# DataRobot docs > Single Markdown file aggregating public English docs for RAG pipelines and deep LLM context. > Generated: 2026-09-02T14:13:35.635112+00:00 | Locale: en | Pages: 1545 > Overview: [https://docs.datarobot.com/llms.txt](https://docs.datarobot.com/llms.txt) Page boundaries below use `---` separators; each page begins with `#` (title) and a `URL:` line. --- # Agent Assist skill URL: https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/agent-assist-skill.html > Install and use the DataRobot Agent Assist skill in third-party coding agents such as Claude Code, Cursor, OpenCode, and VS Code Copilot. The Agent Assist skill ( `datarobot-agent-assist`) packages the same agent design, coding, and deployment workflows as the [dr assist](https://docs.datarobot.com/en/docs/index.html) terminal assistant for use inside third-party coding agents. Instead of running Agent Assist in a dedicated terminal session, you install the skill into your preferred agent to drive the workflow through natural language in that environment. The skill can be found in the [DataRobot Agentic Skills repository](https://github.com/datarobot-oss/datarobot-agent-skills) alongside other DataRobot skills. It guides you through designing an agent, scaffolding from the [Agentic Starter application template](https://github.com/datarobot-community/datarobot-agent-application), implementing tools and code, optionally [testing](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/agent-assist-skill.html#adversarial-swarm-evaluation) the implementation with Adversarial Swarm Evaluation, and deploying to DataRobot. ## Install the skill Install all DataRobot skills (including Agent Assist and the required `datarobot-setup` skill) with the [universal skills installer](https://github.com/skillcreatorai/Ai-Agent-Skills): ``` npx ai-agent-skills install datarobot-oss/datarobot-agent-skills ``` By default, the installer copies skills to all supported coding agents on your machine. To target a specific agent, add the `--agent` flag: ``` npx ai-agent-skills install datarobot-oss/datarobot-agent-skills --agent cursor npx ai-agent-skills install datarobot-oss/datarobot-agent-skills --agent claude ``` You can also install from agent-specific marketplaces or extension catalogs: | Agent | Install surface | | --- | --- | | Claude Code | claude.com/plugins/datarobot-agent-skills | | Cursor | cursor.com/marketplace/datarobot | | Gemini CLI | geminicli.com/extensions | | Skills repository | github.com/datarobot-oss/datarobot-agent-skills | For agent-specific installation details and the full list of supported agents, see [DataRobot agentic skills](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-skills.html). ### Verify installation After installing, ask your coding agent `What DataRobot skills do I have available?`. You should see `datarobot-setup` and `datarobot-agent-assist` listed. The skill source and helper scripts are in [skills/datarobot-agent-assist/](https://github.com/datarobot-oss/datarobot-agent-skills/tree/main/skills/datarobot-agent-assist) in the repository. ## Configure your environment Before designing or coding an agent, run the `datarobot-setup` skill once per workspace. It checks Python and other dependencies, configures your DataRobot API token, and prepares a project directory. There are two ways to trigger setup: - Type: Run datarobot-setup - Or use the slash command: ./datarobot-setup Accept the prompts to complete environment configuration. If setup runs again in a later session, allow it to ensure the prerequisites are current. ## Use Agent Assist in your coding agent Start the Agent Assist workflow with either: - Run datarobot-agent-assist - ./datarobot-agent-assist The skill presents the same three options as `dr assist`: 1. Design an AI agent: Clarify requirements and produce agent_spec.md . 2. Code an AI agent: Scaffold from the agent template and implement the spec. 3. Deploy an AI agent: Deploy the implemented agent to DataRobot. ### Typical workflow The following sequence takes you from idea to deployed agent in one session: 1. Design: Describe what you want to build in plain language. Agent Assist asks clarifying questions and writes an agent_spec.md blueprint before any code is written. 2. Rehearse (optional): Run a simulation to chat with your agent concept as an end user. When satisfied, type Done to move to coding. This design rehearsal is separate from Adversarial Swarm Evaluation , which runs after the agent is implemented. 3. Test locally: SayLet's code it. The skill handles dependencies, tools, and project structure. Watch forLint cleanandTests passedin the output. Run the agent locally with: drrundev Open thelocalhostURL shown in the terminal to verify the agent is running. 4. Test (optional): After coding, choose the option to test your agent. Adversarial Swarm Evaluation runs attack, behavior, and persistence tracks against the implementation, proposes patches for breaches, and writeseval_report.md. SeeAdversarial Swarm Evaluation. 5. Deploy: Enter Deploy my agent to publish it to DataRobot. Deployment typically takes 10 to 30 minutes; keep the session open while it runs. Ask What's the URL for my deployed agent? to retrieve the live URL. For field definitions and examples of `agent_spec.md`, see [Agent specification reference](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/agent-spec-reference.html). For environment variables and configuration used during coding and deployment, see [Environment and commands reference](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html). ## Adversarial Swarm Evaluation After your agent is coded, Agent Assist can automatically test the implementation before you deploy. The coding agent offers it as a next step after implementation; use this section when you want more detail on what it tests and how the run works. The swarm runs three tracks against your implemented agent: | Track | What it probes | | --- | --- | | Attack strategies | Attempts to misuse or exploit the agent's tools and bypass stated restrictions. | | Behavior scenarios | Ambiguous requests, edge cases, and confused-user patterns grounded in your domain. | | Persistence and escalation | Multi-turn pushback to see whether the agent holds its restrictions under sustained pressure. | When a scenario breaches, the skill proposes a minimal fix (usually a system-prompt addition, sometimes a code guard), applies approved changes, and retests. A typical full swarm run takes up to 5 minutes before any fix rounds. ### Prerequisites You need both of the following in the project: - An agent_spec.md with a system_prompt . - Implementation code (for example agent.py , myagent.py , tools.py , or app.py ). If either is missing, finish the Design and Code steps first, then return to battle-testing. ### Run a swarm evaluation 1. Build an agent with Agent Assist in your coding tool (for example DataRobot OpenCode, Claude Code, or Cursor). 2. After the coding step, choose the option to battle-test your agent, or ask in natural language (for example, "Battle-test my agent" or "Run adversarial swarm evaluation"). 3. Answer the configuration prompts. The skill asks for a user persona, optional grounding context (sample queries or real requests), how many fixing rounds to allow (default: 3), evaluation mode ( standard pass/fail or scored by severity), and which model to use. If the spec includes read-only tools, you can optionally let the swarm perform actual calls for those tools instead of simulating their returns. 4. Review the generated scenarios by track. You can add or remove scenarios, ask the skill to explain any of them, then confirm to start the run. 5. Watch the swarm narrate pass, breach, and error outcomes. For each breach, approve or reject the proposed patch; approved prompt changes are applied to both agent_spec.md and the matching system prompt in the implementation so they stay in sync. 6. When convergence finishes, review eval_report.md . The report includes pass/fail outcomes, unresolved or exhausted scenarios, readiness to deploy, and a Changes Applied list of patches made during the run. Intermediate swarm artifacts are written under `.datarobot/swarm/` and removed when the report step completes. Keep `eval_report.md` (and `evaluation_criteria.md` if you want a record of the scenarios) for review before deployment. > [!NOTE] Design rehearsal vs. swarm evaluation > The optional Rehearse step after Design is a chat-style simulation of the spec before code exists. Adversarial Swarm Evaluation runs only against an implemented agent and is meant to harden behavior before deploy. ### After the report From the post-run menu you can: - Review eval_report.md for outcomes and unresolved scenarios. - Re-run the simulation after further changes. - Test locally with the project's usual local-run command. - Deploy the hardened agent to DataRobot. If any scenario is marked exhausted (could not be resolved within the fixing-round limit), address those cases before deploying. The report calls them out explicitly. ## Agent Assist skill vs. dr assist | | dr assist (terminal) | Agent Assist skill (coding agent) | | --- | --- | --- | | Where it runs | DataRobot CLI plugin in the terminal | Inside Claude, Cursor, OpenCode, VS Code Copilot, and other supported agents | | Install | dr plugin install assist | npx ai-agent-skills install datarobot-oss/datarobot-agent-skills | | Start | dr assist | Run or slash-command datarobot-agent-assist | | Workflows | Design, Code, Deploy | Same design, code, and deploy flows, plus design rehearsal and Adversarial Swarm Evaluation after coding | | Best for | Terminal-first development | Teams already working in a coding agent IDE | Both paths use the same `agent_spec.md` format, agent template, and deployment model. Choose the implementation that fits your workflow. Adversarial Swarm Evaluation is available through the Agent Assist skill experience in supported coding agents. --- # Agent specification reference URL: https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/agent-spec-reference.html > Structure and examples for agent_spec.md—the YAML specification Agent Assist creates during design. During the [Design an AI agent](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/workflows-and-prompting.html#design-an-ai-agent) workflow, Agent Assist writes an `agent_spec.md` file in your working directory. The file is YAML and captures what the agent should do before any implementation code exists. You can review it with stakeholders, edit it by hand, or load it in the [Code an AI agent](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/workflows-and-prompting.html#code-an-ai-agent) workflow. This page describes each field. All fields are optional while the spec is still evolving; Agent Assist fills them in as you refine the design. ## Specification fields | Field | Description | | --- | --- | | model | LLM gateway model ID for the coded agent at runtime (for example anthropic/claude-sonnet-4-5-20250929). Browse models with list models. For assist-session LLM settings, see Change the Agent Assist LLM. | | system_prompt | Instructions that define the agent's role, tone, and constraints. | | tools | List of tools the agent can call. Each tool has a function_name, inputs, out, and optionally auth_spec. | | examples | Sample user queries that illustrate intended behavior. | | frontend | UI expectations for the Agentic Starter template (see Frontend options). | ### Tool definition Each entry under `tools` describes one callable function: | Subfield | Description | | --- | --- | | function_name | Name the model uses when requesting the tool. | | inputs | Arguments the tool accepts. Each input has arg_name, type, and optionally object_schema for structured list or dict values. | | out | Values the tool returns. Same structure as inputs. | | auth_spec | Optional. Documents which external service the tool uses and how it authenticates (see Authentication in specs). | Supported `type` values: `str`, `int`, `float`, `bool`, `list`, `dict`. ### Authentication in specs When a tool calls an external API, include `auth_spec` so the design records the integration: ``` auth_spec: service_name: "External API Service" auth_method: api_key ``` | auth_method | Typical use | | --- | --- | | api_key | Static key in a header or query parameter (for example OpenAI, Perplexity). | | oauth2 | User-delegated access with token refresh (for example Salesforce, Google). | | basic_auth | Username and password. | | bearer_token | Static bearer token (for example internal services). | | service_account | Non-human identity with a key file or IAM role (for example GCP, AWS). | | other | Custom or uncommon authentication. | The spec documents what authentication is needed. After you implement the agent, configure actual credentials as runtime parameters in the Agentic Starter template infrastructure code. See the template's `AGENTS.md` for the pattern. ### Frontend options Before simulation or coding, Agent Assist asks whether the default chat UI is enough or you need a custom interface. That choice is stored under `frontend`: | frontend.type | When to use | | --- | --- | | chat | Default single chat window (most agents). | | multi-page | Distinct pages such as dashboards, tabs, or admin views. | | custom | Bespoke layout beyond named pages. | For `multi-page` or `custom`, you can add: - pages —Short descriptions of each page or view. - requirements —Optional free-text UI requirements (theme, charts, filters, and so on). ## Examples ### Simple agent with one tool ``` model: anthropic/claude-sonnet-4-5-20250929 system_prompt: You are a helpful weather assistant. When a user asks about weather, search for current conditions and present them clearly. tools: - function_name: search_weather inputs: - arg_name: location type: str out: - arg_name: search_results type: str auth_spec: service_name: Weather API auth_method: api_key examples: - What's the weather like in New York? - Current conditions in London frontend: type: chat ``` ### Multi-tool agent with authentication ``` model: anthropic/claude-sonnet-4-5-20250929 system_prompt: You are a research assistant. Find and summarize information from internal documents and the web. Always cite your sources. tools: - function_name: search_internal_docs inputs: - arg_name: query type: str out: - arg_name: documents type: list object_schema: "list of {title: str, content: str, url: str}" auth_spec: service_name: Internal Knowledge Base API auth_method: bearer_token - function_name: web_search inputs: - arg_name: query type: str out: - arg_name: results type: str examples: - Find recent papers on LLM hallucination - What does our internal policy say about data retention? frontend: type: chat ``` ### Multi-page dashboard agent ``` model: google/gemini-2.5-pro-preview-05-06 system_prompt: You are a sales analytics assistant. Help users understand pipeline, forecast revenue, and identify at-risk deals. Ground answers in tool data. tools: - function_name: get_pipeline_data inputs: - arg_name: date_range type: str out: - arg_name: deals type: list auth_spec: service_name: Salesforce CRM auth_method: oauth2 examples: - What's our Q2 pipeline look like? - Which deals are at risk of slipping? frontend: type: multi-page pages: - "Pipeline Overview - deals by stage with filtering" - "Revenue Forecast - expected vs actual with confidence bands" requirements: "Dark theme charts. Pipeline table sortable by owner and stage." ``` --- # Environment and commands reference URL: https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html > Environment variables, LLM model selection, files and directories, the list models command, and slash commands for DataRobot Agent Assist. This page provides a quick reference for environment variables, [how the assist LLM is chosen](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#change-the-agent-assist-llm), important files and directories, the `list models` prompt command, and slash commands. If you use Agent Assist through a third-party coding agent instead of `dr assist`, install the skill with: ``` npx ai-agent-skills install datarobot-oss/datarobot-agent-skills ``` See the [Agent Assist skill page](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/agent-assist-skill.html) for installation, setup, and usage guidelines. ## Environment variables The variables that can be set in the environment or in `.env`. | Variable | Required | Description | Default / notes | | --- | --- | --- | --- | | DATAROBOT_API_TOKEN | Yes (unless provided by DR CLI configuration) | DataRobot API key for LLM gateway (default provider). | — | | DATAROBOT_ENDPOINT | No | DataRobot API endpoint. | https://app.datarobot.com/api/v2 | | AGENT_ASSIST_LLM_BASE_URL | No | Base URL for the assist session Completions API. See Change the Agent Assist LLM. | Derived from DATAROBOT_ENDPOINT (for example, …/genai/llmgw) | | AGENT_ASSIST_LLM_MODEL_NAME | No | Model for the assist session. See Change the Agent Assist LLM. | anthropic/claude-sonnet-4-5-20250929 | | AGENT_ASSIST_LLM_API_KEY | No | API key for an external Completions API provider. | Falls back to DATAROBOT_API_TOKEN if unset—even for an external provider. See the warning below. | | AGENT_ASSIST_DISABLE_LLM_GATEWAY | No | Skip the LLM gateway catalog fetch and route the assist session directly to the model named by AGENT_ASSIST_LLM_MODEL_NAME (for example, a DataRobot deployment). Requires AGENT_ASSIST_LLM_BASE_URL and a non-default AGENT_ASSIST_LLM_MODEL_NAME to also be set. | Unset (LLM gateway catalog used) | | LOGFIRE_TOKEN | No | Logfire token for tracing. | — | | DATAROBOT_CLI_CONFIG | No | Override path to DR CLI configuration file. | Default: ~/.config/datarobot/drconfig.yaml | | AGENT_ASSIST_CONFIG | No | Override path to Agent Assist configuration file. | Default: ~/.config/datarobot/agent_assist_config.yaml | > [!TIP] Also read by the OpenCode plugin > The `AGENT_ASSIST_LLM_*` and `AGENT_ASSIST_DISABLE_LLM_GATEWAY` variables above are also read by the [DataRobot OpenCode plugin](https://docs.datarobot.com/en/docs/agentic-ai/cli/opencode-plugin.html) ( `dr opencode`). If they're already set from an Agent Assist setup, OpenCode routes to that same external model automatically—no reconfiguration needed. > [!WARNING] Set AGENT_ASSIST_LLM_API_KEY explicitly for external providers > The fallback to `DATAROBOT_API_TOKEN` applies regardless of what `AGENT_ASSIST_LLM_BASE_URL` is set to. If you configure [an external LLM](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#assist-session-llm) (a non-DataRobot `AGENT_ASSIST_LLM_BASE_URL`) and leave `AGENT_ASSIST_LLM_API_KEY` unset, Agent Assist sends your `DATAROBOT_API_TOKEN` to that external endpoint as the API key instead of failing. Always set `AGENT_ASSIST_LLM_API_KEY` explicitly whenever `AGENT_ASSIST_LLM_BASE_URL` is not the default. ## Files and directories The paths and files used or created by DataRobot Agent Assist. All paths are relative to the current working directory when you start `dr assist`. > [!WARNING] Run dr assist in an empty directory > Only run `dr assist` from a dedicated and empty directory. Running this command in a directory containing code or other files is unsafe. When you use the agent assist coding workflow, the assistant clones the DataRobot Agent Application Template repository into the current directory. This action can overwrite or conflict with existing files, damaging the existing project and degrading the accuracy of the assistant's output. Before running `dr assist`, if you're not in a dedicated directory, create one and open the terminal there (for example, `mkdir my-agent && cd my-agent`, then run `dr assist`). | Path | Description | | --- | --- | | agent_spec.md | Agent specification file (YAML content) in the current working directory; written by the assistant during design. See Agent specification reference for field definitions and examples. | | .env | Optional environment file in the current directory; same variable names as above. | | .datarobot/cli/versions.yaml | Used by dependency check; defines minimum tool versions. | | ~/.config/datarobot/drconfig.yaml | DR CLI configuration (token, endpoint). This path can be overridden with DATAROBOT_CLI_CONFIG. | | ~/.config/datarobot/agent_assist_config.yaml | Agent Assist configuration (optional LLM base URL, model name, API key). This path can be overridden with AGENT_ASSIST_CONFIG. | | ~/.config/datarobot/agent_assist/settings.yaml | Global DataRobot CLI / Agent Assist preferences (for example language). Does not set the assist LLM model; use AGENT_ASSIST_LLM_MODEL_NAME in the environment or .env. For more information, see Change the Agent Assist LLM. | | config.yaml | Optional repository-level configuration; can set repository (url, branch, target_dir, tag) for the template clone. | ## list models command At the `$` prompt, type `list models` (two words, no leading slash) to print the Available LLM Gateway Models list from your [DataRobot LLM gateway](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/dr-llm-gateway.html). Agent Assist shows a table with a row index, Model Name (for example `azure/gpt-5-1-2025-11-13` or `bedrock/anthropic.claude-sonnet-4-5-20250929-v1:0`), Provider (such as Anthropic, Azure OpenAI, Amazon Bedrock, Google Vertex AI, or Together AI), and a short Description. After the table, the output includes how many models are listed, a reminder of which model the session is configured to use, and a prompt to either change the model or continue the conversation. The exact models and providers depend on your organization’s gateway configuration and entitlements. Which model the session uses is determined as described in [Change the Agent Assist LLM](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#change-the-agent-assist-llm). Apart from `list models` and the [slash commands](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#slash-commands) below, use natural language at the `$` prompt for design, coding, and deployment tasks. ## Slash commands Built-in session commands that start with `/` at the `$` prompt. Typing / alone lists these commands and suggests `/help` for details.Simulation session commands apply only inside [simulation](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/workflows-and-prompting.html#simulate-an-agent), not at the main `$` prompt. | Command | Alias | Description | | --- | --- | --- | | Core commands | | | | /help | /? | List commands or show help for a command: /help [command]. | | /quit | /exit | Exit the session (with confirmation). | | Session commands | | | | /reset | /new | Clear session state and start fresh (removes agent_spec.md and template directory, clears conversation). | | /checkpoint | - | List or restore session checkpoints. | | /compact | - | Compact conversation history to reduce token usage. | | /resume | - | Show the resume context injected into the current session. | | /save | - | Save session state to disk. | | Simulation session commands | | | | /quit | /exit | Exit the simulation and return to the main session (no confirmation prompt). | | /f | - | Record feedback about the design: /f . Compiled into a summary when the simulation ends, which the assistant uses to help refine agent_spec.md. | ## Change the Agent Assist LLM Agent Assist uses two LLMs: - Assist session : For design, simulation, and coding conversations. Set AGENT_ASSIST_LLM_* variables to use the DataRobot LLM gateway or an external Completions API. - Coded agent : For the application Agent Assist generates. Set the model field in agent_spec.md to an LLM gateway model ID on your deployment. External AGENT_ASSIST_LLM_* settings do not apply to the coded agent. ### Assist session Set `AGENT_ASSIST_LLM_MODEL_NAME` in the environment or in a `.env` file in the directory where you run `dr assist`. When more than one value is set, Agent Assist uses the first match in this order: 1. Environment variable in the shell. 2. .env in the current working directory. 3. Built-in default: anthropic/claude-sonnet-4-5-20250929 . LLM gateway (default).Leave `AGENT_ASSIST_LLM_BASE_URL` unset. Set `AGENT_ASSIST_LLM_MODEL_NAME` to a model ID from [list models](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#list-models-command) (for example `anthropic/claude-sonnet-4-5-20250929`). **If the default model isn't in your gateway catalog** The built-in default ( `anthropic/claude-sonnet-4-5-20250929`) may not be enabled for your organization's [LLM gateway](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/dr-llm-gateway.html). If `dr assist` can't reach that model, check which models your organization has enabled from a terminal—no running session required—using the DataRobot CLI: ``` dr llm list ``` Copy a value from the MODEL column (for example `azure/gpt-5-1-2025-11-13`), then set it before you start `dr assist`: ``` export AGENT_ASSIST_LLM_MODEL_NAME=azure/gpt-5-1-2025-11-13 ``` `dr llm list` is a shorthand alias for `dr llm-gateway list`. It lists the same active gateway catalog as the in-session [list models](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#list-models-command) command, plus any DataRobot-deployed LLMs available to your account. See the [llm-gatewaycommand reference](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/llm-gateway.html) for the full flag and output reference. If a model isn't in the catalog at all, an org admin must enable it; see [LLM availability](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html#llm-availability). **Use an external LLM for Agent Assist** Set `AGENT_ASSIST_LLM_BASE_URL`, `AGENT_ASSIST_LLM_MODEL_NAME`, and `AGENT_ASSIST_LLM_API_KEY` for any provider that exposes an OpenAI-compatible chat completions endpoint. Set all three—if you omit `AGENT_ASSIST_LLM_API_KEY`, Agent Assist falls back to sending your `DATAROBOT_API_TOKEN` to this external endpoint instead of failing. Add the variables to `.env` or export them before you run `dr assist`: ``` # Example: Anthropic AGENT_ASSIST_LLM_MODEL_NAME=claude-sonnet-4-5 AGENT_ASSIST_LLM_BASE_URL=https://api.anthropic.com/v1 AGENT_ASSIST_LLM_API_KEY=YOUR_ANTHROPIC_API_KEY ``` Use model names and URLs from the provider documentation. The `list models` command lists gateway models only. ### Coded agent The `model` field in `agent_spec.md` must be an LLM gateway model ID. Gateway models must be available in your organization's LLM gateway before you run the [Code an AI agent](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/workflows-and-prompting.html#code-an-ai-agent) workflow. See the [Agent specification reference](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/agent-spec-reference.html#specification-fields). **Model settings** Consider the following when changing agent assist LLM settings: - ~/.config/datarobot/agent_assist/settings.yaml stores CLI preferences (for example language). It does not set AGENT_ASSIST_LLM_MODEL_NAME . - Restart dr assist after you change AGENT_ASSIST_LLM_* variables. There is no in-session /model command. - Credential precedence is described in Set configuration variables . --- # Agent Assist URL: https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/index.html > Agent Assist (dr-assist) is an interactive AI assistant that helps users design, code, and deploy AI agents through natural conversation. > [!NOTE] Premium > DataRobot's Agentic AI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. DataRobot Agent Assist ( `dr-assist`) is an interactive AI assistant optimized for the development of AI agents. It helps users design, code, and deploy agents through natural conversation—users describe the agent they want, and the assistant helps build it on the foundation provided by the [Agentic Starter application template](https://github.com/datarobot-community/datarobot-agent-application). You can use Agent Assist in two ways: - Terminal (dr assist) : Run Agent Assist using the DataRobot CLI . See Prerequisites and installation . - Coding agent (Agent Assist skill) : Install the Agent Assist skill in a third-party coding agent such as Claude Code, Cursor, OpenCode, or VS Code Copilot. See the Agent Assist skill page for setup, verification, and usage in your coding agent. DataRobot Agent Assist integrates with the [DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/index.html) as a plugin. The assist session and the coded agent use separate LLM settings; see [Change the Agent Assist LLM](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#change-the-agent-assist-llm). During the design and code cycle, Agent Assist can outline which tools an agent should call based on the proposed functionality—for straightforward tools, it can implement the tool code; for more complex tools (such as those that consume API tokens or write to a database), it can scaffold the initial file structure for the human-in-the-loop to complete in the editor or development environment of their choice. > [!WARNING] Run dr assist in an empty directory > Only run `dr assist` from a dedicated and empty directory. Running this command in a directory containing code or other files is unsafe. When you use the agent assist coding workflow, the assistant clones the DataRobot Agent Application Template repository into the current directory. This action can overwrite or conflict with existing files, damaging the existing project and degrading the accuracy of the assistant's output. Before running `dr assist`, if you're not in a dedicated directory, create one and open the terminal there (for example, `mkdir my-agent && cd my-agent`, then run `dr assist`). Unlike a general-purpose coding assistant, Agent Assist emphasizes an agent lifecycle: a structured specification ( `agent_spec.md`), simulation before implementation, scaffolding from the Agentic Starter application template, and deployment guidance grounded in that template. DataRobot Agent Assist can: - Design AI agents by helping users think through specifications, ask clarifying questions, and produce an agent specification file ( agent_spec.md ). - Research solutions using file search and analysis (an internal agent can read files, list directories, grep, and glob). - Code AI agents by loading an existing agent_spec.md , cloning the DataRobot agent template repository, and implementing the agent with file edits and shell commands. - Simulate an agent from a specification before coding. In this simulation , the model chooses tools and arguments, but tool calls are not executed. Returns are generated by the LLM so you can validate design (which tools, I/O shapes, model behavior) without calling real deployments or datasets. - Deploy agents to DataRobot following the template’s deployment instructions. | Page | Description | | --- | --- | | Agent Assist skill | Install and use Agent Assist in Claude Code, Cursor, OpenCode, VS Code Copilot, and other supported coding agents. | | Prerequisites and installation | System requirements, required tools and versions, installing the plugin or running standalone, verifying installation. | | Workflows and prompting | Welcome screen, slash commands, Design / Code / Deploy workflows, prompting tips. | | Session persistence | Session persistence details, context injection, session lifecycle details, file drift detection, session reset, session save, session exit (autosave), session slash commands. | | Agent specification reference | Fields, examples, and conventions for agent_spec.md. | | Environment and commands reference | Environment variables table, files and directories, slash commands. | | Troubleshooting | Plugin not discovered, dependency check failed, authentication errors, template bootstrap, LLM API errors, session interruption, and related fixes. | --- # Prerequisites and installation URL: https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/installation.html > System requirements, required tools and versions, and how to install and run DataRobot Agent Assist (plugin or standalone). This page covers system requirements, prerequisite tools, installation, and configuration for DataRobot Agent Assist. ## System requirements Ensure your system meets the minimum requirements for running DataRobot Agent Assist. - Operating system: macOS or Linux (Windows requires WSL or another supported environment) - Python: 3.10 or higher > [!WARNING] Operating system compatibility > This repository is only compatible with macOS and Linux. On Windows, use a [DataRobot codespace](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html), [Windows Subsystem for Linux (WSL)](https://learn.microsoft.com/en-us/windows/wsl/install), or a virtual machine running a supported OS. ## Prerequisite tools At startup, DataRobot Agent Assist runs a dependency check via the DataRobot CLI and a `.datarobot/cli/versions.yaml` file. The following tools must be installed at the versions indicated. > [!TIP] Install tools system-wide > Install tools system-wide so they are available in all terminal sessions and when the DR CLI runs `dr dependencies check`. | Tool | Version | Description | Installation | | --- | --- | --- | --- | | dr-cli | >= 0.2.5 | The DataRobot CLI. | dr-cli installation | | git | >= 2.30.0 | Version control. | git installation | | uv | >= 0.9.0 | Python package manager. | uv installation | | Pulumi | >= 3.163.0 | Infrastructure as Code. | Pulumi installation | | Taskfile | >= 3.43.3 | Task runner. | Taskfile installation | | Node.js | >= 24 | JavaScript runtime (for example, for template frontend). | Node.js installation | > [!TIP] macOS installation > On macOS, several tools can be installed at once: > > ``` > brew install datarobot-oss/taps/dr-cli uv pulumi/tap/pulumi go-task node git python > ``` ## Install and run DataRobot Agent Assist DataRobot Agent Assist can be run as a DataRobot CLI plugin. Install the plugin so that `dr assist` is available wherever the DataRobot CLI is installed. The plugin is discovered when the `dr-assist` executable is on the `PATH` and responds to `--dr-plugin-manifest`. For the plugin published to the CLI plugin index, run: ``` dr plugin install assist ``` To verify the installation, run the following commands: ``` dr plugin list # Should show "assist" when installed as plugin dr assist --help # Show commands ``` The first time you run `dr assist`, it prompts you to select your DataRobot environment. Select the environment that corresponds to the DataRobot instance you want to use. ``` # Output: DataRobot URL configuration dr assist --help 🔌 Running plugin: assist WARN No DataRobot URL configured. Running auth setup... 🌐 DataRobot URL Configuration Choose your DataRobot environment: ┌────────────────────────────────────────────────────────┐ │ [1] 🇺🇸 US Cloud https://app.datarobot.com │ │ [2] 🇪🇺 EU Cloud https://app.eu.datarobot.com │ │ [3] 🇯🇵 Japan Cloud https://app.jp.datarobot.com │ │ 🏢 Custom Enter your custom URL │ └────────────────────────────────────────────────────────┘ 🔗 Don't know which one? Check your DataRobot login page URL in your browser. Enter your choice: ``` If an API key isn't provided, the authentication flow opens the DataRobot application in a browser window. In the Access request: DataRobot CLI dialog box, set a session duration, then click Proceed. > [!WARNING] Run dr assist in an empty directory > Only run `dr assist` from a dedicated and empty directory. Running this command in a directory containing code or other files is unsafe. When you use the agent assist coding workflow, the assistant clones the DataRobot Agent Application Template repository into the current directory. This action can overwrite or conflict with existing files, damaging the existing project and degrading the accuracy of the assistant's output. Before running `dr assist`, if you're not in a dedicated directory, create one and open the terminal there (for example, `mkdir my-agent && cd my-agent`, then run `dr assist`). ## Set configuration variables If necessary, set the following variables in the environment, in `.env`, or in the configuration files. In many cases, this configuration is handled by [the DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/auth.html) through `dr auth login`. | Variable | Required | Description | Default / notes | | --- | --- | --- | --- | | DATAROBOT_API_TOKEN | Yes (unless provided by DR CLI configuration) | DataRobot API key for LLM gateway (default provider). | — | | DATAROBOT_ENDPOINT | No | DataRobot API endpoint. | https://app.datarobot.com/api/v2 | For the full list of variables (including LLM overrides, logging, and config path overrides), see the [Environment and commands reference](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#environment-variables). For details on how Agent Assist resolves `AGENT_ASSIST_LLM_MODEL_NAME`, which files do not control the assist model, and when you must restart `dr assist`, see [Choose and change the LLM model](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#change-the-agent-assist-llm). **Configuration variable priority** DataRobot Agent Assist reads configuration from multiple sources; to do this, it looks for credentials and settings in this order: 1. Environment variables : DATAROBOT_API_TOKEN , DATAROBOT_ENDPOINT , and LLM-related variables such as AGENT_ASSIST_LLM_MODEL_NAME 2. .env file : In the current working directory (where you run dr assist ); same variable names as above 3. DR CLI configuration : ~/.config/datarobot/drconfig.yaml (token and endpoint). Override path with DATAROBOT_CLI_CONFIG 4. Agent Assist configuration : ~/.config/datarobot/agent_assist_config.yaml (optional LLM URL, model, API key). Override path with AGENT_ASSIST_CONFIG `~/.config/datarobot/agent_assist/settings.yaml` is separate: it stores preferences such as language for the CLI and does not set `AGENT_ASSIST_LLM_MODEL_NAME`. Do not rely on that file to choose the assist LLM. For the format and behavior of the DataRobot CLI configuration file ( `drconfig.yaml`), including token and endpoint, see the [DataRobot CLI documentation](https://docs.datarobot.com/en/docs/agentic-ai/cli/configuration.html). ## Startup behavior Before the chat session starts, DataRobot Agent Assist: 1. Verifies the DataRobot CLI is available and runs dr dependencies check (using .datarobot/cli/versions.yaml if present). 2. Checks authentication and can run dr auth login if needed. 3. Ensures DATAROBOT_API_TOKEN (or equivalent from configuration) is set; if not, it displays an error and exits. --- # Session persistence URL: https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/session-management.html > Session persistence details, context injection, session lifecycle details, file drift detection, session reset, session save, session exit (autosave), session slash commands. The assistant automatically saves the session state on exit and offers to resume on next launch. This enables continuity across sessions—the assistant knows what files existed and if they changed, allowing it to pick up where the previous session left off. ## Agent Assist memory The LLM itself does not remember the conversation. There is no server-side chat or conversation ID to preserve. The architecture is stateless from the LLM's perspective: - pydantic-ai + OpenAI-compatible API: The agent usesOpenAIChatModelwhich communicates with the OpenAI Chat Completions API through the DataRobot LLM Gateway. This API is stateless—there is no server-side conversation thread. Every request sends the full message history in the request body, and the server has no memory of prior turns. - message_history: This is alist[ModelMessage]maintained entirely on the client side. Each call toagent.iter(user_input, message_history=message_history)sends the full history to the LLM. When the response arrives,agent_run.all_messages()returns the updated list, including the new turn. The server doesn't track or account for a "chat or conversation ID" during this process. This means: - When the process exits, the message_history list ceases to exist and the LLM doesn't remember the conversation. - Replaying the full raw history when the conversation resumes would be expensive and fragile due to token limits, stale tool results, etc. - Session continuity is achieved through compressed context injection —a summary of the conversation's actions and decisions is injected into the system prompt when the conversation resumes. ## Session persistence details Session persistence saves conversation history in compacted form: a recent-messages tail plus a summary of older turns, alongside lightweight metadata. On resume, the assistant reconstructs a trimmed `message_history` from this (via `reconstruct_history()`) and also injects a resume-context block into the system prompt. The lightweight metadata it tracks: | Artifact | Purpose | | --- | --- | | Session ID and timestamps | Identity and recency—"when did we last talk?". | | agent_spec.md path | Whether the user was designing an agent. | | File manifest (SHA256 hashes) | Detect if files changed while the assistant was offline. Files not in the manifest are not detected (that would require a full workspace scan). | | _session_deleted | Whether the session was deleted by the user. | ## Context injection When the agent conversation resumes, `build_resume_context()` constructs a text block from the previously saved session state and injects it into the agent's system prompt using pydantic-ai's `@agent.system_prompt(dynamic=True)` decorator. This decorator re-evaluates on every agent run, so: - On resume: the agent sees session context (ID, timestamps, file changes, summary, decisions). - After /reset : the dynamic prompt returns "" and the agent sees nothing. - During normal operation: the prompt returns whatever was set at launch. This block serves as invisible system prompt content that gives the agent awareness of prior sessions. ## Session lifecycle details When an exited session resumes, a block similar to the following is injected into the system prompt (invisible to the user): ``` # Resumed Session Context This is a resumed session (ID: a1b2c3d4e5f6). Originally started: 2026-03-30 10:15 UTC Last active: 2026-03-31 14:22 UTC Agent specification file: agent_spec.md ## Previous Session Summary User designed a customer support agent with three tools. Model selection was GPT-5 via the DataRobot model catalog. Spec was saved but not yet implemented in code. ## Key Decisions Made - Selected GPT-5 as the agent model for strong reasoning capability - Chose REST API tools over SDK wrappers for portability - Deferred authentication configuration to implementation phase ## File Changes Since Last Session - agent_spec.md: modified IMPORTANT: You are resuming a previous session. Review the context above before proceeding. If the user's request seems to continue prior work, use this context. If they start a new topic, proceed normally. ``` After running `/reset` ( `/new`), this block disappears—the dynamic prompt returns an empty string. ### Fresh launch (no prior session) The first time the agent launches, when there is no prior session: 1. On launch, SessionState() creates a new session with a random 12-character hex ID. 2. The user interacts with the agent. 3. On exit, SessionLifecycle.snapshot_and_save() captures the current file manifest, then SessionManager.save() writes the session file and active_{pid}.json pointer ### Resume (existing session) When the agent resumes a previous session: 1. On launch, SessionManager.load() reads active_{pid}.json and loads the indicated session file. 2. The user is prompted to resume the previous session: Previous session found (last active: 2026-03-31 14:22 UTC) Resume previous session? [Y/n] 3. Yes : The existing SessionState is carried forward (same ID, same file on disk). build_resume_context() constructs the system prompt block with session info and drift detection results. 4. No : A new SessionState() is created. The old session file remains on disk as an orphan (small, <1KB). The new session's autosave on exit updates active_{pid}.json to point to the new ID. 5. The constructed system prompt is injected into the agent using pydantic-ai's @agent.system_prompt(dynamic=True) . ### Session reset When the user runs `/reset` ( `/new`): 1. If agent_spec.md or template directory exist, the user is prompted for confirmation before deletion. 2. Even if no files exist, /reset always: 3. Clears in-memory conversation history ( message_history = None ). 4. Creates a fresh SessionState() with a new ID. 5. Deletes the persisted session file and active_{pid}.json pointer. 6. Clears the dynamic resume context from the system prompt. 7. Sets _session_deleted = True to prevent autosave resurrection on exit. 8. If the user continues interacting after /reset , the first successful agent response re-enables autosave ( _session_deleted = False ) so the new session is preserved. ### Session save When the user runs `/save`: 1. SessionLifecycle.explicit_save() updates agent_spec_path and rebuilds file_manifest from current workspace 2. SessionManager.save() writes the session file and updates active_{pid}.json 3. Re-enables autosave if it was disabled by a prior /reset ### Session exit (autosave) When the agent exits due to normal shutdown, Ctrl+C, or crash: 1. The finally block in chat() runs on quit, Ctrl+C, or crash. 2. If _session_deleted is True (set by /reset with no subsequent interaction), autosave is skipped 3. Otherwise, snapshots workspace state and persists. ## Session slash commands The assistant supports the following slash commands for session management: | Command | Description | | --- | --- | | /save | Checkpoint the session immediately. Snapshots the current agent_spec.md state and file manifest before writing. | | /reset (/new) | Always clears conversation history, deletes the saved session, and removes resume context from the system prompt. If agent_spec.md or the template directory exist, prompts for confirmation before deleting them. The next agent turn starts completely clean. | | /checkpoint | List session checkpoints, or restore one by number (/checkpoint [number]). | | /compact | Compact conversation history to reduce token usage. | | /resume | Show the resume context currently injected into the session's system prompt. | ## File drift detection On resume, the assistant compares SHA256 hashes of tracked files against the saved manifest. This detects three cases: - Unchanged : The hash matches; reported as a count ("1 tracked file(s) unchanged"). - Modified : The file exists but the hash differs; listed as : modified . - Deleted : The file was in manifest but no longer exists; listed as : deleted . > [!NOTE] File drift detection > New files not in the manifest are not detected (that would require a full workspace scan). This allows the assistant to alert the user (or adjust its own behavior) when workspace files were changed outside the assistant between sessions. ## Component map The following components are responsible for session management: | Component | Location | Responsibility | | --- | --- | --- | | SessionState | session/models.py | Pydantic model—schema for persisted state. | | SessionManager | session/manager.py | CRUD operations on session JSON files. | | SessionLifecycle | session/lifecycle.py | Owns session state, message history, resume context, and deletion flag; enforces state transition invariants; builds resume context for system prompt injection. | | Filesystem helpers | helpers/filesystem.py | SHA256 hashing, path validation, manifest building, drift detection. | | Path helpers | helpers/paths.py | Path-to-directory-name resolution for project settings. | | Confirmation UI | ui/confirm.py | Destructive action confirmation dialog (isolated to break circular import). | | Main loop | main.py | Lifecycle orchestration (resume prompt, delegates to SessionLifecycle). | | Command handlers | commands/handlers.py | /save and /reset session operations. | | CommandContext | commands/base.py | Carries session references to command handlers. | ## Storage layout The session state is stored in the project configuration directory at `~/.config/datarobot/agent_assist/projects/_/sessions/`. ``` ~/.config/datarobot/agent_assist/projects/ my-project_a1b2c3d4/ sessions/ a1b2c3d4e5f6.json # per-session state file active_.json # pointer to current session (one per terminal PID) ``` Each session file contains the following fields: | Field | Description | | --- | --- | | schema_version | Schema version for forward compatibility (currently 3). | | session_id | 12-character hex identifier. | | created_at | UTC timestamp of session creation. | | updated_at | UTC timestamp of last save. | | agent_spec_path | Relative path to agent spec, or null. | | file_manifest | {relative_path: sha256_hex} for tracked files. | | summary | LLM-extracted narrative summary of the prior session, or null. | | decisions | Key decisions extracted from the prior session. | | recent_messages | Serialized recent conversation messages (the compaction tail) reconstructed on resume. | | compacted_history_summary | Summary of older messages compacted away, or null. | | checkpoint_count | Most recent checkpoint number written for the session (informational, not a live file count). | | phase | Current conversation phase (design / code / deploy), or null. | ## Security - Path traversal protection : All manifest paths are validated via is_safe_relative_path() (in helpers/filesystem.py ) to resolve within the workspace directory before any file I/O. Tampered manifests containing ../ sequences are silently skipped. - Symlink rejection : build_file_manifest() and detect_file_drift() skip symlinks as defense-in-depth against time-of-check-to-time-of-use (TOCTOU) symlink attacks. A file replaced by a symlink between sessions is reported as deleted. - Session ID validation : Session IDs are validated against a strict ^[a-f0-9]{12}$ regex before constructing file paths, preventing path injection via crafted pointer files. - No secrets in session files : Session state contains only metadata (timestamps, hashes, paths). API keys, conversation content, and user input are not persisted. --- # Troubleshooting URL: https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/troubleshooting.html > Plugin not discovered, dependency check failed, authentication errors, and related fixes. This page describes how to resolve common issues with DataRobot Agent Assist, including plugin discovery, dependency checks, authentication and API keys, template bootstrap failures, LLM API errors, and session interruption. Follow the sections below for step-by-step fixes. ## Plugin not discovered If `dr assist` does not run or the plugin does not appear in `dr plugin list`: Check that `dr-assist` is on the PATH ``` which dr-assist ``` If it is not found, reinstall the plugin and ensure the directory containing `dr-assist` is on the `PATH`. Check manifest output The CLI discovers plugins by running the executable with `--dr-plugin-manifest` and reading JSON from stdout. ``` dr-assist --dr-plugin-manifest ``` This should print valid JSON. If it hangs or errors, the plugin may not start correctly. Check manifest speed The manifest must respond within 100 ms for reliable discovery; the CLI uses a 500 ms per-manifest timeout. ``` time dr-assist --dr-plugin-manifest ``` If it is slow, look for heavy imports or startup work before the manifest is printed. Debug with the CLI ``` dr --debug plugin list ``` Use this to see why a plugin might not be listed. Common causes: The manifest responds in over 500 ms, the executable lacks execute permission ( `chmod +x`), or the name conflicts with a built-in command. ## Dependency check failed At startup, DataRobot Agent Assist runs `dr dependencies check`, which uses `.datarobot/cli/versions.yaml`. If the file does not exist, the application creates it with default minimum versions (Node 24, git 2.30, task 3.43.3, pulumi 3.163.0). - Error panel:If the check fails, the application shows a "Dependency Check Failed" panel and prints the dependency error output. - What to do:Install or upgrade the missing tools to at least the versions in the table inPrerequisites and installation. Ensure each tool is on thePATHand reports at least the minimum version (for example,git --version,task --version). Fix any issues reported in the panel (for example, wrong executable name or path). ## Authentication / API key errors DataRobot Agent Assist requires a valid DataRobot API token. Use the following for missing or invalid credentials. Missing API key If `DATAROBOT_API_TOKEN` (or equivalent from configuration) is not set, the application shows a "Configuration Error" panel before starting the chat. Do one of the following: - Set the environment variable: export DATAROBOT_API_TOKEN='your-api-key-here' - Create or update a .env file in the current directory with DATAROBOT_API_TOKEN=... - Get the API key from the URL returned by get_api_key_url() (for example, DataRobot account profile). Invalid or expired key On authentication failure (for example, OpenAI `AuthenticationError`), the application shows an "Authentication Error" panel. Verify the key, run `echo $DATAROBOT_API_TOKEN`, update `.env` if needed, and fetch a new key from the provider URL. Using the DataRobot CLI for authentication Log in with the DataRobot CLI; the token is stored in the DR CLI configuration and reloaded by DataRobot Agent Assist: ``` dr auth login ``` If login fails, the application prints "DataRobot CLI authentication failed" and suggests running `dr auth login` manually. ## Template bootstrap failures > [!WARNING] Run dr assist in an empty directory > Only run `dr assist` from a dedicated and empty directory. Running this command in a directory containing code or other files is unsafe. When you use the agent assist coding workflow, the assistant clones the DataRobot Agent Application Template repository into the current directory. This action can overwrite or conflict with existing files, damaging the existing project and degrading the accuracy of the assistant's output. Before running `dr assist`, if you're not in a dedicated directory, create one and open the terminal there (for example, `mkdir my-agent && cd my-agent`, then run `dr assist`). If you see unexpected files, overwrites, or odd behavior after a clone, you may have started in a non-empty directory. If a git clone fails during the coding workflow (for example, when cloning the DataRobot agent template): Verify git is installed and on the PATH: ``` git --version ``` Check network access to the repository: ``` git ls-remote https://github.com/datarobot-community/datarobot-agent-application.git ``` If using SSH, verify SSH keys are configured: ``` ssh -T git@github.com ``` ## LLM API errors For timeouts, rate limiting, or model-not-available errors: - Check DataRobot service status. - Verify your account has LLM gateway access. - If the selected model is unavailable, run dr llm list to see which models your organization's LLM gateway has enabled, then try a different one by setting AGENT_ASSIST_LLM_MODEL_NAME (or .env ) as described in Change the Agent Assist LLM . - Start a new dr assist session to reload the settings. - For timeouts, the assistant prompts you to retry. ## Session interruption If the session ends unexpectedly: - Session state is saved as lightweight metadata that lets the assistant reconstruct enough context to be useful. - The agent_spec.md file is saved to disk and preserved. - Template directory contents are preserved. - Restart the session to continue. ## Getting help For more assistance see the following related documentation: - DataRobot CLI: For CLI installation, configuration file format, and authentication flow: DataRobot CLI documentation . - Plugin build and CI: See the plugin README in the repository for distribution, build, and troubleshooting. --- # Workflows and prompting URL: https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/workflows-and-prompting.html > Welcome screen, list models, slash commands, Design / Code / Deploy workflows, agent_spec.md, simulation, and prompting tips. This page describes the interactive session: the welcome screen, the `list models` command, slash commands, and the three main workflows (Design, Code, Deploy), plus prompting tips. ## Start Agent Assist > [!WARNING] Run dr assist in an empty directory > Only run `dr assist` from a dedicated and empty directory. Running this command in a directory containing code or other files is unsafe. When you use the agent assist coding workflow, the assistant clones the DataRobot Agent Application Template repository into the current directory. This action can overwrite or conflict with existing files, damaging the existing project and degrading the accuracy of the assistant's output. Before running `dr assist`, if you're not in a dedicated directory, create one and open the terminal there (for example, `mkdir my-agent && cd my-agent`, then run `dr assist`). To begin, run the `dr assist` command: ``` dr assist ``` When DataRobot Agent Assist is started, the welcome screen shows the welcome message, three options, and the help footer: ``` # Output: Welcome screen $ dr assist 🔌 Running plugin: assist █████████ █████ ███████ ████████ ██ █████████ ██ ███ ██ ██ ██ ██ ██ █████ ██ ██ █████ █████ █████ ██ ██ █████ ███████ █████ █████ ██████████████ ██ ███ ██ ██ ██ █████████ ██ ██ ██ ██ ██ ██ ██ █████ ██ ██ ███████ ██ ███████ ██ ██ ██ ██ ██ ██ ██ ██ ██ █████████ ██ ████ ██ ██ ██ ██ ██ ██ ███ ██ ██ ██ ██ ██ ██ ██ █████ ████████ ████████ ████ ████████ ██ ███ █████ ████████ █████ ████ █████████ Welcome! I help you design, code, and deploy AI agents. What would you like to do? 1. Design an AI agent → Describe your idea 2. Code an AI agent → Load and implement an existing agent_spec.md 3. Deploy an AI agent → Deploy an implemented agent to DataRobot Type /help for commands | /quit to exit ``` Input is collected at the `$` prompt. Type a number (1, 2, or 3) to choose a category, or, describe the goal in natural language. If the first instruction is one of those numbers, the assistant treats it as referring to the predefined categories. > [!TIP] Built-in commands > Type `list models` at the `$` prompt (no slash) to see the LLM Gateway model catalog, the model your session uses, and whether to switch models or continue. See the [Environment and commands reference](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#list-models-command) for details. Slash commands use the `/` prefix; for the full list and descriptions, see [Slash commands](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#slash-commands). Typing / shows available slash commands and suggests `/help` for details. The `/quit` command exits the session. ### Design an AI agent Use this workflow when starting a new agent idea. The assistant asks clarifying questions about the agent’s purpose, capabilities, and requirements—including business or domain concerns (for example escalation rules, user scenarios, or tone) as well as technical details. It suggests tools and models, and produces an agent specification. The specification is displayed and written to `agent_spec.md` in the current directory (YAML format). A complete spec typically captures what you need to implement and review before code exists: the agent’s behavior and prompts, model choice, tool definitions, and how authentication and integrations should work, so both developers and stakeholders can align on intent. For field definitions and examples, see the [Agent specification reference](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/agent-spec-reference.html). The specification can be refined over several iterations. The assistant limits clarifying questions to about two rounds before proposing an initial draft, then iterates on that draft with you. When choosing a model, it recommends a capable default from your [LLM Gateway catalog](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html#list-models-command) unless you specify cost or other constraints. Type `list models` at any time to browse what is available in your organization. Before simulation or coding, the assistant asks whether the Agentic Starter template’s default chat UI is sufficient or you need a custom frontend (for example a dashboard, data visualization, or multi-page app). That choice is recorded in the `frontend` section of `agent_spec.md`. Once you have a spec, you can [simulate the agent](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/workflows-and-prompting.html#simulate-an-agent) to validate the design before coding. ``` # Example conversation: Design a customer support agent $ I want to create a customer support agent for an e-commerce site Agent Assist: I'll help you design that! I can ask some questions to understand your requirements better. What specific customer issues should this agent handle? User: $ It should handle order status, returns, and product questions DataRobot Agent Assist: Great! What data sources will it need access to? User: $ Our order database and product catalog DataRobot Agent Assist: Perfect. Should it escalate complex issues to humans? User: $ Yes, escalate billing and refund requests DataRobot Agent Assist: I'll create a specification for your customer support agent... [Creates and displays agent_spec.md, saves to file] ``` ### Simulate an agent After you have an `agent_spec.md`, you can simulate the agent before implementing it: you try the design as an end user would, without writing implementation code. The panel that opens is titled Agent Dress Rehearsal. Simulation is for design validation only —no real tool code runs, and no external services are called. - What happens: The model (LLM) uses your spec to decide which tools to call and with what arguments. When the model requests a tool call, Agent Assist does not run your tool implementation or call DataRobot deployments, datasets, or other APIs. Instead, it uses the LLM to generate simulated return values that match the tool’s described inputs and outputs. You see which tools were selected, the arguments passed, and the simulated results—so you can judge whether the design makes sense. - What it’s for: Validate that the right tools exist, that input/output structures are reasonable, and that the selected model tends to call the right tools for the task—before you spend time implementing real tool code. - What it’s not: Simulation does not call real DataRobot deployments, read real datasets, or hit real APIs. To use live data and deployments, implement the agent and run it in the Code or Deploy workflow. Ask the assistant to simulate your agent (for example, “Simulate my agent” or “Let me try the design”) to begin. During the session: - Act as the end user —Type messages the way a real user would (for example order-status questions, research queries, or edge cases you care about). - Try specific scenarios —Describe particular requests or failure modes to see whether the model picks the right tools and arguments. - Capture design notes —If something in the prompt, tools, or examples should change, tell the assistant, or type /f (for example, /f the search tool should also accept a date range ) to log the note directly without interrupting the simulation. - End and review —When you are done, ask the assistant to wrap up simulation. It can summarize how the agent performed and suggest concrete changes to the spec (system prompt, tools, model, or examples). If you recorded notes with /f , that feedback is compiled into the summary the assistant uses to help refine agent_spec.md . Type `/quit` to exit the session when you are finished, or `/f ` at any point during simulation to record feedback without leaving it. Unlike `/quit`, `/f` is only recognized during simulation, not at the main `$` prompt. ### Code an AI agent Use this workflow when the `agent_spec.md` file already exists and is ready to implement. The assistant loads the specification and—if the template repository is not yet present—clones the [DataRobot Agentic Starter repository](https://github.com/datarobot-community/datarobot-agent-application) into your working directory. It then follows the template’s `AGENTS.md` for setup and implementation. The assistant can run shell commands (with approval), edit files, and manage multi-step work with tools. It suggests testing locally and deploying when the implementation is ready. > [!NOTE] Empty working directory > Agent Assist clones the template into the current directory. If the directory already contains files other than `agent_spec.md`, the assistant warns you before proceeding so existing work is not overwritten. When preparing the template for the first time, the assistant typically: 1. Verifies dependencies —Runs dr dependencies check so required tools (Git, Node, Task, Pulumi, and others) meet minimum versions. See Prerequisite tools . 2. Asks for an agentic framework —You choose one of the supported frameworks: LangGraph , CrewAI , LlamaIndex , NeMo Agent Toolkit (NAT) , or Base (minimal scaffold). The choice is saved in the template configuration and determines which example patterns the assistant follows. 3. Initializes the project —Runs dr start (with your approval) to configure credentials, ports, and LLM settings for the Agentic Starter template. 4. Implements from the spec —Adapts template code to match agent_spec.md (tools, prompts, and frontend requirements). Tools that need external API access use the authentication described in the spec; you configure secrets in the template as described in AGENTS.md . > [!WARNING] Windows > Agent Assist runs on macOS and Linux. On Windows, use [WSL](https://learn.microsoft.com/en-us/windows/wsl/install) or a [DataRobot codespace](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html) for the Code workflow. See [System requirements](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/installation.html#system-requirements). The Agentic Starter template is structured for production-style agent applications (for example, an MCP server surface for tools, guided configuration through `dr start`, and deployment-oriented tasks). For details on what the template includes out of the box, see the [Agentic application template](https://docs.datarobot.com/en/docs/wb-apps/app-templates/at-agentic-app.html) overview. After implementation, test the agent locally using the commands in the template’s `AGENTS.md`. Run those commands in a separate terminal in the project directory. When you are ready for production, continue with [Deploy an AI agent](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/workflows-and-prompting.html#deploy-an-ai-agent). ``` # Example conversation: Add a tool to an existing specification User: $ Load my existing agent specification and add a tool for inventory checking Agent Assist: I found your agent_spec.md. I can see it's a customer support agent. Let me add an inventory checking tool... [Reads current spec and adds the requested tool] ``` ### Deploy an AI agent Use this workflow when an implemented agent is ready to deploy to DataRobot. The assistant reads the deployment instructions from the template’s `AGENTS.md` and guides users through the steps. It does not run deployment commands; it indicates what to run (in a separate terminal). For interactive or long-running steps (such as `dr start`), the assistant uses the appropriate tools as defined in the codebase. ``` # Example conversation: Deploy the agent User: $ Help me deploy the agent I implemented Agent Assist: I'll read the deployment section of AGENTS.md and walk you through the steps... [Displays deployment commands and instructions] ``` ## Interact with the DataRobot CLI DataRobot Agent Assist runs certain DataRobot CLI ( `dr`) commands on your behalf and can run other shell commands (including additional `dr` commands) with your approval. - At startup: The assistant runs dr dependencies check to verify that required tools (Python, Node, Git, Task, Pulumi, etc.) are installed and meet minimum versions. See Prerequisites and installation . - Code workflow: When you use the assistant to clone and prepare the template (for example, "Code an AI agent" or "prepare to code"), it runs dr start in the template directory to initialize the DataRobot project. You are prompted to approve before the command runs. - Any workflow: The assistant can run shell commands—including dr subcommands—after you approve. It shows the command and description, then prompts for approval before executing. For full DataRobot CLI usage (authorization, deployment, and other commands), see the [DataRobot CLI documentation](https://docs.datarobot.com/en/docs/agentic-ai/cli/index.html). ## Prompting considerations Consider the following guidelines to get better results from the assistant: - Be specific : "Create a customer support agent that handles order inquiries and escalates billing issues" rather than "Create an agent". - Provide context : "I'm building a travel booking agent and need help with the flight search API integration" rather than "Help me with my code". - Ask for explanations : "Explain why this code isn't working and show me how to fix it" rather than "Fix this". - Iterate : "This is good, but can you make it handle edge cases like cancelled orders?" rather than accepting the first solution. --- # Configure evaluation and moderation URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-deploy/agentic-configure-evaluation-moderation.html > How to configure evaluation and moderation guardrails for a custom text generation model and agentic workflows in the workshop. > [!NOTE] Premium > Evaluation and moderation guardrails are a premium feature. Contact your DataRobot representative or administrator for information on enabling this feature. > > Feature flag: Enable Moderation Guardrails ( Premium), Enable Global Models in the Model Registry ( Premium), Enable Additional Custom Model Output in Prediction Responses Evaluation and moderation guardrails help your organization block prompt injection and hateful, toxic, or inappropriate prompts and responses. It can also prevent hallucinations or low-confidence responses and, more generally, keep the model on topic. In addition, these guardrails can safeguard against the sharing of personally identifiable information (PII). Many evaluation and moderation guardrails connect a deployed text generation model (LLM) or agentic workflow to a deployed guard model. These guard models make predictions on LLM prompts and responses and then report these predictions and statistics to the central LLM or agentic workflow deployment. To use evaluation and moderation guardrails, first create and deploy guard models to make predictions on an LLM's prompts or responses; for example, a guard model could identify prompt injection or toxic responses. Then, when you create a custom model with the Text Generation or Agentic Workflow target type, define one or more evaluation and moderation guardrails. **Important prerequisites** Before configuring evaluation and moderation guardrails for an LLM, follow these guidelines while deploying guard models and configuring your LLM deployment: - If using a custom guard model, before deployment, define moderations.input_column_name and moderations.output_column_name as tag-type key values on the registered model version . If you don't set these key values, any users of the guard model will have to enter the input and output column names manually. - Deploy the global or custom guard models you intend to use to monitor the central LLM before configuring evaluation and moderation. - Deploy the central LLM on a different prediction environment than the deployed guard models. - Set an association ID and enable prediction storage before you start making predictions through the deployed LLM. If you don't set an association ID and provide association IDs alongside the LLM's predictions, the metrics for the moderations won't be calculated on theCustom metricstab. - After you define the association ID, you can enable automatic association ID generation to ensure these metrics appear on the Custom metrics tab. You can enable this setting during or after deployment. - If you plan to use any of the NeMo Evaluator metrics (Agent Goal Accuracy, Context Relevance, Faithfulness (NeMo Evaluator), LLM Judge, Response Groundedness, Response Relevancy, Topic Adherence), create a NeMo evaluator workload and workload deployment via the Workload API first. The Workload API has no UI; you must use the API to create the workload and then create the workload deployment. Each of these metrics requires a NeMo evaluator deployment. **Prediction method considerations** When making predictions outside a [chat generation Q&A application](https://docs.datarobot.com/en/docs/wb-apps/custom-apps/create-qa-app.html), evaluations and moderations are only compatible with [real-time predictions](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-predictions/nxt-pred-api-snippets.html#real-time-prediction-snippet-settings), not [batch predictions](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-predictions/nxt-pred-api-snippets.html#batch-prediction-snippet-settings). In addition, when requesting streaming responses using the [Bolt-on Governance API](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#chat), evaluation and moderation negates the effect of streaming. Guardrails evaluate only the complete response of the LLM and therefore return the response text in one chunk. ## Select evaluation and moderation guardrails When you create a custom model with the Text Generation or Agentic Workflow target type, define one or more evaluation and moderation guardrails. To select and configure evaluation and moderation guardrails: 1. In theWorkshop, open theAssembletab of a custom model with theText GenerationorAgentic Workflowtarget type andassemble a model, eithermanually from a custom model you created outside of DataRobotorautomatically from a model built in a Use Case's LLM playground: When you assemble a text generation model with moderations, ensure you configure any requiredruntime parameters(for example, credentials) orresource settings(for example, public network access). Finally, set theBase environmentto a moderation-compatible environment; for example,[GenAI] Python 3.12 with Moderations: Resource settingsDataRobot recommends creating the LLM custom model using larger resource bundles with more memory and CPU resources. 2. After you've configured the custom model's required settings, navigate to theEvaluation and moderationsection and clickConfigure: 3. On theConfigure evaluation and moderationpanel, in theConfiguration summary, access the following settings: SettingDescriptionShow workflowReview how evaluations are executed in DataRobot. All evaluations and their respective moderations run in parallel.Moderation settingsSet the following:Set moderation timeout: Configure the maximum wait time (in seconds) for moderations before the system automatically times out.Timeout action: Define what happens if the moderation system times out:Score prompt / responseorBlock prompt / response.NeMo evaluator settingsSet theNeMo evaluator deploymentused by the NeMo Evaluator metrics. The dropdown shows "No options available" until you have created a NeMo evaluator workload and workload deployment via the Workload API. You must complete that step before you can configure the NeMo Evaluator metrics. 4. In theConfigure evaluation and moderationpanel, click one of the following metric cards to configure the required properties. The panel has two sections:All MetricsandNeMo metrics. From theConfiguration summarysidebar you can openShow workflow,Moderation settings, orNeMo evaluator settingsto configure the evaluator deployment used by all NeMo evaluator metrics. All MetricsNeMo metricsEvaluation metricRequiresDescriptionContent SafetyA deployed NIM modelllama-3.1-nemoguard-8b-content-safetyimported fromNVIDIA GPU Cloud (NGC) Catalog.Classify prompts and responses as safe or unsafe; return a list of any unsafe categories detected.CostLLM cost settingsCalculate the cost of generating the LLM response using the provided input cost-per-token, and output cost-per-token values. The cost calculation also includes the cost of citations. For more information, seeCost metric settings.Custom DeploymentCustom deploymentUse any deployment to evaluate and moderate your LLM (supported target types: regression, binary classification,multiclass, text generation).Emotions ClassifierEmotions Classifier deploymentClassify prompt or response text by emotion.FaithfulnessLLM, vector databaseMeasure if the LLM response matches the source to identify possible hallucinations.JailbreakA deployed NIM modelnemoguard-jailbreak-detectimported fromNVIDIA GPU Cloud (NGC) Catalog.Classify jailbreak attempts using NemoGuard JailbreakDetect.PII DetectionPresidio PII DetectionDetect Personally Identifiable Information (PII) in text using the Microsoft Presidio library.Prompt InjectionPrompt Injection ClassifierDetect input manipulations, such as overwriting or altering system prompts, intended to modify the model's output.Prompt tokensN/ATrack the number of tokens associated with the input to the LLM and/or retrieved text from the vector database.Response tokensN/ATrack the number of tokens associated with the output from the LLM and/or retrieved text from the vector database.ROUGE-1Vector databaseCalculate the similarity between the response generated from an LLM blueprint and the documents retrieved from the vector database.ToxicityToxicity ClassifierClassify content toxicity to apply moderation techniques, safeguarding against dissemination of harmful content.Agentic workflow metricsAgent Goal AccuracyLLMEvaluate agentic workflow performance in achieving specified objectives in scenarios without a known benchmark. (This agentic workflow metric is distinct from the NeMo Evaluator metric of the same name underNeMo metrics.)Task AdherenceLLMMeasure whether the agentic workflow response is relevant, complete, and aligned with user expectations.Guideline AdherenceLLM, guideline settingEvaluate how well the response follows the defined guideline using a judge LLM. Returnstruewhen the guideline is followed,falseotherwise. You must supply the guideline and select an LLM (from the gateway or a deployment) when configuring.Global models for evaluation metric deploymentsThe deployments required for PII detection, prompt injection detection, emotion classification, and toxicity classification are available asglobal models in RegistryMulticlass custom deployment metric limitsMulticlasscustom deployment metrics can have:Up to10classes defined in theMatcheslist for moderation criteria.Up to100class names in the guard model.TheNeMo Evaluator metrics(Agent Goal Accuracy, Context Relevance, Faithfulness, LLM Judge, Response Groundedness, Response Relevancy, Topic Adherence) require aNeMo evaluator workload deployment, set inNeMo evaluator settingsin the Configuration summary sidebar. Create the workload and workload deployment via the Workload API before you can select it; theSelect a workload deploymentdropdown shows "No options available" until a deployment exists. Each of these metrics also uses an LLM judge (DataRobot deployment or LLM gateway). Response Relevancy additionally requires an embedding deployment. Topic Adherence and LLM Judge have additional configuration.Stay on topic for inputsandStay on topic for outputdo notuse the NeMo evaluator deployment. They use aNIM deploymentof thellama-3.1-nemoguard-8b-topic-controlmodel (like Content safety and Jailbreak use NIM models). Configure them with LLM typeNIM, select the topic-control NIM deployment, and optionally edit the NeMo guardrails configuration files.Evaluator metricRequiresDescriptionAgent Goal AccuracyEvaluator deployment, LLMEvaluate how well the agent fulfills the user's query. This is distinct from the Agent Goal Accuracy metric underAll metrics(agentic workflow).Context RelevanceEvaluator deployment, LLMMeasure how relevant the provided context is to the response.FaithfulnessEvaluator deployment, LLMEvaluate whether the response stays faithful to the provided context using the NeMo Evaluator. This is distinct from the non-NeMo Faithfulness metric listed underAll metrics.LLM JudgeEvaluator deployment, LLMUse a judge LLM to evaluate a user defined metric.Response GroundednessEvaluator deployment, LLMEvaluate whether the response is grounded in the provided context.Response RelevancyEvaluator deployment, LLM, Embedding deploymentMeasure how relevant the response is to the user's query.Topic AdherenceEvaluator deployment, LLM, Metric mode, Reference topicsAssess whether the response adheres to the expected topics.Topic control metricsStay on topic for inputsNIM deployment ofllama-3.1-nemoguard-8b-topic-control, NVIDIA NeMo guardrails configurationUse NVIDIA NeMo Guardrails to provide topic boundaries, ensuring prompts are topic-relevant and do not use blocked terms.Stay on topic for outputNIM deployment ofllama-3.1-nemoguard-8b-topic-control, NVIDIA NeMo guardrails configurationUse NVIDIA NeMo Guardrails to provide topic boundaries, ensuring responses are topic-relevant and do not use blocked terms.To set theNeMo evaluator deploymentused by the NeMo Evaluator metrics, openNeMo evaluator settingsfrom the Configuration summary sidebar. The evaluator deployment will be applied to all NeMo Evaluator metrics. From theSelect a workload deploymentdropdown list, choose the workload deployment for the NeMo evaluator.NeMo evaluator settings panelThe dropdown shows "No options available" until you have created a NeMo evaluator workload and workload deployment via the Workload API. You must complete that step before you can configure the NeMo Evaluator metrics. 5. Depending on the metric selected above, configure the following fields: FieldDescriptionGeneral settingsNameEnter a unique name if adding multiple instances of the evaluation metric.Apply toSelect one or both ofPromptandResponse, depending on the evaluation metric. Note that when you selectPrompt, it's the user prompt, not the final LLM prompt, that is used for metric calculation. This field is only configurable for metrics that apply to both the prompt and the response.Custom Deployment, PII Detection, Prompt Injection, Emotions Classifier, and Toxicity settingsDeployment nameFor evaluation metrics calculated by a guard model, select the custom model deployment.Custom Deployment settingsInput column nameThis name is defined by the custom model creator. Forglobal models created by DataRobot, the default input column name istext. If the guard model for the custom deployment has themoderations.input_column_namekey valuedefined, this field is populated automatically.Output column nameThis name is defined by the custom model creator, and needs to refer to the target column for the model. The target name is listed on the deployment'sOverviewtab (and often has_PREDICTIONappended to it). You can confirm the column names byexporting and viewing the CSV data from the custom deployment. If the guard model for the custom deployment has themoderations.output_column_namekey valuedefined, this field is populated automatically.Guideline Adherence settingGuidelineThe rule or criteria the agent's response should follow. The selected LLM acts as a judge to evaluate whether the response adheres to this guideline and returnstrue(guideline followed) orfalse(guideline not followed). You must supply the guideline and select an LLM (from the gateway or a deployment) when configuring this metric.Faithfulness, Task Adherence, and Guideline Adherence settingsLLMSelect an LLM to evaluate the selected metric. For Faithfulness, once you select an LLM, you have the option of using your ownuser-providedcredentials instead of DataRobot-provided.NeMo Evaluator metric settingsSelect LLM as a judgeSelect an LLM to evaluate the selected metric.Evaluator deploymentFor the NeMo Evaluator metrics only: set in theNeMo evaluator settingssidebar panel (Select a workload deployment). The NeMo evaluator workload deployment is shared by those metrics. Create the workload and workload deployment via the Workload API before configuring; see the prerequisites above.Topic control settingsLLM TypeSelectAzure OpenAI,OpenAI, orNIM. For theAzure OpenAILLM type, additionally enter anOpenAI API deployment; forNIMenter aNIM deployment. If you use the LLM gateway, the default experience, DataRobot-supplied credentials are provided. When LLM type isAzure OpenAIorOpenAI, clickChange credentialsto provide your own authentication.FilesFor theStay on topicevaluations, next to a file, clickto modify the NeMo guardrails configuration files. In particular, updateprompts.ymlwith allowed and blocked topics andblocked_terms.txtwith the blocked terms, providing rules for NeMo guardrails to enforce. Theblocked_terms.txtfile is shared between the input and output topic control metrics; therefore, modifyingblocked_terms.txtin the input metric modifies it for the output metric and vice versa. Only two topic control metrics can exist in a custom model, one for input and one for output.Moderation settingsConfigure and apply moderationEnable this setting to expand theModerationsection and define the criteria that determine when moderation logic is applied. Cost metric settingsFor theCostmetric, define theInputandOutputcost incurrency amount / tokens amountformat, then clickAdd:TheCostmetric doesn't include theModerationsection toConfigure and apply moderation. 6. In theModerationsection, withConfigure and apply moderationenabled, for each evaluation metric, set the following: SettingDescriptionModeration criteriaIf applicable, set the threshold settings evaluated to trigger moderation logic. For numeric metrics (int or float), you can useless than,greater than, orequals towith a value of your choice. For binary metrics (for example, Agent Goal Accuracy), useequals to0 or 1. For the Emotions Classifier, selectMatchesorDoes not matchand define a list of classes (emotions) to trigger moderation logic.Moderation methodSelectReport,Report and block, orReplace(if applicable).Moderation messageIf you selectReport and block, you can optionally modify the default message. 7. After configuring the required fields, clickAddto save the evaluation and return to the evaluation selection page. Then, select and configure another metric, or clickSave configuration. The guardrails you selected appear in theEvaluation and moderationsection of theAssembletab. After you add guardrails to a text generation custom model, you can [test](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-test-custom-model.html), [register](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-register-cus-models.html), and [deploy](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-deploy-models.html) the model to make predictions in production. After making [predictions](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-predictions/index.html), you can view the evaluation metrics on the [Custom metrics](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-custom-metrics.html) tab and prompts, responses, and feedback (if configured) on the [Data exploration](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-exploration.html) tab. > [!NOTE] Data quality tab > When you add moderations to an LLM deployment, you can't view custom metric data by row on the [Data exploration > Data quality](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-exploration.html#explore-deployment-data-quality) tab. ### Change credentials DataRobot provides credentials for [available LLMs](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html) using the LLM gateway. With certain metrics and LLMs or LLM types, you can instead use your own credentials for authentication. Before proceeding, define user-specified credentials on the [credentials management](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html#credentials-management) page. #### Topic control metrics To change credentials for either Stay on topic for inputs or Stay on topic for output, choose the LLM type and click Change credentials. **LLM type: Azure OpenAI:** Provide the Azure OpenAI API deployment and the OpenAI API base URL. Then, from the dropdown, select the set of credentials to apply. [https://docs.datarobot.com/en/docs/images/change-metric-creds-azure.png](https://docs.datarobot.com/en/docs/images/change-metric-creds-azure.png) **LLM type: OpenAI:** From the dropdown, select the set of credentials to apply. [https://docs.datarobot.com/en/docs/images/change-metric-creds-openai.png](https://docs.datarobot.com/en/docs/images/change-metric-creds-openai.png) **LLM type: NIM:** Select the NIM deployment (for example, the topic-control model). Credentials are typically provided via the deployment configuration. To revert to DataRobot-provided credentials, click Revert credentials. #### Faithfulness metric To change credentials for Faithfulness, select the LLM and click Change credentials. The following table lists the required fields: | Provider | Fields | | --- | --- | | Amazon | AWS account (credentials)AWS region | | Azure OpenAI | OpenAI API deploymentOpenAI API base URLCredentials | | Google | Service account (credentials)Google region | | OpenAI | Credentials | To revert to DataRobot-provided credentials, click Revert credentials. ### Considerations for NeMo Evaluator metrics When using NeMo Evaluator metrics, consider the following: - LLM judge output: The NeMo evaluator expects the LLM judge to return data in the correct JSON schema. Some models (for example, certain Llama versions) may return Python code or other formats instead, which can cause the evaluator to fail. Choose an LLM judge that reliably returns the expected format; newer models are often better at following JSON output instructions. - Rate and token limits: Be aware of rate limits and token limits when using NeMo Evaluator guards; hitting these limits can cause evaluation failures. You can use the [Activity log > Moderation](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-activity-log/nxt-moderation.html) tab and evaluator logs to debug why a request was blocked or why a guard failed. ### Global models for evaluation metric deployments The deployments required for PII detection, prompt injection detection, emotion classification, and toxicity classification are available as [global models in Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-global-models.html). The following global models are available: | Model | Type | Target | Description | | --- | --- | --- | --- | | Prompt Injection Classifier | Binary | injection | Classifies text as prompt injection or legitimate. This guard model requires one column named text, containing the text to classify. For more information, see the deberta-v3-base-injection model details. | | Toxicity Classifier | Binary | toxicity | Classifies text as toxic or non-toxic. This guard model requires one column named text, containing the text to classify. For more information, see the toxic-comment-model details. | | Sentiment Classifier | Binary | sentiment | Classifies text sentiment as positive or negative. This model requires one column named text, containing the text to classify. For more information, see the distilbert-base-uncased-finetuned-sst-2-english model details. | | Emotions Classifier | Multiclass | target | Classifies text by emotion. This is a multilabel model, meaning that multiple emotions can be applied to the text. This model requires one column named text, containing the text to classify. For more information, see the roberta-base-go_emotions-onnx model details. | | Refusal Score | Regression | target | Outputs a maximum similarity score, comparing the input to a list of cases where an LLM has refused to answer a query because the prompt is outside the limits of what the model is configured to answer. | | Presidio PII Detection | Binary | contains_pii | Detects and replaces Personally Identifiable Information (PII) in text. This guard model requires one column named text, containing the text to be classified. The types of PII to detect can optionally be specified in a column, 'entities', as a comma-separated string. If this column is not specified, all supported entities will be detected. Entity types can be found in the PII entities supported by Presidio documentation. In addition to the detection result, the model returns an anonymized_text column, containing an updated version of the input with detected PII replaced with placeholders. For more information, see the Presidio: Data Protection and De-identification SDK documentation. | | Zero-shot Classifier | Binary | target | Performs zero-shot classification on text with user-specified labels. This model requires classified text in a column named text and class labels as a comma-separated string in a column named labels. It expects the same set of labels for all rows; therefore, the labels provided in the first row are used. For more information, see the deberta-v3-large-zeroshot-v1 model details. | | Python Dummy Binary Classification | Binary | target | Always yields 0.75 for the positive class. For more information, see the python3_dummy_binary model template. | ## View evaluation and moderation guardrails When a text generation model with guardrails is registered and deployed, you can view the configured guardrails on the [registered model'sOverview](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-view-manage-reg-models.html#view-version-details) tab and the [deployment'sOverview](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-overview/nxt-overview.html) tab: **Registry:** [https://docs.datarobot.com/en/docs/images/nxt-evaluation-moderation-reg-overview.png](https://docs.datarobot.com/en/docs/images/nxt-evaluation-moderation-reg-overview.png) **Console:** [https://docs.datarobot.com/en/docs/images/nxt-evaluation-moderation-deploy-overview.png](https://docs.datarobot.com/en/docs/images/nxt-evaluation-moderation-deploy-overview.png) > [!NOTE] Evaluation and moderation logs > On the [Activity log > Moderation](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-activity-log/nxt-moderation.html) tab of a deployed LLM with evaluation and moderation configured, you can view a history of evaluation and moderation-related events for the deployment to diagnose issues with a deployment's configured evaluations and moderations. --- # Deploy workflows URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-deploy/agentic-workflow-deploy.html > After an agentic workflow is registered and ready for deployment, you can deploy it to Console and configure deployment settings for monitoring and tracing. Once the agentic workflow is registered (listed in the Models tile in Registry), you can deploy it to Console as you would any other model type, providing access to DataRobot monitoring capabilities. To deploy an agentic workflow from Registry: 1. On theRegistry > Modelspage, if the registered agentic workflow isn't already open, click theAgentic workflowtab and locate the workflow to deploy. 2. Click the agentic workflow to open it. If it has theReady for deploymentstatus badge, clickDeploy. 3. Configure the deployment settingsfor the agentic workflow. In particular, review the following sections: Setting configuration post-deploymentYou can configure these settings after the workflow is deployed; however, some settings, like runtime parameters, require temporary deactivation of the deployment. 4. After configuring the deployment settings, clickDeploy model. TheCreating deploymentdialog box appears. Wait for deployment creation, or clickReturn to deploymentsto open Console. --- # Register workflows URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-deploy/agentic-workflow-register.html > After you've connected an agentic workflow custom model to an agentic playground, compared and tested the workflow, and made any required modifications to the workflow, you can register the production-ready agentic workflow for deployment to Console. After experimenting in the agentic playground to build a production-ready agentic workflow, register the custom agentic workflow in the Registry workshop, in preparation for deployment to Console. ## Register from an agentic playground To begin registering an agentic workflow from an agentic playground, find the register option in the comparison chat, or a single-agent chat: **Comparison chat:** Click the actions menu for the agentic workflow you want to register, then click Register agentic workflow. [https://docs.datarobot.com/en/docs/images/agentic-workflow-register-1.png](https://docs.datarobot.com/en/docs/images/agentic-workflow-register-1.png) > [!TIP] Actions menu locations > The actions menu is always available in the Workflows panel for connected agentic workflows. In addition, when a workflow is selected for comparison, the actions menu is available next to the agentic workflow name in the comparison chat area. **Single-agent chat:** Click Register agentic workflow in the upper-right corner of the single-agent chat window. [https://docs.datarobot.com/en/docs/images/agentic-workflow-register-2.png](https://docs.datarobot.com/en/docs/images/agentic-workflow-register-2.png) The agentic workflow opens in the Registry Workshop page. Proceed to [Register from Workshop](https://docs.datarobot.com/en/docs/agentic-ai/agentic-deploy/agentic-workflow-register.html#register-from-workshop). ## Register from Workshop On the Registry > Workshop page, in an open agentic workflow: 1. On theAssembletab, ensure the agentic workflow is fully assembled by reviewing the following sections: LLM gateway access runtime parameter requirementTo use theLLM gatewayfor an agentic workflow, theENABLE_LLM_GATEWAY_INFERENCEruntime parameter must be provided in themodel-metadata.yamlfile and set totrue. 2. (Optional) ClickTest workflowto provide a test dataset andtest the agentic workflow response through the chat and/or score hooks. 3. After the workflow is fully assembled, clickRegister a workflowto open theRegister a workflowpage. 4. UnderConfigure the workflow, theTargetis set based on the workflow you're registering and theTarget typeis set toAgentic Workflow. Select one of the following registration options: 5. ClickRegister a workflow. The agentic workflow version opens on theRegistry > Modelspage with aBuildingstatus. ## Register from the Models page On the Registry > Models page, to register a fully configured agentic workflow: 1. Click theAgentic workflowtab, to filter theModelspage on agentic workflows. 2. Click+ Register a workflow(or thebutton when the registered model or version info panel is open): TheRegister a modelpanel opens to theExternal modeltab. 3. Click theCustom modeltab and then, underConfigure the model, select one of the following options: Then, configure the following fields: FieldDescriptionCustom modelSelect the custom workflow you want to register from theworkshop.Custom model versionSelect the version of the custom workflow to register.Registered model name / Registered modelDo one of the following:Registered model name:When registering a new model, enter a unique and descriptive name for the new registered model. If you choose a name that exists anywhere within your organization, a warning appears.Registered model:When saving as a version of an existing model, select the existing registered model you want to add a new version to.Registered version nameAutomatically populated with the model name, date, and time. Change or modify the name as necessary.Registered model versionAssigned automatically. This displays the expected version number of the version (e.g., V1, V2, V3) you create. This is alwaysV1when you selectRegister as a new model.Optional settingsRegistered version descriptionEnter a description of the business problem this model package solves, or, more generally, describe the model represented by this version.TagsClick+ Add tagand enter aKeyand aValuefor each key-value pair you want to tag the modelversionwith. Tags added when registering a new model are applied toV1. NoteIf you clickCancelon this page to return to theRegistry, you lose the configuration progress on this page. 4. ClickRegister model. The agentic workflow version opens on theRegistry > Modelspage with aBuildingstatus. ## Custom agentic workflow build troubleshooting If the custom workflow build completes with a Build failed status, you can troubleshoot the failure using the model logs. To access the model logs, in the Insight computation failed warning, click Open the workshop: The Workshop opens to the Versions tab for the custom workflow you registered, with the version panel open to the Insights section. Next to Status, find Logs, then click Model logs to open the model logs console: In the Console Log: Model logs modal, review the timestamped log entries: | | Information | Description | | --- | --- | --- | | (1) | Date / time | The date and time the model log event was recorded. | | (2) | Status | The status the log entry reports: INFO: Reports a successful operation.ERROR: Reports an unsuccessful operation. | | (3) | Message | The description of the successful operation (INFO), or the reason for the failed operation (ERROR). This information can help you troubleshoot the root cause of the error. | > [!NOTE] Model logs consideration > In the Registry, a model package's Model logs only report the operations of the underlying model, not the model package operations (e.g., model package deployment time). If you can't locate the log entry for the error you need to fix, it may be an older log entry not shown in the current view. Click Load older logs to expand the Model logs view. > [!TIP] View older logs > Look for the older log entries at the top of the Model logs; they are added to the top of the existing log history. --- # Deploy URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-deploy/index.html > Register and deploy agentic workflows, and configure evaluation and moderation guardrails. > [!NOTE] Premium > DataRobot's Agentic AI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. Register and deploy agentic workflows, and configure evaluation and moderation guardrails. | Topic | Description | | --- | --- | | Register workflows | Register production-ready agentic workflows for deployment to Console. | | Configure evaluation and moderation | Configure evaluation and moderation guardrails for agentic and RAG workflows. | | Deploy workflows | Deploy agentic workflows to Console. | --- # Agent authentication URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-authentication.html > Learn how to implement authentication in DataRobot Agent Templates, covering API tokens, authorization context, OAuth 2.0, and security best practices. AI agents often need to authenticate to external resources to complete tasks. For example, a deployed agent might need to access external APIs, databases, or cloud services to retrieve data or perform operations. This documentation provides comprehensive guidance for implementing authentication in DataRobot Agent Templates, covering API tokens, MCP server authentication, authorization context, OAuth 2.0, and security best practices. ## Authentication methods This section provides an overview of the different authentication methods available in the framework: | Method | Description | Use Case | | --- | --- | --- | | API token authentication | Simple token-based authentication | External APIs, DataRobot services. | | MCP server authentication | Automatic token-based authentication for MCP servers | MCP tool integration. | | OAuth 2.0 | Standard OAuth for external services | Third-party integrations. | ## API token authentication API token authentication is the most common method for authenticating with DataRobot services and external APIs. It uses bearer tokens passed in headers or environment variables. ### DataRobot API authentication Configure API tokens using environment variables or programmatically. The `DATAROBOT_API_TOKEN` is available in the [API keys and tools section of your account settings](https://docs.datarobot.com/en/docs/platform/acct-settings/api-key-mgmt.html). **Environment variables (recommended):** The agent templates automatically use `DATAROBOT_API_TOKEN` and `DATAROBOT_ENDPOINT` environment variables when they are set. This is the recommended approach as it keeps credentials out of your code and works seamlessly with `MyAgent` initialization. The `MyAgent` class automatically falls back to these environment variables if credentials aren't provided explicitly. When using the optional `ToolClient` class for external tool integration, it also uses these environment variables by default. MCP servers also use `DATAROBOT_API_TOKEN` automatically for authentication. ``` DATAROBOT_API_TOKEN= DATAROBOT_ENDPOINT=https://app.datarobot.com ``` **Programmatic:** You can pass API credentials directly when initializing the `MyAgent` class. This is useful when you need to override environment variables or use different credentials for specific instances. The `MyAgent` class will still fall back to environment variables if parameters are not provided. ``` from agent import MyAgent agent = MyAgent( api_key="your_api_key", api_base="https://app.datarobot.com" ) ``` > [!NOTE] ToolClient and MCP tools for external tools > If you're integrating external tools using the `ToolClient` class from the `datarobot-genai` package, you can also pass credentials programmatically. For MCP server-based tool integration, authentication is handled automatically via `DATAROBOT_API_TOKEN`. See the [tool integration documentation](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools-integrate.html) and [MCP tools documentation](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-tools-mcp.html) for details. ### External API authentication When creating custom tools that need to authenticate with external APIs, retrieve API keys from environment variables or runtime parameters within your tool's `run()` method. This keeps credentials out of your source code and follows the same security pattern used throughout the agent templates. For local development, use environment variables. For deployed agents, define runtime parameters in `model-metadata.yaml` and retrieve them using `RuntimeParameters.get()` from `datarobot_drum`. Try `os.environ.get()` first (for local development), then fall back to `RuntimeParameters.get()` for deployed agents., as shown in the example below: ``` # custom-tool-example.py import os import requests from crewai.tools import BaseTool from datarobot_drum import RuntimeParameters class ExternalAPITool(BaseTool): def run(self, query: str) -> str: api_key = os.environ.get("EXTERNAL_API_KEY") if not api_key: api_key = RuntimeParameters.get("EXTERNAL_API_KEY")["apiToken"] headers = {"Authorization": f"Bearer {api_key}"} response = requests.get( "https://api.external-service.com/data", headers=headers, params={"query": query} ) return response.json() ``` > [!NOTE] Define runtime parameters for deployment > To use runtime parameters in deployed agents, define them in your `model-metadata.yaml` file. For credential-type parameters (like API keys), use `type: credential`: > > ``` > runtimeParameterDefinitions: > - fieldName: EXTERNAL_API_KEY > type: credential > credentialType: api_token > description: API key for external service authentication > ``` > > See the [runtime parameters documentation](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-runtime-parameters.html) for more details on configuration options. > > Because `api_token` is a single-field credential type, the platform injects the token into an environment variable named exactly the same as the `fieldName` (for example, `EXTERNAL_API_KEY`), not `EXTERNAL_API_KEY_API_TOKEN`. The sample code above uses `os.environ.get("EXTERNAL_API_KEY")` when running in a deployment. For multi-field credential types, see [runtime parameters](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-runtime-parameters.html). ### MCP server authentication When using MCP (Model Context Protocol) servers to provide tools to your agents, authentication is handled automatically using the `DATAROBOT_API_TOKEN` environment variable. The MCP client in the agent templates automatically uses this token for authenticated requests to the MCP server. Configuration: ``` # .env DATAROBOT_API_TOKEN= DATAROBOT_ENDPOINT=https://app.datarobot.com ``` The MCP client will automatically use `DATAROBOT_API_TOKEN` when making requests to the MCP server. No additional configuration is required beyond setting these environment variables. > [!NOTE] MCP vs. ToolClient authentication > MCP tools use `DATAROBOT_API_TOKEN` directly for server authentication, while `ToolClient` (used for direct tool deployments) can use both API tokens and authorization context. For more information about MCP tool integration, see [Integrate tools using an MCP server](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-tools-mcp.html). ## Authorization context The framework provides an authorization context system for propagating authentication information to downstream tools and services. This allows tokens and credentials to be automatically passed between agent tools without manual configuration. Initialize authorization context in your agent's `chat()` function using `resolve_authorization_context()` from the `datarobot-genai` package. The function returns the authorization context dictionary, which should be assigned to `completion_create_params["authorization_context"]`. Tools can then retrieve the context using `get_authorization_context()` from the `datarobot` SDK: ``` # custom.py from datarobot_genai.core.chat import resolve_authorization_context from datarobot.models.genai.agent.auth import get_authorization_context def chat(completion_create_params, load_model_result, **kwargs): # Initialize the authorization context for downstream agents and tools completion_create_params["authorization_context"] = resolve_authorization_context( completion_create_params, **kwargs ) # ... rest of chat function # In your tools auth_context = get_authorization_context() access_token = auth_context.get("access_token") ``` > [!NOTE] ToolClient and MCP tools with authorization context > When using the optional `ToolClient` class from the `datarobot-genai` package for external tool integration, it automatically propagates authorization context when calling agent tools. MCP tools use `DATAROBOT_API_TOKEN` directly for authentication and don't require authorization context propagation. See the [tool integration documentation](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools-integrate.html) and [MCP tools documentation](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-tools-mcp.html) for details. ## OAuth 2.0 authentication For external services that support OAuth 2.0, implement OAuth flows in your tools. Create an OAuth application in the service's developer console and set environment variables: ``` # .env OAUTH_CLIENT_ID= OAUTH_CLIENT_SECRET= OAUTH_REDIRECT_URI= OAUTH_SCOPE= ``` Then, use those environment variables in the OAuth implementation. This example demonstrates a complete OAuth 2.0 authorization code flow for a CrewAI tool. The tool inherits from `crewai.tools.BaseTool` (the standard base class for tools in CrewAI agent templates), manages its own token lifecycle with in-memory caching, and follows the repository's pattern of retrieving credentials from environment variables. > [!NOTE] Example implementation > Note that this is a CrewAI-specific framework example illustrating a basic OAuth authentication pattern. The `run()` method makes a placeholder API call to demonstrate token usage, but you'll need to implement actual tool logic based on your specific use case. ``` # custom-tool-example.py import os import time import requests from urllib.parse import urlencode from crewai.tools import BaseTool class ExampleToolWithOAuth(BaseTool): def __init__(self): super().__init__() self.client_id = os.getenv("OAUTH_CLIENT_ID") self.client_secret = os.getenv("OAUTH_CLIENT_SECRET") self.redirect_uri = os.getenv("OAUTH_REDIRECT_URI") self._access_token = None self._token_expires_at = None def get_authorization_url(self) -> str: params = { "client_id": self.client_id, "redirect_uri": self.redirect_uri, "response_type": "code", "scope": "read:data" } return f"https://oauth.provider.com/authorize?{urlencode(params)}" def exchange_code_for_token(self, code: str) -> dict: response = requests.post( "https://oauth.provider.com/token", data={ "grant_type": "authorization_code", "client_id": self.client_id, "client_secret": self.client_secret, "code": code, "redirect_uri": self.redirect_uri } ) token_data = response.json() self._access_token = token_data.get("access_token") expires_in = token_data.get("expires_in", 3600) self._token_expires_at = time.time() + expires_in return token_data def get_cached_access_token(self) -> str: if self._access_token and self._token_expires_at and time.time() < self._token_expires_at: return self._access_token # Token expired or not set - refresh or re-authenticate # In production, implement token refresh logic here raise ValueError("Access token expired. Re-authenticate to get a new token.") def run(self, query: str) -> str: access_token = self.get_cached_access_token() response = requests.get( "https://api.provider.com/data", headers={"Authorization": f"Bearer {access_token}"}, params={"query": query} ) return response.json() ``` ## Security best practices Following security best practices is essential when handling authentication in production environments. Adhere to the following guidelines: - API token authentication - Store tokens in environment variables; never hard code secrets in source code. - Use secure token storage solutions in production environments. - Implement token rotation and enforce token expiration policies. - Validate authorization context before using it in tools. - Follow least-privilege access principles. - Log authentication events for audit purposes. - OAuth 2.0 - Use HTTPS for all OAuth communications. - Validate the state parameter to prevent Cross-Site Request Forgery (CSRF) attacks. - Store refresh tokens securely. - Handle token expiration and refresh logic reliably. - Validate authorization context before use. - General Security - Use separate environments for development and production. - Implement robust secret management practices. - Follow container security best practices. - Conduct regular security audits and apply updates. ## Troubleshooting authentication issues This section helps you diagnose and resolve common authentication problems when developing or deploying agents. ### Common issues The following sections describe common authentication errors and how to resolve them: #### Missing API token This error occurs when the DataRobot API token is not configured. Issue: `Error: Missing DataRobot API token. Set the DATAROBOT_API_TOKEN environment variable` Solution: Set the `DATAROBOT_API_TOKEN` environment variable to use your [DataRobot API key](https://docs.datarobot.com/en/docs/platform/acct-settings/api-key-mgmt.html). #### Invalid endpoint This error occurs when the DataRobot endpoint is missing or incorrectly configured. Issue: `Error: Missing DataRobot endpoint. Set the DATAROBOT_ENDPOINT environment variable` Solution: Set the correct `DATAROBOT_ENDPOINT` environment variable. #### Authorization context not set This error occurs when tools try to access the authorization context before it has been initialized. Issue: `Error: Authorization context not available for tool` Solution: Ensure `resolve_authorization_context()` is called in your agent's `chat()` function and the result is assigned to `completion_create_params["authorization_context"]`. #### OAuth token expired This error occurs when an OAuth access token has expired and needs to be refreshed. Issue: `Error: 401 Unauthorized` Solution: Implement token refresh logic or re-authenticate. #### MCP server authentication failure This error occurs when the MCP server cannot authenticate the agent's requests. Issue: `Error: 401 Unauthorized` or `MCP connection failures` Solutions: 1. Verify API token : Ensure DATAROBOT_API_TOKEN is set correctly in your environment 2. Check token permissions : Verify the token has necessary permissions for MCP server access 3. Verify MCP server endpoint : Check that the MCP server endpoint is correct 4. For local development : Ensure the MCP server is running and accessible 5. Review MCP configuration : See MCP tools troubleshooting for more details ### Debugging tips Useful techniques for debugging authentication issues: #### Enable verbose logging Enable verbose logging to get detailed information about authentication operations. To do so, search for where `MyAgent` is instantiated in `custom.py` and set `verbose=True`. ``` # custom.py from agent import MyAgent # ... agent = MyAgent(verbose=True, **completion_create_params) ``` #### Check environment variables Verify that required environment variables are set correctly. ``` import os print(f"API Token: {os.getenv('DATAROBOT_API_TOKEN')}") print(f"Endpoint: {os.getenv('DATAROBOT_ENDPOINT')}") ``` #### Test authentication Test authentication using the `AgentEnvironment` class from `datarobot_genai.core.cli`: ``` from datarobot_genai.core.cli import AgentEnvironment try: env = AgentEnvironment() print("Authentication successful") except ValueError as e: print(f"Authentication failed: {e}") ``` #### Validate authorization context Check authorization context using `get_authorization_context()` from `datarobot.models.genai.agent.auth`: ``` # custom-tool-example.py from datarobot.models.genai.agent.auth import get_authorization_context auth_context = get_authorization_context() print(f"Authorization context: {auth_context}") ``` --- # Debugging agents (PyCharm) URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-debugging-pycharm.html > Use PyCharm's built-in Run Agent configuration to debug agent code locally during development. Debugging agent code is essential for understanding execution flow, inspecting variables, and troubleshooting issues during development. After you clone the repository, PyCharm will automatically have a Run Agent run/debug configuration that points to the `dev.py` script in your agent directory. This script starts the agent development server and gives you an immediate way to run or debug your agent without wiring up a remote debug server. This guide walks you through configuring PyCharm to use your agent's virtual environment, running the autogenerated `Run Agent` configuration, and attaching the debugger while you execute prompts with the CLI. ## Prerequisites Before setting up PyCharm debugging, ensure you have: - PyCharm Professional : Required for full debugging support - Agent application setup : Completed the installation and run dr start to prepare your local environment - Environment configured : Your .env file is properly configured with DATAROBOT_API_TOKEN and DATAROBOT_ENDPOINT ## Configure Python interpreter Before you can debug your agent, you need to configure PyCharm to use the correct Python interpreter for your agent environment. > [!NOTE] Run dr start first > You must run `dr start` before configuring the Python interpreter. This command initializes your workspace and sets up your local virtual environment. ### Set up the Python interpreter 1. Open your agent templates repository in PyCharm. 2. Navigate to PyCharm > Settings . 3. Go to Python > Interpreter . 4. Click the Python Interpreter dropdown (it may showNo interpreter) and selectAdd Interpreter > Add Local Interpreter. 5. In theAdd Python Interpreterdialog, selectSelect existingand choosePythonas the type. Select your agent's virtual environment Python executable in the.venvpath of your agent framework. (The virtual environment is created byuvwhen you rundr startortask install). The example workflow in theDataRobot Agentic Starter repositorycreates two agents:Planner Agent(content planning) andWriter Agent(content writing). For example, the interpreter path for your agent directory should look like: $workspace/agent/.venv/bin/python PyCharm will use this interpreter for running and debugging your agent code, ensuring all dependencies are correctly resolved. ## Use the Run Agent configuration The Run Agent configuration launches the agent development server with your environment variables and interpreter already wired up. 1. Open Run > Edit Configurations to review the configurations for Run Agent . 2. Ensure the configuration points to your agent's dev.py script, sets the working directory to the agent folder, and references your .env file. No additional parameters are required. 3. SelectRun Agentfrom the configuration dropdown and clickRun. PyCharm starts the development server and shows its console output (for example,Running development server on http://127.0.0.1:8842). > [!TIP] Need to recreate the configuration? > If the configuration is missing or out-of-date, create a new Python run configuration with the settings seen in the screenshot. ## Trigger agent execution from the CLI With the development server running in PyCharm, you can execute agent prompts from a terminal window. 1. Open a terminal in your agent application repository root. 2. Run the CLI command for your agent: taskagent:cli--execute--user_prompt"Artificial Intelligence" 3. Watch the PyCharm console for log output as the agent processes the request. Adjust the prompt text or CLI arguments to match your scenario. ## Debug with breakpoints The same Run Agent configuration supports PyCharm's debugger. 1. Set breakpoints in the files you want to inspect (for example, agent/agent/myagent.py ). 2. In PyCharm, choose theRun Agentconfiguration and click theDebugicon (or pressShift+F9). PyCharm launchesdev.pyin debug mode and waits for incoming work. 3. Re-run your CLI command. When execution reaches your breakpoint, PyCharm pauses and opens theDebugtool window so you can inspect state, step through code, and evaluate expressions. ## Use debug features Once execution stops on a breakpoint, you can use PyCharm's debugging capabilities: ### Inspect variables - The Threads & Variables pane shows all variables in the current scope. - Expand objects to see their properties and values. - Right-click variables to add them to Watches for persistent monitoring. ### Evaluate expressions - Click the Evaluate Expression button in the debug toolbar (calculator icon) or right-click in the current context and select Evaluate Expression . - Enter any Python expression to evaluate it in the current context (for example, os.environ ). - Useful for testing conditions, examining complex objects, or calling methods. ### View call stack - The Frames pane shows the call stack, displaying how execution reached the current point. - Click different frames to see variables and code at each level. ## Common issues ### Run Agent configuration is missing Issue: PyCharm does not show the Run Agent configuration. Solution: - Check that your project files (such as .idea/runConfigurations/Run Agent.run.xml ) are not excluded from your VCS or workspace. - Restart PyCharm to force it to reload project metadata. - Re-clone the repository. ### CLI command finishes without hitting breakpoints Issue: The agent finishes processing and never pauses where you expect. Solution: - Make sure you launched the Run Agent configuration in Debug mode, not regular Run. - Verify your breakpoint is active (solid red) and located in code that executes for the selected prompt. - Re-run the CLI command after PyCharm shows that the debugger is listening. ### Environment variables missing Issue: The agent fails due to missing credentials or configuration. Solution: - Confirm your .env file exists at the repository root and includes DATAROBOT_API_TOKEN and DATAROBOT_ENDPOINT . - In the Run Agent configuration, ensure the Paths to ".env" files field points to the correct file. ### Wrong Python version Issue: PyCharm shows "Python version mismatch" errors or modules are missing. Solution: - Confirm you've configured the correct Python interpreter (the one in your agent's virtual environment). - If you recently recreated the environment, click the interpreter dropdown and re-select the .venv Python executable. ### Breakpoints not working Issue: Execution doesn't stop at breakpoints even in Debug mode. Solution: - Confirm the debugger console shows the development server restarted after you clicked Debug. If not, stop and start the configuration again. - Check that you reran the CLI command after entering debug mode—the development server handles one request at a time. - Ensure PyCharm mapped your project correctly. If you're using multiple agent templates, verify you're editing the same copy of the file that the development server runs. --- # Debugging agents (VS Code) URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-debugging-vscode.html > Use VS Code's Run and Debug experience to execute dev.py and inspect agent code locally. VS Code can run the same `dev.py` script that powers the agent development server, so you can step through requests, inspect variables, and replay prompts without wiring up a remote debugger. After cloning the repository, you only need to point VS Code at your agent's virtual environment, create a launch configuration that targets `dev.py`, and execute prompts from the CLI while the debugger listens. This guide mirrors the PyCharm workflow and walks you through preparing VS Code, launching the debugger, and troubleshooting common issues. ## Prerequisites Before configuring VS Code debugging, ensure you have: - VS Code + Python extension : Install the official Microsoft Python extension for debugging support, and see the documentation if you need setup guidance. - Agent application setup : Complete the installation and run dr start to create the agent virtual environment. - Environment variables : Configure .env with DATAROBOT_API_TOKEN , DATAROBOT_ENDPOINT , and any tools-specific variables. - CLI access : You can run task :cli commands from a terminal to trigger executions while the debugger runs. ## Configure Python interpreter > [!NOTE] Run dr start first > `dr start` provisions dependencies and the `.venv` interpreter. Run it before selecting the interpreter in VS Code. 1. Open your agent application folder (for example, datarobot-agent-application ) in VS Code. 2. Press Command+Shift+P (macOS) or Ctrl+Shift+P (Windows/Linux) to open the Command Palette. 3. Run Python: Select Interpreter . 4. Choose the interpreter that lives inside your agent's.venvpath. For theagentdirectory, the entry typically looks like: $workspace/agent/.venv/bin/python VS Code uses this interpreter for linting, the integrated terminal, and the debugger so that the environment matches what `task` created. ## Configure the Run Agent launch configuration The VS Code debugger uses `.vscode/launch.json`. You can reuse the same layout as the PyCharm guide by creating a Run Agent configuration that points to `dev.py`. 1. Open the Run and Debug view (sidebar play icon) and click create a launch.json file . 2. Select Python Debugger as the debugger. 3. Select Python File as the debug configuration. 4. Replace the autogenerated .vscode/launch.json configuration with something similar to the snippet below, adjusting the paths to your agent: ``` { "version": "0.2.0", "configurations": [ { "name": "Run Agent", "type": "python", "request": "launch", "program": "${workspaceFolder}/agent/dev.py", "cwd": "${workspaceFolder}/agent", "envFile": "${workspaceFolder}/.env", "console": "integratedTerminal", "python": "${workspaceFolder}/agent/.venv/bin/python", "justMyCode": true } ] } ``` - program : Points to your agent's dev.py . - cwd : Ensures relative imports and file paths resolve from the agent directory. - envFile : Loads the same .env settings the CLI uses. - python : Forces the debugger to use the .venv interpreter you selected earlier. If your workspace contains multiple agents, duplicate the configuration—with different names and paths—for each agent you debug regularly. ## Run the development server To start up your development server: 1. Open the Run Agent configuration in the Run and Debug view. 2. Click Run (or press F5) to start the development server under the debugger. 3. Watch the Debug Console or Terminal for the startup message (for example, `Running development server on http://localhost:8842`). VS Code now listens for requests and will pause if you set breakpoints. ## Trigger agent execution from the CLI With the development server listening, you can send prompts from any terminal session. For an example CLI call: ``` task agent:cli -- execute --user_prompt "Artificial Intelligence" ``` Every CLI request flows through the VS Code debugger session, so you can repeat the same prompt or adjust arguments until you isolate the issue. ## Debug with breakpoints To set up debugging with breakpoints: 1. Set breakpoints in files such as `agent/agent/myagent.py`. 2. Click Run > Start Debugging (or press F5) to make sure the debugger is active. 3. Re-run your CLI command. When execution hits a breakpoint, VS Code pauses and highlights the line. 4. Use the debug toolbar to step over, step into, or continue execution. ## Use VS Code debug tools The following sections describe the various tools supported for VS Code debugging. ### Inspect variables - The Variables pane shows locals, globals, and environment data for the paused frame. - Right-click variables to Add to Watch for persistent tracking across frames. ### Evaluate expressions - Use the Debug Console to run Python expressions in the paused context (for example, import os then os.environ["DATAROBOT_ENDPOINT"] ). - Add expressions to Watch when you need to monitor them while stepping through code. ### Review the call stack - The Call Stack view shows every frame leading to the breakpoint. - Selecting a frame updates the editor, Variables pane, and Debug Console context so you can inspect earlier calls. ## Common issues ### Launch configuration missing Issue: The Run Agent option does not appear. Solution: - Confirm .vscode/launch.json exists inside your workspace and contains the configuration. - Use the Command Palette option Python Debugger: Debug using launch.json to regenerate the file if it was deleted. ### Wrong interpreter Issue: VS Code cannot import packages or shows Python version errors. Solution: - Run Python: Select Interpreter and choose the .venv interpreter created by dr start . - If you rebuilt the environment, reload the window ( Developer: Reload Window ) and reselect the interpreter. - Confirm the python entry in .vscode/launch.json points to the same .venv/bin/python path you selected so the debugger starts with the correct runtime. ### Environment variables not loading Issue: The agent fails because credentials are missing during debugging. Solution: - Confirm the .env file exists at the repository root and contains DATAROBOT_API_TOKEN and DATAROBOT_ENDPOINT . - Verify the envFile entry in launch.json matches the .env path. ### CLI finishes without hitting breakpoints Issue: The CLI command completes but the debugger never pauses where expected. Solution: - Make sure the debugger is running (green bar in VS Code) before firing the CLI command. - Verify the breakpoint is solid red (not hollow) and located in code that executes for the chosen prompt. - Rerun the CLI command after restarting the debugger; the dev server handles one request at a time. ### Debugger not attaching Issue: VS Code launches but never shows the server banner or stops at breakpoints. Solution: - Stop the session and start it again to ensure dev.py restarted cleanly. - Check that program and cwd reference the same agent directory; mismatched paths cause VS Code to debug the wrong copy of the code. - Inspect the Debug Console for stack traces indicating missing dependencies—rerun dr start if needed. --- # Customize agents URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-development.html > Learn how to modify agent code, test locally, and deploy agentic workflows for production use. Developing an agent involves editing code in the `agent/agent/` directory (primarily `myagent.py`). A variety of tools and commands are provided to help you test and deploy your agent during the development process. > [!NOTE] Generic base template > You can use the `generic_base` template to build an agent using any framework of your choice; however, you need to implement the agent logic and structure yourself, as this template does not include any pre-defined agent code. ## Modify the agent code The first step in developing your agent is to modify the agent code to implement your desired functionality. The main agent code is located in the `agent/agent` directory in your application project. ``` # agent/agent/ directory agent/agent/ ├── __init__.py # Package initialization ├── myagent.py # Main agent implementation, including prompts ├── config.py # Configuration management ├── register.py # DRAgent / NAT registration (framework-specific) ├── workflow.yaml # Declarative workflow config for DRAgent (framework-specific) └── model-metadata.yaml # Agent metadata configuration ``` | File | Description | | --- | --- | | __init__.py | Identifies the directory as a Python package and enables imports. | | model-metadata.yaml | Defines the agent's configuration, runtime parameters, and deployment settings. | | custom.py (in the parent agent/, one level up) | Implements DataRobot integration hooks (load_model, chat) for agent execution. | | myagent.py | Contains the main agent implementation for your framework. In LangGraph, this is typically a graph_factory function plus MyAgent = datarobot_agent_class_from_langgraph(...) rather than a hand-written MyAgent subclass (see the LangGraph tab below). | | config.py | Manages configuration loading from environment variables, runtime parameters, and DataRobot credentials. | | register.py | Wires the DRAgent front server to your framework: LLM selection, MCP tools (mcp_tools_context), and optional workflow tools. | | workflow.yaml | Declares workflow type, LLM component, and optional A2A metadata for the DRAgent server. | The primary implementation you edit is in `myagent.py`; the exact pattern depends on the framework. > [!NOTE] Agentic Starter template updates (11.8.x) > In current [Agentic Starter](https://github.com/datarobot-community/datarobot-agent-application) releases, agent templates (except the generic base template) build `MyAgent` from native framework primitives using helper factories, and MCP tooling is decoupled from the agent class. If you are upgrading an older clone, see [Migrate Agentic Starter agents to 11.8.8](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-starter-migration.html) and the [framework migration guides](https://github.com/datarobot-community/datarobot-agent-application/tree/main/docs/agent) under `docs/agent/` in the repository for more details. The agent template provides a simple multi-step example (for example, a planner node and a writer node in LangGraph). You can modify this code to add more agents, tasks, and tools as needed. LLM selection is not limited to an `llm()` method on a handwritten class. It is usually driven by a combination of the following: - DataRobot runtime and configuration — how the platform resolves the chat model and related settings for your workflow. - Adaptor helpers (LangGraph) — for example, get_llm in the DRUM/DRAgent adaptor path, which supplies the model to your graph. - Declarative and environment settings — for example, workflow.yaml (DRAgent) and environment variables that influence provider and model choice. For details on configuring LLMs, see [Configuring LLM providers](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-llm-providers.html). For the overall structure of agentic workflow templates, see [Agent components](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-overview.html). > [!NOTE] datarobot_genai package > Agent templates use the `datarobot_genai` package to streamline development. This package provides helper functions and base classes that simplify agent implementation, including LLM configuration, response formatting, and integration with DataRobot services. The templates automatically include this package, so you don't need to install it separately. ## Modify agent prompts Each agent template uses different approaches for defining and customizing prompts. Understanding how to modify prompts in your chosen framework is crucial for tailoring agent behavior to your specific use case. **CrewAI:** In CrewAI templates, prompts are defined through several properties in the `MyAgent` class within the `myagent.py` file: Agent prompts : Defined using role , goal , and backstory properties. Task prompts : Defined using description and expected_output properties. ``` @property def agent_planner(self) -> Agent: return Agent( role="Content Planner", goal="Plan engaging and factually accurate content on {topic}", backstory="You're working on planning a blog article about the topic: {topic}. You collect " "information that helps the audience learn something and make informed decisions. Your work is " "the basis for the Content Writer to write an article on this topic.", allow_delegation=False, verbose=self.verbose, llm=self.llm, ) ``` To modify CrewAI agent prompts: Update agent behavior : Modify the role , goal , and backstory properties in agent definitions. Use variables : Leverage {topic} and other variables for dynamic prompt content. ``` @property def task_plan(self) -> Task: return Task( description=( "1. Prioritize the latest trends, key players, and noteworthy news on {topic}.\n" "2. Identify the target audience, considering their interests and pain points.\n" "3. Develop a detailed content outline including an introduction, key points, and a call to action.\n" "4. Include SEO keywords and relevant data or sources." ), expected_output="A comprehensive content plan document with an outline, audience analysis, SEO keywords, " "and resources.", agent=self.agent_planner, ) ``` To modify CrewAI task prompts: Customize task instructions : Update the description property in task definitions. Change expected outputs : Modify the expected_output property to match your requirements. Use variables : Leverage {topic} and other variables for dynamic prompt content. For more advanced CrewAI prompt engineering techniques, see the [CrewAI Agents documentation](https://docs.crewai.com/en/concepts/agents) and [CrewAI Tasks documentation](https://docs.crewai.com/en/concepts/tasks). **LangGraph:** In current LangGraph templates, you define a prompt template ( `ChatPromptTemplate`), a `graph_factory(llm, tools, verbose)` that builds `create_agent` nodes and wires a `StateGraph`, and then `MyAgent = datarobot_agent_class_from_langgraph(graph_factory, prompt_template)`. The [Agentic Starter template](https://github.com/datarobot-community/datarobot-agent-application) follows this pattern in `myagent.py`; see the [LangGraph guide in the repository](https://github.com/datarobot-community/datarobot-agent-application/blob/main/docs/agent/frameworks/langgraph.md) for the full layout. > [!NOTE] LangChaincreate_agentvs other LangGraph examples > DataRobot templates use `langchain.agents.create_agent` with `system_prompt=` (and optional `make_system_prompt`). Some community examples use `langgraph.prebuilt.create_react_agent`, which takes a `prompt=` argument instead. Those are different APIs; follow `create_agent` / `system_prompt` when editing DataRobot agent code. ``` from datarobot_genai.core.agents import make_system_prompt from datarobot_genai.langgraph.agent import datarobot_agent_class_from_langgraph from langchain.agents import create_agent from langgraph.graph import END, START, MessagesState, StateGraph def graph_factory(llm, tools, verbose=False): planner = create_agent( llm, tools=tools, system_prompt=make_system_prompt("You are a content planner..."), name="planner_agent", debug=verbose, ) writer = create_agent( llm, tools=tools, system_prompt=make_system_prompt("You are a content writer..."), name="writer_agent", debug=verbose, ) workflow = StateGraph(MessagesState) workflow.add_node("planner_node", planner) workflow.add_node("writer_node", writer) workflow.add_edge(START, "planner_node") workflow.add_edge("planner_node", "writer_node") workflow.add_edge("writer_node", END) return workflow MyAgent = datarobot_agent_class_from_langgraph(graph_factory, prompt_template) ``` The `tools` argument includes MCP tools and optional workflow tools: the DRUM path builds them in `custompy_adaptor` with `mcp_tools_context`, and the DRAgent path combines `workflow_tools` and MCP tools in `register.py` before invoking `MyAgent` (see the template's `docs/agent/frameworks/langgraph.md`). To modify LangGraph prompts: Update system prompts : Change the strings passed to make_system_prompt() inside graph_factory . Add task-specific instructions : Include detailed instructions within those system prompt strings. Modify graph structure : Add or reconnect nodes and edges in graph_factory . For more advanced LangGraph prompt engineering techniques, see the [LangGraph documentation](https://langchain-ai.github.io/langgraph/). **LlamaIndex:** In LlamaIndex templates, prompts are defined using the `system_prompt` parameter in `FunctionAgent` definitions within the `MyAgent` class: ``` @property def research_agent(self) -> FunctionAgent: return FunctionAgent( name="ResearchAgent", description="Useful for finding information on a given topic and recording notes on the topic.", system_prompt=( "You are the ResearchAgent that can find information on a given topic and record notes on the topic. " "Once notes are recorded and you are satisfied, you should hand off control to the " "WriteAgent to write a report on the topic. You should have at least some notes on a topic " "before handing off control to the WriteAgent." ), llm=self.llm, tools=[self.record_notes], can_handoff_to=["WriteAgent"], ) ``` To modify LlamaIndex prompts: Update system prompts : Modify the system_prompt string in FunctionAgent definitions. Customize agent descriptions : Update the description parameter to change how agents are identified. Modify handoff behavior : Update the can_handoff_to list and system prompt to control agent workflow. Add tool-specific instructions : Include instructions about when and how to use specific tools. For more advanced LlamaIndex prompt engineering techniques, see the [LlamaIndex prompt engineering documentation](https://docs.llamaindex.ai/en/stable/module_guides/models/prompts/). **NAT:** In NAT (NVIDIA NeMo Agent Toolkit) templates, prompts are defined in the `workflow.yaml` file using the `system_prompt` field within function definitions: ``` functions: planner: _type: chat_completion llm_name: datarobot_llm system_prompt: | You are a content planner. You are working with a content writer colleague. You're working on planning a blog article about the topic. You collect information that helps the audience learn something and make informed decisions. Your work is the basis for the Content Writer to write an article on this topic. 1. Prioritize the latest trends, key players, and noteworthy news on the topic. 2. Identify the target audience, considering their interests and pain points. 3. Develop a detailed content outline including an introduction, key points, and a call to action. 4. Include SEO keywords and relevant data or sources. ``` To modify NAT prompts: Update system prompts : Modify the system_prompt field in function definitions within workflow.yaml . Configure LLM per function : Set the llm_name field to reference an LLM defined in the llms section of workflow.yaml . Modify workflow structure : Update the workflow section to change the execution order and tool list. Add new functions : Define additional functions in the functions section to extend agent capabilities. For more advanced NAT usage instructions, see the [NVIDIA NeMo Agent Toolkit documentation](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html). > [!TIP] Best practices for prompt modification > When modifying prompts across any framework: > > Be specific > : Provide clear, detailed instructions for what you want the agent to accomplish. > Use consistent formatting > : Maintain consistent prompt structure across all agents in your workflow. > Test incrementally > : Make small changes and test them before implementing larger modifications. > Consider context > : Ensure prompts work well together in multi-agent workflows. > Document changes > : Keep track of prompt modifications for future reference and team collaboration. ## Enable streaming responses Streaming allows agents to send responses incrementally as they are generated, rather than waiting for the complete response. This provides a better user experience by showing progress in real-time, reducing perceived latency, and enabling users to see agent actions as they happen. Streaming support varies by agent framework. There are three levels of streaming implementation: - Chunk streaming : Each chunk from the LLM is streamed as it's generated (such as tokens/partial text). - Step streaming : Response from each sub-agent is streamed when ready. - Event streaming : Each individual event (starting new step, calling a tool, reasoning) is streamed. | Framework | Streaming | Notes | | --- | --- | --- | | LangGraph | Enabled | Chunk-level streaming is automatically enabled when stream=True is passed. The generated MyAgent from datarobot_agent_class_from_langgraph handles streaming responses. | | Generic Base | Supported | All streaming levels (chunk, step, event) require custom implementation. Example code is provided in myagent.py for chunk streaming. | | CrewAI | Supported | All streaming levels (chunk, step, event) require custom implementation. Event listeners capture agent execution and tool usage events incrementally, which facilitates step and event streaming with custom code. Chunk streaming requires custom implementation to stream from the LLM directly. | | LlamaIndex | Supported | All streaming levels (chunk, step, event) require custom implementation. The framework executes agents incrementally, which facilitates step streaming with custom code. Chunk and event streaming require custom implementation. | | NAT | Supported | All streaming levels (chunk, step, event) require custom implementation. | > [!NOTE] Infrastructure support > All agent templates include infrastructure in `custom.py` that can handle streaming responses. For frameworks that require custom implementation (Generic Base, CrewAI, LlamaIndex, NAT), you need to modify your agent's `invoke()` method to return an `AsyncGenerator` when streaming is requested. If your agent's `invoke()` method returns an `AsyncGenerator`, the infrastructure automatically converts it to the appropriate streaming response format. The `is_streaming` helper function is available to all framework templates via the `datarobot_genai` package by importing `from datarobot_genai.core.agents import is_streaming`. It checks if `stream=True` is present in the chat completion request body parameters. If streaming is implemented for an agent, enable streaming when testing locally (via CLI) or when making predictions with a deployed agent (via API). **CLI (local testing):** Use the `--stream` flag when running the agent CLI: ``` task agent:cli -- execute --user_prompt 'Write a document about the history of AI.' --stream ``` You can also use streaming with structured queries: ``` task agent:cli -- execute --user_prompt '{"topic":"Generative AI"}' --stream ``` **API (deployed agent):** Set `stream=True` in the completion parameters when making API calls: ``` from openai import OpenAI client = OpenAI( base_url=CHAT_API_URL, api_key=API_KEY, ) completion = client.chat.completions.create( model="datarobot-deployed-llm", messages=[ {"role": "user", "content": "What would it take to colonize Mars?"}, ], stream=True, # Enable streaming ) # Process streaming response for chunk in completion: if chunk.choices[0].delta.content: print(chunk.choices[0].delta.content, end="", flush=True) ``` ## Testing the agent during local development You can test your agent locally using the development server provided in the template. This allows you to run and debug your agent code without deploying it to DataRobot. To submit a test query to your agent, the development server must be running. Start it manually or use the auto-start option: **Manual start:** Start the development server manually when running multiple tests. The development server runs continuously and blocks the terminal, so start it in one terminal: ``` task agent:dev ``` Keep this terminal running. Then, in a different terminal, run your test commands: ``` task agent:cli -- execute --user_prompt 'Write a document about the history of AI.' ``` You can also send a structured query as a prompt if your agentic workflow requires it: ``` task agent:cli -- execute --user_prompt '{"topic":"Generative AI"}' ``` **Automatic start:** Auto-start the development server for single tests. Use `START_DEV=1` to automatically start and stop the development server: ``` task agent:cli START_DEV=1 -- execute --user_prompt 'Write a document about the history of AI.' ``` You can also send a structured query as a prompt if your agentic workflow requires it: ``` task agent:cli START_DEV=1 -- execute --user_prompt '{"topic":"Generative AI"}' ``` This command will run the agent locally and print the output to the console. You can modify the query to test different inputs and scenarios. > [!TIP] Fast iteration with runtime dependencies > For rapid development, you can add Python dependencies without rebuilding the Docker image using runtime dependencies. This process improves iteration speed. See the [Add Python packages](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-python-packages.html) documentation for details on adding runtime dependencies. ## Build an agent for testing in the DataRobot LLM Playground To create a custom model that can be refined using the DataRobot LLM Playground, deploy development infrastructure (including playground-related resources) from your template project root. This is different from [running the stack locally](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-development.html#testing-the-agent-during-local-development) with `task dev` or `dr run dev`. ``` dr run deploy-dev ``` You can also run `dr task run deploy-dev` (equivalent to `dr run deploy-dev`). This command runs Pulumi with development targets (for example, LLM Playground and related custom model resources) and does not perform a full production deployment. This is significantly faster for iterative cloud development and testing. For command details, see [dr task](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/task.html) and [dr run](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/run.html) in the CLI documentation. For more examples on working with agents in the DataRobot LLM Playground, see the [Agentic playground documentation](https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-playground.html). > [!NOTE] Build command considerations > The `deploy-dev` task can replace or update existing cloud resources for your stack. Full production resources are created with `dr run deploy` (or `dr task run deploy`). If resources are removed or recreated, new deployment IDs may apply. ## Deploy an agent for production use To create a full production-grade deployment: ``` dr run deploy ``` You can also run `dr task run deploy` (equivalent to `dr run deploy`). This matches the [Agentic Starter template README](https://github.com/datarobot-community/datarobot-agent-application#deploy-your-agent) ( `dr run deploy`). The command builds the custom model and creates a production deployment with the necessary infrastructure, which takes longer but provides a complete production environment. See the [CLI task and run commands](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/task.html) for more options. The deployment is a standard DataRobot deployment that includes full monitoring, logging, and scaling capabilities. For more information about DataRobot deployments, see the [Deployment documentation](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-overview/index.html). ### View logs and traces for a deployed agent Once your agent is deployed, you can view OpenTelemetry (OTel) logs and traces in the DataRobot UI. To view logs, on the Deployed workloads tab, locate and click your deployment, click the Activity log tab, and then click Logs. The logs display in OpenTelemetry format and include log levels ( `INFO`, `DEBUG`, `WARN`, and `ERROR`), time-period filtering (Last 15 min, Last hour, Last day, or Custom range), and export capabilities via the OTel logs API for integration with third-party observability tools like Datadog. > [!NOTE] Access and retention > OTel logs are available for all deployment and target types. Only users with Owner and User roles on a deployment can view these logs. Logs data is stored for a retention period of 30 days, after which it is automatically deleted. To view traces, which follow the end-to-end path of requests to your agent, open the deployment and click the Tracing tab. The trace list shows each request by root span name, timestamp, and status. Select a trace to inspect latency, token usage, span count, and a combined span list and timeline of the agent's execution, including LLM API calls, tool invocations, and agent actions. Select a span to review attributes, logs, input, and output on the same page. For more information, see the [logs documentation](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-activity-log/nxt-otel-logs.html) and [tracing documentation](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-tracing.html). ### Manually deploy an agent using Pulumi If needed, you can manually run Pulumi commands to debug or refine Pulumi code. ``` # Load environment variables set -o allexport && source .env # For build mode only (custom model without deployment) export AGENT_DEPLOY=0 # Or for full deployment mode (default) # export AGENT_DEPLOY=1 # Navigate to the infrastructure directory cd ./infra # Run Pulumi deployment pulumi up ``` The `AGENT_DEPLOY` environment variable controls whether Pulumi creates only the custom model ( `AGENT_DEPLOY=0`) or both the custom model and a production deployment ( `AGENT_DEPLOY=1`). If not set, Pulumi defaults to full deployment mode. Pulumi will prompt you to confirm the resources to be created or updated. ## Make predictions with a deployed agentic workflow After the agentic workflow is deployed, access real-time prediction snippets from the deployment's Predictions > Prediction API tab. For more information on deployment predictions, see the [Prediction API snippets documentation](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-predictions/nxt-pred-api-snippets.html). Alternatively, you can modify the script below to make predictions with the deployed agentic workflow, replacing the placeholders for the `API_KEY`, `DEPLOYMENT_ID`, and `CHAT_API_URL` variables. ``` # datarobot-llm-chat.py import sys import logging import time import os from openai import OpenAI API_KEY = '' # Your API Key DEPLOYMENT_ID = '' # The agentic workflow deployment ID CHAT_API_URL = '' # The chat API URL for the agentic workflow deployment # For example, 'https://app.datarobot.com/api/v2/deployments/68824e9aa1946013exfc3415/' logging.basicConfig( level=logging.INFO, stream=sys.stdout, format='%(asctime)s %(filename)s:%(lineno)d %(levelname)s %(message)s', ) logger = logging.getLogger(__name__) def main(): openai_client = OpenAI( base_url=CHAT_API_URL, api_key=API_KEY, _strict_response_validation=False ) prompt = "What would it take to colonize Mars?" logging.info(f"Trying Simple prompt first: \"{prompt}\"") completion = openai_client.chat.completions.create( model="datarobot-deployed-llm", messages=[ {"role": "system", "content": "Explain your thoughts using at least 100 words."}, {"role": "user", "content": prompt}, ], max_tokens=512, # omit if you want to use the model's default max ) print(completion.choices[0].message.content) return 0 if __name__ == '__main__': sys.exit(main()) ``` ## Next steps After deployment, your agent will be available in your DataRobot environment. You can: 1. Test your deployed agent using task agent:cli -- execute-deployment . 2. Integrate your agent with other DataRobot services. 3. Monitor usage and performance in the DataRobot dashboard. For agentic platform-specific assistance beyond the scope of the examples provided in this repository, see the official documentation for each framework: - CrewAI - LangGraph - LlamaIndex - NVIDIA NeMo Agent Toolkit You can also find more examples and documentation in the public repositories for specific frameworks to help you build more complex agents, add tools, and define workflows and tasks. - CrewAI GitHub repository - LangGraph GitHub repository - LlamaIndex GitHub repository --- # Get started URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html > Install the required components and learn how to build, deploy, and test agentic workflows using DataRobot's pre-built templates for popular AI agent frameworks. The [Agentic Starter template repository](https://github.com/datarobot-community/datarobot-agent-application) provides a ready-to-use application template for building and deploying agentic workflows with multi-agent frameworks, a FastAPI backend server, a React frontend, and an MCP server. The template streamlines the process of setting up new agentic applications with minimal configuration requirements and supports local development and testing, as well as one-command deployments to production environments within DataRobot. This guide covers installing the prerequisite tools and configuring your environment, then creating, deploying, and testing an agentic application using DataRobot's pre-built templates. > [!NOTE] Before you begin > Install DataRobot agent skills in your coding agent—see [Install from agent marketplaces](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-skills.html#install-from-agent-marketplaces) for Claude Code, Cursor, Codex, and Gemini CLI marketplace options. Run the `datarobot-setup` skill once per workspace to configure your DataRobot API token and development environment. ## System requirements Ensure your system meets the minimum requirements for running the Agentic Starter template: - Operating system: macOS, Linux, or Windows - Python: Version 3.10 or higher - Memory: At least 4 GB of RAM > [!NOTE] Windows development > On Windows, complete the [Windows prerequisites](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html#windows-prerequisites) before you clone the repository or run `dr start`. The rest of the installation and quickstart flow matches macOS and Linux. > > If you prefer a Linux environment on Windows, you can also use a [DataRobot codespace](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html), [Windows Subsystem for Linux (WSL)](https://learn.microsoft.com/en-us/windows/wsl/install), a dev container, or a virtual machine. > [!NOTE] Restricted network environments > If you're working from an environment with no direct internet access (for example, an air-gapped environment), see [Restricted network setup](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html#restricted-network-setup) before you install, initialize, or deploy. ## Install prerequisite tools Before you begin, you'll need the following tools installed. If you already have these tools installed, ensure that they are at the required version (or newer) indicated in the table below. For example commands to install the tools, see the [Detailed installation commands](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html#detailed-installation-commands) section. > [!TIP] Install tools system-wide > Make sure to install the tools system-wide, rather than in a virtual environment, so they are available in your terminal sessions. | Tool | Version | Description | Installation guide | | --- | --- | --- | --- | | dr-cli | >= 0.2.77 | The DataRobot CLI. | DataRobot CLI getting started (install and configure); GitHub dr-cli (alternative) | | git | >= 2.30.0 | A version control system. | git installation guide | | uv | >= 0.10.0 | A Python package manager. | uv installation guide | | Pulumi | >= 3.163.0 | An Infrastructure as Code tool. | Pulumi installation guide | | Taskfile | >= 3.43.3 | A task runner. | Taskfile installation guide | | NodeJS | >= 24 | JavaScript runtime for frontend development. | NodeJS installation guide | | C++ build tools | N/A | A C++ compiler and build tools, required to compile some Python packages. | macOS: Xcode Command Line Tools (xcode-select --install); Linux: build-essential (sudo apt-get install build-essential); Windows: Visual Studio Build Tools with the Desktop development with C++ workload | ### Windows prerequisites Complete these steps on Windows before you clone the repository. Skipping them checks symlinks out as plain text files and leaves the working tree broken. 1. Enable symlink support in Git: ``` git config --global core.symlinks true ``` 1. Grant permission to create symlinks using one of the following options: 2. Developer Mode (recommended): On Windows 11, openSettings → System → Advanced → Developer Modeand turn Developer Mode on. SeeEnable your device for developmentfor details. 3. Administrator terminal: Launch PowerShell or Windows Terminal withRun as administratorand run every repo operation from that elevated session. At minimum, use an elevated session forgit clone,dr start,dr run deploy, and anygit checkoutorgit pullthat touches symlinked paths. This template uses Git symlinks at.claude/skills,fastapi_server/core,infra/infra/llm.py, andinfra/infra/oauth.py. 4. After Git is installed, ensure that the following directory is present in your system PATH:[path_of_git_installation]\usr\bin. For example, if Git is installed inC:\Program Files\Git, addC:\Program Files\Git\usr\binto your PATH using the following commands: ``` $dir = 'C:\Program Files\Git\usr\bin' # Change if Git is installed elsewhere. $p = [Environment]::GetEnvironmentVariable('PATH', 'User') [Environment]::SetEnvironmentVariable('PATH', "$p;$dir", 'User') ``` This location provides Linux helper commands required for the Agentic Starter to work correctly. Close and reopen the terminal (or IDE) after running the command so new processes pick up the change. ### Detailed installation commands The following sections provide example installation commands for macOS, Linux (Debian/Ubuntu/DataRobot codespace), and Windows (PowerShell). Click the tab below that corresponds to your operating system: **macOS:** macOS users can install the prerequisite tools using Homebrew. First, install Homebrew if you don't already have it. ``` /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # If homebrew is not already installed ``` Then, install the prerequisite tools with it: ``` brew install datarobot-oss/taps/dr-cli uv pulumi/tap/pulumi go-task node git ``` You also need the Xcode Command Line Tools to compile some Python packages: ``` xcode-select --install ``` **Linux:** Linux users can install the prerequisite tools using the package manager for their distribution. ``` curl https://cli.datarobot.com/install | sh sudo apt-get update sudo apt-get install -y python3 python3-pip python3-venv sudo apt-get install -y build-essential sudo apt-get install -y git curl -LsSf https://astral.sh/uv/install.sh | sh curl -fsSL https://get.pulumi.com | sh sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d sudo apt-get install -y nodejs npm ``` **Windows (PowerShell):** Complete [Windows prerequisites](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html#windows-prerequisites) before you run `dr start`. Windows users can install the prerequisite tools with PowerShell: ``` irm https://cli.datarobot.com/winstall | iex powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" winget install Git.Git winget install Pulumi.Pulumi winget install Task.Task winget install OpenJS.NodeJS winget install Microsoft.VisualStudio.2022.BuildTools --force --override "--wait --passive --add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 --add Microsoft.VisualStudio.Component.Windows11SDK.22621" ``` > [!NOTE] After installing the tools > uv: Run > uv tool update-shell > once so your shell picks up the updated > PATH > before using > uv tool run > or invoking tools installed via > uv tool install > . > Pulumi: If you don't have a Pulumi account, use > pulumi login --local > for local login, or create a free account at > the Pulumi website > . ## Initialize your application > [!WARNING] Installation process > Before starting, complete all installation and setup steps above. Skipping this process can cause errors and prevent your agentic application from running correctly. Run the following command to start the local development environment: ``` dr start ``` This command starts the DataRobot CLI's interactive wizard ( [dr start](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/start.html) in the [CLI command reference](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html)). It automatically clones the application repository and creates a `.env` file in the root directory populated with environment variables you specify. The wizard provides guidance and context for each step; expand the reference below for the complete list of steps. **Full wizard step reference** 1. Specify whether you wish to use the "low-code" agent template: 2. After a few moments, the wizard opens a web browser window to automatically configure your API endpoint and key. Click Proceed to continue. 3. Specify the port for the local web application and press Enter . The default is 8842 . 4. If desired, specify the default execution environment for your agent and press Enter . The default is [DataRobot] Python 3.11 GenAI Agents . 5. Provide a secret key to sign cookies for your session and press Enter . If you do not provide a value, a randomly generated one is used. 6. Enter the URI for a database to use for the application and press Enter . The default is sqlite+aiosqlite:///.data/database.sqlite . 7. Select your backend OAuth provider and press Enter . 8. Specify your authorization server by selecting it from the list and pressing Space . Press Enter to confirm. 9. Enter a passphrase (or leave blank if you don't want to use a passphrase) for your Pulumi stack and press Enter . 10. Specify the ID of a DataRobot Use Case (for example, 69331fad5e07469e7c4f5c6f ), if one is available, and press Enter . 11. Specify your LLM integration and press Enter . 12. Specify the port for the MCP server and press Enter . The default is 9000 . 13. Review the .env configuration summary displayed and press Enter to confirm. 14. Once the configuration finishes, choose a Pulumi stack to use for your application and press Enter . If you wish to create a new stack, press Enter and you are prompted to enter a name for it. The name cannot match any existing stack name. > [!NOTE] First-time initialization > When run for the first time, the `dr start` command prepares your development environment to develop and deploy your application. > This includes both environment and agent component configuration. > After this first initialization, future `dr start` operations only set up your local environment. > For subsequent updates to the configuration of your agent component, run the `dr component update` command. After `dr start` completes successfully, verify the following: - A .env file in your project root. - Your application directory created (typically named datarobot-agent-application or based on your application name). Now that your application is configured, proceed to the next section. ## Run your agent > [!WARNING] Running your agent > Do not proceed to this section until you have run `dr start`, detailed in the previous section. Navigate to the application directory created during `dr start`: ``` cd datarobot-agent-application # or the custom directory name you specified during the wizard, if different ``` Then, run the following command to start all components of the application: ``` dr run dev ``` > [!NOTE] Note > `task dev` runs the same development stack if your template exposes that task. This starts four processes, running in parallel: - Application frontend - Application backend - Agent - MCP server Once all services are running: 1. Open your web browser and navigate to http://localhost:5173 . 2. Confirm that the agent application interface appears. 3. Try sending a test message to verify everything is working. From here, start customizing your agent by adding your own logic and functionality. See the [Develop your agent](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html#develop-your-agent) section for more details. > [!NOTE] Starting individual services > Start individual services in separate terminal windows; for example, `task agent:dev` starts only the agent. ## Develop your agent Now that your agent has been built and tested, you are ready to customize it by adding your own logic and functionality. See the following documentation for more details: - Customize your agent - Add tools to your agent - Configure LLM providers - Add Python requirements - Manage prompts ## Deploy your agent > [!WARNING] Testing your agent > Ensure that you have tested your agent locally before deploying. Next, deploy your agent to DataRobot, which requires a Pulumi login. Run the following command to deploy your agent: ``` dr task run deploy ``` For more on the `task` and `run` commands, see the [CLI task command](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/task.html) and [CLI run command](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/run.html). > [!NOTE] Deployment process > The deployment process takes several minutes to complete. Once deployment is complete, the script displays the deployment details, as shown in the example below. The deployment details vary based on your configuration. ``` Outputs: AGENT_DEPLOYMENT_ID : "69331fad5e07469e7c4f5c6f" Agent Custom Model Chat Endpoint [apptest] [agent]: "https://datarobot.com/api/v2/genai/agents/fromCustomModel/69331f816e1bf9f1890d5d1d/chat/" Agent Deployment Chat Endpoint [apptest] [agent] : "https://datarobot.com/api/v2/deployments/69331fad5e07469e7c4f5c6f/chat/completions" Agent Execution Environment ID [apptest] [agent] : "680fe4949604e9eba46b1775" Agent Playground URL [apptest] [agent] : "https://datarobot.com/usecases/69331e4c3be0efe3b95a7be0/agentic-playgrounds/69331e4d1c036307186c9b16/comparison/chats" Agentic Starter [apptest] : "https://datarobot.com/custom_applications/6933204a9e21e9b59b5a7bee/" DATABASE_URI : "sqlite+aiosqlite:////tmp/agent_app/.data/agent_app.db" DATAROBOT_APPLICATION_ID : "6933204a9e21e9b59b5a7bee" DATAROBOT_OAUTH_PROVIDERS : (json) [] LLM_DEFAULT_MODEL : "azure/gpt-4o-2024-11-20" SESSION_SECRET_KEY : "secretkey123" USE_DATAROBOT_LLM_GATEWAY : "1" [apptest] [mcp_server] Custom Model Id : "69331eebb49131d3d5430ac7" [apptest] [mcp_server] Deployment Id : "69331f1f30548f83b668d9dc" [apptest] [mcp_server] MCP Server Base Endpoint : "https://datarobot.com/api/v2/deployments/69331f1f30548f83b668d9dc/directAccess/" [apptest] [mcp_server] MCP Server MCP Endpoint : "https://datarobot.com/api/v2/deployments/69331f1f30548f83b668d9dc/directAccess/mcp" ``` > [!NOTE] Note > The sample output above reflects an agent using the LLM gateway ( `USE_DATAROBOT_LLM_GATEWAY` is `"1"`). If you use the DataRobot Deployed LLM option instead, `USE_DATAROBOT_LLM_GATEWAY` is automatically set to `0`. ## Restricted network setup Complete this section only if you're installing, initializing, or deploying from an environment with no direct internet access (for example, an air-gapped environment); otherwise, skip it. Configure Pulumi to install the DataRobot plugin from an internal proxy instead of GitHub. Setting these environment variables redirects all Pulumi plugin downloads to your internal proxy and disables external update checks. Add the following variables to your `.env` file: ``` # .env PULUMI_SKIP_UPDATE_CHECK=1 PULUMI_DATAROBOT_DEFAULT_URL=http://internal-proxy-for-pulumi # OPTIONAL PULUMI_DATAROBOT_PLUGIN_VERSION=v0.10.27 ``` | Environment variable | Required | Description | | --- | --- | --- | | PULUMI_SKIP_UPDATE_CHECK | Yes | Enables air-gapped mode when set to 1, disabling external update checks and allowing use of a custom plugin server. | | PULUMI_DATAROBOT_DEFAULT_URL | Yes | The base URL of your internal proxy server hosting the DataRobot Pulumi plugin. This replaces the default GitHub releases source. | | PULUMI_DATAROBOT_PLUGIN_VERSION | No | The specific version of the DataRobot Pulumi plugin to install. If not specified, it defaults to the version bundled with the templates. | > [!NOTE] How it works > When `PULUMI_SKIP_UPDATE_CHECK=1` is set, deployment tasks execute `pulumi plugin install resource datarobot --server `. This ensures that plugin downloads are routed through your internal proxy instead of external sources. > [!NOTE] Internal proxy requirements > The internal proxy must host the DataRobot Pulumi plugin files in a structure compatible with Pulumi's plugin installation. It should mirror the directory and file structure of the [official GitHub releases](https://github.com/datarobot-community/pulumi-datarobot/releases). ### Python packages In restricted network environments, `uv sync` operations fail when attempting to reach the public PyPI. To resolve this, configure `uv` to use your internal PyPI proxy. To configure the proxy, edit the `agent/pyproject.toml` file in your agent project and uncomment the `[tool.uv.pip]` section, replacing the URL with your internal PyPI proxy. For example: ``` # agent/pyproject.toml [tool.uv.pip] extra-index-url = ["https://your-internal-pypi-proxy.example.com/simple/"] ``` > [!NOTE] Configuration impact > Once configured, this setting ensures that all Python package installations are routed through your proxy. This applies to: > > Local development ( > uv sync > ) > Docker image builds > Custom model deployments > Playground operations > Infrastructure deployments > [!TIP] Finding the configuration > The `[tool.uv.pip]` section is located at the end of the `pyproject.toml` file. If it is missing, add it manually. --- # Configure LLM provider fallback URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-llm-fallback.html > Learn how to configure primary and fallback LLM providers for automatic failover in your agentic workflows. You can configure a primary LLM provider with one or more fallback providers for automatic failover. When the primary is unavailable or returns an error, a `litellm.Router` automatically retries with the next fallback provider in the list. The `num_retries` option controls how many retries occur per provider before moving to the next. This works with any DataRobot-supported LLM provider, including the LLM gateway, hosted deployments, NIM deployments, and external APIs. ## Prerequisites - datarobot-genai>=0.15.20 must be available in the execution environment. See Add Python packages for instructions. - A working agent template (CrewAI, LangGraph, LlamaIndex, or DRAgent/NAT). - At least two LLM providers or models configured (one primary, one or more fallbacks). ## Configure fallback in code For DRUM-based templates (CrewAI, LangGraph, LlamaIndex), replace `get_llm()` with `get_router_llm()` in your `myagent.py` file. ### LLMConfig fields The `primary` LLM and each fallback (in `fallbacks`) is defined as an `LLMConfig` object. Set only the fields relevant to your provider type (see [Mixing provider types](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-llm-fallback.html#mixing-provider-types) for a multi-fallback example): | Field | Type | Description | | --- | --- | --- | | use_datarobot_llm_gateway | bool | Use the DataRobot LLM gateway as the provider. | | llm_default_model | str | The model identifier (e.g., azure/gpt-4o-mini). | | llm_deployment_id | str | DataRobot deployment ID for hosted LLM deployments. | | nim_deployment_id | str | DataRobot NIM deployment ID. | | datarobot_endpoint | str | DataRobot API endpoint URL. | | datarobot_api_token | str | DataRobot API token. | ### Framework examples The LLM fallback system follows a similar pattern for each DRUM-based template: **CrewAI:** ``` from datarobot_genai.core.config import LLMConfig from datarobot_genai.crewai.llm import get_router_llm primary = LLMConfig( use_datarobot_llm_gateway=True, llm_default_model="{LLM_DEFAULT_MODEL}", ) fallbacks = [ LLMConfig( use_datarobot_llm_gateway=True, llm_default_model="anthropic/claude-opus-4-20250514", ) ] llm = get_router_llm(primary, fallbacks, {"num_retries": 1}) ``` **LangGraph:** ``` from datarobot_genai.core.config import LLMConfig from datarobot_genai.langgraph.llm import get_router_llm primary = LLMConfig( use_datarobot_llm_gateway=True, llm_default_model="{LLM_DEFAULT_MODEL}", ) fallbacks = [ LLMConfig( use_datarobot_llm_gateway=True, llm_default_model="anthropic/claude-opus-4-20250514", ) ] llm = get_router_llm(primary, fallbacks, {"num_retries": 1}) ``` **LlamaIndex:** ``` from datarobot_genai.core.config import LLMConfig from datarobot_genai.llamaindex.llm import get_router_llm primary = LLMConfig( use_datarobot_llm_gateway=True, llm_default_model="{LLM_DEFAULT_MODEL}", ) fallbacks = [ LLMConfig( use_datarobot_llm_gateway=True, llm_default_model="anthropic/claude-opus-4-20250514", ) ] llm = get_router_llm(primary, fallbacks, {"num_retries": 1}) ``` > [!TIP] Multiple fallbacks > You can specify multiple fallback providers in the `fallbacks` list. The router tries them in order if the primary fails. ## Configure fallback in workflow.yaml For DRAgent/NAT templates, use `_type: datarobot-llm-router` with `primary` and `fallbacks` blocks in `workflow.yaml`: ``` # workflow.yaml llms: datarobot_llm: _type: datarobot-llm-router primary: use_datarobot_llm_gateway: true llm_default_model: "{LLM_DEFAULT_MODEL}" fallbacks: - use_datarobot_llm_gateway: true llm_default_model: anthropic/claude-opus-4-20250514 num_retries: 1 ``` > [!NOTE] LLMConfig fields in YAML > The `primary` and each item in `fallbacks` accept the same fields as `LLMConfig`: `use_datarobot_llm_gateway`, `llm_default_model`, `llm_deployment_id`, `nim_deployment_id`, `datarobot_endpoint`, and `datarobot_api_token`. ## Mixing provider types The primary and fallback providers can use different provider types. For example, you can use the LLM gateway as primary and a deployment as fallback: **Code (DRUM-based):** ``` primary = LLMConfig( use_datarobot_llm_gateway=True, llm_default_model="azure/gpt-4o-mini", ) fallbacks = [ LLMConfig( llm_deployment_id="YOUR_DEPLOYMENT_ID", ), LLMConfig( use_datarobot_llm_gateway=True, llm_default_model="anthropic/claude-opus-4-20250514", ), ] llm = get_router_llm(primary, fallbacks, {"num_retries": 1}) ``` **workflow.yaml (DRAgent/NAT):** ``` llms: datarobot_llm: _type: datarobot-llm-router primary: use_datarobot_llm_gateway: true llm_default_model: azure/gpt-4o-mini fallbacks: - llm_deployment_id: YOUR_DEPLOYMENT_ID - use_datarobot_llm_gateway: true llm_default_model: anthropic/claude-opus-4-20250514 num_retries: 1 ``` > [!WARNING] Retry and latency > Each retry adds latency to the response. Set `num_retries` conservatively (e.g., `1`) to balance reliability and response time. --- # Configure LLM providers with metadata URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-llm-providers-metadata.html > Learn how to configure different LLM providers for your agentic workflows including DataRobot LLM gateway, external APIs, and custom deployments. One of the key components of an LLM agent is the underlying LLM provider. DataRobot allows users to connect to virtually any LLM backend for their agentic workflows. LLM connections can be simplified by using the DataRobot LLM gateway or a DataRobot deployment (including NIM deployments). Alternatively, you can connect to any external LLM provider that supports the OpenAI API standard. The DataRobot Agentic Starter template provides multiple methods for defining an agent LLM: - Use the DataRobot LLM gateway as the agent LLM, allowing you to use any model available in the gateway. - Connect to use a previously-deployed custom model or NIM using the DataRobot API by providing the deployment ID. - Connect directly to an LLM provider API (such as OpenAI, Anthropic, or Gemini) by providing the necessary API credentials, enabling access to providers supporting a compatible API. This document focuses on configuring LLM providers using environment variables and Pulumi (infrastructure-level configuration). This approach allows you to switch between different LLM provider configurations without modifying your agent code. The infrastructure configuration files use the `build_llm()` helper function from `datarobot_genai` package, which automatically handles deployment detection, gateway configuration, and credential management. This simplifies deployment and credential management by leveraging DataRobot's secure credential system. > [!NOTE] Alternative configuration method > If you prefer to manually create LLM instances directly in your `myagent.py` file for fine-grained control, see [Configure LLM providers with code](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-llm-providers.html) for details on that approach. ## Infrastructure-level LLM configuration You can configure LLM providers at the infrastructure level using environment variables and symlinks. This allows you to switch between different LLM provider configurations without modifying your agent code. Both methods create a symbolic link from `infra/infra/llm.py` to your chosen configuration file in `infra/configurations/llm/`. The available configuration options include: | Configuration File | Description | | --- | --- | | gateway_direct.py | A direct LLM gateway integration. | | deployed_llm.py | An LLM deployed in DataRobot. | | blueprint_with_external_llm.py | An LLM blueprint with external LLMs (Azure OpenAI, Amazon Bedrock, Google Gemini Enterprise Agent Platform (formerly Vertex AI), Anthropic). | ### Manual symlink Create the symlink manually for explicit control and immediate visibility of your active configuration. This method is recommended for development. To use this method, navigate to the `infra/infra` folder and create a symbolic link to your chosen configuration: ``` cd infra/infra ln -sf ../configurations/llm/.py ``` Replace `` with one of the available configuration files (e.g., `gateway_direct.py`, `blueprint_with_llm_gateway.py`, etc.). After creating the symlink, you can optionally edit `infra/infra/llm.py` to adjust model parameters ( `temperature`, `top_p`, etc.) or select a specific model. ### Environment variable Set the configuration dynamically using an environment variable. The symlink is automatically created when you run Pulumi commands. This method is recommended for deployment and different environments. To use this method, uncomment the relevant section in your `.env` file. The following examples from `.env.template` show how to configure each LLM provider type. Each configuration may require additional environment variables: > [!TIP] DataRobot credentials in codespaces > If you are using a DataRobot codespace, remove the `DATAROBOT_API_TOKEN` and `DATAROBOT_ENDPOINT` environment variables from the file, as they already exist in the codespace environment. ``` # Your DataRobot API token. # Refer to https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html#configure-your-environment for help. DATAROBOT_API_TOKEN= # The URL of your DataRobot instance API. DATAROBOT_ENDPOINT=https://app.datarobot.com/api/v2 # The Pulumi stack name to use for this project. PULUMI_STACK=dev # If empty, a blank passphrase will be used PULUMI_CONFIG_PASSPHRASE=123 # Set to 1 to skip Pulumi update check and prevent issues with rate limiting PULUMI_SKIP_UPDATE_CHECK=0 # If empty, a new use case will be created DATAROBOT_DEFAULT_USE_CASE= # If empty, a new execution environment will be created for each agent using the docker_context folder DATAROBOT_DEFAULT_EXECUTION_ENVIRONMENT="[DataRobot] Python 3.11 GenAI Agents" # This is set to a specific version of `[DataRobot] Python 3.11 GenAI Agents` to preserve compatibility of the templates DATAROBOT_DEFAULT_EXECUTION_ENVIRONMENT_VERSION_ID="6936d3af440bcb12397f4203" # LLM Configuration: # Agent templates support multiple flexible LLM options including: # - LLM Gateway Direct (default) # - LLM Blueprint with an External LLM # - Already deployed LLM in DataRobot # # You can edit the LLM configuration by manually changing which configuration is # active (recommended option). # Simply run `ln -sf ../configurations/llm/.py` # from the `infra/infra` folder # # If you want to do it dynamically however, you can also set it as a configuration value with: # INFRA_ENABLE_LLM= # from the list of options in the infra/configurations/llm folder # Here are some examples of each of those configuration using the dynamic option described above: # If you want to use the LLM gateway direct (default) # INFRA_ENABLE_LLM=gateway_direct.py # If you want to choose an existing LLM Deployment in DataRobot # uncomment and configure these: # LLM_DEPLOYMENT_ID= # INFRA_ENABLE_LLM=deployed_llm.py # If you want to configure an LLM with an external LLM provider # like Azure, Bedrock, Anthropic, or Google Gemini Enterprise Agent Platform (formerly Vertex AI), or all 4. Here is an # Azure AI example, see: # https://docs.datarobot.com/en/docs/gen-ai/playground-tools/deploy-llm.html # for details on other providers and details: # INFRA_ENABLE_LLM=blueprint_with_external_llm.py # LLM_DEFAULT_MODEL="azure/gpt-4o" # OPENAI_API_VERSION='2024-08-01-preview' # OPENAI_API_BASE='https://.openai.azure.com' # OPENAI_API_DEPLOYMENT_ID='' # OPENAI_API_KEY='' ``` When you run `task infra:build` or `task infra:deploy`, the system reads `INFRA_ENABLE_LLM`, automatically creates or updates the symlink to the specified configuration file, and manages credentials through DataRobot's secure credential system. ## LLM configuration options | Configuration File | Description | | --- | --- | | gateway_direct.py | The default option if not specified in your .env file. Direct LLM gateway integration providing streamlined access to LLMs proxied via DataRobot. Available for both cloud and self-managed users. Requires no additional configuration beyond selecting this file. When using LLM gateway options, your agents can dynamically use any and all models available in the gateway catalog. Each agent or task can specify its own preferred model, allowing you to optimize for different capabilities (e.g., faster models for planning, more capable models for content generation). | | deployed_llm.py | Use a previously deployed DataRobot LLM. Requires LLM_DEPLOYMENT_ID (your deployment ID). | | blueprint_with_external_llm.py | Configure an LLM with an external provider like Azure, Bedrock, Anthropic, or Google Gemini Enterprise Agent Platform (formerly Vertex AI). Requires LLM_DEFAULT_MODEL (e.g., "azure/gpt-4o"). Unlike LLM gateway options, all agents use the same model specified in your configuration. The LLM_DEFAULT_MODEL becomes the primary model since you're connecting to a single external deployment. See the DataRobot documentation for details on other providers. | > [!NOTE] When should I edit the default model? > The default model behavior varies by configuration type: > > LLM gateway configurations > ( > gateway_direct.py > ): The default model is automatically configured and rarely needs changing. Agents can dynamically use any model from the catalog. > Deployed models > ( > deployed_llm.py > ): The default model is automatically set to match your deployment. No manual changes needed. > External LLM configurations > ( > blueprint_with_external_llm.py > ): Edit the > default_model > to match your external LLM deployment and add credentials to connect. This is the model all agents will use. ### Configure the DataRobot LLM gateway The LLM gateway provides a streamlined way to access LLMs proxied via DataRobot. The gateway is available for both cloud and self-managed users. You can retrieve a list of available models for your account using the following methods: **cURL:** ``` curl -X GET -H "Authorization: Bearer $DATAROBOT_API_TOKEN" "$DATAROBOT_ENDPOINT/genai/llmgw/catalog/" | jq '[.data[] | select(.isActive == true) | .model]' ``` **Python SDK:** ``` from datarobot.models.genai import LLMGatewayCatalog print("\n".join(LLMGatewayCatalog.get_available_models())) ``` #### LLM gateway: multiple graph nodes and model selection With the LLM gateway and the current [Agentic Starter](https://github.com/datarobot-community/datarobot-agent-application) LangGraph layout, model selection works as follows: - Default model source: Your infrastructure configuration and helpers such as infra/configurations/llm/gateway_direct.py determine the default catalog model. - What reaches your graph: The chat model is the llm argument to graph_factory(llm, tools, verbose) . It is resolved by get_llm and related helpers in the DRUM/DRAgent adaptor path before your graph runs. - Changing the model: You can still adjust selection using gateway catalog settings , separate LLM deployments , or custom LLM construction in code. For the code-based approach, see Configure LLM providers with code . The examples below use the same LangGraph primitives as the template ( `create_agent` and `make_system_prompt`), but expressed inside `graph_factory` so they match how MCP tools are supplied via the `tools` parameter. Same model for every node (default pattern).Both `create_agent` calls use the same `llm`, so they share one resolved model (typically your configured default): ``` from datarobot_genai.core.agents import make_system_prompt from langchain.agents import create_agent def graph_factory(llm, tools, verbose=False): planner = create_agent( llm, tools=tools, system_prompt=make_system_prompt( "You are a content planner. Plan engaging and factually accurate content on {topic}." ), name="planner_agent", debug=verbose, ) writer = create_agent( llm, tools=tools, system_prompt=make_system_prompt( "You are a content writer. Write insightful and factually accurate opinion piece about the topic: {topic}." ), name="writer_agent", debug=verbose, ) ... ``` Different gateway models per node (optional).When you use the LLM gateway, you can point each node at a different catalog model by building a `ChatLiteLLM` with an explicit `model=` string (same API base and credentials as `llm()`, different model id). Use catalog IDs from the gateway (for example, values returned by `LLMGatewayCatalog.get_available_models()` in the [examples above](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-llm-providers-metadata.html#configure-the-datarobot-llm-gateway)): ``` from datarobot_genai.core.agents import make_system_prompt from langchain.agents import create_agent from langchain_litellm.chat_models import ChatLiteLLM def _llm_for_gateway_model(self, model: str) -> ChatLiteLLM: """Chat model for a specific LLM gateway catalog model.""" api_base = self.litellm_api_base(self.config.llm_deployment_id) return ChatLiteLLM( model=model, api_base=api_base, api_key=self.api_key, timeout=self.timeout, streaming=True, max_retries=3, ) @property def agent_planner(self) -> Any: return create_agent( self._llm_for_gateway_model("datarobot/azure/gpt-5-mini-2025-08-07"), tools=self.mcp_tools, system_prompt=make_system_prompt( "You are a content planner. Plan engaging and factually accurate content on {topic}." ), name="planner_agent", ) @property def agent_writer(self) -> Any: return create_agent( self._llm_for_gateway_model("datarobot/azure/gpt-4o-2024-11-20"), tools=self.mcp_tools, system_prompt=make_system_prompt( "You are a content writer. Write insightful and factually accurate opinion piece about the topic: {topic}." ), name="writer_agent", ) ``` > [!NOTE] Factory-based LangGraph templates > The current [Agentic Starter](https://github.com/datarobot-community/datarobot-agent-application) LangGraph template does not use `MyAgent` subclasses or `self.mcp_tools`. Implement the same pattern inside `graph_factory(llm, tools, verbose)`: call a shared helper (for example, one that mirrors `self._llm_for_gateway_model` but takes configuration from your environment or from `datarobot_genai.langgraph.llm`) to build each `ChatLiteLLM`, pass `tools=tools`, and wire the returned `create_agent` nodes into your `StateGraph`. Replace the model strings with IDs that are valid for your account and gateway catalog. If you use deployed LLM or external LLM infrastructure configuration instead of the gateway, all nodes typically share one deployment or one external model—see the sections below. The default model in gateway configuration (for example in `gateway_direct.py`) applies when you use `self.llm()` without overriding the model string: ``` default_model: str = "datarobot/azure/gpt-5-mini-2025-08-07" # Used when nodes call self.llm() and for fallbacks ``` ### DataRobot hosted LLM deployments You can easily connect to DataRobot-hosted LLM deployments as an LLM provider for your agents. DataRobot hosted LLMs provide access to moderations, guardrails, and advanced monitoring to help you manage and govern your models. When using a deployed LLM, all agents use the same deployment. The configuration automatically sets the correct model identifier—no manual model configuration needed. You can create LLM deployments in several ways: - From the DataRobot playground : Deploy an LLM from the DataRobot Playground - Hugging Face models : Deploy an open source LLM from the Hugging Face Hub as a Workload After deployment, copy the Deployment ID and add it to the `.env` file: ``` LLM_DEPLOYMENT_ID= INFRA_ENABLE_LLM=deployed_llm.py ``` The default model is automatically configured to match the deployment. All agents will use this deployment. > [!NOTE] LLM gateway flag for deployed LLMs > When you configure the agent with the DataRobot Deployed LLM option, `USE_DATAROBOT_LLM_GATEWAY` is automatically set to `0` so inference uses your deployment rather than the LLM gateway. You do not need to set this value manually for that option. ### Configure external LLMs When configured with an external LLM like Azure OpenAI, Amazon Bedrock, etc. all agents in the workflow use the model specified in your configuration. Unlike the LLM gateway, this connection is to a single specific model deployment. **Azure OpenAI:** Azure OpenAI allows you to deploy OpenAI models in your Azure environment. DataRobot can connect to these deployments using the `blueprint_with_external_llm.py` configuration. In the `.env` file, uncomment and provide the following environment variables: ``` # If you want to configure an LLM with an external LLM provider # like Azure, Bedrock, Anthropic, or Google Gemini Enterprise Agent Platform (formerly Vertex AI), or all 4. Here is an # Azure AI example, see: # https://docs.datarobot.com/en/docs/gen-ai/playground-tools/deploy-llm.html # for details on other providers and details: INFRA_ENABLE_LLM=blueprint_with_external_llm.py LLM_DEFAULT_MODEL="azure/gpt-4o" OPENAI_API_VERSION='2024-08-01-preview' OPENAI_API_BASE='https://.openai.azure.com' OPENAI_API_DEPLOYMENT_ID='' OPENAI_API_KEY='' ``` The `LLM_DEFAULT_MODEL` should match your Azure deployment. For example, if you deployed `gpt-4o` in Azure, use `azure/gpt-4o`. > [!NOTE] Credential management > The API key is securely managed through the DataRobot [Credentials Management](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html#credentials-management) system as an API Token credential type. **Amazon Bedrock:** Amazon Bedrock provides access to foundation models from various providers through AWS. Configuration uses AWS credentials managed securely through DataRobot. In the `.env` file, insert and provide the following environment variables: ``` INFRA_ENABLE_LLM=blueprint_with_external_llm.py AWS_ACCESS_KEY_ID='' AWS_SECRET_ACCESS_KEY='' AWS_REGION_NAME='us-east-1' ``` Then, edit `infra/configurations/llm/blueprint_with_external_llm.py` to specify your Bedrock model: ``` external_model_id: str = "bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0" ``` > [!NOTE] Credential management > The AWS credentials are managed through the DataRobot [Credentials Management](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html#credentials-management) system. This system securely stores your access key, secret key, and optional session token. **Google Vertex AI:** Google Vertex AI provides access to Google's foundation models including Gemini. Configuration uses Google Cloud service account credentials. In the `.env` file, insert and provide the following environment variables: ``` INFRA_ENABLE_LLM=blueprint_with_external_llm.py GOOGLE_SERVICE_ACCOUNT='' # or GOOGLE_APPLICATION_CREDENTIALS='/path/to/service-account.json' GOOGLE_REGION='us-west1' ``` Then, edit `infra/configurations/llm/blueprint_with_external_llm.py` to specify your Vertex AI model: ``` external_model_id: str = "vertex_ai/gemini-2.5-pro" ``` > [!NOTE] Credential management > The Google Cloud Platform credentials are managed through the DataRobot [Credentials Management](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html#credentials-management) system. Provide either the service account JSON content directly ( `GOOGLE_SERVICE_ACCOUNT`) or a path to the JSON file ( `GOOGLE_APPLICATION_CREDENTIALS`). **Anthropic:** Anthropic's Claude models can be accessed directly through their API. In the `.env` file, insert and provide the following environment variables: ``` INFRA_ENABLE_LLM=blueprint_with_external_llm.py ANTHROPIC_API_KEY='' ``` Then, edit `infra/configurations/llm/blueprint_with_external_llm.py`: ``` external_model_id: str = "anthropic/claude-3-5-sonnet-20241022" ``` > [!NOTE] Credential management > The API key is securely managed through the DataRobot [Credentials Management](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html#credentials-management) system as an API Token credential type. ### Other LLM providers The template supports 100+ LLM providers through [LiteLLM](https://docs.litellm.ai/), including: - Cohere : Configure with COHERE_API_KEY - Together AI : Configure with TOGETHERAI_API_KEY - Hugging Face : Configure with appropriate provider credentials - Ollama : For locally hosted models - OpenAI Direct : For direct OpenAI API access For providers not natively supported by the template's configuration files, you can implement direct LiteLLM integration by modifying the agent code and using LLM gateway Direct configuration with custom `def llm()` implementation. #### Direct LiteLLM integration with DataRobot credential management This approach gives you full control over LLM initialization while securely storing credentials in DataRobot's credential management system. The example below illustrates using LangGraph with direct Cohere integration. Select the LLM gateway direct configuration in the `.env` file: ``` INFRA_ENABLE_LLM=gateway_direct.py ``` Add a credential to `model-metadata.yaml` (e.g., `agent/agent/model-metadata.yaml`): ``` runtimeParameterDefinitions: - fieldName: COHERE_API_KEY type: credential - fieldName: COHERE_MODEL type: string defaultValue: "command-r-plus" ``` Update `config.py` to load the credential (e.g., `agent/agent/config.py`): ``` from pydantic import Field class Config(BaseConfig): # ... existing fields ... cohere_api_key: str cohere_model: str = "command-r-plus" ``` Replace the `llm()` method in `agent/agent/myagent.py`: ``` from langchain_openai import ChatOpenAI def llm( self, preferred_model: str | None = None, auto_model_override: bool = True, ) -> ChatOpenAI: """Returns a ChatOpenAI instance configured to use Cohere via LiteLLM. Args: preferred_model: The model to use. If None, uses COHERE_MODEL from config. auto_model_override: Ignored for direct LiteLLM integration. """ return ChatOpenAI( model=self.config.cohere_model, # Override the model with the configured one api_key=self.config.cohere_api_key, base_url="https://api.cohere.ai/v1", # Cohere's OpenAI-compatible endpoint timeout=self.timeout, ) ``` Next, add Pulumi credential management in `infra/configurations/llm/gateway_direct.py`: ``` ...existing code # Create the Cohere credential cohere_credential = datarobot.ApiTokenCredential( resource_name=f"{pulumi.get_project()} Cohere API Key Credential", api_token=os.environ.get("COHERE_API_KEY"), ) # Update the runtime parameters arrays app_runtime_parameters = [ datarobot.ApplicationSourceRuntimeParameterValueArgs( key="COHERE_API_KEY", type="credential", value=cohere_credential.id, ), datarobot.ApplicationSourceRuntimeParameterValueArgs( key="COHERE_MODEL", type="string", value=os.environ.get("COHERE_MODEL", "command-r-plus"), ), ] custom_model_runtime_parameters = [ datarobot.CustomModelRuntimeParameterValueArgs( key="COHERE_API_KEY", type="credential", value=cohere_credential.id, ), datarobot.CustomModelRuntimeParameterValueArgs( key="COHERE_MODEL", type="string", value=os.environ.get("COHERE_MODEL", "command-r-plus"), ), ] ``` Finally, set environment variables in the `.env` file: ``` INFRA_ENABLE_LLM=gateway_direct.py COHERE_API_KEY='' COHERE_MODEL='command-r-plus' ``` This approach gives you full control over LLM initialization and credential management while still using DataRobot's secure credential storage. The credentials are managed as runtime parameters and never hard coded in your application code. ## Deploy configuration changes After changing any LLM configuration (updating `.env` variables, switching configuration files, or modifying configuration parameters), you must deploy the updated configuration before running your agent: ``` task infra:deploy ``` This command updates the deployment with the latest configuration and ensures your agent connects to the proper LLM with the correct credentials. Run this command: - After switching between LLM configurations - After updating credentials or API keys - After modifying model parameters ( temperature , top_p , etc.) - Before running your agent locally with the new configuration ## Advanced LLM configuration In addition to the `.env` file settings, you can directly edit the respective `llm.py` configuration file to make additional changes such as: - Temperature settings (applied to the deployed blueprint) - top_p values (applied to the deployed blueprint) - Timeout configurations (useful for GPU-based models) - Other model-specific parameters --- # Configure LLM providers in code URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-llm-providers.html > Learn how to configure different LLM providers for your agentic workflows including DataRobot Gateway, external APIs, and custom deployments. One of the key components of an LLM agent is the underlying LLM provider. DataRobot allows users to connect to virtually any LLM backend for their agentic workflows. LLM connections can be simplified by using the DataRobot LLM gateway or a DataRobot deployment (including NIM deployments). Alternatively, you can connect to any external LLM provider that supports the OpenAI API standard. DataRobot agent templates provide multiple methods for defining an agent LLM: - Use the DataRobot LLM gateway as the agent LLM, allowing you to use any model available in the gateway. - Connect to use a previously deployed custom model or NIM using the DataRobot API by providing the deployment ID. - Connect directly to an LLM provider API (such as OpenAI, Anthropic, or Gemini) by providing the necessary API credentials, enabling access to providers supporting a compatible API. This document focuses on configuring LLM providers by manually creating LLM instances directly in your `myagent.py` file. This approach gives you fine-grained control over LLM initialization and is shown in the framework-specific examples below. > [!NOTE] Alternative configuration method > If you prefer to configure LLM providers using environment variables and Pulumi (infrastructure-level configuration), see [Configure LLM providers with metadata](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-llm-providers-metadata.html). The following sections provide example code snippets for connecting to various LLM providers using the CrewAI, LangGraph, LlamaIndex, and NAT (NVIDIA NeMo Agent Toolkit) frameworks. You can use these snippets as a starting point and modify them as needed to fit your specific use case. ## DataRobot LLM gateway The LLM gateway provides a streamlined way to access LLMs proxied via DataRobot. The gateway is available for both cloud and self-managed users. You can retrieve a list of available models for your account using the following methods: **cURL:** ``` curl -X GET -H "Authorization: Bearer $DATAROBOT_API_TOKEN" "$DATAROBOT_ENDPOINT/genai/llmgw/catalog/" | jq '[.data[] | select(.isActive == true) | .model]' ``` **Python SDK:** ``` from datarobot.models.genai import LLMGatewayCatalog print("\n".join(LLMGatewayCatalog.get_available_models())) ``` The following code examples demonstrate how to programmatically connect to the DataRobot LLM gateway in the CrewAI, LangGraph, and LlamaIndex frameworks. These samples show how to configure the model, API endpoint, and authentication. > [!NOTE] Model format for LLM gateway > When using the DataRobot LLM gateway, the model name format is `datarobot//` (e.g., `datarobot/azure/gpt-5-mini-2025-08-07`). **CrewAI:** ``` from crewai import LLM def llm(self) -> LLM: """Returns a CrewAI LLM instance configured to use DataRobot's LLM gateway.""" return LLM( model="datarobot/azure/gpt-5-mini-2025-08-07", # Define the model name you want to use (format: datarobot//) # Note: The `/chat/completions` endpoint will be automatically appended by LiteLLM api_base="https://app.datarobot.com", # DataRobot endpoint api_key=self.api_key, # Your DataRobot API key timeout=self.timeout, # Optional timeout for requests ) ``` **LangGraph:** ``` from langchain_community.chat_models import ChatLiteLLM def llm(self) -> ChatLiteLLM: """Returns a ChatLiteLLM instance configured to use DataRobot's LLM gateway.""" return ChatLiteLLM( model="datarobot/azure/gpt-5-mini-2025-08-07", # Define the model name you want to use (format: datarobot//) # Note: The `/chat/completions` endpoint will be automatically appended by LiteLLM api_base="https://app.datarobot.com", # DataRobot endpoint api_key=self.api_key, # Your DataRobot API key timeout=self.timeout, # Optional timeout for requests ) ``` **LlamaIndex:** ``` # DataRobotLiteLLM class is included in the `myagent.py` file def llm(self) -> DataRobotLiteLLM: """Returns a DataRobotLiteLLM instance configured to use DataRobot's LLM gateway.""" return DataRobotLiteLLM( model="datarobot/azure/gpt-5-mini-2025-08-07", # Define the model name you want to use (format: datarobot//) # Note: The `/chat/completions` endpoint will be automatically appended by LiteLLM api_base="https://app.datarobot.com", # DataRobot endpoint api_key=self.api_key, # Your DataRobot API key timeout=self.timeout, # Optional timeout for requests ) ``` **NAT:** In NAT templates, LLMs are configured in the `workflow.yaml` file. To use the DataRobot LLM gateway, define an LLM in the `llms` section: ``` llms: datarobot_llm: _type: datarobot-llm-gateway model_name: azure/gpt-4o-mini # Define the model name you want to use temperature: 0.0 ``` Then, define the LLM a specific agent should use through the `llm_name` in the definition of that agent in the `functions` section: ``` functions: planner: _type: chat_completion llm_name: datarobot_llm # Reference the LLM defined below system_prompt: | You are a content planner... ``` If more than one LLM is defined in the `llms` section, the various `functions` can use different LLMs to suit the task. > [!TIP] NAT-provided LLM interfaces > Alternatively, you can use any of the [NAT-provided LLM interfaces](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/llms/index.html) instead of the LLM gateway. To use a NAT LLM interface, add the required configuration parameters such as `api_key`, `url`, and other provider-specific settings directly into the `workflow.yaml` file. ## DataRobot hosted LLM deployments You can easily connect to DataRobot-hosted LLM deployments as an LLM provider for your agents. To do this, [Deploy an LLM from the DataRobot Playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/deploy-llm.html) or [Deploy an open source LLM from the Hugging Face Hub as a Workload](https://docs.datarobot.com/en/docs/workload-api/create-workloads/deploy-llm-gpu.html). DataRobot-hosted LLMs can also provide access to moderations and guardrails for managing and governing models. To use a deployed custom model, manually configure the deployment URL directly in your agent code for `api_base=` param, following the examples below. > [!NOTE] Deployment ID > In the examples below, `DEPLOYMENT_ID` should be replaced with your actual DataRobot deployment ID, which you can obtain from the DataRobot platform. > [!TIP] Model name string construction > DataRobot deployments use an [OpenAI-compatible chat completion endpoint](https://docs.litellm.ai/docs/providers/openai_compatible). Therefore, the `model` name string should start with `openai/` to indicate the use of the OpenAI client. After `openai/`, the model name string should be the name of the model in the deployment. > > For LLMs deployed from the playground, the > model > string should include the provider name and the model name. In the example below, the full model name is > azure/gpt-4o-mini > , provider included, not just > gpt-4o-mini > . This results in a final value of > model="openai/azure/gpt-4o-mini" > . > For NIM models, the > model > string can be found on the NIM deployment's > Predictions > tab or in the NIM documentation. While NIM deployments may work with either > openai > or > meta_llama > interfaces, it's recommended to use > openai > for consistency. **CrewAI:** ``` from crewai import LLM def llm(self) -> LLM: """Returns a CrewAI LLM instance configured to use a DataRobot Deployment.""" return LLM( # Note: For DataRobot deployments, use the openai provider format model="openai/azure/gpt-4o-mini", # Format: openai/ # Note: The `/chat/completions` endpoint will be automatically appended by LiteLLM api_base=f"https://app.datarobot.com/api/v2/deployments/{DEPLOYMENT_ID}/", # Deployment URL api_key=self.api_key, # Your DataRobot API key timeout=self.timeout, # Optional timeout for requests ) ``` **LangGraph:** ``` from langchain_community.chat_models import ChatLiteLLM def llm(self) -> ChatLiteLLM: """Returns a ChatLiteLLM instance configured to use a DataRobot Deployment.""" return ChatLiteLLM( # Note: LangGraph uses datarobot provider format for deployments model="openai/azure/gpt-4o-mini", # Format: openai/ # Note: The `/chat/completions` endpoint will be automatically appended by LiteLLM api_base=f"https://app.datarobot.com/api/v2/deployments/{DEPLOYMENT_ID}/", # Deployment URL api_key=self.api_key, # Your DataRobot API key timeout=self.timeout, # Optional timeout for requests ) ``` **LlamaIndex:** ``` # DataRobotLiteLLM class is included in the `myagent.py` file def llm(self) -> DataRobotLiteLLM: """Returns a DataRobotLiteLLM instance configured to use a DataRobot Deployment.""" return DataRobotLiteLLM( # Note: For DataRobot deployments, use the openai provider format model="openai/azure/gpt-4o-mini", # Format: openai/ # Note: The `/chat/completions` endpoint will be automatically appended by LiteLLM api_base=f"https://app.datarobot.com/api/v2/deployments/{DEPLOYMENT_ID}/", # Deployment URL api_key=self.api_key, # Your DataRobot API key timeout=self.timeout, # Optional timeout for requests ) ``` **NAT:** In NAT templates, LLMs are configured in the `workflow.yaml` file. To use a DataRobot-hosted LLM deployment, define an LLM in the `llms` section: ``` llms: datarobot_deployment: _type: datarobot-llm-deployment model_name: datarobot-deployed-llm # Optional: Define the model name to pass through to the deployment temperature: 0.0 ``` The deployment ID is automatically retrieved from the `LLM_DEPLOYMENT_ID` environment variable or runtime parameter. When you use the DataRobot Deployed LLM option, `USE_DATAROBOT_LLM_GATEWAY` is automatically set to `0` so inference uses your deployment rather than the LLM gateway. To use this deployment, define the LLM a specific agent should use through the `llm_name` in the definition of that agent in the `functions` section: ``` functions: planner: _type: chat_completion llm_name: datarobot_deployment # Reference the LLM defined above system_prompt: | You are a content planner... ``` If more than one LLM is defined in the `llms` section, the various `functions` can use different LLMs to suit the task. > [!TIP] NAT-provided LLM interfaces > Alternatively, you can use any of the [NAT-provided LLM interfaces](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/llms/index.html) instead of the LLM gateway. To use a NAT LLM interface, add the required configuration parameters such as `api_key`, `url`, and other provider-specific settings directly in the `workflow.yaml` file. ## DataRobot NIM deployments The template supports using NIM deployments as an LLM provider, which allows you to use any NIM deployment hosted on DataRobot as an LLM provider for your agent. When using LiteLLM with NIM deployments, use the `openai` provider interface. The model name depends on your specific deployment and can be found in the Predictions tab of your deployment in DataRobot. For example, if the deployment uses a model named `meta/llama-3.2-1b-instruct`, use `openai/meta/llama-3.2-1b-instruct` for the model string. This tells LiteLLM to use the `openai` API adapter and the model name `meta/llama-3.2-1b-instruct`. To create a new NIM deployment, you can follow the instructions in the [DataRobot NIM documentation](https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/genai-nvidia-integration.html). > [!NOTE] Deployment ID > In the examples below, `DEPLOYMENT_ID` should be replaced with your actual DataRobot deployment ID, which you can obtain from the DataRobot platform. > [!TIP] Model name string construction > DataRobot deployments use an [OpenAI-compatible chat completion endpoint](https://docs.litellm.ai/docs/providers/openai_compatible). Therefore, the `model` name string should start with `openai/` to indicate the use of the OpenAI client. After `openai/`, the model name string should be the name of the model in the deployment. > > For LLMs deployed from the playground, the > model > string should include the provider name and the model name. In the example below, the full model name is > azure/gpt-4o-mini > , provider included, not just > gpt-4o-mini > . This results in a final value of > model="openai/azure/gpt-4o-mini" > . > For NIM models, the > model > string can be found on the NIM deployment's > Predictions > tab or in the NIM documentation. While NIM deployments may work with either > openai > or > meta_llama > interfaces, it's recommended to use > openai > for consistency. **CrewAI:** ``` from crewai import LLM def llm(self) -> LLM: """Returns a CrewAI LLM instance configured to use a NIM deployed on DataRobot.""" return LLM( # Use the openai provider with the model name from your deployment's Predictions tab model="openai/meta/llama-3.2-1b-instruct", # Format: openai/ api_base=f"https://app.datarobot.com/api/v2/deployments/{DEPLOYMENT_ID}", # NIM Deployment URL api_key=self.api_key, # Your DataRobot API key timeout=self.timeout, # Optional timeout for requests ) ``` **LangGraph:** ``` from langchain_openai import ChatOpenAI def llm(self) -> ChatOpenAI: """Returns a ChatOpenAI instance configured to use a NIM deployed on DataRobot.""" return ChatOpenAI( # Use the model name from your deployment's Predictions tab model="meta/llama-3.2-1b-instruct", # Model name from deployment's Predictions tab api_base=f"https://app.datarobot.com/api/v2/deployments/{DEPLOYMENT_ID}", # NIM deployment URL api_key=self.api_key, # Your DataRobot API key timeout=self.timeout, # Optional timeout for requests ) ``` **LlamaIndex:** ``` from llama_index.llms.openai_like import OpenAILike def llm(self) -> OpenAILike: """Returns an OpenAILike instance configured to use a NIM deployed on DataRobot.""" return OpenAILike( # Use the model name from your deployment's Predictions tab model="meta/llama-3.2-1b-instruct", # Model name from the deployment's Predictions tab api_base=f"https://app.datarobot.com/api/v2/deployments/{DEPLOYMENT_ID}/v1", # NIM deployment URL with /v1 endpoint api_key=self.api_key, # Your DataRobot API key timeout=self.timeout, # Optional timeout for requests is_chat_model=True, # Enable chat model mode for NIM endpoints ) ``` **NAT:** In NAT templates, LLMs are configured in the `workflow.yaml` file. To use a DataRobot NIM deployment, define an LLM in the `llms` section: ``` llms: datarobot_nim: _type: datarobot-nim model_name: meta/llama-3.2-1b-instruct # Optional: Define the model name to pass through to the deployment temperature: 0.0 ``` The deployment ID is automatically retrieved from the `NIM_DEPLOYMENT_ID` environment variable or runtime parameter. To use this deployment, define the LLM a specific agent should use through the `llm_name` in the definition of that agent in the `functions` section: ``` functions: planner: _type: chat_completion llm_name: datarobot_nim # Reference the LLM defined above system_prompt: | You are a content planner... ``` If more than one LLM is defined in the `llms` section, the various `functions` can use different LLMs to suit the task. > [!TIP] NAT-provided LLM interfaces > Alternatively, you can use any of the [NAT-provided LLM interfaces](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/llms/index.html) instead of the LLM gateway. To use a NAT LLM interface, add the required configuration parameters such as `api_key`, `url`, and other provider-specific settings directly in the `workflow.yaml` file. ## OpenAI API configuration There are cases where you may want to use an external LLM provider that supports the OpenAI API standard, such as OpenAI itself. The template supports connecting to any OpenAI-compatible LLM provider. Here are examples for directly connecting to OpenAI using the CrewAI and LangGraph frameworks. **CrewAI:** ``` from crewai import LLM def llm(self) -> LLM: """Returns a CrewAI LLM instance configured to use OpenAI.""" return LLM( model="gpt-4o-mini", # Define the OpenAI model name api_key="YOUR_OPENAI_API_KEY", # Your OpenAI API key timeout=self.timeout, # Optional timeout for requests ) ``` **LangGraph:** ``` from langchain_openai import ChatOpenAI def llm(self) -> ChatOpenAI: """Returns a ChatOpenAI instance configured to use OpenAI.""" return ChatOpenAI( model="gpt-4o-mini", # Define the OpenAI model name api_key="YOUR_OPENAI_API_KEY", # Your OpenAI API key timeout=self.timeout, # Optional timeout for requests ) ``` **LlamaIndex:** ``` from llama_index.llms.openai import OpenAI def llm(self) -> OpenAI: """Returns an OpenAI instance configured to use OpenAI.""" return OpenAI( model="gpt-4o-mini", # Define the OpenAI model name api_key="YOUR_OPENAI_API_KEY", # Your OpenAI API key timeout=self.timeout, # Optional timeout for requests ) ``` ## Anthropic API configuration You can connect to Anthropic's Claude models using the Anthropic API. The template supports connecting to Anthropic models through both CrewAI and LangGraph frameworks. You'll need an Anthropic API key to use these models. **CrewAI:** ``` from crewai import LLM def llm(self) -> LLM: """Returns a CrewAI LLM instance configured to use Anthropic.""" return LLM( model="claude-3-5-sonnet-20241022", # Define the Anthropic model name api_key="YOUR_ANTHROPIC_API_KEY", # Your Anthropic API key timeout=self.timeout, # Optional timeout for requests ) ``` **LangGraph:** ``` from langchain_anthropic import ChatAnthropic def llm(self) -> ChatAnthropic: """Returns a ChatAnthropic instance configured to use Anthropic.""" return ChatAnthropic( model="claude-3-5-sonnet-20241022", # Define the Anthropic model name anthropic_api_key="YOUR_ANTHROPIC_API_KEY", # Your Anthropic API key timeout=self.timeout, # Optional timeout for requests ) ``` **LlamaIndex:** ``` from llama_index.llms.anthropic import Anthropic def llm(self) -> Anthropic: """Returns an Anthropic instance configured to use Anthropic.""" return Anthropic( model="claude-3-5-sonnet-20241022", # Define the Anthropic model name api_key="YOUR_ANTHROPIC_API_KEY", # Your Anthropic API key timeout=self.timeout, # Optional timeout for requests ) ``` ## Gemini API configuration You can also connect to Google's Gemini models using the Gemini API. The template supports connecting to Gemini models through both CrewAI and LangGraph frameworks. You'll need a Google AI API key to use these models. **CrewAI:** ``` from crewai import LLM def llm(self) -> LLM: """Returns a CrewAI LLM instance configured to use Gemini.""" return LLM( model="gemini/gemini-1.5-flash", # Define the Gemini model name api_key="YOUR_GEMINI_API_KEY", # Your Google AI API key timeout=self.timeout, # Optional timeout for requests ) ``` **LangGraph:** ``` from langchain_google_genai import ChatGoogleGenerativeAI def llm(self) -> ChatGoogleGenerativeAI: """Returns a ChatGoogleGenerativeAI instance configured to use Gemini.""" return ChatGoogleGenerativeAI( model="gemini-1.5-flash", # Define the Gemini model name google_api_key="YOUR_GEMINI_API_KEY", # Your Google AI API key timeout=self.timeout, # Optional timeout for requests ) ``` **LlamaIndex:** ``` from llama_index.llms.gemini import Gemini def llm(self) -> Gemini: """Returns a Gemini instance configured to use Google's Gemini.""" return Gemini( model="gemini-1.5-flash", # Define the Gemini model name api_key="YOUR_GEMINI_API_KEY", # Your Google AI api key timeout=self.timeout, # Optional timeout for requests ) ``` ## Connect to other providers You can connect to any other LLM provider that supports the OpenAI API standard by following the patterns shown in the examples above. For providers that don't natively support the OpenAI API format, you have several options to help bridge the connection: ### Review framework documentation Each framework provides comprehensive documentation for connecting to various LLM providers: - CrewAI : Visit the CrewAI LLM documentation for detailed examples of connecting to different providers - LangGraph : Check the LangChain LLM integrations for extensive provider support - LlamaIndex : Refer to the LlamaIndex LLM modules for various LLM integrations - NAT : Refer to the NVIDIA NeMo Agent Toolkit documentation for LLM configuration in workflow.yaml ### Use LiteLLM for universal connectivity [LiteLLM](https://docs.litellm.ai/) is a library that provides a unified interface for connecting to 100+ LLM providers. It translates requests to match each provider's specific API format, making it easier to connect to providers like: - Azure OpenAI - AWS Bedrock - Google Gemini Enterprise Agent Platform (formerly Vertex AI) - Cohere - Hugging Face - Ollama - And more When using LiteLLM, the model string uses a compound format: `provider/model-name` - Provider : The API adapter/provider to use (e.g., openai , azure , etc.). - Model name : The model name to pass to that provider. For example, if the deployment uses a model named `meta/llama-3.2-1b-instruct`, use `openai/meta/llama-3.2-1b-instruct` for the model string. This tells LiteLLM to use the [openaiAPI adapter](https://docs.litellm.ai/docs/providers/openai_compatible) and the model name `meta/llama-3.2-1b-instruct`. This format allows LiteLLM to route requests to the appropriate provider API while using the correct model identifier for that provider. For the most up-to-date list of supported providers and configuration examples, visit the [LiteLLM documentation](https://docs.litellm.ai/docs/providers). --- # Agentic memory service URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-memory-service.html > Learn when to use DataRobot's built-in chat history and REST integration versus the mem0-compatible memory API. > [!NOTE] Premium > DataRobot's Agentic AI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. The agentic memory surface in DataRobot exposes two different APIs that address different use cases. Choose the path that matches your build, then follow the documentation for that path. You do not need to merge the two in a single application. If you need to persist and retrieve chat-style history (sessions) through DataRobot's REST API and Python API client with a DataRobot model (accessed via `datarobot.models.memory.Session` or `datarobot.models.memory.MemorySpace`), use the chat history API described in the [Chat history API](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-memory-service.html#chat-history-api) section. Read the [REST API](https://docs.datarobot.com/en/docs/api/reference/public-api/agentic_memory.html) and [Python API client](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html) reference documentation for complete details. If you are working with long-term, mem0-style memory (including migrating from the open-source mem0 stack), or you want examples and request shapes that match the mem0 product, use the [mem0 API](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-memory-service.html#mem0-api). For that interface, the canonical resource is the [mem0 documentation](https://docs.mem0.ai/). **Chat history API:** The chat history API is the DataRobot-integrated path: the same [REST API](https://docs.datarobot.com/en/docs/api/reference/public-api/agentic_memory.html) you use for other DataRobot resources, with matching coverage in the [DataRobot Python API client](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html). Python API client: The Python API client adds dedicated types to make build-out straightforward: datarobot.models.memory.MemorySpace offers a basic CRUD for memory spaces (containers for stored conversation data). datarobot.models.memory.Session and datarobot.models.memory.Event work with sessions and events (messages within a session) when you build chat-style agentic applications. REST API: Use the REST API for these resources alongside the same authentication and base URL patterns as the rest of the DataRobot API. Details appear in the [REST API](https://docs.datarobot.com/en/docs/api/reference/public-api/agentic_memory.html) reference for the relevant operations and schemas. Choose this path when you want a DataRobot chat history and session model aligned with the platform's API and Python package. **mem0 API:** DataRobot also offers a mem0-compatible REST API that is intended as a one-to-one match to the [open-source mem0](https://github.com/mem0ai/mem0) product's interface. For conceptual overviews, endpoint behavior, and especially examples (including language- and framework-specific snippets), use the official mem0 documentation, which is maintained for that product. Access the primary mem0 documentation at [https://docs.mem0.ai](https://docs.mem0.ai/) to learn request and response shapes, how to add and search memory, and how to integrate with common agent frameworks. Because the DataRobot service is built to be compatible with the same API surface, those guides apply with your DataRobot endpoint and authentication. Choose this path when you are targeting mem0's semantics and ecosystem, or porting an integration that was written against the open-source service. ## Quickstart Review a quickstart workflow below. ### Prerequisites - Python 3.9+ - A DataRobot account, an API key, and the endpoint URL (e.g. https://app.datarobot.com/api/v2 ) - pip install "datarobot>=3.17" "mem0ai>=2.0" Set credentials once, either via environment ( `DATAROBOT_API_TOKEN`, `DATAROBOT_ENDPOINT`) or `~/.config/datarobot/drconfig.yaml`. The snippets below pass them explicitly. ### Connect ``` import datarobot as dr dr.Client(token="", endpoint="https://app.datarobot.com/api/v2") ``` ### Create a memory space A memory space is the isolation boundary for one application or one end user. ``` from datarobot.models.memory import MemorySpace space = MemorySpace.create( description="Acme support assistant", llm_model_name="vertex_ai/claude-sonnet-4-6", # used for mem0 fact extraction ) print(space.id) ``` `llm_model_name` selects the LLM used for mem0 fact extraction. It is validated against the models available through the DataRobot LLM Gateway; an unknown name is rejected with a 422 response that lists the allowed models. See [LLM availability](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html) for the models offered per provider. Non-reasoning models are recommended, as reasoning models are significantly slower for fact extraction without producing better results. Alternatively, `llm_base_url` points the memory space at a custom OpenAI-compatible endpoint instead of the LLM Gateway; its host must match the allowlist configured by your administrator. ### Create a session A session is one conversation.`participants` is a list of MongoDB-ObjectId strings - typically your end user. ``` from datarobot.models.memory import Session session = Session.create( memory_space_id=space.id, participants=["67500a000000000000000001"], description="Triage chat #42", ) print(session.id) ``` The server attaches a default soft-delete lifecycle strategy (180 days) if you do not supply one. ### Post messages Each message is an event.`body` is free-form JSON; `emitter` identifies who sent it. ``` event = session.post_event( body={"content": "Hello, my deployment is failing"}, emitter={"type": "user", "id": "67500a000000000000000001"}, event_type="message", ) print(event.sequence_id) # 0 ``` Batched append (up to 200 events, single transaction): ``` session.post_events([ { "body": {"content": "Looking at it now."}, "emitter": {"type": "agent", "id": "67500a000000000000000002"}, "event_type": "message", }, { "body": {"content": "Found the issue - rolling back."}, "emitter": {"type": "agent", "id": "67500a000000000000000002"}, "event_type": "message", }, ]) ``` ### Read and edit events ``` # Newest 20 events: recent = session.events(last_n=20) for e in recent: print(e.sequence_id, e.emitter_type, e.body) # Edit message #1: session.update_event( sequence_id=1, body={"content": "Looking at it now (ETA 5min)."}, ) ``` `update_event` accepts an optional `created_at` for optimistic concurrency: if the event has changed since that timestamp, the server rejects the update and the caller must reload and retry. ### Memory extraction via stock mem0 client The service exposes a mem0-compatible sub-route per memory space. Point a stock `mem0ai.MemoryClient` at it and the standard mem0 client works unchanged - `add()`, `get()`, `search()`, `get_all()`, `delete()`. ``` from mem0 import MemoryClient endpoint = f"https://app.datarobot.com/api/v2/memory/{space.id}" memory = MemoryClient(host=endpoint, api_key="") memory.add( [{"role": "user", "content": "I deploy on Fridays using ArgoCD"}], user_id="alice", agent_id="support-bot", ) hits = memory.search( "when do we deploy", filters={"user_id": "alice", "agent_id": "support-bot"}, ) ``` The `api_key` is your DataRobot API token. Memory-space scoping is entirely in the URL path - no extra headers. ## Session lifecycle and retention Every session carries one or more lifecycle strategies that control what happens to it over time. A strategy pairs an action— `soft_delete` (hide the session and its events) or `extract_memories` (promote the conversation to long-term memory)—with a trigger that decides when the action fires. If you do not supply any strategies at creation time, the server attaches a default `soft_delete` strategy with a 180-day TTL. The following triggers are available: | Trigger | Fires | | --- | --- | | ttl | When the session age exceeds the given number of seconds (maximum two years). | | eventCount | When the number of events in the session reaches the threshold. | | tokenCount | When the total token count of the session's events reaches the threshold. | | idle | When no event has been added to the session for the given number of seconds (maximum two years). | | never | Never. The session is retained until you delete it explicitly. | ``` session = Session.create( memory_space_id=space.id, participants=["67500a000000000000000001"], description="Triage chat #42", lifecycle_strategies=[{"type": "soft_delete", "trigger": {"ttl": 604800}}], ) ``` ### Never-expire sessions The `never` trigger opts a session out of automatic lifecycle execution entirely: the session and its events are retained until an explicit session delete (or an administrator-triggered permanent deletion) removes them. ``` session = Session.create( memory_space_id=space.id, participants=["67500a000000000000000001"], description="Application settings store", lifecycle_strategies=[{"type": "soft_delete", "trigger": {"never": True}}], ) ``` Lifecycle strategies are set when the session is created and cannot be changed afterwards, so the `never` trigger applies to new sessions only. Sessions created earlier keep the expiry they were given; to hold existing data indefinitely, create a new session with the `never` trigger and write the data to it. Use the `never` trigger for: - Application system data: Records such as user profiles, settings, and workspaces stored in session metadata, where expiry would break the application rather than clean up stale data. A dormant installation survives any retention window without keep-alive traffic. - Compliance retention beyond two years: The ttl and idle triggers cap at two years, so a longer retention floor (for example, a six-year regulatory requirement for chat history) is only expressible with never . Avoid the `never` trigger for ordinary conversational history: nothing collects never-expire sessions automatically, so deleting them becomes your application's responsibility. Prefer `ttl` or `idle` for data with a natural end of life. > [!NOTE] Availability information > The `never` trigger is available on Self-Managed AI Platform and Single-tenant SaaS installations only; it is not available on the Multi-tenant SaaS AI Platform. The trigger is disabled by default and is enabled per installation by an administrator—see the [memory service configuration](https://docs.datarobot.com/en/docs/install/install-config/advanced-configuration/generative-ai-agentic/memory-service.html). Where it is not enabled, session creation with a `never` trigger is rejected with a 422 response. --- # Agent components URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-overview.html > This overview details the components required to create an agent using DataRobot's agent framework. This overview details the components required to create an agent using DataRobot's agent framework. An agent artifact includes several standard files that contain metadata, hooks/functions, classes, and properties. | Section | Description | | --- | --- | | Agent file structure | Describes important files and their organization for a DataRobot agent. | | Functions and hooks | Details the mandatory functions and integration hooks needed for agent operation. | | Agent class implementation | Details the general structure of the main agent class and its methods and properties. | | Tool integration | Explains how agents use tools via the ToolClient class and framework-specific tool APIs. | ## Agent file structure Every DataRobot agent requires a specific set of files in the `agent/agent/` directory (for example, in the [DataRobot Agentic Starter template](https://github.com/datarobot-community/datarobot-agent-application)). These files work together to create a complete agent that can be deployed and executed by DataRobot. ``` # agent/agent/ directory agent/agent/ ├── __init__.py # Package initialization ├── myagent.py # Main agent implementation, including prompts ├── config.py # Configuration management ├── register.py # DRAgent / NAT registration (framework-specific) ├── workflow.yaml # Declarative workflow config for DRAgent (framework-specific) └── model-metadata.yaml # Agent metadata configuration ``` Parent directory `agent/` also contains `custom.py` (DRUM hooks), `dev.py`, `cli.py`, and other infrastructure files. | File | Description | | --- | --- | | __init__.py | Identifies the directory as a Python package and enables imports. | | model-metadata.yaml | Defines the agent's configuration, runtime parameters, and deployment settings. | | custom.py (under the parent agent/, one level up) | Implements DataRobot integration hooks (load_model, chat) for agent execution. | | myagent.py | Contains the main agent workflow. Depending on the framework, this may be a MyAgent class or a factory-produced agent (for example, LangGraph uses datarobot_agent_class_from_langgraph). | | config.py | Manages configuration loading from environment variables, runtime parameters, and DataRobot credentials. | | register.py | Connects the DRAgent front server to your agent: LLM wrappers, MCP tools, and optional workflow tools. | | workflow.yaml | Declares workflow type, LLM component, and related settings for DRAgent. | ### Agent metadata (model-metadata.yaml) The `model-metadata.yaml` file tells DataRobot how to configure and deploy the agent. It defines the agent's type, name, and any required runtime parameters. ``` # model-metadata.yaml --- name: agent_name type: inference targetType: agenticworkflow runtimeParameterDefinitions: - fieldName: LLM_DEPLOYMENT_ID defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string - fieldName: LLM_DEFAULT_MODEL defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string - fieldName: LLM_DEFAULT_MODEL_FRIENDLY_NAME defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string - fieldName: USE_DATAROBOT_LLM_GATEWAY defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string - fieldName: MCP_DEPLOYMENT_ID defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string - fieldName: EXTERNAL_MCP_URL defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string - fieldName: SESSION_SECRET_KEY defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string ``` | Field | Description | | --- | --- | | name | The agent's display name in DataRobot (used for identification and deployment). | | type | The agent model type. Must be inference for all DataRobot agents. | | targetType | The agent target type. Must be agenticworkflow for agentic workflow deployments. | | runtimeParameterDefinitions | Defines optional runtime parameters for LLM configuration, MCP server connections, and other agent settings. | > [!TIP] LLM Provider Configuration > Agents support multiple LLM provider configurations including: > > LLM gateway direct > : Use DataRobot's LLM gateway directly > LLM blueprint with external LLMs > : Connect to external providers (Azure OpenAI, Amazon Bedrock, Google Gemini Enterprise Agent Platform (formerly Vertex AI), Anthropic, Cohere, TogetherAI) > Deployed models > : Use a DataRobot-deployed LLM via > LLM_DEPLOYMENT_ID > > When you use the DataRobot Deployed LLM option, `USE_DATAROBOT_LLM_GATEWAY` is automatically set to `0` so the workflow targets your deployment instead of the gateway. ## Functions and hooks (custom.py) Agents use specific function signatures called "hooks" to integrate with DataRobot. The `custom.py` file contains the required functions that DataRobot calls to execute the agent. These functions connect DataRobot and the agent's logic. For more information, see the [structured model hooks documentation](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html). The following DataRobot custom model hooks are implemented in `custom.py`: | Component | Description | | --- | --- | | load_model() | One-time initialization function called when DataRobot starts the agent. | | chat() | Main execution function called for each user interaction/chat message. | > [!TIP] Other DataRobot hooks > The `score()` and `score_unstructured()` functions can be implemented if required for specific use cases. ### load_model() hook The `load_model()` hook is called once to initialize the agent. This is where any one-time configuration can be defined. ``` # custom.py def load_model(code_dir: str) -> tuple[ThreadPoolExecutor, asyncio.AbstractEventLoop]: """The agent is instantiated in this function and returned. Args: code_dir: Path to the agentic workflow directory Returns: tuple[ThreadPoolExecutor, asyncio.AbstractEventLoop]: Thread pool executor and event loop for async operations """ thread_pool_executor = ThreadPoolExecutor(1) event_loop = asyncio.new_event_loop() thread_pool_executor.submit(asyncio.set_event_loop, event_loop).result() return (thread_pool_executor, event_loop) ``` ### chat() hook The main entry point for the agent. DataRobot calls this function every time a user sends a message to the agent. ``` # custom.py def chat( completion_create_params: CompletionCreateParams | CompletionCreateParamsNonStreaming | CompletionCreateParamsStreaming, load_model_result: tuple[ThreadPoolExecutor, asyncio.AbstractEventLoop], **kwargs: Any, ) -> Union[CustomModelChatResponse, Iterator[CustomModelStreamingResponse]]: """Main entry point for agent execution via chat endpoint. Args: completion_create_params: OpenAI-compatible completion parameters load_model_result: Result from load_model() function **kwargs: Additional keyword arguments (e.g., headers) Returns: Union[CustomModelChatResponse, Iterator[CustomModelStreamingResponse]]: Formatted response with agent output """ ``` ## Agent class implementation (myagent.py) The `myagent.py` file contains the workflow logic for your agent. In many templates this is a `MyAgent` class; in current LangGraph [Agentic Starter](https://github.com/datarobot-community/datarobot-agent-application) projects, `MyAgent` is produced by `datarobot_agent_class_from_langgraph` from a `graph_factory` and prompt template. This is where you define how the agent behaves, which tools it receives, and how it processes inputs. | Component | Description | | --- | --- | | init() | Method to initialize the agent with credentials, configuration, and framework-specific setup. | | invoke() | Main execution method that processes inputs and returns framework-specific results. | | llm or llm() | Property or method (depending on framework template) that returns the configured LLM instance for agent operations. | Every agent follows this basic pattern, though the specific implementation varies by framework. > [!NOTE] Framework-specific implementations > CrewAI, LangGraph, LlamaIndex, and NAT (NVIDIA NeMo Agent Toolkit) templates include framework-specific `llm` or `llm()` implementations and return types. The Generic Base template provides a minimal implementation that can be customized for any framework. NAT templates configure LLMs in `workflow.yaml` rather than through Python `llm` configuration. ``` # myagent.py class MyAgent: """Agent implementation following DataRobot patterns.""" def __init__( self, api_key: Optional[str] = None, api_base: Optional[str] = None, model: Optional[str] = None, verbose: Optional[Union[bool, str]] = True, timeout: Optional[int] = 90, **kwargs: Any, ): """Initialize agent with credentials and configuration.""" self.api_key = api_key or os.environ.get("DATAROBOT_API_TOKEN") self.api_base = api_base or os.environ.get("DATAROBOT_ENDPOINT") self.model = model self.timeout = timeout # ... other initialization @property def llm(self) -> LLM: # Framework-specific type """Primary LLM configuration.""" if os.environ.get("LLM_DEPLOYMENT_ID"): return self.llm_with_datarobot_deployment else: return self.llm_with_datarobot_llm_gateway def invoke(self, completion_create_params: CompletionCreateParams) -> Union[ Generator[tuple[str, Any | None, dict[str, int]], None, None], tuple[str, Any | None, dict[str, int]], ]: """Main execution method - REQUIRED.""" # Extract inputs inputs = create_inputs_from_completion_params(completion_create_params) # Execute agent workflow # Return results return response_text, pipeline_interactions, usage_metrics ``` ### __init__() method The method that initializes the agent with configuration and credentials from DataRobot and framework-specific setup. ``` # myagent.py class MyAgent: def __init__(self, api_key: Optional[str] = None, api_base: Optional[str] = None, model: Optional[str] = None, verbose: Optional[Union[bool, str]] = True, timeout: Optional[int] = 90, **kwargs: Any): """Initialize agent with DataRobot credentials and configuration.""" ``` ### invoke() method The core execution method that DataRobot calls to run the agent. This method must be implemented and should contain the agent's main workflow logic. All frameworks use the same return type pattern. ``` # myagent.py def invoke(self, completion_create_params: CompletionCreateParams) -> Union[ Generator[tuple[str, Any | None, dict[str, int]], None, None], tuple[str, Any | None, dict[str, int]], ]: """Main execution method - REQUIRED for DataRobot integration. Args: completion_create_params: Input parameters from DataRobot Returns: Union of generator (for streaming) or tuple (for non-streaming): - response_text: str - The agent's response - pipeline_interactions: Any | None - Event tracking data - usage_metrics: dict[str, int] - Token usage statistics """ ``` ### llm property Defines which language model the agent uses for generating responses. The return type and implementation varies by framework. The CrewAI and LlamaIndex templates implement API base URL logic directly within their `llm` properties. LangGraph starter templates typically obtain the chat model through helpers such as `get_llm` in the DRUM/DRAgent adaptor path rather than an `llm` property on a handwritten class. ``` # myagent.py (CrewAI/LlamaIndex) @property def llm(self) -> LLM: # Framework-specific type """Primary LLM instance for agent operations. Returns: Framework-specific LLM type: - CrewAI: LLM - LlamaIndex: DataRobotLiteLLM """ api_base = urlparse(self.api_base) if os.environ.get("LLM_DEPLOYMENT_ID"): # Handle deployment-specific URL construction # ... implementation details ... return LLM(model="openai/gpt-4o-mini", api_base=deployment_url, ...) else: # Handle LLM gateway URL construction # ... implementation details ... return LLM(model="datarobot/azure/gpt-5-mini-2025-08-07", api_base=api_base.geturl(), ...) ``` The Generic Base template implements the `llm` property as follows: ``` # myagent.py (Generic Base) @property def llm(self) -> Any: """Primary LLM instance for agent operations. Returns: Any: Minimal implementation for custom frameworks """ if os.environ.get("LLM_DEPLOYMENT_ID"): return self.llm_with_datarobot_deployment else: return self.llm_with_datarobot_llm_gateway ``` The NAT template configures LLMs in `workflow.yaml` rather than through a Python property. You can use the DataRobot LLM gateway ( `_type: datarobot-llm-gateway`), DataRobot deployments ( `_type: datarobot-llm-deployment`), or DataRobot NIM deployments ( `_type: datarobot-nim`). The example below uses the DataRobot LLM gateway: ``` # workflow.yaml (NAT) llms: datarobot_llm: _type: datarobot-llm-gateway model_name: azure/gpt-4o-mini # Define the model name you want to use temperature: 0.0 ``` The LLM a specific agent uses is defined through the `llm_name` in the definition of that agent in the `functions` section: ``` # workflow.yaml (NAT) functions: planner: _type: chat_completion llm_name: datarobot_llm # Reference the LLM defined above system_prompt: | You are a content planner... ``` If more than one LLM is defined in the `llms` section, the various `functions` can use different LLMs to suit the task. > [!TIP] NAT-provided LLM interfaces > Alternatively, you can use any of the [NAT-provided LLM interfaces](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/llms/index.html) instead of the LLM gateway. To use a NAT LLM interface, add the required configuration parameters such as `api_key`, `url`, and other provider-specific settings directly in the `workflow.yaml` file. For information about using DataRobot deployments with NAT templates, see [Configure LLM providers in code](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-llm-providers.html#datarobot-hosted-llm-deployments). ## Tool integration Agents can use tools to extend their capabilities with the `ToolClient` class, which enables agents to call DataRobot tool deployments. The functionality of `helpers.py` present in previous versions was moved to the [datarobot-genaipackage](https://github.com/datarobot-oss/datarobot-genai) along with other adapter code that connects agents to DataRobot's DRUM server in version 11.3.1. > [!NOTE] ToolClient usage consideration > The `ToolClient` is specifically designed for calling user-deployed global tools within DataRobot. As this client serves that specialized use case, it is not required for the majority of agent tool implementations. ### Framework-specific tools Each framework provides its own native tool APIs for defining custom tools: - CrewAI : Tools are passed to Agent instances via the tools parameter. - LangGraph : Tools are defined as part of graph nodes and edges. - LlamaIndex : Tools are defined as functions and passed to agent constructors. - NAT : Tools are defined in workflow.yaml as functions and referenced in the workflow's tool_list (the available tool types are defined by thenat_toolsubmodules ). > [!TIP] NAT agent configuration > Use [react_agent](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/about/react-agent.html) instead of [sequential_executor](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/about/sequential-executor.html) for flexible agents that decide which tools to run in which order, depending on the query. ### Authorization context The `resolve_authorization_context()` function from the `datarobot-genai` package is called in `custom.py` to automatically handle authentication for tools that require access tokens. The function returns an authorization context dictionary that is assigned to `completion_create_params["authorization_context"]`. This ensures tools can securely access external services using DataRobot's credential management system. --- # Add Python packages URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-python-packages.html > Add required Python packages to agentic workflows using pyproject.toml dependency management. To add additional Python packages to your agent environment, add them to your `pyproject.toml` file using uv, a modern Python package and environment manager. > [!WARNING] Keep existing packages > We recommend keeping existing packages in `pyproject.toml` to ensure consistent behavior across playground and deployment environments. > [!TIP] Fast iteration with runtime dependencies > For rapid development, you can add dependencies using [runtime dependencies](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-python-packages.html#add-runtime-dependencies-fast-iteration) without rebuilding the Docker image. This is ideal for quick testing during development. The typical workflow for adding packages is: 1. Add the package: uv add 2. Test locally: dr run dev or task dev (runs the development server) 3. Build Docker image: dr task agent:build-docker-context (if using custom Docker image) 4. Deploy: Use your deployment pipeline > [!NOTE] Automatic synchronization > During `dr run deploy`, `dr run deploy-dev`, or `dr run dev`, the `pyproject.toml` file is automatically synchronized with the agentic workflow code. Package installation is auto-synchronized with `uv` by default. You don't need to manually update the `pyproject.toml` file in these directories. ## Add packages to your agent 1. Navigate to your agent directory and use uv to add the new package: ``` cd agent uv add ``` For example, to add the `requests` package: ``` uv add requests ``` You can also specify version constraints: ``` uv add "requests>=2.32.0" ``` 1. Create a custom execution environment to test your agent in the playground: 2. Open the.envfile. 3. SetDATAROBOT_DEFAULT_EXECUTION_ENVIRONMENT=to empty or delete the line completely. 4. Optional: If you need to test with a custom Docker image, first create the docker context: ``` # Create docker_context directory if needed dr task agent:create-docker-context # Then build and test the Docker image cd agent/docker_context docker build -f Dockerfile . -t docker_context_test ``` After completing these steps, when you run `dr run dev`, `dr run deploy-dev`, or `dr run deploy`, the new environment will be automatically built the first time. Subsequent builds will use the cached environment if the requirements have not changed. The new environment will be automatically linked and used for all your agent components, models, and deployments. You can manually test building your agent image by running the following command, ensuring the new dependency is defined successfully: ``` dr task agent:build-docker-context ``` > [!NOTE] docker_context is optional > The `docker_context` directory is no longer included by default. If you need to build a custom Docker image, first create the docker context using `dr task agent:create-docker-context`. This command downloads the necessary Docker files and creates the `docker_context` directory in your agent folder. The `dr task agent:build-docker-context` command will then copy the updated `pyproject.toml` to the `docker_context/` directory, build the Docker image with the new dependencies, and save the image. > [!WARNING] Don't remove requirements > Don't remove any packages from the `pyproject.toml` file unless you are certain they aren't needed. Removing packages may lead to unexpected behavior with playground or deployment interactions, even if the local execution environment works correctly. ## Agent-specific considerations Different agent types may have different dependency management approaches: | Agent Type | Dependency Management Approach | | --- | --- | | agent_generic_base | Uses pyproject.toml. | | agent_llamaindex | Uses pyproject.toml with auto-generated requirements.in and requirements.txt files. | | agent_crewai | Uses pyproject.toml. | | agent_langgraph | Uses pyproject.toml. | | agent_nat | Uses pyproject.toml. | For all agent types, the recommended approach is to use `uv add ` to modify the `pyproject.toml` file, which will automatically handle dependency resolution and version constraints. The `pyproject.toml` file serves as the single source of truth for all dependencies, and the build process automatically handles the Docker environment setup. ## Add runtime dependencies (Fast iteration) For rapid development and testing, add dependencies at runtime without rebuilding the Docker image. Dependencies added to the `extras` group in your `pyproject.toml` file will be installed when the prompt is first executed in the Playground or when the deployment starts. Runtime dependencies are ideal for: - Quick iteration during development - Testing new packages without rebuilding images - Adding lightweight dependencies that don't require compilation Add runtime dependencies directly using `uv`: ``` cd agent uv add --active --no-upgrade --group extras "chromadb>=1.1.1" ``` > [!TIP] Use --no-upgrade flag > The `--no-upgrade` flag is crucial to minimize the time required to install extra dependencies. It ensures that only new dependencies are added without upgrading existing ones, which helps to keep the runtime installation fast and minimize differences from the execution environment. ### Feature considerations Runtime dependencies have several important limitations to consider: - Internet access required: Since extra dependencies are installed at runtime,internet access is required. In restricted network environments, you mustconfigure an internal PyPI proxyor build custom execution environments to include custom dependencies. - Installation complexity: If a dependency has a sophisticated installation process (for example, compilation of C bindings, setting up cache, or custom build steps), the runtime installation may fail due to restrictions in the execution environment. In such cases, building a custom execution environment is the only option. - Security considerations: Using runtime dependencies makes it possible to quickly fix CVE issues in runtime by upgrading library versions directly in the agent. However, it's also possible to introduce vulnerabilities by downgrading libraries to vulnerable versions, even if the execution environment doesn't have those vulnerabilities. You must update both execution environments andpyproject.toml/uv.lockfiles in your agents to ensure vulnerabilities are properly fixed. Consider the following best practices for using runtime dependencies: - Use runtime dependencies for development and testing. - For production deployments, consider building custom execution environments with all required dependencies installed. - Keep the pyproject.toml file synchronized with all dependencies to ensure consistency across environments. - Minimize upgrades by using the --no-upgrade flag to keep installation times fast. --- # Access request headers URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-request-headers.html > Learn how to access HTTP request headers in your deployed agents for authentication, tracking, and custom metadata. When your agent is deployed, you may need to access HTTP request headers for authentication, tracking, or custom metadata. DataRobot makes headers available to your agent code through the `chat()` function's `**kwargs` parameter. ## Extracting X-Untrusted-* headers Headers with the `X-Untrusted-*` prefix are passed through from the original request. They are available in the `kwargs` dictionary, located in `agent/agent/custom.py`: ``` # agent/agent/custom.py import asyncio from concurrent.futures import ThreadPoolExecutor from typing import Any, Iterator, Union from openai.types.chat import CompletionCreateParams from openai.types.chat.completion_create_params import ( CompletionCreateParamsNonStreaming, CompletionCreateParamsStreaming, ) def chat( completion_create_params: CompletionCreateParams | CompletionCreateParamsNonStreaming | CompletionCreateParamsStreaming, load_model_result: tuple[ThreadPoolExecutor, asyncio.AbstractEventLoop], **kwargs: Any, ) -> Union[CustomModelChatResponse, Iterator[CustomModelStreamingResponse]]: # Extract all headers from kwargs headers = kwargs.get("headers", {}) # Access specific X-Untrusted-* headers authorization_header = headers.get("X-Untrusted-Authorization") custom_header = headers.get("X-Untrusted-Custom-Metadata") # Use headers in your agent logic if authorization_header: # Pass to downstream services, tools, etc. # Do not log raw authorization tokens; log only non-sensitive metadata. request_id = headers.get("X-Untrusted-Request-ID") if request_id: print(f"Processing request {request_id} with authorization header present") else: print("Processing request with authorization header present") # Continue with agent logic... ``` ## Common use cases Request headers can be used for various purposes in your agent workflows. The following examples demonstrate common patterns for extracting and using header information: ### Passing authentication to external services Extract authentication tokens from request headers and pass them to external services. This allows you to forward authentication credentials from incoming requests to downstream tools and services: ``` # agent/agent/custom.py headers = kwargs.get("headers", {}) auth_token = headers.get("X-Untrusted-Authorization") # Use the token to authenticate with external APIs tool_client = MyTool(auth_token=auth_token) ``` ### Tracking request metadata Use headers to track request IDs, user IDs, or other metadata for debugging and analytics. This helps you trace requests through your agent workflow: ``` # agent/agent/custom.py headers = kwargs.get("headers", {}) request_id = headers.get("X-Untrusted-Request-ID") user_id = headers.get("X-Untrusted-User-ID") # Log metadata for debugging or analytics logging.info(f"Processing request {request_id} for user {user_id}") ``` ### Conditional agent behavior Adjust agent behavior based on request context from headers. This enables you to customize agent behavior for different regions, users, or deployment contexts: ``` # agent/agent/custom.py headers = kwargs.get("headers", {}) region = headers.get("X-Untrusted-Region") # Adjust agent behavior based on request context if region == "EU": agent = MyAgent(enable_gdpr_mode=True) ``` ## Important considerations When working with request headers in your agent code, keep the following in mind: - Only headers with the X-Untrusted-* prefix are passed through to your agent code. - Headers are case-sensitive when accessing them from the dictionary. - Always provide default values or check for None when accessing headers that may not be present. - Headers are available in both streaming and non-streaming chat responses. ## Log request headers for debugging Log all available headers to inspect what's being passed to your agent: ``` # agent/agent/custom.py import asyncio import logging from concurrent.futures import ThreadPoolExecutor from typing import Any def chat(completion_create_params, load_model_result: tuple[ThreadPoolExecutor, asyncio.AbstractEventLoop], **kwargs: Any): headers = kwargs.get("headers", {}) # Log all headers to inspect what's available if headers: logging.warning(f"All headers: {dict(headers)}") # Continue with agent logic... ``` --- # DataRobot agentic skills URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-skills.html > Install and use DataRobot agentic skills in Claude Code, Cursor, Codex, Gemini CLI, and other coding agents. DataRobot agentic skills are modular capability packages that help a coding agent execute DataRobot workflows reliably. Each skill bundles instructions, examples, and supporting resources in a `SKILL.md` file so the agent loads only what it needs for the current task. Skills are maintained in the [DataRobot Agentic Skills repository](https://github.com/datarobot-oss/datarobot-agent-skills) and work with Claude Code, Cursor, Codex, Gemini CLI, VS Code Copilot, OpenCode, and other coding agents. | Section | Description | | --- | --- | | Quick start | Install skills, verify the installation, and configure the environment. | | Install from agent marketplaces | Recommended install path for Claude Code, Cursor, Gemini CLI, and Codex. | | Install with the universal installer | Install all skills or target a specific skill or agent with one command. | | Use skills in a coding agent | Prompt examples and helper scripts. | | Available skills | Full skill catalog grouped by workflow. | | Next steps | Continue with Agent Assist or the Agentic Starter template. | ## Quick start Complete these steps in order on first install: 1. Install from a marketplace for Claude Code, Cursor, Gemini CLI, or Codex, or install with the universal installer for other agents. 2. Verify installation by asking the coding agent which DataRobot skills are available. 3. Configure the environment by running the datarobot-setup skill once per workspace. 4. Use a skill in a prompt, or continue to Agent Assist or the Agentic Starter . > [!NOTE] Skills nomenclature > "Skills" is an Anthropic term used in Claude AI and Claude Code, but the concept applies more broadly. OpenAI Codex uses `AGENTS.md` to define agent instructions, and Gemini uses `gemini-extension.json` for extensions. The DataRobot skills repository is compatible with all of them, and more. **List of supported agents** Supported agents for DataRobot skills include [Claude Code](https://www.anthropic.com/claude-code/), [Cursor](https://cursor.com), [Codex](https://developers.openai.com/codex/), [Amp](https://ampcode.com/), [VS Code Copilot (GitHub Copilot)](https://github.com/features/copilot), [Gemini CLI](https://geminicli.com/), [Goose](https://block.github.io/goose/), [Letta](https://www.letta.com/), [Kilo Code](https://kilocode.ai/), [OpenCode](https://opencode.ai/), [Windsurf](https://windsurf.com/), and [Devin](https://devin.ai/). ## Install from agent marketplaces Install DataRobot skills from the coding agent marketplace or extension catalog. This is the recommended onboarding path for Claude Code, Cursor, Gemini CLI, and Codex. | Agent | Install surface | | --- | --- | | Claude Code | claude.com/plugins/datarobot-agent-skills | | Cursor | cursor.com/marketplace/datarobot | | Gemini CLI | geminicli.com/extensions | | Codex | developers.openai.com/codex/plugins | Select the tab for the coding agent in use: **Claude Code:** Install all DataRobot skills from the official Claude plugins marketplace. From the terminal: ``` claude plugin install datarobot-agent-skills@claude-plugins-official ``` From within a Claude Code CLI session: ``` /plugin install datarobot-agent-skills@claude-plugins-official ``` Alternative: register the repository as a plugin marketplace from within a Claude Code CLI session: ``` /plugin marketplace add datarobot-oss/datarobot-agent-skills ``` To install a specific skill: ``` /plugin install datarobot-model-training@datarobot-skills ``` **Cursor:** Open the [DataRobot marketplace entry](https://cursor.com/marketplace/datarobot) and click Add to Cursor, or run the following in Cursor chat: ``` /add-plugin datarobot-agent-skills ``` When the [DataRobot Agentic Skills repository](https://github.com/datarobot-oss/datarobot-agent-skills) is open as the workspace, Cursor also reads `AGENTS.md` and makes skills available without additional configuration. To verify that the skills are loaded, open the AI chat panel ( `Cmd/Ctrl + L`) and ask: "What DataRobot skills are available?" **Gemini CLI:** This repository includes `gemini-extension.json` for Gemini CLI integration. Install from the GitHub URL: ``` gemini extensions install https://github.com/datarobot-oss/datarobot-agent-skills.git --consent ``` Or install locally after cloning the repository: ``` gemini extensions install . --consent ``` For more information, see the [Gemini CLI extensions](https://geminicli.com/docs/extensions/) documentation. **Codex:** Codex identifies skills through the `AGENTS.md` file in the workspace. Install skills with the universal installer, targeting Codex: ``` npx ai-agent-skills install datarobot-oss/datarobot-agent-skills --agent codex ``` Verify that the instructions are loaded: ``` codex --ask-for-approval never "Summarize the current instructions." ``` For plugin discovery and installation in ChatGPT and Codex, see the [Codex plugins](https://developers.openai.com/codex/plugins) documentation. For details on how Codex reads agent instructions, see the [CodexAGENTS.mdguide](https://developers.openai.com/codex/guides/agents-md). ## Install with the universal installer Install all DataRobot skills, or only the ones needed, for all supported AI agents with one command by using the [universal skills installer](https://github.com/skillcreatorai/Ai-Agent-Skills). To install all skills: ``` npx ai-agent-skills install datarobot-oss/datarobot-agent-skills ``` To install a specific skill: ``` npx ai-agent-skills install datarobot-oss/datarobot-agent-skills/skills/datarobot-predictions ``` To target a specific agent: ``` npx ai-agent-skills install datarobot-oss/datarobot-agent-skills --agent cursor npx ai-agent-skills install datarobot-oss/datarobot-agent-skills --agent claude ``` > [!NOTE] Default behavior > By default, the installer copies skills to all supported agents at the same time. No configuration is required. > For agent-specific installation methods, see [Install from agent marketplaces](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-skills.html#install-from-agent-marketplaces). ### Install on other coding agents Use the following methods to install DataRobot skills on additional supported coding agents. #### VS Code Copilot (GitHub Copilot) VS Code with GitHub Copilot automatically detects and uses skills from this repository through the `AGENTS.md` file. 1. Open this repository in VS Code. 2. Ensure that the GitHub Copilot extension is installed and activated. 3. Skills are automatically available through the AGENTS.md file. Open Copilot Chat ( `Cmd/Ctrl + I`) and ask "What DataRobot skills are available?" to verify that the skills are loaded. > [!TIP] Tip > The `@workspace` agent in Copilot Chat provides full context about the repository and available skills. #### OpenCode Add to `~/.config/opencode/opencode.json`: ``` { "plugin": ["opencode-datarobot-skills"] } ``` OpenCode automatically installs the plugin on startup. For more information, see the [DataRobot OpenCode plugin](https://docs.datarobot.com/en/docs/agentic-ai/cli/opencode-plugin.html). #### Windsurf and Devin [Windsurf](https://windsurf.com/) and [Devin](https://devin.ai/) support the Agent Skills format. Install DataRobot skills using the [universal installer](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-skills.html#install-with-universal-installer). ## Verify installation After installing, ask the coding agent: ``` What DataRobot skills do I have available? ``` The response includes skills such as `datarobot-setup` and `datarobot-agent-assist`. ## Configure the environment Before using platform-connected skills, run the `datarobot-setup` skill once per workspace. It checks Python and other dependencies, authenticates `dr-cli`, and optionally adds `DATAROBOT_ENDPOINT` and `DATAROBOT_API_TOKEN` to the shell configuration. Trigger setup with either: - Run datarobot-setup - /datarobot-setup ## Use skills in a coding agent After a skill is installed, mention it directly in instructions to the coding agent: - "Use the DataRobot model training skill to create a new project and start AutoML training." - "Use the DataRobot predictions skill to generate a prediction dataset template for deployment abc123." - "Use the DataRobot feature engineering skill to analyze feature importance for my model." - "Use the DataRobot model monitoring skill to check Data Drift for deployment xyz789." - "Use the DataRobot external agent monitoring skill to instrument my agent for DataRobot monitoring." The coding agent loads the corresponding `SKILL.md` instructions and any helper scripts needed while completing the task. ### Helper scripts Some skills include helper scripts that an agent can run directly: - datarobot-predictions : get_deployment_features.py , generate_prediction_data_template.py , validate_prediction_data.py , make_prediction.py - datarobot-model-training : create_project.py , start_training.py , list_models.py - datarobot-model-explainability : compute_shap_matrix.py - datarobot-data-preparation : upload_dataset.py - datarobot-external-agent-monitoring : create_shell_deployment.py , verify_otel_connection.py - datarobot-agent-assist : select_framework.py , clone_template.py , setup_template.py , list_llm_models.py , rehearsal.py , env_utils.py These scripts are located in each skill `scripts/` directory and can be executed directly or used as references when writing code. ## Available skills The [DataRobot Agentic Skills repository](https://github.com/datarobot-oss/datarobot-agent-skills) contains skills for common DataRobot workflows. Contributors can also add their own skills. > [!TIP] Start with these skills > datarobot-setup > : Configure local development tools, authentication, and environment variables. > datarobot-agent-assist > : Design, build, simulate, and deploy agents to DataRobot. > datarobot-discover > : Find skills, MCP servers, agents, and platform resources. ### Onboarding skills The following skills support first-time setup and agent onboarding: | Skill Folder | Description | Documentation | | --- | --- | --- | | skills/datarobot-setup/ | Local DataRobot development setup (SDK, dr-cli, Agent Assist). | SKILL.md | | skills/datarobot-agent-assist/ | Design, build, simulate, and deploy AI agents to DataRobot. | SKILL.md | | skills/datarobot-discover/ | Discover DataRobot skills, MCP servers, agents, and platform resources. | SKILL.md | ### Agent development The following skills support agent design, deployment, and observability: | Skill Folder | Description | Documentation | | --- | --- | --- | | skills/datarobot-agent-llm-selection/ | Configure LLM integration for a DataRobot agent application and safely synchronize provider credentials. | SKILL.md | | skills/datarobot-app-framework-cicd/ | Set up CI/CD pipelines for DataRobot application templates with GitLab and GitHub Actions. | SKILL.md | | skills/datarobot-external-agent-monitoring/ | Instrument external agents with OpenTelemetry for DataRobot monitoring and observability. | SKILL.md | ### ML workflows The following skills support model training, deployment, predictions, and monitoring: | Skill Folder | Description | Documentation | | --- | --- | --- | | skills/datarobot-model-training/ | Instructions and utilities for training models, managing projects, and running AutoML experiments. | SKILL.md | | skills/datarobot-model-deployment/ | Tools for deploying models, managing deployments, and configuring prediction environments. | SKILL.md | | skills/datarobot-predictions/ | Guidance for making predictions, batch scoring, real-time predictions, and generating prediction datasets. | SKILL.md | | skills/datarobot-feature-engineering/ | Instructions for feature engineering, Feature Discovery, and feature importance analysis. | SKILL.md | | skills/datarobot-model-monitoring/ | Tools for monitoring model performance, tracking Data Drift, and managing model health. | SKILL.md | | skills/datarobot-model-explainability/ | Tools for model explainability, Prediction Explanations, SHAP values, and model diagnostics. | SKILL.md | | skills/datarobot-data-preparation/ | Utilities for data upload, dataset management, and data validation. | SKILL.md | ### Platform and integration The following skills support platform integration and workload management: | Skill Folder | Description | Documentation | | --- | --- | --- | | skills/datarobot-workload-api/ | Create, configure, debug, observe, and roll out container workloads on the Workload API. | SKILL.md | ### Skill structure Skills are self-contained folders that package instructions, scripts, and resources for a specific use case. Each folder includes a `SKILL.md` file with YAML frontmatter ( `name` and `description`), followed by the guidance the coding agent uses while the skill is active. > [!NOTE] Skill naming convention > All DataRobot skills follow the naming convention `datarobot-`, where `` describes the skill focus area. This provides clear identification of DataRobot-specific skills, consistent naming across the skill library, and easy discovery and organization. ## Next steps After installing skills and running setup, choose an onboarding path to create a first agentic workflow: | Topic | Description | | --- | --- | | Start with Agent Assist | Design, code, and deploy an agent with dr assist through natural conversation. | | Get started with the Agentic Starter | Install prerequisites, build, deploy, and test agentic workflows using DataRobot pre-built templates with dr start. | | Agent Assist reference | Workflows, environment variables, slash commands, and troubleshooting for Agent Assist. | ## See also - For the latest instructions, scripts, and templates, see datarobot-oss/datarobot-agent-skills . - For the libraries and workflows referenced in each skill, see the DataRobot documentation . - For API reference, see the DataRobot Python SDK documentation . --- # Migrate existing agents to current version URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-starter-migration.html > Migrate Agentic Starter projects from pre-11.8.1 class-based agents to the factory-based layout, by framework. Users with agents constructed using the [Agentic Starter template](https://github.com/datarobot-community/datarobot-agent-application) prior to [version 11.8.1](https://github.com/datarobot-community/datarobot-agent-application/releases/tag/v11.8.1) must perform a one-time update to maintain compatibility with newer releases. This is due to a change in the way the template constructs agents and how tools reach the workflow: - LangGraph, CrewAI, and LlamaIndex no longer rely on a handwritten MyAgent subclass. Build the native framework object (graph, crew, or workflow) first, then wrap it with a datarobot_agent_class_from_* factory that produces MyAgent . - Base (generic) still subclasses BaseAgent , with simplified construction and LLM wiring. custompy_adaptor passes llm=get_llm(...) , consistent with other templates. - NAT (NVIDIA NeMo Agent Toolkit) still subclasses NatAgent . __init__ is smaller, custompy_adaptor omits model= , and LLMs stay declarative in workflow.yaml . Across these patterns, the LLM is decoupled from ad hoc class state. Framework-specific `get_llm()` helpers resolve the model inside `custompy_adaptor` (and, for DRAgent, in `register.py`). MCP and workflow tools are injected at runtime instead of being read from `self.mcp_tools` on a `LangGraphAgent` -style subclass. This page summarizes migration steps by framework. For step-by-step diffs, before-and-after examples, and test updates, see the [framework migration guides](https://github.com/datarobot-community/datarobot-agent-application/tree/main/docs/agent) in the template repository ( `docs/agent/`). ## Before you start - Back up your repository and .env file before you merge template updates. - Update the template (pull the latest Agentic Starter changes, or run dr component update for the agent component) so pyproject.toml , uv.lock , and generated stubs match the version you target. - Rename environment variables if you still use deprecated names. Deployed LLM configuration expects LLM_DEPLOYMENT_ID (not TEXTGEN_DEPLOYMENT_ID ). See Configure LLM providers with metadata . - Keep the MyAgent export—infrastructure and tests expect that symbol. Do not rename it. ## Changes that apply to most frameworks ### custompy_adaptor and the LLM In `custompy_adaptor`, replace passing a `model=` string into `MyAgent(...)` with passing `llm=` from `get_llm(...)` in the appropriate module: - LangGraph / Base: datarobot_genai.langgraph.llm.get_llm - CrewAI: datarobot_genai.crewai.llm.get_llm (CrewAI may need extra parameters , for example {"stream_options": None} ) - LlamaIndex: datarobot_genai.llama_index.llm.get_llm Filter placeholder model names with a small set such as `_PLACEHOLDER_MODELS = frozenset({"unknown"})` so a placeholder `model_name` from DataRobot does not force an invalid model. ### Configuration In `myagent.py`, remove `from agent.config import Config` where the new pattern delegates configuration to `get_llm()` and the runtime environment. If your template still loads MCP-related settings in `custom.py`, keep `Config` there. ### Tests Across frameworks, tests that constructed `MyAgent(model=..., api_key=...)` should use `MyAgent(llm=Mock(), ...)` instead. Remove tests that target deleted `llm()` methods or old `__init__` signatures. Add tests for `graph_factory` or other module-level wiring where applicable. ## Framework-specific migration ### LangGraph - Replace class MyAgent(LangGraphAgent) with: - A module-level prompt_template - Remove llm() , __init__ boilerplate, and Config imports from myagent.py as described in the LangGraph migration guide . ### CrewAI - Move agents, tasks, crew, and kickoff_inputs to module level; call get_llm() once at module scope where appropriate. - Replace the class with MyAgent = datarobot_agent_class_from_crew(crew, agents, tasks, kickoff_inputs) . - Tools are injected at runtime; do not rely on self.tools on the old class. - Crew stream : often False at module scope; DRAgent can adjust streaming at runtime (see the migration guide in the template repository). ### LlamaIndex - Use a module-level LiteLLM(model="placeholder") (or equivalent) for agents until custompy_adaptor injects the real LLM. - Move agents, workflow, state tools, and extract_response_text to module level. Remove make_input_message (the factory handles it). - MyAgent = datarobot_agent_class_from_llamaindex(workflow, agents, extract_response_text) . ### Base (generic) - MyAgent still subclasses BaseAgent . __init__ is simplified because BaseAgent owns initialization. - Remove manual _llm , self.config , and Config usage from the class. Pass llm=get_llm(...) from custompy_adaptor as in the Base migration guide . ### NAT - NAT requires fewer structural changes than other frameworks: MyAgent(NatAgent) still subclasses NatAgent , but __init__ forwards *args / **kwargs to super() and supplies a default workflow_path . - In custompy_adaptor , omit model= when constructing MyAgent ; LLMs are defined in workflow.yaml . ## After you migrate 1. Run dr task run agent:install (or your template’s install task), then dr task run agent:test . 2. Start the local stack with dr run dev or task dev , exercise the app, and run task agent:cli as described in Customize agents . 3. Deploy to a non-production environment first. Confirm traces and MCP behavior in the DataRobot UI. For ongoing development tasks after migration, see [Customize agents](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-development.html) and [Agent components](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-overview.html). --- # Add tools to agents URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools-integrate.html > Learn how to add local tools, predefined tools, and DataRobot global tools to your agentic workflows. To add local tools to your agents, you can start by modifying the `myagent.py` file or adding new files to the `agent` directory; the following examples will help you get started. Note that the structure and implementation of a tool is framework-specific. The examples provided here are for CrewAI, LangGraph, LlamaIndex, and NAT (NVIDIA NeMo Agent Toolkit). If you are using another framework, you will need to refer to the framework's documentation for details on how to implement tools. ## Call tools from an agent Once you have defined your tool, you can call it from your agent by adding it to the list of tools available to the agent. This is typically done in the `myagent.py` file where the agent is defined. You will need to import the tool class and add it to the agent's tool list. The following simple examples illustrate one way of doing this for each framework. **CrewAI:** To add a tool to a CrewAI agent, you can modify an agent in the `myagent.py` file in the `agent` directory. An example of modifying the `agent_planner` agent to use the sample tool from the [Local Datetime Tool Example](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools-integrate.html#local-datetime-tool-example) is shown below: ``` @property def agent_planner(self) -> Agent: datetime_tool = DateTimeTool() # Import and instantiate your tool here return Agent( role="Content Planner", goal="Plan engaging and factually accurate content on {topic}", backstory="...", # truncated for brevity in this example allow_delegation=False, verbose=self.verbose, llm=self.llm(), tools=[datetime_tool] # Add your tool to the tools list here ) ``` **LangGraph:** To add a tool to a LangGraph agent, extend the `tools` list inside `graph_factory` in `myagent.py` (current [Agentic Starter](https://github.com/datarobot-community/datarobot-agent-application) templates use `graph_factory(llm, tools, verbose)` rather than `self.mcp_tools` on a subclass). The snippet below adds the sample tool from the [Local Datetime Tool Example](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools-integrate.html#local-datetime-tool-example) alongside MCP and workflow tools already passed in as `tools`: ``` from datarobot_genai.core.agents import make_system_prompt from langchain.agents import create_agent def graph_factory(llm, tools, verbose=False): datetime_tool = DateTimeTool() # Import and instantiate your tool here all_tools = [datetime_tool] + list(tools) planner = create_agent( llm, tools=all_tools, system_prompt=make_system_prompt( "...", # truncated for brevity in this example ), name="planner_agent", debug=verbose, ) ... ``` **LlamaIndex:** To add a tool to a LlamaIndex agent, you can modify an agent in the `myagent.py` file in the `agent` directory. An example of modifying the `agent_planner` agent to use the sample tool from the [Local Datetime Tool Example](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools-integrate.html#local-datetime-tool-example) is shown below: ``` @property def agent_planner(self) -> FunctionAgent: datetime_tool = DateTimeTool() return FunctionAgent( name="PlannerAgent", description="...", # truncated for brevity in this example system_prompt=( "..." # truncated for brevity in this example ), llm=self.llm(), tools=[self.planner_notes_tool, *self.mcp_tools, datetime_tool], can_handoff_to=["WriterAgent"], ) ``` **NAT:** In NAT templates, tools are defined as functions in the `workflow.yaml` file. Functions are defined in the `functions` section with `_type: chat_completion` and referenced in the workflow's `tool_list`. Refer to the [NVIDIA NeMo Agent Toolkit documentation](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html) for details on implementing tools in NAT. ## Use predefined tools Some frameworks provide predefined tools that you can use directly in your agents. For example, CrewAI provides a `SearchTool` that can be used to perform web searches. You can refer to the framework documentation for a list of predefined tools and how to use them. These tools can be added to your agent by simply importing them from the framework and adding them to the agent's tool list [as shown in the examples above](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools-integrate.html#call-tools-from-an-agent). > [!NOTE] LangChain compatibility > Most agentic workflow frameworks (including CrewAI, LangGraph, and many others) are natively compatible with LangChain tools. This means you can import and use them without any modifications. You can refer to the [LangChain Tools Documentation](https://python.langchain.com/docs/concepts/tools) for more information on using LangChain tools in your agents. ### Local datetime tool The following examples show how to create a custom local datetime tool for your agents. This tool returns the current date and time, allowing the agent to be aware of and use the current date and time in its responses or actions. This tool does not require any network or file access and can be implemented and run without any additional permissions or credentials. > [!NOTE] Starting point for local tools > This example can be used as a starting point for creating other local tools that do not require external access. **CrewAI:** To add a local datetime tool to a CrewAI agent, you can modify the `myagent.py` file in the `agent` directory. You can add the following code to define the datetime tool: ``` from datetime import datetime from typing import Optional, Type from zoneinfo import ZoneInfo from pydantic import BaseModel, Field from crewai.tools import BaseTool class DateTimeToolInput(BaseModel): """Input schema for DateTimeTool.""" timezone: Optional[str] = Field( None, description="IANA timezone string (e.g., 'America/New_York', 'Europe/London', 'Asia/Tokyo'). " "If not provided, returns local time.", ) class DateTimeTool(BaseTool): name: str = "datetime_tool" description: str = ( "Returns the current date and time. Optionally accepts a timezone parameter as an IANA timezone " "string (e.g., 'America/New_York', 'Europe/London', 'Asia/Tokyo'). If no timezone is provided, " "returns local time.") args_schema: Type[BaseModel] = DateTimeToolInput def _run(self, timezone: Optional[str] = None) -> str: try: # If the agent provides the timezone parameter, use it to get the current time in that timezone if timezone: # Use the specified timezone tz = ZoneInfo(timezone) current_time = datetime.now(tz) return f"{current_time.strftime('%Y-%m-%d %H:%M:%S')} ({timezone})" # Return the current local time if the agent does not have or provide the timezone parameter else: # Use local time return datetime.now().strftime("%Y-%m-%d %H:%M:%S") # Gracefully handle errors and exceptions so the agent can understand the error # and attempt to correct in a followup call of the tool if needed. except Exception: return ( f"Error: Invalid timezone '{timezone}'. " f"Please use a valid IANA timezone string (e.g., 'America/New_York', 'Europe/London')." ) ``` **LangGraph:** To add a local datetime tool to a LangGraph agent, you can modify the `myagent.py` file in the `agent` directory. You can add the following code to define the datetime tool: ``` from datetime import datetime from typing import Optional from zoneinfo import ZoneInfo from langchain.tools import BaseTool class DateTimeTool(BaseTool): name: str = "datetime_tool" description: str = ( "Returns the current date and time. Optionally accepts a timezone parameter as an IANA timezone " "string (e.g., 'America/New_York', 'Europe/London', 'Asia/Tokyo'). If no timezone is provided, " "returns local time.") def _run(self, timezone: Optional[str] = None) -> str: try: # If the agent provides the timezone parameter, use it to get the current time in that timezone if timezone: # Use the specified timezone tz = ZoneInfo(timezone) current_time = datetime.now(tz) return f"{current_time.strftime('%Y-%m-%d %H:%M:%S')} ({timezone})" # Return the current local time if the agent does not have or provide the timezone parameter else: # Use local time return datetime.now().strftime("%Y-%m-%d %H:%M:%S") # Gracefully handle errors and exceptions so the agent can understand the error # and attempt to correct in a followup call of the tool if needed. except Exception: return ( f"Error: Invalid timezone '{timezone}'. " f"Please use a valid IANA timezone string (e.g., 'America/New_York', 'Europe/London')." ) ``` **LlamaIndex:** To add a local datetime tool to a LlamaIndex agent, you can modify the `myagent.py` file in the `agent` directory. You can add the following code to define the datetime tool: ``` from datetime import datetime from typing import Optional from zoneinfo import ZoneInfo from llama_index.core.tools import FunctionTool def _datetime_run(timezone: Optional[str] = None) -> str: """Returns the current date and time. Optionally accepts a timezone parameter as an IANA timezone string (e.g., 'America/New_York', 'Europe/London', 'Asia/Tokyo'). If no timezone is provided, returns local time.""" try: # If the agent provides the timezone parameter, use it to get the current time in that timezone if timezone: # Use the specified timezone tz = ZoneInfo(timezone) current_time = datetime.now(tz) return f"{current_time.strftime('%Y-%m-%d %H:%M:%S')} ({timezone})" # Return the current local time if the agent does not have or provide the timezone parameter else: # Use local time return datetime.now().strftime("%Y-%m-%d %H:%M:%S") # Gracefully handle errors and exceptions so the agent can understand the error # and attempt to correct in a followup call of the tool if needed. except Exception: return ( f"Error: Invalid timezone '{timezone}'. " f"Please use a valid IANA timezone string (e.g., 'America/New_York', 'Europe/London')." ) def DateTimeTool() -> FunctionTool: return FunctionTool.from_defaults( fn=_datetime_run, name="datetime_tool", description=( "Returns the current date and time. Optionally accepts a timezone parameter as an IANA timezone " "string (e.g., 'America/New_York', 'Europe/London', 'Asia/Tokyo'). If no timezone is provided, " "returns local time." ), ) ``` **NAT:** Refer to the [NVIDIA NeMo Agent Toolkit documentation](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html) for details on implementing local tools in NAT. ### Weather API tool The following examples show how to create a custom tool that fetches weather information from a public API. This tool will require network access to fetch the weather data. You will need to ensure that your agent has the necessary permissions to make network requests and the appropriate API credentials required by the weather service. > [!NOTE] Starting point for external API tools > This example can be used as a starting point for tools that need to communicate with external APIs. **CrewAI:** To add a weather API tool to a CrewAI agent, you can modify the `myagent.py` file in the `agent` directory. You can add the following code to define the weather tool: ``` import requests from typing import Type from pydantic import BaseModel, Field from crewai.tools import BaseTool class WeatherToolInput(BaseModel): """Input schema for WeatherTool.""" city: str = Field( ..., description="The name of the city to fetch weather for (e.g., 'London', 'New York')." ) class WeatherTool(BaseTool): name: str = "weather_tool" description: str = ( "Fetches the current weather for a specified city. Usage: weather_tool(city='City Name'). " "Requires an API key from OpenWeatherMap. Sign up at https://openweathermap.org/api.") args_schema: Type[BaseModel] = WeatherToolInput def _run(self, city: str) -> str: api_key = "YOUR_API_KEY" # Replace with your OpenWeatherMap API key base_url = "http://api.openweathermap.org/data/2.5/weather" params = {"q": city, "appid": api_key, "units": "metric"} try: # Submit a query to an API using requests response = requests.get(base_url, params=params, timeout=10) response.raise_for_status() # Collect and format the response data = response.json() weather = data['weather'][0] main = data['main'] # Format and return the response to the agent return ( f"Current weather in {data['name']}, {data['sys']['country']}:\n" f"Temperature: {main['temp']}°C (feels like {main['feels_like']}°C)\n" f"Condition: {weather['main']} - {weather['description']}\n" f"Humidity: {main['humidity']}%\n" f"Pressure: {main['pressure']} hPa" ) # Gracefully handle errors and exceptions so the agent can understand the error # and attempt to correct in a followup call of the tool if needed. except requests.exceptions.RequestException as e: return f"Error fetching weather data: {str(e)}" except KeyError as e: return f"Error parsing weather data: Missing key {str(e)}" except Exception as e: return f"Unexpected error: {str(e)}" ``` **LangGraph:** To add a weather API tool to a LangGraph agent, you can modify the `myagent.py` file in the `agent` directory. You can add the following code to define the weather tool: ``` import requests from langchain.tools import BaseTool class WeatherTool(BaseTool): name: str = "weather_tool" description: str = ( "Fetches the current weather for a specified city. Usage: weather_tool(city='City Name'). " "Requires an API key from OpenWeatherMap. Sign up at https://openweathermap.org/api.") def _run(self, city: str) -> str: api_key = "YOUR_API_KEY" # Replace with your OpenWeatherMap API key base_url = "http://api.openweathermap.org/data/2.5/weather" params = {"q": city, "appid": api_key, "units": "metric"} try: # Submit a query to an API using requests response = requests.get(base_url, params=params, timeout=10) response.raise_for_status() # Collect and format the response data = response.json() weather = data['weather'][0] main = data['main'] # Format and return the response to the agent return ( f"Current weather in {data['name']}, {data['sys']['country']}:\n" f"Temperature: {main['temp']}°C (feels like {main['feels_like']}°C)\n" f"Condition: {weather['main']} - {weather['description']}\n" f"Humidity: {main['humidity']}%\n" f"Pressure: {main['pressure']} hPa" ) # Gracefully handle errors and exceptions so the agent can understand the error # and attempt to correct in a followup call of the tool if needed. except requests.exceptions.RequestException as e: return f"Error fetching weather data: {str(e)}" except KeyError as e: return f"Error parsing weather data: Missing key {str(e)}" except Exception as e: return f"Unexpected error: {str(e)}" ``` **LlamaIndex:** To add a weather API tool to a LlamaIndex agent, you can modify the `myagent.py` file in the `agent` directory. You can add the following code to define the weather tool: ``` import requests from llama_index.core.tools import FunctionTool def _weather_run(city: str) -> str: """Fetches the current weather for a specified city. Requires a city name as input.""" api_key = "YOUR_API_KEY" # Replace with your OpenWeatherMap API key base_url = "http://api.openweathermap.org/data/2.5/weather" params = {"q": city, "appid": api_key, "units": "metric"} try: # Submit a query to an API using requests response = requests.get(base_url, params=params, timeout=10) response.raise_for_status() # Collect and format the response data = response.json() weather = data['weather'][0] main = data['main'] # Format and return the response to the agent return ( f"Current weather in {data['name']}, {data['sys']['country']}:\n" f"Temperature: {main['temp']}°C (feels like {main['feels_like']}°C)\n" f"Condition: {weather['main']} - {weather['description']}\n" f"Humidity: {main['humidity']}%\n" f"Pressure: {main['pressure']} hPa" ) # Gracefully handle errors and exceptions so the agent can understand the error # and attempt to correct in a followup call of the tool if needed. except requests.exceptions.RequestException as e: return f"Error fetching weather data: {str(e)}" except KeyError as e: return f"Error parsing weather data: Missing key {str(e)}" except Exception as e: return f"Unexpected error: {str(e)}" def WeatherTool() -> FunctionTool: return FunctionTool.from_defaults( fn=_weather_run, name="weather_tool", description=( "Fetches the current weather for a specified city. Usage: weather_tool(city='City Name'). " "Requires an API key from OpenWeatherMap. Sign up at https://openweathermap.org/api." ), ) ``` **NAT:** Refer to the [NVIDIA NeMo Agent Toolkit documentation](https://docs.nvidia.com/nemo/agent-toolkit/latest/index.html) for details on implementing external API tools in NAT. ## Integrate DataRobot deployed tools DataRobot provides a set of global tools that can be used across different agent frameworks. These tools are designed to interact with DataRobot's platform and services. The following examples demonstrate a global tool that searches the DataRobot Data Registry for datasets. This tool will require network access to fetch the data from DataRobot. Before integrating these tools into an agent, [deploy them in DataRobot](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools.html). > [!NOTE] DataRobot service integration > This example can be modified to create any tool that interacts with DataRobot's services. Source code for many global tools can be found in the [agent-tool-templatesrepository](https://github.com/datarobot-oss/agent-tool-templates). When building agents with external tools—either global tools or custom tools created as custom models in Workshop—the following components are required to call deployed tools from an agent: - The ToolClient class instance from the datarobot-genai package. - The resolve_authorization_context function from the datarobot-genai package (called in custom.py ). - A deployed tool's deployment_id , defined in model-metadata.yaml . - The tool's metadata, defined in a tool module; for example, tool_ai_catalog_search.py . The examples in this documentation show how to add tool integration capabilities to agent templates. The base agent templates provided in this repository do not include tool implementations by default—these examples demonstrate the patterns you can follow to integrate deployed tools into your own agent implementations. To assemble an agentic workflow using agentic tools deployed from Registry, an example workflow could include the following files: | File | Contents | | --- | --- | | __init__.py | Python package initialization file, making the directory a Python package. | | custom.py | The custom model code implementing the Bolt-on Governance API (the chat hook) to call the LLM and also passing those parameters to the agent (defined in myagent.py). | | myagent.py | The agent code, implementing the agentic workflow in the MyAgent class with the required invoke method. Tool integration properties can be added to this class to interface with deployed tools. | | config.py | The code for loading the configuration from environment variables, runtime parameters, and DataRobot credentials. | | mcp_client.py | The code providing MCP server connection management for tool integration (optional, only needed when using MCP tools). | | tool_deployment.py | The BaseTool class code, containing all necessary metadata for implementing tools. | | tool.py | The code for interfacing with the deployed tool, defining the input arguments and schema. Often, this file won't be named tool.py, as you may implement more than one tool. In this example, this functionality is defined in tool_ai_catalog_search.py. | | model-metadata.yaml | The custom model metadata and runtime parameters required by the agentic workflow. | | pyproject.toml | The libraries (and versions) required by the agentic workflow, using modern Python packaging standards. | ### Implement the ToolClient class instance Every agent template and framework requires the `ToolClient` class from the `datarobot-genai` package to offload tool call processing to deployed global tools. The tool client calls the deployed tool and returns the results to the agent. To import the `ToolClient` module into a `myagent.py` file, use the following import statement: ``` # agent/agent/myagent.py from datarobot_genai.core.chat.client import ToolClient ``` The `ToolClient` is available in the [datarobot-genaipackage](https://github.com/datarobot-oss/datarobot-genai). It defines the API endpoint and deployment ID for the deployed tool, gets the authorization context (if required), and provides interfaces for the `score`, `score_unstructured`, and `chat` hooks. After you import the `ToolClient` into `myagent.py`, you can add a `tool_client` property to your `MyAgent` class. The `ToolClient` automatically uses environment variables `DATAROBOT_API_TOKEN` and `DATAROBOT_ENDPOINT` for authentication, but you can also pass these explicitly if needed. ``` # agent/agent/myagent.py from datarobot_genai.core.chat.client import ToolClient class MyAgent: # More agentic workflow code. @property def tool_client(self): """ToolClient instance for calling deployed tools.""" return ToolClient( api_key=self.api_key, base_url=self.api_base, ) # More agentic workflow code. ``` ### (Optional) Initialize authorization context for external tools Authorization context is required to allow downstream agents and tools to retrieve access tokens when connecting to external services. The authorization context functionality is available in the `datarobot-genai` package alongside the `ToolClient` class. The `resolve_authorization_context` function is available in the `datarobot-genai` package and handles resolving the authorization context from the completion parameters and request headers. This function returns an authorization context dictionary that should be assigned to `completion_create_params["authorization_context"]`. The authorization context uses utility methods from the `datarobot` SDK: * `set_authorization_context`: A method to set the authorization context for the current process. * `get_authorization_context`: A method to retrieve the authorization context for the current process. > [!NOTE] OAuth utility method availability > These utility methods are available in the DataRobot Python API client starting with version 3.8.0. You can review the `resolve_authorization_context` function in the [datarobot-genaipackage](https://github.com/datarobot-oss/datarobot-genai). The `resolve_authorization_context` function is available in the `datarobot-genai` package. It resolves the authorization context for the agent, which is required for propagating information needed by downstream agents and tools to retrieve access tokens to connect to external services. When set, authorization context will be automatically propagated when using the `ToolClient` class. In the `custom.py` example below, the `chat()` hook calls `resolve_authorization_context` (imported from `datarobot-genai`) each time a chat request is made to the agentic workflow, and assigns the result to `completion_create_params["authorization_context"]`, providing any credentials required for external tools. ``` # agent/custom.py import asyncio from concurrent.futures import ThreadPoolExecutor from typing import Any, Iterator, Union from agent import MyAgent from config import Config from datarobot_genai.core.chat import ( CustomModelChatResponse, CustomModelStreamingResponse, resolve_authorization_context, to_custom_model_chat_response, to_custom_model_streaming_response, ) from openai.types.chat import CompletionCreateParams from openai.types.chat.completion_create_params import ( CompletionCreateParamsNonStreaming, CompletionCreateParamsStreaming, ) def load_model(code_dir: str) -> tuple[ThreadPoolExecutor, asyncio.AbstractEventLoop]: """The agent is instantiated in this function and returned.""" thread_pool_executor = ThreadPoolExecutor(1) event_loop = asyncio.new_event_loop() thread_pool_executor.submit(asyncio.set_event_loop, event_loop).result() return (thread_pool_executor, event_loop) def chat( completion_create_params: CompletionCreateParams | CompletionCreateParamsNonStreaming | CompletionCreateParamsStreaming, load_model_result: tuple[ThreadPoolExecutor, asyncio.AbstractEventLoop], **kwargs: Any, ) -> Union[CustomModelChatResponse, Iterator[CustomModelStreamingResponse]]: """When using the chat endpoint, this function is called. Agent inputs are in OpenAI message format and defined as the 'user' portion of the input prompt. """ # Load configuration and runtime parameters _ = Config() # Initialize the authorization context for downstream agents and tools to retrieve # access tokens for external services. completion_create_params["authorization_context"] = resolve_authorization_context( completion_create_params, **kwargs ) # Instantiate the agent, all fields from the completion_create_params are passed to the agent # allowing environment variables to be passed during execution agent = MyAgent(**completion_create_params) if completion_create_params.get("stream"): streaming_response_generator = agent.invoke( completion_create_params=completion_create_params ) return to_custom_model_streaming_response( streaming_response_generator, model=completion_create_params.get("model") ) else: # Synchronous non-streaming response response_text, pipeline_interactions, usage_metrics = agent.invoke( completion_create_params=completion_create_params ) return to_custom_model_chat_response( response_text, pipeline_interactions, usage_metrics, model=completion_create_params.get("model"), ) ``` When authorization context is set, it is automatically propagated by the `ToolClient` class from the `datarobot-genai` package. The `ToolClient.call()` method automatically includes the authorization context when calling deployed tools. The `ToolClient` class provides methods to call the custom model tool using various hooks: `score`, `score_unstructured`, and `chat`. When the `authorization_context` is set, the client automatically propagates it to the agent tool. The `authorization_context` is required for retrieving access tokens to connect to external services. If you're implementing a custom tool reliant on an external service, you can use the `@datarobot_tool_auth` decorator to streamline the process of retrieving the authorization context, extracting the relevant data, and connecting to the DataRobot API to obtain the OAuth access token from [an OAuth provider configured in DataRobot](https://docs.datarobot.com/en/docs/platform/acct-settings/manage-oauth.html). When only one OAuth provider is configured the decorator doesn't require the `provider` parameter, as it will use the only available provider; however, if multiple providers are (or will be) available, you should define this parameter. ``` # tool.py from datarobot.models.genai.agent.auth import datarobot_tool_auth, AuthType # More tool code. @datarobot_tool_auth( type=AuthType.OBO, # on-behalf-of provider="google", # required with multiple OAuth providers ) def list_files_in_google_drive(folder_name: str, token: str = "") -> list[dict]: """The value for token parameter will be injected by the decorator.""" # More tool code. ``` ### Interface with tool deployments Global tools and tools custom-built in the Registry workshop must be deployed for the agent to call them. When these tools are deployed, communicating with them requires a deployment ID, used to interface with the tool through the DataRobot API. The primary method for providing a deployed tool's deployment ID to the agent is through environment variables, defined as runtime parameters in the agent's metadata. To provide this metadata, create or modify a `model-metadata.yaml` file to add the runtime parameter for each deployed tool the agent needs to communicate with. Define runtime parameters in `runtimeParameterDefinitions`. ``` # agent/model-metadata.yaml runtimeParameterDefinitions: - fieldName: AI_CATALOG_SEARCH_TOOL_DEPLOYMENT_ID defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string ``` The example below illustrates a `model-metadata.yaml` file configured for an `agenticworkflow` implementing two global tools, Search Data Registry and Get Data Registry Dataset. The field names in the example below are used by DataRobot agent templates implementing these tools; however, the `fieldName` is configurable, and must match the implementation in the agent's code, located in the `myagent.py` file. ``` # agent/model-metadata.yaml --- name: agent_with_tools type: inference targetType: agenticworkflow runtimeParameterDefinitions: - fieldName: OTEL_SDK_ENABLED defaultValue: true type: boolean - fieldName: LLM_DEPLOYMENT_ID defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string - fieldName: DATA_REGISTRY_SEARCH_TOOL_DEPLOYMENT_ID defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string - fieldName: DATA_REGISTRY_READ_TOOL_DEPLOYMENT_ID defaultValue: SET_VIA_PULUMI_OR_MANUALLY type: string ``` Unless you provided the required values as the `defaultValue`, you must set the runtime parameter to provide the deployment IDs to the agent's code. You can do this in two ways: - Manually: Configuring the values in the UI, in the Registry workshop, before deploying the agent. - Automatically: Configuring the values in a Pulumi script and passing them to the custom model. When the parameters are set, they're accessible in the agent code, as long as the `RuntimeParameters` class is imported from `datarobot_drum`. The following example shows how you can add tool properties to your `MyAgent` class to interface with deployed tools: ``` # agent/agent/myagent.py import os from datarobot_drum import RuntimeParameters from datarobot_genai.core.chat.client import ToolClient class MyAgent: # More agentic workflow code. @property def tool_client(self) -> ToolClient: """ToolClient instance for calling deployed tools.""" return ToolClient( api_key=self.api_key, base_url=self.api_base, ) @property def tool_ai_catalog_search(self) -> BaseTool: """Search AI Catalog tool using deployed tool.""" deployment_id = os.environ.get("AI_CATALOG_SEARCH_TOOL_DEPLOYMENT_ID") if not deployment_id: deployment_id = RuntimeParameters.get("AI_CATALOG_SEARCH_TOOL_DEPLOYMENT_ID") return SearchAICatalogTool( tool_client=self.tool_client, deployment_id=deployment_id ) @property def tool_ai_catalog_read(self) -> BaseTool: """Read AI Catalog tool using deployed tool.""" deployment_id = os.environ.get("AI_CATALOG_READ_TOOL_DEPLOYMENT_ID") if not deployment_id: deployment_id = RuntimeParameters.get("AI_CATALOG_READ_TOOL_DEPLOYMENT_ID") return ReadAICatalogTool( tool_client=self.tool_client, deployment_id=deployment_id, ) # More agentic workflow code. ``` ### Define tool metadata When building tools for an agent, the metadata defines how the agent LLM should call the tool. The more details the metadata provides, the more effectively the LLM uses the tool. The metadata includes the tool description and each arguments' schema and related description. Each framework has a unique way to define this metadata; however, in most cases you can leverage `pydantic` to import `BaseModel` to define the tool's arguments. ``` # tool_ai_catalog_search.py from pydantic import BaseModel as PydanticBaseModel, Field class SearchAICatalogArgs(PydanticBaseModel): search_terms: str = Field( default="", description="Terms for the search. Leave blank to return all datasets." ) limit: int = Field( default=20, description="The maximum number of datasets to return. " "Set to -1 to return all." ) ``` The example below implements a simple `BaseTool` class for CrewAI, implemented in `tool_deployment.py`, containing all necessary metadata and available for reuse across multiple CrewAI tools. ``` # tool_deployment.py from abc import ABC from crewai.tools import BaseTool from datarobot_genai.core.chat.client import ToolClient class BaseToolWithDeployment(BaseTool, ABC): model_config = { "arbitrary_types_allowed": True } """Adds support for arbitrary types in Pydantic models, needed for the ToolClient.""" tool_client: ToolClient """The tool client initialized by the agent with access to the ToolClient authorization context.""" deployment_id: str """The DataRobot deployment ID of the custom model executing tool logic.""" ``` The `SearchAICatalogTool`, defined in `tool_ai_catalog_search.py`, invokes `tool_deployment` to build off the `BaseToolWithDeployment` module. ``` # tool_ai_catalog_search.py import json from typing import Dict, List, Type from pydantic import BaseModel as PydanticBaseModel from tool_deployment import BaseToolWithDeployment class SearchAICatalogTool(BaseToolWithDeployment): name: str = "Search Data Registry" description: str = ( "This tool provides a list of all available dataset names and their associated IDs from the Data Registry. " "You should always check to see if the dataset you are looking for can be found here. " "For future queries, you should use the associated dataset ID instead of the name to avoid ambiguity." ) args_schema: Type[PydanticBaseModel] = SearchAICatalogArgs def _run(self, **kwargs) -> List[Dict[str, str]]: # Validate and parse the input arguments using the defined schema. validated_args = self.args_schema(**kwargs) # Call the tool deployment with the generated payload. result = self.tool_client.call( deployment_id=self.deployment_id, payload=validated_args.model_dump() ) # Format and return the results. return json.loads(result.data).get("datasets", []) ``` The example below uses the CrewAI framework to implement a tool through the `BaseTool`, `Agent`, and `Task` classes. The following methods show how you can add tool properties to your `MyAgent` class to initialize the Data Registry searching tool, define an LLM agent to search the Data Registry, and then define a task for the agent: ``` # agent/agent/myagent.py from crewai.tools import BaseTool from crewai import Agent, Task from tool_ai_catalog_search import SearchAICatalogTool from datarobot_genai.core.chat.client import ToolClient class MyAgent: # More agentic workflow code. @property def tool_client(self) -> ToolClient: """ToolClient instance for calling deployed tools.""" return ToolClient( api_key=self.api_key, base_url=self.api_base, ) @property def search_ai_catalog_tool(self) -> BaseTool: """Search AI Catalog tool using deployed tool.""" deployment_id = self.search_ai_catalog_deployment_id if not deployment_id: raise ValueError("Configure a deployment ID for the Search Data Registry tool.") return SearchAICatalogTool( tool_client=self.tool_client, deployment_id=deployment_id ) @property def agent_ai_catalog_searcher(self) -> Agent: """Agent configured to search the AI Catalog.""" return Agent( role="Expert Data Registry Searcher", goal="Search for and retrieve relevant files from Data Registry.", backstory="You are a meticulous analyst that is skilled at examining lists of files and " "determining the most appropriate file based on the context.", verbose=self.verbose, allow_delegation=False, llm=self.llm_with_datarobot_llm_gateway, ) @property def task_ai_catalog_search(self) -> Task: """Task for searching the AI Catalog.""" return Task( description=( "You should search for a relevant dataset id in the Data Registry " "based on the provided dataset topic: {dataset_topic}." ), expected_output=( "Search for a list of relevant files in the Data Registry and " "determine the most relevant dataset id that matches the given topic. " "You should return the entire dataset id." ), agent=self.agent_ai_catalog_searcher, tools=[self.search_ai_catalog_tool], ) # More agentic workflow code. ``` ## Framework-specific documentation You can also refer to the framework repositories and documentation for more information on constructing more advanced tools and agents: - CrewAI tools documentation - LangChain and LangGraph tools documentation - LlamaIndex tools documentation - NVIDIA NeMo Agent Toolkit documentation ## Agentic tool considerations When deploying the application and agent separately from the agentic tool (for example, deploying the application and agent via Pulumi after local development and the tool manually in DataRobot), all components must be deployed by the same user. Custom models use the creator's API key and require a common identity to store and retrieve authentication data from the OAuth Providers Service. --- # Deploy agentic tools URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools.html > Deploy tools to handle tasks critical to the agent workflow. When building agents, you often need to integrate tools to handle tasks critical to the agent workflow—typically for complex use cases involving communication with external services. While some tools are embedded directly in the code of an agentic workflow, other tools are deployed externally and called by the agent process. Because externally deployed tools can scale independently, they are well-suited for resource-intensive operations, I/O-bound tasks, and reusable functionality. Deploying tools externally also enables production-ready monitoring, mitigation, and moderation capabilities in Console. ## Global agentic tools The following global tools are available for deployment to Console: > [!TIP] Identifying tools > All global tools are prefixed with the [Tool]identifier. Use this identifier to filter the global models and tools list to show only tools. | Tool | Description | Notes | | --- | --- | --- | | Get Data Registry Dataset | Retrieves datasets from the DataRobot Data Registry using a dataset_id and returns the dataset in CSV format as raw bytes. | N/A | | Make AutoML Predictions | Accepts a pandas.DataFrame and uses that data to return a prediction from the specified predictive model. | The argument columns_to_return_with_predictions tells the tool to return columns from the input dataset. Use this to make sure you can interpret the predictions. For example, you may want to return an ID or other identifying column so that you can see which prediction is which because you can't rely on the index or order of the predictions. | | Make Text Generation Predictions | Accepts a string and returns a prediction from the specified DataRobot text generation model (LLM). | Suitable for tasks like summarization or text completion. This tool should only be used for TextGeneration deployments and not for regression, classification, or other target types. | | Make Time Series Predictions | Returns forecasts from a time series model. | Before using this tool, verify that you have all the data needed. Time series models require a forecast point. They also have specific requirements for the input data. | | Render Plotly Chart | Returns a JSON object containing a rendered Plotly chart object generated based on the provided specification and dataset ID. | When generating the Plotly chart, placeholders in the specification—indicated by double braces enclosing a column name (for example, {{ column_name }})—are replaced by the corresponding values from the specified column in the Data Registry dataset. The Data Registry dataset is identified by the dataset_id input parameter. | | Render Vega-Lite Chart | Generates a Vega-Lite chart by passing in the Vega-Lite specification in JSON format and returns JSON with a base64-encoded image of the chart. | To provide data for the chart, pass in the Data Registry dataset_id for the dataset you want to chart. | | Search Data Registry | Searches for datasets in the DataRobot Data Registry using search terms. Returns matching datasets as a pandas.DataFrame. | The Data Registry does not support partial matching. If this tool doesn't return the expected results, try again with a more specific search query. | | Summarize DataFrame | Provides a detailed summary of a pandas.DataFrame in Markdown format, including statistics and data insights. | N/A | > [!NOTE] Agentic tool target type > All global tools have an Unstructured target type and a Target of `target`. To learn more about a tool, you can access the source code in the [publicagent-tool-templatesrepository](https://github.com/datarobot-oss/agent-tool-templates). Each tool is tagged with the `global.model.source` tag, linking to the directory containing the source files for that tool. This allows you to explore its contents to learn more about the model, review its input and output schema, or use the code as a template for building a customized tool. To find the repository link: 1. Apply theGlobalfilter and look for a[Tool]in the list. 2. Open a version and in the version, scroll down to theKey valuessection. 3. Open theTagspanel and locate theglobal.model.sourcetag. 4. Hover over the tag value to view the full URL, or, click the link to open the repository to the directory for that tool. --- # Implement tracing URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html > Learn how to add OpenTelemetry tracing and custom span instrumentation to your agent tools for monitoring, debugging, and observability. OpenTelemetry (OTel) provides comprehensive observability for your agents, allowing you to monitor, trace, and debug agent execution in real-time. This guide explains how to add custom tracing to your agent tools to capture detailed execution information. OpenTelemetry tracing helps: - Monitor agent performance and execution flow. - Debug issues by tracking detailed execution traces. - Understand tool execution patterns and timing. - View custom attributes and metadata from your tools. The agent templates already include OpenTelemetry instrumentation for frameworks like CrewAI, LangGraph, and Llama-Index. This instrumentation automatically captures spans for: - Agent execution - Tool invocations - LLM API calls - HTTP requests You can enhance this default tracing by adding custom spans and attributes in your tools. ## Add custom tracing to tools Add custom OpenTelemetry tracing to your tools to capture additional information about tool execution. This allows you to track custom attributes, intermediate outputs, and execution details that are specific to your use case. The basic pattern for adding custom tracing to a tool is: ``` from opentelemetry import trace tracer = trace.get_tracer(__name__) # Within your tool's execution with tracer.start_as_current_span("my_custom_span_name"): current_span = trace.get_current_span() current_span.set_attribute("tool_name", "my_tool_name") current_span.set_attribute("gen_ai.prompt", "input passed to this step") current_span.set_attribute("datarobot.moderation.cost", 0.0) # Your tool logic here result = perform_tool_action() current_span.set_attribute("gen_ai.completion", str(result)) # Optionally add more attributes about the result current_span.set_attribute("result.status", "success") current_span.set_attribute("result.size", len(result)) return result ``` ### Tool examples See the code examples below to learn how to add custom OpenTelemetry tracing to agentic tools: **CrewAI:** ``` import requests from crewai.tools import BaseTool from opentelemetry import trace tracer = trace.get_tracer(__name__) class WeatherTool(BaseTool): name: str = "weather_tool" description: str = ( "Fetches the current weather for a specified city. " "Requires an API key from OpenWeatherMap." ) def _run(self, city: str) -> str: with tracer.start_as_current_span("weather_tool_fetch"): current_span = trace.get_current_span() current_span.set_attribute("tool_name", "weather_tool") current_span.set_attribute("gen_ai.prompt", f"weather lookup for {city}") current_span.set_attribute("datarobot.moderation.cost", 0.0) # Set custom attributes current_span.set_attribute("weather.city", city) current_span.set_attribute("weather.api", "openweathermap") api_key = "YOUR_API_KEY" # Replace with your API key base_url = "http://api.openweathermap.org/data/2.5/weather" params = {"q": city, "appid": api_key, "units": "metric"} try: response = requests.get(base_url, params=params, timeout=10) response.raise_for_status() data = response.json() weather = data['weather'][0] main = data['main'] # Add result attributes current_span.set_attribute("weather.temperature", main['temp']) current_span.set_attribute("weather.condition", weather['main']) result = ( f"Current weather in {data['name']}, {data['sys']['country']}:\n" f"Temperature: {main['temp']}°C (feels like {main['feels_like']}°C)\n" f"Condition: {weather['main']} - {weather['description']}\n" f"Humidity: {main['humidity']}%\n" f"Pressure: {main['pressure']} hPa" ) current_span.set_attribute("gen_ai.completion", result) return result except requests.exceptions.RequestException as e: current_span.set_attribute("weather.error", str(e)) err = f"Error fetching weather data: {str(e)}" current_span.set_attribute("gen_ai.completion", err) return err ``` **LangGraph:** ``` import requests from langchain.tools import BaseTool from opentelemetry import trace tracer = trace.get_tracer(__name__) class WeatherTool(BaseTool): name: str = "weather_tool" description: str = ( "Fetches the current weather for a specified city. " "Requires an API key from OpenWeatherMap." ) def _run(self, city: str) -> str: with tracer.start_as_current_span("weather_tool_fetch"): current_span = trace.get_current_span() current_span.set_attribute("tool_name", "weather_tool") current_span.set_attribute("gen_ai.prompt", f"weather lookup for {city}") current_span.set_attribute("datarobot.moderation.cost", 0.0) # Set custom attributes current_span.set_attribute("weather.city", city) api_key = "YOUR_API_KEY" # Replace with your API key base_url = "http://api.openweathermap.org/data/2.5/weather" params = {"q": city, "appid": api_key, "units": "metric"} try: response = requests.get(base_url, params=params, timeout=10) response.raise_for_status() data = response.json() # Add result attributes current_span.set_attribute("weather.temperature", data['main']['temp']) result = f"Temperature in {city}: {data['main']['temp']}°C" current_span.set_attribute("gen_ai.completion", result) return result except requests.exceptions.RequestException as e: current_span.set_attribute("weather.error", str(e)) err = f"Error: {str(e)}" current_span.set_attribute("gen_ai.completion", err) return err ``` **Llama-Index:** ``` import requests from llama_index.core.tools import FunctionTool from opentelemetry import trace tracer = trace.get_tracer(__name__) def _weather_run(city: str) -> str: """Fetches the current weather for a specified city. Requires an API key from OpenWeatherMap.""" with tracer.start_as_current_span("weather_tool_fetch"): current_span = trace.get_current_span() current_span.set_attribute("tool_name", "weather_tool") current_span.set_attribute("gen_ai.prompt", f"weather lookup for {city}") current_span.set_attribute("datarobot.moderation.cost", 0.0) # Set custom attributes current_span.set_attribute("weather.city", city) api_key = "YOUR_API_KEY" # Replace with your API key base_url = "http://api.openweathermap.org/data/2.5/weather" params = {"q": city, "appid": api_key, "units": "metric"} try: response = requests.get(base_url, params=params, timeout=10) response.raise_for_status() data = response.json() # Add result attributes current_span.set_attribute("weather.temperature", data['main']['temp']) result = f"Temperature in {city}: {data['main']['temp']}°C" current_span.set_attribute("gen_ai.completion", result) return result except requests.exceptions.RequestException as e: current_span.set_attribute("weather.error", str(e)) err = f"Error: {str(e)}" current_span.set_attribute("gen_ai.completion", err) return err def WeatherTool() -> FunctionTool: return FunctionTool.from_defaults( fn=_weather_run, name="weather_tool", description=( "Fetches the current weather for a specified city. " "Requires an API key from OpenWeatherMap." ), ) ``` ## Create nested spans Create nested spans to represent complex tool execution with multiple steps: ``` from opentelemetry import trace tracer = trace.get_tracer(__name__) def complex_tool_workflow(input_data): with tracer.start_as_current_span("complex_tool_main"): current_span = trace.get_current_span() current_span.set_attribute("input.size", len(input_data)) # First step in the workflow with tracer.start_as_current_span("data_processing"): processed_data = process_data(input_data) trace.get_current_span().set_attribute("processed_items", len(processed_data)) # Second step in the workflow with tracer.start_as_current_span("data_validation"): validated_data = validate_data(processed_data) trace.get_current_span().set_attribute("validated_items", len(validated_data)) # Third step in the workflow with tracer.start_as_current_span("result_generation"): result = generate_result(validated_data) current_span.set_attribute("result.size", len(result)) return result ``` ### Add events to spans Add events to your spans to mark important moments in tool execution: ``` from opentelemetry import trace from datetime import datetime tracer = trace.get_tracer(__name__) def tool_with_events(): with tracer.start_as_current_span("tool_execution"): current_span = trace.get_current_span() # Add an event for when processing starts current_span.add_event( "Processing started", {"timestamp": datetime.utcnow().isoformat()} ) # Your tool logic intermediate_result = perform_action() # Add an event for mid-execution current_span.add_event( "Intermediate result ready", {"result_count": len(intermediate_result)} ) # More processing final_result = complete_processing(intermediate_result) # Add final event current_span.add_event( "Processing completed", {"output_size": len(final_result)} ) return final_result ``` ## Add custom tracing to agent You can set up a custom trace to capture how your agent starts up, including configurations and environment details. Follow the steps below to surface runtime parameters (like environment variables) on a span: 1. Update your.envfile to contain the following environment variable so it's available during local development and when you package the model: EXAMPLE_ENV_VAR=my_example_value 2. Add the parameter toagent/model-metadata.yamlso DataRobot can inject it when the agent runs: runtimeParameterDefinitions:-fieldName:EXAMPLE_ENV_VARtype:stringdefaultValue:SET_VIA_PULUMI_OR_MANUALLY 3. Add the parameter to your Config class inagent/agent/config.py(for example,example_env_var: str = "") so the value is available asconfig.example_env_varin your agent code. 4. Updateinfra/infra/llm.pyto forward the runtime parameter into the custom model environment: custom_model_runtime_parameters=[# ...existing parameters...datarobot.CustomModelRuntimeParameterValueArgs(key="EXAMPLE_ENV_VAR",type="string",value=os.environ.get("EXAMPLE_ENV_VAR"),),] 5. Wrap the configuration loading code in a span and attach the values withset_attributeandadd_event. The property name depends on your template (for example,agent_plannerin CrewAI templates): @propertydefagent_planner(self)->Any:withtracer.start_as_current_span("config_variables"):current_span=trace.get_current_span()current_span.set_attribute("config.example_env_var",config.example_env_var)current_span.add_event("config attribute set on span")# ...agent code continued... When you run the agent locally during development, the trace visualizer shows a `config_variables` span with attributes such as `config.example_env_var=my_example_value`. This makes it easy to confirm that runtime parameters and other environment values were loaded correctly. After you deploy your agent as a custom application, users with [Owner or Editor](https://docs.datarobot.com/en/docs/wb-apps/custom-apps/manage-custom-app.html#share-applications) permissions on the application can review these spans on the Tracing tab in the DataRobot UI. See [Tracing for custom applications](https://docs.datarobot.com/en/docs/wb-apps/custom-apps/monitor-app.html#tracing). If you deploy the agent as a Console deployment, review traces on the dedicated Tracing tab; see [Tracing](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-tracing.html). ## Map spans and attributes to the tracing table On a deployment, the tracing table displays several columns derived from OpenTelemetry span attributes. The following naming conventions apply across the trace: | Tracing table column | Attribute | How it is derived | | --- | --- | --- | | Cost | datarobot.moderation.cost | Summed across all spans in the trace. | | Prompt | gen_ai.prompt | If multiple spans set this attribute, the first value in trace order is used. | | Completion | gen_ai.completion | If multiple spans set this attribute, the last value in trace order is used. | | Tools | tool_name | Every distinct tool_name found on any span in the trace is listed. | ### Surface tool names in the tracing table The Tools column is populated from the span attribute `tool_name`. Some frameworks set it on tool spans automatically; others do not. If your traces show tool execution in the span timeline, but Tools is empty, create a span around the tool body (or use the active span) and set `tool_name` explicitly. **Inside awithspan:** ``` from opentelemetry import trace tracer = trace.get_tracer(__name__) def my_tool_impl(query: str) -> str: with tracer.start_as_current_span("my_tool"): span = trace.get_current_span() # Attributes that map to deployment Tracing table columns (see above) span.set_attribute("tool_name", "my_tool_name") span.set_attribute("gen_ai.prompt", query) span.set_attribute("datarobot.moderation.cost", 0.0) # numeric; summed per trace # ... tool logic ... result = "result" span.set_attribute("gen_ai.completion", result) return result ``` **Current span only:** If a span is already active (for example, from upstream instrumentation), you can set the attribute on that span: ``` from opentelemetry import trace span = trace.get_current_span() span.set_attribute("tool_name", "my_tool_name") span.set_attribute("gen_ai.prompt", "user input or request text") span.set_attribute("gen_ai.completion", "model or tool output text") span.set_attribute("datarobot.moderation.cost", 0.0) ``` For LangGraph and similar frameworks, tool calls are sometimes wired through callbacks in a way that does not add `tool_name` to spans; manual instrumentation can allow the name to appear in the Tools column. ## Best practices Use descriptive span names: - Use clear, descriptive names for spans (e.g., "weather_fetch" rather than "span1" ). - Include the tool name in the span name when relevant. Set meaningful attributes: - Add attributes that provide context about the execution. - Use consistent attribute naming conventions (e.g., tool.input , tool.output , tool.error ). - Include relevant metadata like sizes, counts, or statuses. - To populate Cost , Prompt , Completion , and Tools in the deployment Tracing table, set datarobot.moderation.cost , gen_ai.prompt , gen_ai.completion , and tool_name on the relevant spans. Keep spans focused: - Create spans for significant operations, not every line of code. - Each span should represent a meaningful unit of work. - Use nested spans to represent sub-operations. --- # Troubleshooting URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-troubleshooting.html > Troubleshoot common issues when working with DataRobot Agent Templates. This guide helps you diagnose and resolve common issues when working with DataRobot Agent Templates. ## Prerequisites and setup issues Common issues related to system requirements and initial setup for DataRobot Agent Templates. ### Windows compatibility Issue: Symlinks appear as plain text files on Windows, or commands fail with missing-module errors after clone. Symptoms: Files such as `infra/infra/llm.py` or `fastapi_server/core` contain a path string instead of linking to another file. Solution: 1. Complete Windows prerequisites before you clone the repository. 2. If you already cloned without symlink support, delete the local repository and clone again after you configure Git and Developer Mode (or use an administrator terminal). 3. Verify Git symlink settings: ``` git config --global --get core.symlinks ``` The command should print `true`. ### Missing prerequisites Issue: Required tools are not installed. Solution: Install missing tools following the [prerequisites guide](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html). ### Build tools missing Issue: Build tools are not available on your system. Solution: Install Xcode Command Line Tools (macOS), build-essential (Linux), or [Visual Studio Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools/) with the Desktop development with C++ workload (Windows). ## Environment configuration issues Problems with environment variables and DataRobot endpoint configuration. ### Missing environment variables Issue: Required environment variables are not set. Solution: Create `.env` file with `DATAROBOT_API_TOKEN` and `DATAROBOT_ENDPOINT`. ### Invalid endpoint configuration Issue: Incorrect DataRobot endpoint configured. Solution: Use correct endpoint for your region (cloud) or contact support (self-managed). ## Authentication issues Issues related to API authentication and authorization context setup. ### API token authentication failed Issue: API token is invalid or lacks proper permissions. Solution: Verify API token is valid and has proper permissions. ### Authorization context not set Issue: Authorization context is not initialized in your agent. Solution: Ensure `initialize_authorization_context()` is called in your agent. ## Deployment issues Problems encountered during the deployment process and infrastructure management. ### Pulumi login required Issue: Pulumi authentication is not configured. Solution: Run `pulumi login --local` or `pulumi login`. ### Deployment ID not found Issue: Cannot locate deployment ID after deployment. Solution: Check terminal output or DataRobot UI → Console → Deployed workloads. ## CLI and testing issues Issues with command-line interface usage and local testing of agents. ### CLI command not found Issue: CLI commands are not available. Solution: Ensure the DataRobot CLI is installed and on your PATH. See [Getting started with the DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html) for installation, then run `dr start` to select a framework. For CLI-specific issues, see [CLI troubleshooting](https://docs.datarobot.com/en/docs/agentic-ai/cli/troubleshooting.html). ### Local testing fails Issue: Local testing encounters errors when trying to run your agent during development. Solution: Use the CLI command to test your agent locally. This allows you to run and debug your agent code without deploying it to DataRobot. The `execute` command requires a development server to be running. You can either start it manually with `task agent:dev` (it runs continuously, so use a separate terminal), or use `START_DEV=1` to automatically start and stop it: - Test with a basic text query: task agent:cli START_DEV=1 -- execute --user_prompt "Hello" (or start task agent:dev first, then run task agent:cli -- execute --user_prompt "Hello" ) - Test with JSON input: task agent:cli START_DEV=1 -- execute --user_prompt '{"topic": "Artificial Intelligence"}' - Test with a completion JSON file: task agent:cli START_DEV=1 -- execute --completion_json example-completion.json - Enable verbose logging: Add "verbose": true to the extra_body field in your completion JSON Common issues include missing environment variables ( `DATAROBOT_API_TOKEN`, `DATAROBOT_ENDPOINT`), import issues in `myagent.py` (see [Import issues](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-troubleshooting.html#import-issues-in-myagent-py)), and missing dependencies in `pyproject.toml`. ## Infrastructure issues Problems with containerization, build processes, and infrastructure state management. ### Docker build fails Issue: Docker container build encounters errors. Solution: Test locally and check `pyproject.toml` dependencies. ### Pulumi state issues Issue: Pulumi state is out of sync. Solution: Run `task infra:refresh` to sync state. ### Import issues in myagent.py Issue: Imports in `myagent.py` to files in the same folder cause silent failures in DRUM. Solution: Use relative imports instead of package imports. ## LLM gateway issues Issues specific to LLM gateway connectivity, model access, and configuration. ### Model access error Issue: LLM gateway cannot connect to the model or fails with "no model access" error. Solution: Ensure you have access to the model. If you specified a model you don't have access to (or a retired model), you can connect to the gateway, but then the action fails with a "no model access" error. ### LLM gateway configuration Issue: LLM gateway is not properly configured. Solution: Ensure that your organization has access to the [LLM gateway](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html) and that the `ENABLE_LLM_GATEWAY_INFERENCE` runtime parameter is provided in the `model-metadata.yaml` file and set to `true`. ### Non-gateway model deployment Issue: Non-gateway model deployment fails. Solution: If you are using a non-gateway model, run `task deploy` once even if it fails to get your LLM deployment to run. ## Accessing request headers When your agent is deployed, you may need to access HTTP request headers for authentication, tracking, or custom metadata. DataRobot makes headers available to your agent code through the `chat()` function's `**kwargs` parameter. ### Extracting X-Untrusted-* headers Headers with the `X-Untrusted-*` prefix are passed through from the original request and are available in the `kwargs` dictionary: ``` import asyncio from concurrent.futures import ThreadPoolExecutor from typing import Any, Iterator, Union from openai.types.chat import CompletionCreateParams from openai.types.chat.completion_create_params import ( CompletionCreateParamsNonStreaming, CompletionCreateParamsStreaming, ) def chat( completion_create_params: CompletionCreateParams | CompletionCreateParamsNonStreaming | CompletionCreateParamsStreaming, load_model_result: tuple[ThreadPoolExecutor, asyncio.AbstractEventLoop], **kwargs: Any, ) -> Union[CustomModelChatResponse, Iterator[CustomModelStreamingResponse]]: # Extract all headers from kwargs headers = kwargs.get("headers", {}) # Access specific X-Untrusted-* headers authorization_header = headers.get("X-Untrusted-Authorization") custom_header = headers.get("X-Untrusted-Custom-Metadata") # Use headers in your agent logic if authorization_header: # Pass to downstream services, tools, etc. print(f"Authorization header: {authorization_header}") ``` ### Common use cases Passing authentication to external services: ``` headers = kwargs.get("headers", {}) auth_token = headers.get("X-Untrusted-Authorization") # Use the token to authenticate with external APIs tool_client = MyTool(auth_token=auth_token) ``` Tracking request metadata: ``` headers = kwargs.get("headers", {}) request_id = headers.get("X-Untrusted-Request-ID") user_id = headers.get("X-Untrusted-User-ID") # Log metadata for debugging or analytics logging.info(f"Processing request {request_id} for user {user_id}") ``` Conditional agent behavior: ``` headers = kwargs.get("headers", {}) region = headers.get("X-Untrusted-Region") # Adjust agent behavior based on request context if region == "EU": agent = MyAgent(enable_gdpr_mode=True) ``` ### Important considerations - Only headers with the X-Untrusted-* prefix are passed through to your agent code. - Headers are case-sensitive when accessing them from the dictionary. - Always provide default values or check for None when accessing headers that may not be present. - Headers are available in both streaming and non-streaming chat responses. ## Debugging tips Useful techniques and commands for troubleshooting and debugging agent issues. ### Enable verbose logging ``` agent = MyAgent(verbose=True) ``` ### Test authentication ``` from datarobot_genai.core.cli import AgentEnvironment env = AgentEnvironment() ``` ### Check environment variables ``` echo $DATAROBOT_API_TOKEN echo $DATAROBOT_ENDPOINT ``` ### Log request headers for debugging ``` import asyncio import logging from concurrent.futures import ThreadPoolExecutor from typing import Any def chat(completion_create_params, load_model_result: tuple[ThreadPoolExecutor, asyncio.AbstractEventLoop], **kwargs: Any): headers = kwargs.get("headers", {}) # Log all headers to inspect what's available if headers: logging.warning(f"All headers: {dict(headers)}") # Continue with agent logic... ``` ## Getting help For additional assistance: - Check the documentation for your chosen agentic framework. - Contact DataRobot for support. - Open an issue on the GitHub repository . - Framework Documentation: --- # Build URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/index.html > Build agentic workflows with DataRobot using existing templates or by modifying those templates to suit your specific use case. > [!NOTE] Premium > DataRobot's Agentic AI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. The [DataRobot Agentic Starter template repository](https://github.com/datarobot-community/datarobot-agent-application) provides ready-to-use templates for building and deploying AI agents with multi-agent frameworks. These templates streamline the process of setting up your own agents with minimal configuration requirements and support both local development and testing, as well as deployment to production environments within DataRobot. To start building and deploying AI agents from DataRobot-provided templates, review the [Get started](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html) guide, which covers installing the required components and using the [DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/index.html) ( `dr start`, `dr task run`) for setup and deployment—see [Getting started with the DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html) if you need to install or configure the CLI first. | Topic | Description | | --- | --- | | Get started | Install required components, then build, deploy, and test agentic workflows using DataRobot's pre-built templates. | | DataRobot agentic skills | Install DataRobot-supported agent skills from marketplaces or the universal installer for Claude Code, Cursor, Codex, Gemini CLI, and other coding agents. | | Agent components | Learn about the components required to create an agent using DataRobot's agent framework. | | Agent authentication | Learn how to implement authentication in your DataRobot agentic application, covering API tokens, authorization context, OAuth 2.0, and security best practices. | | Customize agents | Customize agent code, test locally, and deploy agentic application for production use. | | Add Python packages | Add required Python packages to agentic application using execution environment or custom model requirements. | | Configure LLM providers in code | Configure different LLM providers for your agentic application including DataRobot gateway, external APIs, and custom deployments. | | Configure LLM providers with metadata | Configure LLM providers using environment variables and Pulumi for infrastructure-level configuration without modifying agent code. | | Configure LLM provider fallback | Configure primary and fallback LLM providers for automatic failover when the primary provider is unavailable. | | Add tools to agents | Add local tools, predefined tools, and DataRobot global tools to your agentic application, including detailed integration patterns. | | Deploy agentic tools | Deploy global agentic tools from the DataRobot Registry to handle tasks critical to the agent application. | | Agentic memory service | Learn when to use DataRobot's built-in chat history and REST integration versus the mem0-compatible memory API. | | Implement tracing | Add custom tracing to your agent tools for monitoring, debugging, and observability. | | Access request headers | Learn how to access HTTP request headers in your deployed agents for authentication, tracking, and custom metadata. | | Debug agents in PyCharm | Use PyCharm's Run configuration in debug mode to debug agent code locally. | | Debug agents in VS Code | Use VS Code's Run and Debug configuration to start dev.py and step through agent code locally. | | Troubleshooting | Diagnose and resolve common issues when working with DataRobot agentic application. | --- # Chat with agents URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-chatting.html > Describes working with single-agent chats in the playground. Chatting with an agent primarily involves testing the agent's output against expectations—it allows human evaluation of responses. In the agentic playground chat you provide prompts and assess if the responses align with desired outcomes. Did the playground metrics and tools produce the expected output? Agentic chatting does not include [context-aware](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html#context-aware-chatting) capabilities, and so while a "chat" is a collection of chat prompts, each response is individual and not dependent on previous responses. ## Single- vs multi-agent chats A single agent chat is accessed by clicking the agent tile (not the checkbox) in the Agentic playground > Workflows tab. To return to the multi-agent view, click the Agentic playground tile (or the back arrow above). A multi-agent chat: - Is accessed from the Agentic playground > Workflows tab (1). - Uses check boxes to select agents (2). Selecting only one of the agents does not make it a single-agent chat. - Displays Agentic workflow comparison in the top of the chat window (3). There is no functional difference between single- and multi-agent chat responses; however, [tracing](https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-tracing.html) is available only in single-agent chats. ## Entering and sending prompts Enter a prompt to start a chat for either single or multi-agent chatting from the Agentic playground > Workflows tab. Click Send to request a response. ## Agent actions Most actions, with the exception of [tracing](https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-chatting.html#single-agent-tracing), are shared for both single- and multi-agent chats, as described in the sections and tables below. ### Actions menu Click the actions menu either under the Workflows tab or within the chat window to act on the agent. **Workflows tab:** [https://docs.datarobot.com/en/docs/images/agent-actions.png](https://docs.datarobot.com/en/docs/images/agent-actions.png) **Chat window:** [https://docs.datarobot.com/en/docs/images/agent-chat-window-actions.png](https://docs.datarobot.com/en/docs/images/agent-chat-window-actions.png) | Action | Description | Location | | --- | --- | --- | | Edit agentic workflow name | Edit the display name of the agentic workflow. | Workflows tab action menu. | | Open in codespace | Opens the agent in a codespace, where you can directly edit the existing files, upload new files, or use any of the codespace functionality. | Workflows tab action menu Button in the top right of a single-agent chat windowChat window action menu | | Register agentic workflow | After experimenting in the agentic playground to build a production-ready agentic workflow, register the custom agentic workflow in the Registry workshop, in preparation for deployment to Console. | Workflows tab action menu Button in the top right of a single-agent chat windowChat window action menu | | Remove from playground | Delete the agentic workflow from the playground. Removing an agentic workflow deletes it from the Use Case but does not delete it from Workshop. All playground-related information stored with the workflow, including metrics and chats, is also removed. | Workflows tab action menu Chat window action menu | You can remove both the prompt and all responses from the chat history from within the Agentic workflow comparison window: ### Aggregation tools The following table lists the shared [aggregation tools](https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-evaluation-tools.html#add-aggregated-metrics) and tabs available: | | Element | Description | | --- | --- | --- | | (1) | Agentic aggregated metrics | View aggregated metrics and scores calculated not using an evaluation dataset. These metrics originate in the Registry workshop. | | (2) | Evaluation aggregated metrics | View aggregated metrics and scores calculated using an evaluation dataset to provide a baseline for comparison. These metrics originate in the playground. | | (3) | Configure aggregation | Combine metrics across many prompts and/or responses to get a more comprehensive approach to evaluation. | ## Single-agent tracing There are two types of [agentic tracing](https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-tracing.html) available for single-agent chats. The output of the trace depends on the execution location. **Prompt header:** From the actions menu, in the prompt header, select Open tracing table. [https://docs.datarobot.com/en/docs/images/trace-from-chat-window.png](https://docs.datarobot.com/en/docs/images/trace-from-chat-window.png) View a log that traces all components used in LLM response generation. [https://docs.datarobot.com/en/docs/images/tracing-log.png](https://docs.datarobot.com/en/docs/images/tracing-log.png) **Response header:** Click Review tracing in the response header: [https://docs.datarobot.com/en/docs/images/trace-from-response-window.png](https://docs.datarobot.com/en/docs/images/trace-from-response-window.png) The tracing details panel opens. [https://docs.datarobot.com/en/docs/images/tracing-chart.png](https://docs.datarobot.com/en/docs/images/tracing-chart.png) | Location | Option | Description | | --- | --- | --- | | Prompt header actions menu | Open tracing | Opens the tracing table log, which shows all components and prompting activity used in generating agentic responses. | | Response header | Review tracing | Opens the tracing details panel, illustrating the path a single request takes through the agentic workflow. | ## Response feedback Use the response feedback "thumbs" to rate the prompt answer. Responses are recorded in the User feedback column on the [Tracing](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-eval-metrics.html#tracing) tab. The response, as part of the exported feedback sent to the AI Catalog, can be used, for example, to train a predictive model. --- # Evaluate metrics URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-evaluation-tools.html > Configure evaluation metrics, add evaluation datasets, and review tracing for agentic workflows in a playground. The playground's agentic evaluation tools include evaluation metrics and datasets, aggregated metrics, compliance tests, and tracing. The agentic evaluation metric tools include: | Agentic workflow evaluation tool | Description | | --- | --- | | Evaluation metrics | Report an array of performance, safety, and operational metrics for prompts and responses in the playground and define moderation criteria and actions for any configured metrics. | | Evaluation datasets | Upload or generate the evaluation datasets used to evaluate an agentic workflow through evaluation dataset metrics and aggregated metrics. | | Aggregated metrics | Combine evaluation metrics across many prompts and responses to evaluate an agentic workflow at a high level, as only so much can be learned from evaluating a single prompt or response. | | Tracing table | Trace the execution of agentic workflows through a log of all components and prompting activity used in generating responses in the playground. | ## Configure evaluation metrics With evaluation metrics, you can configure performance and operational metrics for agents. You can view these metrics in comparison chats and in chats with individual agents. Playground metrics require reference information provided through an evaluation dataset and are useful for assessing if an agentic workflow is operating as expected. Because they require an evaluation dataset, they are only available in the playground. Agentic workflow metrics don't require reference data, so they are available in production and configured in the [Workshop](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-configure-evaluation-moderation.html). | Playground metrics | Agentic workflow metrics | | --- | --- | | Are configured in a playground. | Are configured in Workshop. | | Require reference data provided as an evaluation dataset. | Don't require reference data. | | Can't be computed in production. | Can be computed in production. | | Can only be applied to the top level agentic workflow. | Can be applied to the top-level agent and sub-agents and sub-tools of the workflow (if they are separate custom models). | > [!NOTE] Agent moderation > Agentic workflow-specific metrics don't support setting moderation criteria. ### View agentic workflow metrics To enable agentic workflow metrics for a workflow, configure [evaluation and moderation in Workshop](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-configure-evaluation-moderation.html). Click the Evaluation tile to see, on the Agentic workflow metrics tab, the configured metrics that are enabled for the agentic workflow. ### Configure playground metrics To enable playground metrics for your workflows, add one or more evaluation metrics to the agentic playground. In addition, you must provide reference data using [evaluation datasets](https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-evaluation-tools.html#add-evaluation-datasets). 1. To select and configure playground evaluation metrics for an agentic playground, do either of the following: Without connected agentsWith connected agentsIf you haven't connected any agents to the playground, in theEvaluate with metricstile, clickConfigure metricsto configure metrics before adding an agent:If you've added one or more agents to the playground, on the side navigation bar click theEvaluationtile: 2. On theEvaluation and moderationpage, click thePlayground metricstab, then clickConfigure metrics. 3. On theConfigure evaluation and moderationpage, click anOperational metric: Playground metricDescriptionAgent latencyTotal latency of running the agent workflow for a given request. Includes time for completions, tool calls, and metrics calculated by the moderations library. Always available when the agent is configured for OTel.Agent total tokensFor agents using the LLM gateway, total tokens are reported in OTel; current templates already do this. For agents using a deployed LLM, that LLM must have token count metrics enabled in themoderations configuration.Agent costAvailable only for calls to deployed LLMs. The deployed LLM must have acost metric configuredin the moderations configuration. Operational metrics and Agentic workflow comparisonAgent Cost, Agent Latency, and Agent Total Tokens use data from theOTel collector, which is configured by default in theagent templates. These metrics aggregate the reported OTel data and require tracing enabled to associate the correct spans with that data. Tracing is not available on the Agentic workflow comparison screen, so these operational evaluation metrics are only available when assessing per-agent responses, not when comparing agentic workflows. Then, optionally modify the metricNameand clickAdd. TheApply tosetting is preconfigured for these metrics. 4. On theConfigure evaluation and moderationpage, click aQuality metric: Playground metricDescriptionAgent Goal Accuracy with ReferenceUse a known benchmark to evaluate agentic workflow performance in achieving specified objectives. Requires an evaluation dataset containing an agent goal column.Tool Call AccuracyMeasure agentic workflow performance when identifying and calling the required tools for a given task. Requires an evaluation dataset containing an expected tool calls column. Evaluation dataset exampleThe example evaluation dataset below includes an expected tool calls column (toolCalls) required by theTool Call Accuracymetric and an agent goal column (agentGoal) required by theAgent Goal Accuracy with Referencemetric:example_evaluation_dataset.csvid,promptText,expectedResponse,toolCalls,agentGoal 1,What is the weather like in New York today?,It is 24 C and sunny in New York today.,"[{""name"":""weather_check"",""args"":{""location"":""New York""}},{""name"":""temperature_conversion"",""args"":{""temperature_fahrenheit"":75}}]",A concise answer to a question about weather. 2,How many planets are in the solar system?,Our solar system has 8 planets.,[],A concise answer to a question about the solar system.In addition, the DataRobot Python client providesutility classes for constructing the expected tool calls column. Then, configure the following settings, depending on the metric you selected: Playground metricDescriptionAgent Goal Accuracy with Reference(Optional) Enter a metricName.Select a playground or deployed LLM to evaluate goal accuracy.Tool Call Accuracy(Optional) Enter a metricName. After configuring the settings, clickAdd. TheApply tosetting is preconfigured for these metrics. 5. Select and configure another metric, or clickSave configuration. Edit configuration summaryAfter you add one or more metrics to the playground configuration, you can edit or delete those metrics. ### Copy metric configurations To copy an evaluation metrics configuration to or from an agentic playground: 1. In the upper-right corner of theEvaluation and moderationpage, next toConfigure metrics, click, and then clickCopy configuration. 2. In theCopy evaluation and moderation configurationmodal, select one of the following options: From an existing playgroundTo an existing playgroundTo a new playgroundIf you selectFrom an existing playground, choose toAdd to existing configurationorReplace existing configurationand then select a playground toCopy from.If you selectTo an existing playground, choose toAdd to existing configurationorReplace existing configurationand then select a playground toCopy to.If you selectTo a new playground, enter aNew playground name. 3. Select if you want toInclude evaluation datasets, and then clickCopy configuration. > [!NOTE] Duplicate evaluation metrics > Selecting Add to existing configuration can result in duplicate metrics. ### Add evaluation datasets To enable playground evaluation metrics and aggregated metrics, you must add one or more evaluation datasets to the playground to serve as reference data. The dataset must be a CSV file, in the Data Registry, and have at least one text or categorical column. 1. To add evaluation datasets in an agentic playground, do either of the following: 2. On theEvaluation and moderationpage, click theEvaluation datasetstab to view any existing datasets, or, clickAdd evaluation datasetfrom any tab, and select one of the following methods: MethodDescriptionSelect an existing datasetClick a dataset in theData Registrytable.Upload a new datasetClickUploadto register and select a new dataset from your local filesystem.ClickUpload from URL, then, enter theURLfor a hosted dataset and clickAdd. After you select a dataset, in theEvaluation dataset configurationright-hand sidebar, define the following columns: ColumnDescriptionPrompt column nameThe name of the reference dataset column containing the user prompt.Response (target) column nameThe name of the reference dataset column containing an expected agent response.Reference goals column nameThe name of the reference dataset column containing a description of the expected (goal) output of the agent. This data is used for theConfigure Agent Goal Accuracy with Referencemetric.Reference tools column nameThe name of the reference dataset column containing the expected agentic tool calls. This data is used for theConfigure Tool Call Accuracymetric. Then, clickAdd evaluation dataset. Evaluation dataset exampleThe example evaluation dataset below includes an expected tool calls column (toolCalls) required by theTool Call Accuracymetric and an agent goal column (agentGoal) required by theAgent Goal Accuracy with Referencemetric:example_evaluation_dataset.csvid,promptText,expectedResponse,toolCalls,agentGoal 1,What is the weather like in New York today?,It is 24 C and sunny in New York today.,"[{""name"":""weather_check"",""args"":{""location"":""New York""}},{""name"":""temperature_conversion"",""args"":{""temperature_fahrenheit"":75}}]",A concise answer to a question about weather. 2,How many planets are in the solar system?,Our solar system has 8 planets.,[],A concise answer to a question about the solar system.In addition, the DataRobot Python client providesutility classes for constructing the expected tool calls column. 3. After you add an evaluation dataset, it appears on theEvaluation datasetstab of theEvaluation and moderationpage, where you can: ## Add aggregated metrics When a playground includes more than one metric, you can begin creating aggregated metrics. Aggregation is the act of combining metrics across many prompts and/or responses, which helps to evaluate agents at a high level (only so much can be learned from evaluating a single prompt/response). Aggregation provides a more comprehensive approach to evaluation. Aggregation either averages the raw scores, counts the boolean values, or surfaces the number of categories in a multiclass model. DataRobot does this by generating the metrics for each individual prompt/response and then aggregating using one of the methods listed, based on the metric. To configure aggregated metrics for an agentic playground: 1. In the agentic playground, clickConfigure aggregationbelow the prompt input (from theWorkflowstab, or in an individual agentChatstab): Workflows tabAgent Chats tabFrom theWorkflowstab, each agentic workflow selected is included in the aggregation job.From theChatstab for a single agent, only the current agentic workflow is included in the aggregation job. Aggregation job run limitOnly one aggregated metric job can run at a time. If an aggregation job is currently running, theConfigure aggregationbutton is disabled and the "Aggregation job in progress; try again when it completes" tooltip appears. 2. On theGenerate aggregated metricspanel, select metrics to include in aggregation and configure theAggregate bysettings. In the right-hand panel, enter a newChat name, select anEvaluation dataset(to generate prompts in the new chat), and select theWorkflowsfor which the metrics should be generated. These fields are pre-populated based on the current playground: Playground vs agentic workflow metricsIn the example below,Agent Goal Accuracy with ReferenceandTool Call Accuracyare playground metrics, whileROUGE-1andResponse Tokens are agentic workflow metrics (fromWorkshop). After you complete theMetrics selectionandConfigurationsections, clickGenerate metrics. This results in a chat, identified as aMetricchat, containing all associated prompts and responses: Aggregated metrics are run against an evaluation dataset, not individual prompts in a standard chat. Therefore, you can only view aggregated metrics in the generatedaggregated metrics chat, added to the agent'sAll Chatslist (on the agent's individualChatstab). Aggregation metric calculation for multiple agentsIf many agents are included in the metric aggregation request, aggregated metrics are computed sequentially, agent-by-agent. 3. Once an aggregated chat is generated, you can explore the resulting aggregated metrics, scores, and related assets on theAgentic aggregated metricstab andEvaluation aggregated metricstab. These tabs are available when comparing agentic chats, and when viewing a single-agent chat. You can filter byAggregation method,Evaluation dataset, andMetric: Agentic aggregated metricsEvaluation aggregated metrics --- # Connect to a playground URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-playground.html > The agentic playground, an asset of a Use Case, is the space for connecting and interacting with agentic workflows. An agentic playground, a type of Use Case asset, is the space for connecting and interacting with agentic workflows. Within the playground you compare agentic workflow responses to determine which agent to use in production for solving a business problem. Multiple playgrounds can exist in one Use Case and multiple agentic workflows can exist within a single playground. The suggested, simplified workflow for working with playgrounds is as follows: 1. Add an agentic playground . 2. Connect an agentic workflow . 3. Chat with a single agent to test, tune, and view tracing or with multiple agents to compare results.. 4. Connect additional agentic workflows. 5. Add datasets and metrics to a playground or to individual agentic workflows to evaluate responses. ## Add an agentic playground To add an agentic playground, [create a Use Case](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/usecases/usecase-overview.html#create-a-use-case) (if a new Use Case is needed), then, in the left navigation bar, click the Playgrounds tile. From the Playgrounds tab, the following options are available, depending on how many playgrounds already exist: - If you haven't added a playground yet,click theAdd Playgrounddropdown in the center of the page, then clickAdd RAG playground. This button is only available for the first playground added to a Use Case; use theAdd Playgrounddropdown for subsequent playgrounds. - If you've already added one or more playgrounds (RAG or agentic),click theAdd Playgrounddropdown in the upper-right corner of the page, then clickAdd RAG playground. **Playground naming** The playground is named, by default, `Playground `. You can change the name from the Use Case directory by choosing Edit playground info in the Actions menu. ## Navigate the playground After you create and open an agentic playground, you can access several areas of the playground from the left navigation bar. Click the icons to navigate the playground: | Icon | Component | Description | | --- | --- | --- | | | Agentic playground | Connect, compare, and chat with agentic workflows. | | | Playground info | Display playground summary information including the name, description, creation time and date, creator, last modification time and date, and number of agentic workflows. | | | Evaluation* | Configure evaluation metrics in a playground. | | | Tracing* | Display an exportable log that traces all components used in agent response generation. | * Available only if LLM/agentic assessment is enabled. > [!NOTE] Return to the Playgrounds tile > Also in the left navigation, you can click back to return to the Playgrounds tile in the current Use Case. ## Connect an agentic workflow Agentic workflows can be created manually in the Registry [workshop](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-create-custom-model.html#assemble-an-agentic-workflow), or programmatically using the [agent templates in the publicdatarobot-agent-applicationrepository](https://github.com/datarobot-community/datarobot-agent-application). This repository provides ready-to-use templates for building and deploying AI agents with multi-agent frameworks, streamlining the process of setting up your own agents—with minimal configuration requirements. To connect an agentic workflow to a playground, select the Agentic playground tile and then the Workflows tab. The following options are available: - If you haven't added a workflow yet,clickConnect workflow from workshopin the center of the page. This button is only available for the first workflow added to a playground. - If you've already added one or more workflows,click theConnect agentic workflowdropdown in the upper-left corner of the page. In the Agentic workflow modal, from the Workshop dropdown, select an agentic workflow that you previously assembled in the workshop, then click Connect. ## Manage connected agentic workflows To access management actions for connected agentic workflows, click the actions menu for an agentic workflow, either in the Workflows panel, or in the header of the Agentic workflow comparison, next to the agent name. | Action | Description | | --- | --- | | Edit agentic workflow name | Change the current name for the agentic workflow. The default name is the name defined in the Registry's Workshop. This option is only available in the actions menu in the Workflows panel. | | Open in codespace | Open a codespace containing the agentic workflow's files to fine-tune the agentic workflow by editing the underlying code. | | Register agentic workflow | Open the agentic workflow in the workshop to make any final changes to the configuration, register the workflow, and deploy to production. | | Remove from playground | Disconnect the selected agentic workflow from the current playground. This does not remove the agentic workflow from the workshop, and you can restore the connection at any time. | --- # Review tracing URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-tracing.html > Review tracing for agentic workflows in a playground. Tracing the execution of agentic workflows is a powerful tool for understanding how most parts of the GenAI stack work. The Tracing tile provides a log of all components and prompting activity used in generating agent responses in the playground. Insights from tracing provide full context of everything the agent evaluated, including prompts, vector database chunks, and past interactions within the context window. For example: - DataRobot metadata: Reports the timestamp, status, Use Case, playground, agentic workflow, and the creator name. These help pinpoint the sources of trace records if you need to surface additional information from DataRobot objects interacting with the agentic workflow. - Prompts and responses: Provides a history of user prompts, agentic workflow responses, and user feedback. - Evaluations (if configured): Illustrates how evaluation metrics are scoring prompts or responses. To locate specific information in the Tracing table, click Filters and filter by any combination of Timestamp, User name, LLM Blueprint name, LLM, Vector database, Chat name, and Evaluation dataset. > [!TIP] Send tracing data to the Data Registry > Click Upload to Data Registry to export data from the tracing table to the Data Registry. A warning appears on the tracing table when it includes results from running the toxicity test and the toxicity test results are excluded from the Data Registry upload. ### Individual chat request tracing In addition to the logged information about all prompts and responses on the Tracing tile, you can also access tracing details on the path a single chat request takes through the agentic workflow. Do this from the Chats tab for a single agentic workflow. In a single-agent chat, click Review tracing next to the agent's name. Traces represent the path taken by a request to a model or agentic workflow. DataRobot uses the [OpenTelemetry framework for tracing](https://opentelemetry.io/docs/concepts/signals/traces/). A trace follows the entire end-to-end path of a request, from origin to resolution. Each trace contains one or more spans, starting with the root span. The root span represents the entire path of the request and contains a child span for each individual step in the process. The root (or parent) span and each child span share the same Trace ID. In the tracing details panel header, review the Trace ID for the request, the execution time (in ms), and the span services involved. On the List and Chart tabs, review the [spans](https://opentelemetry.io/docs/concepts/signals/traces/#spans) contained in the trace, along with trace details. The span colors correspond to a Span service. The first span service is the experiment container, sometimes followed by one or more deployments.Restricted span appears when you don’t have access to the deployment or service associated with the span. You can view spans in Chart format or List format. > [!TIP] Span detail controls > From either view, you can click Hide details panel to return to the single-agent-chat. **Chart view:** [https://docs.datarobot.com/en/docs/images/agentic-tracing-table-spans-chart.png](https://docs.datarobot.com/en/docs/images/agentic-tracing-table-spans-chart.png) **List view:** [https://docs.datarobot.com/en/docs/images/agentic-tracing-table-spans-list.png](https://docs.datarobot.com/en/docs/images/agentic-tracing-table-spans-list.png) > [!NOTE] Trace details > In list view, you can click Trace details to view the Input/Output ( Prompt and Completion) and Evaluation details about the trace associated with the current span. For either view, click the Span service name to access the deployment or resource (if you have access). Additional information, dependent on the configuration of the generative AI model or agentic workflow, is available on the Info, Resources, Events, Input/Output, and Error tabs. The Error tab only appears when an error occurs in a trace. --- # Build workflows URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-workflow-build.html > After you've connected an agentic workflow custom model to an agentic playground, as you compare and test the workflow, you can modify the agentic workflow's code in a codespace or in Workshop. After you've [connected an agentic workflow](https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/agentic-playground.html#connect-an-agentic-workflow) custom model to an agentic playground, as you compare and test the workflow, you can modify the workflow's code in a codespace or in the Registry workshop. Changes to the code are reflected in the agentic playground, allowing agentic workflow builders to experiment until they build a production-ready agentic workflow for deployment to Console. ## Agentic workflow components To assemble an agentic workflow, a standard workflow's custom model code could include the following components: | File | Contents | | --- | --- | | __init__.py | Python package initialization file, making the directory a Python package. | | custom.py | The custom model code implementing the Bolt-on Governance API (the chat hook) to call the LLM and also passing those parameters to the agent (defined in myagent.py). | | myagent.py | The agent code, implementing the agentic workflow in the MyAgent class with the required invoke method. Tool integration properties can be added to this class to interface with deployed tools. | | config.py | The code for loading the configuration from environment variables, runtime parameters, and DataRobot credentials. | | mcp_client.py | The code providing MCP server connection management for tool integration (optional, only needed when using MCP tools). | | tool_deployment.py | The BaseTool class code, containing all necessary metadata for implementing tools. | | tool.py | The code for interfacing with the deployed tool, defining the input arguments and schema. Often, this file won't be named tool.py, as you may implement more than one tool. In this example, this functionality is defined in tool_ai_catalog_search.py. | | model-metadata.yaml | The custom model metadata and runtime parameters required by the agentic workflow. | | pyproject.toml | The libraries (and versions) required by the agentic workflow, using modern Python packaging standards. | For more information on assembling agentic workflows, review the following resources: | Resource | Description | | --- | --- | | datarobot-agent-application repository | Documentation and ready-to-use templates for building and deploying AI agents with multi-agent frameworks. These templates streamline the process of setting up your own agents with minimal configuration requirements. | | agent-tool-templates repository | Documentation and source code for the global agentic tools available in the Registry. The source code for these tools can serve as templates for creating your own custom model tools for agentic workflows. | | datarobot-user-models repository | Tools, templates, and information for assembling, debugging, testing, and running your custom models, custom tasks and custom notebook environments with DataRobot. The custom model infrastructure is the foundation of agentic workflows. | | Custom model assembly documentation | Documentation for assembling, testing, and running custom models. | | Workshop documentation | Documentation for using the DataRobot UI to upload model artifacts to create, test, and deploy custom models to Console, a centralized model management and deployment hub. | ## Modify an agentic workflow Agentic workflows connected to the agentic playground can be continuously modified and developed as you prompt, compare, and evaluate metrics. This works for custom tools built manually, and tools built programmatically using the [agent templates in the publicdatarobot-agent-applicationrepository](https://github.com/datarobot-community/datarobot-agent-application). ### Codespace Codespaces are the primary method for developing your agentic workflow while testing in an agentic playground. In DataRobot, a codespace is a development environment you can use to view, modify, and run your agent's files. Changes made in the codespace are passed on to the agentic workflow in the agentic playground, as well as the custom agent in the Registry workshop. To develop an agentic workflow in a codespace, locate the Open in Codespace option in one of the following locations: **Workflows comparison (header):** [https://docs.datarobot.com/en/docs/images/agentic-playground-open-codespace-1.png](https://docs.datarobot.com/en/docs/images/agentic-playground-open-codespace-1.png) **Workflows comparison (sidebar):** [https://docs.datarobot.com/en/docs/images/agentic-playground-open-codespace-2.png](https://docs.datarobot.com/en/docs/images/agentic-playground-open-codespace-2.png) **Single model chat:** [https://docs.datarobot.com/en/docs/images/agentic-playground-open-codespace-3.png](https://docs.datarobot.com/en/docs/images/agentic-playground-open-codespace-3.png) Any of these options opens a codespace in a panel for the selected agentic workflow, where you can modify your agent's code to address issues identified in the playground. > [!NOTE] Codespace loading > The panel displays a Waiting for the codespace to start...message while the agent's files are loaded. When you finish modifying the agent code, click Save to pass those changes to the agentic workflow connected to the playground and in the Registry workshop. ### Workshop Similar to modifying an agentic workflow in a codespace, you can also modify your agent code directly in the Registry's workshop. For more information, see the [Workshop](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/index.html) documentation. When you save the [changes to your agentic workflow](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-create-custom-model.html#assemble-the-custom-model) in the workshop, a new version is created and passed to the agentic workflow connected to the playground. From the workshop you can also [configure evaluation metrics and moderations](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-configure-evaluation-moderation.html) not available in an agentic playground. --- # Evaluate URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-eval/index.html > Chat with a workflow, compare flows, and implement evaluation metrics. > [!NOTE] Premium > DataRobot's Agentic AI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. Create, connect, and test agentic workflows, integrate tools, and implement evaluation metrics. | Topic | Description | | --- | --- | | Connect to a playground | Connect to and interact with agentic workflows. | | Chat with agents | Chat with a single agentic workflow or with multiple workflows for comparison purposes. | | Evaluate metrics | Use the playground's evaluation tools, including evaluation metrics and datasets, aggregated metrics, and compliance tests. | | Review tracing | Review tracing of agentic workflow execution in a playground. | | Build workflows | Modify the agentic workflow's code in a codespace or in the Workshop. | --- # Connect agentic coding environments to MCP servers URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-clients.html > Configure Cursor, Claude Desktop, and VS Code to use your DataRobot MCP server for tools and resources in agentic coding workflows. You can connect your DataRobot MCP server to standard agentic coding environments—such as Cursor, Claude Desktop, and VS Code—to allow AI assistants in those environments to discover and call your MCP tools, prompts, and resources. This enables you to use DataRobot capabilities (e.g., projects, deployments, predictions, third-party tools) directly from your IDE or chat client. This guide explains how to configure each client to use an MCP server that you run locally or that is deployed to DataRobot. It applies to any of the following MCP connection options: - An MCP server deployed using the DataRobot Agentic Starter template ( datarobot-agent-application ). - A standalone MCP server deployed using the DataRobot MCP template ( af-component-datarobot-mcp ). - An MCP server implemented by the DataRobot Global MCP. > [!NOTE] MCP server vs. agent application > This page focuses on connecting MCP clients (Cursor, Claude Desktop, VS Code) to an MCP server. For integrating an MCP server into a DataRobot agentic workflow (e.g., LangGraph agent in the Agentic Starter template), see [Integrate tools using an MCP server](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-tools-mcp.html). ## Using the DataRobot Global MCP The DataRobot Global MCP is a service automatically deployed to your DataRobot instance that agentic workflows can use to access tools. > [!NOTE] Available tools > The DataRobot Global MCP currently supports tools for predictive AI. This limitation will be removed in a future release. ### Configuration The DataRobot Global MCP requires an API key to authenticate requests. You can obtain your API key from the DataRobot UI by opening the user menu and selecting API keys and tools. See [API key management](https://docs.datarobot.com/en/docs/platform/acct-settings/api-key-mgmt.html) for more information. Once you have your API key, configure the MCP client to use the DataRobot Global MCP endpoint. Refer to the steps that correspond to your MCP client in the tabs below. **Cursor:** ``` { "mcpServers": { "datarobot-mcp": { "url": "https://{DATAROBOT_URL}/api/v2/genai/globalmcp/mcp", "headers": { "Authorization": "Bearer " } } } } ``` To verify the connection, save `.cursor/mcp.json` in the correct location, restart Cursor or reload the window, then in Chat or Composer ask the AI to list or use tools from the DataRobot MCP server. **Claude Desktop:** ``` { "mcpServers": { "datarobot-mcp": { "command": "npx", "args": [ "-y", "mcp-remote@latest", "https://{DATAROBOT_URL}/api/v2/genai/globalmcp/mcp", "--header", "Authorization: ${AUTH_HEADER}", "--transport", "http" ], "env": { "AUTH_HEADER": "Bearer " } } } } ``` To verify the connection, save `claude_desktop_config.json` in the correct location, restart Claude Desktop, then ask Claude to use tools from the DataRobot MCP server. **VS Code:** ``` { "servers": { "datarobotMcp": { "type": "http", "url": "https://{DATAROBOT_URL}/api/v2/genai/globalmcp/mcp", "headers": { "Authorization": "Bearer " } } } } ``` To verify the connection, save `.vscode/mcp.json` under `.vscode/` or your user profile ( MCP: Open User Configuration in the Command Palette), reload the window if prompted, then in Copilot Chat ask the AI to list or use tools from the DataRobot MCP server. ## Using a standalone MCP server You can also use a standalone MCP server that you deploy to your own infrastructure. To use a standalone MCP server, you need to configure your MCP client to use the standalone MCP server endpoint. ### Prerequisites Before configuring your coding environment, ensure that you have: - A running MCP server (local or deployed): - Local - Deployed - Endpoint and auth - "Authorization": "Bearer " (required for authentication with the MCP server) - "x-datarobot-api-token": " " (required for tool execution) > [!TIP] Finding the deployed MCP endpoint > To find the endpoint of the deployed MCP server: > > For the > DataRobot MCP template > , after deploying run > task infra:info > or check the Pulumi/output step for > MCP_SERVER_MCP_ENDPOINT > . > For the > Agentic Starter template > , the deployment output includes an MCP server endpoint. Use the URL shown there for your client. ### Endpoint reference | Context | Base URL | Notes | | --- | --- | --- | | Agentic Starter (local) | http://localhost:9000/mcp | Default port is 9000; set MCP_SERVER_PORT to change it. | | MCP template (local) | http://localhost:8080/mcp | Default port is 8080; set MCP_SERVER_PORT to change it. | | Deployed to DataRobot | https://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp | Use the exact URL from your deployment output. | ### Configure your environment You can configure your MCP client to use the MCP server endpoint. Refer to the steps that correspond to your MCP client in the tabs below. **Cursor:** > [!NOTE] Cursor MCP docs > For Cursor's current MCP options, see [Cursor's MCP documentation](https://cursor.com/docs/context/mcp). Configuration file location: Project-specific: /.cursor/mcp.json Global: ~/.cursor/mcp.json For a local MCP server using the [DataRobot Agentic Starter](https://github.com/datarobot-community/datarobot-agent-application) on port 9000: ``` { "mcpServers": { "datarobot-mcp": { "url": "http://localhost:9000/mcp", "headers": { "Authorization": "Bearer ", "x-datarobot-api-token": "" } } } } ``` For a local MCP server using the [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp) on port 8080: ``` { "mcpServers": { "datarobot-mcp": { "url": "http://localhost:8080/mcp", "headers": { "Authorization": "Bearer ", "x-datarobot-api-token": "" } } } } ``` For a deployed MCP server: ``` { "mcpServers": { "datarobot-mcp": { "url": "https://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp", "headers": { "Authorization": "Bearer ", "x-datarobot-api-token": "" } } } } ``` **Claude Desktop:** Claude Desktop can connect to remote MCP servers over HTTP. Configure the MCP server in `claude_desktop_config.json`. > [!NOTE] Claude Desktop MCP docs > For Claude Desktop's current MCP options, see [Claude Desktop's MCP documentation](https://docs.anthropic.com/en/docs/build-with-claude/mcp). Configuration file location: macOS: ~/Library/Application Support/Claude/claude_desktop_config.json Windows: %APPDATA%\Claude\claude_desktop_config.json For a local MCP server using the [DataRobot Agentic Starter](https://github.com/datarobot-community/datarobot-agent-application) on port 9000: ``` { "mcpServers": { "datarobot-mcp": { "command": "npx", "args": [ "-y", "mcp-remote@latest", "http://localhost:9000/mcp", "--header", "Authorization: ${AUTH_HEADER}", "--header", "x-datarobot-api-token: ${DR_API_TOKEN}", "--transport", "http" ], "env": { "AUTH_HEADER": "Bearer ", "DR_API_TOKEN": "" } } } } ``` For a local MCP server using the [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp) on port 8080: ``` { "mcpServers": { "datarobot-mcp": { "command": "npx", "args": [ "-y", "mcp-remote@latest", "http://localhost:8080/mcp", "--header", "Authorization: ${AUTH_HEADER}", "--header", "x-datarobot-api-token: ${DR_API_TOKEN}", "--transport", "http" ], "env": { "AUTH_HEADER": "Bearer ", "DR_API_TOKEN": "" } } } } ``` For a deployed MCP server: ``` { "mcpServers": { "datarobot-mcp": { "command": "npx", "args": [ "-y", "mcp-remote@latest", "https://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp", "--header", "Authorization: ${AUTH_HEADER}", "--header", "x-datarobot-api-token: ${DR_API_TOKEN}", "--transport", "http" ], "env": { "AUTH_HEADER": "Bearer ", "DR_API_TOKEN": "" } } } } ``` If your deployment requires authentication, add the appropriate headers or token mechanism as supported by Claude Desktop and your MCP server (e.g., environment variables or config fields; check [Claude Desktop documentation](https://docs.anthropic.com/en/docs/build-with-claude/mcp) and your server docs). > [!TIP] Debugging Claude Desktop MCP > On macOS, MCP-related logs are often under `~/Library/Logs/Claude/` (e.g., `mcp*.log`). Use them to troubleshoot connection or auth issues. **VS Code:** VS Code (with GitHub Copilot) includes built-in MCP client support. Configure remote MCP servers in an `mcp.json` file (HTTP servers use `type`, `url`, and optional `headers`). > [!NOTE] VS Code Copilot MCP docs > For VS Code's current MCP options, the configuration schema, and security notes, see [Use MCP servers in VS Code](https://code.visualstudio.com/docs/copilot/customization/mcp-servers) and the [MCP configuration reference](https://code.visualstudio.com/docs/copilot/reference/mcp-configuration). Configuration file location: Workspace: /.vscode/mcp.json (appropriate for shared, non-secret settings that you may commit to source control). User profile: run MCP: Open User Configuration from your coding environment's Command Palette ( ⇧⌘P / Ctrl+Shift+P ). Prefer this location, or another secret-management mechanism, for API keys and other credentials. Do not commit API keys If your MCP configuration includes headers such as Authorization or x-datarobot-api-token , do not commit those secrets to source control. Keep credentialed configuration in your user-profile mcp.json or another secure secret store. You can also use MCP: Add Server in your coding environment's Command Palette for a guided setup. For more options, see [Add and manage MCP servers in VS Code](https://code.visualstudio.com/docs/copilot/customization/mcp-servers). For a local MCP server using the [DataRobot Agentic Starter](https://github.com/datarobot-community/datarobot-agent-application) on port 9000: ``` { "servers": { "datarobotMcp": { "type": "http", "url": "http://localhost:9000/mcp", "headers": { "Authorization": "Bearer ", "x-datarobot-api-token": "" } } } } ``` For a local MCP server using the [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp) on port 8080: ``` { "servers": { "datarobotMcp": { "type": "http", "url": "http://localhost:8080/mcp", "headers": { "Authorization": "Bearer ", "x-datarobot-api-token": "" } } } } ``` For a deployed MCP server: ``` { "servers": { "datarobotMcp": { "type": "http", "url": "https://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp", "headers": { "Authorization": "Bearer ", "x-datarobot-api-token": "" } } } } ``` If your deployed server requires a token, configure authentication using HTTP headers supported by your MCP server. Avoid passing tokens in URLs or query parameters to reduce the risk of credential leakage; see [Use MCP servers in VS Code](https://code.visualstudio.com/docs/copilot/customization/mcp-servers) and your server docs. ## Apply and verify To confirm connectivity: 1. Save your MCP configuration in the correct location. 2. Restart your MCP client (or reload the window). 3. In your MCP client, ask the AI to list or use tools from the DataRobot MCP server. ## Troubleshooting ### Client cannot connect to MCP server Symptoms: The IDE or Claude reports that the MCP server is unavailable, or tools do not appear. Solutions: 1. Confirm the MCP server is running: 2. Local: run curl -i http://localhost:9000/ or curl -i http://localhost:8080/ (adjust the port to match your local MCP server setup). A successful response indicates the server is up. 3. Deployed: run curl -i using the exact URL from your deployment output. If the server requires authentication, you may need to pass a bearer token or other headers; see your deployment documentation. A successful response indicates the server is up. 4. Check URL and path: Use the exact base URL and path (e.g., /mcp ) required by your client and server. 5. Confirm firewall and network access: For deployed servers, ensure your network allows outbound HTTPS to the DataRobot host. ### Tools do not appear in the client Symptoms: Connection seems OK, but the client does not list MCP tools. Solutions: 1. Restart the client (Cursor, Claude Desktop, or VS Code) after changing the MCP config. 2. Check client and MCP server logs for errors (e.g., Cursor: Output → MCP Logs; Claude: ~/Library/Logs/Claude/ ). A successful connection will show MCP server logs with tool registration and availability information. ### Authentication errors Symptoms: Requests to the deployed MCP server return 401 or similar. Solutions: 1. Confirm the deployed server's auth requirements (e.g., bearer token). 2. Ensure the client is configured with the correct token or credentials (environment variables or extension settings). 3. For DataRobot deployments, ensure DATAROBOT_API_TOKEN (or the equivalent used by the server) is valid and has access to the deployment. ## Additional resources - Integrate tools using an MCP server — Use an MCP server from a DataRobot agentic workflow (e.g., LangGraph agent). - DataRobot MCP template — Build and deploy standalone MCP servers with DataRobot integration. - DataRobot MCP template — Configure your MCP client — Client setup details from the template repo. - DataRobot Agentic Starter template — Full agentic application including an MCP server. - Model Context Protocol — Official MCP specification. --- # Dynamic tool registration URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-dynamic-tool-registration.html > Learn how the MCP server discovers DataRobot deployments and exposes them as MCP tools automatically. Dynamic tool registration lets the MCP server discover DataRobot deployments and expose them as MCP tools automatically. When a tool is invoked, the server proxies the request to the registered deployment. This capability is available in MCP servers built with the [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp) and the [DataRobot Agentic Starter template](https://github.com/datarobot-community/datarobot-agent-application). For MCP server setup and agent integration, see [Integrate tools using an MCP server](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-tools-mcp.html). ## Quick start 1. Deploy your model or service to DataRobot. 2. Tag the deployment with tool as both the tag name and the tag value. 3. Start the server with dynamic tool registration enabled, if needed. To enable auto-discovery on startup (optional): ``` MCP_SERVER_REGISTER_DYNAMIC_TOOLS_ON_STARTUP=true ``` For most deployment types, no additional configuration is needed. ## Supported deployment types | Deployment type | Configuration needed | | --- | --- | | DataRobot native predictive models | None. Tag the deployment as tool. | | DRUM structured predictions | None. Optionally define inputSchema in model-metadata.yaml. | | DRUM agentic workflows | None. Optionally define inputSchema in model-metadata.yaml. | | DRUM unstructured models | Define inputSchema in model-metadata.yaml. | | Custom servers, such as FastAPI services | Expose an /info/ endpoint with tool metadata. | Registering other MCP servers as tools through dynamic tool registration is not supported. ## Registration requirements All deployments must: - Be active. - Be tagged with tool as both the name and value. Additional requirements depend on the deployment type: - DataRobot native models : No extra requirements. - DRUM unstructured models : Define inputSchema in model-metadata.yaml . - Custom servers : Expose /info/ and return endpoint , method , and input_schema . ### Runtime API Use these endpoints on the MCP server to manage registrations at runtime: - GET /registeredDeployments : List registered tools. - PUT /registeredDeployments/{deployment_id} : Register a tool. - DELETE /registeredDeployments/{deployment_id} : Remove a tool. For example, the `GET /registeredDeployments` endpoint can be implemented as follows: ``` # List registered deployments (excerpt) @mcp.custom_route(prefix_mount_path("/registeredDeployments"), methods=["GET"]) async def list_deployments(_: Request) -> JSONResponse: """List all deployments.""" try: deployments = await get_registered_tool_deployments() formatted_deployments = [ {"deploymentId": k, "toolName": v} for k, v in deployments.items() ] return JSONResponse( status_code=HTTPStatus.OK, content={ "deployments": formatted_deployments, "count": len(deployments), }, ) except Exception as e: return JSONResponse( status_code=HTTPStatus.INTERNAL_SERVER_ERROR, content={"error": f"Failed to retrieve deployments: {str(e)}"}, ) ``` ## DRUM deployments [DataRobot DRUM](https://pypi.org/project/datarobot-drum/) deployments usually work with little or no additional configuration. ### Zero configuration (for most cases) For these deployment types, tag the deployment as `tool` and the server can register it automatically: - Structured predictions, including binary, regression, and multiclass models. - Agentic workflows. - DataRobot native predictive models. ### Unstructured models For an `unstructured` target type, add `inputSchema` to `model-metadata.yaml`: ``` # model-metadata.yaml name: "Fetch dataset" description: "Fetches a dataset from DataRobot Data Registry" type: inference targetType: unstructured inputSchema: type: object properties: json: type: object properties: dataset_id: type: string description: Dataset ID from Data Registry limit: type: integer default: 100 required: - dataset_id ``` > [!NOTE] Unstructured model schemas > For unstructured models, define request parameters under the > json > property. > Exposing input schemas from > model-metadata.yaml > requires > datarobot-drum > version > 1.17.2 > or later. ### Optional custom schema You can override fallback schemas to give the LLM better guidance or tighter control over the request shape: ``` # model-metadata.yaml (custom inputSchema) inputSchema: type: object properties: data: type: string description: "CSV with columns: transaction_amount, user_age, merchant_category" required: - data ``` ## Custom server deployments For FastAPI, Flask, and similar services, expose an `/info/` endpoint that returns tool metadata. ### Required /info/ response ``` { "endpoint": "directAccess/weather/{city}", "method": "GET", "input_schema": { "type": "object", "properties": { "path_params": { "type": "object", "properties": { "city": { "type": "string", "description": "City name" } }, "required": ["city"] } } } } ``` ### FastAPI example ``` # FastAPI custom server (excerpt) from fastapi import FastAPI from pydantic import BaseModel, Field app = FastAPI() class WeatherRequest(BaseModel): class PathParams(BaseModel): city: str = Field(description="City name") class QueryParams(BaseModel): units: str = Field(default="metric", description="metric or imperial") path_params: PathParams query_params: QueryParams | None = None @app.get("/info/") async def metadata(): return { # Custom model deployments expose custom server routes behind directAccess/. "endpoint": "directAccess/weather/{city}", "method": "GET", "input_schema": WeatherRequest.model_json_schema(), } @app.get("/weather/{city}") async def get_weather(city: str, units: str = "metric"): return {"city": city, "temp": 22, "units": units} ``` ### Request mapping Given this tool call: ``` { "path_params": {"city": "paris"}, "query_params": {"units": "imperial"} } ``` the MCP server generates a request like: ``` GET /directAccess/weather/paris?units=imperial ``` Where: - base_url is derived from the DataRobot deployment URL. - directAccess/ is the prefix used for custom server endpoints in custom model deployments. ## Input schema reference ### Parameter groups Parameters map to HTTP requests as follows: | Group | Purpose | Example | | --- | --- | --- | | path_params | Substitutes values into the URL path. | {city} → "paris" | | query_params | Adds query-string parameters. | ?units=imperial | | data | Sends a raw request body, such as CSV. | Not used in the weather example. | | json | Sends a JSON request body. | Not used in the weather example. | ### Rules - path_params and query_params must be flat objects. - data and json can contain nested structures. - Every {param} in the endpoint must be present in path_params . - Empty schemas are allowed only when MCP_SERVER_TOOL_REGISTRATION_ALLOW_EMPTY_SCHEMA=true . ### What the server does Internally, the MCP server transforms tool calls into HTTP requests. For the weather example, the request looks like this: ``` async with session.request( method="GET", url="/directAccess/weather/paris", params={"units": "imperial"}, ) as response: return await response.json() ``` ## Troubleshooting ### Tool does not register Use these commands to inspect the deployment and, for DRUM or custom servers, test the `/info/` endpoint: ``` # Check that the deployment is active and tagged correctly. curl -H "Authorization: Bearer $DATAROBOT_API_TOKEN" \ "$DATAROBOT_ENDPOINT/api/v2/deployments/{deployment-id}/" | jq . # Check the /info/ endpoint for DRUM and custom server deployments. curl -H "Authorization: Bearer $DATAROBOT_API_TOKEN" \ "$DATAROBOT_ENDPOINT/api/v2/deployments/{deployment-id}/directAccess/info/" | jq . ``` ### Common errors | Error | Fix | | --- | --- | | Missing input_schema | Add input_schema to the /info/ response for custom servers, or add inputSchema to model-metadata.yaml for DRUM unstructured models. | | Unsupported top-level property | Use only path_params, query_params, data, and json. | | Nested structure in path_params | Flatten the structure or move it to json. | | Missing path parameter | Define every path variable from the endpoint in path_params. | ## Additional resources - Integrate tools using an MCP server —connect MCP tools to agentic workflows. - Connect agentic coding environments to MCP servers —configure Cursor, Claude Desktop, and VS Code. - DataRobot MCP template —build and deploy standalone MCP servers. - DataRobot Agentic Starter template —agentic application with MCP integration. --- # MCP server overview URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-overview.html > Learn what Model Context Protocol (MCP) servers are and the types of MCP servers DataRobot provides, including the Global MCP, standalone MCP template, and Agentic Starter bundled server. The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is an open standard that lets AI agents, large language models (LLMs), and coding assistants discover and interact with external tools, data sources, and services through a consistent interface. Instead of embedding tool logic inside every agent or client, you deploy an MCP server that hosts tools centrally. Clients connect to that server at runtime to list and invoke tools without custom integration code per client. DataRobot provides several MCP server options—from a managed platform service to templates you deploy yourself—so you can expose DataRobot capabilities and custom logic to agentic workflows, IDEs, and chat clients. ## How MCP works MCP uses a client-server model: - The client —the reasoning engine or assistant that plans tasks. Examples include a LangGraph agent in the DataRobot Agentic Starter template , or an IDE assistant in Cursor, Claude Desktop, or VS Code. - The MCP server —a web service that hosts tool logic, resources, and prompts. The server executes tasks such as calling the DataRobot API, running predictions, or querying external systems. - The protocol —a standard interface for discovery and invocation. Clients ask the server which tools are available and send structured requests to run them. ``` flowchart LR Client["MCP client
(agent, IDE, chat app)"] Server["MCP server
(tools, resources, prompts)"] DR["DataRobot platform
and external APIs"] Client <-->|"MCP protocol"| Server Server --> DR ``` ### Why use an MCP server? Using an MCP server to provide tools offers several advantages over embedding tools directly in agent code or deploying each tool separately: - Centralized tool management —define and host tools in one place that multiple agents or clients can share. - Standardized interface —tools follow the MCP protocol, so they work across MCP-compatible frameworks and clients. - Dynamic discovery —clients list available tools at runtime; you can add or change tools on the server without redeploying every client. - Separation of concerns —scale and update the tool server independently from your agents and applications. For tool integration patterns inside agentic workflows, see [Integrate tools using an MCP server](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-tools-mcp.html). For connecting IDEs and chat clients, see [Connect agentic coding environments to MCP servers](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-clients.html). ## DataRobot MCP server types DataRobot supports three main MCP server options. The right choice depends on whether you need a managed platform endpoint, a fully customizable server, or an MCP server bundled with an agentic application. | Option | Deployment | Best for | | --- | --- | --- | | DataRobot Global MCP | Automatically available on your DataRobot instance. | Quick access to platform tools without deploying your own server. | | Standalone MCP server | Local development or DataRobot deployment via the MCP template. | Full control, custom tools, integrations, and dynamic tool registration. | | Agentic Starter MCP server | Bundled with the Agentic Starter template. | Agentic workflows that connect to MCP tools at runtime via built-in client support. | ### DataRobot Global MCP The DataRobot Global MCP is a persistently deployed MCP server that DataRobot automatically provisions on your instance. Agentic workflows and MCP clients can connect to it using a fixed platform endpoint without deploying a separate MCP application. Endpoint: ``` https://{DATAROBOT_URL}/api/v2/genai/globalmcp/mcp ``` Key characteristics: - Managed service —no separate MCP deployment to create or maintain. - Platform-native access —connect from agentic workflows or coding environments using your DataRobot API key. - Predictive AI tools (initial release) —the Global MCP currently exposes DataRobot predictive AI tools; additional tool categories are planned for future releases. > [!NOTE] Global MCP tool availability > The DataRobot Global MCP currently supports tools for predictive AI. This limitation will be removed in a future release. Authentication: Requests require a DataRobot API key passed as a Bearer token. Obtain your key from the user menu under API keys and tools. See [API key management](https://docs.datarobot.com/en/docs/platform/acct-settings/api-key-mgmt.html). When to use it: - You want immediate access to DataRobot predictive tools from Cursor, Claude Desktop, VS Code, or an agentic workflow. - You do not need custom tools, third-party integrations, or deployment-specific configuration. - Your environment provides the Global MCP endpoint (available in recent DataRobot releases). For client configuration examples, see [Using the DataRobot Global MCP](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-clients.html#using-the-datarobot-global-mcp). ### Standalone MCP server The [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp) ( `af-component-datarobot-mcp`) is an [App Framework](https://af.datarobot.com) component that deploys a FastMCP-based MCP server as a DataRobot custom model application. You can run it locally for development or deploy it to DataRobot for production. Endpoints: | Environment | URL | | --- | --- | | Local (default) | http://localhost:8080/mcp | | DataRobot deployment | https://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp | Key characteristics: - Comprehensive built-in tools —pre-built tools for DataRobot platform operations, third-party integrations, and web search. - Dynamic tool registration —automatically expose tagged DataRobot deployments as MCP tools. - Custom tool authoring —add domain-specific tools using FastMCP decorators and Python type hints. - Repeatable instances —apply the component multiple times in one project under different names for separate MCP backends. - OpenTelemetry tracing —optional observability for tool calls in production. Built-in tool categories include: - DataRobot platform —catalog and datasets, modeling and projects, deployments, batch and real-time predictions, vector databases, use cases, and documentation lookup. - Data connectors —Confluence, Jira, Google Drive, and Microsoft 365 (OAuth configuration required). - Web search —Perplexity and Tavily (API keys required). ### Supported built-in MCP tools The standalone [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp) currently supports the following built-in tools. > [!NOTE] Tool inventory source > This list is based on the current `af-component-datarobot-mcp` codebase. Tool availability can vary by feature flags and credentials (OAuth providers, Perplexity/Tavily API keys, and DataRobot API access). DataRobot - Catalog (datasets and datastores): - catalog_upload_dataset , catalog_list_datasets , catalog_get_preview , catalog_list_datastores , catalog_browse_datastore - catalog_query_datastore , catalog_check_timeseries_eligibility , catalog_analyze_dataset , catalog_suggest_ml_problems , catalog_get_eda_insights DataRobot - Modeling (projects and models): - models_get_bestmodel , modeling_score_dataset , modeling_list_models , modeling_get_modeldetails , modeling_list_projects - modeling_get_project_dataset , modeling_start_autopilot , modeling_get_model_roc , modeling_get_model_feature_impact , modeling_get_model_lift_chart DataRobot - Deployments: - deployment_get_list , deployment_get_model_info , deployment_create_deployment , deployment_get_prediction_history , deployment_get_info - deployment_generate_prediction_sample , deployment_validate_prediction_data , deployment_get_features DataRobot - Predictions: - predict_batch_predictions_from_dataset , predict_batch_predictions_from_partition , predict_get_batch_job_status - predict_get_batch_results , predict_score_catalog_realtime , predict_score_inline_realtime DataRobot - Vector databases, use cases, and docs: - vdb_list , vdb_query , datarobot_usecases_list , usecases_list_assets , datarobot_docs_fetch_page Data connectors - Confluence: - confluence_get_page , confluence_create_page , confluence_add_comment , confluence_search_space , confluence_update_page Data connectors - Jira: - jira_search_issues , jira_get_issue , jira_create_issue , jira_update_issue , jira_transition_issue Data connectors - Google Drive: - gdrive_find_contents , gdrive_read_and_export_content , gdrive_create_file , gdrive_update_metadata , gdrive_manage_access Data connectors - Microsoft 365: - microsoft_graph_search_content , microsoft_graph_share_item , microsoft_graph_create_file , microsoft_graph_update_metadata Web search tools: - perplexity_search , perplexity_sonar , tavily_search_web , tavily_extract_text , tavily_list_links , tavily_crawl_site When to use it: - You need custom tools, integration with collaboration platforms, or full control over the tool surface. - Multiple agents or MCP clients should share one tool backend. - You want to turn DataRobot deployments into tools automatically via dynamic tool registration . For deployment and development, see the [MCP Server application template](https://docs.datarobot.com/en/docs/wb-apps/app-templates/at-mcp-server.html) and the [MCP template repository](https://github.com/datarobot-community/af-component-datarobot-mcp). ### Agentic Starter MCP server The [DataRobot Agentic Starter template](https://github.com/datarobot-community/datarobot-agent-application) includes an MCP server alongside its FastAPI backend, React frontend, and LangGraph agent workflows. The server can run locally during development or deploy to DataRobot with the rest of the application. Endpoints: | Environment | URL | | --- | --- | | Local (default) | http://localhost:9000/mcp/ | | DataRobot deployment | https://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp | Set `MCP_SERVER_PORT` in your `.env` file to change the local port. Key characteristics: - Integrated agent client —LangGraph workflows receive MCP tools at runtime through mcp_tools_context in custompy_adaptor (DRUM) and register.py (DRAgent). - Co-deployed architecture —agent, API, frontend, and MCP server ship together in one template. - Composable —the agent can also connect to a standalone MCP server or the Global MCP instead of (or in addition to) the bundled server. When to use it: - You are building an agentic application with the Agentic Starter template and want MCP tools available out of the box. - You prefer a single template that includes both the agent and its tool server for local development and deployment. The bundled MCP server follows the same MCP protocol as the standalone template. For agent-side integration details, see [Integrate tools using an MCP server](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-tools-mcp.html). ## Endpoint reference Use the following patterns when configuring agents or MCP clients. Always use the exact URL from your deployment output when available. | MCP server type | Context | Base URL | | --- | --- | --- | | Global MCP | Platform | https://{DATAROBOT_URL}/api/v2/genai/globalmcp/mcp | | Standalone MCP template | Local | http://localhost:8080/mcp | | Standalone MCP template | Deployed | https://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp | | Agentic Starter | Local | http://localhost:9000/mcp | | Agentic Starter | Deployed | https://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp | > [!TIP] Finding deployed endpoints > For the [MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp), run `task infra:info` or check deployment output for MCP_SERVER_MCP_ENDPOINT. For the [Agentic Starter template](https://github.com/datarobot-community/datarobot-agent-application), deployment output includes the MCP server endpoint directly. ## Authentication Remote MCP connections to DataRobot typically require: - Authorization: Bearer —DataRobot API key for authentication. - x-datarobot-api-token: —required for tool execution on many standalone and Agentic Starter deployments. The Global MCP generally requires the Bearer token only. See [Connect agentic coding environments to MCP servers](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-clients.html) for per-client configuration examples. ## Choose an MCP server Use this guidance to select the right option: - Start with the Global MCP if you need quick access to DataRobot predictive tools from an IDE or agent and your instance provides the endpoint. - Use the standalone MCP template if you need custom tools, third-party integrations (Jira, Confluence, Google Drive, Microsoft 365), web search, dynamic deployment registration, or a dedicated tool server shared across multiple applications. - Use the Agentic Starter MCP server if you are building on the Agentic Starter template and want MCP tools integrated with minimal setup alongside your LangGraph agent. You can combine options: for example, an Agentic Starter agent can connect to its bundled MCP server for development and to a production standalone MCP server or the Global MCP in deployed environments. ## MCP vs. direct tool integration DataRobot also supports integrating tools directly into agents without an MCP server (for example, via ToolClient and direct tool deployments). MCP is preferable when you want centralized tool management, protocol-standard compatibility, and runtime tool discovery. | Consideration | MCP server | Direct tool deployment | | --- | --- | --- | | Tool management | Centralized in one server. | Each tool deployed separately. | | Protocol | MCP standard. | DataRobot-specific. | | Tool discovery | Automatic at runtime. | Manual per tool. | | Dynamic updates | Update server without redeploying agents. | Often requires agent redeployment. | | Client compatibility | Any MCP-compatible client. | DataRobot agent framework. | For direct tool integration, see [Add tools to agents](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools-integrate.html). ## Next steps | Topic | Description | | --- | --- | | Integrate tools using an MCP server | Connect MCP tools to LangGraph and other agentic workflows. | | Connect agentic coding environments to MCP servers | Configure Cursor, Claude Desktop, and VS Code for Global MCP and standalone servers. | | MCP Server application template | Overview of the standalone MCP template and key features. | | DataRobot MCP template repository | Source, component setup, and in-repo developer documentation. | | DataRobot Agentic Starter template | Full agentic application with bundled MCP server. | | Model Context Protocol | Official MCP specification. | --- # Integrate tools using an MCP server URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-tools-mcp.html > Learn how to integrate tools into your agentic workflows using the Model Context Protocol (MCP) server instead of direct tool deployments. The Model Context Protocol (MCP) provides a standardized interface for AI agents to interact with external systems, tools, and data sources. Using an MCP server to provide tools to your agents offers several advantages over direct tool deployments: - Centralized tool management : All tools are managed in one MCP server deployment - Standardized interface : Tools follow the MCP protocol standard, making them compatible across different agent frameworks - Dynamic tool registration : Tools can be added or modified without redeploying the agent (see Dynamic tool registration ) - Better separation of concerns : Tools are deployed separately from agents, enabling independent scaling and updates This guide explains how to integrate tools using an MCP server with your agentic workflows, as implemented in the [DataRobot Agentic Starter template](https://github.com/datarobot-community/datarobot-agent-application). MCP endpoints can come from the Agentic Starter app, a standalone server built with the [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp), or the [DataRobot Global MCP](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-clients.html#using-the-datarobot-global-mcp); this page focuses on connecting your agent application to the MCP URL your deployment uses. > [!NOTE] MCP vs. local tool integration > This documentation covers MCP server-based tool integration. For information about integrating tools using local tools (direct tool deployments), see [Add tools to agents](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tools-integrate.html). To connect MCP clients in Cursor, Claude Desktop, or VS Code—including the DataRobot Global MCP and deployment URLs—see [Connect agentic coding environments to MCP servers](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-clients.html). ## Overview The Model Context Protocol (MCP) fundamentally changes how agents access tools. Instead of defining tool logic directly within the agent's code, you deploy a standalone "Tool Server." Your agent then connects to this server as a client, requesting available tools and executing them via a standardized protocol. The [DataRobot Agentic Starter template](https://github.com/datarobot-community/datarobot-agent-application) connects MCP tools at runtime: LangGraph workflows receive MCP (and optional workflow) tools as the `tools` argument to `graph_factory`, using `mcp_tools_context` in `custompy_adaptor` (DRUM) and in `register.py` (DRAgent). You do not import the MCP server into agent code. Refer to the sections below for more information on how to use MCP tools in your agentic workflows. ### MCP architecture - The agent (client) : The reasoning engine (e.g., a LangGraph application) that plans tasks. It does not contain tool code; it only knows how to ask the MCP server what tools are available. - The MCP server : A web service hosting the logic for tools, resources, and prompts. It handles the actual execution of tasks (e.g., querying a database, calling an external API). - The protocol : A standard interface that allows the agent and MCP server to communicate securely, regardless of the underlying infrastructure. ### Integration workflow To integrate tools using this architecture, follow this high-level process: - Deploy or select an MCP endpoint : Create and deploy an MCP server containing your tool logic (using the DataRobot MCP template ), use the MCP server bundled with the DataRobot Agentic Starter template , or connect through the DataRobot Global MCP when your environment uses that service. - Connect the agent : Configure your agent application to point to the MCP server's URL (for example, a deployment directAccess URL, a gateway URL, or a local dev URL). - Execute : The agent automatically discovers tools at runtime. You do not need to redeploy the agent to add new tools; you only update the MCP server. ## Prerequisites Before integrating MCP tools, ensure you have: - A reachable MCP endpoint: a server built with the DataRobot MCP template , the MCP server included with the DataRobot Agentic Starter template , or the DataRobot Global MCP (depending on your setup). - An agentic workflow based on the DataRobot Agentic Starter template . - The MCP endpoint URL and authentication credentials (if required). For deploying a standalone MCP server, refer to the [MCP template documentation](https://github.com/datarobot-community/af-component-datarobot-mcp/blob/main/README.md). For URL patterns (local ports, deployment `directAccess`, gateway), see [Connect agentic coding environments to MCP servers](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-clients.html). ## MCP server deployment The MCP server can be deployed either locally for development or to DataRobot for production use. ### Local deployment For local development, start the MCP server using the development command: ``` cd mcp_server dr task run mcp_server:dev ``` The MCP server will start on the configured port (default: `9000`) and be accessible at `http://localhost:9000/mcp/`. ### Production deployment To deploy the MCP server to DataRobot: 1. Configure deployment settings: Update your Pulumi configuration or deployment settings. 2. Deploy the server: Use the deployment command: drrundeploy You can also usedr task run deploy, which is equivalent. For more on these commands, see the CLItaskandruncommands. 3. Get the deployment endpoint: After deployment, note the deployment ID and construct the MCP endpoint URL: https://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp Some environments use theDataRobot Global MCPinstead of a per-deploymentdirectAccessURL: https://{DATAROBOT_URL}/api/v2/genai/globalmcp/mcp SeeUsing the DataRobot Global MCPfor how Global MCP URLs fit alongside deployment endpoints. 4. Configure the agent: Update your agent's configuration to use the production MCP endpoint (whichever URL your deployment provides). For detailed deployment instructions, refer to the [MCP template README](https://github.com/datarobot-community/af-component-datarobot-mcp/blob/main/README.md). ## Configure MCP server connection To connect your agent to an MCP server, you need to configure the MCP server endpoint and authentication in your agent's environment variables or configuration. ### Local development For local development, configure the MCP server connection in your `.env` file: ``` # .env # MCP Server Configuration MCP_SERVER_PORT=9000 ``` The MCP server uses one of two ports by default: - Port 8080 when deployed using the DataRobot MCP template - Port 9000 when deployed using the DataRobot Agentic Starter template Make sure to use the correct port when configuring the agent. ### Production configuration For production deployments, configure the MCP server using environment variables. This process is handled automatically by the template. > [!NOTE] MCP server authentication > If your MCP server requires authentication, ensure that `DATAROBOT_API_TOKEN` is set in your environment, as the MCP client will use it automatically for authenticated requests. HTTP MCP endpoints often expect both `Authorization: Bearer` and `x-datarobot-api-token` headers; the client typically supplies these from your DataRobot API credentials. For the same headers in Cursor, Claude Desktop, and VS Code, see [Connect agentic coding environments to MCP servers](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-clients.html). ## Use MCP tools in agents In the current [DataRobot Agentic Starter template](https://github.com/datarobot-community/datarobot-agent-application) (LangGraph), MCP tools are passed into your `graph_factory(llm, tools, verbose)` as the `tools` list. The template merges MCP discovery with any workflow tools before calling `MyAgent`. ### LangGraph agents For LangGraph agents, pass the `tools` argument through to each `create_agent` node: ``` # agent/agent/myagent.py (excerpt) from datarobot_genai.core.agents import make_system_prompt from langchain.agents import create_agent def graph_factory(llm, tools, verbose=False): planner = create_agent( llm, tools=tools, system_prompt=make_system_prompt( "You are a content planner. Use available tools to gather information and plan content." ), name="planner_agent", debug=verbose, ) ... ``` At runtime, `tools` includes MCP tools resolved via `mcp_tools_context` (see the template's `custompy_adaptor` and `register.py`). That flow: - Connects to the configured MCP server - Discovers available tools - Exposes LangChain-compatible tools to the graph - Applies authentication headers and authorization context when required ### Supported built-in MCP tools For the standalone [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp), built-in tools currently include: - DataRobot platform tools : catalog_* , modeling_* , models_get_bestmodel , deployment_* , predict_* , vdb_* , datarobot_usecases_list , usecases_list_assets , and datarobot_docs_fetch_page - Data connector tools : confluence_* , jira_* , gdrive_* , and microsoft_graph_* (OAuth configuration required) - DataRobot platform tools : catalog_* , modeling_* , models_get_bestmodel , deployment_* , predict_* , vdb_* , datarobot_usecases_list , usecases_list_assets , and datarobot_docs_fetch_page . - Data connector tools : confluence_* , jira_* , gdrive_* , and microsoft_graph_* (OAuth configuration required). - Web search tools : perplexity_* and tavily_* (API keys required). For the full tool-by-tool inventory, see [Supported built-in MCP tools](https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/agentic-mcp-overview.html#supported-built-in-mcp-tools) in the MCP overview. ### Combine MCP tools with local tools Add local tools inside `graph_factory` by extending the list (MCP tools are already in `tools`): ``` # agent/agent/myagent.py (excerpt) from langchain_core.tools import Tool class DateTimeTool(Tool): """Local datetime tool.""" name = "datetime_tool" description = "Returns the current date and time." def run(self, query: str = "") -> str: from datetime import datetime return datetime.now().strftime("%Y-%m-%d %H:%M:%S") def graph_factory(llm, tools, verbose=False): local_tools = [DateTimeTool()] all_tools = list(tools) + local_tools planner = create_agent( llm, tools=all_tools, system_prompt=make_system_prompt("You are a content planner..."), name="planner_agent", debug=verbose, ) ... ``` ## Create custom MCP tools To add custom tools to your MCP server, use the [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp). The template provides a structured approach for creating tools, resources, and prompts. ### Tool structure Custom MCP tools are defined using the `@dr_mcp_tool` decorator: ``` # dr_mcp/app/recipe/tools/my_custom_tool.py from app.base.core.mcp_instance import dr_mcp_tool from app.base.core.common import get_sdk_client @dr_mcp_tool(tags={"custom", "recipe", "your-domain"}) async def my_custom_tool(input_param: str, optional_param: int = 10) -> str: """ Brief description of what your tool does. This description helps LLMs understand when and how to use your tool. Be specific about the tool's purpose and behavior. Args: input_param: Description of the required parameter optional_param: Description of the optional parameter Returns: Description of what the tool returns """ # Use the DataRobot SDK client for API operations client = get_sdk_client() # Your custom logic here result = f"Processed {input_param} with {optional_param}" return result ``` > [!NOTE] MCP template repository > The path and imports above refer to the [DataRobot MCP template](https://github.com/datarobot-community/af-component-datarobot-mcp) repository structure. ### Tool best practices When creating MCP tools, follow these best practices: - Clear descriptions : Provide detailed docstrings—LLMs use these to understand tool capabilities - Type hints : Always use type hints for parameters and return values - Error handling : Implement proper error handling and return meaningful error messages - Async functions : Tools should be async functions for better performance - Tags : Use descriptive tags to categorize tools (helps with tool filtering) - SDK client : Use get_sdk_client() for DataRobot API access For more information on creating custom MCP tools, see the [MCP template custom tools documentation](https://github.com/datarobot-community/af-component-datarobot-mcp/blob/main/docs/custom_tools.md). ## Comparison: MCP vs. ToolClient | Feature | MCP Server | ToolClient (Direct Deployment) | | --- | --- | --- | | Tool management | Centralized in one server. | Each tool deployed separately. | | Protocol | MCP standard protocol. | DataRobot custom protocol. | | Tool discovery | Automatic via MCP. | Manual configuration per tool. | | Dynamic updates | Tools can be added/modified without agent redeployment. | Requires agent redeployment for new tools. | | Framework compatibility | Works with any MCP-compatible framework. | DataRobot-specific. | | Deployment complexity | Single MCP server deployment. | Multiple tool deployments. | | Scaling | Scale MCP server independently. | Scale each tool independently. | Choose MCP server integration when: - You want centralized tool management - You need dynamic tool registration - You're using multiple agents that share tools - You want standard protocol compliance Choose ToolClient integration when: - You need fine-grained control over individual tool deployments - Tools have very different scaling requirements - You prefer DataRobot-specific tool management features ## Troubleshooting ### Agent can't connect to MCP server Symptoms: Agent errors mention MCP connection failures or tools not available. Solutions: 1. Verify MCP server is running: # For local developmentcurl-ihttp://localhost:9000/mcp# For production (deployment MCP endpoint)curl-ihttps://{DATAROBOT_URL}/api/v2/deployments/{DEPLOYMENT_ID}/directAccess/mcp# If your environment uses the DataRobot Global MCPcurl-ihttps://{DATAROBOT_URL}/api/v2/genai/globalmcp/mcp 2. Check environment variables: 3. Verify network connectivity: ### Tools not appearing Symptoms: MCP server is connected, but the agent cannot use tools. Solutions: 1. Check tool registration: Verify that tools are properly registered in the MCP server. 2. Review tool metadata: Ensure tool descriptions and schemas are correctly defined. 3. Check server logs: Review MCP server logs for tool registration errors. 4. Verify agent configuration: Confirm that the mcp_tools property is being used correctly. ### Authentication issues Symptoms: MCP server requests fail with authentication errors. Solutions: 1. Verify API token: Ensure DATAROBOT_API_TOKEN is set and valid. 2. Check token permissions: Verify the token has necessary permissions for MCP server access. 3. Review server configuration: Check that the MCP server is configured to accept the authentication method used. ## Additional resources - Connect agentic coding environments to MCP servers — DataRobot Global MCP, deployment and local URLs, and client configuration (Cursor, Claude Desktop, VS Code) - DataRobot MCP template - template for creating and deploying MCP servers - DataRobot Agentic Starter template - Application template with MCP integration - MCP Protocol Documentation - Official MCP protocol specification - Add tools to agents - Documentation for ToolClient-based tool integration - MCP Client Setup Guide - Guide for configuring MCP clients --- # MCP URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-mcp/index.html > Integrate tools using MCP servers and connect agentic coding environments to MCP. > [!NOTE] Premium > DataRobot's Agentic AI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. The Model Context Protocol (MCP) provides a standardized interface for AI agents to interact with external systems, tools, and data sources. Use MCP to centralize tool management, integrate tools into agentic workflows, and connect IDEs and chat clients to your DataRobot MCP server. | Topic | Description | | --- | --- | | MCP server overview | Learn what MCP servers are and compare the DataRobot Global MCP, standalone MCP template, and Agentic Starter bundled server. | | Integrate tools using an MCP server | Integrate tools into your agentic workflows using an MCP server for centralized tool management and a standardized interface. | | Dynamic tool registration | Discover DataRobot deployments and expose them as MCP tools automatically through the MCP server. | | Connect agentic coding environments to MCP servers | Configure Cursor, Claude Desktop, and VS Code to use your DataRobot MCP server for tools and resources. | --- # Custom metrics URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-custom-metrics.html > Create and monitor custom metrics and guard metrics for generative and agentic deployments. On a deployment's Monitoring > Custom metrics tab, you can use the data you collect from the [Data exploration](https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-data-exploration.html) tab (or data calculated through other custom metrics) to compute and monitor custom business or performance metrics. When you configure [evaluation and moderation guardrails](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-configure-evaluation-moderation.html) (including NeMo Evaluator metrics) for a text generation or agentic workflow deployment, the guards also report metrics to this tab—for example, guard latency, average score for prompt or response, and blocked count per guard—so you can monitor and debug guard behavior over time. These metrics are recorded on the configurable Custom metrics summary dashboard, where you monitor, visualize, and export each metric's change over time. This feature allows you to implement your organization's specialized metrics alongside [service health](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-service-health.html) for the deployment. > [!NOTE] Custom metrics limits > You can have up to 50 custom metrics per deployment, and of those 50, 5 can be hosted custom metrics. To view and add custom metrics, in the Console, open the deployment for which you want to create custom metrics and click the Monitoring > Custom metrics tab: **Q: What types of custom metrics are supported?** Three types of custom metrics are available for use: | Custom metric type | Description | | --- | --- | | External custom metrics | What: A custom metric where the calculations of the metric are not directly hosted by DataRobot. An external metric is a simple API used to submit a metric value for DataRobot to save and visualize. The metric calculation is handled externally, by the user. External metrics can be combined with other tools in DataRobot like Notebooks, Jobs, or Custom Models, or external tools like Airflow or cloud providers to provide the hosting and calculation needed for a particular metric.Where: On the Custom Metrics tab of any deployment, click Add custom metric > New external metric .Why: Provides a simple option to save a number from your AI solution that you want for tracking and visualization in DataRobot. For example, you could track the change in LLM cost, calculated by your LLM provider, over time. | | Hosted custom metrics | What: A custom metric where the metric calculations are hosted in a custom job within DataRobot. For hosted metrics, DataRobot orchestrates pulling the data, computing the metric values, saving the values to storage, and visualizing the data. No outside tools or infrastructure are required.Where: On the Custom Metrics tab of any deployment, click Add custom metric > New hosted metric .Why: Provides a complete end-to-end workflow for building business-specific metrics and dashboards in DataRobot. | | Hosted custom metric templates | What: A template or ready-to-use example of a hosted custom metric, where DataRobot provides the user the code and automates the creation process for a hosted custom metric. For metric templates, the result is a hosted metric, without starting from scratch. Templates are provided by DataRobot and can be used as-is or modified to calculate new metrics.Where: On the Custom Metrics tab of any deployment in NextGen only, click Add custom metric > Create new from template.Why: Provides the simplest way to get started with custom metrics, where DataRobot provides an example implementation and a complete end-to-end workflow. They are ready to use in just a few clicks. | ## Add custom metrics To add a metric, in a text generation, agentic workflow, VDB, or MCP deployment, click the Monitoring > Custom metrics tab. Then, on the Custom metrics tab, click + Add custom metric, select one of the following custom metric types, and proceed to the configuration steps linked in the table: **With existing metrics:** [https://docs.datarobot.com/en/docs/images/nxt-new-cus-metric-with-existing.png](https://docs.datarobot.com/en/docs/images/nxt-new-cus-metric-with-existing.png) **Without existing metrics:** [https://docs.datarobot.com/en/docs/images/nxt-new-cus-metric-without-existing.png](https://docs.datarobot.com/en/docs/images/nxt-new-cus-metric-without-existing.png) | Custom metric type | Description | | --- | --- | | New external metric | Add a custom metric where the calculations of the metric are not directly hosted by DataRobot. An external metric is a simple API used to submit a metric value for DataRobot to save and visualize. The metric calculation is handled externally, by the user. External metrics can be combined with other tools in DataRobot like notebooks, jobs, or custom models, or external tools like Airflow or cloud providers to provide the hosting and calculation needed for a particular metric.External custom metrics provide a simple option to save a value from your AI solution for tracking and visualization in DataRobot. For example, you could track the change in LLM cost, calculated by your LLM provider, over time. | | New hosted metric | Add a custom metric where the metric calculations are hosted in a custom job within DataRobot. For hosted metrics, DataRobot orchestrates pulling the data, computing the metric values, saving the values to storage, and visualizing the data. No outside tools or infrastructure are required.Hosted custom metrics provide a complete end-to-end workflow for building business-specific metrics and dashboards in DataRobot. | | Create new from template | Add a custom metric from a template, or ready-to-use example of a hosted custom metric, where DataRobot provides the code and automates the creation process. With metric templates, the result is a hosted metric, without starting from scratch. Templates are provided by DataRobot and can be used as-is or modified to calculate new metrics.Hosted custom metric templates provide the simplest way to get started with custom metrics, where DataRobot provides an example implementation and a complete end-to-end workflow. They are ready to use in just a few clicks. | ### Add external custom metrics External custom metrics allow you to create metrics with calculations occurring outside of DataRobot. With an external metric, you can submit a metric value for DataRobot to save and visualize. External metrics can be combined with other tools in DataRobot like notebooks, jobs, or custom models, or external tools like Airflow or cloud providers to provide the hosting and calculation needed for a particular metric. To add an external custom metric, in the Add custom metric dialog box, configure the metric settings, and then click + Add custom metric: **Numeric:** [https://docs.datarobot.com/en/docs/images/nxt-custom-metric-fields.png](https://docs.datarobot.com/en/docs/images/nxt-custom-metric-fields.png) **Categorical:** [https://docs.datarobot.com/en/docs/images/nxt-custom-metric-fields-categorical.png](https://docs.datarobot.com/en/docs/images/nxt-custom-metric-fields-categorical.png) | Field | Description | | --- | --- | | Name | A descriptive name for the metric. This name appears on the Custom metrics summary dashboard. | | Description | (Optional) A description of the custom metric; for example, you could describe the purpose, calculation method, and more. | | Name of Y-axis (label) | A descriptive name for the dependent variable. This name appears on the custom metric's chart on the Custom Metric Summary dashboard. | | Default interval | The default interval used by the selected Aggregation type. Only HOUR is supported. | | Metric type | The type of metric to create, Numeric or Categorical. The available metric settings change based on this selection. | | Numeric metric settings | | | Baseline | (Optional) The value used as a basis for comparison when calculating the x% better or x% worse values. | | Aggregation type | The type of metric calculation. Select from Sum, Average, or Gauge—a metric with a distinct value measured at single point in time. | | Metric direction | The directionality of the metric, controlling how changes to the metric are visualized. You can select Higher is better or Lower is better. For example, if you choose Lower is better, a 10% decrease in the calculated value of your custom metric will be considered 10% better, and displayed in green. | | Categorical metric settings | | | Class name | For each class added, a descriptive name (maximum of 200 characters). | | Baseline | (Optional) For each class added, the value used as a basis for comparison when calculating the x% better or x% worse values. | | Class direction | For each class added, the directionality of the metric, controlling how changes to the metric are visualized. You can select Higher is better or Lower is better. For example, if you choose Lower is better, a 10% decrease in the calculated value of your custom metric will be considered 10% better, and displayed in green. | | + Add class | To define each class needed for the categorical metric, click + Add class and configure the required class settings listed above. You can add up to ten classes. To remove a class, click Delete class. | | Model specific aggregation setting | | | Is model-specific | When enabled, links the metric to the model with the Model Package ID (the Registered Model Version ID) provided in the dataset. This setting influences when values are aggregated (or uploaded). For example: Model-specific (enabled): Model accuracy metrics are model-specific, so the values are aggregated separately. When you replace a model, the chart for your custom accuracy metric only shows data for the days after the replacement.Not model-specific (disabled): Revenue metrics aren't model-specific, so the values are aggregated together. When you replace a model, the chart for your custom revenue metric doesn't change. This field can't be edited after you create the metric. | | Column name definitions for standard deployments | | | Timestamp column | The column in the dataset containing a timestamp. | | Value column | The column in the dataset containing the values used for custom metric calculation. | | Date format | (Optional) The date format used by the timestamp column. | > [!NOTE] Note > You can override the Column names definition settings when you upload data to a custom metric, [as described below](https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-custom-metrics.html#upload-data-to-custom-metrics). ### Add hosted custom metrics Hosted custom metrics allow you to implement up to 5 of your organization's specialized metrics in a deployment, uploading the custom metric code using [DataRobot Notebooks](https://docs.datarobot.com/en/docs/workbench/wb-notebook/index.html) and hosting the metric calculation on custom jobs infrastructure. After creation, these custom metrics can be reused for other deployments. > [!NOTE] Custom metrics limits > You can have up to 50 custom metrics per deployment, and of those 50, 5 can be hosted custom metrics. > [!WARNING] Time series support > The [DataRobot Model Metrics (DMM)](https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/index.html) library does not support time series models, specifically [data export](https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/dmm-data-sources.html#export-prediction-data) for time series models. To export and retrieve data, use the [DataRobot API client](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest-release/reference/mlops/data_exports.html). To add a hosted custom metric, in the Add Custom Metric dialog box configure the metric settings, and then click Add custom metric from notebook: | Field | Description | | --- | --- | | Name | (Required) A descriptive name for the metric. This name appears on the Custom Metric Summary dashboard. | | Description | A description of the custom metric; for example, you could describe the purpose, calculation method, and more. | | Name of y-axis (label) | (Required) A descriptive name for the dependent variable. This name appears on the custom metric's chart on the Custom Metric Summary dashboard. | | Default interval | Determines the default interval used by the selected Aggregation type. Only HOUR is supported. | | Baseline | Determines the value used as a basis for comparison when calculating the x% better or x% worse values. | | Aggregation type | Determines if the metric is calculated as a Sum, Average, or Gauge—a metric with a distinct value measured at single point in time. | | Metric direction | Determines the directionality of the metric, which controls how changes to the metric are visualized. You can select Higher is better or Lower is better. For example, if you choose Lower is better a 10% decrease in the calculated value of your custom metric will be considered 10% better, displayed in green. | | Is model-specific | When enabled, this setting links the metric to the model with the Model Package ID (Registered Model Version ID) provided in the dataset. This setting influences when values are aggregated (or uploaded). For example: Model-specific (enabled): Model accuracy metrics are model specific, so the values are aggregated completely separately. When you replace a model, the chart for your custom accuracy metric only shows data for the days after the replacement.Not model-specific (disabled): Revenue metrics aren't model specific, so the values are aggregated together. When you replace a model, the chart for your custom revenue metric doesn't change. This field can't be edited after you create the metric. | | Schedule | Defines when the custom metrics are populated. Select a frequency (hourly, daily, monthly, etc.) and a time. Select Use advanced scheduler for more precise scheduling options. | After configuring a custom metric, DataRobot loads the notebook that contains the metric's code. The notebook contains one custom metric cell. A custom metric cell is a unique notebook cell, containing Python code defining how the metric is exported and calculated, code for scoring, and code to populate the metric. Modify the code in the custom metric cell as needed. Then, test the code by clicking Test custom metric code at the bottom of the cell. The test creates a custom job. If the test runs successfully, click Deploy custom metric code to add the custom metric to your deployment. > [!NOTE] Availability information > Notebooks for hosted custom metrics are off by default. Contact your DataRobot representative or administrator for information on enabling this feature. > > Feature flag: Enable Notebooks Custom Environments If the code does not run properly, you will receive the Testing custom metric code failed warning after testing completes. Click Open custom metric job to access the job and check the logs to troubleshoot the issue: To troubleshoot a custom metric's code, navigate to the job's Runs tab, containing a log of the failed test. In the failed run, click View log. ### Add hosted custom metrics from the gallery The custom metrics gallery provides a centralized library containing pre-made, reusable, and shareable code implementing a variety of hosted custom metrics for predictive and generative models. These metrics are recorded on the configurable Custom Metric Summary dashboard, alongside any external custom metrics. From this dashboard, you can monitor, visualize, and export each metric's change over time. This feature allows you to implement your organization's specialized metrics, expanding on the insights provided by DataRobot's built-in service health, data drift, and accuracy metrics. To add a pre-made custom metric to a deployment: 1. In theAdd custom metricpanel, select a custom metric template applicable to your use case. DataRobot provides three different categories ofMetric type: Binary ClassificationRegressionLLM (Generative)Agentic workflow / LLMCustom metric templateDescriptionRecall for top x%Measures model performance limited to a certain top fraction of the sorted predicted probabilities. Recall is a measure of a model's performance that calculates the proportion of actual positives that are correctly identified by the model.Precision for top x%Measures model performance limited to a certain top fraction of the sorted predicted probabilities. Precision is a measure of a model's performance that calculates the proportion of correctly predicted positive observations from the total predicted positive.F1 for top x%Measures model performance limited to a certain top fraction of the sorted predicted probabilities. F1 score is a measure of a model's performance which considers both precision and recall.AUC (Area Under the ROC Curve) for top x%Measures model performance limited to a certain top fraction of the sorted predicted probabilities.Custom metric templateDescriptionMean Squared Logarithmic Error (MSLE)Calculates the mean of the squared differences between logarithms of the predicted and actual values. It is a loss function used in regression problems when the target values are expected to have exponential growth, like population counts, average sales of a commodity over a time period, and so on.Median Absolute Error (MedAE)Calculates the median of the absolute differences between the target and the predicted values. It is a robust metric used in regression problems to measure the accuracy of predictions.Custom metric templateDescriptionCompletion Reading TimeEstimates the average time it takes a person to read text generated by the LLM.Completion Tokens MeanCalculates the mean number of tokens in completions for the time period requested. The cl100k_base encoding used only supports OpenAI models: gpt-4, gpt-3.5-turbo, and text-embedding-ada-002. If you use a different model, change the encoding.Cosine Similarity AverageCalculates the mean cosine similarity between each prompt vector and corresponding context vectors.Cosine Similarity MaximumCalculates the maximum cosine similarity between each prompt vector and corresponding context vectors.Cosine Similarity MinimumCalculates the minimum cosine similarity between each prompt vector and corresponding context vectors.CostEstimates the financial cost of using the LLM by calculating the number of tokens in the input, output, and retrieved text, and then applying token pricing. The cl100k_base encoding used only supports OpenAI models: gpt-4, gpt-3.5-turbo, and text-embedding-ada-002. If you use a different model, change the encoding.Dale Chall ReadabilityMeasures the U.S. grade level required to understand a text based on the percentage of difficult words and average sentence length.Euclidean AverageCalculates the mean Euclidean distance between each prompt vector and corresponding context vectors.Euclidean MaximumCalculates the maximum Euclidean distance between each prompt vector and corresponding context vectors.Euclidean MinimumCalculates the minimum Euclidean distance between each prompt vector and corresponding context vectors.Flesch Reading EaseMeasures the readability of text based on the average sentence length and average number of syllables per word.Prompt Injection [sidecar metric]Detects input manipulations, such as overwriting or altering system prompts, that are intended to modify the model's output. This metric requires an additional deployment of the Prompt Injection Classifierglobal model.Prompt Tokens MeanCalculates the mean number of tokens in prompts for the time period requested. The cl100k_base encoding used only supports OpenAI models: gpt-4, gpt-3.5-turbo, and text-embedding-ada-002. If you use a different model, change the encoding.Sentence CountCalculates the total number of sentences in user prompts and text generated by the LLM.SentimentClassifies text sentiment as positive or negativeSentiment [sidecar metric]Classifies text sentiment as positive or negative using a pre-trained sentiment classification model. This metric requires an additional deployment of the Sentiment Classifierglobal model.Syllable CountCalculates the total number of syllables in the words in user prompts and text generated by the LLM.Tokens MeanCalculates the mean of tokens in prompts and completions. The cl100k_base encoding used only supports OpenAI models: gpt-4, gpt-3.5-turbo, and text-embedding-ada-002. If you use a different model, change the encoding.Toxicity [sidecar metric]Measures the toxicity of text using a pre-trained hate speech classification model to safeguard against harmful content. This metric requires an additional deployment of the Toxicity Classifierglobal model.Word CountCalculates the total number of words in user prompts and text generated by the LLM.Japanese text metrics[JP] Character CountCalculates the total number of characters generated while working with the LLM.[JP] PII occurrence countCalculates the total number of PII occurrences while working with the LLM.Custom metric templateDescriptionAgentic completion tokensCalculates the total completion tokens of agent-based LLM calls.Agentic costCalculates the total cost of agent-based LLM calls. Requires that each LLM span reports token usage so the metric can compute cost from the trace.Agentic prompt tokensCalculates the total prompt tokens of agent-based LLM calls. 2. After you select a metric from the list, in theCustom metric configurationsidebar, configure a metric calculation schedule or run the metric calculation immediately, and, optionally, set a metric baseline value. Sidecar metricsIf you selected a[sidecar metric], when you open theAssembletab, navigate to theRuntime Parameterssection to set theSIDECAR_DEPLOYMENT_ID, associating the sidecar metric with the connected deployment required to calculate that metric. If you haven't deployed a model to calculate the metric, you can find pre-defined models for these metrics asglobal models. 3. ClickCreate metric. The new metric appears on theCustom metricsdashboard. 4. After you create a custom metric, you can view the custom job associated with the metric. This job runs on the metric's defined schedule, in the same way ashosted custom metrics(those not from the gallery). To access and manage the associated custom job, click theActions menuand then clickOpen Custom Job: ## Upload data to custom metrics After you create a custom metric, you can provide data to calculate the metric: 1. On theCustom metricstab, locate the custom metric for which you want to upload data and click theUpload Dataicon. 2. In theUpload datadialog box, select an upload method and clickNext: Upload methodDescriptionUse Data RegistryIn theSelect a datasetpanel, upload a dataset or click a dataset from the list, and then clickConfirm. The Data Registry includes datasets from theData explorationtab.Use APIIn theUse API Clientpanel, clickCopy to clipboard, and then modify and use the API snippet to upload a dataset. You can upload up to 10,000 values in one API call. 3. In theSelect dataset columnsdialog box, configure the following: FieldDescriptionTimestamp column(Required) The column in the dataset containing a timestamp.Value column(Required) The column in the dataset containing the values used for custom metric calculation.Association IDThe row containing the association ID required by the custom metric to link predicted values to actuals.Date formatThe date format used by the timestamp column. 4. ClickUpload data. ### Report custom metrics via chat requests For DataRobot-deployed text generation and agentic workflow custom models that implement the `chat()` hook, custom metric values can be reported directly in chat completion requests using the `extra_body` field. This allows reporting custom metrics at the same time as making chat requests, without needing to upload data separately. > [!TIP] Manual chat request construction > The OpenAI client converts the `extra_body` parameter contents to top-level fields in the JSON payload of the chat `POST` request. When manually constructing a chat payload, without the OpenAI client, include `"datarobot_metrics": {...}` in the top level of the payload. To report custom metrics via chat requests: 1. Ensure the deployment has an association ID column defined and moderation configured. These are required for custom metrics to be processed. 2. Define custom metrics on theCustom Metricstab as described inAdd external custom metrics. 3. When making a chat completion request using the OpenAI client, includedatarobot_metricsin theextra_bodyfield with the metric names and values to report: ``` from openai import OpenAI openai_client = OpenAI( base_url="https:///api/v2/deployments/{deployment_id}/", api_key="", ) extra_body = { # These values pass through to the LLM "llm_id": "azure-gpt-6", # If set here, replaces the auto-generated association ID "datarobot_association_id": "my_association_id_0001", # DataRobot captures these for custom metrics "datarobot_metrics": { "field1": 24, "field2": 25 } } completion = openai_client.chat.completions.create( model="datarobot-deployed-llm", messages=[ {"role": "system", "content": "Explain your thoughts using at least 100 words."}, {"role": "user", "content": "What would it take to colonize Mars?"}, ], max_tokens=512, extra_body=extra_body ) print(completion.choices[0].message.content) ``` > [!NOTE] Custom metric requirements > A matching custom metric for each name in > datarobot_metrics > must already be defined for the deployment. > Custom metric values reported this way must be numeric. > The deployed custom model must have an association ID column defined and moderation configured for the metrics to be processed. For more information about using `extra_body` with chat requests, including how to specify association IDs, see the [chat()hook documentation](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#association-id). ## Manage custom metrics On the Custom metrics dashboard, after you've added your custom metrics, you can edit or delete them: On the Custom metrics tab, locate the custom metric you want to manage, and then click the Actions menu: - To edit a metric, clickEdit, update any configurable settings, and then clickUpdate custom metric. - To delete a metric, clickDelete. ## Configure the custom metric dashboard display settings Configure the following settings to specify the custom metric calculations you want to view on the dashboard: > [!TIP] Custom metrics for evaluation and moderation require an association ID > For the metrics added when you configure evaluations and moderations, to view data on the Custom metrics tab, ensure that you set an association ID and enable prediction storage before you start making predictions through the deployed LLM.If you don't set an association ID and provide association IDs alongside the LLM's predictions, the metrics for the moderations won't be calculated on the [Custom metrics](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-custom-metrics.html) tab.After you define the association ID, you can enable automatic association ID generation to ensure these metrics appear on the Custom metrics tab. You can enable this setting [during](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-deploy-models.html#custom-metrics) or [after](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-custom-metrics-settings.html) deployment. | | Setting | Description | | --- | --- | --- | | (1) | Model | Select the deployment's model, current or previous, to show custom metrics for. | | (2) | Range (UTC) / Date Slider | Select the start and end dates of the period from which you want to view custom metrics. | | (3) | Resolution | Select the granularity of the date slider. Select from hourly, daily, weekly, and monthly granularity based on the time range selected. If the time range is longer than 7 days, hourly granularity is not available. | | (4) | Segment attribute / Segment value | Sets the individual attribute and value to filter the data drift visualizations for segment analysis. | | (5) | Refresh | Refresh the custom metric dashboard. | | (6) | Reset | Reset the custom metric dashboard's display settings to the default. | ### Arrange or hide metrics on the dashboard To arrange or hide metrics on the Custom metrics summary dashboard, locate the custom metric you want to move or hide: - To move a metric, click the grid iconon the left side of the metric tile and then drag the metric to a new location. - To hide a metric chart, clear the checkbox next to the metric name. ## Explore deployment data tracing Tracing for custom and external model deployments is on the dedicated Tracing tab. For search, filter, and span details, see [Tracing](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-tracing.html). --- # Data exploration URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-data-exploration.html > Export deployment data and review data quality for generative and agentic deployments. On a deployment's Monitoring > Data exploration tab, you can interact with a deployment's stored data to gain insight into model or agent performance. You can also download deployment data to use in custom metric calculations. The Data exploration summary includes the following functionality, depending on the deployment type: | Functionality | Description | | --- | --- | | Data export | For all deployments, download a deployment's stored data including training data, prediction data, actuals, and custom metric data. | | Data quality | For generative AI and agentic workflow deployments, assess the quality of a generative AI model's responses based on user feedback and custom metrics. | > [!NOTE] Data requirements > To use the Data exploration tab, the deployment must store prediction data. Ensure that you [enable prediction row storage](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-data-exploration-settings.html) in the data exploration (or challenger) settings. The Data exploration tab doesn't store or export Prediction Explanations, even if they are requested with the predictions. ## Configure data exploration range In the deployment from which you want to export stored training data, prediction data, or actuals, click the Monitoring > Data exploration tab and configure the following settings to specify the stored training data, prediction data, or actuals you want to export: | | Setting | Description | | --- | --- | --- | | (1) | Model | Select the deployment's model, current or previous, to export prediction data for. | | (2) | Range (UTC) | Select the start and end dates of the period you want to export prediction data from. | | (3) | Resolution | Select the granularity of the date slider. Select from hourly, daily, weekly, and monthly granularity based on the time range selected. If the time range is longer than 7 days, hourly granularity is not available. | | (4) | Refresh | Refresh the data exploration tab's data. | | (5) | Reset | Reset the data exploration settings to the default. | ## Export deployment data On the Data exploration summary page (or the Data export tab of the Data exploration summary), you can download a deployment's stored data. This can include training data, prediction data, actuals, and custom metric data. Use the exported data to compute and monitor custom business or performance metrics on the [Custom metrics](https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-custom-metrics.html) tab or outside of DataRobot. To export deployment data for custom metrics, verify that the deployment stores prediction data, generate data for a specified time range, and then view or download that data. ### Export a deployment's production data To access deployment data export for prediction data, actuals, or custom metric data, on the Data exploration summary page, locate the Production data panel. On the Production data panel, in the Generate button, click the down arrow and select one of the data generation options. The availability of the following options depends on the data stored in the deployment for the model and time range selected. | Option | Description | | --- | --- | | All production data | For generative AI deployments, generate all available production data (predictions, actuals, custom metrics) for the specified model and time range. | | Custom metrics | For generative AI deployments, generate available custom metric data for the specified model and time range. | > [!NOTE] Premium > Custom metric data export is off by default. Contact your DataRobot representative or administrator for information on enabling this feature. Production data appears in the table below the panels. You can identify the data type in the Exported data column. **Prediction data and actuals considerations** When generating prediction data or actuals, consider the following: - When generating prediction data, you can export up to 200,000 rows per export. If the time range you set exceeds 200,000 rows of prediction data, decrease the range. - In the Data Registry, you can have up to 100 prediction export items. If generating prediction data for export would cause the number of prediction export items in the Data Registry to exceed that limit, delete old prediction export Data Registry items. - When generating actuals, you can export up to 1,000,000 rows per export. If the time range you set exceeds 1,000,000 rows of actuals, decrease the time range. - In the Data Registry, you can have up to 100 actuals export items. If generating actuals data for export would cause the number of actuals export items in the Data Registry to exceed that limit, delete old actuals export Data Registry items. - Up to 10,000,000 actuals are stored for a deployment; therefore, exporting old actuals can result in an error if no actuals are currently stored for that time period. ### Export a deployment's training data To access deployment data export for training data, on the Data exploration summary page, locate the Training data panel and click Generate training data to generate data for the specified model and time range: Options for interacting with the training data appear in the Training data panel. Click the down arrow to choose between Open training data and Download training data: ### Review and download data After the production or training data are generated, you can view or download the data. Production data appears in the table below the panels, where you can identify the data type in the Exported data column. Training data appears in the Training data panel. | Option | Description | | --- | --- | | | Open the exported data in the Data Registry. | | | Download the exported data. | > [!NOTE] Export to notebook > You can also click Export to notebook to open a [DataRobot notebook](https://docs.datarobot.com/en/docs/workbench/wb-notebook/index.html) with cells for exporting training data, prediction data, and actuals. ## Explore deployment data tracing Tracing for custom model deployments is on the dedicated Tracing tab. For search, filter, and span details, see [Tracing](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-tracing.html). ## Explore deployment data quality > [!NOTE] Premium > Data quality is a premium feature. Contact your DataRobot representative or administrator for information on enabling this feature. On the Data exploration tab of a generative AI deployment, click Data quality to explore prompts and responses alongside user ratings and custom metrics, if implemented, providing insight into the quality of the generative AI model. Prompts, responses, and any available metrics are matched by association ID: To configure the rows displayed in the data quality table, click Settings to open the Column management panel, where columns can be selected, hidden, or rearranged. > [!NOTE] Prompt and response matching > To use the data quality table, [define an association ID](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-custom-metrics-settings.html) to match prompts with responses in the same row. Tracing analysis is only available for prompts and responses matched in the same row by association ID; aggregate custom metric data is excluded. Locate specific rows in the Data quality table by searching. Click Search by and select Prompt values, Response, or Actual values. Then, click Search: In addition, you can filter the Data quality table on a single custom metric value from one of the [custom metrics created for the current deployment](https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-custom-metrics.html). To filter the table, click Filter, select a Metric, enter a Metric value, and then click Apply filters: > [!TIP] Sorting the data quality table > You can sort the Data quality table by clicking the column for Prompt created at, Association ID, or any [custom metrics created for the current deployment](https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-custom-metrics.html). Click the open icon to expand the details panel. The display shows a row's full Prompt and the Response matched with the prompt by association ID. It also shows custom metric values and citations (if configured): To export columns for external use, click Export all in selected range to export every row in the time range defined at the top of the Data quality view, or click Export selected rows if you've selected one or more rows in the table: --- # Moderation events URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-moderation.html > Review evaluation and moderation events for guarded LLM and agentic deployments. For a deployed text generation or agentic workflow model with evaluation and moderation configured, on the deployment's Activity log > Moderation tab, view a history of evaluation and moderation-related events for the deployment. These events can help diagnose issues with a deployment's configured evaluations and moderations. Use this tab together with the [Custom metrics](https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-custom-metrics.html) tab and evaluator logs to debug why a request was blocked or why a guard (including a NeMo Evaluator guard) failed. To view the moderation events log, navigate to the Activity log > Moderation tab. The most recent events appear at the top of the list. Each event shows the time it occurred, a description, and an icon indicating its status. > [!NOTE] Status for moderation events > All moderation events have a failure event type. Moderation events represent deployment actions and can help you review the status of your deployment and the health of the management agent. Currently, the following events can appear as moderation events: | Event type | Description | | --- | --- | | Moderation metric creation error | Reports errors encountered while creating a custom metric definition. For example: Failed to create custom metric. Maximum number of custom metrics reached.Failed to create custom metric for another reason (with details). | | Moderation metric reporting error | Reports errors encountered while reporting a custom metric value. For example:Failed to upload custom metrics. | | Moderation model scoring error | Reports errors encountered during the scoring phase of the model. For example: Failed to execute user score function.Cannot execute postscore guards. | | Moderation model configuration error | Reports errors encountered while configuring moderation for the model. | | Moderation model runtime error | Reports errors encountered while running moderation for the model. For example:Model Guard timeout.Model Guard predictions failed.Faithfulness calculations failed.ROUGE-1 guard configured without citation columns.NeMo guard calculation failed. | --- # OpenTelemetry logs URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-otel-logs.html > View OpenTelemetry log events for deployments. A deployment's Logs tab receives logs from models and agentic workflows in the OpenTelemetry (OTel) standard format, centralizing the relevant logging information for deeper analysis, troubleshooting, and understanding of application performance and errors. Additionally, you can filter and [view span-specific logs](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-tracing.html#filter-tracing-logs) on the Tracing tab. The collected logs provide time-period filtering capabilities and the OTel logs API is available to programmatically export logs with similar filtering capabilities. Because the logs are OTel-compliant, they're standardized for export to third-party observability tools like Datadog. > [!NOTE] Access and retention > OTel logs are available for all deployment and target types. Only users with Owner and User roles on a deployment can view these logs. Logs data is stored for a retention period of 30 days, after which it is automatically deleted. To access the logs for a deployment, on the Deployed workloads tab, locate and click the deployment, click the Activity log tab, and then click Logs. The logging levels available are `INFO`, `DEBUG`, `WARN`, `CRITICAL` and `ERROR`. | Control | Description | | --- | --- | | Range UTC | Select the logging date range Last 15 min, Last hour, Last day, or Custom range. | | Level | Select the logging level to view: Debug, Info, Warning, Error, or Critical. | | Refresh | Refresh the contents of the Logs tab to load new logs. | | Copy logs | Copy the contents of the current Logs tab view. | | Search | Search the text contents of the logs tab. | ## Export OTel logs The code example below uses the OTel logs API to get the OpenTelemetry-compatible logs for a deployment, print a preview and the number of `ERROR` logs, and then write logs to an output file. Before running the code, configure the `entity_id` variable with your deployment ID, replacing `` with the deployment ID from the deployment Overview tab or URL. In addition, you can modify the `export_logs_to_json` function to match your target observability service's expected format. > [!TIP] DataRobot Python Client version > The following script requires `datarobot` version `3.11.0` or higher installed to support OpenTelemetry logging submodules. | Export OTel logs to JSON | | --- | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | --- # OpenTelemetry metrics URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-otel-metrics.html > Visualize OpenTelemetry metrics alongside DataRobot native metrics. The OTel metrics tab provides comprehensive OpenTelemetry (OTel) metrics monitoring capabilities for your deployment, enabling centralized observability by visualizing external metrics from your applications and agentic workflows alongside DataRobot's native metrics, all in OTel-compliant format for export to third-party observability tools. > [!NOTE] Access and retention > OTel metrics are available for all deployment and target types. Only users with Owner and User roles on a deployment can view and configure these metrics. Metrics data is stored for a retention period of 30 days, after which it is automatically deleted. To access OTel metrics for a deployment, on the Deployed workloads tab, locate and click the deployment, click the Monitoring tab, and then click OTel metrics. ## Select OTel metrics The OTel metrics tab can display up to 50 metrics. A customization dialog box displays all available metrics and supports searching by metric name. From this list, select the most important metrics for the dashboard. To select an OTel metric: 1. On theMonitoring > OTel metricstab, click+ Select OTel metrics(orCustomize tilesif metrics are already added). No metrics on the dashboardMetrics on the dashboard 2. In theCustomize tilesdialog box, in theMetricslist, do any of the following: 3. To add more metrics, click+ Add another metricand repeat the step above. Select up to 50 OTel metrics to display for the deployment. Remove and reorder metricsClick the up arrowand down arrowicons to reorder the metrics on the dashboard. Click the remove iconto remove a metric from the dashboard. 4. ClickSaveto update theOTel metricsdashboard configuration and review the metric visualizations. Supported time resolution settingsThe supported time resolution settings for OTel metric visualization are minute, hour, or day. ## Edit OTel metrics The OTel metrics tab enables the customization of how individual metrics are displayed and aggregated on your monitoring dashboard. After selecting metrics to monitor, fine-tune their presentation by editing display names, choosing aggregation methods, and toggling between trend charts and summary values. To edit an OTel metric: 1. On theMonitoring > OTel metricstab, with metrics already added, click the edit iconin the upper-right corner of a tile. 2. In theEdit metricdialog box, configure the following settings: Counter/gaugeHistogram SettingDescriptionDisplay nameDefines the name displayed on the dashboard tile and chart. The original name is preserved as theKey name, or the name defined in the monitored system.Aggregation typeSets the mathematical method for summarizing OTel metric data points across a given time period. The default aggregation depends on the metric type:Histogram(default for histograms) orAverage(default for counters/gauges). Available aggregation methods:Histogram: Displays the distribution of values as a histogram.Percentile: Calculates percentile values from the metric data.Average: The average of all reported values for the metric within the time period.Sum: The total of all reported values for the metric within the time period.Minimum: The lowest value reported for the metric within the time period.Maximum: The highest value reported for the metric within the time period.Percentile(Available whenHistogramorPercentileaggregation type is selected) Specifies the percentile value to calculate, ranging from 0 to 1. The default value is 0.5 (50th percentile).Show metric values over timeToggle the display of an OTel metric between an "over time" chart for trend analysis and a single, summarized numeric value for an at-a-glance summary. When this setting is disabled, the chart cannot be displayed. 3. To save the new metric settings, clickEdit. --- # Deployment overview URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-overview.html > Review deployment details, lineage, tags, runtime parameters, and evaluation and moderation when guardrails are configured. When you select a deployment from the Deployed workloads dashboard, DataRobot opens the Overview page for that deployment. The Overview page provides a model- and environment-specific summary that describes the deployment, including the information you supplied when creating the deployment and any model replacement activity. ## Details The Details section of the Overview tab lists an array of information about the deployment, including the deployment's model and environment-specific information. At the top of the Overview page, you can view the deployment name and description; click the edit icon to update this information. > [!NOTE] Note > The information included in this list depends on the registered custom model, target type (for example, text generation or agentic workflow), and environment. | Field | Description | | --- | --- | | Deployment ID | The ID number of the current deployment. Click the copy icon to save it to your clipboard. | | Predictions | A visual representation of the relative prediction frequency, per day, over the past week. | | Importance | The importance level assigned during deployment creation. Click the edit icon to update the deployment importance. | | Approval status | The deployment's approval policy status for governance purposes. | | Prediction environment | The environment on which the deployed model makes predictions. | | Build environment | The build environment used by the deployment's current model (e.g., DataRobot, Python, R, or Java). | | Flags | Indicators providing a variety of deployment metadata, including deployment status—Active, Inactive, Errored, Warning, Launching—and deployment type (for example, LLM, text generation, or agentic workflow). | | Created by | The name of the user who created the model. | | Last prediction | The number of days since the last prediction. Hover over the field to see the full date and time. | | Custom model information | | | Custom model | The name and version of the custom model registered and deployed from the workshop. | | Custom environment | The name and version of the custom model environment on which the registered custom model runs. | | Resource bundle | Preview feature. The CPU or GPU bundle selected for the custom model in the resource settings. | | Resource replicas | Preview feature. The number of replicas defined for the custom model in the resource settings. | | Generative model information | | | Target | The feature name of the target column used by the deployment's current generative model. This feature is the generative model's answer to a prompt; for example, resultText, answer, completion, etc. | | Prompt column name | The feature name of the prompt column used by the deployment's current generative model. This feature is the prompt the generative model responds to; for example, promptText, question, prompt, etc. | ## Lineage The Lineage section provides visibility into the assets and relationships associated with a deployment. This section helps understand the complete context of a deployment, including the models, datasets, experiments, and other MLOps assets connected to it. The Lineage section contains two tabs: - Graph: An interactive, end-to-end visualization of the relationships and dependencies between MLOps assets. This DAG (Directed Acyclic Graph) view helps audit complex workflows, track asset lifecycles, and manage components of agentic and generative AI systems. The graph displays nodes (assets) and edges (relationships/connections), enabling the exploration of connections and navigation through the asset ecosystem. - List: A list of the assets associated with a deployment, including registered models, model versions, experiments, datasets, and other related items. Each item displays its name, ID, creator, and creation date. ClickViewto open any related item, or use the list to quickly identify and access connected assets. For custom model deployments (including text generation, agentic workflows, vector databases, and MCP integrations), the list emphasizes registered models, custom model versions, and related training data. **Graph:** The Lineage section in the Overview tab includes a Graph view that provides an end-to-end visualization of the relationships and dependencies between your MLOps assets. This feature is essential for auditing complex workflows, tracking asset lifecycles, and managing the various components of agentic and generative AI systems. The Graph view serves as a central hub for reviewing your systems. The lineage is presented as a Directed Acyclic Graph (DAG) consisting of nodes (assets) and edges (relationships). When reviewing nodes, the asset you are currently viewing is distinguished by a purple outline. Nodes display key information such as ID, name (or version number), creator, and the last modification information (user and date). When reviewing edges, solid lines represent concrete, persistent relationships within the platform, such as a registered model used to create a deployment.Dashed lines indicate relationships inferred from runtime parameters. These are considered less reliable as they may change if a user modifies the underlying code or parameters. Arrows generally flow from the "ancestor" or container to the "descendant" or content (e.g., Registered model version to Deployment). > [!NOTE] Inaccessible assets > If an asset exists but you do not have permission to view it, the node only displays the asset ID and is marked with an Asset restricted notice. The view is highly interactive, allowing for deep exploration of your asset ecosystem. To interact with the graph area, use the following controls: [https://docs.datarobot.com/en/docs/images/lineage-graph-view-controls.png](https://docs.datarobot.com/en/docs/images/lineage-graph-view-controls.png) Control Description Legend View the legend defining how lines correspond to edges. and Control the magnification level of the graph view. Reset the magnification level and center the graph view on the focused node. Open a fullscreen view of the related items lineage graph. and In fullscreen view, navigate the history of selected nodes (assets/nodes viewed ). Graph area navigation To navigate the graph, click and drag the graph area. To control the zoom level, scroll up and down. To interact with the related item nodes, use the following controls when they appear: [https://docs.datarobot.com/en/docs/images/lineage-graph-node-controls.png](https://docs.datarobot.com/en/docs/images/lineage-graph-node-controls.png) Control Description Navigate to the asset in a new tab. Open a fullscreen view of the related items lineage graph centered on the selected asset node. Copy the asset's associated ID. > [!NOTE] One-to-many list view > If an asset is used by many other assets (e.g., one dataset version used for many projects), in the fullscreen view, the graph shows a preview of the 5 most recent items. Additional assets are viewable in a paginated and searchable list. If you don't have permission to view the ancestor of a paginated group, you can only view the 5 most recent items, without the option to change pages or search. > > [https://docs.datarobot.com/en/docs/images/one-to-many-asset-list.png](https://docs.datarobot.com/en/docs/images/one-to-many-asset-list.png) **List:** The Lineage section in the Overview tab also includes a List view. On the List tab, click Show more to reveal all related items. Each item in the list displays its name, ID, the user who created it, and the date it was created. Click View to open the related item. For custom model deployments (including text generation, agentic workflows, vector databases, and MCP integrations), the list emphasizes registered models, custom model versions, and related training data. [https://docs.datarobot.com/en/docs/images/nxt-overview-tab-items.png](https://docs.datarobot.com/en/docs/images/nxt-overview-tab-items.png) Field Description Registered model The name and ID of the registered model associated with the deployment. Click to open the registered model in Registry. Registered model version The name and ID of the registered model version associated with the deployment. Click to open the registered model version in Registry. Custom model information Custom model The name, version, and ID of the custom model associated with the deployment. Click to open the workshop to the Assemble tab for the custom model. Custom model version The version and ID of the custom model version associated with the deployment. Click to open the workshop to the Versions tab for the custom model. Training dataset The filename and ID of the training dataset used to create the currently deployed custom model. > [!NOTE] Inaccessible related items > If you don't have access to a related item, a lock icon appears at the end of the item's row. ## Evaluation and moderation > [!NOTE] Availability information > Evaluation and moderation guardrails are a premium feature. Contact your DataRobot representative or administrator for information on enabling this feature. > > Feature flag: Enable Moderation Guardrails ( Premium), Enable Global Models in the Model Registry ( Premium), Enable Additional Custom Model Output in Prediction Responses When a text generation or agentic workflow model with guardrails is registered and deployed, you can view the Evaluation and moderation section on the deployment's Overview tab: ## Tags In the Tags section, click + Add new and enter a Name and a Value for each key-value pair you want to tag the deployment with. Deployment tags can help you categorize and search for deployments in the [dashboard](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-overview/nxt-dashboard.html). ## Runtime parameters > [!NOTE] Preview > The ability to edit custom model runtime parameters on a deployment is on by default. > > Feature flag: Enable Editing Custom Model Runtime-Parameters on Deployments On a custom model deployment's Overview tab, you can access the Runtime parameters section. Runtime parameters are injected into containers as standard environment variables, without requiring prefixes or JSON parsing for simple types, meaning that developers can retrieve parameters using standard Python methods (e.g., `os.getenv`) rather than relying on the `datarobot-drum` library and its associated dependencies. Parameters [created via theWorkshopUI](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-create-custom-model.html#define-runtime-parameters) persist and merge when you upload new code versions, ensuring a seamless development flow. From this section, manage these parameters on an inactive deployment. To do this, first make sure the deployment is inactive, then, click Edit: In the Runtime parameters table, edit the Value. To discard an individual change, click Revert changes. If you edit any of the runtime parameters, to save your changes, click Save. For more information on how to define runtime parameters and use them in custom model code, see the [Define custom model runtime parameters](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-runtime-parameters.html) documentation. ## Setup checklist and approval When governance management functionality is enabled for your organization, the Setup checklist panel appears on the deployment Overview. This checklist includes the settings required by your administrator, any additional guidance they provided when configuring the checklist, and the status of the checklist setting: Not enabled, Partially enabled, or Enabled. Complete this checklist before requesting deployment approval from an administrator. Click a tile in the checklist to open the relevant [deployment setting](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/index.html) page. > [!NOTE] Default setup checklist > By default, the Setup checklist displays all available setting groups for the current deployment; however, the default list is customized by the organization-level administrator. If a deployment is subject to a configured approval policy, the deployment is created in a Draft state, with an Approval status of Needs approval, as shown above. After you complete the approval checklist, you can click Request approval in the Draft deployment notice on the deployment Overview page. When you click Request approval, the Submit request for approval dialog box appears, where you can enter Additional comments for the approver. Then, click Request approval to complete your request. After approval, the deployment is automatically moved out of the draft state and activated. On the Deployed workloads tab, draft deployments awaiting approval are shown with a Draft tag and an Inactive tag: > [!NOTE] Draft deployment limitations > With a draft deployment, you can't make predictions, upload actuals or custom metric data, or create scheduled jobs. --- # Prediction API snippets URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-pred-api-snippets.html > Use Prediction API snippets for real-time scoring and chat completions from generative custom model deployments. DataRobot provides sample Python code containing the commands and identifiers required to submit a CSV or JSON file for scoring. You can use this code with the [DataRobot Prediction API](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html). To use the Prediction API Scripting Code, open the deployment you want to make predictions through and click Predictions > Prediction API. On the Prediction API Scripting Code page, configure Real-time scripts for chat completions and predictions from text generation, agentic workflow, VDB, and MCP custom model deployments. Follow the sample provided and make the necessary changes when you want to integrate the model, via API, into your production application. > [!NOTE] Dormant prediction servers > Prediction servers become dormant after a prolonged period of inactivity. If you see the Prediction server is dormant alert, contact support@datarobot.com for reactivation. ### Real-time prediction snippet settings To find and access the real-time prediction script required for your use case, configure the following settings: | | Content | Description | | --- | --- | --- | | (1) | Prediction type | Determines the prediction method used. Select Real time. | | (2) | Language | Determines the language of the real-time prediction script generated. Select a format:Python: An example real-time prediction script using DataRobot's Python package.cURL: A script using cURL, a command-line tool for transferring data using various network protocols, available by default in most Linux distributions and macOS. | | (3) | Show secrets | Displays any secrets hidden by ***** in the code snippet. Revealing the secrets in a code snippet can provide a convenient way to retrieve your API key or datarobot-key; however, these secrets are hidden by default for security reasons, so ensure that you handle them carefully. | | (4) | Copy script to clipboard | Copies the entire code snippet to your clipboard. | | (5) | Open in a codespace | Open the snippet in a codespace to edit it, share with others, and incorporate additional files. | | (6) | Code overview screen | Displays the example code you can download and run on your local machine. Edit this code snippet to fit your needs. | ### Open snippets in a codespace You can open a Prediction API code snippet in a [codespace](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html) to edit the snippet directly, share it with other users, and incorporate additional files. To open a Prediction API snippet, click Open in a codespace. DataRobot generates a codespace instance and populates the snippet inside as a python file. In the codespace, you can upload files and edit the snippet as needed. For example, you may want to add CLI arguments in order to execute the snippet. The codespace allows for full access to file storage. You can use the Upload button to add additional datasets for scoring, and have the prediction output ( `output.json`, `output.csv`, etc.) return to the codespace file directory after executing the snippet. This example uploads `10k_diabetes_small.csv` to the codespace as an input file. To add CLI arguments to the snippet, click Add CLI arguments. This example references `10k_diabetes_small.csv` as the input file for scoring, and names the output file `output.csv`. The snippet is now configured to run and return predictions. When you have finished working in the codespace, click Exit and save codespace. Codespaces belong to [Use Cases](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/usecases/usecase-overview.html), so you must specify an existing Use Case or create a new one to save the codespace to. When a Use Case has been selected, click Exit and save codespace again. Your snippet is now saved in a codespace as part of a Use Case. --- # Deployment reports URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-reports.html > Generate on-demand or scheduled deployment reports in Console. Ongoing monitoring reports are a critical step in the deployment and governance process. DataRobot allows you to download deployment reports from MLOps, compiling deployment status, charts, and overall quality into a shareable report. Deployment reports are compatible with all deployment types. ## Generate a deployment report To generate a report for a deployment, select it from the Deployments inventory, navigate to the Monitoring > Reports tab and click Generate report now: In the Settings for report generation panel, select the Model, Date range, and Date resolution, then click Generate report: When the report generation is finished, click the view icon to open the report in your browser or the download icon to open it locally: ## Schedule deployment reports In addition to manual creation, DataRobot allows you to manage a schedule to generate deployment reports automatically. To schedule report generation for a deployment, select the deployment from the Deployments inventory and navigate to the Monitoring > Reports tab. On the Deployment reports page, click + Create new report schedule: In the report panel, configure the Report Schedule (UTC) and Report Contents and Recipients: > [!TIP] Advanced scheduling > For more granular scheduling controls, you can click Use advanced schedule. After defining the report schedule and recipients, click Save report schedule. The reports automatically generate at the configured dates and times. The generated report appears on the Monitoring > Reports tab. You can edit or delete the report schedule from the list: --- # Resource monitoring URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-resource-monitoring.html > Monitor CPU, memory, and replica utilization for serverless custom model deployments. The Monitoring > Resource monitoring tab provides visibility into resource utilization metrics for deployed custom models and agentic workflows, helping you monitor performance, identify bottlenecks, and understand auto-scaling behavior. Use this tab to evaluate resource usage, navigate tradeoffs between speed and cost, and ensure your deployments efficiently utilize available hardware resources. To access Resource monitoring, select a deployment from the Deployments inventory and then click Monitoring > Resource monitoring. The tab displays summary tiles showing aggregated and current values for key metrics, along with interactive charts that visualize resource utilization over time. ## Resource utilization summary tiles The Resource monitoring tab displays summary tiles at the top of the page, showing both the aggregated value over the selected timespan (the primary value) and the current Live value at the time of the request. The aggregated value represents the average value over the selected timespan. Clicking on a metric tile updates the chart below to display that metric over time. The following metrics are displayed as summary tiles: | Metric | Description | | --- | --- | | Replicas | The number of active compute instances (replicas) out of the maximum available for the deployment. For text generation, agentic workflow, VDB, and MCP custom model deployments, this counts custom model pods. | | CPU utilization | The percentage of CPU cores being used across all compute instances for the deployment. | | Memory usage | The amount of memory (in bytes or appropriate units) being used across all compute instances for the deployment. | ## Resource utilization charts The Resource monitoring tab displays interactive charts that visualize resource utilization metrics over time, helping you identify patterns and understand resource consumption trends. The chart displays the selected metric over time, with the following elements: | | Chart element | Description | | --- | --- | --- | | (1) | Time (X-axis) | Displays the time represented by each data point, based on the selected resolution (1 minute, 5 minutes, hourly, or daily). | | (2) | Metric value (Y-axis) | Displays the value (cardinality for Replicas and average for all other metrics) of the selected metric (Replicas, CPU utilization, or Memory usage) for each time period. | | (3) | Containers | For deployments with multiple compute instances, you can filter resource utilization metrics by specific compute instance. | To view additional information on the chart, hover over a data point to see the time range and metric value: You can configure the Resource monitoring dashboard to focus on specific time frames and metrics. The following controls are available: | Control | Description | | --- | --- | | Range (UTC) | Sets the date range displayed for the deployment date slider. You can also drag the date slider to set the range. The range selector only allows you to select dates and times between the start date of the deployment's current version of a model and the current date. | | Resolution | Sets the time granularity of the deployment date slider. The following resolution settings are available, based on the selected range: Hourly: If the range is less than 7 days.Daily: If the range is between 1-60 days (inclusive).Weekly: If the range is between 1-52 weeks (inclusive).Monthly: If the range is at least 1 month and less than 120 months. | | Refresh | Initiates an on-demand update of the dashboard with new data. Otherwise, DataRobot refreshes the dashboard every 15 minutes. | | Reset | Reverts the dashboard controls to the default settings. | > [!NOTE] Time range limitations > The Resource monitoring tab is limited to displaying data from the last 30 days. This limitation ensures optimal performance when querying and displaying resource utilization metrics. ## Filter by compute instance For deployments with multiple compute instances, you can filter resource utilization metrics by specific compute instances. Filtering by compute instance allows you to: - Identify which instances are experiencing high resource utilization. - Troubleshoot issues affecting specific instances. - Understand resource distribution across instances. To filter by compute instance, use the Containers selector in the dashboard controls. Metrics are grouped by compute instance and are filtered by LRS ID or inference ID. > [!NOTE] Compute instance filtering > Compute instance filtering is available for deployments with multiple instances. For single-instance deployments, the filtering selector is not available. ## Understanding resource utilization metrics The following sections provide detailed explanations of each resource utilization metric displayed on the Resource monitoring tab. Understanding these metrics helps you evaluate resource usage, identify bottlenecks, and make informed decisions about resource bundle sizing. ### Replicas The Replicas metric shows the number of active compute instances (replicas) currently running for your deployment, out of the maximum available. This metric helps you: - Monitor changes in the number of replicas over time to understand scaling behavior. - Correlate the number of replicas with resource utilization metrics. - Identify when additional capacity is needed or when resources are underutilized. For custom model deployments on serverless, this metric counts custom model pods. ### CPU utilization The CPU utilization metric shows the percentage of CPU cores being utilized across all compute instances. This metric helps you: - Identify CPU bottlenecks that may be affecting model performance. - Understand CPU usage patterns over time. - Make informed decisions about CPU resource bundle sizing. High CPU utilization may indicate that your deployment needs more CPU resources or that the workload is CPU-intensive. Low CPU utilization may suggest that you can reduce the CPU resource bundle size to optimize costs. ### Memory usage The Memory usage metric shows the amount of memory being used across all compute instances. This metric helps you: - Monitor memory usage to prevent out-of-memory errors. - Identify memory leaks or excessive memory consumption. - Make informed decisions about memory resource allocation. Memory usage is displayed in bytes or appropriate units (KB, MB, GB) based on the scale of usage. --- # Standard output logs URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-runtime-logs.html > View runtime logs from the custom model container for debugging. When you deploy a custom model, it generates log reports unique to this type of deployment, allowing you to debug custom code and troubleshoot prediction request failures from within DataRobot. These logs are accessible on the Activity logs > Standard output tab. To view the logs for a deployed custom model: - On theDeployed workloadstab, locate the deployment, click theActions menu(oron the deploymentOverview), and then clickView logs. - In a deployment, click theActivity logtab, and then clickStandard output. From this tab, you can troubleshoot failed prediction requests. The logs are captured from the Docker container running the deployed custom model and contain up to 1MB of data. > [!NOTE] No logs available > Standard output can only be retrieved when the custom model deployment is active; if the deployment is inactive, the Standard output tab and the action menu button are disabled for inactive deployments. > > In addition, even when the Standard output tab is accessible, DataRobot only provides logs from the Docker container running the custom model; therefore, it's possible for specific event logs to be unavailable when a failure occurs outside the Docker container. You can re-request logs by clicking Refresh. Use the Search bar to find specific references within the logs. Click Download Log to save a local copy of the logs. --- # Deployment service health URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-service-health.html > Track latency, throughput, and error rate for generative and agentic custom model deployments. The Service health tab tracks metrics about a deployment's ability to respond to prediction requests quickly and reliably. This helps identify bottlenecks and assess capacity, which is critical to proper provisioning. For example, if a model seems to have generally slowed in its response times, the Service health tab for the model's deployment can help. You might notice in the tab that median latency goes up with an increase in prediction requests. If latency increases when a new model is switched in, you can consult with your team to determine whether the new model can instead be replaced with one offering better performance. To access Service health, select an individual deployment from the deployment inventory page and then, from the Overview, click Monitoring > Service health. The tab provides informational [tiles and a chart](https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-service-health.html#understand-metric-tiles-and-chart) to help assess the activity level and health of the deployment. > [!NOTE] Time of Prediction > The Time of Prediction value differs between the [Data drift](https://docs.datarobot.com/en/docs/classic-ui/mlops/monitor/data-drift.html) and [Accuracy](https://docs.datarobot.com/en/docs/classic-ui/mlops/monitor/deploy-accuracy.html) tabs and the [Service health](https://docs.datarobot.com/en/docs/classic-ui/mlops/monitor/service-health.html) tab: > > On the > Service health > tab, the "time of prediction request" is > always > the time the prediction server > received > the prediction request. This method of prediction request tracking accurately represents the prediction service's health for diagnostic purposes. > On the > Data drift > and > Accuracy > tabs, the "time of prediction request" is, > by default > , the time you > submitted > the prediction request, which you can override with the prediction timestamp in the > Prediction History and Service Health > settings. ## Understand metric tiles and chart DataRobot displays informational statistics based on your current settings for model and time frame. That is, tile values correspond to the same units as those selected on the slider. If the slider interval values are weekly, the displayed tile metrics show values corresponding to weeks. Clicking a metric tile updates the chart below. The Service health tab reports the following metrics on the dashboard: > [!NOTE] Service health information for external models and monitoring jobs > Service health information is unavailable for external [agent-monitored deployments](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/mlops-agent/monitoring-agent/index.html) and deployments with predictions uploaded through a [prediction monitoring job](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/pred-monitoring-jobs/index.html). | Statistic | Reports (for selected time period) | | --- | --- | | Total Predictions | The number of predictions the deployment has made (per prediction node). | | Total Requests | The number of prediction requests the deployment has received (a single request can contain multiple prediction requests). | | Requests over x ms | The number of requests where the response time was longer than the specified number of milliseconds. The default is 2000 ms; click in the box to enter a time between 10 and 100,000 ms or adjust with the controls. | | Response Time | The time (in milliseconds) DataRobot spent receiving a prediction request, calculating the request, and returning a response to the user. The report does not include time due to network latency. Select the median prediction request time or 90th, 95th, or 99th percentile. The display reports a dash if you have made no requests against it or if it's an external deployment. | | Execution Time | The time (in milliseconds) DataRobot spent calculating a prediction request. Select the median prediction request time or 90th, 95th, or 99th percentile. | | Median/Peak Load | The median and maximum number of requests per minute. | | Data Error Rate | The percentage of requests that result in a 4xx error (problems with the prediction request submission). This is a component of the value reported as the Service Health Summary on the Deployed workloads dashboard top banner. | | System Error Rate | The percentage of well-formed requests that result in a 5xx error (problem with the DataRobot prediction server). This is a component of the value reported as the Service Health Summary on the Deployed workloads dashboard top banner. | | Consumers | The number of distinct users (identified by API key) who have made prediction requests against this deployment. | You can configure the dashboard to focus the visualized statistics on specific segments and time frames. The following controls are available: | Control | Description | | --- | --- | | Model | Updates the dashboard displays to reflect the model you selected from the dropdown. | | Range (UTC) | Sets the date range displayed for the deployment date slider. You can also drag the date slider to set the range. The range selector only allows you to select dates and times between the start date of the deployment's current version of a model and the current date. | | Resolution | Sets the time granularity of the deployment date slider. The following resolution settings are available, based on the selected range: Hourly: If the range is less than 7 days.Daily: If the range is between 1-60 days (inclusive).Weekly: If the range is between 1-52 weeks (inclusive).Monthly: If the range is at least 1 month and less than 120 months. | | Refresh | Initiates an on-demand update of the dashboard with new data. Otherwise, DataRobot refreshes the dashboard every 15 minutes. | | Reset | Reverts the dashboard controls to the default settings. | The chart below the metric tiles displays individual metrics over time, helping to identify patterns in the quality of service. Clicking on a metric tile updates the chart to represent that information; adjusting the data range slider focuses on a specific period: > [!TIP] Export charts > Click Export to download a `.csv` or `.png` file of the currently selected chart, or a `.zip` archive file of both (and a `.json` file). The Median | Peak Load (calls/minute) chart displays two lines, one for Peak load and one for Median load over time: ## Service health status indicators Service health tracks metrics about a deployment’s ability to respond to prediction requests quickly and reliably. You can view the service health status in the [deployment inventory](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-overview/nxt-dashboard.html#health-indicators) and visualize service health on the [Service health](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-service-health.html) tab. Service health monitoring represents the occurrence of 4XX and 5XX errors in your prediction requests or prediction server: - 4xx errors indicate problems with the prediction request submission. - 5xx errors indicate problems with the DataRobot prediction server. | Color | Service Health | Action | | --- | --- | --- | | Green / Passing | Zero 4xx or 5xx errors. | No action needed. | | Yellow / At risk | At least one 4xx error and zero 5xx errors. | Concerns found, but no immediate action needed; monitor. | | Red / Failing | At least one 5xx error. | Immediate action needed. | | Gray / Disabled | Unmonitored deployment. | Enable monitoring and make predictions. | | Gray / Not started | No service health events recorded. | Make predictions. | | Gray / Unknown | No predictions made. | Make predictions. | ## Explore deployment data tracing Tracing for custom model deployments (including text generation, agentic workflow, VDB, and MCP deployments) is on the dedicated Tracing tab. For search, filter, and span details, see [Tracing](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-tracing.html). --- # Deployment usage URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/agent-usage.html > Monitor quota usage, tokens, and rate limits for agentic workflow and related serverless deployments. For text generation, VDB, and MCP custom model deployments, the Usage tab follows the standard prediction-processing views described in [Usage](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-usage.html). For agentic workflow (and NIM) deployments, the Quota monitoring experience below is the primary usage view. ## Quota usage monitoring On the Monitoring > Usage tab for agentic workflow and NIM deployments, the Quota monitoring dashboard visualizes historical usage segmented by user or agent. Other serverless generative deployments use the same Console controls when quota monitoring applies. The Quota monitoring dashboard displays three key metric tiles at the top of the page: | Metric | Description | | --- | --- | | Total requests | The total number of requests made during the selected time range, along with the average requests per minute. | | Total rate limited requests | The total number of requests that were rate limited during the selected time range, along with the average rate limited requests per minute. | | Total token count | The total number of tokens consumed during the selected time range, along with the average tokens per minute. | | Average concurrent requests | The average number of simultaneous API calls processed by the agent service over the defined interval, tracked as a key metric for observability and used to enforce the system's quota limit on simultaneous operations. | Each metric displays the value for the selected time frame and the average per minute in green. Click the metric tile to review the corresponding chart below: - Total requests - Total rate limited requests - Total token count - Average concurrent requests You can configure the Quota monitoring dashboard to focus the visualized statistics on specific entities and time frames. The following controls are available: | Filter | Description | | --- | --- | | Model | Select the model version to monitor. The Current option displays data for the active model version. | | Range (UTC) | Select the date and time range for the data displayed. Use the date pickers to set the start and end times in UTC. | | Resolution | Select the time resolution for aggregating data: Hourly, Daily, or Weekly. | | Entity | Filter by entity type: All, User, or Agent. | | Refresh | Updates the dashboard with the latest data based on the current filter settings. | | Reset | Resets all filters to their default values. | ### Quota monitoring charts The Quota monitoring charts display an area chart showing the distribution of requests over time, rate limited requests over time, or token count over time. This chart is a stacked chart (or stacked graph), a chart stacking multiple data series on top of each other to visualize how each entity contributes to the total over time and across categories. Each chart is segmented by entity (user or agent). Each entity is represented by a different color in the chart legend. | | Chart element | Description | | --- | --- | --- | | (1) | Entity filter | Displays all entities (users or agents) included in the selected time range. Each entity is represented by a dot that matches the area in the chart. | | (2) | Entity legend | Displays all entities (users or agents) included in the selected time range. Each entity is represented by a dot that matches the area in the chart. | | (3) | Time range (X-axis) | Displays the time range selected in the filters, showing the date range from start to end. | | (4) | Metric (Y-axis) | Displays the number of requests, rate limited requests, or tokens on the vertical axis. | | (5) | Request areas | Overlapping areas show the volume of requests per entity over time. The height of each area at any point represents the number of requests for that entity at that time. This chart is a stacked chart (or stacked graph), a chart stacking multiple data series on top of each other to visualize how each entity contributes to the total over time and across categories. | | (6) | Tracing | Click Show tracing to view tracing data for the requests. | | (7) | Export | Click Export to download a .csv file. | Hover over the chart to view detailed information about the number of requests for each entity at specific time points. ### Request tracing table On any Quota monitoring chart, click Show tracing to view tracing data for the deployment. This tracing chart uses the same interface as the dedicated [Tracing](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-tracing.html) tab. ### Rate limited requests table The Rate limited requests table provides a detailed breakdown of rate limiting by entity: | | Table element | Description | | --- | --- | --- | | (1) | Entity type filter | Filter the table by entity type (user or agent). | | (2) | Rate limited percentage filter | Filter entities by their rate limited percentage threshold (zero, low, medium, or high). | | (3) | Search box | Search for specific entities by name or identifier. | | (4) | Entity column | Displays the entity identifier (user email or agent name). | | (5) | Rate limited requests column | Shows the number of rate limited requests and the percentage of total requests that were rate limited. The percentage is highlighted in red when it exceeds a threshold, or displayed in gray when it is 0%. | | (6) | Requests column | Displays the number of requests that were rate limited due to exceeding the request quota. | | (7) | Token count column | Displays the number of requests that were rate limited due to exceeding the token quota. | | (8) | Concurrent requests column | Displays the number of requests that were rate limited due to exceeding the concurrent requests quota. | The table helps identify which entities are experiencing rate limiting and to what extent, allowing you to adjust quotas or usage patterns accordingly. --- # Monitor URL: https://docs.datarobot.com/en/docs/agentic-ai/agentic-monitor/index.html > Monitor an agentic artifact deployment's performance and behavior. > [!NOTE] Premium > DataRobot's Agentic AI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. After you've deployed an agentic artifact to Console, monitor the deployment's performance and behavior. In Console, on the Deployed workloads page, click the Agentic Workflow tab to view a list of agentic workflow deployments. A fully configured agentic workflow deployment has access to the following Console features: | Feature | Description | | --- | --- | | Overview | | | Deployment overview | Review deployment details, lineage, tags, runtime parameters, and—when guardrails are configured—evaluation and moderation summary information. | | Tracing | | | Tracing | Inspect agent traces from a dedicated Tracing tab. Search and filter traces, review span duration and token usage, and expand traces to view attributes, logs, input, and output on one page. | | Monitoring | | | Deployment service health | Track model-specific deployment latency, throughput, and error rate. | | Deployment usage | Track prediction processing progress for use in accuracy, data drift, and predictions over time analysis. For agentic workflow deployments, includes quota usage monitoring segmented by user or agent. | | Custom metrics | Create and monitor custom business or performance metrics or add pre-made metrics. When you configure evaluation and moderation for the workflow, guard metrics (for example, guard latency and blocked counts) are reported here. | | Data exploration | Explore and export stored prediction data, actuals, and training data, and assess response quality. | | Deployment reports | Generate reports, immediately or on a schedule, to summarize the details of a deployment, such as its owner, how the model was built, the model age, and the humility monitoring status. | | Resource monitoring | Monitor CPU, memory, and replica utilization for the serverless deployment. | | OpenTelemetry metrics | Visualize OpenTelemetry metrics from your application alongside DataRobot native metrics. | | Predictions | | | Prediction API snippets | Use downloadable snippets to call the deployment's prediction and chat completion APIs from your application (including real-time scoring integrations). | | Activity log | | | Standard output logs | View runtime logs from the custom model container to debug scoring and request failures. | | OpenTelemetry logs | View OpenTelemetry log events for troubleshooting and deeper analysis (span-related logs can also be filtered from the Tracing tab). | | Moderation events | When evaluation and moderation guardrails are enabled, review guard-related events to diagnose blocked requests and guard failures. | --- # Authentication management URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/auth.html > Manage authentication with DataRobot. Manage authentication with DataRobot. ## Synopsis ``` dr auth [flags] ``` ## Description The `auth` command provides authentication management for the DataRobot CLI. It handles login, logout, and URL configuration for connecting to your DataRobot instance. ## Commands ### login Authenticate with DataRobot using OAuth. ``` dr auth login ``` Behavior: 1. Starts a local web server (typically on port 8080) 2. Opens your default browser to DataRobot's OAuth page 3. Prompts you to authorize the CLI 4. Receives and stores the API key 5. Closes the browser and server automatically Example: ``` $ dr auth login Opening browser for authentication... Waiting for authentication... ✓ Successfully authenticated! ``` Stored Credentials: - Location: `~/.config/datarobot/drconfig.yaml` (Linux/macOS) or `%USERPROFILE%\.config\datarobot\drconfig.yaml` (Windows) - Format: Encrypted API key Troubleshooting: ``` # If browser doesn't open automatically # The CLI will display a URL to visit manually: $ dr auth login Failed to open browser automatically. Please visit: https://app.datarobot.com/oauth/authorize?client_id=... # Port already in use # The CLI will try alternative ports automatically ``` On Windows, if your DataRobot host uses a private CA, pass `--export-windows-certs` before `auth`. This global flag is available only in Windows installations of the CLI: ``` dr --export-windows-certs auth login ``` ### logout Remove stored authentication credentials. ``` dr auth logout ``` Example: ``` $ dr auth logout ✓ Successfully logged out ``` Effect: - Removes API key from config file - Keeps DataRobot URL configuration - Next API call will require re-authentication ### set-url Configure the DataRobot instance URL. ``` dr auth set-url [url] ``` Arguments: - `url` (optional) - DataRobot instance URL Interactive Mode: If no URL is provided, enters interactive mode: ``` $ dr auth set-url Please specify your DataRobot URL, or enter the numbers 1 - 3 if you are using that multi tenant cloud offering Please enter 1 if you are using https://app.datarobot.com Please enter 2 if you are using https://app.eu.datarobot.com Please enter 3 if you are using https://app.jp.datarobot.com Otherwise, please enter the URL you use > _ ``` Direct Mode: Specify URL directly: ``` # Using cloud shortcuts $ dr auth set-url 1 # Sets to https://app.datarobot.com $ dr auth set-url 2 # Sets to https://app.eu.datarobot.com $ dr auth set-url 3 # Sets to https://app.jp.datarobot.com # Using full URL $ dr auth set-url https://app.datarobot.com $ dr auth set-url https://my-company.datarobot.com ``` Validation: ``` $ dr auth set-url invalid-url Error: Invalid URL format ``` ## Global flags These flags work with all `auth` commands: ``` -v, --verbose Enable verbose output --debug Enable debug output --skip-auth Skip authentication checks (for advanced users) -h, --help Show help for command ``` > ⚠️ Warning:The--skip-authflag bypasses all authentication checks. This is intended for advanced use cases where authentication is handled externally or not required. When this flag is used, commands that require authentication may fail with API errors. ## Examples ### Initial setup ``` # Set URL and login (recommended workflow) $ dr auth set-url https://app.datarobot.com ✓ DataRobot URL set to: https://app.datarobot.com $ dr auth login Opening browser for authentication... ✓ Successfully authenticated! ``` ### Using cloud instance shortcuts ``` # US Cloud $ dr auth set-url 1 $ dr auth login # EU Cloud $ dr auth set-url 2 $ dr auth login # Japan Cloud $ dr auth set-url 3 $ dr auth login ``` ### Self-managed instance ``` $ dr auth set-url https://datarobot.mycompany.com $ dr auth login ``` ### Re-authentication ``` # Logout and login again $ dr auth logout ✓ Successfully logged out $ dr auth login Opening browser for authentication... ✓ Successfully authenticated! ``` ### Switching instances ``` # Switch to different DataRobot instance $ dr auth set-url https://staging.datarobot.com $ dr auth login ``` ### Debug authentication issues ``` # Use verbose flag for details $ dr auth login --verbose [INFO] Starting OAuth server on port 8080 [INFO] Opening browser to: https://app.datarobot.com/oauth/... [INFO] Waiting for callback... [INFO] Received authorization code [INFO] Exchanging code for token... [INFO] Token saved successfully ✓ Successfully authenticated! # Use debug flag for even more details $ dr auth login --debug [DEBUG] Config file: /Users/username/.datarobot/config.yaml [DEBUG] Current URL: https://app.datarobot.com [DEBUG] Starting server on: 127.0.0.1:8080 ... ``` ## Authentication flow ``` ┌──────────┐ │ User │ └────┬─────┘ │ │ dr auth login │ v ┌─────────────────┐ ┌──────────────┐ │ Local Server │◄──────┤ Browser │ │ (Port 8080) │ │ │ └────┬────────────┘ └──────▲───────┘ │ │ │ │ Opens │ │ v │ ┌─────────────────┐ │ │ DataRobot │───────────────┘ │ OAuth Server │ └────┬────────────┘ │ │ Returns API Key │ v ┌─────────────────┐ │ Config File │ │ (~/.config/ │ │ datarobot/ │ │ drconfig.yaml)│ └─────────────────┘ ``` ## Configuration file After authentication, credentials are stored in: Location: - Linux/macOS: `~/.config/datarobot/drconfig.yaml` - Windows: `%USERPROFILE%\.config\datarobot\drconfig.yaml` Format: ``` datarobot: endpoint: https://app.datarobot.com token: # User preferences preferences: default_timeout: 30 verify_ssl: true ``` Permissions: - File is created with restricted permissions (0600) - Only the user who created it can read/write ## Security best practices ### 1. Protect your config file ``` # Verify permissions ls -la ~/.config/datarobot/drconfig.yaml # Should show: -rw------- (600) # Fix if needed chmod 600 ~/.config/datarobot/drconfig.yaml ``` ### 2. Do not share credentials Never commit or share: - `~/.config/datarobot/drconfig.yaml` - API keys - OAuth tokens ### 3. Use per-environment authentication ``` # Development export DATAROBOT_CLI_CONFIG=~/.config/datarobot/dev-config.yaml dr auth set-url https://dev.datarobot.com --config $DATAROBOT_CLI_CONFIG dr auth login # Production export DATAROBOT_CLI_CONFIG=~/.config/datarobot/prod-config.yaml dr auth set-url https://prod.datarobot.com --config $DATAROBOT_CLI_CONFIG dr auth login ``` ### 4. Regular re-authentication ``` # Logout when finished dr auth logout # Login only when needed dr auth login ``` ## Environment variables Override configuration with environment variables: ``` # Override URL export DATAROBOT_ENDPOINT=https://app.datarobot.com # Override API key (not recommended) export DATAROBOT_API_TOKEN=your-api-token # Custom config file location export DATAROBOT_CLI_CONFIG=~/.config/datarobot/custom-config.yaml ``` ## Common issues ### Browser doesn't open Problem: Browser fails to open automatically. Solution: ``` # Copy the URL from the output and open manually $ dr auth login Failed to open browser automatically. Please visit: https://app.datarobot.com/oauth/authorize?... ``` ### Port already in use Problem: Port 8080 is already in use. Solution: The CLI automatically tries alternative ports (8081, 8082, etc.) ### Invalid credentials Problem:"Authentication failed" error. Solution: ``` # Clear credentials and try again dr auth logout dr auth login ``` ### Connection refused Problem: Cannot connect to DataRobot. Solution: ``` # Verify URL is correct cat ~/.config/datarobot/drconfig.yaml # Try setting URL again dr auth set-url https://app.datarobot.com # Check network connectivity ping app.datarobot.com ``` ### SSL certificate issues Problem: SSL verification fails. Solution: ``` # For self-signed certificates (not recommended for production) export DATAROBOT_VERIFY_SSL=false dr auth login ``` ## See also - Getting Started - Initial setup guide - Configuration - Configuration file details - templates - Template management commands --- # Shell completion URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/completion.html > Generate shell completion scripts for command auto-completion. Generate shell completion scripts for command auto-completion. ## Synopsis ``` dr self completion ``` ## Description The `self completion` command generates shell completion scripts that enable auto-completion for the DataRobot CLI. Completions provide command, subcommand, and flag suggestions when you press Tab. ## Supported shells - bash —Bourne Again Shell. - zsh —Z Shell. - fish —Friendly Interactive Shell. - powershell —PowerShell. ## Usage ### Bash Linux: ``` # Install system-wide dr self completion bash | sudo tee /etc/bash_completion.d/dr # Reload shell source ~/.bashrc ``` macOS: ``` # Install via Homebrew's bash-completion brew install bash-completion@2 dr self completion bash > $(brew --prefix)/etc/bash_completion.d/dr # Reload shell source ~/.bash_profile ``` Temporary (current session only): ``` source <(dr self completion bash) ``` ### Zsh Setup: First, ensure completion is enabled: ``` # Add to ~/.zshrc if not present autoload -U compinit compinit ``` Installation: ``` # Option 1: User completions directory mkdir -p ~/.zsh/completions dr self completion zsh > ~/.zsh/completions/_dr echo 'fpath=(~/.zsh/completions $fpath)' >> ~/.zshrc # Option 2: System directory dr self completion zsh > "${fpath[1]}/_dr" # Clear cache and reload rm -f ~/.zcompdump source ~/.zshrc ``` Temporary (current session only): ``` source <(dr self completion zsh) ``` ### Fish ``` # Install completion dr self completion fish > ~/.config/fish/completions/dr.fish # Reload Fish source ~/.config/fish/config.fish ``` Temporary (current session only): ``` dr self completion fish | source ``` ### PowerShell Persistent: ``` # Generate completion script dr self completion powershell > dr.ps1 # Add to PowerShell profile Add-Content $PROFILE ". C:\path\to\dr.ps1" # Reload profile . $PROFILE ``` Temporary (current session only): ``` dr self completion powershell | Out-String | Invoke-Expression ``` ## Examples ### Generate completion script ``` # View the generated script dr self completion bash # Save to a file dr self completion bash > dr-completion.bash # Save for all shells dr self completion bash > dr-completion.bash dr self completion zsh > dr-completion.zsh dr self completion fish > dr-completion.fish dr self completion powershell > dr-completion.ps1 ``` ### Install for multiple shells If you use multiple shells: ``` # Bash dr self completion bash > ~/.bash_completions/dr # Zsh dr self completion zsh > ~/.zsh/completions/_dr # Fish dr self completion fish > ~/.config/fish/completions/dr.fish ``` ### Update completions After updating the CLI: ``` # Bash dr self completion bash | sudo tee /etc/bash_completion.d/dr # Zsh dr self completion zsh > ~/.zsh/completions/_dr rm -f ~/.zcompdump exec zsh # Fish dr self completion fish > ~/.config/fish/completions/dr.fish ``` ## Completion behavior ### Command completion ``` $ dr auth completion dotenv run templates version $ dr auth login logout set-url $ dr templates clone list setup status ``` ### Flag completion ``` $ dr run -- --concurrency --dir --exit-code --help --list --parallel --silent --watch --yes $ dr -- --debug --help --verbose ``` ### Argument completion Some commands support argument completion: ``` # Template names (when connected to DataRobot) $ dr templates clone python-streamlit react-frontend fastapi-backend # Task names (when in a template directory) $ dr run build dev deploy lint test ``` ## Troubleshooting ### Completions not working Bash: 1. Verify bash-completion is installed: # macOSbrewlistbash-completion@2# Linuxdpkg-l|grepbash-completion 2. Check if completion script exists: ls-l/etc/bash_completion.d/dr 3. Ensure .bashrc sources completions: grepbash_completion~/.bashrc 4. Reload shell: source~/.bashrc Zsh: 1. Verify compinit is called: grepcompinit~/.zshrc 2. Check fpath includes completion directory: echo$fpath 3. Clear completion cache: rm-f~/.zcompdump* compinit 4. Reload shell: execzsh Fish: 1. Check completion file: ls-l~/.config/fish/completions/dr.fish 2. Verify Fish recognizes it: complete-Cdr 3. Reload Fish: source~/.config/fish/config.fish PowerShell: 1. Check execution policy: Get-ExecutionPolicy If restricted: ``` Set-ExecutionPolicy RemoteSigned -Scope CurrentUser ``` 1. Verify profile loads completion: cat$PROFILE 2. Reload profile: .$PROFILE ### Permission denied Use user-level installation instead of system-wide: ``` # Bash - user level mkdir -p ~/.bash_completions dr self completion bash > ~/.bash_completions/dr echo 'source ~/.bash_completions/dr' >> ~/.bashrc # Zsh - user level mkdir -p ~/.zsh/completions dr self completion zsh > ~/.zsh/completions/_dr echo 'fpath=(~/.zsh/completions $fpath)' >> ~/.zshrc ``` ### Outdated completions After updating the CLI, regenerate completions: ``` # Bash dr self completion bash | sudo tee /etc/bash_completion.d/dr source ~/.bashrc # Zsh dr self completion zsh > ~/.zsh/completions/_dr rm -f ~/.zcompdump exec zsh # Fish dr self completion fish > ~/.config/fish/completions/dr.fish ``` ## Completion features ### Intelligent suggestions Completions are context-aware: ``` # Only shows valid subcommands dr auth # Shows: login logout set-url (not other commands) # Only shows valid flags dr run --l # Shows: --list (not all flags) ``` ### Description support In Fish and PowerShell, completions include descriptions: ``` $ dr templates clone (Clone a template repository) list (List available templates) setup (Interactive template setup wizard) status (Show current template status) ``` ### Dynamic completion Some completions are generated dynamically: ``` # Template names from DataRobot API dr templates clone # Task names from current Taskfile dr run # Available shells dr self completion ``` ## Advanced configuration ### Custom completion scripts You can extend or modify generated completions: ``` # Generate base completion dr self completion bash > ~/dr-completion-custom.bash # Edit to add custom logic vim ~/dr-completion-custom.bash # Source your custom version source ~/dr-completion-custom.bash ``` ### Completion performance For faster completions, especially with dynamic suggestions: ``` # Cache template list dr templates list > ~/.dr-templates-cache # Use cached list in custom completion script ``` ## See also - Shell completion guide —detailed setup instructions. - Getting started —initial setup. - Command completion is powered by Cobra . --- # dotenv command URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/dotenv.html > Manage environment variables and .env files in DataRobot templates. Manage environment variables and `.env` files in DataRobot templates. ## Overview The `dr dotenv` command provides tools for creating, editing, validating, and updating environment configuration files. It includes an interactive wizard for guided setup and a text editor for direct file manipulation. ## Commands ### dr dotenv setup Launch the interactive wizard to configure environment variables. ``` dr dotenv setup ``` Features: - Interactive prompts for all required variables. - Context-aware questions based on template configuration. - Automatic discovery of configuration from .datarobot/prompts.yaml files. - Smart defaults from .env.template . - Secure handling of secret values. - DataRobot authentication integration. - Automatic state tracking of completion timestamp. Prerequisites: - Must be run inside a git repository. - Requires authentication with DataRobot. State tracking: Upon successful completion, `dr dotenv setup` records the timestamp in the state file. This allows `dr templates setup` to intelligently skip dotenv configuration if it has already been completed. The state is stored in the same location as other CLI state (see [Configuration - State tracking](https://docs.datarobot.com/en/docs/agentic-ai/cli/configuration.html#state-tracking)). Keep in mind that `dr dotenv setup` will always prompt for configuration if run manually, regardless of state. To force the setup wizard to run again (ignoring the state file), use the `--force-interactive` flag: ``` dr templates setup --force-interactive ``` This is useful for testing or when you need to reconfigure your environment from scratch. Example: ``` cd my-template dr dotenv setup ``` The wizard guides you through: 1. DataRobot credentials (auto-populated if authenticated). 2. Application-specific configuration. 3. Optional features and integrations. 4. Validation of all inputs. 5. Generation of `.env` file. ### dr dotenv edit Open the `.env` file in an interactive editor or wizard. ``` dr dotenv edit ``` Behavior: - If `.env` exists, opens it in the editor. - If no extra variables are detected, opens text editor mode. - If template prompts are found, offers wizard mode. - Can switch between editor and wizard modes. Editor mode controls: - `e` —edit in text editor. - `w` —switch to wizard mode. - `Enter` —save and exit. - `Esc` —save and exit. Wizard mode controls: - Navigate prompts with arrow keys. - Enter values or select options. - `Esc` —return to previous screen. Example: ``` cd my-template dr dotenv edit ``` ### dr dotenv update Automatically refresh DataRobot credentials in the `.env` file. ``` dr dotenv update ``` Features: - Updates `DATAROBOT_ENDPOINT` and `DATAROBOT_API_TOKEN`. - Preserves all other environment variables. - Automatically authenticates if needed. - Uses current authentication session. Prerequisites: - Must be run inside a git repository. - Must have a `.env` or `.env.template` file. - Requires authentication with DataRobot. Example: ``` cd my-template dr dotenv update ``` Use cases: - Refresh expired API tokens. - Switch DataRobot environments. - Update credentials after re-authentication. ### dr dotenv validate Validate that all required environment variables are properly configured. ``` dr dotenv validate ``` Features: - Validates against template requirements defined in `.datarobot/prompts.yaml`. - Checks both `.env` file and environment variables. - Verifies core DataRobot variables ( `DATAROBOT_ENDPOINT`, `DATAROBOT_API_TOKEN`). - Reports missing or invalid variables with helpful error messages. - Respects conditional requirements based on selected options. Prerequisites: - Must be run inside a git repository. - Must have a `.env` file. Example: ``` cd my-template dr dotenv validate ``` Output: Successful validation: ``` Validating required variables: APP_NAME: my-app DATAROBOT_ENDPOINT: https://app.datarobot.com DATAROBOT_API_TOKEN: *** DATABASE_URL: postgresql://localhost:5432/db Validation passed: all required variables are set. ``` Validation errors: ``` Validating required variables: APP_NAME: my-app DATAROBOT_ENDPOINT: https://app.datarobot.com Validation errors: Error: required variable DATAROBOT_API_TOKEN is not set Description: DataRobot API token for authentication Set this variable in your .env file or run `dr dotenv setup` to configure it. Error: required variable DATABASE_URL is not set Description: PostgreSQL database connection string Set this variable in your .env file or run `dr dotenv setup` to configure it. ``` Use cases: - Verify configuration before running tasks. - Debug missing environment variables. - CI/CD pipeline checks. - Troubleshoot application startup issues. ## File structure ### .env.template Template file committed to version control: ``` # Required Configuration APP_NAME= DATAROBOT_ENDPOINT= DATAROBOT_API_TOKEN= # Optional Configuration # DEBUG=false # PORT=8080 ``` ### .env Generated configuration file (never committed): ``` # Required Configuration APP_NAME=my-awesome-app DATAROBOT_ENDPOINT=https://app.datarobot.com DATAROBOT_API_TOKEN=*** # Optional Configuration DEBUG=true PORT=8000 ``` ## Interactive configuration ### Prompt types The wizard supports multiple input types defined in `.datarobot/prompts.yaml`: Text input: ``` prompts: - key: "app_name" env: "APP_NAME" help: "Enter your application name" ``` Secret string: ``` prompts: - key: "api_key" env: "API_KEY" type: "secret_string" help: "Enter your API key" generate: true # Auto-generate a random secret ``` Single selection: ``` prompts: - key: "environment" env: "ENVIRONMENT" help: "Select deployment environment" options: - name: "Development" value: "dev" - name: "Production" value: "prod" ``` Multiple selection: ``` prompts: - key: "features" env: "ENABLED_FEATURES" help: "Select features to enable" multiple: true options: - name: "Analytics" - name: "Monitoring" ``` ### Conditional prompts Prompts can be shown based on previous selections: ``` prompts: - key: "enable_database" help: "Enable database?" options: - name: "Yes" requires: "database_config" - name: "No" - key: "database_url" section: "database_config" env: "DATABASE_URL" help: "Database connection string" ``` ## Common workflows ### Initial setup Set up a new template with all configuration: ``` cd my-template dr dotenv setup ``` ### Quick updates Update just the DataRobot credentials: ``` dr dotenv update ``` ### Manual editing Edit variables directly: ``` dr dotenv edit # Press 'e' for editor mode # Make changes # Press Enter to save ``` ### Validation Check configuration before running tasks: ``` dr dotenv validate dr run dev ``` ### Switch wizard to editor Start with wizard, switch to editor: ``` dr dotenv edit # Press 'w' for wizard mode # Complete some prompts # Press 'e' to switch to editor for fine-tuning ``` ## Configuration discovery The CLI automatically discovers configuration from: 1. .env.template —base template with variable names. 2. .datarobot/prompts.yaml —interactive prompts and validation. 3. Existing.env —current values (if present). 4. Environment variables —system environment (override .env ). Priority order (highest to lowest): 1. System environment variables. 2. User input from wizard. 3. Existing `.env` file values. 4. Default values from prompts. 5. Template values from `.env.template`. ## Security ### Secret handling - Secret values are masked in the UI. - Variables containing "PASSWORD", "SECRET", "KEY", or "TOKEN" are automatically treated as secrets. - The secret_string prompt type enables secure input with masking. - .env files should never be committed (add to .gitignore ). ### Auto-generation Secret strings with `generate: true` are automatically generated: ``` prompts: - key: "session_secret" env: "SESSION_SECRET" type: "secret_string" generate: true help: "Session encryption key" ``` This generates a cryptographically secure random string when no value exists. ## Error handling ### Not in repository ``` Error: not inside a git repository Run this command from within an application template git repository. To create a new template, run `dr templates setup`. ``` Solution: Navigate to a git repository or use `dr templates setup`. ### Missing .env file ``` Error: .env file does not exist at /path/to/.env Run `dr dotenv setup` to create one. ``` Solution: Run `dr dotenv setup` to create the file. ### Authentication required ``` Error: not authenticated Run `dr auth login` to authenticate. ``` Solution: Authenticate with `dr auth login`. ### Validation failures ``` Validation errors: Error: required variable DATABASE_URL is not set Description: PostgreSQL database connection string Set this variable in your .env file or run `dr dotenv setup` to configure it. ``` Solution: Set the missing variables or run `dr dotenv setup`. ## Exit codes | Code | Meaning | | --- | --- | | 0 | Success. | | 1 | Error (file not found, validation failed, not in repo). | | 130 | Interrupted (Ctrl+C). | ## Examples ### Create configuration from scratch ``` cd my-template dr dotenv setup ``` ### Update after re-authentication ``` dr auth login dr dotenv update ``` ### Validate before deployment ``` dr dotenv validate && dr run deploy ``` ### Edit specific variables ``` dr dotenv edit # Press 'e' for editor mode # Update DATABASE_URL # Press Enter to save ``` ### Check configuration ``` cat .env dr dotenv validate ``` ## Integration with other commands ### With templates ``` dr templates setup # Automatically runs dotenv setup ``` ### With run ``` dr dotenv validate dr run dev ``` ### With auth ``` dr auth login dr dotenv update ``` ## See also - Environment variables guide —managing .env files. - Interactive configuration —configuration wizard details. - Template structure —template organization. - auth command —authentication management. - run command —executing tasks. --- # Command reference URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html > Complete reference documentation for all DataRobot CLI commands. Complete reference documentation for all DataRobot CLI commands. ## Global flags These flags are available for all commands: ``` -v, --verbose Enable verbose output (info level logging) --debug Enable debug output (debug level logging) --skip-auth Skip authentication checks (for advanced users) --force-interactive Force the setup wizard to run even if already completed -h, --help Show help information ``` Warning: The `--skip-auth` flag is intended for advanced use cases only. Using this flag will bypass all authentication checks, which may cause API calls to fail. Use with caution. Note: The `--force-interactive` flag forces commands to behave as if setup has never been completed, while still updating the state file. This is useful for testing or forcing re-execution of setup steps. For more on these flags from a configuration perspective, see [Configuration - Advanced flags](https://docs.datarobot.com/en/docs/agentic-ai/cli/configuration.html#advanced-flags) and [Getting started - Getting help](https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html#getting-help). ## Choosing commands - dr start — Use for first-time setup or quickstart: runs task start if present, then a quickstart script from .datarobot/cli/bin/ , or falls back to the template setup wizard. Best when you want one command to "get running." - dr run — Use to run specific tasks (e.g., dr run dev , dr run test ). Use when you already have a template and want to execute a single task. - dr task list — Lists all available tasks. Use when you want to see what tasks a template provides. - dr task compose — Composes a unified Taskfile from component Taskfiles. Use when developing or customizing templates that use multiple Taskfiles. ## Commands ### Main commands | Command | Description | | --- | --- | | auth | Authenticate with DataRobot. | | templates | Manage application templates. | | start | Run the application quickstart process. | | run | Execute application tasks. | | task | Manage Taskfile composition and task execution. | | dotenv | Manage environment variables. | | llm-gateway | List LLMs and set the CLI's default LLM. | | self | CLI utility commands (update, version, completion). | ### Command tree ``` dr ├── auth Authentication management │ ├── login Log in to DataRobot │ ├── logout Log out from DataRobot │ └── set-url Set DataRobot URL ├── templates Template management │ ├── list List available templates │ ├── clone Clone a template │ ├── setup Interactive setup wizard │ └── status Show template status ├── start Run quickstart process (alias: quickstart) ├── run Task execution ├── task Taskfile composition and execution │ ├── compose Compose unified Taskfile │ ├── list List available tasks │ └── run Execute tasks ├── dotenv Environment configuration ├── llm-gateway LLM model management (alias: llm) │ ├── list List available LLMs │ └── select Set the default LLM └── self CLI utility commands ├── completion Shell completion │ ├── bash Generate bash completion │ ├── zsh Generate zsh completion │ ├── fish Generate fish completion │ └── powershell Generate PowerShell completion ├── config Display configuration settings ├── update Update CLI to latest version └── version Version information ``` ## Quick examples ### Authentication ``` # Set URL and login dr auth set-url https://app.datarobot.com dr auth login # Logout dr auth logout ``` ### Templates ``` # List templates dr templates list # Clone template dr templates clone python-streamlit # Interactive setup dr templates setup # Check status dr templates status ``` ### Quickstart ``` # Run quickstart process (interactive) dr start # Run with auto-yes dr start --yes # Using the alias dr quickstart ``` ### Environment configuration ``` # Interactive wizard dr dotenv setup # Editor mode dr dotenv edit # Validate configuration dr dotenv validate ``` ### Running tasks ``` # List available tasks dr task list # Run a task dr run dev # Run multiple tasks dr run lint test --parallel ``` ### Shell completions ``` # Bash (Linux) dr self completion bash | sudo tee /etc/bash_completion.d/dr # Zsh dr self completion zsh > "${fpath[1]}/_dr" # Fish dr self completion fish > ~/.config/fish/completions/dr.fish ``` ### CLI management ``` # Update to latest version dr self update # Check version dr self version ``` ## Command details For detailed documentation on each command, see: - auth —authentication management. - login —OAuth authentication. - logout —remove credentials. - set-url—configure DataRobot URL. - templates—template operations. - list —list available templates. - clone —clone a template repository. - setup —interactive wizard for full setup. - status—show current template status. - run—task execution. - Execute template tasks (e.g., dr run dev ). - List available tasks via dr task list . - Parallel execution support. - Watch mode for development. - task—Taskfile composition and management. - compose —generate unified Taskfile from components. - list —list all available tasks. - run—execute tasks. - dotenv—environment management. - Interactive configuration wizard. - Direct file editing. - Variable validation. - completion—shell completions. - Bash, Zsh, Fish, PowerShell support. - Auto-complete commands and flags. - version—version information. - Show CLI version. - Build information. - Runtime details. ## Getting help ``` # General help dr --help dr -h # Command help dr auth --help dr templates --help dr run --help # Subcommand help dr auth login --help dr templates clone --help ``` ## Environment variables Global environment variables that affect all commands: ``` # Configuration DATAROBOT_ENDPOINT # DataRobot URL DATAROBOT_API_TOKEN # API token (not recommended) ``` ## Exit codes | Code | Meaning | | --- | --- | | 0 | Success. | | 1 | General error. | | 2 | Command usage error. | | 130 | Interrupted (Ctrl+C). | --- # LLM model management URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/llm-gateway.html > List LLM Gateway models and DataRobot-deployed LLMs, and set the CLI's default LLM. List available large language models (LLMs) and configure which one the DataRobot CLI uses by default. ## Synopsis ``` dr llm-gateway [flags] dr llm [flags] # alias ``` ## Description The `llm-gateway` command group exposes two subcommands: - list —fetch available LLMs from two sources and display them as a table or JSON: active LLM gateway catalog models ( /api/v2/genai/llmgw/catalog/ ) and DataRobot-deployed LLMs ( /api/v2/deployments/ , deployments whose champion model is a text-generation model). A SOURCE column (or source field in JSON) distinguishes the two. - select —choose a default LLM, either by ID or through an interactive picker. The selection is persisted to drconfig.yaml and read by other CLI commands. Each source is best-effort: if one source can't be reached (for example, an empty LLM gateway on an on-prem install, or no deployment access), the command logs a warning and lists the other. It errors only when both sources fail. Aliases: `llm`, `llm-gateways` ## Subcommands ### list Fetch available LLMs—LLM gateway catalog models and DataRobot-deployed LLMs—and display them. ``` dr llm-gateway list [flags] dr llm ls # shortest alias ``` Flags: | Flag | Description | | --- | --- | | --output-format json | Emit machine-parseable JSON instead of a table. | Table columns: | Column | Description | | --- | --- | | ID | LLM identifier—a gateway model ID or a deployment ID. Prefixed with * if it's the current default, two spaces otherwise. | | NAME | Human-readable model name (a deployment's label for deployed LLMs). | | SOURCE | gateway for LLM gateway catalog models, deployed for DataRobot-deployed LLMs. | | PROVIDER | Model provider (for example azure, anthropic, google). - for deployed LLMs. | | MODEL | Underlying model identifier (for example azure/gpt-5-1-2025-11-13). - for deployed LLMs, since the deployment ID in ID is the routing key instead. | | CONTEXT | Context-window size in tokens. - when not reported (always - for deployed LLMs). | The table is sized to its content and capped at the terminal width to avoid overflow.`description` is long enough to wrap unreadably in a table, so it's included in JSON output only. JSON output ( `--output-format json`) returns an envelope with an `llms` array. Each entry includes: ``` { "id": "llm-abc123", "name": "GPT-4o", "source": "gateway", "provider": "azure", "model": "azure/gpt-5-1-2025-11-13", "description": "OpenAI's flagship multimodal model.", "context_size": 128000, "deployment_id": "", "selected": true } ``` For a deployed LLM, `source` is `deployed`, `deployment_id` carries the deployment ID, and `model` is the internal routing sentinel `datarobot/datarobot-deployed-llm`. Examples: ``` # Table view dr llm-gateway list # JSON output dr llm-gateway list --output-format json # Aliases dr llm list dr llm ls ``` ### select Set the default LLM. The chosen ID—a gateway model ID or a deployment ID—is written to `drconfig.yaml` under the key `default-llm-id` and is also readable through `DATAROBOT_CLI_DEFAULT_LLM_ID`. ``` dr llm-gateway select [llm-id] dr llm select [llm-id] # alias ``` Arguments: - llm-id (optional)—an ID from the ID column of dr llm list . When provided, it's validated against the available LLMs (gateway models and deployed LLMs) and persisted immediately. When omitted, an interactive picker opens. Interactive picker: - Arrow keys to navigate, / to filter by name. - Enter to confirm the selection. - Ctrl+C or Esc to cancel without saving. Examples: ``` # Interactive picker dr llm-gateway select # Set directly by ID dr llm-gateway select llm-abc123 # Error: ID not found among available LLMs dr llm-gateway select unknown-id # Error: LLM "unknown-id" not found ``` ## Global flags All `dr` [global flags](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html#global-flags) apply to `dr llm-gateway` subcommands: - -v, --verbose — Verbose output - --debug — Debug output - --skip-auth — Skip authentication checks (advanced; API calls may fail) - -h, --help — Help ## Configuration The selected LLM ID is stored in `drconfig.yaml`. It's a gateway model ID or a DataRobot deployment ID, depending on which was selected: ``` default-llm-id: llm-abc123 ``` It can also be set or overridden with an environment variable: ``` export DATAROBOT_CLI_DEFAULT_LLM_ID=llm-abc123 ``` `dr llm-gateway list` uses this value to mark the current default with `*` in the `ID` column. ## Authentication Both subcommands require valid DataRobot credentials. Run `dr auth login` first if you haven't already. --- # dr run URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/run.html > Execute tasks defined in application templates. Execute tasks defined in application templates. ## Synopsis The `dr run` command executes tasks defined in Taskfiles within DataRobot application templates. It automatically discovers component Taskfiles and aggregates them into a unified task execution environment. ``` dr run [TASK_NAME...] [flags] ``` ## Description The `run` command provides a convenient way to execute common application tasks such as starting development servers, running tests, building containers, and deploying applications. It works by discovering Taskfiles in your template directory and generating a consolidated task runner configuration. Key features: - Automatic discovery —finds and aggregates Taskfiles from template components. - Template validation —verifies you're in a DataRobot template directory. - Conflict detection —prevents dotenv directive conflicts in nested Taskfiles. - Parallel execution —run multiple tasks simultaneously. - Watch mode —automatically re-run tasks when files change. ## Template requirements To use `dr run`, your directory must meet these requirements: 1. Contains a .env file —indicates you're in a DataRobot template directory. 2. Contains Taskfiles —component directories with Taskfile.yaml or Taskfile.yml files. 3. No dotenv conflicts —component Taskfiles cannot have their own dotenv directives. If these requirements aren't met, the command provides clear error messages explaining the issue. ## Options ``` -l, --list List all available tasks -d, --dir string Directory to look for tasks (default ".") -p, --parallel Run tasks in parallel -C, --concurrency int Number of concurrent tasks to run (default 2) -w, --watch Enable watch mode for the given task -y, --yes Assume "yes" as answer to all prompts -x, --exit-code Pass-through the exit code of the task command -s, --silent Disable echoing -h, --help Help for run ``` ## Global options ``` -v, --verbose Enable verbose output --debug Enable debug output ``` ## Examples ### List available tasks ``` dr run --list ``` Output: ``` Available tasks: * dev Start development server * test Run tests * lint Run linters * build Build Docker container * deploy Deploy to DataRobot ``` ### Run a single task ``` dr run dev ``` Starts the development server defined in your template's Taskfile. ### Run multiple tasks sequentially ``` dr run lint test ``` Runs the lint task, then the test task in sequence. ### Run multiple tasks in parallel ``` dr run lint test --parallel ``` Runs lint and test tasks simultaneously. ### Run with watch mode ``` dr run dev --watch ``` Runs the development server and automatically restarts it when source files change. ### Control concurrency ``` dr run task1 task2 task3 --parallel --concurrency 3 ``` Runs up to 3 tasks concurrently. ### Silent execution ``` dr run build --silent ``` Runs the build task without echoing commands. ### Pass-through exit codes ``` dr run test --exit-code ``` Exits with the same code as the task command (useful in CI/CD). ## Task discovery The `dr run` command discovers tasks in this order: 1. Check for .env file —verifies you're in a template directory. 2. Scan for Taskfiles —finds Taskfile.yaml or Taskfile.yml files up to 2 levels deep. 3. Validate dotenv directives —ensures component Taskfiles don't have conflicting dotenv declarations. 4. Generate Taskfile.gen.yaml —creates a unified task configuration. 5. Execute tasks —runs the requested tasks using the task binary. ### Directory structure ``` my-template/ ├── .env # Required: template marker ├── Taskfile.gen.yaml # Generated: consolidated tasks ├── backend/ │ ├── Taskfile.yaml # Component tasks (no dotenv) │ └── src/ └── frontend/ ├── Taskfile.yaml # Component tasks (no dotenv) └── src/ ``` ### Generated Taskfile The CLI generates `Taskfile.gen.yaml` with this structure: ``` version: '3' dotenv: [".env"] includes: backend: taskfile: ./backend/Taskfile.yaml dir: ./backend frontend: taskfile: ./frontend/Taskfile.yaml dir: ./frontend ``` This allows you to run component tasks with prefixes: ``` dr run backend:build dr run frontend:dev ``` ## Error handling ### Not in a template directory If you run `dr run` outside a DataRobot template: ``` You don't seem to be in a DataRobot Template directory. This command requires a .env file to be present. ``` Solution: Navigate to a template directory or run `dr templates setup` to create one. ### Dotenv directive conflict If a component Taskfile has its own `dotenv` directive: ``` Error: Cannot generate Taskfile because an existing Taskfile already has a dotenv directive. existing Taskfile already has dotenv directive: backend/Taskfile.yaml ``` Solution: Remove the `dotenv` directive from component Taskfiles. The generated `Taskfile.gen.yaml` handles environment variables. ### Task binary not found If the `task` binary isn't installed: ``` "task" binary not found in PATH. Please install Task from https://taskfile.dev/installation/ ``` Solution: Install Task following the instructions at [taskfile.dev/installation](https://taskfile.dev/installation/). ### No tasks found If no Taskfiles exist in component directories: ``` file does not exist Error: failed to list tasks: exit status 1 ``` Solution: Add Taskfiles to your template components or use `dr templates clone` to start with a pre-configured template. ## Task definitions Tasks are defined in component `Taskfile.yaml` files using Task's syntax. ### Basic task ``` version: '3' tasks: dev: desc: Start development server cmds: - python -m uvicorn src.app.main:app --reload ``` ### Task with dependencies ``` tasks: build: desc: Build Docker container cmds: - docker build -t {{.APP_NAME}} . deploy: desc: Deploy application deps: [build] cmds: - docker push {{.APP_NAME}} - kubectl apply -f deploy.yaml ``` ### Task with environment variables ``` tasks: test: desc: Run tests with coverage env: PYTEST_ARGS: "--cov=src --cov-report=html" cmds: - pytest {{.PYTEST_ARGS}} ``` ### Task with preconditions ``` tasks: deploy: desc: Deploy to production preconditions: - sh: test -f .env msg: ".env file is required" - sh: test -n "$DATAROBOT_ENDPOINT" msg: "DATAROBOT_ENDPOINT must be set" cmds: - ./deploy.sh ``` ## Best practices ### Descriptive task names Use clear, action-oriented task names: ``` tasks: dev: # ✅ Clear and concise desc: Start development server test:unit: # ✅ Namespaced for organization desc: Run unit tests lint:python: # ✅ Specific and descriptive desc: Run Python linters ``` ### Useful descriptions Provide helpful task descriptions: ``` tasks: deploy: desc: Deploy application to DataRobot (requires authentication) cmds: - ./deploy.sh ``` ### Common task names Use standard names for common operations: - dev —start development server. - build —build application or container. - test —run test suite. - lint —run linters and formatters. - deploy —deploy to target environment. - clean —clean build artifacts. ### Environment variable usage Reference `.env` variables in tasks: ``` tasks: deploy: desc: Deploy {{.APP_NAME}} to {{.DEPLOYMENT_TARGET}} cmds: - echo "Deploying to $DATAROBOT_ENDPOINT" - ./deploy.sh ``` ### Silent tasks Use `silent: true` for tasks that don't need output: ``` tasks: check:version: desc: Check CLI version silent: true cmds: - dr version ``` ## Integration with other commands ### With dr templates ``` # Clone and set up template dr templates clone python-streamlit my-app cd my-app # Configure environment dr dotenv setup # Run tasks dr run dev ``` ### With dr dotenv ``` # Update environment variables dr dotenv setup # Run with updated configuration dr run deploy ``` ### In CI/CD pipelines ``` #!/bin/bash # ci-pipeline.sh set -e # Run tests dr run test --exit-code --silent # Run linters dr run lint --exit-code --silent # Build dr run build --silent ``` ## Troubleshooting ### Tasks not found Problem: `dr run --list` shows no tasks. Causes: - No Taskfiles in component directories. - Taskfiles at wrong depth (deeper than 2 levels). Solution: ``` # Check for Taskfiles find . -name "Taskfile.y*ml" -maxdepth 3 # Ensure Taskfiles are in component directories # Correct: ./backend/Taskfile.yaml # Wrong: ./backend/src/Taskfile.yaml ``` ### Environment variables not loading Problem: Tasks can't access environment variables. Causes: - Missing `.env` file. - Variables not exported. Solution: ``` # Verify .env exists ls -la .env # Check variables are set source .env env | grep DATAROBOT ``` ### Task execution fails Problem: Task runs but fails with errors. Solution: ``` # Enable verbose output dr run task-name --verbose # Enable debug output dr run task-name --debug # Check task definition cat component/Taskfile.yaml ``` ### Permission denied errors Problem: Tasks fail with permission errors. Solution: ``` # Make scripts executable chmod +x scripts/*.sh # Check file permissions ls -l scripts/ ``` ## See also - Template system overview —understanding templates. - Task definitions —creating Taskfiles. - Environment variables —managing configuration. - dr dotenv —environment variable management. - Task documentation —official Task runner docs. --- # CLI utility commands URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/self.html > Commands for managing and configuring the DataRobot CLI itself. Commands for managing and configuring the DataRobot CLI itself. ## Synopsis ``` dr self ``` ## Description The `self` command provides utility functions for managing the CLI tool itself, including updating to the latest version, checking version information, and setting up shell completion. ## Subcommands ### completion Generate or manage shell completion scripts for command auto-completion. ``` dr self completion ``` See the [completion documentation](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/completion.html) for detailed usage. Quick examples: ``` # Install completions interactively dr self completion install # Generate completions for bash dr self completion bash > /etc/bash_completion.d/dr # Generate completions for zsh dr self completion zsh > "${fpath[1]}/_dr" ``` ### config Display current configuration settings from config file and environment variables. ``` dr self config ``` This command shows all configuration values currently in use by the CLI, including settings from: - Configuration file ( ~/.config/datarobot/drconfig.yaml or custom path) - Environment variables (prefixed with DATAROBOT_CLI_ or standard DATAROBOT_ variables) - Command-line flags Sensitive values like API tokens are automatically redacted for security. Examples: ``` # Display current configuration dr self config ``` Sample output: ``` Configuration initialized. Using config file: /Users/username/.config/datarobot/drconfig.yaml debug: false endpoint: https://app.datarobot.com/api/v2 external_editor: vim token: **** verbose: false ``` Use cases: - Verify which configuration file is being used - Check that environment variables are being recognized - Debug configuration issues - Confirm API endpoint and settings before deployment ### update Update the DataRobot CLI to the latest version. ``` dr self update ``` This command automatically detects your installation method and uses the appropriate update mechanism: - Homebrew (macOS) —uses brew update && upgrade dr-cli if installed via Homebrew - Windows —runs the PowerShell installation script - macOS/Linux —runs the shell installation script The update process will download and install the latest version while preserving your configuration and credentials. Examples: ``` # Update to latest version dr self update ``` Note: This command requires an active internet connection and appropriate permissions to install software on your system. ### version Display version information about the CLI. ``` dr self version ``` Options: - -f, --format —output format ( text or json ) Examples: ``` # Show version (default text format) dr self version # Show version in JSON format dr self version --format json ``` ## Global flags All `dr` global flags are available: - -v, --verbose —enable verbose output - --debug —enable debug output - -h, --help —show help information ## Examples ### Update CLI to latest version ``` $ dr self update Downloading latest version... Installing DataRobot CLI... ✓ Successfully updated to version 1.1.0 ``` ### Check CLI version ``` $ dr self version DataRobot CLI version: 1.0.0 ``` ### View current configuration ``` $ dr self config Configuration initialized. Using config file: /Users/username/.config/datarobot/drconfig.yaml debug: false endpoint: https://app.datarobot.com/api/v2 external_editor: vim token: **** verbose: false ``` ### Install shell completions ``` # Interactive installation $ dr self completion install ✓ Detected shell: zsh ✓ Installing completions to: ~/.zsh/completions/_dr ✓ Completions installed successfully! # Manual installation $ dr self completion bash | sudo tee /etc/bash_completion.d/dr ``` ### Get version in JSON ``` $ dr self version --format json { "version": "1.0.0", "commit": "abc123", "buildDate": "2025-11-10T12:00:00Z" } ``` ## See also - Shell completions guide —detailed completion setup - Completion command —completion command reference - Getting started —initial CLI setup --- # dr start URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/start.html > Run the application quickstart process for the current template. Run the application quickstart process for the current template. ## Synopsis The `start` command (also available as `quickstart`) provides an automated way to initialize and launch your DataRobot application. It performs several checks and either executes a template-specific quickstart script or seamlessly launches the interactive template setup wizard. It is the main entry point for setting up [Agentic AI](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html) workflows—run `dr start` from the get started guide to clone and configure your agent template. ``` dr start [flags] ``` ## Aliases - dr start - dr quickstart ## Description The `start` command streamlines the process of getting your DataRobot application up and running. It automates the following workflow: 1. Prerequisite checks —verifies that required tools are installed and validates your environment. 2. CLI version check —verifies your CLI version meets the template's minimum requirements. 3. Repository check —checks if you're in a DataRobot repository (if not, launches template setup). 4. Execution —executes a start command in this order: 5. Taskfile —runs task start from the Taskfile (if available). 6. Quickstart script —runs a script from .datarobot/cli/bin/ (if available). 7. Fallback —launches the interactive dr templates setup wizard if neither exists. This command is designed to work intelligently with your template's structure. If you're not in a DataRobot repository or no Taskfile/quickstart exists, the command gracefully falls back to the standard setup wizard. ## Flags ``` -y, --yes Skip confirmation prompts and execute immediately -h, --help Show help information ``` ### Global flags All [global flags](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html#global-flags) are also available. ## Quickstart scripts ### Location Quickstart scripts must be placed in: ``` .datarobot/cli/bin/ ``` ### Naming convention Scripts must start with `quickstart` (case-sensitive): - ✅ quickstart - ✅ quickstart.sh - ✅ quickstart.py - ✅ quickstart-dev - ❌ Quickstart.sh (wrong case) - ❌ start.sh (wrong name) If there are multiple scripts matching the pattern, the first one found in lexicographical order will be executed. ### Platform-specific requirements Unix/Linux/macOS: - Script must have executable permissions ( chmod +x ) - Can be any executable file (shell script, Python, compiled binary, etc.) Windows: - Must have executable extension: .exe , .bat , .cmd , or .ps1 ## Examples ### Basic usage Run the quickstart process interactively: ``` dr start ``` If a quickstart script is found: ``` DataRobot Quickstart ✓ Starting application quickstart process... ✓ Checking template prerequisites... ✓ Locating quickstart script... → Executing quickstart script... Quickstart found at: .datarobot/cli/bin/quickstart.sh. Will proceed with execution... Press 'y' or ENTER to confirm, 'n' to cancel ``` If no quickstart script is found: ``` DataRobot Quickstart ✓ Starting application quickstart process... ✓ Checking template prerequisites... ✓ Locating quickstart script... → Executing quickstart script... No quickstart script found. Will proceed with template setup... ``` The command will then seamlessly launch the interactive setup wizard. ### Non-interactive mode Skip all prompts and execute immediately: ``` dr start --yes ``` or ``` dr start -y ``` This is useful for: - CI/CD pipelines - Automated deployments - Scripted workflows ### Using the alias ``` dr quickstart ``` ## Behavior ### State tracking The `dr start` command automatically tracks when it runs successfully by updating a state file with: - Timestamp of when the command last started (ISO 8601 format) - CLI version used This state information is stored in `.datarobot/cli/state.yaml` within the template directory. State tracking is automatic and transparent. No manual intervention is required. The state file helps other commands (like `dr templates setup`) know that you've already run `dr start`, allowing them to skip redundant setup steps. ### Execution order The command tries the following in order: 1. task start —if a Taskfile with a start task exists, runs it. 2. Quickstart script —if a script is found in .datarobot/cli/bin/ , runs it (after user confirmation unless --yes is used). 3. Setup wizard —if neither is available (or not in a DataRobot repository), launches dr templates setup . ### When a quickstart script runs 1. No task start in Taskfile (or no Taskfile), but script is detected in .datarobot/cli/bin/ 2. User is prompted for confirmation (unless --yes or -y is used) 3. If user confirms (or --yes is specified), script executes with full terminal control 4. Command completes when script finishes 5. State file is updated with current timestamp and CLI version If the user declines to execute the script, the command exits gracefully and still updates the state file. ### When setup wizard runs 1. No task start and no quickstart script (or not in a DataRobot repository) 2. User is notified 3. User is prompted for confirmation (unless --yes or -y is used) 4. If user confirms (or --yes is specified), interactive dr templates setup wizard launches automatically 5. User completes template configuration through the wizard 6. State file is updated with current timestamp and CLI version If the user declines, the command exits gracefully and still updates the state file. ### Prerequisites checked Before proceeding, the command verifies: - ✅ Required tools are installed (Git, etc.) When searching for a quickstart script, the command checks: - ✅ Current directory is within a DataRobot repository (contains .datarobot/ directory) If the repository check fails, the command automatically launches the template setup wizard instead of exiting with an error. ## Error handling ### Not in a DataRobot repository If you're not in a DataRobot repository, the command automatically launches the template setup wizard: ``` $ dr start # Automatically launches: dr templates setup ``` No manual intervention is needed - the command handles this gracefully. ### Missing prerequisites ``` $ dr start Error: required tool 'git' not found # Solution: Install the missing tool ``` ### Script execution failure If a quickstart script fails, the error is displayed and the command exits. Check the script's output for details. ## When to use dr start ### ✅ Good use cases - First-time setup —initializing a newly cloned template or starting from scratch. - Quick restart —restarting development after a break. - Onboarding —helping new team members get started quickly. - CI/CD —automating application initialization in pipelines. - General entry point —universal command that works whether you have a template or not. ### ❌ When not to use - Making configuration changes —use dr dotenv to modify environment variables. - Running specific tasks —use dr run for targeted task execution. ## See also - dr templates setup —interactive template setup wizard. - dr run —execute specific application tasks. - dr dotenv —manage environment configuration. - Template Structure —understanding template organization. ## Tips ### Creating a custom quickstart script 1. Create the directory structure: ``` mkdir -p .datarobot/cli/bin ``` 1. Create your script: ``` # Create the script cat > .datarobot/cli/bin/quickstart.sh <<'EOF' #!/bin/bash echo "Starting my custom quickstart..." dr run build dr run dev EOF ``` 1. Make it executable: ``` chmod +x .datarobot/cli/bin/quickstart.sh ``` 1. Test it: ``` dr start --yes ``` ### Best practices - Keep scripts simple —focus on essential initialization steps. - Provide clear output —use echo statements to show progress. - Handle errors gracefully —use set -e in bash scripts to exit on errors. - Check prerequisites —verify .env exists and required tools are installed. - Make it idempotent —script should be safe to run multiple times. - Document behavior —add comments explaining what the script does. --- # dr task URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/task.html > Manage Taskfile composition and task execution for DataRobot templates. Manage Taskfile composition and task execution for DataRobot templates. ## Synopsis ``` dr task [command] [flags] ``` ## Description The `task` command group provides utilities for working with Taskfiles in DataRobot application templates. It includes subcommands for composing unified Taskfiles from multiple component Taskfiles and executing tasks. In [Agentic AI](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html) workflows, you use `dr task run deploy` to deploy agents and `dr task run dev` (or `task dev`) for local development. ## Subcommands - compose —compose a unified Taskfile from component Taskfiles. - list —list available tasks. - run —execute template tasks. ## dr task compose Generate a root `Taskfile.yaml` by discovering and aggregating Taskfiles from subdirectories. ### Synopsis ``` dr task compose [flags] ``` ### Description The `compose` command automatically discovers Taskfiles in component directories and generates a unified root `Taskfile.yaml` that includes all components and aggregates common tasks. This allows you to run tasks across multiple components from a single entry point. Key features: - Automatic discovery —finds Taskfiles up to 2 levels deep in subdirectories. - Task aggregation —discovers common tasks (lint, install, dev, deploy) and creates top-level tasks that delegate to components. - Template support —uses customizable Go templates for flexible Taskfile generation. - Auto-discovery —automatically detects .Taskfile.template in the root directory. - Gitignore integration —adds generated Taskfile to .gitignore automatically. ### Options ``` -t, --template string Path to custom Taskfile template -h, --help Help for compose ``` ### Global options ``` -v, --verbose Enable verbose output --debug Enable debug output ``` ### Template requirements To use `dr task compose`, your directory must meet these requirements: 1. Contains a .env file —indicates you're in a DataRobot template directory. 2. Contains component Taskfiles —subdirectories with Taskfile.yaml or Taskfile.yml files. 3. No dotenv conflicts —component Taskfiles cannot have their own dotenv directives. ### Directory structure Expected directory structure: ``` my-template/ ├── .env # Required: template marker ├── .Taskfile.template # Optional: custom template ├── .taskfile-data.yaml # Optional: template configuration ├── Taskfile.yaml # Generated: unified taskfile ├── backend/ │ ├── Taskfile.yaml # Component tasks │ └── src/ ├── frontend/ │ ├── Taskfile.yaml # Component tasks │ └── src/ └── infra/ ├── Taskfile.yaml # Component tasks └── terraform/ ``` ### Examples #### Basic composition Generate a Taskfile using the default embedded template: ``` dr task compose ``` Output: ``` Generated file saved to: Taskfile.yaml Added /Taskfile.yaml line to .gitignore ``` This creates a `Taskfile.yaml` with: - Environment configuration - Includes for all discovered components - Aggregated tasks (lint, install, dev, deploy) #### With auto-discovered template If `.Taskfile.template` exists in the root directory: ``` dr task compose ``` Output: ``` Using auto-discovered template: .Taskfile.template Generated file saved to: Taskfile.yaml ``` The command automatically uses your custom template. #### With custom template Specify a custom template explicitly: ``` dr task compose --template templates/custom.yaml ``` This uses your custom template for generation instead of the embedded default. #### Template in subdirectory ``` dr task compose --template .datarobot/taskfile.template ``` ### Generated Taskfile structure The default generated `Taskfile.yaml` includes: ``` --- # https://taskfile.dev version: '3' env: ENV: testing dotenv: ['.env', '.env.{{.ENV}}'] includes: backend: taskfile: ./backend/Taskfile.yaml dir: ./backend frontend: taskfile: ./frontend/Taskfile.yaml dir: ./frontend infra: taskfile: ./infra/Taskfile.yaml dir: ./infra tasks: default: desc: "ℹ️ Show all available tasks (run `task --list-all` to see hidden tasks)" cmds: - task --list --sort none silent: true start: desc: "💻 Prepare local development environment" cmds: - dr dotenv setup - task: install lint: desc: "🧹 Run linters" cmds: - task: backend:lint - task: frontend:lint install: desc: "🛠️ Install all dependencies" cmds: - task: backend:install - task: frontend:install - task: infra:install test: desc: "🧪 Run tests across all components" cmds: - task: backend:test - task: frontend:test dev: desc: "🚀 Run all services together" cmds: - | task backend:dev & sleep 3 task frontend:dev & sleep 8 echo "✅ All servers started!" wait deploy: desc: "🚀 Deploy all services" cmds: - task: infra:deploy - task: backend:deploy deploy-dev: desc: "🚀 Deploy all services to development" cmds: - task: infra:deploy-dev - task: backend:deploy-dev ``` ### Task aggregation The compose command discovers these common tasks in component Taskfiles: - lint —code linting and formatting. - install —dependency installation. - test —running test suites. - dev —development server startup. - deploy —production deployment operations. - deploy-dev —development deployment operations. For each discovered task type, it creates a top-level task that delegates to all components that have that task. ### Custom templates Create a custom template to control the generated Taskfile structure. #### Template file example Save as `.Taskfile.template`: ``` --- version: '3' env: ENV: production dotenv: ['.env', '.env.{{.ENV}}'] includes: {{- range .Includes }} {{ .Name }}: taskfile: {{ .Taskfile }} dir: {{ .Dir }} {{- end }} tasks: default: desc: "Show available tasks" cmds: - task --list {{- if .HasLint }} lint: desc: "Run linters" cmds: {{- range .LintComponents }} - task: {{ . }}:lint {{- end }} {{- end }} {{- if .HasInstall }} install: desc: "Install dependencies" cmds: {{- range .InstallComponents }} - task: {{ . }}:install {{- end }} {{- end }} {{- if .HasTest }} test: desc: "Run tests" cmds: {{- range .TestComponents }} - task: {{ . }}:test {{- end }} {{- end }} # Custom task check: desc: "Run all checks" cmds: - task: lint - task: test ``` #### Template variables Templates have access to these variables: Includes (array): - `.Name` —component name (e.g., "backend"). - `.Taskfile` —relative path to Taskfile (e.g., "./backend/Taskfile.yaml"). - `.Dir` —relative directory path (e.g., "./backend"). Task flags (boolean): - `.HasLint` —true if any component has a lint task. - `.HasInstall` —true if any component has an install task. - `.HasTest` —true if any component has a test task. - `.HasDev` —true if any component has a dev task. - `.HasDeploy` —true if any component has a deploy task. - `.HasDeployDev` —true if any component has a deploy-dev task. Task components (arrays): - `.LintComponents` —component names with lint tasks. - `.InstallComponents` —component names with install tasks. - `.TestComponents` —component names with test tasks. - `.DevComponents` —component names with dev tasks. - `.DeployComponents` —component names with deploy tasks. - `.DeployDevComponents` —component names with deploy-dev tasks. Development ports (array): - `.DevPorts[].Name` —service name. - `.DevPorts[].Port` —port number. #### Example: minimal template ``` version: '3' dotenv: ['.env'] includes: {{- range .Includes }} {{ .Name }}: taskfile: {{ .Taskfile }} dir: {{ .Dir }} {{- end }} tasks: default: cmds: - task --list ``` #### Example: extensive aggregation ``` version: '3' dotenv: ['.env'] includes: {{- range .Includes }} {{ .Name }}: taskfile: {{ .Taskfile }} dir: {{ .Dir }} {{- end }} tasks: {{- if .HasLint }} lint: desc: "Run all linters" cmds: {{- range .LintComponents }} - task: {{ . }}:lint {{- end }} {{- end }} {{- if .HasInstall }} install: desc: "Install all dependencies" cmds: {{- range .InstallComponents }} - task: {{ . }}:install {{- end }} {{- end }} {{- if .HasTest }} test: desc: "Run all tests" cmds: {{- range .TestComponents }} - task: {{ . }}:test {{- end }} {{- end }} {{- if .HasDev }} dev: desc: "Start all services" cmds: {{- range .DevComponents }} - task: {{ . }}:dev {{- end }} {{- end }} {{- if .HasDeploy }} deploy: desc: "Deploy to production" cmds: {{- range .DeployComponents }} - task: {{ . }}:deploy {{- end }} {{- end }} {{- if .HasDeployDev }} deploy-dev: desc: "Deploy to development" cmds: {{- range .DeployDevComponents }} - task: {{ . }}:deploy-dev {{- end }} {{- end }} ci: desc: "Run CI pipeline" cmds: - task: lint - task: test - task: build ``` ### Gitignore integration The compose command automatically adds the generated Taskfile to `.gitignore`: ``` /Taskfile.yaml ``` This prevents committing the generated file to version control. Each developer generates their own version based on their local component structure. If you want to commit the generated Taskfile, remove it from `.gitignore`. ### Error handling #### Not in a template directory ``` You don't seem to be in a DataRobot Template directory. This command requires a .env file to be present. ``` Solution: Navigate to a template directory or run `dr templates setup`. #### No Taskfiles found ``` no Taskfiles found in child directories ``` Solution: Add Taskfiles to component directories or adjust your directory structure. #### Dotenv conflict ``` Error: Cannot generate Taskfile because an existing Taskfile already has a dotenv directive. existing Taskfile already has dotenv directive: backend/Taskfile.yaml ``` Solution: Remove `dotenv` directives from component Taskfiles. The root Taskfile handles environment loading. #### Template not found ``` Error: template file not found: /path/to/template.yaml ``` Solution: Check the template path and ensure the file exists. ### Best practices #### Keep components independent Each component Taskfile should be self-contained: ``` # backend/Taskfile.yaml version: '3' tasks: dev: desc: Start backend server cmds: - python -m uvicorn src.app.main:app --reload test: desc: Run tests cmds: - pytest lint: desc: Run linters cmds: - ruff check . - mypy . ``` #### Use consistent task names Use the same task names across components for automatic aggregation: - lint —linting. - install —dependency installation. - test —testing. - dev —development server. - build —building artifacts. - deploy —production deployment. - deploy-dev —development deployment. #### Commit custom templates If using a custom template, commit it to version control: ``` git add .Taskfile.template git commit -m "Add custom Taskfile template" ``` #### Configure development ports Optionally create a `.taskfile-data.yaml` file to display service URLs in the dev task. See [Taskfile data configuration](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/task.html#taskfile-data-configuration) for complete documentation. #### Document custom variables If your template uses custom variables, document them: ``` # .Taskfile.template # # Custom variables: # - PROJECT_NAME: Set in .env # - DEPLOY_TARGET: Set in .env # version: '3' # ... ``` #### Test template changes After modifying a template, regenerate and test: ``` dr task compose --template .Taskfile.template task --list task dev ``` ### Taskfile data configuration Template authors can provide additional configuration for Taskfile generation by creating a `.taskfile-data.yaml` file in the template root directory. #### File location ``` my-template/ ├── .env ├── .taskfile-data.yaml # Configuration file ├── Taskfile.yaml # Generated └── components/ ``` #### Configuration format ``` # .taskfile-data.yaml # Optional configuration for dr task compose # Development server ports # Displayed when running the dev task ports: - name: Backend API port: 8080 - name: Frontend port: 5173 - name: Worker Service port: 8842 - name: MCP Server port: 9000 ``` #### Port configuration Purpose: The `ports` array allows template authors to specify which ports their services use. When developers run `task dev`, they see URLs for each service. Example output: When developers run `task dev` with port configuration: ``` task mcp_server:dev & sleep 3 task web:dev & sleep 3 task writer_agent:dev & sleep 3 task frontend_web:dev & sleep 8 ✅ All servers started! 🔗 Backend API: http://localhost:8080 🔗 Frontend: http://localhost:5173 🔗 Worker Service: http://localhost:8842 🔗 MCP Server: http://localhost:9000 ``` DataRobot Notebook integration: The generated dev task automatically detects DataRobot Notebook environments and adjusts URLs: ``` 🔗 Backend API: https://app.datarobot.com/notebook-sessions/abc123/ports/8080 🔗 Frontend: https://app.datarobot.com/notebook-sessions/abc123/ports/5173 ``` This happens automatically when the `NOTEBOOK_ID` environment variable is present. Benefits: - Improved onboarding —new developers immediately know where services are running. - Self-documenting —ports are visible in generated Taskfile and command output. - Notebook support —URLs work correctly in DataRobot Notebooks. - Reduced confusion —no need to check logs or documentation for port numbers. Best practices: 1. List all services —include every service that starts in dev mode. 2. Use descriptive names —"Backend API" is clearer than "Backend". 3. Match actual ports —ensure ports match what's in component Taskfiles. 4. Update when changing —keep configuration in sync with service changes. #### When to use this file Use `.taskfile-data.yaml` when: - Your template has multiple services with different ports. - Services use non-standard ports that aren't obvious. - You want to improve developer experience. - Your template targets DataRobot Notebooks. You can skip it when: - Your template has a single service. - Ports are obvious or standard (e.g., 3000 for Node.js). - You use custom Taskfile templates with hardcoded values. - Port information is already well-documented elsewhere. #### File is optional The `.taskfile-data.yaml` file is completely optional. If not present: - The dev task still works correctly. - Services start normally. - Port URLs simply aren't displayed. This allows template authors to add port configuration incrementally without breaking existing templates. #### Future extensibility The `.taskfile-data.yaml` file uses an extensible format. Future CLI versions may support additional configuration options such as: - Custom environment variables for templates. - Service metadata (descriptions, dependencies). - Deployment configuration. - Build optimization hints. Template authors can future-proof their templates by using this configuration file even if only specifying ports initially. #### Example templates Minimal example: ``` # .taskfile-data.yaml ports: - name: App port: 8000 ``` Full-stack application: ``` # .taskfile-data.yaml ports: - name: Backend API port: 8080 - name: Frontend port: 5173 - name: Database Admin port: 8081 - name: Redis Commander port: 8082 ``` Microservices architecture: ``` # .taskfile-data.yaml ports: - name: API Gateway port: 8080 - name: Auth Service port: 8081 - name: User Service port: 8082 - name: Order Service port: 8083 - name: Frontend port: 3000 - name: Admin Dashboard port: 3001 ``` ### Workflow integration #### Initial setup ``` # Clone template dr templates clone python-fullstack my-app cd my-app # Set up environment dr dotenv setup # Generate Taskfile dr task compose # View available tasks task --list ``` #### Development workflow ``` # Add new component mkdir new-service cat > new-service/Taskfile.yaml << 'EOF' version: '3' tasks: dev: desc: Start new service cmds: - echo "Starting service..." EOF # Regenerate Taskfile dr task compose # Run all services task dev ``` #### Template updates When components change: ``` # Regenerate Taskfile dr task compose # Verify new structure task --list ``` ## dr task list List all available tasks from composed Taskfile. ### Synopsis ``` dr task list [flags] ``` ### Description Lists all tasks available in the current template, including tasks from all component Taskfiles. ### Examples ``` # List all tasks dr task list # Show with full task tree task --list-all ``` ## dr task run Execute template tasks. This is an alias for `dr run`. ### Synopsis ``` dr task run [TASK_NAME...] [flags] ``` ### Description Execute one or more tasks defined in component Taskfiles. See [dr run](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/run.html) for full documentation. ### Examples ``` # Run single task dr task run dev # Run multiple tasks dr task run lint test # Run in parallel dr task run lint test --parallel ``` ## See also - dr run —task execution documentation. - Template system —template structure overview. - Environment variables —configuration management. - Task documentation —official Task runner documentation. --- # Template management URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/templates.html > List, clone, set up, and check status of DataRobot application templates. ## Template management Manage DataRobot application templates: list available templates, clone repositories, run the interactive setup wizard, and check template status. ## Synopsis ``` dr templates [arguments] [flags] ``` ## Description The `templates` command provides subcommands for discovering, cloning, and configuring application templates from your DataRobot instance. Templates are pre-configured application scaffolds that you customize into your own application. ## Subcommands ### list List available templates from your DataRobot instance. ``` dr templates list ``` Behavior: - Requires authentication. Run dr auth login if not already authenticated. - Fetches templates from the DataRobot API that are available to your user and organization. - Displays template names and descriptions. Example: ``` $ dr templates list Available templates: * python-streamlit - Streamlit application template * react-frontend - React frontend template * fastapi-backend - FastAPI backend template ``` Flags: All [global flags](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html#global-flags) apply (e.g., `--verbose`, `--debug`, `--skip-auth`). ### setup Run the interactive setup wizard to select, clone, and configure a template. ``` dr templates setup ``` Behavior: 1. Displays a list of templates available to you. 2. You select a template and specify a directory name; the wizard clones the template there. 3. Guides you through environment configuration (or skips it if already completed; see State tracking ). 4. Optionally runs dr dotenv setup if the template has environment prompts. For details on the template system and configuration wizard, see [Template system](https://docs.datarobot.com/en/docs/agentic-ai/cli/template-system/index.html) and [Interactive configuration](https://docs.datarobot.com/en/docs/agentic-ai/cli/template-system/interactive-config.html). ### Template selection screen When the setup wizard runs, you see a template list screen: - Navigate the list with the arrow keys (↑/↓). - Filter templates by pressing / and typing a search term. - Select a template by pressing Enter . - At the next prompt, enter the desired directory name for the cloned template and press Enter to clone. Only templates that are available to your user account are shown. After cloning, the wizard continues with configuration steps as needed. Flags: All [global flags](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html#global-flags) apply. Use `--force-interactive` to run the wizard even if setup was previously completed. ### clone Clone a template repository by name (and optional target directory). ``` dr templates clone [directory] ``` Arguments: - template-name — Name or identifier of the template (as shown by dr templates list ). - directory (optional) — Local directory name or path for the clone. If omitted, the template name is used. Examples: ``` # Clone into a directory named after the template dr templates clone python-streamlit # Clone into a custom directory dr templates clone python-streamlit my-app ``` Behavior: - Requires authentication. - Clones the template's Git repository to the current working directory. - After cloning, run dr dotenv setup to configure environment variables, or use dr templates setup to run the full wizard (which includes clone + configuration). ### status Show the current template's status (version, modifications, updates). ``` dr templates status ``` Behavior: - Must be run from within a cloned template directory. - Shows current version, latest available version (if applicable), modified files, and whether updates are available. Example: ``` $ dr templates status Template: python-streamlit Current version: 1.0.0 Modified files: .env (local only) ``` ## Global flags All `dr` [global flags](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html#global-flags) apply to `dr templates` subcommands: - -v, --verbose — Verbose output - --debug — Debug output (creates dr-tui-debug.log in current directory for TUI debugging) - --skip-auth — Skip authentication checks (advanced; API calls may fail) - --force-interactive — Force setup wizard to run even if already completed - -h, --help — Help ## Choosing templates vs. setup - Use dr templates setup when you want a guided, all-in-one flow: select template, clone, and configure in one session. Recommended for most users. - Use dr templates list and dr templates clone when you prefer to clone first and configure manually (e.g., dr dotenv setup afterward). ## See also - Template system — How templates are structured and configured - Interactive configuration — Wizard behavior and keyboard controls - Getting started — Initial setup and first template - dotenv — Environment variable management after cloning --- # Configuration files URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/configuration.html Understanding DataRobot CLI configuration files and settings. ## Configuration location The CLI stores configuration in a platform-specific location: | Platform | Location | | --- | --- | | Linux | ~/.config/datarobot/drconfig.yaml | | macOS | ~/.config/datarobot/drconfig.yaml | | Windows | %USERPROFILE%\.config\datarobot\drconfig.yaml | ## Configuration structure ### Main configuration file `~/.config/datarobot/drconfig.yaml`: ``` # DataRobot Connection endpoint: https://app.datarobot.com token: api key here ``` ### Environment-specific configs You can maintain multiple configurations: ``` # Development ~/.config/datarobot/dev-config.yaml # Staging ~/.config/datarobot/staging-config.yaml # Production ~/.config/datarobot/prod-config.yaml ``` Switch between them: ``` export DATAROBOT_CLI_CONFIG=~/.config/datarobot/dev-config.yaml dr templates list ``` ## Configuration options ### Connection settings ``` # Required: DataRobot instance URL endpoint: https://app.datarobot.com # Required: API authentication key token: api key here ``` ## Environment variables Override configuration with environment variables: ### Connection ``` # DataRobot endpoint URL export DATAROBOT_ENDPOINT=https://app.datarobot.com # API token (not recommended for security) export DATAROBOT_API_TOKEN=your_api_token ``` ### CLI behavior ``` # Custom config file path export DATAROBOT_CLI_CONFIG=~/.config/datarobot/custom-config.yaml # Editor for text editing export EDITOR=nano # Force setup wizard to run even if already completed export DATAROBOT_CLI_FORCE_INTERACTIVE=true ``` ### Environment variables reference | Variable | Purpose | Scope | | --- | --- | --- | | DATAROBOT_ENDPOINT | DataRobot instance URL (API endpoint) | Connection; overrides config | | DATAROBOT_API_TOKEN | API token (not recommended; prefer dr auth login) | Connection; overrides config | | DATAROBOT_CLI_CONFIG | Path to config file | CLI behavior | | DATAROBOT_CLI_FORCE_INTERACTIVE | Force setup wizard to run (e.g., true) | CLI behavior; setup/dotenv | | DATAROBOT_CLI_SKIP_AUTH | Skip authentication checks (e.g., true); advanced use | CLI behavior | | DATAROBOT_VERIFY_SSL | Disable SSL verification if false; not recommended for production | Auth / connection | | DR_TEMPLATES_DIR | Default directory for cloning templates; see Custom templates directory | Templates | | EDITOR | Editor used for text editing (e.g., vim, nano) | dotenv edit | ### Advanced flags The CLI supports advanced command-line flags for special use cases: ``` # Skip authentication checks (advanced users only) dr templates list --skip-auth # Force setup wizard to run (ignore completion state) dr templates setup --force-interactive # Enable verbose logging dr templates list --verbose # Enable debug logging dr templates list --debug ``` > ⚠️ Warning:The--skip-authflag bypasses all authentication checks and should only be used when you understand the implications. Commands requiring API access will likely fail without valid credentials. For the full list of global flags (including `--verbose`, `--debug`, and `--force-interactive`), see [Command reference - Global flags](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html#global-flags). ## Configuration priority Settings are loaded in order of precedence: 1. flags (command-line arguments, i.e. --config ) 2. environment variables (i.e. DATAROBOT_CLI_CONFIG=... ) 3. config files (i.e. ~/.config/datarobot/drconfig.yaml ) 4. defaults (built-in defaults) ## Security best practices ### 1. Protect configuration files ``` # Verify permissions (should be 600) ls -la ~/.config/datarobot/drconfig.yaml # Fix permissions if needed chmod 600 ~/.config/datarobot/drconfig.yaml chmod 700 ~/.config/datarobot/ ``` ### 2. Don't commit credentials Add to `.gitignore`: ``` # DataRobot credentials .config/datarobot/ drconfig.yaml *.yaml !.env.template ``` ### 3. Use environment-specific configs ``` # Never use production credentials in development # Keep separate config files ~/.config/datarobot/ ├── dev-config.yaml # Development ├── staging-config.yaml # Staging └── prod-config.yaml # Production ``` ### 4. Avoid environment variables for secrets ``` # ❌ Don't do this (visible in process list) export DATAROBOT_API_TOKEN=my_secret_token # Do this instead (use config file) dr auth login ``` ## Advanced configuration ### Custom templates directory ``` templates: default_clone_dir: ~/workspace/datarobot ``` Or via environment: ``` export DR_TEMPLATES_DIR=~/workspace/datarobot ``` ### Debugging configuration Enable debug logging: ``` debug: true ``` Or temporarily: ``` dr --debug templates list ``` ## Configuration examples ### Development environment `~/.config/datarobot/dev-config.yaml`: ``` endpoint: https://dev.datarobot.com token: api token for dev ``` Usage: ``` export DATAROBOT_CLI_CONFIG=~/.config/datarobot/dev-config.yaml dr templates list ``` ### Production environment `~/.config/datarobot/prod-config.yaml`: ``` endpoint: https://app.datarobot.com token: api key for prod ``` Usage: ``` export DATAROBOT_CLI_CONFIG=~/.config/datarobot/prod-config.yaml dr run deploy ``` ### Enterprise with proxy `~/.config/datarobot/enterprise-config.yaml`: ``` datarobot: endpoint: https://datarobot.enterprise.com token: enterprise_key proxy: http://proxy.enterprise.com:3128 verify_ssl: true ca_cert_path: /etc/ssl/certs/enterprise-ca.pem timeout: 120 preferences: log_level: warn ``` ## Troubleshooting ### Configuration not loading ``` # Check if config file exists ls -la ~/.config/datarobot/drconfig.yaml # Verify it's readable cat ~/.config/datarobot/drconfig.yaml # Check environment variables env | grep DATAROBOT ``` ### Invalid configuration ``` # The CLI will report syntax errors $ dr templates list Error: Failed to parse config file: yaml: line 5: could not find expected ':' # Fix syntax and try again vim ~/.config/datarobot/drconfig.yaml ``` ### Permission denied ``` # Fix file permissions chmod 600 ~/.config/datarobot/drconfig.yaml # Fix directory permissions chmod 700 ~/.config/datarobot/ ``` ### Multiple configs ``` # List all config files find ~/.config/datarobot -name "*.yaml" # Switch between them export DATAROBOT_CLI_CONFIG=~/.config/datarobot/dev-config.yaml ``` ## State tracking The CLI maintains state information about your interactions with repositories to provide a better user experience. State is tracked per-repository and stores metadata about command executions. ### What counts as a template directory A directory is treated as a DataRobot template directory when it contains a `.env` file (or, for some commands, a `.datarobot/` directory). This affects: - dr run — Requires a .env file in the current directory to discover and run tasks. - dr start — If not in a template directory, launches the template setup wizard instead of running a quickstart. - dr task compose / dr task list — Expect a template directory (e.g., with .env and component Taskfiles). - State tracking — State is stored per template directory in .datarobot/cli/state.yaml within that directory. Cloned templates created by `dr templates setup` or `dr templates clone` include `.datarobot/` and, after configuration, a `.env` file, so they are recognized automatically. ### State file location The CLI stores state locally within each template directory: - .datarobot/cli/state.yaml in the template directory (current working directory) ### Tracked information The state file tracks: - CLI version : Version of the CLI used for the last successful execution - Last start : Timestamp of the last successful dr start execution - Last dotenv setup : Timestamp of the last successful dr dotenv setup execution ### State file format ``` cli_version: "0.2.38" last_start: 2026-01-15T00:02:07.615186Z last_dotenv_setup: 2026-01-15T00:15:30.123456Z ``` All timestamps are in ISO 8601 format (UTC). ### How state is used - dr start : Updates state after successful execution - dr dotenv setup : Records when environment setup was completed - dr templates setup : Skips dotenv setup if it was already completed (based on state) ### Managing state State files are automatically created and updated. To reset state for a template directory: ``` # Remove template state rm .datarobot/cli/state.yaml ``` You can also force the wizard to run without deleting the state file by using the `--force-interactive` flag: ``` # Force re-execution of setup wizard while preserving state dr templates setup --force-interactive # Or via environment variable export DATAROBOT_CLI_FORCE_INTERACTIVE=true dr templates setup ``` This flag makes commands behave as if setup has never been completed, while still updating the state file. This is useful for: - Testing setup flows - Forcing reconfiguration without losing state history - Development and debugging State files are small and do not require manual management under normal circumstances. Each repository maintains its own state independently. ## See also - Getting started —initial setup. - Authentication —managing credentials. --- # Authentication flow URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/development/authentication.html > Reusable authentication mechanism for DataRobot CLI commands. ## Overview The CLI provides a reusable authentication mechanism that you can use with any command that requires valid DataRobot credentials. Authentication is handled using Cobra's `PreRunE` hooks, which ensure credentials are valid before a command executes. ## Using authentication in commands ### PreRunE hook (recommended) The recommended approach is to use the `auth.EnsureAuthenticatedE()` function in your command's `PreRunE` hook: ``` import "github.com/datarobot/cli/cmd/auth" var MyCmd = &cobra.Command{ Use: "mycommand", Short: "My command description", PreRunE: func(_ *cobra.Command, _ []string) error { return auth.EnsureAuthenticatedE() }, Run: func(_ *cobra.Command, _ []string) { // Command implementation // Authentication is guaranteed to be valid here }, } ``` ### How it works 1. Checks for valid credentials —first checks if a valid API key already exists. 2. Auto-configures URL if missing —if no DataRobot URL is configured, prompts you to set it up. 3. Retrieves new credentials —if credentials are missing or expired, automatically triggers the browser-based login flow. 4. Fails early —if authentication cannot be established, the command won't run and returns an error. ### Direct call (for non-command code) For code that isn't a Cobra command, you can use `auth.EnsureAuthenticated()` directly: ``` import "github.com/datarobot/cli/cmd/auth" func MyFunction() error { // Ensure valid authentication before proceeding. if !auth.EnsureAuthenticated() { return errors.New("authentication failed") } // Continue with authenticated operations. apiKey := config.GetAPIKey() // ... use apiKey for API calls return nil } ``` ### When to use Add authentication to any command that: - Makes API calls to DataRobot endpoints. - Needs to populate DataRobot credentials in configuration files. - Requires valid authentication to function correctly. ### Commands with authentication The following commands use `PreRunE` to ensure authentication: - dr dotenv update —automatically ensures authentication before updating environment variables. - dr templates list —requires authentication to fetch templates from the API. - dr templates clone —requires authentication to fetch template details. ## Skipping authentication For advanced use cases where authentication is handled externally or not required, you can bypass authentication checks using the `--skip-auth` global flag. ### Using the skip-auth flag ``` # Skip authentication for any command dr templates list --skip-auth dr dotenv update --skip-auth # Skip authentication with environment variable DATAROBOT_CLI_SKIP_AUTH=true dr templates setup ``` ### Behavior When `--skip-auth` is enabled: 1. Bypasses all authentication checks —the EnsureAuthenticated() function returns true immediately without validating credentials. 2. Emits a warning —logs a warning message: "Authentication checks are disabled via --skip-auth flag. This may cause API calls to fail." 3. May cause API failures —commands that make API calls will likely fail if no valid credentials are present. ### When to use skip-auth The `--skip-auth` flag is intended for advanced scenarios such as: - Testing —testing command logic without requiring valid credentials. - CI/CD pipelines —when authentication is managed through environment variables ( DATAROBOT_API_TOKEN ). - Offline development —working in environments without internet access or access to DataRobot. - Debugging —isolating authentication issues from other command behavior. > ⚠️ Warning:This flag should only be used when you understand the implications. Most users should rely on the standard authentication flow viadr auth login. ## Manual login You can still manually run `dr auth login` to refresh credentials or change accounts. The `LoginAction()` function provides the interactive login experience with confirmation prompts for overwriting existing credentials. --- # Development guide URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/development/building.html > Building, testing, and developing the DataRobot CLI. This guide covers building, testing, and developing the DataRobot CLI. ## Table of contents - Building from source - Project architecture - Coding standards - Development workflow - Testing - Debugging - Release process ## Building from source ### Prerequisites - Go 1.25.3+ — Download . - Git —version control. - Task —task runner ( install ). ### Quick build ``` # Clone repository git clone https://github.com/datarobot-oss/cli.git cd cli # Install development tools task dev-init # Build binary task build # Binary is at ./dist/dr ./dist/dr version ``` ### Available tasks ``` # Show all tasks task --list # Common tasks task build # Build the CLI binary task test # Run all tests task test-coverage # Run tests with coverage task lint # Run linters (includes formatting) task clean # Clean build artifacts task dev-init # Setup development environment task install-tools # Install development tools task run # Run CLI without building ``` ### Build options Always use `task build` for building the CLI. This ensures proper version information and build flags are applied: ``` # Standard build (recommended) task build # Run without building (for quick testing) task run -- templates list ``` The `task build` command automatically includes: - Version information from git - Git commit hash - Build timestamp - Proper ldflags configuration For cross-platform builds and releases, we use GoReleaser (see [Release Process](https://docs.datarobot.com/en/docs/agentic-ai/cli/development/building.html#release-process)). ## Project architecture ### Directory structure ``` cli/ ├── cmd/ # Command implementations (Cobra) │ ├── root.go # Root command and global flags │ ├── auth/ # Authentication commands │ │ ├── cmd.go # Auth command group │ │ ├── login.go # Login command │ │ ├── logout.go # Logout command │ │ └── setURL.go # Set URL command │ ├── dotenv/ # Environment variable management │ │ ├── cmd.go # Dotenv command │ │ ├── model.go # TUI model (Bubble Tea) │ │ ├── promptModel.go # Prompt handling │ │ ├── template.go # Template parsing │ │ └── variables.go # Variable handling │ ├── run/ # Task execution │ │ └── cmd.go # Run command │ ├── templates/ # Template management │ │ ├── cmd.go # Template command group │ │ ├── clone/ # Clone subcommand │ │ ├── list/ # List subcommand │ │ ├── setup/ # Setup wizard │ │ └── status.go # Status command │ └── self/ # CLI utility commands │ ├── cmd.go # Self command group │ ├── completion.go # Completion generation │ └── version.go # Version command ├── internal/ # Private packages (not importable) │ ├── assets/ # Embedded assets │ │ └── templates/ # HTML templates │ ├── config/ # Configuration management │ │ ├── config.go # Config loading/saving │ │ ├── auth.go # Auth config │ │ └── constants.go # Constants │ ├── drapi/ # DataRobot API client │ │ ├── llmGateway.go # LLM gateway API │ │ └── templates.go # Templates API │ ├── envbuilder/ # Environment configuration │ │ ├── builder.go # Env file building │ │ └── discovery.go # Prompt discovery │ ├── task/ # Task runner integration │ │ ├── discovery.go # Taskfile discovery │ │ └── runner.go # Task execution │ └── version/ # Version information │ └── version.go ├── tui/ # Terminal UI shared components │ ├── banner.go # ASCII banner │ └── theme.go # Color theme ├── docs/ # Documentation ├── main.go # Application entry point ├── go.mod # Go module dependencies ├── go.sum # Dependency checksums ├── Taskfile.yaml # Task definitions └── goreleaser.yaml # Release configuration ``` ### Key components #### Command layer (cmd/) The CLI is built using the [Cobra](https://github.com/spf13/cobra) framework. Commands are organized hierarchically, and there should be a one-to-one mapping between commands and files/directories. For example, the `templates` command group is in `cmd/templates/`, with subcommands in their own directories. Code in the `cmd/` folder should primarily handle command-line parsing, argument validation, and orchestrating calls to internal packages. There should be minimal to no business logic here.Consider this the UI layer of the application. ``` // cmd/root.go - Root command definition var RootCmd = &cobra.Command{ Use: "dr", Short: "DataRobot CLI", Long: "Command-line interface for DataRobot", } // Register subcommands RootCmd.AddCommand( auth.Cmd(), templates.Cmd(), // ... ) ``` #### TUI layer (cmd/dotenv/, cmd/templates/setup/) Uses [Bubble Tea](https://github.com/charmbracelet/bubbletea) for interactive UIs: ``` // Bubble Tea Model type Model struct { // State screen screens // Sub-models textInput textinput.Model list list.Model } // Required methods func (m Model) Init() tea.Cmd func (m Model) Update(tea.Msg) (tea.Model, tea.Cmd) func (m Model) View() string ``` #### Internal packages (internal/) Houses core business logic, API clients, configuration management, etc. #### Configuration (internal/config/) Uses [Viper](https://github.com/spf13/viper) for configuration as well as a state registry: ``` // Load config viper.SetConfigName("config") viper.SetConfigType("yaml") viper.AddConfigPath("~/.datarobot") viper.ReadInConfig() // Access values endpoint := viper.GetString("datarobot.endpoint") ``` #### API client (internal/drapi/) HTTP client for DataRobot APIs: ``` // Make API request func GetTemplates() (*TemplateList, error) { resp, err := http.Get(endpoint + "/api/v2/templates") // ... handle response } ``` ### Design patterns #### Command pattern Each command is self-contained: ``` // cmd/templates/list/cmd.go var Cmd = &cobra.Command{ Use: "list", Short: "List templates", GroupID: "core", RunE: func(cmd *cobra.Command, args []string) error { // Implementation return listTemplates() }, } ``` `RunE` is the main execution function. Cobra also provides `PreRunE`, `PostRunE`, and other hooks. Prefer to use these for setup/teardown, validation, etc.: ``` PersistPreRunE: func(cmd *cobra.Command, args []string) error { // Setup logging return setupLogging() }, PreRunE: func(cmd *cobra.Command, args []string) error { // Validate args return validateArgs(args) }, PostRunE: func(cmd *cobra.Command, args []string) error { // Cleanup return nil }, ``` Each command can be assigned to a group via `GroupID` for better organization in `dr help` views. Commands without a `GroupID` are listed under "Additional Commands". #### Model-View-Update (Bubble Tea) Interactive UIs use MVU pattern: ``` // Update handles events func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) { switch msg := msg.(type) { case tea.KeyMsg: return m.handleKey(msg) case dataLoadedMsg: return m.handleData(msg) } return m, nil } // View renders current state func (m Model) View() string { return lipgloss.JoinVertical( lipgloss.Left, m.header(), m.content(), m.footer(), ) } ``` ## Coding standards ### Go style requirements Critical: All code must pass `golangci-lint` with zero errors. Follow these whitespace rules strictly: 1. Never cuddle declarations : Always add a blank line before var , const , type declarations when they follow other statements 2. Separate statement types : Add blank lines between different statement types (assign, if, for, return, etc.) 3. Blank line after block start : Add blank line after opening braces of functions/blocks when they follow declarations 4. Blank line before multi-line statements : Add blank line before if/for/switch statements Example of correct spacing: ``` func example() { x := 1 if x > 0 { y := 2 fmt.Println(y) } var result string result = "done" return result } ``` Common mistakes to avoid: ``` // ❌ BAD: Cuddled declaration func bad() { x := 1 var y int // Missing blank line before declaration } // ✅ GOOD: Properly spaced func good() { x := 1 var y int } ``` ### TUI development standards When building terminal user interfaces: 1. Always wrap TUI models with InterruptibleModel —ensures global Ctrl-C handling: ``` import "github.com/datarobot/cli/tui" // Wrap your model interruptible := tui.NewInterruptibleModel(yourModel) program := tea.NewProgram(interruptible) ``` 1. Reuse existing TUI components—checktui/package first before creating new components. Also explore theBubbles libraryfor pre-built components. 2. Use common lipgloss styles—defined intui/theme.gofor visual consistency: ``` import "github.com/datarobot/cli/tui" // Use theme styles title := tui.TitleStyle.Render("My Title") error := tui.ErrorStyle.Render("Error message") ``` ### Quality tools All code must pass these tools without errors: - go mod tidy —dependency management - go fmt —basic formatting - go vet —suspicious constructs - golangci-lint —comprehensive linting (includes wsl, revive, staticcheck, etc.) - goreleaser check —release configuration validation Before committing code, verify it follows wsl (whitespace) rules. ### Running quality checks ``` # Run all quality checks at once task lint # Individual checks go mod tidy go fmt ./... go vet ./... task install-tools # Install golangci-lint ./tmp/bin/golangci-lint run ./... ./tmp/bin/goreleaser check ``` ## Development workflow ### Important: Use Taskfile, not direct Go commands Always use Taskfile tasks for development operations rather than direct `go` commands. This ensures consistency, proper build flags, and correct environment setup. ``` # ✅ CORRECT: Use task commands task build task test task lint # ❌ INCORRECT: Don't use direct go commands go build go test ``` ### 1. Setup development environment ``` # Clone and setup git clone https://github.com/datarobot-oss/cli.git cd cli task dev-init ``` ### 2. Create feature branch ``` git checkout -b feature/my-feature ``` ### 3. Make changes ``` # Edit code vim cmd/templates/new-feature.go # Run linters (includes formatting) task lint ``` ### 4. Test changes ``` # Run tests task test # Run specific test (direct go test is acceptable for specific tests) go test -run TestMyFeature ./cmd/templates # Test manually using task run task run -- templates list # Or build and test the binary task build ./dist/dr templates list ``` ### 5. Commit and push ``` git add . git commit -m "feat: add new feature" git push origin feature/my-feature ``` ## Testing ### Unit tests ``` // cmd/auth/login_test.go package auth import ( "testing" "github.com/stretchr/testify/assert" ) func TestLogin(t *testing.T) { // Arrange mockAPI := &MockAPI{} // Act err := performLogin(mockAPI) // Assert assert.NoError(t, err) } ``` ### Integration tests ``` // internal/config/config_test.go func TestConfigReadWrite(t *testing.T) { // Create temp config tmpDir := t.TempDir() configPath := filepath.Join(tmpDir, "config.yaml") // Write config err := SaveConfig(configPath, &Config{ Endpoint: "https://test.datarobot.com", }) assert.NoError(t, err) // Read config config, err := LoadConfig(configPath) assert.NoError(t, err) assert.Equal(t, "https://test.datarobot.com", config.Endpoint) } ``` ### TUI tests Using [teatest](https://github.com/charmbracelet/x/tree/main/exp/teatest): ``` // cmd/dotenv/model_test.go func TestDotenvModel(t *testing.T) { m := Model{ // Setup model } tm := teatest.NewTestModel(t, m) // Send keypress tm.Send(tea.KeyMsg{Type: tea.KeyEnter}) // Wait for update teatest.WaitFor(t, tm.Output(), func(bts []byte) bool { return bytes.Contains(bts, []byte("Expected output")) }) } ``` ### Running tests ``` # All tests (recommended) task test # With coverage (opens HTML report) task test-coverage # Specific package (direct go test is fine for targeted testing) go test ./internal/config # Verbose go test -v ./... # With race detection (task test already includes this) go test -race ./... # Specific test go test -run TestLogin ./cmd/auth ``` Note: `task test` automatically runs tests with race detection and coverage enabled. ### Running smoke tests using GitHub Actions We have smoke tests that are not currently run on Pull Requests however can be using PR comments to trigger them. These are the appropriate comments to trigger respective tests: - /trigger-smoke-test or /trigger-test-smoke - Run smoke tests on this PR - /trigger-install-test or /trigger-test-install - Run installation tests on this PR ## Debugging ### Using Delve ``` # Install delve go install github.com/go-delve/delve/cmd/dlv@latest # Debug with arguments dlv debug main.go -- templates list # In debugger (dlv) break main.main (dlv) continue (dlv) print variableName (dlv) next ``` ### Debug logging ``` # Enable debug mode (use task run) task run -- --debug templates list # Or with built binary task build ./dist/dr --debug templates list ``` ### Add debug statements ``` import "github.com/charmbracelet/log" // Debug logging log.Debug("Variable value", "key", value) log.Info("Processing started") log.Warn("Unexpected condition") log.Error("Operation failed", "error", err) ``` ### Quick release ``` # Tag version git tag v1.0.0 git push --tags # GitHub Actions will: # 1. Build for all platforms # 2. Run tests # 3. Create GitHub release # 4. Upload binaries ``` --- # Release process URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/development/releasing.html > How to create and publish releases of the DataRobot CLI. This document describes how to create and publish releases of the DataRobot CLI. ## Overview The project uses [GoReleaser](https://goreleaser.com/) for automated releases. Releases are triggered by creating and pushing Git tags, which automatically builds binaries for multiple platforms and publishes them to GitHub. ## Prerequisites - Write access to the repository - All changes merged to the main branch - Familiarity with Semantic Versioning ## Versioning We follow [Semantic Versioning](https://semver.org/) (SemVer): - MAJOR.MINOR.PATCH (e.g., v1.2.3 ) - Pre-releases : v1.2.3-rc.1 , v1.2.3-beta.1 , v1.2.3-alpha.1 ### Version guidelines MAJOR version when making incompatible API changes: - Breaking changes to command-line interface - Removing commands or flags - Changing default behavior that breaks existing workflows MINOR version when adding functionality in a backward-compatible manner: - New commands or subcommands - New flags or options - New features PATCH version when making backward-compatible bug fixes: - Bug fixes - Documentation updates - Performance improvements ## Creating a release ### Step 1: Ensure main branch is ready ``` # Switch to main branch git checkout main # Pull latest changes git pull origin main # Verify all tests pass task test # Verify linting passes task lint ``` ### Step 2: Determine next version Review recent changes and decide on the next version number based on SemVer guidelines above. ### Step 3: Create and push tag ``` # Create a new version tag git tag v0.2.0 # Push the tag to trigger the release git push origin v0.2.0 ``` Note: The tag must start with `v` (e.g., `v1.0.0`, not `1.0.0`). ### Step 4: Monitor release process 1. Go to the Actions tab in GitHub 2. Watch the release workflow run 3. The workflow will: 4. Build binaries for multiple platforms (macOS, Linux, Windows) 5. Run tests 6. Generate release notes from commit messages 7. Create a GitHub release 8. Upload artifacts ### Step 5: Verify release Once the workflow completes: 1. Go to Releases 2. Verify the new release appears with: 3. Correct version number 4. Generated release notes 5. Binary artifacts for all platforms 6. Checksums file ### Step 6: Update release notes (optional) Edit the release notes on GitHub to: - Add highlights of major changes - Include upgrade instructions if needed - Add breaking change warnings - Include acknowledgments ## Pre-release versions For testing releases before making them generally available: ``` # Create a pre-release tag git tag v0.2.0-rc.1 # Push the tag git push origin v0.2.0-rc.1 ``` Pre-release versions are marked as "Pre-release" on GitHub and can be used for testing. ## Testing the release process To test the release process without publishing: ``` # Dry run (builds but doesn't publish) goreleaser release --snapshot --clean # Check output in dist/ directory ls -la dist/ ``` This creates build artifacts locally without creating a GitHub release. ## Rollback If a release has issues: ### Delete the tag locally and remotely ``` # Delete local tag git tag -d v0.2.0 # Delete remote tag git push origin :refs/tags/v0.2.0 ``` ### Delete the GitHub release - Go to Releases page - Click on the problematic release - Click "Delete this release" ### Fix the issues and create a new patch release ## Release configuration The release process is configured in `goreleaser.yaml`. Key configurations: - Builds : Defines target platforms and architectures - Archives : Creates distribution archives - Checksums : Generates checksum files - Release notes : Automatic generation from commits - Artifacts : Files to include in the release To validate the configuration: ``` goreleaser check ``` ## Automated release workflow The GitHub Actions workflow ( `.github/workflows/release.yml`) automatically: 1. Triggers on tag push matching v* 2. Checks out the code 3. Sets up Go environment 4. Runs GoReleaser 5. Creates GitHub release 6. Uploads all artifacts ## Best practices 1. Always test before releasing: 2. Run full test suite: task test 3. Run linters: task lint 4. Build locally:task build 5. Use meaningful commit messages: 6. They're used to generate release notes 7. Follow conventional commit format when possible 8. Update CHANGELOG.md: 9. Document significant changes 10. Include migration notes for breaking changes 11. Communicate breaking changes: 12. Update documentation 13. Add prominent notes in release description 14. Consider a major version bump 15. Test installation: 16. Test the install script after release 17. Verify binaries work on target platforms ## Troubleshooting ### Release workflow fails - Check the Actions tab for error messages - Verify goreleaser.yaml is valid: goreleaser check - Ensure all required secrets are configured ### Tag already exists ``` # Delete and recreate if needed git tag -d v0.2.0 git push origin :refs/tags/v0.2.0 git tag v0.2.0 git push origin v0.2.0 ``` ### Missing artifacts - Verify build configuration in goreleaser.yaml - Check build logs in GitHub Actions - Test locally with goreleaser release --snapshot --clean ## Next steps - Setup guide —development environment setup - Building guide —detailed build information --- # Development setup URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/development/setup.html > Setting up your development environment for building and developing the DataRobot CLI. This guide covers setting up your development environment for building and developing the DataRobot CLI. ## Prerequisites - Go 1.25.3+ — Download - Git —version control - Task —task runner ( install ) ## Installation ### Installing Task Task is required for running development tasks. #### macOS ``` brew install go-task/tap/go-task ``` #### Linux ``` sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b /usr/local/bin ``` #### Windows ``` choco install go-task ``` ## Setting up the development environment ### Clone the repository ``` git clone https://github.com/datarobot-oss/cli.git cd cli ``` ### Install development tools ``` task dev-init ``` This will install all necessary development tools including linters and code formatters. ### Build the CLI ``` task build ``` The binary will be available at `./dist/dr`. ### Verify the build ``` ./dist/dr version ``` ## Available development tasks View all available tasks: ``` task --list ``` ### Common tasks | Task | Description | | --- | --- | | task build | Build the CLI binary | | task test | Run all tests | | task test-coverage | Run tests with coverage report | | task lint | Run linters and code formatters | | task fmt | Format code | | task clean | Clean build artifacts | | task dev-init | Setup development environment | | task install-tools | Install development tools | | task run | Run CLI without building (e.g., task run -- templates list) | ## Building Always use `task build` for building the CLI.This ensures: - Version information from git is included - Git commit hash is embedded - Build timestamp is recorded - Proper ldflags configuration is applied ``` # Standard build (recommended) task build # Run without building (for quick testing) task run -- templates list ``` ## Running tests ``` # Run all tests task test # Run tests with coverage task test-coverage # Run specific test go test ./cmd/auth/... ``` ## Linting and formatting ``` # Run all linters (includes formatting) task lint # Format code only task fmt ``` The project uses: - golangci-lint for comprehensive linting - go fmt for basic formatting - go vet for suspicious constructs - goreleaser check for release configuration validation ## Next steps - Project structure —understand the codebase organization - Building guide —detailed build information and architecture - Release process —creating releases and publishing --- # Project structure URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/development/structure.html > Organization of the DataRobot CLI codebase. This document describes the organization of the DataRobot CLI codebase. ## Directory overview ``` cli/ ├── cmd/ # Command implementations (Cobra) │ ├── root.go # Root command and global flags │ ├── auth/ # Authentication commands │ ├── component/ # Component management commands │ ├── dotenv/ # Environment variable management │ ├── run/ # Task execution │ ├── self/ # Self-management commands │ ├── start/ # Application startup │ ├── task/ # Task commands │ └── templates/ # Template management ├── internal/ # Private application code │ ├── assets/ # Embedded assets │ ├── config/ # Configuration management │ ├── copier/ # Template copying utilities │ ├── drapi/ # DataRobot API client │ ├── envbuilder/ # Environment builder │ ├── misc/ # Miscellaneous utilities │ ├── repo/ # Repository detection │ ├── shell/ # Shell utilities │ ├── task/ # Task discovery and execution │ ├── tools/ # Tool prerequisites │ └── version/ # Version information ├── tui/ # Terminal UI components │ ├── banner.go # Banner display │ ├── interrupt.go # Interrupt handling │ └── theme.go # Visual theme ├── docs/ # Documentation │ ├── commands/ # Command reference │ ├── development/ # Development guides │ ├── template-system/ # Template system docs │ └── user-guide/ # User documentation ├── smoke_test_scripts/ # Smoke tests ├── main.go # Application entry point ├── Taskfile.yaml # Task definitions ├── go.mod # Go module definition └── goreleaser.yaml # Release configuration ``` ## Key directories ### cmd/ Contains all CLI command implementations using the Cobra framework. Each subdirectory represents a command or command group. Structure: - root.go —root command setup and global flags - Each command has its own subdirectory with cmd.go as the entry point - Commands that have subcommands organize them in the same directory Example: - cmd/auth/cmd.go —auth command group - cmd/auth/login.go —login subcommand - cmd/auth/logout.go —logout subcommand ### internal/ Private application code that cannot be imported by other projects. This follows Go's convention for internal packages. #### config/ Configuration management including: - Reading/writing configuration files - Authentication state - User preferences #### drapi/ DataRobot API client implementation for: - Template listing and retrieval - API authentication - API endpoint communication #### envbuilder/ Environment configuration builder that: - Discovers environment variables from templates - Validates configuration - Generates .env files - Provides interactive prompts #### task/ Task discovery and execution: - Taskfile detection - Task parsing - Task running - Output handling ### tui/ Terminal UI components built with Bubble Tea: - Reusable UI models - Theme definitions - Interrupt handling for graceful exits - Banner displays ### docs/ Documentation organized by audience: - commands/ —detailed command reference - development/ —development guides for contributors - template-system/ —template configuration system - user-guide/ —end-user documentation ## Code organization patterns ### Command structure Each command follows this pattern: ``` // cmd/example/cmd.go package example import "github.com/spf13/cobra" var Cmd = &cobra.Command{ Use: "example", Short: "Example command", Long: `Detailed description`, PreRunE: func(cmd *cobra.Command, args []string) error { // Validation and setup return nil }, RunE: func(cmd *cobra.Command, args []string) error { // Command implementation return nil }, } func init() { // Flag definitions Cmd.Flags().StringP("flag", "f", "", "Flag description") } ``` ### TUI models TUI components use the Bubble Tea framework and are wrapped with `InterruptibleModel` for consistent Ctrl-C handling: ``` // cmd/example/model.go package example import ( tea "github.com/charmbracelet/bubbletea" "github.com/datarobot/cli/tui" ) type model struct { // State fields } func (m model) Init() tea.Cmd { return nil } func (m model) Update(msg tea.Msg) (tea.Model, tea.Cmd) { // Handle messages return m, nil } func (m model) View() string { // Render UI return "" } // Usage in command func runInteractive() error { m := model{} wrapped := tui.NewInterruptibleModel(m) _, err := tea.NewProgram(wrapped).Run() return err } ``` ### Configuration Configuration is managed through Viper and stored in: - ~/.config/datarobot/config.yaml —global configuration - ~/.config/datarobot/credentials.json —authentication tokens Access configuration through the `internal/config` package: ``` import "github.com/datarobot/cli/internal/config" // Get configuration values apiKey := config.GetAPIKey() endpoint := config.GetEndpoint() // Set configuration values config.SetAPIKey("new-key") config.SaveConfig() ``` ## Testing structure Tests are colocated with the code they test: - Unit tests: *_test.go files in the same package - Test helpers in same directory when needed - Smoke tests in smoke_test_scripts/ directory ## Build artifacts Generated files and artifacts: - dist/ —build output (created by Task/GoReleaser) - tmp/ —temporary build files - coverage.txt —test coverage report ## Next steps - Setup guide —setting up your development environment - Building guide —detailed build information and architecture --- # Getting started with DataRobot CLI URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html This guide will help you install and start using the DataRobot CLI ( `dr`) for managing custom applications. ## Prerequisites Before you begin, ensure you have: - DataRobot account —access to a DataRobot instance (cloud or self-managed). If you don't have an account, sign up at DataRobot or contact your organization's DataRobot administrator. - Git —for cloning templates (version 2.0+). Install Git from git-scm.com if not already installed. Verify installation: git --version - Task —for running tasks. Install Task from taskfile.dev if not already installed. Verify installation: task --version - Terminal —command-line interface access. - macOS/Linux: Use Terminal, iTerm2, or your preferred terminal emulator. - Windows: Use PowerShell, Command Prompt, or Windows Terminal. ## Installation Install the latest version with a single command: macOS/Linux ``` curl https://cli.datarobot.com/install | sh ``` Windows (PowerShell) ``` irm https://cli.datarobot.com/winstall | iex ``` For alternative installation methods (Homebrew, download binary, specific version, or build from source), see the sections below. ### Install via Homebrew / Linuxbrew ``` brew install datarobot-oss/taps/dr-cli ``` ### Download binary Download the latest release for your operating system: #### macOS ``` # Intel Macs curl -LO https://github.com/datarobot-oss/cli/releases/latest/download/dr-darwin-amd64 chmod +x dr-darwin-amd64 sudo mv dr-darwin-amd64 /usr/local/bin/dr # Apple Silicon (M1/M2) curl -LO https://github.com/datarobot-oss/cli/releases/latest/download/dr-darwin-arm64 chmod +x dr-darwin-arm64 sudo mv dr-darwin-arm64 /usr/local/bin/dr ``` #### Linux ``` # x86_64 curl -LO https://github.com/datarobot-oss/cli/releases/latest/download/dr-linux-amd64 chmod +x dr-linux-amd64 sudo mv dr-linux-amd64 /usr/local/bin/dr # ARM64 curl -LO https://github.com/datarobot-oss/cli/releases/latest/download/dr-linux-arm64 chmod +x dr-linux-arm64 sudo mv dr-linux-arm64 /usr/local/bin/dr ``` #### Windows Download `dr-windows-amd64.exe` from the [releases page](https://github.com/datarobot-oss/cli/releases/latest) and add it to your PATH. ### Install a specific version To install a specific version, pass the version number to the installer: #### macOS/Linux ``` curl https://cli.datarobot.com/install | sh -s -- v0.2.38 ``` #### Windows (PowerShell) ``` $env:VERSION = "v0.2.38"; irm https://cli.datarobot.com/winstall | iex ``` ### Build from source If you have Go 1.25.6 or later installed: ``` # Clone the repository git clone https://github.com/datarobot-oss/cli.git cd cli # Install Task (if not already installed) go install github.com/go-task/task/v3/cmd/task@latest # Build task build # The binary will be at ./dist/dr sudo mv ./dist/dr /usr/local/bin/dr ``` ### Verify installation ``` dr --version ``` You should see output similar to: ``` DataRobot CLI version: v0.2.38 ``` ## Updating the CLI To update to the latest version of the DataRobot CLI, use the built-in update command: ``` dr self update ``` This command will automatically: - Detect your installation method (Homebrew, manual installation, etc.) - Download the latest version - Install it using the appropriate method for your system - Preserve your existing configuration and credentials The update process supports: - Homebrew (macOS) —automatically upgrades via brew upgrade --cask dr-cli - Windows —runs the latest PowerShell installation script - macOS/Linux —runs the latest shell installation script After updating, verify the new version: ``` dr self version ``` You can also check the installed version with `dr --version` at any time. ## Uninstalling the CLI How you uninstall depends on how you installed: Installed via install script (curl/irm): - macOS/Linux: Remove the binary (e.g., sudo rm /usr/local/bin/dr if that is where it was installed). Alternatively, run the uninstall script from the CLI repository if you have it cloned. - Windows: Remove the dr executable from your PATH (the install script typically places it in a user directory). Installed via Homebrew: ``` brew uninstall dr-cli ``` Optional: remove configuration and state - User config: Delete ~/.config/datarobot/ (Linux/macOS) or %USERPROFILE%\.config\datarobot\ (Windows) to remove drconfig.yaml and stored credentials. - Template state: In any cloned template directory, you can remove .datarobot/cli/state.yaml to clear local state for that template. ## Initial setup ### 1. Configure DataRobot URL First, configure your DataRobot credentials by setting your DataRobot URL. For steps to locate your DataRobot URL (API endpoint) and manage API keys, see the [DataRobot documentation](https://docs.datarobot.com/). Set your DataRobot instance URL: ``` dr auth set-url ``` You'll be prompted to enter your DataRobot URL. You can use shortcuts for cloud instances: - Enter 1 for https://app.datarobot.com - Enter 2 for https://app.eu.datarobot.com - Enter 3 for https://app.jp.datarobot.com - Or enter your custom URL (e.g., https://your-instance.datarobot.com ) Alternatively, set the URL directly: ``` dr auth set-url https://app.datarobot.com ``` ### 2. Authenticate Log in to DataRobot using OAuth: ``` dr auth login ``` This will: 1. Open your default web browser. 2. Redirect you to the DataRobot login page. 3. Request authorization. 4. Automatically save your credentials. Your API key will be securely stored in `~/.config/datarobot/drconfig.yaml`. ### 3. Verify authentication Check that you're logged in: ``` dr templates list ``` This should display a list of available templates from your DataRobot instance. ## Your first template Now that you're set up, let's create your first application from a template. ### Using the setup wizard (recommended) The easiest way to get started: ``` dr templates setup ``` This interactive wizard will: 1. Display available templates. 2. Help you select and clone a template. 3. Guide you through environment configuration. 4. Set up all required variables. Follow the on-screen prompts to complete the setup. ### Manual setup If you prefer manual control: ``` # 1. List available templates. dr templates list # 2. Set up a template (this clones and configures it). dr templates setup # 3. Navigate to the template directory. cd TEMPLATE_NAME # 4. Configure environment variables (if not done during setup). dr dotenv setup ``` ## Running your application Once your template is set up, you have several options to run it: ### Quick start (recommended) Use the `start` command for automated initialization: ``` dr start ``` This command will: - Check prerequisites and validate your environment. - Verify your CLI version meets the template's minimum requirements. - Check if you're in a DataRobot repository (if not, launches template setup). - Execute a start command in this order: - task start from the Taskfile (if available) - A quickstart script from .datarobot/cli/bin/ (if available) - Fall back to the setup wizard if neither exists. For non-interactive mode (useful in scripts or CI/CD): ``` dr start --yes ``` ### Running specific tasks For more control, execute individual tasks: ``` # List available tasks dr task list # Run the development server dr run dev # Or execute specific tasks dr run build dr run test ``` ## Next steps - Authentication guide —learn about authentication options. - Working with templates —detailed template management. - Shell completions —set up command auto-completion. - Agentic AI get started guide —build and deploy AI agents from DataRobot templates using dr start and dr task run . ## Common issues ### "dr: command not found" Why it happens: The CLI binary isn't in your system's PATH, so your shell can't find it. How to fix: ``` # Check if dr is in PATH which dr # If not found, verify the binary location ls -l /usr/local/bin/dr # Add it to your PATH (for current session) export PATH="/usr/local/bin:$PATH" # For permanent fix, add to your shell config file: # Bash: echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc # Zsh: echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.zshrc ``` How to prevent: Re-run the installation script or ensure the binary is installed to a directory in your PATH. ### "Failed to read config file" Why it happens: The configuration file doesn't exist yet or is in an unexpected location. This typically occurs on first use before authentication. How to fix: ``` # Set your DataRobot URL (creates config file if missing) dr auth set-url https://app.datarobot.com # Authenticate (saves credentials to config file) dr auth login ``` How to prevent: Run `dr auth set-url` and `dr auth login` as part of your initial setup. The config file is automatically created at `~/.config/datarobot/drconfig.yaml`. ### "Authentication failed" Why it happens: Your API token may have expired, been revoked, or the DataRobot URL may have changed. This can also occur if the config file is corrupted. How to fix: ``` # Clear existing credentials dr auth logout # Re-authenticate dr auth login # If issues persist, verify your DataRobot URL dr auth set-url https://app.datarobot.com # or your instance URL dr auth login ``` How to prevent: Regularly update the CLI ( `dr self update`) and re-authenticate if you change DataRobot instances or if your organization rotates API keys. ## Getting help For additional help: ``` # General help dr --help # Command-specific help dr auth --help dr templates --help dr run --help # Enable verbose output for debugging dr --verbose templates list # Enable debug output for detailed information dr --debug templates list ``` When you enable debug mode, the CLI creates a `dr-tui-debug.log` file in the current directory for terminal UI debug information. For advanced options such as `--skip-auth` (skip authentication checks) and `--force-interactive` (force the setup wizard to run again), see the [Command reference - Global flags](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html#global-flags). ## Configuration location Configuration files are stored in: - Linux/macOS — ~/.config/datarobot/drconfig.yaml . - Windows — %USERPROFILE%\.config\datarobot\drconfig.yaml . See [Configuration files](https://docs.datarobot.com/en/docs/agentic-ai/cli/configuration.html) for more details. --- # CLI URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/index.html > Guides and references for using and developing the DataRobot CLI tool. This guide provides a comprehensive overview of all available documentation for the DataRobot CLI. The CLI (Command Line Interface) enables you to interact with DataRobot from your terminal, providing powerful automation capabilities for managing projects, templates, and workflows. ## Quick links ### For setup Essential guides to get you started with the DataRobot CLI: - Get started : Installation and initial setup - Quick reference : One-page command and path reference - Shell completions : Set up command auto-completion for your shell - Configuration : Understand and manage config files - DataRobot experimentation plugin : Local dashboard for agent development—install, configure, and open the GUI. See Access the GUI for the full walkthrough. - Local tracing : Inspect OpenTelemetry spans in the Traces tab during development. - Batch agent evaluation : Score agent responses in the Evaluation tab (preview; requires --enable-evaluation ). - Connect a local IDE to a codespace : Install the codespace CLI plugin and open a codespace from Visual Studio Code over SSH - Troubleshooting : Common issues and where to find solutions ### For templates Learn how to work with the template system: - Template structure : Understand how templates work - Interactive configuration : Overview of the configuration wizard - Environment variables : Manage .env files ### For agentic workflows Building or deploying AI agents with DataRobot templates: - Agentic AI : Create and evaluate agentic workflows - Agentic get started guide : Set up and run agents with dr start and dr task run - Develop agentic workflows : Installation, customization, tools, and deployment ### For developers Resources for contributing and building: - Building from source : Compile and build the CLI from source - Development setup : Local development environment - Project structure : Codebase organization - Releasing : Release process ## Key features documented ### Shell completions Comprehensive documentation for setting up [auto-completion in multiple shells](https://docs.datarobot.com/en/docs/agentic-ai/cli/shell-completions.html): - Bash (Linux and macOS) - Zsh - Fish - PowerShell ### Template system A detailed explanation of the template system including: - Template repository structure - .datarobot/prompts.yaml format - Interactive configuration wizard - Conditional prompts and sections - Multi-level configuration ### Interactive configuration In-depth coverage of the [interactive configuration system](https://docs.datarobot.com/en/docs/agentic-ai/cli/template-system/interactive-config.html): - Bubble Tea architecture - Prompt types (text, selection, multi-select) - Conditional logic with sections - State management - Keyboard controls - Advanced features ### Environment management A complete guide to managing [environment variables](https://docs.datarobot.com/en/docs/agentic-ai/cli/template-system/environment-variables.html): - .env vs .env.template - Variable types (required, optional, secret) - Interactive wizard - Security best practices - Common patterns ## Getting started If you're new to the DataRobot CLI, start here: 1. Installation and setup : Get the CLI installed and configured 2. Shell completions : Enable command auto-completion for faster workflows 3. Configuration : Understand how to configure the CLI for your environment --- # Batch agent evaluation URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-evaluation.html > Run batch agent evaluations from the local experimentation dashboard using the evaluation component and NeMo Evaluator. > [!NOTE] Preview > The evaluation component is in preview. APIs, configuration, and outputs may change between releases. When an App Framework project includes the [evaluation component](https://github.com/datarobot-community/af-component-evaluation), the experimentation dashboard can run batch evaluations against the agent. Each run sends test prompts to the agent OpenAI-compatible endpoint and scores every response using the [NeMo Evaluator](https://github.com/NVIDIA-NeMo/evaluator) BYOB (Bring Your Own Benchmark) framework. The evaluator treats the agent as a black box—only the final OpenAI-compatible response is scored. Each run uses exactly one benchmark. For installation, GUI access, and port configuration, see the [DataRobot experimentation plugin](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html) landing page. > [!NOTE] Note > These benchmarks are development-time testing aids for catching regressions before release. They are not security, privacy, or compliance controls, and a passing score is not evidence of compliance with any standard. ## Enable batch evaluation workflows Batch evaluation workflows in the GUI are controlled by the `--enable-evaluation` flag. Without it, the dashboard exposes Traces only—even when the evaluation component is installed in the project. Pass the flag when starting the dashboard: ``` dr xp --enable-evaluation --entity-id ``` Alternatively, set the environment variable before launch: ``` export DR_EXPERIMENT_ENABLE_EVALUATION=true dr xp --entity-id ``` For Agentic Starter projects that use `dr run dev` or `task infra:dev`, add the variable to the project root `.env` file instead. Taskfiles load `.env` automatically: ``` # .env (project root) DR_EXPERIMENT_ENABLE_EVALUATION=true ``` Bundled startup runs `dr xp --plain-output` without `--enable-evaluation`. The `.env` variable enables the Evaluation tab without changing the Taskfile. When the evaluation component is present and evaluation is enabled, `/api/info` reports `"features":["traces","evaluation"]`, and the Evaluation tab appears in the GUI. When disabled, `features` contains `"traces"` only, and evaluation API routes return `404`. If a dashboard is already running without the flag, stop it and restart with `--enable-evaluation`. See [Evaluation troubleshooting](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-evaluation.html#evaluation-troubleshooting) for common symptoms. ## Prerequisites | Requirement | Notes | | --- | --- | | base component | Install af-component-base before adding the evaluation component. | | Evaluation component in project | dr component add https://github.com/datarobot-community/af-component-evaluation . | | Dashboard started with evaluation enabled | dr xp --enable-evaluation or DR_EXPERIMENT_ENABLE_EVALUATION=true; see Enable batch evaluation workflows. | | Running agent endpoint | Typically http://localhost:8842/v1 via dr run dev. | | Credentials in project .env | DATAROBOT_API_TOKEN and DATAROBOT_ENDPOINT (for example https://app.datarobot.com; required for judge-based benchmarks and dataset generation). | Judge-free benchmarks ( `answer_correctness`, `instruction_following`, `prompt_injection`, `pii_leakage`, `tool_grounding`) run without judge credentials. ## Run an evaluation 1. Installaf-component-base, then add the evaluation component to the project (one-time): drcomponentaddhttps://github.com/datarobot-community/af-component-evaluation. 2. From the component directory (for exampleevaluations/), runtask install. 3. Start the dashboard with batch evaluation workflows enabled: drxp--enable-evaluation--entity-id 4. From thedr xpdashboard, select a pipeline, dataset, and agent endpoint, then start the run. 5. Alternatively, from the project root: drtaskrunevaluations:eval--\--endpointhttp://localhost:8842/v1\--pipelineanswer_quality.yaml\--datasetuser_datasets/sample_answer_quality.json Replaceevaluationswith the component task namespace if the folder was renamed during setup. Rundr task compose && task --listif the namespace is unknown. > [!TIP] Tip > Add `--dry-run` to validate the endpoint, pipeline, and dataset without scoring any cases or incurring judge costs. Set `AGENT_API_KEY` in `.env` only when the agent endpoint requires authentication; local DRUM agents usually need none. ## Benchmarks at a glance Each run uses one pipeline YAML from the component `user_pipelines/` directory. Three benchmarks use an LLM judge; five use deterministic checks and need no judge model. | Pipeline | Judge? | Measures | | --- | --- | --- | | answer_quality.yaml | Yes | General response quality. | | safety_refusal.yaml | Yes | Harmful-request refusal. | | faithfulness.yaml | Yes | RAG grounding. | | answer_correctness.yaml | No | Known-answer regression. | | instruction_following.yaml | No | Structural constraints. | | prompt_injection.yaml | No | Injection resistance. | | pii_leakage.yaml | No | PII in responses. | | tool_grounding.yaml | No | Tool-use evidence. | ## Key concepts - Judge-based vs judge-free. Judge-based benchmarks call an LLM to grade responses; judge-free benchmarks use deterministic checks and do not require a judge model. Given the same agent response, a deterministic check produces the same score. - Pass threshold. A case passes at score >= 0.5 . Cases that cannot be scored (for example, when a judge call fails) are marked inconclusive and excluded from pass rates. - Output vs internals. This workflow batch-tests agent outputs as a black box. To inspect tool calls, trajectories, or RAG retrieval, use NAT /evaluate instead. See NAT vs. NeMo . - Pre-release vs runtime. Batch evaluation catches regressions on a fixed dataset before release. Runtime guardrails enforce policy on live traffic; they solve different problems and are not substitutes. ## Evaluation troubleshooting > [!NOTE] How it works > The evaluation component runs in an isolated `uv` environment. The CLI discovers it via `[tool.af-component]` in `pyproject.toml` and invokes it as a subprocess. During a run, `dr xp` polls `/output/eval_status.json` for progress and reads `/output/eval_results.json` when the run completes. | Symptom | Fix | | --- | --- | | No Evaluation tab or evaluation APIs return 404 | Restart with --enable-evaluation or set DR_EXPERIMENT_ENABLE_EVALUATION=true in .env; confirm the evaluation component is installed. | | Agent unreachable | Start the agent with dr run dev; confirm http://localhost:8842/v1. | | Judge returns 400 on Bedrock models | NeMo sends both temperature and top_p; use an Azure GPT judge in the pipeline YAML. | | Wrong judge model name | Use gateway catalog names with no datarobot/ prefix (for example azure/gpt-5-5-2026-04-23). | | Task namespace not found | Run dr task compose && task --list. | | Adversarial cases marked inconclusive | Expected for judge-based benchmarks such as safety_refusal when the judge endpoint content-filters the prompt. | ## Further reading | Topic | GitHub doc | | --- | --- | | Full benchmark reference | benchmarks.md | | Pipeline YAML schema | pipelines.md | | Dataset format | datasets.md | | Output schema | outputs.md | | Custom benchmarks | writing-benchmarks.md | | NAT /evaluate comparison | nat-vs-nemo.md | ## See also - DataRobot experimentation plugin —install, configure, and open the GUI. - Local tracing —inspect OpenTelemetry spans during development. - af-component-evaluation repository —evaluation component source and reference docs. --- # Local tracing URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-tracing.html > Inspect OpenTelemetry traces in the local experimentation dashboard during agent development. The Traces tab in the local experimentation dashboard lists every request the agent handled during a development session. Use it to inspect spans, tool calls, latency, token counts, and errors without deploying to the DataRobot platform. For installation, GUI access, and port configuration, see the [DataRobot experimentation plugin](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html) landing page. ## Why use local tracing Local tracing supports agent development in several ways: - Faster iteration. Review traces, logs, and spans without leaving the local environment or breaking the development flow. - Behavior verification. Validate agent behavior across multiple real requests before deployment, not only on a single demo input. - Span-level debugging. Inspect individual spans to explain agent decisions instead of inferring them from the final response. - Earlier issue detection. Catch errors, guardrail violations, and performance problems while changes are still local, before they reach production. For guidance on adding custom spans and attributes that appear in local and deployed traces, see [Implement tracing](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-tracing-code.html). After deployment, review traces in the DataRobot UI; see [Tracing for custom applications](https://docs.datarobot.com/en/docs/wb-apps/custom-apps/monitor-app.html#tracing). ## Inspect OpenTelemetry traces After installing the plugin, start the agent locally with `task agent:dev`. If the dashboard did not start automatically, run `dr xp` in another terminal. By default, it is available at `http://localhost:8090`. Configure the port with `--port` or `DR_EXPERIMENT_PORT`. The dashboard lists every request the agent handled during the session. Each row corresponds to a trace. Select a trace to open a detailed breakdown of its spans, including: - Tool invocations and their execution order. - Duration and latency for each span. - Token counts, when reported by instrumentation. - Associated error logs when a span or trace fails. This is the same class of observability normally available only after deployment. With the experimentation plugin, it is available on the local machine from the start of development. ## Filter and search traces As request volume grows, use the dashboard controls to narrow results: | Control | Description | | --- | --- | | Attribute filter | Isolate traces by tool name, model, or a custom span attribute. | | Status filter | Separate successful runs from traces that ended in error. | | Date range | Scope the table to a specific window of activity. | | Search | Jump directly to a trace by request or trace identifier. | ## Tracing troubleshooting | Symptom | Fix | | --- | --- | | No traces in the dashboard | Start the agent with dr run dev or task agent:dev, send requests to the agent, then refresh the Traces tab. | | Dashboard not reachable | Confirm the server is running; see GUI access troubleshooting. | | Port 8090 already in use | List the process with lsof -i :8090 (macOS/Linux), note the PID, and run kill . On Windows, run netstat -ano \| findstr :8090, then taskkill /PID /F. Alternatively, start on another port with --port 8091. | | Duplicate startup message | When task dev runs, dr xp may start from both agent:dev and infra:dev. The message Already running at http://127.0.0.1:8090 is expected. | ## See also - DataRobot experimentation plugin —install, configure, and open the GUI. - Batch agent evaluation —score agent responses on a fixed dataset. - Implement tracing —add custom spans and attributes. --- # DataRobot experimentation plugin URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html > Install and use the DataRobot experimentation CLI plugin to open a local dashboard for agent development, tracing, and batch evaluation. The DataRobot experimentation plugin ( `dr xp`) exposes a local dashboard during agentic application development. The dashboard runs on port `8090` by default and organizes observability into tabs. The Traces tab is available as soon as the plugin is installed; additional tabs require feature flags and optional components. The plugin uses the same OpenTelemetry standard that carries telemetry across the rest of the stack. Local traces follow the same tracing model as deployed agents on the DataRobot platform, so local inspection reflects what the agent actually did during development without deploying to the platform or switching to a separate UI tab. ## Access the dashboard Complete these steps to open the local experimentation dashboard: 1. Install the CLI— curl https://cli.datarobot.com/install | sh ; verify with dr self version . See Install the CLI and authenticate . 2. Authenticate— dr auth login . 3. Install the plugin— dr plugin install xp ; verify with dr plugin list . See Install the experimentation plugin . 4. Create or open a project— dr start from an Agentic Starter template; confirm DATAROBOT_USE_CASE_ID in pulumi_config.json . See Set up the App Framework project . 5. Start the agent and dashboard—Run dr run dev from the project root. To launch the dashboard separately, start the agent with task agent:dev in one terminal, then run dr xp --entity-id in another. See Start the experimentation dashboard . 6. Open the GUI—browse to http://localhost:8090 . See Open the GUI in a browser . 7. Confirm traces—send a test message to the agent, refresh the Traces tab. See Confirm end-to-end behavior . For batch evaluation workflows, also install the [evaluation component](https://github.com/datarobot-community/af-component-evaluation) and set `DR_EXPERIMENT_ENABLE_EVALUATION=true` in `.env`. See [Batch agent evaluation](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-evaluation.html). ## Choose your path | Goal | Start here | | --- | --- | | Inspect agent traces during local development | Local tracing—Traces tab (default). | | Score agent responses on a fixed benchmark dataset | Batch agent evaluation—Evaluation tab (preview; requires --enable-evaluation). | Both paths share the same setup: install the CLI, install the `xp` plugin, set up an App Framework project, and open the GUI. ## Capabilities | Tab / capability | Doc | Requires | | --- | --- | --- | | Traces | Local tracing | Plugin installed; agent running. | | Evaluation | Batch agent evaluation | --enable-evaluation, evaluation component installed (preview). | Future tabs (logs, metrics, MCP tool collections) follow the same pattern: one doc page per tab, enabled by a feature flag. Run `dr xp --help` for the current flag list. ## Local development stack Agentic Starter templates run several services during local development. The commands and ports referenced throughout this page assume this stack: | Port | Service | | --- | --- | | 8090 | Experimentation dashboard (dr xp GUI). | | 8842 | Local agent OpenAI-compatible endpoint. | | 5173 | Frontend dev server. | | 8080 | FastAPI backend. | | 9000 | MCP server (configurable via MCP_SERVER_PORT). | Ports `8090`, `8080`, and `5173` are fixed in Agentic Starter templates. The agent port ( `8842`) is set during the `dr start` wizard. ## Quick start `` below is your App Framework project's use case ID—see [Prerequisites](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html#gui-prerequisites). ``` # Install the plugin dr plugin install xp # Show available options dr xp --help # Start with a use case (traces only) dr xp --entity-id # Enable batch evaluation workflows in the GUI dr xp --enable-evaluation --entity-id # Start with a deployment on a custom port dr xp --entity-type deployment --entity-id --port 8091 ``` > [!NOTE] Important > Batch evaluation workflows in the GUI are off by default. Pass `--enable-evaluation` (or set `DR_EXPERIMENT_ENABLE_EVALUATION=true`) when starting `dr xp` to expose the Evaluation tab and evaluation API routes. Traces work without this flag. See [Enable batch evaluation workflows](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-evaluation.html#enable-batch-evaluation). When [dr run dev](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/run.html) runs from an Agentic Starter project, the local tracing dashboard starts automatically on port `8090`. The bundled dashboard starts with traces only unless `DR_EXPERIMENT_ENABLE_EVALUATION=true` is set in `.env`. ## Access the GUI The experimentation plugin serves a web UI from a local HTTP server. The CLI installs and launches the plugin; the browser connects to the server URL after startup completes. ### Prerequisites Before starting the dashboard, confirm the following: | Requirement | Notes | | --- | --- | | DataRobot CLI | Install with the getting started guide; verify with dr self version. | | CLI authentication | Run dr auth login or configure credentials in ~/.config/datarobot/drconfig.yaml. | | Experimentation plugin | Install with dr plugin install xp; verify with dr plugin list. | | App Framework project | Required for entity context; create with dr start or clone an Agentic Starter template. | | Use case ID | Stored in pulumi_config.json as DATAROBOT_USE_CASE_ID, or pass --entity-id explicitly. | After confirming the prerequisites, complete the following sections in order (or jump straight to a step using the links in [Access the dashboard](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html#access-the-dashboard) above). ### Install the CLI and authenticate 1. Install the DataRobot CLI: ``` curl https://cli.datarobot.com/install | sh ``` 1. Verify the installation: ``` dr self version ``` 1. Authenticate with DataRobot: ``` dr auth login ``` The plugin requires authentication. The CLI passes `DATAROBOT_ENDPOINT` and `DATAROBOT_API_TOKEN` to the plugin process at launch. ### Install the experimentation plugin Install the `xp` plugin once per machine: ``` dr plugin install xp ``` Verify discovery: ``` dr plugin list ``` Confirm the list includes `xp` with a path under `~/.config/datarobot/plugins/xp/` (Linux and macOS) or the equivalent Windows config directory. ### Set up the App Framework project 1. Create or open an Agentic Starter project: drstart 2. Confirmpulumi_config.jsonat the project root contains a use case ID: "DATAROBOT_USE_CASE_ID":"" 3. (Optional)To run batch evaluation workflows in the GUI, add theevaluation component: drcomponentaddhttps://github.com/datarobot-community/af-component-evaluation. 4. From the evaluation component directory (for exampleevaluations/), install dependencies: taskinstall See [Batch agent evaluation](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-evaluation.html) for prerequisites and setup details. ### Configure the dashboard Set options with flags, environment variables, or a project-level `.dr-xp.yaml` file. Flag values take precedence over environment variables, then config files, then `pulumi_config.json`. The following table summarizes the most common settings: | Setting | Flag | Environment variable | Default | | --- | --- | --- | --- | | Port | --port | DR_EXPERIMENT_PORT | 8090 | | Entity type | --entity-type | DR_EXPERIMENT_ENTITY_TYPE | experiment_container (use case) | | Entity ID | --entity-id | DR_EXPERIMENT_ENTITY_ID | DATAROBOT_USE_CASE_ID from pulumi_config.json | | Evaluation workflows | --enable-evaluation | DR_EXPERIMENT_ENABLE_EVALUATION=true | Off; pass the flag to enable batch evaluation workflows. | | Plain terminal output | --plain-output | — | Off (TUI panel shown by default). | Example project-level config file: ``` # .dr-xp.yaml (project root) entity_type: experiment_container port: 8090 # enable_evaluation: true # or set DR_EXPERIMENT_ENABLE_EVALUATION=true in .env ``` For Agentic Starter projects, add evaluation support to `.env`: ``` # .env (project root) — enables Evaluation tab with dr run dev / task infra:dev DR_EXPERIMENT_ENABLE_EVALUATION=true ``` For judge-based evaluations, also set credentials in `.env`: ``` DATAROBOT_API_TOKEN= DATAROBOT_ENDPOINT=https://app.datarobot.com/api/v2 ``` Judge-free benchmarks run without judge credentials. See [Batch agent evaluation](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-evaluation.html). ### Start the local agent The dashboard visualizes traces from agent activity. Batch evaluations require a running agent endpoint. Start the full development stack from the project root: ``` dr run dev ``` This typically starts the agent on `http://localhost:8842/v1` and may also launch the dashboard on port `8090`. See [Quick start](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html#quick-start) for the flag/env var that enables the Evaluation tab. To start services individually: ``` task agent:dev # Agent on port 8842. task infra:dev # Dashboard on port 8090. ``` When `task dev` runs, `dr xp` may start from both `agent:dev` and `infra:dev`. The message `Already running at http://127.0.0.1:8090` is expected. ### Start the experimentation dashboard Choose one of the following options. #### Standalone start From the App Framework project root: ``` dr xp --entity-id ``` Add `--enable-evaluation` when the Evaluation tab is needed. See [Enable batch evaluation workflows](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-evaluation.html#enable-batch-evaluation). #### Infra task (Agentic Starter projects) ``` task infra:dev ``` This runs `dr xp --plain-output` without `--enable-evaluation`. To enable batch evaluation workflows, set `DR_EXPERIMENT_ENABLE_EVALUATION=true` in the project `.env`. #### Bundled with full dev ``` dr run dev ``` Expected startup output: ``` Starting server... Local Experimentation running on http://127.0.0.1:8090 for experiment_container: ``` If a server is already listening on the port, the CLI prints: ``` Already running at http://127.0.0.1:8090 — no new server started. ``` The server runs in the foreground. Stop it with Ctrl + C. ### Verify the server (optional) Confirm the dashboard responds: ``` curl -s http://127.0.0.1:8090/api/info ``` Example response with evaluation enabled: ``` {"entity_type":"experiment_container","entity_id":"","features":["traces","evaluation"]} ``` When evaluation is disabled, `features` contains `"traces"` only and evaluation API routes return `404`. ### Open the GUI in a browser Open one of the following URLs: | Environment | URL | | --- | --- | | Local machine | http://localhost:8090 or http://127.0.0.1:8090. | | DataRobot codespace or notebook | Exposed-port URL for port 8090 (shown when dr run dev or task dev completes). | The page title is Experimentation UI. Available tabs depend on enabled features: | Tab | Doc | Requires | | --- | --- | --- | | Traces | Local tracing | Plugin installed and agent running. | | Evaluation | Batch agent evaluation | --enable-evaluation (or DR_EXPERIMENT_ENABLE_EVALUATION=true) and the evaluation component installed. | ### Confirm end-to-end behavior 1. Send a test message to the agent (for example at http://localhost:5173 ). 2. Refresh the dashboard. A new trace appears in the Traces tab. See Local tracing . 3. (Optional) When evaluation is enabled, open the Evaluation tab, select a pipeline, dataset, and agent endpoint ( http://localhost:8842/v1 ), then start a run. See Batch agent evaluation . ## GUI access troubleshooting | Symptom | Fix | | --- | --- | | dr: command not found | Reinstall the CLI or add the binary directory to PATH; see Getting started. | | Plugin not found | Run dr plugin install xp. | | Missing entity ID | Run dr start to create pulumi_config.json, or pass --entity-id . | | Port 8090 already in use | List the process with lsof -i :8090 (macOS/Linux), note the PID, and run kill . On Windows, run netstat -ano \| findstr :8090, then taskkill /PID /F. Alternatively, start on another port with --port 8091. | | No Evaluation tab | See Evaluation troubleshooting. | | No traces in the dashboard | See Tracing troubleshooting. | | Evaluation APIs return 404 | The running instance was started without --enable-evaluation; restart with the flag enabled. | ## Configuration reference Use these options when non-default entity types, ports, or environment-based configuration is required. ### Command options ``` dr xp --entity-type [TYPE] --entity-id [ID] --port [PORT] [--enable-evaluation] [--plain-output] ``` The server runs in the foreground and logs to the terminal. Stop it with Ctrl + C. For a step-by-step walkthrough, see [Access the GUI](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html#access-the-gui). - --entity-id is required (falls back to DATAROBOT_USE_CASE_ID in pulumi_config.json if present). - --entity-type defaults to experiment_container (use case). Also accepts the alias use_case . - --enable-evaluation enables batch evaluation workflows in the GUI when the evaluation component is present in the project. Without this flag, the dashboard exposes traces only. - --plain-output streams logs directly without the persistent TUI panel (used by task infra:dev ). - Value precedence: flag > environment variable > config file ( .dr-xp.yaml ) > pulumi_config.json > default. Run `dr xp --help` for the full flag list, including optional feature gates ( `--enable-logs`, `--enable-metrics`, `--enable-mcp-inspector`, and others). ### Environment variables Add these variables to the `.env` file as needed. - DR_EXPERIMENT_ENTITY_TYPE (default: experiment_container ) - DR_EXPERIMENT_ENTITY_ID - DR_EXPERIMENT_PORT (default: 8090 ) - DR_EXPERIMENT_ENABLE_EVALUATION (set to true for evaluation workflows) ### Supported entity types | Entity type | Description | | --- | --- | | experiment_container (alias: use_case) | Use case experiment container (default). | | deployment | Model deployment. | | custom_application | Custom application. | | workload | Workload. | ## See also - Local tracing —inspect OpenTelemetry spans during development. - Batch agent evaluation —score agent responses on a fixed dataset. - Implement tracing —add custom spans and attributes. - DataRobot CLI getting started —install and configure the CLI. - af-component-evaluation repository —evaluation component source and reference docs. --- # DataRobot OpenCode plugin URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/opencode-plugin.html > Try OpenCode, an open source terminal coding agent, wired to the DataRobot LLM Gateway with DataRobot agent skills preinstalled. The `dr opencode` plugin connects [OpenCode](https://opencode.ai/) (an open source AI coding agent that runs in your terminal) directly into the DataRobot LLM Gateway. Every model your account has access to through DataRobot is available inside a single tool, allowing you to switch models to suit the task at hand. The plugin also installs [DataRobot agent skills](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-skills.html), including the [Agent Assist](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/index.html) skill. ## Quick start To use the DataRobot OpenCode plugin, in a [DataRobot codespace](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html) or on your local machine, ensure you have the following: - A DataRobot account with LLM Gateway access. - TheDataRobot CLI. If you're running OpenCode locally (not in acodespace) and don't havedryet, install it with Homebrew: brewinstalldatarobot-oss/taps/dr-cli SeeGetting startedfor other installation methods (Linux, Windows, or a specific version). Run the commands outlined below in the terminal of your preferred environment. First, if you are using an older version of the CLI, update it to a version that supports the `opencode` plugin (v0.2.76 or later): ``` dr self update --force ``` Then, install the plugin and launch OpenCode: ``` # Install the OpenCode plugin dr plugin install opencode # Launch OpenCode dr opencode ``` If you haven't already authenticated the CLI, `dr opencode` runs `dr auth login` first, since the plugin requires DataRobot credentials to reach the LLM Gateway. From there, the first launch installs the OpenCode binary for your platform and starts the terminal UI in your current directory. A DataRobot theme is active by default the first time you launch OpenCode. If you switch to a different theme with `/themes` (or the keybind Ctrl + X, then T), that choice is remembered on future `dr opencode` launches—there's no config file to edit by hand. > [!TIP] Codespace terminal > Inside a [codespace](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html), open the integrated terminal and run the same three commands. If `DATAROBOT_API_TOKEN` and `DATAROBOT_ENDPOINT` are already set in that environment, you may not need to run `dr auth login` separately. ## Plugin configuration Each time you run `dr opencode`, the plugin generates a fresh OpenCode config in its own config directory— `$XDG_CONFIG_HOME/datarobot/opencode` (or `~/.config/datarobot/opencode` if `XDG_CONFIG_HOME` isn't set)—and points OpenCode at it for that session. > [!NOTE] Config separation > The plugin's config is separate from OpenCode's own default config location, so it never conflicts with an existing global OpenCode setup. Because the plugin's manifest requires authentication, `dr` automatically forwards your DataRobot endpoint and API token to that process. The generated config uses them to register a `datarobot` provider backed by your organization's LLM Gateway catalog (defaulting to a Claude Sonnet 4.6 model from whichever provider offers one, or otherwise the first model in your catalog, until you pick one yourself). Inside an OpenCode session, list and switch models with the `/models` command (or the keybind Ctrl + X, then M): ``` /models ``` Models appear as `datarobot/`. You can also cycle recently used models with F2, or pin a specific model for a single launch: ``` dr opencode --model datarobot/ ``` This lets you pick a different model per task—for example, a fast model for quick edits and a stronger reasoning model for a tricky refactor—without leaving your terminal. **Config files are regenerated on every launch** The generated `opencode.json` and `tui.json` are rebuilt from scratch each time you run `dr opencode`, so don't hand-edit them—changes won't survive the next launch. The one exception is your model and theme picks: whatever you last selected with `/models` or `/themes` is read back from OpenCode's own state ( `~/.local/state/opencode`) and carried forward automatically, instead of resetting to the DataRobot default every time. ## Use DataRobot agent skills The generated config also registers the `opencode-datarobot-skills` plugin, which installs the [DataRobot agent skills](https://github.com/datarobot-oss/datarobot-agent-skills) (including [Agent Assist](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/agent-assist-skill.html)) and DataRobot theme variants into OpenCode's skills and themes directories the first time it runs. OpenCode surfaces installed skills to the agent automatically through its built-in `skill` tool—you don't need to load them manually. To see what's available or trigger one, just ask in the session, for example: ``` What DataRobot skills are available? Use the datarobot-model-deployment skill to deploy this model. ``` ## Already using Agent Assist? If you've already configured [Agent Assist](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/index.html)'s external LLM settings, `dr opencode` picks up the same environment variables and routes to that model instead of the LLM Gateway catalog—no extra setup required: | Variable | Effect in dr opencode | | --- | --- | | AGENT_ASSIST_LLM_BASE_URL | Adds the external model as an agent-assist provider and sets it as the default, alongside the LLM Gateway models. | | AGENT_ASSIST_LLM_MODEL_NAME | The model name used for that provider. | | AGENT_ASSIST_LLM_API_KEY | Credential for the external provider; also becomes the LLM Gateway catalog credential if no base URL is set. | | AGENT_ASSIST_DISABLE_LLM_GATEWAY | Skips the LLM Gateway catalog fetch entirely and uses only the external model (requires AGENT_ASSIST_LLM_BASE_URL and AGENT_ASSIST_LLM_MODEL_NAME). | See the [environment and commands reference](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/env-and-commands-reference.html) for what each variable does in Agent Assist itself. ## Update or remove the plugin To update the plugin to the latest version, run: ``` dr plugin update opencode ``` ``` dr plugin uninstall opencode ``` ## Related documentation For more information, see the following documentation: - dr plugin command reference - dr llm-gateway command reference - DataRobot agentic skills - OpenCode documentation --- # DataRobot CLI overview URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/overview.html > Learn about what the DataRobot CLI is for, how it's used, and how the CLI documentation is organized. The DataRobot CLI ( `dr`) is an open source tool for working with DataRobot from your terminal. It is designed for developers and operators who want repeatable, scriptable workflows instead of doing everything from the DataRobot UI. ## What the CLI is for Use the CLI to: - Authenticate against your DataRobot instance (cloud or self-managed) and locally manage credentials. - Work with application templates. Browse, clone, and configure projects built from templates, including an interactive setup when a template defines prompts and environment variables. - Run local development tasks. Execute the same Task-based workflows your template expects (for example, dr run dev , dr run build , dr run test ) so your machine matches how the app is meant to be built and run. - Support Agentic AI workflows. Bootstrap and run agent-oriented setups with commands such as dr start and dr task run . The CLI does not replace the DataRobot web application for every task; it focuses on local development, template-driven apps, and automation (including CI/CD) where a terminal interface fits best. ## Common workflow Most users follow a path like this: 1. Install dr in your workspace (see Quick install or the full Getting started guide). 2. Point the CLI at your environment and sign in (see Authentication management ){ target=_blank }. 3. Clone or set up a template if you are building from an application template (see Working with templates ). 4. Run tasks defined for that project ( dr task list , dr run ) as you develop. 5. Inspect local traces with dr xp during agent development (see Local tracing ). If you are building [Agentic AI](https://docs.datarobot.com/en/docs/agentic-ai/index.html) workflows, the [Agentic AI get started guide](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html) walks through `dr start` and `dr task run` end to end. ## Where to go next | If you want to… | Start here | | --- | --- | | Install and configure the CLI for the first time | Getting started | | Look up commands without narrative | Quick reference | | Understand template layout and .env behavior | Template system | | Read per-command details | Command reference (below) | > [!TIP] Building agentic workflows? > The CLI is used to set up, run, and deploy [Agentic AI](https://docs.datarobot.com/en/docs/agentic-ai/index.html) workflows. See the [Agentic AI get started guide](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-get-started.html) to get started with `dr start` and `dr task run`. ## Quick install Install the latest version with a single command that auto-detects your operating system: macOS/Linux: ``` curl https://cli.datarobot.com/install | sh ``` Windows (PowerShell): ``` irm https://cli.datarobot.com/winstall | iex ``` For more installation options, see [Getting Started](https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html). ## Documentation structure ### User guide End-user documentation for using the CLI: - Getting started —installation and initial setup. - Quick reference —one-page command and path reference. - Authentication —setting up DataRobot credentials. - Working with templates —clone and manage application templates. - Shell completions —set up command auto-completion. - Configuration files —understanding config file structure. - Troubleshooting —common issues and where to find solutions. - DataRobot experimentation plugin —local dashboard for agent development; see Access the GUI for installation through opening the browser. - Local tracing —inspect OpenTelemetry spans in the Traces tab. - Batch agent evaluation —score agent responses in the Evaluation tab (preview). - DataRobot OpenCode plugin —run the open source OpenCode coding agent wired to the DataRobot LLM Gateway with agent skills preinstalled. ### Template system Understanding the interactive template configuration: - Template structure —how templates are organized. - Interactive configuration —the wizard system explained. - Environment variables —managing .env files. ### Command reference The [command reference](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/index.html) lists every `dr` command, global flags, and guidance on when to use `dr start` versus `dr run`, plus links into each topic. Common entry points: - auth —authentication management. - run —task execution. - dotenv —environment variable management. - self —CLI utility commands (version, completion). ### Development guide For contributors and developers: - Building from source —compile and build the CLI. - Development setup —local development environment. - Project structure —codebase organization. - Releasing —release process. ## Getting help If the answer isn't in this documentation, the [GitHub repository](https://github.com/datarobot-oss/cli) provides additional support: 1. Search existing issues . 2. Open a new issue . 3. Email: oss-community-management@datarobot.com. --- # Quick reference URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/quick-reference.html > One-page reference for common DataRobot CLI commands and paths. ## Quick reference One-page reference for the most common DataRobot CLI ( `dr`) commands and paths. ## Installation ``` # macOS/Linux curl https://cli.datarobot.com/install | sh # Windows (PowerShell) irm https://cli.datarobot.com/winstall | iex ``` ## Authentication ``` dr auth set-url https://app.datarobot.com # or 1=US, 2=EU, 3=JP dr auth login dr auth logout ``` ## Templates ``` dr templates list dr templates setup dr templates clone [directory] dr templates status ``` ## Running applications ``` dr start # Quickstart or setup wizard dr task list # List tasks dr run dev # Run dev task dr run build dr run test ``` ## Environment ``` dr dotenv setup dr dotenv edit dr dotenv validate dr dotenv update ``` ## Local experimentation (dr xp) ``` dr plugin install xp dr xp --entity-id # Traces tab (default) dr xp --enable-evaluation --entity-id # Evaluation tab (preview) curl -s http://127.0.0.1:8090/api/info # Verify enabled features ``` | Setting | Flag / env | Default | | --- | --- | --- | | Port | --port / DR_EXPERIMENT_PORT | 8090 | | Entity ID | --entity-id / DR_EXPERIMENT_ENTITY_ID | DATAROBOT_USE_CASE_ID in pulumi_config.json | | Evaluation tab | --enable-evaluation / DR_EXPERIMENT_ENABLE_EVALUATION=true | Off | See [DataRobot experimentation plugin](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/index.html), [Local tracing](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-tracing.html), and [Batch agent evaluation](https://docs.datarobot.com/en/docs/agentic-ai/cli/local-experimentation/experimentation-evaluation.html). ## CLI self-management ``` dr --version dr self version dr self update dr self config dr self completion bash | sudo tee /etc/bash_completion.d/dr ``` ## Global flags | Flag | Description | | --- | --- | | -v, --verbose | Verbose output | | --debug | Debug output (creates dr-tui-debug.log) | | --skip-auth | Skip authentication (advanced) | | --force-interactive | Force setup wizard to run again | | -h, --help | Help | ## Config and state paths | Platform | Config file | | --- | --- | | Linux/macOS | ~/.config/datarobot/drconfig.yaml | | Windows | %USERPROFILE%\.config\datarobot\drconfig.yaml | | Location | Purpose | | --- | --- | | .datarobot/cli/state.yaml | Template state (per template directory) | ## See also - Getting started - Command reference - Configuration - Troubleshooting --- # Shell completions URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/shell-completions.html The DataRobot CLI supports auto-completion for Bash, Zsh, Fish, and PowerShell. Shell completions provide: - Command and subcommand suggestions. - Flag and option completions. - Faster command entry via tab completion. - Discovery of available commands. ## Installation ### Automatic installation You can use three different methods to install shell completions: 1. Installation script: Recommended for first-time installs. The installer automatically detects your shell and configures completions. ``` curl -fsSL https://raw.githubusercontent.com/datarobot-oss/cli/main/install.sh | sh ``` 1. Interactive command: Recommended for managing completions. This command detects your shell and installs completions to the appropriate location. ``` dr self completion install ``` 1. Manual installation : Recommended for advanced users. Follow the shell-specific instructions below. ### Interactive commands The CLI provides commands to easily manage completions. ``` # Install completions for your current shell dr self completion install # Force reinstall (useful after updates) dr self completion install --force # Uninstall completions dr self completion uninstall ``` ### Manual installation If you prefer manual installation or the automatic methods do not work, follow the instructions below. ### Bash #### Linux ``` # Generate and install the completion script dr self completion bash | sudo tee /etc/bash_completion.d/dr # Reload your shell source ~/.bashrc ``` ``` # If shell completion is not already enabled in your environment you will need # to enable it. You can execute the following command once: # echo "autoload -U compinit; compinit" >> ~/.zshrc # To load completions for each session, execute the following command once: $ dr self completion zsh > "${fpath[1]}/_dr" ``` #### macOS The default shell in MacOS is `zsh`. Shell completions for `zsh` are typically stored in one of the following directories: - /usr/local/share/zsh/site-functions/ - /opt/homebrew/share/zsh/site-functions/ - ${ZDOTDIR:-$HOME}/.zsh/completions/ Run `echo $fpath` to see all possibilities. For example, if you wish to put CLI completions into ZDOTDIR, then run: ``` dr self completion zsh > ${ZDOTDIR:-$HOME}/.zsh/completions/_dr ``` #### Temporary session For the current session only: ``` source <(dr self completion bash) ``` ### Zsh #### Setup First, ensure completion is enabled in your `~/.zshrc`: ``` # Add these lines if not already present autoload -U compinit compinit ``` #### Installation ``` # Create completions directory if it doesn't exist mkdir -p ~/.zsh/completions # Generate completion script dr self completion zsh > ~/.zsh/completions/_dr # Add to fpath in ~/.zshrc (if not already there) echo 'fpath=(~/.zsh/completions $fpath)' >> ~/.zshrc # Reload your shell source ~/.zshrc ``` #### Alternative (using system directory) ``` # Generate and install the completion script dr self completion zsh > "${fpath[1]}/_dr" # Clear completion cache rm -f ~/.zcompdump # Reload your shell source ~/.zshrc ``` #### Temporary session For the current session only: ``` source <(dr self completion zsh) ``` ### Fish ``` # Generate and install the completion script dr self completion fish > ~/.config/fish/completions/dr.fish # Reload fish configuration source ~/.config/fish/config.fish ``` #### Temporary session For the current session only: ``` dr self completion fish | source ``` ### PowerShell #### Persistent installation To add the CLI to your PowerShell profile: ``` # Generate the completion script dr self completion powershell > dr.ps1 # Find your profile location echo $PROFILE # Add the following line to your profile . C:\path\to\dr.ps1 ``` Alternatively, you can install it directly: ``` # Add to profile dr self completion powershell >> $PROFILE # Reload profile . $PROFILE ``` #### Temporary session For the current session only: ``` dr self completion powershell | Out-String | Invoke-Expression ``` ## Usage Once installed, completions work automatically when you press `Tab`: ### Command completion ``` # Type 'dr' and press Tab to see all commands dr # Shows: auth, completion, dotenv, run, templates, version # Type 'dr auth' and press Tab to see subcommands dr auth # Shows: login, logout, set-url # Type 'dr templates' and press Tab dr templates # Shows: clone, list, setup, status ``` ### Flag completion ``` # Type a command and -- then Tab to see flags dr run -- # Shows: --concurrency, --dir, --exit-code, --help, --list, --parallel, --silent, --watch, --yes # Partial flag matching works too dr run --par # Completes to: dr run --parallel ``` ### Argument completion For commands that support it: ``` # Template names when using clone dr templates clone # Shows available template names from DataRobot # Task names when using run (if in a template directory) dr run # Shows available tasks from Taskfile ``` ## Verification Test that completions are working: ``` # Try command completion dr te # Should complete to: dr templates # Try flag completion dr run --l # Should complete to: dr run --list ``` ## Troubleshooting ### Completions not working #### Bash Important: Bash completions require the `bash-completion` package to be installed first. 1. Install bash-completion if not already installed: ``` # macOS (Homebrew) brew install bash-completion@2 # Ubuntu/Debian sudo apt-get install bash-completion # RHEL/CentOS sudo yum install bash-completion ``` 1. Check that bash-completion is loaded: ``` # macOS brew list bash-completion@2 # Linux (Ubuntu/Debian) dpkg -l | grep bash-completion ``` 1. Verify completion script location: ``` ls -l /etc/bash_completion.d/dr # or on macOS ls -l $(brew --prefix)/etc/bash_completion.d/dr ``` 1. Check .bashrc sources completion: ``` grep bash_completion ~/.bashrc ``` 1. Reload your shell: source~/.bashrc #### Zsh 1. Verifycompinitis in~/.zshrc: grepcompinit~/.zshrc 2. Check completion file location: ``` ls -l ~/.zsh/completions/_dr # or echo $fpath[1] ls -l $fpath[1]/_dr ``` 1. Clear completion cache: rm-f~/.zcompdump 2. Reload Zsh: execzsh #### Fish 1. Check that the completion file exists: ls-l~/.config/fish/completions/dr.fish 2. Verify that Fish recognizes it: complete-Cdr 3. Reload Fish: source~/.config/fish/config.fish #### PowerShell 1. Check execution policy: Get-ExecutionPolicy If it's `Restricted`, change it: ``` Set-ExecutionPolicy RemoteSigned -Scope CurrentUser ``` 1. Verify profile loads completion: cat$PROFILE 2. Reload profile: .$PROFILE ### Permission denied If you get permission errors when installing: ``` # Use sudo for system-wide installation sudo dr completion bash > /etc/bash_completion.d/dr # Or use user-level installation dr completion bash > ~/.bash_completions/dr source ~/.bash_completions/dr ``` ### Completion cache issues For Zsh, if completions are outdated: ``` # Clear cache rm -f ~/.zcompdump* # Rebuild cache compinit ``` ## Advanced configuration ### Custom completion behavior You can customize how completions work by modifying the generated script. For example, in the Bash completion script, you can add custom completion logic: ``` # Extract the generated script dr completion bash > ~/dr-completion.bash # Edit the script to add custom logic vim ~/dr-completion.bash # Source it in your .bashrc source ~/dr-completion.bash ``` ### Multiple shell support If you use multiple shells, install completions for each: ``` # Install for all shells you use dr completion bash > ~/.bash_completions/dr dr completion zsh > ~/.zsh/completions/_dr dr completion fish > ~/.config/fish/completions/dr.fish ``` ## Updating completions When the CLI is updated, regenerate completions: ``` # Bash dr completion bash | sudo tee /etc/bash_completion.d/dr # Zsh dr completion zsh > ~/.zsh/completions/_dr rm -f ~/.zcompdump # Fish dr completion fish > ~/.config/fish/completions/dr.fish # PowerShell dr completion powershell > $PROFILE ``` ## See also - Getting started setup guide - Command reference - Cobra documentation (underlying completion framework) --- # Environment variables URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/template-system/environment-variables.html > Manage environment variables and .env files in DataRobot templates with interactive configuration tools. This page outlines how to manage environment variables and `.env` files in DataRobot templates. DataRobot templates use `.env` files to store configuration variables needed by your application. The CLI provides tools to: - Create .env files from templates - Interactively edit variables - Validate configuration - Securely manage secrets ## File structure ### .env.template The template provided by the repository (committed to Git): ``` # Required configuration APP_NAME= DATAROBOT_ENDPOINT= DATAROBOT_API_TOKEN= # Optional configuration # DEBUG=false # LOG_LEVEL=info # PORT=8080 # Database configuration # DATABASE_URL= # DATABASE_POOL_SIZE=10 # Cache configuration # CACHE_ENABLED=false # CACHE_URL= ``` #### Characteristics - Committed to version control - Contains empty required variables - Comments indicate optional variables - Includes documentation comments ### .env The actual configuration file (never committed): ``` # Required configuration APP_NAME=my-awesome-app DATAROBOT_ENDPOINT=https://app.datarobot.com DATAROBOT_API_TOKEN=*** # Optional configuration DEBUG=true LOG_LEVEL=debug PORT=8000 # Database configuration DATABASE_URL=postgresql://localhost:5432/mydb DATABASE_POOL_SIZE=5 ``` Characteristics: - Generated from `.env.template`. - Contains actual values. - Never committed (in `.gitignore`). - User-specific configuration. ## Create environment files ### Use the wizard The interactive wizard guides you through configuration. ``` # In a template directory dr dotenv setup ``` or ``` # During template setup dr templates setup ``` #### Wizard workflow 1. Loads .env.template . 2. Discovers configuration prompts. 3. Shows interactive questions. 4. Validates inputs. 5. Generates an .env file. ### Manual creation To copy and edit a template manually: ``` # Copy the template cp .env.template .env # Edit the template with your preferred editor vim .env # Alternatively, use the CLI editor dr dotenv ``` ## Manage variables ### Interactive editor Launch the built-in editor to manage variables: ``` dr dotenv ``` #### Features - List all variables - Mask secrets (passwords, API keys) - Start wizard mode - Directly edit variables #### Commands ``` Variables found in .env: APP_NAME: my-awesome-app DATAROBOT_ENDPOINT: https://app.datarobot.com DATAROBOT_API_TOKEN: *** DEBUG: true Press w to set up variables interactively. Press e to edit the file directly. Press enter to finish and exit. ``` ### Wizard mode You can also interactively configure a template with prompts. ``` dr dotenv setup ``` #### Advantages - Guided setup - Built-in validation - Conditional prompts - Help text for each variable ### Direct editing To edit the file directly: ``` dr dotenv edit # Press 'e' to enter editor mode # Or use external editor vim .env ``` ## Variable types ### Required variables The following variables must be set before running the application: ``` # .env.template shows these without comments APP_NAME= DATAROBOT_ENDPOINT= DATAROBOT_API_TOKEN= ``` The wizard enforces that an application name must be provided. ``` Enter your application name > _ (Cannot proceed without entering a value) ``` ### Optional variables The following variables are optional and can be left empty (shown as comments): ``` # .env.template shows these with # prefix # DEBUG=false # LOG_LEVEL=info ``` The wizard allows you to skip binding these variables: ``` Enable debug mode? (optional) > None (leave blank) Yes No ``` ### Secret variables Sensitive values that should be masked during input and display. To define secret variables: ``` # In .datarobot/prompts.yaml prompts: - key: "api_key" env: "API_KEY" type: "secret_string" help: "Enter your API key" ``` #### Auto-detection Variables with names containing `PASSWORD`, `SECRET`, `KEY`, or `TOKEN` are automatically treated as secrets. #### Display behavior - The wizard input's secrets are masked with bullet characters (••••). - The editor view displays secrets as *** . - The actual file contains secrets as plain text values. #### Security best practices - Always add .env to .gitignore . - Use secret_string type for all sensitive values. - Never commit .env files to version control. ### Auto-generated secrets You can cryptographically secure random values for application secrets: ``` prompts: - key: "session_secret" env: "SESSION_SECRET" type: "secret_string" generate: true help: "Session encryption key (auto-generated)" ``` #### Features - Generates 32-character random string if no value exists. - Uses base64 URL-safe encoding. - Preserves existing values (only generates when empty). - User can override secrets with a custom value. ### Conditional variables These variables are only shown or required based on your other selections: ``` # In .datarobot/prompts.yaml prompts: - key: "enable_database" options: - name: "Yes" requires: "database_config" - name: "No" - key: "database_url" section: "database_config" env: "DATABASE_URL" help: "Database connection string" ``` If `Enable database = No`, then `DATABASE_URL` is not shown. ## Environment variable discovery The CLI discovers variables from multiple sources: ### 1. Template file (.env.template) ``` # Variables defined in template APP_NAME= PORT=8080 ``` ### 2. Prompt definitions (.datarobot/prompts.yaml) ``` prompts: - key: "app_name" env: "APP_NAME" help: "Application name" ``` ### 3. Existing .env file ``` # Previously configured values APP_NAME=my-app ``` ### 4. Current environment ``` # Shell environment variables export PORT=3000 ``` ### Merge priority The CLI merges in the following order of priority (highest priority first): 1. User input from wizard. 2. Current shell environment. 3. Existing .env values. 4. Template defaults. ## Common patterns ### Database configuration ``` # PostgreSQL DATABASE_URL=postgresql://user:password@localhost:5432/dbname DATABASE_POOL_SIZE=10 DATABASE_TIMEOUT=30 # MySQL DATABASE_URL=mysql://user:password@localhost:3306/dbname # MongoDB DATABASE_URL=mongodb://localhost:27017/dbname ``` ### Authentication ``` # API Key DATAROBOT_API_TOKEN=your_api_token_here # OAuth AUTH_PROVIDER=oauth2 AUTH_CLIENT_ID=client_id AUTH_CLIENT_SECRET=*** AUTH_REDIRECT_URL=http://localhost:8080/callback # JWT JWT_SECRET=*** JWT_EXPIRATION=3600 ``` ### Feature flags ``` # Enable/disable features FEATURE_ANALYTICS=true FEATURE_MONITORING=false FEATURE_CACHING=true # Or as comma-separated list ENABLED_FEATURES=analytics,caching ``` ### Logging ``` # Log level LOG_LEVEL=debug # debug, info, warn, error # Log format LOG_FORMAT=json # json, text # Log output LOG_OUTPUT=stdout # stdout, file # Log file path LOG_FILE=/var/log/app.log ``` ## Security best practices ### Never commit .env files Ensure that `.gitignore` includes: ``` # Environment variables .env .env.local .env.*.local # Keep templates !.env.template !.env.example ``` ### Use strong secrets ``` # ✓ Good - strong random secret JWT_SECRET=a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6 # ✗ Bad - weak secret JWT_SECRET=secret123 ``` Generate secure secrets: ``` # Random 32-byte hex string openssl rand -hex 32 ``` ### Restrict file permissions ``` # Only the owner can read/write chmod 600 .env # Verify ls -la .env # Should show: -rw------- (600) ``` ### Use different configs per environment ``` # Development .env.development # Staging .env.staging # Production .env.production ``` Load based on the environment: ``` export ENV=production dr run deploy ``` ### Avoid hardcoding in code ``` # ✗ Bad api_token = "abc123" # ✓ Good import os api_token = os.getenv("DATAROBOT_API_TOKEN") ``` ## Validation ### Validate with dr dotenv To validate your environment configuration against template requirements: ``` dr dotenv validate ``` This command validates the following: - All required variables defined in .datarobot/prompts.yaml . - Core DataRobot variables ( DATAROBOT_ENDPOINT , DATAROBOT_API_TOKEN ). - Conditional requirements based on selected options. - Both .env file and environment variables. #### Example output Successful validation: ``` Validating required variables: APP_NAME: my-app DATAROBOT_ENDPOINT: https://app.datarobot.com DATAROBOT_API_TOKEN: *** DATABASE_URL: postgresql://localhost:5432/db Validation passed: all required variables are set. ``` Validation errors: ``` Validating required variables: APP_NAME: my-app DATAROBOT_ENDPOINT: https://app.datarobot.com Validation errors: Error: required variable DATAROBOT_API_TOKEN is not set Description: DataRobot API token for authentication Set this variable in your .env file or run `dr dotenv setup` to configure it. Error: required variable DATABASE_URL is not set Description: PostgreSQL database connection string Set this variable in your .env file or run `dr dotenv setup` to configure it. ``` #### Use cases - Pre-flight checks before running tasks. - CI/CD pipeline validation. - Debugging missing configuration. - Troubleshooting application startup issues. ### Required variables check Commands like `dr run` automatically validate required variables. ``` $ dr run dev Error: Missing required environment variables: - APP_NAME - DATAROBOT_API_TOKEN Please run: dr dotenv setup ``` ### Format validation For variables with specific formats: ``` # URL validation DATAROBOT_ENDPOINT=https://app.datarobot.com # ✓ Valid DATAROBOT_ENDPOINT=not-a-url # ✗ Invalid # Port validation PORT=8080 # ✓ Valid PORT=99999 # ✗ Invalid (out of range) # Email validation EMAIL=user@example.com # ✓ Valid EMAIL=invalid # ✗ Invalid ``` ## Advanced features ### Variable substitution Reference other variables: ``` # Base URL BASE_URL=https://app.datarobot.com # API endpoint uses base URL API_ENDPOINT=${BASE_URL}/api/v2 # Full URL becomes: https://app.datarobot.com/api/v2 ``` ### Multi-line values For long values: ``` # Single line PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\nMIIE..." # Or use actual newlines PRIVATE_KEY="-----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC... -----END PRIVATE KEY-----" ``` ### Comments Document your configuration: ``` # Application Configuration APP_NAME=my-app # Application identifier PORT=8080 # HTTP server port # Database Configuration # Format: protocol://user:password@host:port/database DATABASE_URL=postgresql://localhost:5432/mydb ``` ## Troubleshooting ### Variables not loading ``` # Check .env exists ls -la .env # Verify format cat .env # Check for syntax errors # Each line should be: KEY=value ``` ### Secrets exposed ``` # Check .gitignore includes .env cat .gitignore | grep .env # Check Git status git status # Should NOT show .env # If .env is tracked, remove it git rm --cached .env git commit -m "Remove .env from tracking" ``` ### Permission errors ``` # Fix permissions chmod 600 .env # Verify ls -la .env ``` ### Variables not expanding ``` # Ensure proper syntax for variable substitution # Works: API_URL=${BASE_URL}/api # Doesn't work: API_URL=$BASE_URL/api # Missing braces ``` ### Configuration not working Use `dr dotenv validate` to diagnose issues: ``` # Validate configuration dr dotenv validate # If validation passes but issues persist, check: # 1. Environment variables override .env env | grep DATAROBOT # 2. Ensure .env is in correct location (repository root) pwd ls -la .env # 3. Check if application is loading .env file # Some applications need explicit .env loading ``` ## Common workflows ### Initial setup ``` cd my-template dr dotenv setup dr dotenv validate dr run dev ``` ### Update credentials ``` dr auth login dr dotenv update dr dotenv validate ``` ### Validate before deployment ``` dr dotenv validate && dr run deploy ``` ### Edit and validate ``` dr dotenv edit dr dotenv validate ``` ## See also - Interactive configuration : Configuration wizard details. - Template structure : Template organization. - dotenv command : dotenv command reference. --- # Template system URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/template-system/index.html > Pre-configured application scaffolds for building and deploying custom applications to DataRobot. DataRobot templates are pre-configured application scaffolds that help you quickly build and deploy custom applications to DataRobot. Each template includes: - Application source code - Configuration prompts - Environment setup tools - Task definitions - Documentation ## Documentation ### Core concepts - Template structure: How templates are organized. - Interactive configuration: The configuration wizard. - Environment variables: Managing .env files. ## Quickstart ### Using a template ``` # List available templates dr templates list # Interactive setup (recommended) dr templates setup # Manual setup dr templates clone my-template cd my-template dr dotenv setup dr run dev ``` ### Create a template ``` # 1. Create structure mkdir my-template cd my-template # 2. Add metadata mkdir .datarobot cat > .datarobot/prompts.yaml < .env.template < Taskfile.gen.yaml < Interactive configuration wizard for setting up DataRobot templates with smart prompts, validation, and conditional logic. The DataRobot CLI features a powerful interactive configuration system that guides users through setting up application templates with smart prompts, validation, and conditional logic. ## Overview The interactive configuration system is built using [Bubble Tea](https://github.com/charmbracelet/bubbletea), a Go framework for building terminal user interfaces. It provides: - Guided setup: A step-by-step wizard for configuration - Smart prompts: Context-aware questions with validation - Conditional logic: Show/hide prompts based on previous answers - Multiple input types: Text fields, checkboxes, and selection lists - Visual feedback: Beautiful terminal UI with progress indicators ## Architecture ### Components The configuration system consists of three main layers: ``` ┌─────────────────────────────────────────┐ │ User interface layer │ │ (Bubble Tea models and views) │ ├─────────────────────────────────────────┤ │ Business logic layer │ │ (Prompt processing and validation) │ ├─────────────────────────────────────────┤ │ Data layer │ │ (Environment discovery and storage) │ └─────────────────────────────────────────┘ ``` ### Key files - cmd/dotenv/model.go : The main dotenv editor model - cmd/dotenv/promptModel.go : Individual prompt handling - internal/envbuilder/discovery.go : Prompt discovery from templates - cmd/templates/setup/model.go : Template setup wizard orchestration ## Configuration flow ### 1. Template setup wizard When you run `dr templates setup`, the wizard flow is: Template selection: On the template list screen, use the arrow keys to navigate, press `/` to filter by search term, and press Enter to select a template and then enter the target directory name. Only templates available to your user are shown. For full details, see [Templates command - Template selection screen](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/templates.html#template-selection-screen). ``` Welcome screen ↓ DataRobot URL configuration (if needed) ↓ Authentication (if needed) ↓ Template selection ↓ Template cloning ↓ Environment configuration (skipped if previously completed) ↓ Completion ``` State-aware behavior: If `dr dotenv setup` has been successfully run in the past (tracked via state file), the Environment configuration step is automatically skipped. This allows you to re-run the template setup without re-configuring your environment variables. See [Configuration - State tracking](https://docs.datarobot.com/en/docs/agentic-ai/cli/configuration.html#state-tracking) for details. ### 2. Environment configuration The environment configuration phase (dotenv wizard): ``` Load .env template ↓ Discover user prompts (from .datarobot files) ↓ Initialize response map ↓ For each required prompt: ├── Display prompt with help text ├── Show options (if applicable) ├── Capture user input ├── Validate input ├── Update required sections (conditional) └── Move to next prompt ↓ Generate .env file ↓ Save configuration ``` ## Prompt types ### Text input prompts Simple text entry for values: ``` # Example from .datarobot/prompts.yaml prompts: - key: "database_url" env: "DATABASE_URL" help: "Enter your database connection string" default: "postgresql://localhost:5432/mydb" optional: false ``` User experience: ``` Enter your database connection string > postgresql://localhost:5432/mydb█ Default: postgresql://localhost:5432/mydb ``` ### Secret string prompts Secure text entry with input masking for sensitive values: ``` prompts: - key: "api_key" env: "API_KEY" type: "secret_string" help: "Enter your API key" optional: false ``` User experience: ``` Enter your API key > ••••••••••••••••••█ Input is masked for security ``` Features: - Input is masked with bullets (•). - Prevents shoulder-surfing and accidental exposure. - Stored as plain text in .env file (file should be in .gitignore ). ### Auto-generated secrets Secret strings can be automatically generated: ``` prompts: - key: "session_secret" env: "SESSION_SECRET" type: "secret_string" generate: true help: "Session encryption key (auto-generated)" optional: false ``` Behavior: - If no value exists, a cryptographically secure random string is generated. - Generated secrets are 32 characters long. - Uses base64 URL-safe encoding. - Only generates when value is empty (preserves existing secrets). User experience: ``` Session encryption key (auto-generated) > ••••••••••••••••••••••••••••••••█ A random secret was generated. Press Enter to accept or type a custom value. ``` ### Single selection prompts Choose one option from a list: ``` prompts: - key: "environment" env: "ENVIRONMENT" help: "Select your deployment environment" optional: false multiple: false options: - name: "Development" value: "dev" - name: "Staging" value: "staging" - name: "Production" value: "prod" ``` User experience: ``` Select your deployment environment > Development Staging Production ``` ### Multiple selection prompts Choose multiple options (checkboxes): ``` prompts: - key: "features" env: "ENABLED_FEATURES" help: "Select features to enable (space to toggle, enter to confirm)" optional: false multiple: true options: - name: "Analytics" value: "analytics" - name: "Monitoring" value: "monitoring" - name: "Caching" value: "caching" ``` User experience: ``` Select features to enable (Use Space to toggle and Enter to confirm) > [x] Analytics [ ] Monitoring [x] Caching ``` ### Optional prompts Prompts that can be skipped: ``` prompts: - key: "cache_url" env: "CACHE_URL" help: "Enter cache server URL (optional)" optional: true options: - name: "None (leave blank)" blank: true - name: "Redis" value: "redis://localhost:6379" - name: "Memcached" value: "memcached://localhost:11211" ``` ## Conditional prompts Prompts can be shown or hidden based on previous selections using the `requires` and `section` fields. ### Section-based conditions ``` prompts: - key: "enable_database" help: "Do you want to use a database?" multiple: true options: - name: "Yes" value: "yes" requires: "database_config" # Enables this section - name: "No" value: "no" database_config: # Only shown if enabled - key: "database_type" help: "Select database type" options: - name: "PostgreSQL" value: "postgres" - name: "MySQL" value: "mysql" - env: "DATABASE_URL" help: "Enter database connection string" ``` ### How it works 1. Initial state: All sections start as disabled 2. User selection: When you select an option with requires: "section_name" 3. Section activation: That section becomes enabled 4. Prompt display: Prompts with matching section: "section_name" are shown 5. Cascade: Newly shown prompts can activate additional sections ### Example flow ``` Q: Do you want to use a database? [x] Yes ← User selects this (requires: "database_config") → Section "database_config" is now enabled Q: Select database type (Now shown because section is enabled) > PostgreSQL Q: Enter database connection string (Also shown because section is enabled) > postgresql://localhost:5432/db ``` ## Prompt discovery The CLI automatically discovers prompts from `.datarobot` directories in your template. ### Discovery process ``` // From internal/envbuilder/discovery.go func GatherUserPrompts(rootDir string) ([]UserPrompt, []string, error) { // 1. Recursively find all .datarobot directories // 2. Load prompts.yaml from each directory // 3. Parse and validate prompt definitions // 4. Build dependency graph (sections and requires) // 5. Return ordered prompts with root sections } ``` ### Prompt file structure Create `.datarobot/prompts.yaml` in any directory: ``` my-template/ ├── .datarobot/ │ └── prompts.yaml # Root level prompts ├── backend/ │ └── .datarobot/ │ └── prompts.yaml # Backend-specific prompts ├── frontend/ │ └── .datarobot/ │ └── prompts.yaml # Frontend-specific prompts └── .env.template ``` Each `prompts.yaml`: ``` prompts: - key: "unique_key" env: "ENV_VAR_NAME" # Optional: Environment variable to set type: "secret_string" # Optional: "string" (default) or "secret_string" help: "Help text shown to user" default: "default value" # Optional optional: false # Optional: Can be skipped multiple: false # Optional: Allow multiple selections generate: false # Optional: Auto-generate random value (secret_string only) section: "section_name" # Optional: Only show if section enabled options: # Optional: List of choices - name: "Display Name" value: "actual_value" requires: "other_section" # Optional: Enable section if selected ``` ## UI components ### Prompt model Each prompt is rendered by a `promptModel` that handles: - Input capture (text field or list) - Visual rendering - State management - Validation - Success callback ``` type promptModel struct { prompt envbuilder.UserPrompt input textinput.Model // For text prompts list list.Model // For selection prompts Values []string // Captured values successCmd tea.Cmd // Callback when complete } ``` ### List rendering Custom item delegate for beautiful list rendering: ``` type itemDelegate struct { multiple bool // Show checkboxes } func (d itemDelegate) Render(w io.Writer, m list.Model, index int, listItem list.Item) { // Renders items with: // - Checkboxes for multiple selection // - Highlighting for current selection // - Proper spacing and styling } ``` ### State management The main model manages screen transitions: ``` type Model struct { screen screens // Current screen variables []variable // Loaded variables prompts []envbuilder.UserPrompt requires map[string]bool // Active sections envResponses map[string]string // User responses currentPromptIndex int currentPrompt promptModel } ``` ## Keyboard controls ### List navigation - ↑/↓ or j/k - Navigate list items - Space - Toggle checkbox (multiple selection) - Enter - Confirm selection - Esc - Go back to previous screen ### Text input - Type normally to enter text - Enter - Confirm input - Esc - Go back to previous screen ### Editor mode - w - Start wizard mode - e - Open text editor - Enter - Finish and save - Esc - Save and exit editor ## Advanced features ### Default values Prompts can have default values: ``` prompts: - key: "port" env: "PORT" help: "Application port" default: "8080" ``` Shown as: ``` Application port > 8080█ Default: 8080 ``` ### Secret values The CLI provides secure handling for sensitive values using the `secret_string` type: ``` prompts: - key: "api_key" env: "API_KEY" type: "secret_string" help: "Enter your API key" ``` Features: - Input is masked with bullet characters (••••) during entry. - Prevents accidental exposure of sensitive data. - Cryptographic auto-generation secures random values with generate: true . Auto-detection: Variables with names containing "PASSWORD", "SECRET", "KEY", or "TOKEN" are automatically treated as secrets in the editor view, displaying as `***` instead of the actual value. ### Generated secrets You can automatically generate secrets. ``` prompts: - key: "session_secret" env: "SESSION_SECRET" type: "secret_string" generate: true help: "Session encryption key" ``` When `generate: true` is set: - A 32-character cryptographically secure random string is generated if no value exists. - Uses base64 URL-safe encoding. - Preserves existing values (only generates for empty fields). - User can still override with a custom value. ### Merge environment variables The wizard intelligently merges: 1. Existing values from an .env file 2. Environment variables from the current shell 3. User responses from the wizard 4. Template defaults from .env.template Priority (highest to lowest): 1. User wizard responses 2. Current environment variables 3. Existing .env values 4. Template defaults ## Error handling ### Validation Prompts can validate input: ``` func (pm promptModel) submitInput() (promptModel, tea.Cmd) { pm.Values = pm.GetValues() // Don't submit if required and empty if !pm.prompt.Optional && len(pm.Values[0]) == 0 { return pm, nil // Stay on prompt } return pm, pm.successCmd // Proceed } ``` ### User feedback ``` // Visual feedback for errors if err != nil { sb.WriteString(errorStyle.Render("❌ " + err.Error())) } // Success indicators sb.WriteString(successStyle.Render("✓ Configuration saved")) ``` ## Integration example To add the interactive wizard to your template: ### 1. Create a prompts file `.datarobot/prompts.yaml`: ``` prompts: - key: "app_name" env: "APP_NAME" help: "Enter your application name" optional: false - key: "features" help: "Select features to enable" multiple: true options: - name: "Authentication" value: "auth" requires: "auth_config" - name: "Database" value: "database" requires: "db_config" - key: "auth_provider" section: "auth_config" env: "AUTH_PROVIDER" help: "Select authentication provider" options: - name: "OAuth2" value: "oauth2" - name: "SAML" value: "saml" - key: "database_url" section: "db_config" env: "DATABASE_URL" help: "Enter database connection string" default: "postgresql://localhost:5432/myapp" ``` ### 2. Create an environment template `.env.template`: ``` # Application settings APP_NAME= # Features ENABLED_FEATURES= # Authentication (if enabled) # AUTH_PROVIDER= # Database (if enabled) # DATABASE_URL= ``` ### 3. Run setup ``` dr templates setup ``` The wizard automatically discovers and uses your prompts. ## Best practices ### 1. Clear help text ``` # ✓ Good help: "Enter your PostgreSQL connection string (e.g., postgresql://user:pass@host:5432/db)" # ✗ Bad help: "Database URL" ``` ### 2. Sensible defaults ``` # Provide reasonable defaults default: "postgresql://localhost:5432/myapp" ``` ### 3. Organize with sections ``` # Group related prompts - key: "enable_monitoring" options: - name: "Yes" requires: "monitoring_config" - key: "monitoring_url" section: "monitoring_config" help: "Monitoring service URL" ``` ### 4. Use descriptive keys ``` # ✓ Good key: "database_connection_pool_size" # ✗ Bad key: "pool" ``` ### 5. Validate input Use `optional: false` for required fields: ``` prompts: - key: "api_token" env: "API_TOKEN" type: "secret_string" help: "Enter your DataRobot API key" optional: false # Required! ``` ### 6. Use secret types for sensitive data Always use `secret_string` for passwords, API keys, and tokens. ``` # ✓ Good prompts: - key: "database_password" env: "DATABASE_PASSWORD" type: "secret_string" help: "Database password" # ✗ Bad (exposes password during input) prompts: - key: "database_password" env: "DATABASE_PASSWORD" help: "Database password" ``` ### 7. Auto-generate secrets when possible Use `generate: true` for application secrets that don't need to be memorized. ``` # ✓ Good for session keys, encryption keys prompts: - key: "jwt_secret" env: "JWT_SECRET" type: "secret_string" generate: true help: "JWT signing key" # ✗ Don't auto-generate user credentials prompts: - key: "admin_password" env: "ADMIN_PASSWORD" type: "secret_string" help: "Administrator password" help: "Enter your DataRobot API key" optional: false # Required! ``` ## Testing prompts Test your prompt configuration: ``` # Dry run without saving dr dotenv setup # Check discovered prompts dr templates status # View generated .env cat .env ``` ## See also - Template structure : How templates are organized - Environment variables : Manage .env files - Command reference: dotenv : dotenv command documentation --- # Template system structure URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/template-system/structure.html > Understanding how DataRobot organizes and configures application templates. This page provides an understanding of how DataRobot organizes and configures application templates. ## Overview DataRobot templates are Git repositories that contain application code, configuration, and metadata to deploy custom applications to DataRobot. The CLI provides tools to clone, configure, and manage these templates. ## Template repository structure A typical template repository: ``` my-datarobot-template/ ├── .datarobot/ # Template metadata │ ├── prompts.yaml # Configuration prompts │ ├── config.yaml # Template settings │ └── cli/ # CLI-specific files │ └── bin/ # Quickstart scripts │ └── quickstart.sh ├── .env.template # Environment variable template ├── .taskfile-data.yaml # Taskfile configuration (optional) ├── .gitignore ├── README.md ├── Taskfile.gen.yaml # Generated task definitions ├── src/ # Application source code │ ├── app/ │ │ └── main.py │ └── tests/ ├── requirements.txt # Python dependencies └── package.json # Node dependencies (if applicable) ``` ## Template metadata ### .datarobot directory The `.datarobot` directory contains template-specific configuration: ``` .datarobot/ ├── prompts.yaml # User prompts for setup wizard ├── config.yaml # Template metadata └── README.md # Template-specific docs ``` ### prompts.yaml Defines interactive configuration prompts. See [Interactive configuration](https://docs.datarobot.com/en/docs/agentic-ai/cli/template-system/interactive-config.html) for more details. Review the example prompt yaml configuration below. ``` prompts: - key: "app_name" env: "APP_NAME" help: "Enter your application name" default: "my-app" optional: false - key: "deployment_target" env: "DEPLOYMENT_TARGET" help: "Select deployment target" options: - name: "Development" value: "dev" - name: "Production" value: "prod" ``` ### config.yaml Template metadata and settings: ``` name: "My DataRobot Template" version: "1.0.0" description: "A sample DataRobot application template" author: "DataRobot" repository: "https://github.com/datarobot/template-example" # Minimum CLI version required min_cli_version: "0.1.0" # Tags for discovery tags: - python - streamlit - machine-learning # Required DataRobot features requirements: features: - custom_applications permissions: - CREATE_CUSTOM_APPLICATION ``` ## Environment configuration ### .env.template Review a template for environment variables. Note that the commented lines are optional. ``` # Required configuration APP_NAME= DATAROBOT_ENDPOINT= # Optional configuration (commented out by default) # DEBUG=false # LOG_LEVEL=info # Database configuration (conditional) # DATABASE_URL=postgresql://localhost:5432/mydb # DATABASE_POOL_SIZE=10 # Authentication # AUTH_ENABLED=false # AUTH_PROVIDER=oauth2 ``` ### .env (Generated) Created by the CLI during setup, the `.env` file contains actual values. Note that `.env` should be in `.gitignore` and never committed. ``` # Required configuration APP_NAME=my-awesome-app DATAROBOT_ENDPOINT=https://app.datarobot.com # Optional configuration DEBUG=true LOG_LEVEL=debug # Database configuration DATABASE_URL=postgresql://localhost:5432/mydb DATABASE_POOL_SIZE=5 ``` ## Quickstart scripts Templates can optionally provide quickstart scripts to automate application initialization. These scripts are executed by the `dr start` command. Quickstart scripts must be placed in `.datarobot/cli/bin/`. ### Naming conventions Scripts must start with `quickstart` (case-sensitive): - ✅ quickstart - ✅ quickstart.sh - ✅ quickstart.py - ✅ quickstart-dev - ❌ Quickstart.sh (wrong casing) - ❌ start.sh (wrong name) If there are multiple scripts matching the pattern, the first one found in lexicographical order will be executed. ### Platform requirements Review the requirements for different platforms below. #### Unix/Linux/macOS - Must have executable permissions ( chmod +x ) - Can be any executable file (shell script, Python script, compiled binary, etc.) #### Windows - Must have an executable extension: .exe , .bat , .cmd , or .ps1 ### When to use quickstart scripts Quickstart scripts are useful for: - Multi-step initialization: When your application requires several setup steps - Dependency management: Install packages or tools before starting - Environment validation: Check prerequisites before launch - Custom workflows: Template-specific initialization logic ### Fallback behavior If `dr start` does not find a quickstart, it automatically launches the interactive `dr templates setup` wizard instead to ensure that you can always get started even without a custom script. ## Task definitions ### Taskfile.gen.yaml The CLI automatically generates `Taskfile.gen.yaml` to aggregate component tasks. This file includes a `dotenv` directive to load environment variables from `.env`. Important: Component taskfiles cannot have their own `dotenv` directives. The CLI detects conflicts and prevents generation if a component taskfile already has a `dotenv` declaration. The generated structure is shown below. ``` version: '3' dotenv: [".env"] includes: backend: taskfile: ./backend/Taskfile.yaml dir: ./backend frontend: taskfile: ./frontend/Taskfile.yaml dir: ./frontend ``` ### Component taskfiles Component directories define their own tasks: Review the structure of `backend/Taskfile.yaml` below. ``` version: '3' # Note: No dotenv directive are allowed here tasks: dev: desc: Start development server cmds: - python -m uvicorn src.app.main:app --reload test: desc: Run tests cmds: - pytest src/tests/ build: desc: Build application cmds: - docker build -t {{.APP_NAME}} . ``` ### Running tasks The `dr run` command requires a `.env` file to be present: ``` # List all available tasks dr run --list # Run a specific task dr run dev # Run multiple tasks dr run lint test # Run tasks in parallel dr run lint test --parallel ``` If you're not in a DataRobot template directory (no `.env` file), you'll see the following message: ``` You don't seem to be in a DataRobot Template directory. This command requires a .env file to be present. ``` ### Taskfile configuration data Template authors can optionally provide a `.taskfile-data.yaml` file to configure the generated Taskfile. This file allows specifying port numbers for development servers and other configuration data. See [dr task compose documentation](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/task.html#taskfile-data-configuration) for complete details on the file format and usage. ## Multi-level configuration Templates can have nested `.datarobot` directories for component-specific configuration: ``` my-template/ ├── .datarobot/ │ └── prompts.yaml # Root level prompts ├── backend/ │ ├── .datarobot/ │ │ └── prompts.yaml # Backend prompts │ └── src/ ├── frontend/ │ ├── .datarobot/ │ │ └── prompts.yaml # Frontend prompts │ └── src/ └── .env.template ``` ### Discovery order The CLI discovers prompts in this order: 1. Root .datarobot/prompts.yaml 2. Subdirectory prompts (depth-first search, up to depth 2) 3. Merged and deduplicated ### Example: backend prompts `backend/.datarobot/prompts.yaml`: ``` prompts: - key: "api_port" env: "API_PORT" help: "Backend API port" default: "8000" section: "backend" - key: "database_url" env: "DATABASE_URL" help: "Database connection string" section: "backend" ``` ### Example: frontend prompts `frontend/.datarobot/prompts.yaml`: ``` prompts: - key: "ui_port" env: "UI_PORT" help: "Frontend UI port" default: "3000" section: "frontend" - key: "api_endpoint" env: "API_ENDPOINT" help: "Backend API endpoint" default: "http://localhost:8000" section: "frontend" ``` ## Template lifecycle ### 1. Discovery Templates are discovered from DataRobot: ``` # List available templates dr templates list ``` Output: ``` Available templates: * python-streamlit - Streamlit application template * react-frontend - React frontend template * fastapi-backend - FastAPI backend template ``` ### 2. Cloning Clone a template to your local machine: ``` # Clone a specific template dr templates clone python-streamlit # Clone to a custom directory dr templates clone python-streamlit my-app ``` This: - Clones the Git repository - Sets up directory structure - Initializes configuration files ### 3. Configuration Configure the template interactively: ``` # Full setup wizard dr templates setup # Or configure existing template cd my-template dr dotenv setup ``` ### 4. Development Work on your application: ``` # Run development server (requires .env file) dr run dev # Run tests dr run test # Build for deployment dr run build ``` Note: All `dr run` commands require a `.env` file in the current directory. If you see an error about not being in a template directory, run `dr dotenv setup` to create your `.env` file. ### 5. Deployment Deploy to DataRobot: ``` dr run deploy ``` ## Template types ### Python templates ``` python-template/ ├── .datarobot/ ├── requirements.txt ├── setup.py ├── src/ │ └── app/ │ └── main.py ├── tests/ └── .env.template ``` #### Key features - Python dependencies in requirements.txt - Source code in src/ - Tests in tests/ ### Node.js templates ``` node-template/ ├── .datarobot/ ├── package.json ├── src/ │ └── index.js ├── tests/ └── .env.template ``` #### Key features - Node dependencies in package.json - Source code in src/ - npm scripts integration ### Multi-language templates ``` full-stack-template/ ├── .datarobot/ ├── backend/ │ ├── .datarobot/ │ ├── requirements.txt │ └── src/ ├── frontend/ │ ├── .datarobot/ │ ├── package.json │ └── src/ ├── docker-compose.yml └── .env.template ``` #### Key features - Separate backend and frontend - Component-specific configuration - Docker composition ## Best practices ### Version control Note: Always exclude `.env` and `Taskfile.gen.yaml` from version control. The CLI generates `Taskfile.gen.yaml` automatically. ``` # .gitignore should include: .env Taskfile.gen.yaml *.log __pycache__/ node_modules/ dist/ ``` ### Documentation Include a clear README. ``` # My template ## Quick start {: #quick-start } 1. Clone: `dr templates clone my-template` 2. Configure: `dr templates setup` 3. Run: `dr run dev` ## Available tasks {: #available-tasks } - `dr run dev`: development server. - `dr run test`: run tests. - `dr run build`: build for production. ``` ### Sensible defaults Provide defaults in `.env.template`. ``` # Good defaults for local development API_PORT=8000 DEBUG=true LOG_LEVEL=info ``` ### Clear prompts Use descriptive help text. ``` prompts: - key: "database_url" help: "PostgreSQL connection string (format: postgresql://user:pass@host:5432/dbname)" ``` ### 5. Organized structure Keep related files together. ``` src/ ├── api/ # API endpoints ├── models/ # Data models ├── services/ # Business logic └── utils/ # Utilities ``` ## Template updates ### Checking for updates ``` # Check current template status dr templates status # Shows: # - Current version # - Latest available version # - Modified files # - Available updates ``` ### Updating templates ``` # Update to latest version git pull origin main # Re-run configuration if needed dr dotenv setup ``` ## Creating your own template ### 1. Start with base structure ``` mkdir my-new-template cd my-new-template git init ``` ### 2. Add template files Create the necessary files: ``` # Configuration mkdir .datarobot touch .datarobot/prompts.yaml touch .env.template # Application structure mkdir -p src/app touch src/app/main.py # Tasks touch Taskfile.gen.yaml ``` ### 3. Define prompts `.datarobot/prompts.yaml`: ``` prompts: - key: "app_name" env: "APP_NAME" help: "Enter your application name" optional: false ``` ### 4. Create an environment template `.env.template`: ``` APP_NAME= DATAROBOT_ENDPOINT= ``` ### 5. Define tasks Create component Taskfiles (e.g., `backend/Taskfile.yaml`): ``` version: '3' tasks: dev: desc: Start development server cmds: - echo "Starting {{.APP_NAME}}" ``` ### 6. Configure Taskfile data Optional. Create `.taskfile-data.yaml` to provide additional configuration for the generated root taskfile: ``` # .taskfile-data.yaml # Optional configuration for dr task compose # Ports to display when running dev task ports: - name: Backend port: 8080 - name: Frontend port: 5173 ``` This allows developers using your template to see which ports services run on when they execute `task dev`. ### 7. Test the template ``` # Test the setup locally dr templates setup # Verify configuration dr run --list ``` ### 8. Publish the template ``` # Push to GitHub git add . git commit -m "Initial template" git push origin main # Register with DataRobot (contact your admin) ``` ## See also - Interactive configuration : Configuration wizard details. - Environment variables : Manage .env files. - dr run : Task execution. - dr task compose : Taskfile composition and configuration. - Command reference: templates : Template commands. --- # Troubleshooting URL: https://docs.datarobot.com/en/docs/agentic-ai/cli/troubleshooting.html > Index of common issues and where to find solutions in the DataRobot CLI documentation. ## Troubleshooting This page links to troubleshooting and common-issue sections across the CLI documentation. ## Installation and setup | Issue | Where to look | | --- | --- | | "dr: command not found" | Getting started - Common issues: dr: command not found | | "Failed to read config file" | Getting started - Common issues: Failed to read config file | | Uninstalling the CLI | Getting started - Uninstalling the CLI | ## Authentication | Issue | Where to look | | --- | --- | | "Authentication failed" | Getting started - Common issues: Authentication failed | | Browser doesn't open for login | Auth command - Common issues: Browser doesn't open | | Port already in use (OAuth) | Auth command - Common issues: Port already in use | | Invalid credentials, connection refused, SSL issues | Auth command - Common issues | ## Configuration | Issue | Where to look | | --- | --- | | Config not loading, invalid config, permission denied | Configuration - Troubleshooting | | Multiple configs, state tracking | Configuration - State tracking | | Environment variables | Configuration - Environment variables reference | ## Templates and setup wizard | Issue | Where to look | | --- | --- | | Template selection (navigate, filter, directory prompt) | Templates command - Template selection screen | | Interactive configuration and keyboard controls | Interactive configuration - Keyboard controls | ## Tasks and running | Issue | Where to look | | --- | --- | | "Not in a DataRobot Template directory" / no .env | dr run - Error handling, Configuration - What counts as a template directory | | Dotenv directive conflict (Taskfile) | dr run - Dotenv directive conflict | | Task binary not found | dr run - Task binary not found | | Tasks not found, env vars not loading | dr run - Troubleshooting | | dr task compose errors | dr task - Error handling | ## Experimentation plugin (dr xp) | Issue | Where to look | | --- | --- | | Install plugin, configure, start dashboard, open browser, ports | DataRobot experimentation plugin - Access the GUI | | Port conflicts, missing entity ID, plugin not found | DataRobot experimentation plugin - GUI access troubleshooting | | No traces, filter, and search | Local tracing - Tracing troubleshooting | | Enable batch evaluation (--enable-evaluation), no Evaluation tab, judge errors | Batch agent evaluation | | Evaluation APIs return 404 | Batch agent evaluation - Enable batch evaluation workflows | ## Environment variables (dotenv) | Issue | Where to look | | --- | --- | | Not in repository, missing .env, auth required | dotenv command - Error handling | | Validation failures | dotenv command - Validation failures | ## Debugging | Tip | Where to look | | --- | --- | | Verbose and debug output | Getting started - Getting help | | Global flags (--verbose, --debug, --skip-auth) | Command reference - Global flags | | View current config | dr self config — self command | ## Get help - In-app help: dr --help , dr --help - Issues and discussions: GitHub Issues and GitHub Discussions - Email: oss-community-management@datarobot.com ## See also - Getting started - Command reference - Configuration - Using Agentic AI templates? If you're building or running agentic workflows, see Agentic AI troubleshooting for agent-specific issues (prerequisites, deployment, local testing). --- # Agentic workflow with code URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/agentic-example.html This notebook demonstrates a simple agentic workflow in DataRobot, showing how MLOps can be used to serve, monitor, and govern the workflow. After running the notebook, you will have DataRobot MLOps deployments for a simple agent that can also reliably perform basic arithmetic. This example also produces a separate deployment for the calculator tool used by the agent. ## Code asset overview The following code-based assets are used in this workflow, all of which you can download [here](https://datarobot-doc-assets.s3.us-east-1.amazonaws.com/simple_agent.zip). - calculator/custom.py : Custom deployment logic for the "calculator" tool that will allow the LLM to reliably perform simple arithmetic operations. - agent/custom.py : Custom deployment logic for the agent that will either respond directly to user prompts or alternatively first delegate to the calculator tool. - agent/requirements.txt : Python dependencies for the agent custom deployment. - agent/model-metadata.yaml : A configuration file for the agent deployment that specifies Azure OpenAI credentials and the identifier of the calculator deployment. - create_deployments.ipynb : This notebook file; includes code for creating and testing the deployments. ## Workflow outline 1. Create the calculator deployment 2. Update model-metadata.yaml and requirements.txt 3. Create the agent deployment 4. Test and make predictions with the agent deployment ## 1. Create the calculator deployment The following cell deploys the files in the calculator directory ( `calculator/custom.py`). Create a custom model deployment by importing the DataRobot package and using DataRobot MLOps' deployment creation methods. This model functions as a calculator. You can bring two numbers and a mathematic operation to the deployment, and the model will return the answer. ``` import datarobot as dr default_prediction_server_id = '' # Specify your prediction server here execution_environment_id = "5e8c889607389fe0f466c72d" cm_calc = dr.CustomInferenceModel.create(name='Calculator', target_name='result', target_type='TextGeneration') cmv_calc = dr.CustomModelVersion.create_clean(cm_calc.id, base_environment_id=execution_environment_id, folder_path='./calculator') rmv_calc = dr.RegisteredModelVersion.create_for_custom_model_version(cmv_calc.id) d_calc = dr.Deployment.create_from_registered_model_version(rmv_calc.id, 'Calculator', default_prediction_server_id=default_prediction_server_id) ``` ## 2. Provide credentials In your text editor of choice, update `agent/model-metadata.yaml` with your Azure Open AI credentials and the calculator deployment ID from step 1 ( `d_calc.id`). In production you should use the DataRobot credential store to expose secrets in the deployment. Update package versions in `agent/requirements.txt` to the following: ``` openai==1.55.3 pydantic==2.5.2 datarobot-predict==1.13.5 datarobot==3.4.0 ``` ## 3. Create the agent deployment Use the code below to create a deployment for the agent. ``` cm_agent = dr.CustomInferenceModel.create(name='Agent', target_name='completion', target_type='TextGeneration') cmv_agent = dr.CustomModelVersion.create_clean(cm_agent.id, base_environment_id=execution_environment_id, folder_path='./agent') dr.CustomModelVersionDependencyBuild.start_build(cm_agent.id, cmv_agent.id) rmv_agent = dr.RegisteredModelVersion.create_for_custom_model_version(cmv_agent.id) d_agent = dr.Deployment.create_from_registered_model_version(rmv_agent.id, 'Agent', default_prediction_server_id=default_prediction_server_id) ``` ## 4. Test the deployment The cells below communicate with the deployments by asking the calculator model a math problem ( `what is 4 x 752`) and receiving the answer retrieved by the agent. ``` from datarobot_predict.deployment import predict import pandas as pd import json messages = [ {'role': 'user', 'content': 'what is 4*752',} ] df, _ = predict(d_agent, pd.DataFrame([{'messages': json.dumps(messages)}])) df ``` ``` messages = [ {'role': 'user', 'content': 'hello',} ] df, _ = predict(d_agent, pd.DataFrame([{'messages': json.dumps(messages)}])) df ``` --- # Build and host a ChromaDB vector database URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/chromadb-vdb.html The following notebook provides an example of how you can build, validate, and register a vector database to the DataRobot platform using DataRobot's Python client. It describes how to load in and host a ChromaDB in-memory vector store, with metadata filtering, within a custom model. This notebook is designed for use with DataRobot Notebooks; DataRobot recommends downloading this notebook and uploading it for use in the platform. Note that when using ChromaDB-hosted documents with custom models, maximum file size is 1GB per file. ## Setup The following steps outline the necessary configuration to integrate vector databases with the DataRobot platform. 1. This workflow uses the following feature flags. Contact your DataRobot representative or administrator for information on enabling these features. 2. Use a codespace, not a DataRobot Notebook, to ensure this notebook has access to a filesystem. Use Python 3.12 to match the GenAI execution environment used for deployment. 3. Set the notebook session timeout to 180 minutes. 4. Restart the notebook container using at least a "Medium" (16GB RAM) instance. 5. Optionally, upload your documents archive to the notebook filesystem. ### Install libraries Install the following libraries: ``` # Upgrade pip to fix langchain installation issues !pip install --upgrade pip "setuptools<82" ``` If you are running this notebook on MacOS or Windows substitute `pysqlite3` for `pysqlite3-binary`. ``` !pip install "langchain" \ "langchain-community" \ "langchain-chroma" \ "langchain-text-splitters" \ "sentence-transformers==3.0.0" \ "datarobot" \ "datarobot-predict" \ "unstructured" \ "pysqlite3-binary" ``` ``` # replace sqlite3 with pysqlite3 to fix chroma issues __import__('pysqlite3') import sys sys.modules['sqlite3'] = sys.modules.pop('pysqlite3') ``` ``` import datarobot as dr from datarobot.models.genai.vector_database import CustomModelVectorDatabaseValidation from datarobot.models.genai.vector_database import VectorDatabase ``` ### Connect to DataRobot Read more about options for [connecting to DataRobot from the Python client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ### Download sample data This example references a sample dataset made from the DataRobot english documentation. To experiment with your own data, modify this section and/or the "Load and split text" section to reference your local dataset. Note: If you are a self-managed user, you must modify code samples that reference `app.datarobot.com` to the appropriate URL for your instance. ``` import requests, zipfile, io SOURCE_DOCUMENTS_ZIP_URL = "https://s3.amazonaws.com/datarobot_public_datasets/ai_accelerators/datarobot_english_documentation_5th_December.zip" UNZIPPED_DOCS_DIR = "datarobot_english_documentation" STORAGE_DIR = "storage" r = requests.get(SOURCE_DOCUMENTS_ZIP_URL) z = zipfile.ZipFile(io.BytesIO(r.content)) z.extractall(f"{STORAGE_DIR}/") ``` ### Load and split text Next, load the DataRobot documentation dataset and split it into chunks. If you are applying this recipe to a different use case, consider the following: - Use additional or alternative document loaders. - Filter out extraneous and noisy documents. - Choose an appropriate chunk_size and overlap . These are counted by number of characters, not tokens. ``` import re from langchain_community.document_loaders import DirectoryLoader, TextLoader from langchain_text_splitters import RecursiveCharacterTextSplitter SOURCE_DOCUMENTS_DIR = f"{STORAGE_DIR}/{UNZIPPED_DOCS_DIR}/" SOURCE_DOCUMENTS_FILTER = "**/*.txt" loader = DirectoryLoader( f"{SOURCE_DOCUMENTS_DIR}", glob=SOURCE_DOCUMENTS_FILTER, loader_cls=TextLoader, ) splitter = RecursiveCharacterTextSplitter( chunk_size=128, chunk_overlap=0, ) print(f"Loading {SOURCE_DOCUMENTS_DIR} directory") data = loader.load() print(f"Splitting {len(data)} documents") docs = splitter.split_documents(data) for doc in docs: doc.metadata['source'] = re.sub( rf'{STORAGE_DIR}/{UNZIPPED_DOCS_DIR}/datarobot_docs/en/(.+)\.md', r'https://docs.datarobot.com/en/docs/\1.html', doc.metadata['source'] ) doc.metadata["category"] = doc.metadata["source"].split("|")[-1].replace(".txt", "") print(f"Created {len(docs)} documents") ``` ## Create a vector database from documents Use the following cell to build a vector database from the DataRobot documentation dataset. Note that this notebook uses ChromaDB, an open source, in-memory vector store with metadata filtering support that is compatible with DataRobot Notebooks. Additionally, this notebook uses the HuggingFace `jina-embedding-t-en-v1` [embeddings model](https://huggingface.co/jinaai/jina-embedding-t-en-v1) (open source). ``` from datetime import datetime from langchain_chroma import Chroma from langchain_community.embeddings.sentence_transformer import (SentenceTransformerEmbeddings) CHROMADB_DATA_PATH = f"{STORAGE_DIR}/chromadb" CHROMADB_EMBEDDING_CACHE_FOLDER = STORAGE_DIR + '/sentencetransformers' CHROMADB_EMBEDDING_FUNCTION = SentenceTransformerEmbeddings(model_name="jinaai/jina-embedding-t-en-v1", cache_folder=CHROMADB_EMBEDDING_CACHE_FOLDER) def create_chromadb_from_documents(docs, embedding_function, persist_directory): start_time = datetime.now() print(f'>>> BEGIN ({start_time.strftime("%H:%M:%S")}): Creating ChromaDB from documents') print(f'Embedding function: {embedding_function}') print(f'ChromaDB data directory: {persist_directory}') print(' ') print(f'Documents for loading: {len(docs)}') db = Chroma.from_documents(docs, embedding_function, persist_directory=persist_directory) end_time = datetime.now() print(' ') print(f'>>> END ({end_time.strftime("%H:%M:%S")}): Creating ChromaDB from documents') print(f'Loaded {len(docs)} documents.') print(f"Chroma VectorDB now has {db._collection.count()} documents") total_elapsed_min = (end_time - start_time).total_seconds() / 60 document_average_sec = (end_time - start_time).total_seconds() / len(docs) print("Total Elapsed", "%.2f" % total_elapsed_min, "minutes") print("Document Average", "%.2f" % document_average_sec, "seconds") return db print(f"Created {len(docs)} documents") db = create_chromadb_from_documents(docs, CHROMADB_EMBEDDING_FUNCTION, CHROMADB_DATA_PATH) print(db._collection.count()) ``` ### Test the vector database Use the following cell to test the vector database by having the model perform a similarity search with metadata filtering; it will return the top five documents matching the query provided. ``` question = "What is MLOps?" top_k = 5 metadata_filter = {"category": {"$eq": "index"}} results_with_scores = db.similarity_search_with_score( question, k=top_k, filter=metadata_filter, ) print(len(results_with_scores)) for doc, score in results_with_scores: print("********************************************************************************") print(" ") print("----------") print(f"METADATA: {doc.metadata}, Score: {score}") print(" ") print("----------") print(f"CONTENT: {doc.page_content}") print(" ") ``` ## Define hooks for deploying an unstructured custom model The following cell defines the methods used to deploy an unstructured custom model. These include loading the custom model and using the model for scoring. ``` def load_model(input_dir): """Custom model hook for loading our knowledge base.""" import os print("Loading model") chromadb_data_path = os.path.join(input_dir, "chromadb") chromadb_embedding_cache_folder = os.path.join(input_dir, "sentencetransformers") chromadb_embedding_model_name = "jinaai/jina-embedding-t-en-v1" # https://docs.trychroma.com/troubleshooting#sqlite __import__('pysqlite3') import sys sys.modules['sqlite3'] = sys.modules.pop('pysqlite3') from langchain_chroma import Chroma from langchain_community.embeddings.sentence_transformer import SentenceTransformerEmbeddings print(f'CHROMADB_DATA_PATH = {chromadb_data_path}') print(f'CHROMADB_EMBEDDING_CACHE_FOLDER = {chromadb_embedding_cache_folder}') print(f'CHROMADB_EMBEDDING_MODEL_NAME = {chromadb_embedding_model_name}') chromadb_embedding_function = SentenceTransformerEmbeddings( model_name=chromadb_embedding_model_name, cache_folder=chromadb_embedding_cache_folder, ) db = Chroma( persist_directory=chromadb_data_path, embedding_function=chromadb_embedding_function, ) print(f'Loaded ChromaDB with {db._collection.count()} chunks') return db def score_unstructured(model, data, **kwargs) -> str: """Custom model hook for retrieving relevant docs with our knowledge base. When requesting predictions from the deployment, pass a dictionary with the following keys: - 'question' the question to be passed to the vector store retriever - 'metadata' the metadata filter to be passed to the vector store retriever - 'top_k' the number of results to return datarobot-user-models (DRUM) handles loading the model and calling this function with the appropriate parameters. Returns: -------- rv : str Json dictionary with keys: - 'question' user's original question - 'relevant' the generated answer to the question - 'metadata' - metadata for each document - 'error' - error message if exception in handling request """ import json try: data_dict = json.loads(data) question = data_dict['question'] top_k = data_dict.get("k", 10) metadata_filter = data_dict.get("filter", None) results_with_scores = model.similarity_search_with_score( question, k=top_k, filter=metadata_filter, ) print(f'Returned {len(results_with_scores)} results') relevant, metadata = [], [] for doc, score in results_with_scores: relevant.append(doc.page_content) doc.metadata["similarity_score"] = score metadata.append(doc.metadata) rv = { "question": question, "relevant": relevant, "metadata": metadata, } except Exception as e: rv = {'error': f"{e.__class__.__name__}: {str(e)}"} return json.dumps(rv), {"mimetype": "application/json", "charset": "utf8"} ``` ### Test hooks locally Before proceeding with deployment, use the cell below to test that the custom model hooks function correctly. ``` import json ## Test the hooks locally score_unstructured( load_model(f"{STORAGE_DIR}"), json.dumps( { "question": "How do I replace a custom model on an existing custom environment?", "filter": {"category": {"$eq": "index"}}, "k": 1, } ), ) ``` ## Deploy the knowledge base The cells below use the DataRobot Python client to: - Package the custom model hooks and vector database artifacts. - Create an unstructured custom model in the workshop. - Upload a new custom model version to the GenAI Python 3.12 execution environment. - Deploy the model and return a dr.Deployment object for predictions. ``` import inspect from pathlib import Path CUSTOM_MODEL_NAME = "External DR Knowledge Base using ChromaDB" DEPLOY_DIR = Path("storage/deploy_model") DEPLOY_DIR.mkdir(parents=True, exist_ok=True) (DEPLOY_DIR / "custom.py").write_text( inspect.getsource(load_model) + "\n\n" + inspect.getsource(score_unstructured) ) (DEPLOY_DIR / "requirements.txt").write_text( "langchain_chroma\n" "pysqlite3-binary\n" "langchain-community\n" "sentence-transformers==3.0.0\n" ) model_files = [ (str(DEPLOY_DIR / "custom.py"), "custom.py"), (str(DEPLOY_DIR / "requirements.txt"), "requirements.txt"), ] for artifact_dir in ("chromadb", "sentencetransformers"): source_root = Path("storage") / artifact_dir for path in source_root.rglob("*"): if path.is_file(): model_files.append( (str(path), f"{artifact_dir}/{path.relative_to(source_root).as_posix()}") ) genai_environment = dr.ExecutionEnvironment.list( search_for="[GenAI] Python 3.12 with Moderations" )[0] existing_models = [m for m in dr.CustomInferenceModel.list() if m.name == CUSTOM_MODEL_NAME] if existing_models: custom_model = existing_models[0] else: custom_model = dr.CustomInferenceModel.create( name=CUSTOM_MODEL_NAME, target_type=dr.TARGET_TYPE.UNSTRUCTURED, language="python", ) print("Uploading custom model version") model_version = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=genai_environment.id, files=model_files, maximum_memory=8 * 1024 * 1024 * 1024, max_wait=60 * 30, ) print("Building custom model dependency image") try: build_info = dr.CustomModelVersionDependencyBuild.start_build( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, max_wait=60 * 60, ) except dr.errors.ClientError as e: if "already has a dependency image" in str(e): print("Dependency build already exists, skipping build step") build_info = dr.CustomModelVersionDependencyBuild.get_build_info( custom_model.id, model_version.id, ) else: raise if build_info is not None and build_info.build_status != "success": raise RuntimeError( f"Dependency build failed with status '{build_info.build_status}'.\n" f"{build_info.get_log()}" ) print("Creating deployment") pred_server = dr.PredictionServer.list()[0] deployment = dr.Deployment.create_from_custom_model_version( model_version.id, label=CUSTOM_MODEL_NAME, default_prediction_server_id=pred_server.id, max_wait=60 * 30, ) print(f"Deployment ID: {deployment.id}") ``` ### Test the deployment Test that the deployment can successfully provide responses to questions using the [datarobot-predict](https://datarobot.github.io/datarobot-predict/) library. ``` import json from datarobot_predict.deployment import predict_unstructured # deployment = dr.Deployment.get("ADD_VALUE_HERE") content, response_headers = predict_unstructured( deployment=deployment, data={ "question": "How do I replace a custom model on an existing custom environment?", "filter": {"category": {"$eq": "index"}}, "k": 1, }, ) json.loads(content) ``` ## Validate and create the vector database These methods execute, validate, and integrate the vector database. This example associates a Use Case with the validation and creates the vector database within that Use Case. Set the `use_case_id` to specify an existing Use Case or create a new one with that name. ``` use_case_id = "ADD_VALUE_HERE" use_case = dr.UseCase.get(use_case_id) # UNCOMMENT if you want to create a new Use Case # use_case = dr.UseCase.create() ``` ### Validate the vector database The `CustomModelVectorDatabaseValidation.create` function executes the validation of the vector database. Be sure to provide the deployment ID. ``` external_vdb_validation = CustomModelVectorDatabaseValidation.create( prompt_column_name="question", target_column_name="relevant", deployment_id=deployment.id, use_case=use_case, wait_for_completion=True ) ``` ``` assert external_vdb_validation.validation_status == "PASSED" ``` ### Create the vector database After validation completes, use `VectorDatabase.create_from_custom_model()` to integrate the vector database. You must provide the Use Case name (or Use Case ID), a name for the external vector database, and the validation ID returned from the previous cell. ``` vdb = VectorDatabase.create_from_custom_model( name="DR Vector Database", use_case=use_case, validation_id=external_vdb_validation.id ) ``` ``` assert vdb.execution_status == "COMPLETED" ``` ``` print(f"Vector Database ID: {vdb.id}") ``` This vector database ID can now be used in the [GenAI E2E how-to](https://docs.datarobot.com/en/docs/gen-ai/genai-code/genai-e2e.html) to create the LLM blueprint with a vector database. --- # Create and deploy a DataRobot vector database URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/create-deploy-vdb-builtin-embeddings.html This notebook demonstrates how to use the DataRobot Python SDK to create and deploy a vector database (VDB) using DataRobot's built-in embeddings. This is the simplest approach for creating vector databases and doesn't require creating custom models. If you need to use your own embedding model (BYO embeddings), see the [Create vector databases from BYO embeddings](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/create-vdb-byo-embedding.ipynb) notebook for a complete example. This example workflow does the following: - Creates a Use Case and uploads a dataset. - Configures chunking parameters for your documents. - Creates a vector database using DataRobot's built-in embedding models. - Deploys vector databases for use in production. Note: For self-managed users, code samples that reference `app.datarobot.com` need to be changed to the appropriate URL for your instance. ## Setup ### Prerequisites This workflow requires the following feature flags. Contact your DataRobot representative or administrator for information on enabling these features: - Enable MLOps - Enable Public Network Access for all Custom Models (Premium) - Enable Monitoring Support for Generative Models - Enable Custom Inference Models - Enable GenAI Experimentation ### Import libraries This section imports Python libraries needed to interact with DataRobot, configure document chunking, and manage the creation and deployment of vector databases. These libraries supply the necessary interfaces for connection, configuration, and orchestration of the workflow. ``` import datarobot as dr from datarobot.models.genai.vector_database import VectorDatabase from datarobot.models.genai.vector_database import ChunkingParameters from datarobot.enums import VectorDatabaseEmbeddingModel from datarobot.enums import VectorDatabaseChunkingMethod from datarobot.enums import PredictionEnvironmentPlatform from datarobot.enums import PredictionEnvironmentModelFormats import time import requests ``` ### Connect to DataRobot This section managed the connection to the DataRobot client. Read more about different options for [connecting to DataRobot from the Python client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` # Option 1: Use environment variables (recommended) # The client will automatically use DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN dr.Client() # Option 2: Explicitly provide endpoint and token # endpoint = "https://app.datarobot.com/api/v2" # token = "" # dr.Client(endpoint=endpoint, token=token) ``` ## Create a vector database with DataRobot built-in embeddings This approach uses DataRobot's built-in embedding models to create and deploy a vector database from the uploaded document. ### Get the Use Case Vector databases must be associated with a Use Case in DataRobot. This step uses the Use Case that the notebook is running in (if available), or creates a new one if the notebook is not in a Use Case. Set `USE_CASE_NAME` to define a more specific Use Case name. ``` # Create the Use Case USE_CASE_NAME = "VDB Example Use Case" use_case = dr.UseCase.create(name=USE_CASE_NAME) print(f"Created Use Case: {use_case.name}") ``` ### Upload a dataset This section uploads the documents dataset. The dataset should be a ZIP file containing text files ( `.txt`, `.md`, etc.). Each file is processed and chunked for the vector database. ``` # Get or create the dataset DATASET_NAME = "pirate_resumes.zip" dataset = None # Search for existing dataset by exact name try: all_datasets = dr.Dataset.list(filter_failed=True) for d in all_datasets: if d.name == DATASET_NAME: dataset = d print(f"Found existing dataset: {dataset.name}") break except Exception: pass # Upload if not found if not dataset: dataset_url = "https://s3.amazonaws.com/datarobot_public_datasets/genai/pirate_resumes.zip" dataset = dr.Dataset.create_from_url(dataset_url) print(f"Uploaded new dataset: {dataset.name}") # Add dataset to Use Case if not already added try: use_case_datasets = use_case.get_datasets() dataset_ids = [d.id for d in use_case_datasets] if dataset.id not in dataset_ids: use_case.add(dataset) print(f"Added dataset to Use Case") except Exception: pass # Already added or error adding ``` ### Configure chunking parameters Configure how your documents will be split into chunks. The chunking parameters determine: - Chunk size : Maximum number of characters per chunk. - Chunk overlap : Percentage of overlap between chunks (helps preserve context). - Chunking method : The algorithm used to split text (recursive, fixed, etc.). - Embedding model : The model used to generate embeddings (optional, defaults to Jina). ``` chunking_parameters = ChunkingParameters( embedding_model=VectorDatabaseEmbeddingModel.JINA_EMBEDDING_T_EN_V1, chunking_method=VectorDatabaseChunkingMethod.RECURSIVE, chunk_size=256, chunk_overlap_percentage=25, separators=["\n\n", "\n", " ", ""], ) ``` ### Create the vector database Create the vector database using the dataset and chunking parameters. This process: 1. Splits the documents into chunks. 2. Generates embeddings for each chunk using the specified embedding model. 3. Stores the chunks and embeddings in the vector database. This process typically takes 30-60 seconds depending on the size of the dataset. ``` vdb = VectorDatabase.create( dataset_id=dataset.id, chunking_parameters=chunking_parameters, use_case=use_case, name="My Vector Database" ) ``` Check the status of the vector database until it completes successfully. ``` max_wait_time = 600 check_interval = 5 start_time = time.time() print("Waiting for vector database creation...") while time.time() - start_time < max_wait_time: vdb = VectorDatabase.get(vdb.id) status = vdb.execution_status if status == "COMPLETED": print(f"Vector database created: {vdb.name}") break elif status == "FAILED": error_msg = getattr(vdb, 'error_message', 'Unknown error') raise Exception(f"Vector database creation failed: {error_msg}") else: # Show progress if available percentage = getattr(vdb, 'percentage', None) if percentage is not None: print(f" Status: {status} ({percentage}%)") else: print(f" Status: {status}...") time.sleep(check_interval) else: raise Exception(f"Vector database creation timed out after {max_wait_time} seconds") assert vdb.execution_status == "COMPLETED", f"Vector database creation failed with status: {vdb.execution_status}" ``` ## Deploy the vector database Once the vector database is created, deploy it for production use. There are two main ways to deploy vector databases: - Direct deployment : Deploy the vector database directly to a prediction environment using the Python SDK - Send to Workshop : Register the vector database as a custom model first, then deploy it ### Create a prediction environment First, you need a prediction environment. DataRobot Serverless is typically used for vector database deployments. This section creates a new prediction environment if one doesn't already exist. In addition, this section selects the resource bundle required to run the vector database. ``` PREDICTION_ENVIRONMENT_NAME = "Vector Database Prediction Environment" # Get or create prediction environment prediction_environment = None for env in dr.PredictionEnvironment.list(): if env.name == PREDICTION_ENVIRONMENT_NAME: prediction_environment = env break if prediction_environment is None: prediction_environment = dr.PredictionEnvironment.create( name=PREDICTION_ENVIRONMENT_NAME, platform=PredictionEnvironmentPlatform.DATAROBOT_SERVERLESS, supported_model_formats=[ PredictionEnvironmentModelFormats.DATAROBOT, PredictionEnvironmentModelFormats.CUSTOM_MODEL ], ) print(f"Created prediction environment: {prediction_environment.name}") else: print(f"Using existing prediction environment: {prediction_environment.name}") # Select 3XL resource bundle resource_bundle_id = None try: dr_client = dr.Client() bundles_url = f"{dr_client.endpoint}/mlops/compute/bundles/" headers = {"Authorization": f"Bearer {dr_client.token}"} bundles_response = requests.get(bundles_url, headers=headers, params={"useCases": "customModel"}) if bundles_response.status_code == 200: bundles_data = bundles_response.json() if bundles_data.get("data"): bundles = bundles_data["data"] # Look for 3XL bundle bundle_3xl = next((b for b in bundles if "3XL" in b.get("name", "").upper()), None) if bundle_3xl: resource_bundle_id = bundle_3xl["id"] print(f"Selected 3XL bundle: {bundle_3xl['name']}") else: # Fallback to largest available sorted_bundles = sorted(bundles, key=lambda b: b.get("memoryBytes", 0), reverse=True) if sorted_bundles: resource_bundle_id = sorted_bundles[0]["id"] print(f"Warning: 3XL bundle not found. Using largest available: {sorted_bundles[0]['name']}") else: print("Using memory settings (no resource bundles available)") else: print("Using memory settings (resource bundles not enabled)") except (ImportError, KeyError) as e: print(f"Using memory settings (error checking bundles): {e}") except requests.RequestException as e: print(f"Using memory settings (network error checking bundles): {e}") ``` ### Send vector database to workshop Before deploying, send the vector database to the custom model workshop. This creates a custom model version that can be registered and deployed. The code uses the resource configuration determined when the prediction environment was created or checked (resource bundles if available, otherwise memory settings). ``` assert vdb.execution_status == "COMPLETED", f"Vector database must be completed. Current status: {vdb.execution_status}" # Send to workshop with 3XL bundle or memory settings if resource_bundle_id: custom_model_version = vdb.send_to_custom_model_workshop( resource_bundle_id=resource_bundle_id, replicas=1, network_egress_policy=dr.NETWORK_EGRESS_POLICY.PUBLIC, ) else: custom_model_version = vdb.send_to_custom_model_workshop( maximum_memory=4096*1024*1024, replicas=1, network_egress_policy=dr.NETWORK_EGRESS_POLICY.PUBLIC, ) print(f"Custom model version created: {custom_model_version}") ``` ### Register the model Next, register the custom model version. If a registered model with the same name already exists, this step adds a new version to the existing model instead of creating a duplicate. ``` REGISTERED_MODEL_NAME = f"Vector Database - {vdb.name}" # Register model (adds new version if model already exists) existing_models = [m for m in dr.RegisteredModel.list() if m.name == REGISTERED_MODEL_NAME] if existing_models: registered_model_version = dr.RegisteredModelVersion.create_for_custom_model_version( custom_model_version_id=custom_model_version.id, registered_model_id=existing_models[0].id, ) print(f"Added new version to existing registered model: {REGISTERED_MODEL_NAME}") else: registered_model_version = dr.RegisteredModelVersion.create_for_custom_model_version( custom_model_version_id=custom_model_version.id, registered_model_name=REGISTERED_MODEL_NAME, ) print(f"Created new registered model: {REGISTERED_MODEL_NAME}") ``` ### Wait for model build to complete Wait for the registered model version to finish building before deploying. ``` registered_model = dr.RegisteredModel.get(registered_model_version.registered_model_id) max_wait_time = 600 check_interval = 10 start_time = time.time() print("Waiting for model build to complete...") while time.time() - start_time < max_wait_time: version = registered_model.get_version(registered_model_version.id) build_status = getattr(version, 'build_status', None) or getattr(version, 'buildStatus', None) if build_status in ('READY', 'complete', 'COMPLETE'): print(f"Model build completed (status: {build_status})") break elif build_status in ('FAILED', 'ERROR', 'error'): raise Exception(f"Model build failed. Status: {build_status}") else: print(f" Build status: {build_status}...") time.sleep(check_interval) else: version = registered_model.get_version(registered_model_version.id) build_status = getattr(version, 'build_status', None) or getattr(version, 'buildStatus', None) raise Exception(f"Model build timed out. Current status: {build_status}") # Verify ready status version = registered_model.get_version(registered_model_version.id) final_status = getattr(version, 'build_status', None) or getattr(version, 'buildStatus', None) if final_status not in ('READY', 'complete', 'COMPLETE'): raise Exception(f"Model not ready for deployment. Status: {final_status}") ``` ### Deploy the registered model Deploy the registered model version to the prediction environment created earlier. The model must be in READY status before deployment process can resolve successfully. ``` deployment = dr.Deployment.create_from_registered_model_version( registered_model_version.id, label=f"Vector Database Deployment - {vdb.name}", description="Vector database deployment for RAG applications", prediction_environment_id=prediction_environment.id, max_wait=600, ) print(f"Deployment created: {deployment.id}") ``` ### Use vector databases in LLM Playgrounds Vector databases created through the Python SDK are automatically available in LLM Playgrounds for use in RAG workflows. See the [genai-e2e.ipynb](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/genai-e2e.ipynb) notebook for examples of using vector databases with LLMs. For detailed deployment instructions and UI options, see the [Register and deploy vector databases](https://docs.datarobot.com/en/docs/gen-ai/vector-database/vector-dbs-register-deploy.html) documentation. ## List and manage vector databases Using the command below list all vector databases associated with a Use Case and manage them programmatically. ``` # List all vector databases in a Use Case vdbs = VectorDatabase.list(use_case=use_case) print(f"Found {len(vdbs)} vector database(s) in Use Case '{use_case.name}'") ``` ### Next steps - Use your vector database in an LLM Playground . - Learn about creating vector databases with custom embedding models (BYO embeddings). - Explore external vector databases like ChromaDB. --- # Create vector databases from BYO embeddings URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/create-vdb-byo-embedding.html The following notebook outlines how you can build and validate a vector database from a bring-your-own (BYO) embedding and register the vector database to the DataRobot platform using the Python client. This notebook is designed for use with DataRobot notebooks; DataRobot recommends downloading this notebook and uploading it for use in the platform. ## Setup The following steps outline the necessary configuration for integrating vector databases with the DataRobot platform. 1. This workflow uses the following feature flags. Contact your DataRobot representative or administrator for information on enabling these features. 2. Use a codespace, not a DataRobot Notebook, to ensure this notebook has access to a filesystem. 3. Set the notebook session timeout to 180 minutes. 4. Restart the notebook container using at least a "Medium" (16GB RAM) instance. 5. Optionally, upload your documents archive to the notebook filesystem. ### Install requirements Import the following libraries and modules to interface with the DataRobot platform, prediction environments, and vector database functionality. ``` import datarobot as dr from datarobot.enums import PredictionEnvironmentPlatform from datarobot.enums import PredictionEnvironmentModelFormats from datarobot.models.genai.custom_model_embedding_validation import CustomModelEmbeddingValidation from datarobot.models.genai.vector_database import VectorDatabase from datarobot.models.genai.vector_database import ChunkingParameters ``` ### Connect to DataRobot Provide a DataRobot `endpoint` and `token` to connect to DataRobot through the DataRobot Python client. Read more about options for [connecting to DataRobot from the Python client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` # Option 1: Use environment variables (recommended) # The client will automatically use DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN dr.Client() # Option 2: Explicitly provide endpoint and token # endpoint = "https://app.datarobot.com/api/v2" # token = "" # dr.Client(endpoint=endpoint, token=token) ``` ## Select an environment Using `dr.ExecutionEnvironment.list()`, iterate through the environments available to your organization, selecting the environment named `[GenAI] Python 3.12 with Moderations` to be the base environment of the vector database and assigning it to `base_environment`. ``` execution_environments = dr.ExecutionEnvironment.list() base_environment = None environment_versions = None for execution_environment in execution_environments: # print(execution_environment) if execution_environment.name == "[GenAI] Python 3.12 with Moderations": base_environment = execution_environment environment_versions = dr.ExecutionEnvironmentVersion.list( execution_environment.id ) break environment_version = environment_versions[0] print(base_environment) print(environment_version) ``` ## Create a custom embedding model Using `dr.CustomInferenceModel.list()` search the available custom models for `all-MiniLM-L6-v2-embedding-model`. If the custom model doesn't exist, create it as `custom_model` using `dr.CustomInferenceModel.create()`. If the custom model does exist, assign it to `custom_model`. ``` CUSTOM_MODEL_NAME = "all-MiniLM-L6-v2-embedding-model_20260218-01" if CUSTOM_MODEL_NAME not in [c.name for c in dr.CustomInferenceModel.list()]: # Create a new custom model print("Creating new custom model") custom_model = dr.CustomInferenceModel.create( name=CUSTOM_MODEL_NAME, target_type=dr.TARGET_TYPE.UNSTRUCTURED, is_training_data_for_versions_permanently_enabled=True ) else: print("Custom Model Exists") custom_model = [c for c in dr.CustomInferenceModel.list() if c.name == CUSTOM_MODEL_NAME].pop() ``` ### Write custom embedding model code Create a directory called `custom_embedding_model` to write custom embedding model code into. Write custom embedding model code into the `custom.py` file, creating an unstructured model from `all-MiniLM-L6-v2`. ``` import os os.mkdir('custom_embedding_model') ``` ``` %%writefile ./custom_embedding_model/custom.py import os os.environ["HF_HOME"] = "/tmp/hf_cache" os.environ["TRANSFORMERS_CACHE"] = "/tmp/hf_cache" os.environ["SENTENCE_TRANSFORMERS_HOME"] = "/tmp/hf_cache" from sentence_transformers import SentenceTransformer def load_model(input_dir): return SentenceTransformer("all-MiniLM-L6-v2") def score_unstructured(model, data, query, **kwargs): import json data_dict = json.loads(data) outputs = model.encode(data_dict["input"]) return json.dumps( { "result": outputs.tolist(), "device": str(model._target_device) } ) ``` ### Write a requirements file Write the requirements for the custom embedding model into the `requirements.txt` file, ensuring the custom model environment includes the embedding model's dependencies. ``` %%writefile ./custom_embedding_model/requirements.txt sentence-transformers==3.0.0 ``` ### Create a custom model version Using `dr.CustomModelVersion.create_clean`, create a custom model version with the `custom_model`, `base_environment`, and `files` defined in previous steps. In addition, enable public network access using `dr.NETWORK_EGRESS_POLICY.PUBLIC`. ``` # Create a new custom model version in DataRobot print("Upload new model version to DataRobot") model_version = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=base_environment.id, files=[ ("./custom_embedding_model/custom.py", "custom.py"), ("./custom_embedding_model/requirements.txt", "requirements.txt"), ], network_egress_policy=dr.NETWORK_EGRESS_POLICY.PUBLIC, ) ``` ### Build a custom model environment Using `dr.CustomModelVersionDependencyBuild`, build a custom model environment with the required dependencies installed. ``` # Build the custom model environment to ensure dependencies from `requirements.txt` are installed. build_info = dr.CustomModelVersionDependencyBuild.start_build( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, max_wait=60*10, # Set a long timeout ) ``` ## Register the custom model Using `dr.RegisteredModel.list()`, search the available custom models for `all-MiniLM-L6-v2-embedding-model` (assigned to `CUSTOM_MODEL_NAME`). If the registered model doesn't exist, create a `registered_model_version` using `dr.RegisteredModelVersion.create_for_custom_model_version`. If the registered model does exist, assign it to `registered_model` and create a `registered_model_version` using `dr.RegisteredModelVersion.create_for_custom_model_version`. ``` if CUSTOM_MODEL_NAME not in [m.name for m in dr.RegisteredModel.list()]: print("Creating New Registered Model") registered_model_version = dr.RegisteredModelVersion.create_for_custom_model_version( model_version.id, name=CUSTOM_MODEL_NAME, registered_model_name=CUSTOM_MODEL_NAME ) else: print("Using Existing Model") registered_model = [m for m in dr.RegisteredModel.list() if m.name == CUSTOM_MODEL_NAME].pop() registered_model_version = dr.RegisteredModelVersion.create_for_custom_model_version( model_version.id, name=CUSTOM_MODEL_NAME, registered_model_id=registered_model.id ) ``` ## Create a prediction environment for embedding models Using `dr.PredictionEnvironment.list()`, search the available prediction environments for `Prediction environment for BYO embeddings models`. If the prediction environment doesn't exist, create a DataRobot Serverless `prediction_environment` using `dr.PredictionEnvironment.create`. If the prediction environment does exist, assign it to `prediction_environment`. ``` PREDICTION_ENVIRONMENT_NAME = "Prediction environment for BYO embeddings models" prediction_environment = None for _prediction_environment in dr.PredictionEnvironment.list(): if _prediction_environment.name == PREDICTION_ENVIRONMENT_NAME: prediction_environment = _prediction_environment if prediction_environment is None: prediction_environment = dr.PredictionEnvironment.create( name=PREDICTION_ENVIRONMENT_NAME, platform=PredictionEnvironmentPlatform.DATAROBOT_SERVERLESS, supported_model_formats=[ PredictionEnvironmentModelFormats.DATAROBOT, PredictionEnvironmentModelFormats.CUSTOM_MODEL ], ) ``` ## Deploy the custom embedding model Using `dr.Deployment.list()`, search the available deployments for `Deployment for all-MiniLM-L6-v2`. If the prediction environment doesn't exist, create a `deployment`, deploying the `registered_model_version` created in a previous section with `dr.Deployment.create_from_registered_model_version`. If the deployment does exist, assign it to `deployment`. ``` MODEL_DEPLOYMENT_NAME = "Deployment for all-MiniLM-L6-v2" if MODEL_DEPLOYMENT_NAME not in [d.label for d in dr.Deployment.list()]: deployment = dr.Deployment.create_from_registered_model_version( registered_model_version.id, label=MODEL_DEPLOYMENT_NAME, max_wait=1000, prediction_environment_id=prediction_environment.id ) else: deployment = [d for d in dr.Deployment.list() if d.label == MODEL_DEPLOYMENT_NAME][0] ``` ## Create a Use Case for BYO embeddings Using `dr.UseCase.create`, create the Use Case to use the vector database with and assign it to `use_case`. When working with your own vector database, if [PostgreSQL](https://docs.datarobot.com/en/docs/gen-ai/vector-database/vector-dbs.html#connect-to-postgresql) is the connection method, the output dimension must be less than 2000. ``` use_case = dr.UseCase.create(name="For BYO embeddings") ``` ### Upload a dataset to the Use Case Using `dr.Dataset.create_from_url`, upload the example dataset for the vector database and assign it to `dataset`. ``` # this can be updated with any public URL that is pointing to a .zip file # in the expected format dataset_url = "https://s3.amazonaws.com/datarobot_public_datasets/genai/pirate_resumes.zip" # We will use a vector database with our GenAI models. Let's upload a dataset with our documents. # If you wish to use a local file as dataset, change this to # `dataset = dr.Dataset.create_from_file(local_file_path)` dataset = dr.Dataset.create_from_url(dataset_url) ``` Then, add the dataset to the `use_case` created in the previous section. ``` # Attach dataset to use case. use_case.add(dataset) ``` ## Validate and create the custom embedding model The `CustomModelVectorDatabaseValidation.create` function executes the validation of the vector database, setting the required settings and associating the custom embedding model with the `use_case` and `deployment` created earlier in this notebook. This step stores the validation ID in `custom_model_embedding_validation`. ``` # Create BYO embeddings validation using prepared deployment custom_model_embedding_validation = CustomModelEmbeddingValidation.create( prompt_column_name="input", target_column_name="result", deployment_id=deployment.id, use_case = use_case, name="BYO embeddings", wait_for_completion=True, prediction_timeout=300, ) ``` ### Set chunking parameters and create a vector database After validation completes, set the `ChunkingParameter()` and use `VectorDatabase.create()` to integrate the vector database. This step uses the `custom_model_embedding_validation`, `dataset`, and `use_case` defined in previous sections. ``` # Use created validation to set up chunking parameters chunking_parameters = ChunkingParameters( embedding_validation=custom_model_embedding_validation, chunking_method="recursive", chunk_size=256, chunk_overlap_percentage=50, separators=["\n\n", "\n", " ", ""], embedding_model=None, ) vdb = VectorDatabase.create( dataset_id=dataset.id, chunking_parameters=chunking_parameters, use_case=use_case ) ``` ``` import time max_wait_time = 600 check_interval = 5 start_time = time.time() print("Waiting for vector database creation...") while time.time() - start_time < max_wait_time: vdb = VectorDatabase.get(vdb.id) status = vdb.execution_status if status == "COMPLETED": print(f"Vector database created: {vdb.name}") break elif status == "FAILED": error_msg = getattr(vdb, 'error_message', 'Unknown error') raise Exception(f"Vector database creation failed: {error_msg}") else: # Show progress if available percentage = getattr(vdb, 'percentage', None) if percentage is not None: print(f" Status: {status} ({percentage}%)") else: print(f" Status: {status}...") time.sleep(check_interval) else: raise Exception(f"Vector database creation timed out after {max_wait_time} seconds") assert vdb.execution_status == "COMPLETED", f"Vector database creation failed with status: {vdb.execution_status}" ``` --- # Use the DataRobot LLM gateway URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/dr-llm-gateway.html The DataRobot LLM gateway is a service that unifies and simplifies LLM access across DataRobot. It provides a DataRobot API endpoint to interface with LLMs hosted by external LLM providers. To request LLM responses from the DataRobot LLM gateway, you can use any API client that supports OpenAI-compatible chat completion API, for example, the [OpenAI Python API library](https://github.com/openai/openai-python). Note: Provisioning LLMs using the LLM gateway is available for DataRobot-managed (Cloud) instances and single-tenant SaaS (STS) self-managed instances with supported pricing plans; it is not available for on-premise installations. Contact your DataRobot representative for details. The gateway itself, as a service for calling LLMs, is available on all platforms. ## Setup The DataRobot LLM gateway is a premium feature; contact your DataRobot representative or administrator for information on enabling it. To use the DataRobot LLM gateway with the OpenAI Python API library, first, make sure the OpenAI client package is installed and imported. You must also import the DataRobot Python client. ``` %pip install openai ``` ``` import datarobot as dr from openai import OpenAI from datarobot.models.genai.llm_gateway_catalog import LLMGatewayCatalog from pprint import pprint ``` ## Connect to the LLM gateway Next, initialize the OpenAI client. The base URL is the DataRobot LLM gateway endpoint. The example below assembles this URL by combining your DataRobot API endpoint (retrieved from [dr_client](https://docs.datarobot.com/en/docs/api/reference/sdk/client-setup.html)) and `/genai/llmgw`. Usually this results in `https://app.datarobot.com/api/v2/genai/llmgw`, `https://app.eu.datarobot.com/api/v2/genai/llmgw`, `https://app.jp.datarobot.com/api/v2/genai/llmgw`, or your organization's DataRobot API endpoint URL with `/genai/llmgw` appended to it. The API key is your [DataRobot API key](https://docs.datarobot.com/en/docs/get-started/acct-mgmt/acct-settings/api-key-mgmt.html). ``` dr_client = dr.Client() DR_API_TOKEN = dr_client.token LLM_GATEWAY_BASE_URL = f"{dr_client.endpoint}/genai/llmgw" client = OpenAI( base_url=LLM_GATEWAY_BASE_URL, api_key=DR_API_TOKEN, ) print(f"Your LLM gateway URL is {LLM_GATEWAY_BASE_URL}.") ``` ## Select models and make requests In your code, you can specify any supported provider LLM and set up the message to send to the LLM as a prompt. An optional argument is `client_id`, where you can specify the caller service to use for metering: `genai-playground`, `custom-model`, or `moderations`. This example calls the LLM gateway catalog endpoint to get one of the latest supported LLMs from each provider. ``` # Get list of catalog model names using the SDK (active, non-deprecated models by default) catalog_entries = LLMGatewayCatalog.get_available_models() first_model_by_provider = {} # Iterate through the available models to select the first supported LLM from each provider for model_string in catalog_entries: # Extract the provider name (the part before the first '/') provider_name = model_string.split('/')[0] # If the provider isn't already recorded, store the full model string if provider_name not in first_model_by_provider: first_model_by_provider[provider_name] = model_string # Store the list of models models = list(first_model_by_provider.values()) print(f"Selected models from {len(first_model_by_provider)} providers:") pprint(first_model_by_provider, sort_dicts=False) ``` After you define the `client`, `models` and `message`, you can make chat completion requests to the LLM gateway. The authentication uses DataRobot-provided credentials. ``` from IPython.display import display, Markdown # Store a message to send to the LLM message = [{"role": "user", "content": "Hello! What is your name and who made you?"}] for model in models: response = client.chat.completions.create( model=model, messages=message, ) response_text = response.choices[0].message.content output_as_markdown = f""" **{model}:** {response_text} --- """ display(Markdown(output_as_markdown)); ``` To further configure your chat completion request when making direct calls to an LLM gateway, specify LLM parameter settings like `temperature`, `max_completion_tokens`, and more. These parameters are also supported for custom models. For more information on the available parameters, see the [OpenAI chat completion documentation](https://platform.openai.com/docs/api-reference/chat/create). ``` model2 = models[0] if len(models) > 0 else "openai/gpt-4o" # Use first model or fallback message2 = [{"role": "user", "content": "Hello! What is your name and who made you? How do you feel about Agentic AI"}] extra_body2 = { "temperature": 0.8, "max_completion_tokens": 2000, } response2 = client.chat.completions.create( model=model2, messages=message2, extra_body=extra_body2, ) response_text2 = response2.choices[0].message.content output_as_markdown2 = f""" **{model2}:** {response_text2} --- """ display(Markdown(output_as_markdown2)); ``` ## Identify supported LLMs To provide a list of LLMs supported by the LLM gateway, this example uses the LLM gateway catalog SDK to get the available models. ``` # Get all available models using the SDK convenience method supported_llms = LLMGatewayCatalog.get_available_models() print(f"Found {len(supported_llms)} available models:") pprint(supported_llms[:10]) # Show first 10 models if len(supported_llms) > 10: print(f"... and {len(supported_llms) - 10} more models") ``` If you try to use an unsupported LLM, the LLM gateway returns an error message, relaying that the specified LLM is not in the LLM catalog. ``` # Verify model availability unsupported_model = "unsupported-provider/random-llm" try: # Check if the model is available before making the request model_entry = LLMGatewayCatalog.verify_model_availability(unsupported_model) print(f"Model {unsupported_model} is available: {model_entry.name}") except ValueError as e: print(f"Model {unsupported_model} is not available: {e}") # Alternative: still show the original error handling for comparison messages3 = [ {"role": "user", "content": "Hello!"} ] try: response = client.chat.completions.create( model=unsupported_model, messages=messages3, ) response.choices[0].message.content except Exception as e: print(f"Direct API call error: {str(e)}") ``` You can also verify if a specific model is available before attempting to use it. This is useful for error handling and validation: ``` # Test different model IDs test_models = [ "azure/gpt-4o-2024-11-20", # Example Azure model "openai/gpt-4o", # Example OpenAI model "non-existent-model" # This should fail ] for model_id in test_models: try: model_entry = LLMGatewayCatalog.verify_model_availability(model_id) print(f"✓ {model_id} is available:") pprint({ "name": model_entry.name, "provider": model_entry.provider, "context_size": f"{model_entry.context_size:,} tokens", "active": model_entry.is_active, "deprecated": model_entry.is_deprecated }, sort_dicts=False) except ValueError as e: print(f"✗ {model_id} is not available: {e}") print() ``` ## Advanced filtering The LLM gateway Catalog SDK provides advanced filtering capabilities to help you find the right models for your needs: ``` # Get all models including deprecated ones all_entries = LLMGatewayCatalog.list( only_active=False, only_non_deprecated=False, limit=20 ) active_count = sum(1 for entry in all_entries if entry.is_active) deprecated_count = sum(1 for entry in all_entries if entry.is_deprecated) print(f"Found {len(all_entries)} total entries:") print(f" - Active: {active_count}") print(f" - Deprecated: {deprecated_count}") # Show deprecated models with replacement info deprecated_models = [e for e in all_entries if e.is_deprecated] if deprecated_models: print("\nDeprecated models with replacements (first 3 models):") for entry in deprecated_models[:3]: deprecated_info = { "model": entry.model, "name": entry.name, "provider": entry.provider } if entry.retirement_date: deprecated_info["retirement_date"] = entry.retirement_date if entry.suggested_replacement: deprecated_info["suggested_replacement"] = entry.suggested_replacement pprint(deprecated_info, sort_dicts=False) print() ``` ``` # Get detailed catalog entries with rich information catalog_entries = LLMGatewayCatalog.list(limit=5) print("Detailed catalog entries:") for entry in catalog_entries: entry_info = { "name": entry.name, "model": entry.model, "provider": entry.provider, "context_size": f"{entry.context_size:,} tokens", "active": entry.is_active, "deprecated": entry.is_deprecated } pprint(entry_info, sort_dicts=False) print() ``` --- # Create external LLMs with code URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/ext-llm.html The following, designed for use with DataRobot Notebooks, outlines how you can build and validate an external LLM using the DataRobot Python client. DataRobot recommends downloading this notebook and uploading it for use in the platform. Note: For self-managed users, when code samples reference `app.datarobot.com`, change them to the appropriate URL for your instance. ## Setup The following steps outline the configuration necessary for integrating an external LLM with the DataRobot platform. 1. This workflow requires that the following feature flags are enabled. Contact your DataRobot representative or administrator for information on enabling these features. 2. Create a new credential in theDataRobot Credentials Management tool: 3. Use a codespace, not a DataRobot Notebook, to ensure this notebook has access to a filesystem. 4. Set the notebook session timeout to 180 minutes. 5. Restart the notebook container using at least a "Medium" (16GB RAM) instance. ## Install libraries Install the following libraries: ``` !pip install openai datarobot-drum datarobot-predict ``` ``` import datarobot as dr from datarobot.models.genai.custom_model_llm_validation import CustomModelLLMValidation ``` ## Connect to DataRobot Read more about different options for [connecting to DataRobot from the Python client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` dr.Client() token = dr.Client().token endpoint = f"{dr.Client().endpoint}" dr.Client(endpoint=endpoint, token=token) ``` ## Create a directory for custom code Create a directory called `custom_model` that will hold your OpenAI wrapper code. ``` !mkdir custom_model ``` ## Define environment variables In the cell below, define each environment variable required to run this notebook. These environment variables are added as runtime parameters to the custom model in the workshop. ``` %%writefile .env # Define required environment variables for the codespace environment OPENAI_API_KEY= # For codespace testing, the API key for the Azure OpenAI API OPENAI_API_VERSION= OPENAI_API_BASE= OPENAI_DEPLOYMENT_NAME= DATAROBOT_CREDENTIAL_OPENAI= # For the deployed model, the ObjectId of the Azure OpenAI API Token credential ``` The value for `DATAROBOT_CREDENTIAL_OPENAI` environment variable must be the `ObjectId` of the Azure OpenAI API Token credential. You can find the ID of the credential using the DataRobot Python API client's credentials list method. ``` dr.Credential.list() ``` ## Define hooks The following cell defines the methods used to deploy a text generation custom model. These include loading the custom model and using the model for scoring. ``` import os import pandas as pd from typing import Any, Iterator, Union, Dict, List from openai import AzureOpenAI from openai.types.chat import ChatCompletion, ChatCompletionChunk # Try to load dotenv for codespace testing try: from dotenv import load_dotenv load_dotenv() except ImportError: pass # Don't load the .env file CompletionCreateParams = Dict[str, Any] def get_config(): # Get configuration from runtime parameters or environment variables try: # Try DataRobot runtime parameters first from datarobot_drum import RuntimeParameters return { "api_key": RuntimeParameters.get("OPENAI_API_KEY")["apiToken"], "api_base": RuntimeParameters.get("OPENAI_API_BASE"), "api_version": RuntimeParameters.get("OPENAI_API_VERSION"), "deployment_name": RuntimeParameters.get("OPENAI_DEPLOYMENT_NAME") } except Exception: # Fallback to environment variables for codespace testing return { "api_key": os.environ.get("OPENAI_API_KEY", ""), "api_base": os.environ.get("OPENAI_API_BASE", ""), "api_version": os.environ.get("OPENAI_API_VERSION", ""), "deployment_name": os.environ.get("OPENAI_DEPLOYMENT_NAME", "") } # Implement the load_model hook. def load_model(*args, **kwargs): config = get_config() return AzureOpenAI( api_key=config["api_key"], azure_endpoint=config["api_base"], api_version=config["api_version"] ) # Load the Azure OpenAI client def load_client(*args, **kwargs): return load_model(*args, **kwargs) # Get supported LLM models def get_supported_llm_models(client: AzureOpenAI) -> List[Dict[str, Any]]: azure_models = client.models.list() model_ids = [m.id for m in azure_models] return model_ids if model_ids else ["datarobot-deployed-llm"] # On-demand chat requests def chat( completion_create_params: CompletionCreateParams, client: AzureOpenAI ) -> Union[ChatCompletion, Iterator[ChatCompletionChunk], Dict]: try: if completion_create_params.get("model") == "datarobot-deployed-llm": config = get_config() completion_create_params["model"] = config["deployment_name"] return client.chat.completions.create(**completion_create_params) except Exception as e: return { "error": f"{e.__class__.__name__}: {str(e)}" } # Batch chat requests PROMPT_COLUMN_NAME = "promptText" COMPLETION_COLUMN_NAME = "resultText" ERROR_COLUMN_NAME = "error" def score(data, client, **kwargs): prompts = data["promptText"].tolist() responses = [] errors = [] for prompt in prompts: try: # Get model config config = get_config() # Attempt to get a completion from the client response = client.chat.completions.create( model=config["deployment_name"], messages=[{"role": "user", "content": f"{prompt}"},], max_tokens=20, temperature=0 ) # On success, append the content and a null error responses.append(response.choices[0].message.content or "") errors.append("") except Exception as e: # On failure, format the error message error = f"{e.__class__.__name__}: {str(e)}" responses.append("") errors.append(error) return pd.DataFrame({ PROMPT_COLUMN_NAME: prompts, COMPLETION_COLUMN_NAME: responses, ERROR_COLUMN_NAME: errors }) ``` ## Test hooks locally Before proceeding with the deployment, use the cell below to test that the custom model hooks function correctly. ``` # Provide test data test_data = pd.DataFrame({PROMPT_COLUMN_NAME: ["What is a large language model (LLM)?"]}) ``` ``` # Test get_supported_llm_models() models = get_supported_llm_models(load_client()) print(f"Available models: {models}") ``` ``` # Test chat() chat( { "model": "datarobot-deployed-llm", "messages": [{"role": "user", "content": "What is a large language model (LLM)?"}], "max_tokens": 20, "temperature": 0, }, client=load_client(), ) ``` ``` # Test score() score(test_data, client=load_client()) ``` ## Save the custom model code Next, save the hooks above as `custom_model/custom.py`. This python file will be executed by DataRobot using your credentials. The following is a copy of the cell where you previously defined the hooks. ``` %%writefile custom_model/custom.py import os import pandas as pd from typing import Any, Iterator, Union, Dict, List from openai import AzureOpenAI from openai.types.chat import ChatCompletion, ChatCompletionChunk # Try to load dotenv for codespace testing try: from dotenv import load_dotenv load_dotenv() except ImportError: pass # Don't load the .env file CompletionCreateParams = Dict[str, Any] def get_config(): # Get configuration from runtime parameters or environment variables try: # Try DataRobot runtime parameters from datarobot_drum import RuntimeParameters return { "api_key": RuntimeParameters.get("OPENAI_API_KEY")["apiToken"], "api_base": RuntimeParameters.get("OPENAI_API_BASE"), "api_version": RuntimeParameters.get("OPENAI_API_VERSION"), "deployment_name": RuntimeParameters.get("OPENAI_DEPLOYMENT_NAME") } except Exception: # Fallback to environment variables for codespace testing return { "api_key": os.environ.get("OPENAI_API_KEY", ""), "api_base": os.environ.get("OPENAI_API_BASE", ""), "api_version": os.environ.get("OPENAI_API_VERSION", ""), "deployment_name": os.environ.get("OPENAI_DEPLOYMENT_NAME", "") } # Implement the load_model hook. def load_model(*args, **kwargs): config = get_config() return AzureOpenAI( api_key=config["api_key"], azure_endpoint=config["api_base"], api_version=config["api_version"] ) # Load the Azure OpenAI client. def load_client(*args, **kwargs): return load_model(*args, **kwargs) # Get supported LLM models def get_supported_llm_models(client: AzureOpenAI) -> List[Dict[str, Any]]: azure_models = client.models.list() model_ids = [m.id for m in azure_models] return model_ids if model_ids else ["datarobot-deployed-llm"] # On-demand chat requests def chat( completion_create_params: CompletionCreateParams, client: AzureOpenAI ) -> Union[ChatCompletion, Iterator[ChatCompletionChunk], Dict]: try: if completion_create_params.get("model") == "datarobot-deployed-llm": config = get_config() completion_create_params["model"] = config["deployment_name"] return client.chat.completions.create(**completion_create_params) except Exception as e: return { "error": f"{e.__class__.__name__}: {str(e)}" } # Batch chat requests PROMPT_COLUMN_NAME = "promptText" COMPLETION_COLUMN_NAME = "resultText" ERROR_COLUMN_NAME = "error" def score(data, client, **kwargs): prompts = data["promptText"].tolist() responses = [] errors = [] for prompt in prompts: try: # Get model config config = get_config() # Attempt to get a completion from the client response = client.chat.completions.create( model=config["deployment_name"], messages=[{"role": "user", "content": f"{prompt}"},], max_tokens=20, temperature=0 ) # On success, append the content and a null error responses.append(response.choices[0].message.content or "") errors.append("") except Exception as e: # On failure, format the error message error = f"{e.__class__.__name__}: {str(e)}" responses.append("") errors.append(error) return pd.DataFrame({ PROMPT_COLUMN_NAME: prompts, COMPLETION_COLUMN_NAME: responses, ERROR_COLUMN_NAME: errors }) ``` Save requirements and metadata files to help describe the model's environment and usage. ``` %%writefile custom_model/requirements.txt openai datarobot-drum pandas python-dotenv ``` ``` %%writefile custom_model/model-metadata.yaml --- name: OpenAI gpt-4o type: inference targetType: textgeneration runtimeParameterDefinitions: - fieldName: OPENAI_API_KEY type: credential credentialType: api_token description: OpenAI API key stored in DataRobot allowEmpty: false - fieldName: OPENAI_API_VERSION type: string description: OpenAI API version string allowEmpty: false - fieldName: OPENAI_API_BASE type: string description: OpenAI API base URL string allowEmpty: false - fieldName: OPENAI_DEPLOYMENT_NAME type: string description: OpenAI API deployment ID string allowEmpty: false ``` ## Test the code locally The DataRobot `DRUM` library allows you to test the code as if DataRobot were running it via a simple CLI. To do this, supply a test file and then run it. ``` # Create the test file test_data.to_csv("custom_model/test_data.csv", index=False) ``` ``` os.putenv("TARGET_NAME", COMPLETION_COLUMN_NAME) !drum score --code-dir custom_model/ --target-type textgeneration --input custom_model/test_data.csv ``` ## Create a custom model in DataRobot The code below performs a few steps to register your code with DataRobot: - Creates a custom model to contain the versioned code. - Creates a custom model version with the code in the custom_model folder and adds the required Runtime Parameters. - Builds the environment to run the model by installing the requirements.txt file. - Tests the entire setup. ``` # List all existing base environments execution_environments = dr.ExecutionEnvironment.list() execution_environments BASE_ENVIRONMENT = None for execution_environment in execution_environments: if execution_environment.name == "[GenAI] Python 3.12 with Moderations": BASE_ENVIRONMENT = execution_environment environment_versions = dr.ExecutionEnvironmentVersion.list( execution_environment.id ) break if BASE_ENVIRONMENT is None: raise ValueError( "Required execution environment '[GenAI] Python 3.12 with Moderations' not found. Please check your DataRobot instance." ) BASE_ENVIRONMENT_VERSION = environment_versions[0] print(BASE_ENVIRONMENT) print(BASE_ENVIRONMENT_VERSION) print(BASE_ENVIRONMENT.id) ``` ``` CUSTOM_MODEL_NAME = "External LLM OpenAI Wrapper Model" if CUSTOM_MODEL_NAME not in [c.name for c in dr.CustomInferenceModel.list()]: # Create a new custom model print("Creating a new custom model") custom_model = dr.CustomInferenceModel.create( name=CUSTOM_MODEL_NAME, target_type=dr.TARGET_TYPE.TEXT_GENERATION, target_name=COMPLETION_COLUMN_NAME, description="Wrapper for OpenAI completion", language="Python", is_training_data_for_versions_permanently_enabled=True, ) else: print("Custom model exists") custom_model = [ c for c in dr.CustomInferenceModel.list() if c.name == CUSTOM_MODEL_NAME ].pop() ``` ``` # Create a new custom model version in DataRobot with required runtime parameters print("Upload new version of model to DataRobot") model_version = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=BASE_ENVIRONMENT.id, files=[ "./custom_model/custom.py", "./custom_model/requirements.txt", "./custom_model/model-metadata.yaml", ], network_egress_policy=dr.NETWORK_EGRESS_POLICY.PUBLIC, runtime_parameter_values=[ dr.models.custom_model_version.RuntimeParameterValue(field_name="OPENAI_API_KEY", type="credential", value=os.environ.get("DATAROBOT_CREDENTIAL_OPENAI", "")), dr.models.custom_model_version.RuntimeParameterValue(field_name="OPENAI_API_VERSION", type="string", value=os.environ.get("OPENAI_API_VERSION", "")), dr.models.custom_model_version.RuntimeParameterValue(field_name="OPENAI_API_BASE", type="string", value=os.environ.get("OPENAI_API_BASE", "")), dr.models.custom_model_version.RuntimeParameterValue(field_name="OPENAI_DEPLOYMENT_NAME", type="string", value=os.environ.get("OPENAI_DEPLOYMENT_NAME", "")), ] ) ``` ``` try: build_info = dr.CustomModelVersionDependencyBuild.start_build( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, max_wait=3600, ) print("Finished new dependency build") except dr.errors.ClientError as e: if "already has a dependency image" in str(e): print("Dependency build already exists, skipping build step") build_info = None else: raise e ``` ## Test the custom model in DataRobot Next, use the environment to run the model with prediction test data to verify that the custom model is functional before deployment. To do this, upload the inference dataset for testing predictions. ``` pred_test_dataset = dr.Dataset.create_from_in_memory_data(test_data) pred_test_dataset.modify(name="LLM Test Data") pred_test_dataset.update() ``` After uploading the inference dataset, you can test the custom model. ``` # Test a new version in DataRobot print("Run test of new version in DataRobot") custom_model_test = dr.CustomModelTest.create( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, dataset_id=pred_test_dataset.id, max_wait=3600, # 1 hour timeout ) custom_model_test.overall_status HOST = "https://app.datarobot.com" for name, test in custom_model_test.detailed_status.items(): print("Test: {}".format(name)) print("Status: {}".format(test["status"])) print("Message: {}".format(test["message"])) print( "Finished testing: " + HOST + "model-registry/custom-models/" + custom_model.id + "/assemble" ) ``` ## Register and deploy the LLM Next, register the model in the Registry. The model registry contains entries from all models (predictive, generative, built in DataRobot, and externally hosted). ``` if CUSTOM_MODEL_NAME not in [m.name for m in dr.RegisteredModel.list()]: print("Creating New Registered Model") registered_model_version = ( dr.RegisteredModelVersion.create_for_custom_model_version( model_version.id, name=CUSTOM_MODEL_NAME, description="LLM Wrapper Example from DataRobot Docs", registered_model_name=CUSTOM_MODEL_NAME, ) ) else: print("Using Existing Model") registered_model = [ m for m in dr.RegisteredModel.list() if m.name == CUSTOM_MODEL_NAME ].pop() registered_model_version = ( dr.RegisteredModelVersion.create_for_custom_model_version( model_version.id, name=CUSTOM_MODEL_NAME, description="LLM Wrapper Example from DataRobot Docs", registered_model_id=registered_model.id, ) ) ``` Now, deploy the model. If you are a DataRobot multitenant SaaS user, you must select a prediction environment. ``` pred_server = [s for s in dr.PredictionServer.list()][0] print(f"Prediction server ID: {pred_server}") ``` ``` MODEL_DEPLOYMENT_NAME = "LLM Wrapper Deployment" if MODEL_DEPLOYMENT_NAME not in [d.label for d in dr.Deployment.list()]: deployment = dr.Deployment.create_from_registered_model_version( registered_model_version.id, label=MODEL_DEPLOYMENT_NAME, description="Your new deployment", max_wait=1000, # Only needed for DataRobot Managed AI Platform default_prediction_server_id=pred_server.id, ) else: deployment = [d for d in dr.Deployment.list() if d.label == MODEL_DEPLOYMENT_NAME][ 0 ] ``` ## Test the deployment Test that the deployment can successfully provide responses to prompts. ``` from datarobot_predict.deployment import predict input_df = pd.DataFrame( { PROMPT_COLUMN_NAME: [ "Give me some context on large language models and their applications?", "What is AutoML?", "Tell me a joke", ], } ) result_df, response_headers = predict(deployment, input_df) result_df ``` ## Validate the external LLM The following methods execute and validate the external LLM. This example associates a Use Case with the validation and creates the vector database within that Use Case. Set the `use_case_id` to specify an existing Use Case or create a new one with that name. ``` # Option 1: Create a new Use Case (default approach) use_case = dr.UseCase.create() # Option 2: Use an existing Use Case (replace with a Use Case ID) # use_case_id = # use_case = dr.UseCase.get(use_case_id) ``` `CustomModelLLMValidation.create` executes the validation of the external LLM. Be sure to provide the deployment ID. ``` external_llm_validation = CustomModelLLMValidation.create( prompt_column_name=PROMPT_COLUMN_NAME, target_column_name=COMPLETION_COLUMN_NAME, deployment_id=deployment.id, name="My External LLM", use_case=use_case, wait_for_completion=True, ) ``` ``` assert external_llm_validation.validation_status == "PASSED" ``` ``` print(f"External LLM Validation ID: {external_llm_validation.id}") ``` This external LLM can now be used in the [GenAI E2E walkthrough](https://docs.datarobot.com/en/docs/gen-ai/genai-code/genai-e2e.html), for example to create the LLM blueprint. --- # Use the Bolt-on Governance API URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/genai-chat-completion-api.html This notebook outlines how to use the Bolt-on Governance API with [deployed LLM blueprints](https://docs.datarobot.com/en/docs/gen-ai/deploy-llm.html). LLM blueprints deployed from the playground implement the [chat()hook](https://docs.datarobot.com/en/docs/mlops/deployment/custom-models/drum/structured-custom-models.html#chat) in the custom model's `custom.py` file by default. You can use [the official Python library for the OpenAI API](https://github.com/openai/openai-python) to make chat completion requests to DataRobot LLM blueprint deployments: ``` !pip install openai ``` ``` from openai import OpenAI ``` Specify the ID of the LLM blueprint deployment and your DataRobot API token: ``` DEPLOYMENT_ID = "" DATAROBOT_API_TOKEN = "" DEPLOYMENT_URL = f"https://app.datarobot.com/api/v2/deployments/{DEPLOYMENT_ID}" ``` Use the code below to create an OpenAI client: ``` client = OpenAI(base_url=DEPLOYMENT_URL, api_key=DATAROBOT_API_TOKEN) ``` Use the code below to request a chat completion. See the [considerations](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/genai-chat-completion-api.html#considerations) below for more information on specifying the `model` parameter. Specifying the system message in the request overrides the system prompt configured in the LLM blueprint. Specifying other settings in the request, such as `max_completion_tokens`, overrides the settings of the LLM blueprint. ``` completion = client.chat.completions.create( model="datarobot-deployed-llm", messages=[ {"role": "system", "content": "Answer with just a number."}, {"role": "user", "content": "What is 2+3?"}, {"role": "assistant", "content": "5"}, {"role": "user", "content": "Now multiply the result by 4."}, {"role": "assistant", "content": "20"}, {"role": "user", "content": "Now divide the result by 2."}, ], ) ``` ``` print(completion) ``` This returns a `ChatCompletion` object if streaming is disabled and `Iterator[ChatCompletionChunk]` if streaming is enabled. Use the following cell to request a chat completion with a streaming response. ``` streaming_response = client.chat.completions.create( model="datarobot-deployed-llm", messages=[ {"role": "system", "content": "Explain your thoughts using at least 100 words."}, {"role": "user", "content": "What would it take to colonize Mars?"}, ], stream=True, ) ``` ``` for chunk in streaming_response: content = chunk.choices[0].delta.content if content is not None: print(content, end="") ``` To return `citations`, the deployed LLM must have a vector database associated with it.`completion` returns keys related to citations and accessible to custom models. ## Specify association ID and custom metrics When making a chat request to a DataRobot-deployed text generation or agentic workflow custom model, a custom association ID can be optionally specified for chat requests in place of the auto-generated ID by setting `datarobot_association_id` in the `extra_body` field of the chat request. Values can also be reported for arbitrary custom metrics defined for the deployment by setting `datarobot_metrics` in the `extra_body` field. To do this, define these values in the optional `extra_body` field of the chat request. The `extra_body` field is a standard way to add more parameters to an OpenAI chat request, allowing the chat client to pass model-specific parameters to an LLM. If the `datarobot_association_id` field is found in `extra_body`, DataRobot uses that value instead of the automatically generated one. If the `datarobot_metrics` field is found in `extra_body`, DataRobot reports a custom metric for all the `name:value` pairs found inside. A matching custom metric for each name must already be defined for the deployment. Custom metric values reported this way must be numeric. The deployed custom model must have an association ID column defined for DataRobot to process custom metrics from chat requests, regardless of whether `extra_body` is specified. Moderation must be configured for the custom model for the metrics to be processed. ``` extra_body = { # These values pass through to the LLM "llm_id": "azure-gpt-6", # If set here, replaces the auto-generated association ID "datarobot_association_id": "my_association_id_0001", # DataRobot captures these for custom metrics "datarobot_metrics": { "field1": 24, "field2": 25 } } completion = client.chat.completions.create( model="datarobot-deployed-llm", messages=[ {"role": "system", "content": "Explain your thoughts using at least 100 words."}, {"role": "user", "content": "What would it take to colonize Mars?"}, ], max_tokens=512, extra_body=extra_body ) print(completion.choices[0].message.content) ``` ## Moderation and guardrails Moderation guardrails help your organization block prompt injection and hateful, toxic, or inappropriate prompts and responses. To return `datarobot_moderations`, the deployed LLM must be running in an execution environment that has the moderation library installed, and the custom model code directory must contain `moderation_config.yaml` to configure the moderations. When using the Bolt-on Governance API with moderations configured, consider the following: - If there are no guards for the response stage, the moderation library returns the existing stream obtained from the LLM. - Not all response guards are applied to a chunk. The faithfulness, rouge-1, and nemo guards are not applied to chunk. Instead they are applied to the whole response when available because these guards need the whole response to be present in order to evaluate. - If moderation is enabled and the streaming response is requested, the first chunk will always contain the information about prompt guards (if configured) and response guards (excluding faithfulness, rouge-1, and nemo). Access the chunk via chunk.datarobot_moderations . - For every subsequent chunk that is not the last chunk, response guards (excluding faithfulness, rouge-1, and NeMo) are applied and can be accessed from chunk.datarobot_moderations . - The last chunk has all response guards (excluding faithfulness, rouge-1 and nemo) applied to the chunk. Faithfulness, rouge-1, and nemo are applied to the whole response. - If streaming is the aggregration, the following custom metrics are reported: ## Considerations When using the Bolt-on Governance API, consider the following: - If you implement the chat completion hook without modification, the chat() hook behaves differently than the score() hook. Specifically, the unmodified chat() hook passes in themodelparameter through thecompletion_create_paramsargument while the score() hook specifies the model in the custom model code. - If you add a deployed LLM to the playground , the validation uses the value entered into the "Chat model ID" field as the model parameter value. Ensure the LLM deployment accepts this value as the model parameter. Alternatively, you can modify the implementation of the chat() hook to override the value of the model parameter, defining the intended model (for example, using a runtime parameter ). For more information, see GenAI troubleshooting . - GPU-backed inference is also available via the Workload API , which deploys open source LLMs from the Hugging Face Hub as managed vLLM services. --- # End-to-end code-first generative AI experimentation URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/genai-e2e.html This notebook outlines a comprehensive overview of the generative AI features of DataRobot. Note: For self-managed users, code samples that reference `app.datarobot.com` need to be changed to the appropriate URL for your instance. ``` import datarobot as dr from datarobot.models.genai.vector_database import VectorDatabase from datarobot.models.genai.vector_database import ChunkingParameters from datarobot.enums import PromptType from datarobot.enums import VectorDatabaseEmbeddingModel from datarobot.enums import VectorDatabaseChunkingMethod from datarobot.models.genai.playground import Playground from datarobot.models.genai.llm import LLMDefinition from datarobot.models.genai.llm_blueprint import LLMBlueprint from datarobot.models.genai.llm_blueprint import VectorDatabaseSettings from datarobot.models.genai.chat import Chat from datarobot.models.genai.chat_prompt import ChatPrompt from datarobot.models.genai.vector_database import CustomModelVectorDatabaseValidation from datarobot.models.genai.comparison_chat import ComparisonChat from datarobot.models.genai.comparison_prompt import ComparisonPrompt from datarobot.models.genai.custom_model_llm_validation import CustomModelLLMValidation from pprint import pprint ``` ### Connect to DataRobot Read more about different options for [connecting to DataRobot from the Python client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` endpoint="https://app.datarobot.com/api/v2" token="" dr.Client(endpoint=endpoint, token=token) ``` ### Create a Use Case Generative AI works within a DataRobot Use Case, so to begin, create a new use case. ``` use_case = dr.UseCase.create() ``` This workflow uses a vector database with GenAI models. Next, you will upload a dataset with a set of documents. ``` # this can be updated with any public URL that is pointing to a .zip file # in the expected format dataset_url = "https://s3.amazonaws.com/datarobot_public_datasets/genai/pirate_resumes.zip" # We will use a vector database with our GenAI models. Let's upload a dataset with our documents. # If you wish to use a local file as dataset, change this to # `dataset = dr.Dataset.create_from_file(local_file_path)` dataset = dr.Dataset.create_from_url(dataset_url) ``` ### Create a vector database ``` chunking_parameters = ChunkingParameters( embedding_model=VectorDatabaseEmbeddingModel.JINA_EMBEDDING_T_EN_V1, chunking_method=VectorDatabaseChunkingMethod.RECURSIVE, chunk_size=128, chunk_overlap_percentage=25, separators=[], ) vdb = VectorDatabase.create(dataset.id, chunking_parameters, use_case) ``` Use the following cell to retrieve the vector database and make sure it has completed. Approximate completion time is 30-60 seconds. ``` vdb = VectorDatabase.get(vdb.id) assert vdb.execution_status == "COMPLETED" ``` ### Create an LLM playground Create an LLM playground that will host large language models (LLMs). ``` playground = Playground.create(name="New playground for example", use_case=use_case) ``` Use the following cell to see what kind of LLMs are available. By default this method returns as a list of dicts for easy readability. It includes a list of allowed settings for each LLM along with any constraints on the values. ``` llms_dict = LLMDefinition.list(use_case=use_case) print(f"Number of LLMs available: {len(llms_dict)}") pprint(llms_dict[0]) ``` As an example, use the first LLM in the list. ``` llms = LLMDefinition.list(use_case=use_case, as_dict=False) llm = llms[0] ``` To interact with the LLM, you need to create an LLM blueprint with the settings that you want. Since the allowed settings depend on the LLM type, take a generic dict with those values that will be validated during the creation of the blueprint. Review the allowed settings for the selected LLM. ``` pprint([setting.to_dict() for setting in llm.settings]) ``` ### Create an LLM blueprint ``` llm_settings = { "system_prompt": ( "You are a pirate who begins each response with 'Arrr' " "and ends each response with 'Matey!'" ), "max_completion_length": 1024, # Let's ask for short responses "temperature": 1.0, # Stay within common LLM temperature bounds } # PromptType.ONE_TIME_PROMPT is an alternative if you don't wish # for previous chat prompts (history) to be included in each subsequent prompt prompting_strategy = PromptType.CHAT_HISTORY_AWARE ``` Use some vector database settings that could be different from the defaults when the VDB was created. ``` vector_database_settings = VectorDatabaseSettings( max_documents_retrieved_per_prompt=2, max_tokens=128, ) llm_blueprint = LLMBlueprint.create( playground=playground, name=llm.name, llm=llm, prompt_type=prompting_strategy, llm_settings=llm_settings, vector_database=vdb, vector_database_settings=vector_database_settings, ) ``` ### Create a chat Now create a chat and associate it with the LLM blueprint you just created. A chat is the entity that groups together a set of prompt messages. It can be thought of as a conversation with a particular LLM blueprint. ``` chat = Chat.create( name="Resume review chat with a pirate", llm_blueprint=llm_blueprint ) ``` ### Chat with the LLM blueprint Use the following cell to chat with the LLM blueprint. ``` prompt1 = ChatPrompt.create( chat=chat, text="How can a pirate resume be evaluated?", wait_for_completion=True, ) print(prompt1.result_text) ``` Lower the temperature to reduce the variability in the response. ``` llm_settings["temperature"] = 0.8 prompt2 = ChatPrompt.create( text="Please summarize the best pirate resume from those retrieved.", chat=chat, llm_settings=llm_settings, wait_for_completion=True, ) print(prompt2.result_text) ``` Verify that the new llm_settings are saved. ``` llm_blueprint = LLMBlueprint.get(llm_blueprint.id) print(llm_blueprint.llm_settings) ``` Confirm that the LLM is using information from the vector database. ``` print([citation.text for citation in prompt2.citations]) ``` Next, create a new LLM blueprint from the existing one so that you can change some settings and compare the two. ``` new_llm_blueprint = LLMBlueprint.create_from_llm_blueprint(llm_blueprint, name="new blueprint") ``` Remove the vector database from the new blueprint, modify the settings as shown below, and save the blueprint. ``` llm_settings["system_prompt"] = "You are an AI assistant who helps to evaluate resumes." new_llm_blueprint = new_llm_blueprint.update( llm_settings=llm_settings, remove_vector_database=True, ) ``` Once you are satisfied with the blueprint, in the UI you can perform an [LLM blueprint comparison](https://docs.datarobot.com/en/docs/gen-ai/compare-llm.html) or [deploy it from the playground](https://docs.datarobot.com/en/docs/gen-ai/deploy-llm.html). ### Add an external vector database Now add an external vector database from a deployment. Note that creating the deployment is outside the scope of this example. To continue with this portion of the example, first create an external vector database deployment, validate and deploy it, and update the `external_vdb_id` value here. You can create an external vector database and deploy it by first following the guide to create a [Chroma VDB](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/chromadb-vdb.html) or [Qdrant VDB.](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/qdrantdb-vdb.html) Once you have done that, simply update your external vector database ID here to use it. ``` external_vdb_id = "" ``` ``` # Now create a vector database external_model_vdb = VectorDatabase.get( vector_database_id=external_vdb_id ) assert external_model_vdb.execution_status == "COMPLETED" ``` ### Add an external model LLM Now add an external LLM blueprint from a deployment. Because creating the deployment is outside the scope of this example, it instead assumes a previously deployed text generation model. To continue with this portion of the example, first create an external LLM deployment, validate it, and update the `external_llm_validation_id` value here. You can create an external LLM and validate it by first following [this guide.](https://docs.datarobot.com/en/docs/gen-ai/genai-code/ext-llm.html) Once you have done that, update your external LLM validation ID here to use it. ``` external_llm_validation_id = "" ``` ``` external_llm_validation = CustomModelLLMValidation.get(validation_id=external_llm_validation_id) assert external_llm_validation.validation_status == "PASSED" ``` Now you can create an LLM blueprint using the custom model LLM and the custom model vector database. ``` custom_model_llm_blueprint = LLMBlueprint.create( playground=playground, name="custom model LLM with custom model vdb", llm="custom-model", llm_settings={ "validation_id": external_llm_validation.id, # NOTE - update this value based on the context size in tokens # of the external LLM that you are deploying "external_llm_context_size": 4096 }, vector_database=external_model_vdb, ) ``` ### Create a comparison chat Finally create a `comparison_chat` and associate it with a playground. A comparison chat, analogous to the chat for a chat prompt, is the entity that groups together a set of comparison prompt messages. It can be thought of as a conversation with one or more LLM blueprints. ``` comparison_chat = ComparisonChat.create( name="Resume review comparison chat with a pirate", playground=playground ) ``` Now compare the LLM blueprints. The response of the `custom_model_llm_blueprint` as it is set up in the documentation will not be related to the pirate resume and that is expected. Feel free to modify the external VDB creation and/or external LLM to suit your use case. ``` comparison_prompt = ComparisonPrompt.create( llm_blueprints=[llm_blueprint, new_llm_blueprint, custom_model_llm_blueprint], text="Summarize the best resume", comparison_chat=comparison_chat, wait_for_completion=True, ) for result in comparison_prompt.results: print(result.result_text) print("\n\n") ``` --- # Code walkthroughs URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/index.html > Use code to create and validate external vector databases and LLMs; walk through a comprehensive GenAI overview. > [!NOTE] Availability information > DataRobot's GenAI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. For code-first users, the following sections provide code samples that create and validate external vector databases and LLMs. Additionally, use the end-to-end notebook to walk through a comprehensive overview of GenAI features. See the [list of considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html) to keep in mind when working with DataRobot GenAI. | Topic | Description | | --- | --- | | End-to-end code-first generative AI experimentation | A comprehensive overview of the generative AI features DataRobot has to offer with the Python API client. | | Create and deploy a vector database | How to use the Python SDK to create and deploy DataRobot vector databases using built-in embeddings. For custom embedding models (BYO embeddings), see the separate notebook below. | | Create vector databases from BYO embeddings | How to build, validate, and register an external vector database from bring-your-own (BYO) embeddings. | | Create external LLMs with code | How to set up and validate an external LLM using DataRobot's Python API client. | | Use the DataRobot LLM gateway | How to use the OpenAI Python library to make chat completion requests directly to the DataRobot LLM gateway. | | Agentic workflow with code | How to use a simple agentic workflow to serve, monitor, and govern the workflow. | | Use the Bolt-on Governance API | How to use the OpenAI Python library to make chat completion requests to a deployed LLM blueprint. | | Create a ChromaDB vector database | How to load in and host a ChromaDB in-memory vector store, with metadata filtering, within a custom model. | | Build and host a Qdrant vector database | How to build, validate, and register a Qdrant vector database to the DataRobot application using DataRobot's Python API client. | --- # Build and host a Qdrant vector database URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-code/qdrantdb-vdb.html This page provides an example of how you can build, validate, and register a vector database to the DataRobot application using DataRobot's Python API client. It describes how to load and host a Qdrant vector store with metadata filtering as part of a custom model. This page is designed for use within a DataRobot [codespace](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html). DataRobot recommends downloading this notebook and uploading it for use in the application. ## Setup The following steps outline the necessary configuration to integrate vector databases with the DataRobot platform. 1. This workflow uses the following feature flags. Contact your DataRobot representative or administrator for information on enabling these features. 2. Use a codespace, not a DataRobot Notebook, to ensure this notebook has access to a filesystem. Use Python 3.12 to match the GenAI execution environment used for deployment. 3. Usepip installto install the packages outlined in the following section if they are not already in the codespace's environment image. 4. Set the notebook session timeout to 180 minutes. 5. Restart the notebook container using at least a "Medium" (16GB RAM) instance. ### Install libraries Install the following libraries: ``` !pip install "langchain-community==0.4.1" \ "langchain_text_splitters==1.0.0" \ "qdrant-client==1.16.1" \ "sentence-transformers==5.1.2" \ "datarobot" \ "datarobot-predict" ``` ``` import datarobot as dr from datarobot.models.genai.vector_database import CustomModelVectorDatabaseValidation from datarobot.models.genai.vector_database import VectorDatabase import os from pathlib import Path from qdrant_client import QdrantClient, models from sentence_transformers import SentenceTransformer from langchain_community.document_loaders import TextLoader from langchain_text_splitters import RecursiveCharacterTextSplitter import json import requests import zipfile import io import re ``` ### Connect to DataRobot Read more about options for [connecting to DataRobot from the Python client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ### Download sample data This example references a sample dataset made from the DataRobot english documentation. Feel free to use your own data here. Note: If you are a self-managed user, you must modify code samples that reference `app.datarobot.com` to the appropriate URL for your instance. ## Configuration ``` QDRANT_DATA_PATH = "qdrant" EMBEDDING_MODEL_NAME = "all-MiniLM-L6-v2" COLLECTION_NAME = "my_documents" SOURCE_DOCUMENTS_ZIP_URL = "https://s3.amazonaws.com/datarobot_public_datasets/ai_accelerators/datarobot_english_documentation_5th_December.zip" UNZIPPED_DOCS_DIR = "datarobot_english_documentation" ``` ## Verify docs exist or download them ``` doc_path = Path(UNZIPPED_DOCS_DIR) if doc_path.exists(): txt_files = list(doc_path.rglob("*.txt")) print(f"✓ Found {len(txt_files)} .txt files in {UNZIPPED_DOCS_DIR}/") if txt_files: print(f" Sample files:") for f in txt_files[:5]: print(f" - {f}") else: # Download some example docs r = requests.get(SOURCE_DOCUMENTS_ZIP_URL) z = zipfile.ZipFile(io.BytesIO(r.content)) z.extractall() ``` ## Create a vector database from documents Use the following cell to build a vector database from the DataRobot documentation dataset. Note that this notebook uses Qdrant, an open source vector database with metadata filtering support. Additionally, this notebook uses the HuggingFace `all-MiniLM-L6-v2` [embeddings model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) (also open source). ``` # Create vectorDB def create_database(): """Create and populate the Qdrant database""" # Initialize encoder encoder = SentenceTransformer(EMBEDDING_MODEL_NAME) # Initialize Qdrant client client = QdrantClient(path=QDRANT_DATA_PATH) try: # Create collection print("Creating collection...") client.create_collection( collection_name=COLLECTION_NAME, vectors_config=models.VectorParams( size=encoder.get_sentence_embedding_dimension(), distance=models.Distance.COSINE, ), ) # Load text files print(f"Loading documents from {UNZIPPED_DOCS_DIR}/...") docs = [] doc_path = Path(UNZIPPED_DOCS_DIR) for file_path in doc_path.rglob("*.txt"): loader = TextLoader(str(file_path)) loaded_docs = loader.load() docs.extend(loaded_docs) print(f"Loaded {len(docs)} documents") # Split documents into chunks splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=50) split_docs = splitter.split_documents(docs) print(f"Split into {len(split_docs)} chunks") # Process metadata for doc in split_docs: # Convert file path to documentation URL doc.metadata['source'] = re.sub( r'datarobot_english_documentation/datarobot_docs/en/(.+)\.(txt|md)', r'https://docs.datarobot.com/en/docs/\1.html', doc.metadata.get('source', '') ) # Extract category from source doc.metadata["category"] = doc.metadata["source"].split("/")[-1].split(".")[0] # Batch encode all documents print("Encoding all documents (this may take a few minutes)...") all_contents = [doc.page_content for doc in split_docs] all_vectors = encoder.encode( all_contents, show_progress_bar=False, batch_size=32, convert_to_numpy=True ) # Create points print("Creating point structures...") points_to_upload = [ models.PointStruct( id=idx, vector=all_vectors[idx].tolist(), payload={ "content": doc.page_content, "source": doc.metadata.get("source", ""), "category": doc.metadata.get("category", ""), **doc.metadata } ) for idx, doc in enumerate(split_docs) ] # Upload to Qdrant print(f"Uploading {len(points_to_upload)} points to Qdrant...") client.upload_points( collection_name=COLLECTION_NAME, points=points_to_upload ) # Verify collection_info = client.get_collection(collection_name=COLLECTION_NAME) print(f"✓ Collection '{COLLECTION_NAME}' created with {collection_info.points_count} points") print(f"✓ Data saved to: {QDRANT_DATA_PATH}/") finally: # ALWAYS close the client client.close() print("✓ Database client closed") # Run the creation (only once) create_database() ``` ### Test the vector database Use the following cell to test the vector database by having the model perform a similarity search. It returns the top five documents matching the query provided. ``` # Test a query to the vectorDB def test_database(): """Test the database with a sample query""" encoder = SentenceTransformer(EMBEDDING_MODEL_NAME) client = QdrantClient(path=QDRANT_DATA_PATH) try: question = "What is MLOps?" query_vector = encoder.encode(question).tolist() results = client.query_points( collection_name=COLLECTION_NAME, query=query_vector, limit=5, ).points print(f"Found {len(results)} results for: '{question}'\n") for hit in results: print(f"Score: {hit.score:.4f}") print(f"Content: {hit.payload.get('content', '')[:200]}...") print(f"Source: {hit.payload.get('source', '')}") print(f"Category: {hit.payload.get('category', '')}\n") finally: # ALWAYS close the client client.close() print("✓ Test client closed") # Run the test test_database() ``` ## Define hooks to deploy an unstructured custom model The following cell defines the methods used to deploy an unstructured custom model. These include loading the custom model and using the model for scoring. In this notebook, the vector database is loaded to DataRobot's infrastructure. Alternatively, you could use a proxy endpoint in `load_model` that points to where the vector database is actually stored. ``` def load_model(input_dir): """Custom model hook for loading our Qdrant knowledge base.""" import os from qdrant_client import QdrantClient from sentence_transformers import SentenceTransformer print("Loading model") EMBEDDING_MODEL_NAME = 'all-MiniLM-L6-v2' COLLECTION_NAME = "my_documents" # When deploying model="qdrant/", the files are at input_dir/qdrant/ # not directly at input_dir if input_dir: QDRANT_DATA_PATH = os.path.join(input_dir, "qdrant") else: QDRANT_DATA_PATH = "qdrant" print(f'QDRANT_DATA_PATH = {QDRANT_DATA_PATH}') print(f'EMBEDDING_MODEL_NAME = {EMBEDDING_MODEL_NAME}') print(f'COLLECTION_NAME = {COLLECTION_NAME}') # Initialize the embedding model (downloads if needed) encoder = SentenceTransformer(EMBEDDING_MODEL_NAME) # Initialize Qdrant client client = QdrantClient(path=QDRANT_DATA_PATH) # Get collection info to verify it loaded collection_info = client.get_collection(collection_name=COLLECTION_NAME) print(f'Loaded Qdrant collection "{COLLECTION_NAME}" with {collection_info.points_count} points') return { "client": client, "encoder": encoder, "collection_name": COLLECTION_NAME } def score_unstructured(model, data, **kwargs) -> str: """Custom model hook for retrieving relevant docs with our Qdrant knowledge base. When requesting predictions from the deployment, pass a dictionary with the following keys: - 'question' the question to be passed to the vector store retriever - 'filter' any metadata filter - 'k' the number of results to return (default: 10) datarobot-user-models (DRUM) handles loading the model and calling this function with the appropriate parameters. Returns: -------- rv : str Json dictionary with keys: - 'question' user's original question - 'relevant' the retrieved document contents - 'metadata' - metadata for each document including similarity scores - 'error' - error message if exception in handling request """ import json from qdrant_client import models try: # Loading data data_dict = json.loads(data) question = data_dict['question'] metadata_filter = data_dict.get("filter", None) top_k = data_dict.get("k", 10) # Defining info client = model["client"] encoder = model["encoder"] collection_name = model["collection_name"] # Encode the question query_vector = encoder.encode(question).tolist() # Build query parameters query_params = { "collection_name": collection_name, "query": query_vector, "limit": top_k, } # Only add filter if it exists if metadata_filter is not None: query_params["query_filter"] = models.Filter(**metadata_filter) # Perform the search results = client.query_points(**query_params).points print(f'Returned {len(results)} results') relevant, metadata = [], [] for hit in results: # Extract the content from payload content = hit.payload.get('content', '') relevant.append(content) # Add similarity score to metadata hit_metadata = dict(hit.payload) hit_metadata["similarity_score"] = hit.score metadata.append(hit_metadata) rv = { "question": question, "relevant": relevant, "metadata": metadata, } except Exception as e: rv = {'error': f"{e.__class__.__name__}: {str(e)}"} return json.dumps(rv), {"mimetype": "application/json", "charset": "utf8"} ``` ### Test hooks locally Before proceeding with deployment, use the cell below to test that the custom model hooks function correctly. ``` # Testing to ensure they work locally before deploying def test_hooks(): """Test the DataRobot hooks locally""" # Load the model - pass None so it uses "qdrant" as the path model = load_model(None) try: # Test scoring print("=" * 80) print("TEST: Basic search") print("=" * 80) result = score_unstructured( model, json.dumps({ "question": "What is MLOps?", "filter": models.Filter( must=[ models.FieldCondition( key="category", match=models.MatchValue(value="datarobot_docs|en|more-info|eli5") ) ] ).model_dump(), # converting to be json serializable (will be convert back before scoring) "k": 3, }) ) response = json.loads(result[0]) print(f"Question: {response['question']}") print(f"Found {len(response['relevant'])} results:\n") for idx, (content, meta) in enumerate(zip(response['relevant'], response['metadata'])): print(f"{idx+1}. Score: {meta['similarity_score']:.4f}") print(f" Source: {meta.get('source', 'N/A')}") print(f" Category: {meta.get('category', 'N/A')}") print(f" Content: {content[:150]}...") print() finally: # ALWAYS close the client after testing model["client"].close() print("✓ Test client closed") # Run the test test_hooks() ``` ## Deploy the knowledge base The cell below uses the DataRobot Python client to: - Package the custom model hooks and qdrant/ vector database artifacts. - Create an unstructured custom model in the workshop. - Upload a new custom model version using the GenAI Python 3.12 execution environment. - Deploy the model and return a dr.Deployment object for predictions. This example uses a pre-built environment. You can also provide an `environment_id` and instead use an existing custom model environment for shorter iteration cycles on the custom model hooks. See your account's existing pre-built environments from the [DataRobot Workshop](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-environment-workshop/nxt-drop-in-envs.html). ``` import inspect from pathlib import Path CUSTOM_MODEL_NAME = "Qdrant Vector Database" DEPLOY_DIR = Path("deploy_model") DEPLOY_DIR.mkdir(parents=True, exist_ok=True) (DEPLOY_DIR / "custom.py").write_text( inspect.getsource(load_model) + "\n\n" + inspect.getsource(score_unstructured) ) (DEPLOY_DIR / "requirements.txt").write_text( "qdrant-client\n" "sentence-transformers\n" ) model_files = [ (str(DEPLOY_DIR / "custom.py"), "custom.py"), (str(DEPLOY_DIR / "requirements.txt"), "requirements.txt"), ] source_root = Path("qdrant") for path in source_root.rglob("*"): if path.is_file(): model_files.append( (str(path), f"qdrant/{path.relative_to(source_root).as_posix()}") ) genai_environment = dr.ExecutionEnvironment.list( search_for="[GenAI] Python 3.12 with Moderations" )[0] existing_models = [m for m in dr.CustomInferenceModel.list() if m.name == CUSTOM_MODEL_NAME] if existing_models: custom_model = existing_models[0] else: custom_model = dr.CustomInferenceModel.create( name=CUSTOM_MODEL_NAME, target_type=dr.TARGET_TYPE.UNSTRUCTURED, language="python", ) print("Uploading custom model version") model_version = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=genai_environment.id, files=model_files, maximum_memory=8 * 1024 * 1024 * 1024, max_wait=60 * 30, ) print("Building custom model dependency image") try: build_info = dr.CustomModelVersionDependencyBuild.start_build( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, max_wait=60 * 60, ) except dr.errors.ClientError as e: if "already has a dependency image" in str(e): print("Dependency build already exists, skipping build step") build_info = dr.CustomModelVersionDependencyBuild.get_build_info( custom_model.id, model_version.id, ) else: raise if build_info is not None and build_info.build_status != "success": raise RuntimeError( f"Dependency build failed with status '{build_info.build_status}'.\n" f"{build_info.get_log()}" ) print("Creating deployment") pred_server = dr.PredictionServer.list()[0] deployment = dr.Deployment.create_from_custom_model_version( model_version.id, label=CUSTOM_MODEL_NAME, default_prediction_server_id=pred_server.id, max_wait=60 * 30, ) print(f"Deployment ID: {deployment.id}") ``` ### Test the deployment Test that the deployment can successfully provide responses to questions using the [datarobot-predict](https://datarobot.github.io/datarobot-predict/1.13/deployment/) library. ``` from datarobot_predict.deployment import predict_unstructured # Now with metadata filtering data = { "question": "How do I replace a custom model on an existing custom environment?", "filter": models.Filter( must=[ models.FieldCondition( key="category", match=models.MatchValue(value="datarobot_docs|en|modeling|special-workflows|cml|cml-custom-env") ) ] ).model_dump(), "k": 5, } # Prediction request content, response_headers = predict_unstructured( deployment=deployment, data=data, ) # Check output content ``` ## Validate and create the vector database These methods execute, validate, and integrate the vector database. This example associates a Use Case with the validation and creates the vector database within that Use Case. Use the current Use Case when running in a DataRobot Codespace, or set `use_case_id` to specify an existing Use Case. Uncomment the create line to make a new Use Case instead. ``` use_case_id = os.environ['DATAROBOT_DEFAULT_USE_CASE'] use_case = dr.UseCase.get(use_case_id) # UNCOMMENT if you want to create a new Use Case # use_case = dr.UseCase.create() ``` ### Validate the vector database The `CustomModelVectorDatabaseValidation.create` function executes the validation of the vector database. Be sure to provide the deployment ID. ``` external_vdb_validation = CustomModelVectorDatabaseValidation.create( prompt_column_name="question", target_column_name="relevant", deployment_id=deployment.id, use_case=use_case, wait_for_completion=True ) external_vdb_validation ``` ``` assert external_vdb_validation.validation_status == "PASSED" ``` ### Create the vector database After validation completes, use `VectorDatabase.create_from_custom_model()` to integrate the vector database. You must provide the Use Case name (or Use Case ID), a name for the external vector database, and the validation ID returned from the previous cell. ``` vdb = VectorDatabase.create_from_custom_model( name="Qdrant Vector Database", use_case=use_case, validation_id=external_vdb_validation.id ) vdb ``` ``` assert vdb.execution_status == "COMPLETED" ``` ``` print(f"Vector Database ID: {vdb.id}") ``` This vector database ID can now be used alongside [LLM blueprints](https://docs.datarobot.com/en/docs/gen-ai/playground-tools/build-llm-blueprints.html#add-a-vector-database) to create RAG workflows. --- # NVIDIA NIM gallery information URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/genai-nim-gallery-support.html > Comprehensive table of NIM gallery information including model names, types, chat model IDs, playground support, platform support, and documentation links for unstructured models. This is a comprehensive table of NVIDIA NIM gallery information, including model names and types, chat model IDs, playground support information, platform support information, and documentation links for unstructured models. | NIM | Type | Chat model ID | Supported in playground | Platform support | | --- | --- | --- | --- | --- | | codellama-13b-instruct | Text Generation | codellama/codellama-13b-instruct | Yes | Cloud, 11.1 and later | | codellama-34b-instruct | Text Generation | codellama/codellama-34b-instruct | Yes | Cloud, 11.1 and later | | codellama-70b-instruct | Text Generation | codellama/codellama-70b-instruct | Yes | Cloud, 11.1 and later | | deepseek-r1-distill-llama-8b | Text Generation | deepseek-ai/deepseek-r1-distill-llama-8b | Yes | Cloud, 11.1 and later | | deepseek-r1-distill-qwen-7b | Text Generation | deepseek-ai/deepseek-r1-distill-qwen-7b | Yes | Cloud, 11.1 and later | | deepseek-r1-distill-qwen-14b | Text Generation | deepseek-ai/deepseek-r1-distill-qwen-14b | Yes | Cloud, 11.1 and later | | deepseek-r1-distill-qwen-32b | Text Generation | deepseek-ai/deepseek-r1-distill-qwen-32b | Yes | Cloud, 11.1 and later | | deepseek-v4-flash | Text Generation | deepseek-ai/deepseek-v4-flash | No | 11.10 and later | | deepseek-v4-pro | Text Generation | deepseek-ai/deepseek-v4-pro | No | 11.10 and later | | gemma-2-2b-instruct | Text Generation | google/gemma-2-2b-instruct | Yes | Cloud, 11.1 and later | | gemma-2-9b-it | Text Generation | google/gemma-2-9b-it | Yes | Cloud, 11.1 and later | | gpt-oss-120b | Text Generation | openai/gpt-oss-120b | Yes | Cloud, 11.2 and later | | gpt-oss-20b | Text Generation | openai/gpt-oss-20b | Yes | Cloud, 11.2 and later | | kimi-k2.6 | Text Generation | moonshotai/kimi-k2.6 | No | 11.10 and later | | llama-2-13b-chat | Text Generation | meta/llama-2-13b-chat | Yes | Cloud, 11.1 and later | | llama-2-7b-chat | Text Generation | meta/llama-2-7b-chat | Yes | Cloud, 11.1 and later | | llama-2-70b-chat | Text Generation | meta/llama-2-70b-chat | No | 11.1 and later | | llama-3-sqlcoder-8b | Text Generation | defog/llama-3-sqlcoder-8b | Yes | Cloud, 11.1 and later | | llama-3-swallow-70b-instruct-v0.1 | Text Generation | tokyotech-llm/llama-3-swallow-70b-instruct-v0.1 | No | 11.1 and later | | llama-3-taiwan-70b-instruct | Text Generation | yentinglin/llama-3-taiwan-70b-instruct | No | 11.1 and later | | llama-3.1-70b-instruct | Text Generation | meta/llama-3.1-70b-instruct | Yes | Cloud, 11.1 and later | | llama-3.1-8b-instruct | Text Generation | meta/llama-3.1-8b-instruct | Yes | Cloud, 11.1 and later | | llama-3.1-8b-instruct-pb24h2 | Text Generation | meta/llama-3.1-8b-instruct-pb24h2 | Yes | Cloud, 11.1 and later | | llama-3.1-70b-instruct-pb24h2 | Text Generation | meta/llama-3.1-70b-instruct-pb24h2 | Yes | Cloud, 11.1 and later | | llama-3.1-nemotron-nano-8b-v1 | Text Generation | nvidia/llama-3.1-nemotron-nano-8b-v1 | Yes | Cloud, 11.1 and later | | llama-3.1-nemotron-70b-instruct | Text Generation | nvidia/llama-3.1-nemotron-70b-instruct | Yes | Cloud, 11.1 and later | | llama-3.1-nemotron-ultra-253b-v1 | Text Generation | nvidia/llama-3.1-nemotron-ultra-253b-v1 | No | 11.1 and later | | llama-3.1-swallow-70b-instruct-v0.1 | Text Generation | tokyotech-llm/llama-3.1-swallow-70b-instruct-v0.1 | Yes | Cloud, 11.1 and later | | llama-3.2-1b-instruct | Text Generation | meta/llama-3.2-1b-instruct | Yes | Cloud, 11.1 and later | | llama-3.2-3b-instruct | Text Generation | meta/llama-3.2-3b-instruct | Yes | Cloud, 11.1 and later | | llama-3.2-11b-vision-instruct | Text Generation | meta/llama-3.2-11b-vision-instruct | Yes | Cloud, 11.1 and later | | llama-3.2-90b-vision-instruct | Text Generation | meta/llama-3.2-90b-vision-instruct | No | 11.1 and later | | llama-3.3-70b-instruct | Text Generation | meta/llama-3.3-70b-instruct | Yes | Cloud, 11.1 and later | | llama-3.3-nemotron-super-49b-v1 | Text Generation | nvidia/llama-3.3-nemotron-super-49b-v1 | Yes | Cloud, 11.1 and later | | llama-3.3-nemotron-super-49b-v1.5 | Text Generation | nvidia/llama-3-3-nemotron-super-49b-v1-5 | Yes | Cloud, 11.2 and later | | llama-4-scout-17b-16e-instruct | Text Generation | meta/llama-4-scout-17b-16e-instruct | Yes | 11.2 and later | | llama3-70b-instruct | Text Generation | meta/llama3-70b-instruct | No | 11.1 and later | | llama3-8b-instruct | Text Generation | meta/llama3-8b-instruct | Yes | Cloud, 11.1 and later | | mistral-7b-instruct-v0.3 | Text Generation | mistralai/mistral-7b-instruct-v0.3 | Yes | Cloud, 11.1 and later | | mistral-nemo-12b-instruct | Text Generation | mistral-nemo-12b-instruct | Yes | Cloud, 11.1 and later | | mistral-nemo-minitron-8b-8k-instruct | Text Generation | nv-mistralai/mistral-nemo-minitron-8b-8k-instruct | Yes | Cloud, 11.1 and later | | mixtral-8x7b-instruct-v01 | Text Generation | mistralai/mixtral-8x7b-instruct-v0.1 | Yes | Cloud, 11.1 and later | | mixtral-8x22b-instruct-v01 | Text Generation | mistralai/mixtral-8x22b-instruct-v01 | No | 11.1 and later | | nemotron-3-nano | Text Generation | nvidia/nemotron-3-nano | Yes | Cloud | | nemotron-3-super-120b-a12b | Text Generation | nvidia/nemotron-3-super-120b-a12b | Yes | Cloud, 11.7 and later | | nemotron-3-super-120b-a12b-fp8-4xl40s | Text Generation | nvidia/nemotron-3-super-120b-a12b | No | Cloud, 11.10 and later | | nvidia-nemotron-nano-9b-v2 | Text Generation | nvidia/nvidia-nemotron-nano-9b-v2 | Yes | 11.1 and later | | phi-3-mini-4k-instruct | Text Generation | microsoft/phi-3-mini-4k-instruct | Yes | Cloud, 11.1 and later | | qwen-2.5-7b-instruct | Text Generation | qwen/qwen-2.5-7b-instruct | Yes | Cloud, 11.1 and later | | qwen3-32b | Text Generation | qwen/qwen3-32b | Yes | 11.2 and later | | qwen3-next-80b-a3b-thinking | Text Generation | qwen/qwen3-next-80b-a3b-thinking | Yes | 11.2 and later | | starcoder2-7b | Text Generation | bigcode/starcoder2-7b | Yes | Cloud, 11.1 and later | | cosmos-predict1-7b-text2world | Unstructured | - | - | 11.2 and later | | cosmos-predict1-7b-video2world | Unstructured | - | - | 11.2 and later | | cosmos-reason2-2b | Unstructured | - | - | Cloud, 11.7 and later | | cosmos-reason2-8b | Unstructured | - | - | Cloud, 11.7 and later | | cuopt | Unstructured | - | - | Cloud, 11.1 and later | | diffdock | Unstructured | - | - | Cloud, 11.7 and later | | genmol | Unstructured | - | - | Cloud, 11.1 and later | | arctic-embed-l | Embedding/Unstructured | - | - | Cloud, 11.1 and later | | llama-3.1-nemotron-nano-vl-8b-v1 | Unstructured | - | - | 11.2 and later | | llama-3.2-nv-embedqa-1b-v2 | Embedding/Unstructured | - | - | Cloud, 11.1 and later | | nv-embedqa-e5-v5 | Embedding/Unstructured | - | - | Cloud, 11.1 and later | | nv-embedqa-e5-v5-pb24h2 | Embedding/Unstructured | - | - | Cloud, 11.1 and later | | nv-embedqa-mistral-7b-v2 | Embedding/Unstructured | - | - | Cloud, 11.1 and later | | nvclip | Embedding/Unstructured | - | - | Cloud, 11.1 and later | | llama-3.2-nv-rerankqa-1b-v2 | Unstructured | - | - | Cloud, 11.1 and later | | molmim | Unstructured | - | - | Cloud, 11.1 and later | | nemoretriever-graphic-elements-v1 | Unstructured | - | - | Cloud, 11.1 and later | | nemoretriever-page-elements-v2 | Unstructured | - | - | Cloud, 11.1 and later | | nemoretriever-parse | Unstructured | - | - | Cloud, 11.1 and later | | nemoretriever-table-structure-v1 | Unstructured | - | - | Cloud, 11.1 and later | | nv-rerankqa-mistral-4b-v3 | Unstructured | - | - | Cloud, 11.1 and later | | openfold2 | Unstructured | - | - | 11.1 and later | | openfold3 | Unstructured | - | - | Cloud, 11.7 and later | | boltz2 | Unstructured | - | - | Cloud, 11.7 and later | | paddleocr | Unstructured | - | - | Cloud, 11.1 and later | | proteinmpnn | Unstructured | - | - | Cloud, 11.1 and later | | rfdiffusion | Unstructured | - | - | Cloud, 11.1 and later | | llama-3.1-nemoguard-8b-content-safety | Evaluation | - | - | Cloud, 11.1 and later | | llama-3.1-nemoguard-8b-topic-control | Evaluation | - | - | Cloud, 11.1 and later | | nemoguard-jailbreak-detect | Evaluation | - | - | Cloud, 11.1 and later | ## Feature considerations - Chat model ID : For NIM model deployments, the chat model ID can be set to datarobot-deployed-llm for dynamic population, or hard-coded using the values in the table. - Playground support : Models marked as "No" in the playground support column are not supported in the playground. - Embedding/unstructured models with chat support:The following embedding/unstructured models support both direct access endpoint and chat completions endpoint: - Evaluation metrics: --- # NVIDIA AI Enterprise integration URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/genai-nvidia-integration.html > NVIDIA AI Enterprise and DataRobot provide an integrated, pre-built AI stack solution, designed to fit existing infrastructure. > [!NOTE] Premium > The use of NVIDIA Inference Microservices (NIM) in DataRobot requires access to premium features for GenAI experimentation and GPU inference. Contact your DataRobot representative or administrator for information on enabling the required features. NVIDIA AI Enterprise and DataRobot provide a pre-built AI stack solution, designed to integrate with your organization's existing DataRobot infrastructure, providing access to robust evaluation, governance, and monitoring features. This integration includes a comprehensive array of tools for end-to-end AI orchestration, accelerating your organization's data science pipelines to rapidly deploy production-grade AI applications on NVIDIA GPUs in DataRobot Serverless Compute. In DataRobot, create custom AI applications tailored to your organization's needs by selecting NVIDIA Inference Microservices (NVIDIA NIM) from a gallery of AI applications and agents. NVIDIA NIM provides pre-built and pre-configured microservices within NVIDIA AI Enterprise, designed to accelerate the deployment of generative AI across enterprises. The DataRobot moderation framework provides out-of-the-box guards, allowing you to customize your applications with simple rules, code, or models to ensure GenAI applications perform to your organization's standards. NVIDIA NeMo Guardrails are tightly integrated into DataRobot, providing an easy way to build state-of-the-art guardrails into your application. For more information on the capabilities provided by NVIDIA AI Enterprise and DataRobot, review the documentation listed below, or read the workflow summary on this page. | Task | Description | | --- | --- | | Create an inference endpoint for NVIDIA NIM | Register and deploy with NVIDIA NIM to create inference endpoints accessible through code or the DataRobot UI. | | Evaluate a text generation NVIDIA NIM in the playground | Add a deployed text generation NVIDIA NIM to a blueprint in the playground to access an array of comparison and evaluation tools. | | Use an embedding NVIDIA NIM to create a vector database | Add a registered or deployed embedding NVIDIA NIM to a Use Case with a vector database to enrich prompts in the playground with relevant context before they are sent to the LLM. | | Use NVIDIA NeMo Guardrails in a moderation framework to secure your application | Connect NVIDIA NeMo Guardrails to deployed text generation models to guard against off-topic discussions, unsafe content, and jailbreaking attempts. | | Use a text generation NVIDIA NIM in an application template | Customize application templates from DataRobot to use a registered or deployed NVIDIA NIM text generation model. | ## Create an inference endpoint for NVIDIA NIM The NVIDIA AI Enterprise integration with DataRobot starts in Registry, where you can import NIM containers from the NVIDIA AI Enterprise catalog. The resulting registered model is optimized for deployment to Console and is compatible with the DataRobot monitoring and governance framework. NVIDIA NIM provides optimized foundational models you can add to a playground in Workbench for evaluation and inclusion in agentic blueprints, embedding models used to create vector databases, and NVIDIA NeMo Guardrails used in the DataRobot moderation framework to secure your agentic application. On the Models tab in Registry, register NVIDIA NIM models from the NVIDIA GPU Cloud (NGC) gallery, selecting the model name and performance profile and reviewing the information provided on the model card. After the model is registered, deploy it to a DataRobot Serverless prediction environment. To deploy a registered model to a DataRobot Serverless environment, on the Models tab, locate and click the registered NIM, and then click the version to deploy. Then in the registered model version, you can [review the version information](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-view-manage-reg-models.html#view-version-details) and click Deploy. After the model is deployed to a DataRobot Serverless prediction environment, you can access real-time prediction snippets from the deployment's Predictions tab. The requirements for running the prediction snippet depends on the model type: text generation or unstructured. When you add a NIM to Registry in DataRobot, LLMs are imported as text generation models, allowing you to use the [Bolt-on Governance API](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/genai-chat-completion-api.html) to communicate with the deployed NIM. Other types of models are imported as unstructured models, where endpoints provided by the NIM containers are exposed to communicate with the deployed NIM. This provides the flexibility required to deploy any NIM on GPU infrastructure using DataRobot Serverless Compute. For more information, see the [documentation](https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/nvidia-ngc-nim-import.html). ## Evaluate a text generation NVIDIA NIM in the playground Optimized foundational models are available in Registry through NVIDIA NIM, imported with a text generation target type. In Workbench, you can add these optimized foundational models to a playground, where you can create, interact with, and compare LLM blueprints. For more information, see the [documentation](https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/nvidia-nim-deploy.html). ## Use an embedding NVIDIA NIM to create a vector database Embedding models are available in Registry through NVIDIA NIM. In Workbench, you can add a deployed embedding model to a Use Case as a vector database. Vector databases can optionally be used to ground the LLM responses to specific information and can be assigned to an LLM blueprint to leverage during a Retrieval-Augmented Generation (RAG) operation. For more information, see the [documentation](https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/nvidia-nim-vdb-embed.html). ## Use NVIDIA NeMo Guardrails in a moderation framework to secure your application Connect NVIDIA NeMo Guardrails to deployed text generation models to guard against off-topic discussions, unsafe content, and jailbreaking attempts. To use a deployed NVIDIA NIM with the moderation framework provided by DataRobot, first, register and deploy a NeMo model, then, when you create a custom model with the text generation target type, configure evaluation and moderation. For more information, see the [documentation](https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/nvidia-nim-evaluation-moderation.html). ## Use a text generation NVIDIA NIM in an application template Applications templates can integrate capabilities provided by NVIDIA AI Enterprise. To use this integration, you can customize a DataRobot application template to programmatically generate a GenAI Use Case built on NVIDIA NIM. With minimal edits, the following application templates can be updated to use NVIDIA NIM, selecting a registered or deployed text generation NIM: | Application template | Description | | --- | --- | | Guarded RAG Assistant | Build a RAG-powered chatbot using any knowledge base as its source. The Guarded RAG Assistant template logic contains prompt injection guardrails, sidecar models to evaluate responses, and a customizable interface that is easy to host and share. Example use cases: product documentation, HR policy documentation. | | Predictive Content Generator | Generates prediction content using prediction explanations from a classification model. The Predictive Content Generator template returns natural language-based personalized outreach. Example use cases: next-best-offer, loan approvals, and fraud detection. | | Talk to My Data Agent | Provides a talk-to-your-data experience. Upload a .csv, ask a question, and the agent recommends business analyses. It then produces charts and tables to answer your question (including the source code). This experience is paired with MLOps to host, monitor, and govern the components. | | Forecast Assistant | Leverage predictive and generative AI to analyze a forecast and summarize important factors in predictions. The Forecast Assistant template provides explorable explanations over time and supports "what-if" scenario analysis. Example use case: store sales forecasting. | To use an existing text generation model or deployment with these application templates, select one of the templates above from the Application Gallery. Then, you can make minimal modifications to the template files, locally or in a DataRobot codespace, to customize the template to use a registered or deployed NVIDIA NIM. With the template customized, you can proceed with the standard workflow outlined in the template's `README.md`. For more information, see the [documentation](https://docs.datarobot.com/en/docs/wb-apps/app-templates/index.html). --- # Import and deploy with NVIDIA NIM URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/nvidia-ngc-nim-import.html > Import, register, and deploy models with NVIDIA NIM to create an inference endpoint. Interact with inference endpoints using code or the DataRobot UI. > [!NOTE] Premium > The use of NVIDIA Inference Microservices (NIM) in DataRobot requires access to premium features for GenAI experimentation and GPU inference. Contact your DataRobot representative or administrator for information on enabling the required features. The DataRobot integration with the NVIDIA AI Enterprise Suite enables users to perform one-click deployment of NVIDIA Inference Microservices (NIM) on GPUs in DataRobot Serverless Compute. This process starts in Registry, where you can import NIM containers from the NVIDIA AI Enterprise model catalog. The registered model is optimized for deployment to Console and is compatible with the DataRobot monitoring and governance framework. NVIDIA NIM provides optimized foundational models you can add to a playground in Workbench for evaluation and inclusion in agentic blueprints, embedding models used to create vector databases, and NVIDIA NeMo Guardrails used in the DataRobot moderation framework to secure your agentic application. ## Import from NVIDIA GPU Cloud (NGC) On the Models tab in Registry, create a registered model from the gallery of available NIM models, selecting the model name and performance profile and reviewing the information provided on the model card. To import from NVIDIA NGC: 1. On theRegistry > Modelstab, next to+ Register a model, clickand thenImport from NVIDIA NGC. 2. In theImport from NVIDIA NGCpanel, on theSelect NIMtab, click a NIM in the gallery. Search the galleryTo direct your search, you canSearch, filter byPublisher, or clickSort byto order the gallery by date added or alphabetically (ascending or descending). 3. Review the model information from the NVIDIA NGC source, then clickNext. 4. On theRegister modeltab, configure the following fields and clickRegister: FieldDescriptionRegistered model name / Registered modelConfigure one of the following:Registered model name:When registering a new model, enter auniqueand descriptive name for the new registered model. If you choose a name that exists anywhere within your organization, a warning appears.Registered model:When saving as a version of an existing model, select the existing registered model you want to add a new version to.Registered version nameAutomatically populated with the model name and the wordversion. Change the version name or modify the default version name as necessary.Registered model versionAssigned automatically. This displays the expected version number of the version (e.g., V1, V2, V3) you create. This is alwaysV1when you selectRegister as a new model.Resource bundleRecommended automatically. If possible, DataRobot translates the GPU requirements for the selected model into a resource bundle. In some cases, DataRobot can't detect a compatible resource bundle. To identify a resource bundle with sufficient VRAM, review the documentation for that NIM.For Managed AI Platform installations, note that theGPU - 5XLresource bundle can be difficult to procure on-demand. If possible, consider a smaller resource bundle.NVIDIA NGC API keySelect the credential associated with your NVIDIA NGC API key. Ensure that the selected NVIDIA NGC API key exists in your DataRobot organization, as cross-organization sharing of NVIDIA NGC API keys is unsupported. In addition, due to this restriction, cross-organization sharing of global models created with NVIDIA NIM is unsupported.Optional settingsRegistered version descriptionEnter a description of the business problem this model package solves, or, more generally, describe the model represented by this version.TagsClick+ Add tagand enter aKeyand aValuefor each key-value pair you want to tag the modelversionwith. Tags added when registering a new model are applied toV1. ## Deploy the registered NVIDIA NIM After the NVIDIA NIM is registered, deploy it to a DataRobot Serverless prediction environment. To deploy a registered model to a DataRobot Serverless environment: 1. On theRegistry > Modelstab, locate and click the registered NIM, and then click the version to deploy. 2. In the registered model version, you canreview the version information, then clickDeploy. 3. In thePrediction history and service healthsection, underChoose prediction environment, verify that the correct prediction environment withPlatform: DataRobot Serverlessis selected. Change DataRobot Serverless environmentsIf the correct DataRobot Serverless environment isn't selected, clickChange. On theSelect prediction environmentpanel'sDataRobot Serverlesstab, select a different serverless prediction environment from the list. 4. Optionally, configure additional deployment settings. Then, when the deployment is configured, clickDeploy model. Enable the tracing tableTo enable thetracing tablefor the NIM deployment, ensure that youenable prediction row storagein the data exploration (or challenger) settings and configure the deployment settings required todefine an association ID. ## Make predictions with the deployed NVIDIA NIM After the model is deployed to a DataRobot Serverless prediction environment, you can access real-time prediction snippets from the deployment's Predictions tab. The requirements for running the prediction snippet depend on the model type: text generation or unstructured. **Text generation:** [https://docs.datarobot.com/en/docs/images/nxt-predict-nvidia-nim.png](https://docs.datarobot.com/en/docs/images/nxt-predict-nvidia-nim.png) **Unstructured:** [https://docs.datarobot.com/en/docs/images/nxt-predict-nvidia-nim-unstructured.png](https://docs.datarobot.com/en/docs/images/nxt-predict-nvidia-nim-unstructured.png) When you add a NIM to Registry in DataRobot, LLMs are imported as text generation models, allowing you to use the [Bolt-on Governance API](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/genai-chat-completion-api.html) to communicate with the deployed NIM. Other types of models are imported as unstructured models and endpoints provided by the NIM containers are exposed to communicate with the deployed NIM. This provides the flexibility required to deploy any NIM on GPU infrastructure using DataRobot Serverless Compute. | Target type | Supported endpoint type | Description | | --- | --- | --- | | Text generation | /chat/completions | Deployed text generation NIM models provide access to the /chat/completions endpoint. Use the code snippet provided on the Predictions tab to make predictions. | | Unstructured | /directAccess/nim/ | Deployed unstructured NIM models provide access to the /directAccess/nim/ endpoint. Modify the code snippet provided on the Predictions tab to provide a NIM URL suffix and a properly formed payload. | | Unstructured (embedding model) | Both | Deployed unstructured NIM embedding models can provide access to both the /directAccess/nim/ and /chat/completions endpoints. Modify the code snippet provided on the Predictions tab to suit your intended usage. | > [!NOTE] CSV predictions endpoint use > With an imported text generation NIM, it is also possible to make requests to the `/predictions` endpoint (accepting CSV input). For CSV input submitted to the `/predictions` endpoint, ensure that you use `promptText` as the column name for user prompts to the text generation model. If the CSV input isn't provided in this format, those predictions do not appear in the deployment's [tracing table](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-tracing.html). ### Text generation model endpoints Access the Prediction API scripting code on the deployment's Predictions > Prediction API tab. For a text generation model, the endpoint link required is the base URL of the DataRobot deployment. For more information, see the [Bolt-on Governance API](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/genai-chat-completion-api.html) documentation. **Prediction snippets for Private CA environments** For Self-managed AI Platform installations in a Private Certificate Authority (Private CA) environment, the snippets provided on the Predictions tab may need to be updated, depending on how your organization's IT team configured the Private CA environment. If your organization's Private CA environment requires modifications to the provided prediction snippet, locate the following code: | Standard Prediction API scripting code | | --- | | 1 2 3 4 5 | Update the code above, making the following changes to allow the prediction snippet to access the Private CA bundle file: | Private CA Prediction API scripting code | | --- | | 1 2 3 4 5 6 7 8 | ### Unstructured model endpoints Access the Prediction API scripting code from the deployment's Predictions > Prediction API tab. For unstructured models, endpoints provided by the NIM containers are exposed to enable communication with the deployed NIM. To determine how to construct the correct endpoint URL and send a request to a deployed NVIDIA NIM instance, refer to the documentation for the registered and deployed NIM, [listed below](https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/nvidia-ngc-nim-import.html#nim-documentation-list). > [!NOTE] Observability for direct access endpoints > Most unstructured models from NVIDIA NIM only provide access to the `/directAccess/nim/` endpoint. This endpoint is compatible with a limited set of observability features. For example, accuracy and drift tracking is not supported for the `/directAccess/nim/` endpoint. To use the Prediction API scripting code, perform the following steps and use the `send_request` function to communicate with the model: 1. Review the BASE_API_URL (line 4). This is the prefix of the endpoint. It automatically populates with the deployment's base URL. 2. Retrieve the appropriate NIM_SUFFIX (line 10). This is the suffix of the NIM endpoint. Locate this suffix in the NVIDIA NIM documentation for the deployed model . 3. Construct the request payload ( sample_payload , line 45). This request payload must be structured based on the model’s API specifications from the NVIDIA NIM documentation for the deployed model . | Prediction API scripting code | | --- | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | **Unstructured model NVIDIA NIM documentation list** For unstructured models, the required NIM endpoint can be found in the NIM documentation. The list below provides the documentation link required to assemble the `NIM_SUFFIX` and `sample_payload`. - arctic-embed-l - boltz-2 - cuopt - diffdock - genmol - llama-3.2-nv-embedqa-1b-v2 - llama-3.2-nv-rerankqa-1b-v2 - molmim - nemoguard-jailbreak-detect - nemoretriever-graphic-elements-v1 - nemoretriever-page-elements-v2 - nemoretriever-parse - nemoretriever-table-structure-v1 - nv-embedqa-e5-v5 - nv-embedqa-e5-v5-pb24h2 - nv-embedqa-mistral-7b-v2 - nv-rerankqa-mistral-4b-v3 - nvclip - openfold3 - paddleocr - proteinmpnn - rfdiffusion ### Unstructured models with text generation support Embedding models are imported and deployed as unstructured models while maintaining the ability to request chat completions. The following embedding models support both a direct access endpoint and a chat completions endpoint: - arctic-embed-l - llama-3.2-nv-embedqa-1b-v2 - nv-embedqa-e5-v5 - nv-embedqa-e5-v5-pb24h2 - nv-embedqa-mistral-7b-v2 - nvclip Each embedding NIM is deployed as an unstructured model, providing a REST interface at `/directAccess/nim/`. In addition, these models are capable of returning chat completions, so the code snippet provides a `BASE_API_URL` with the `/chat/completions` endpoint used by (structured) text generation models. To use the Prediction API scripting code, review the table below to determine how to modify the prediction snippet to access each endpoint type: | Endpoint type | Requirements | | --- | --- | | Direct access | Update the BASE_API_URL (on line 4), replacing /chat/completions with /directAccess/nim/. To structure the request payload, review the model’s API specifications from the NVIDIA NIM documentation for the deployed model. | | Chat completion | Update the DEPLOYMENT_URL (on line 13), removing /{NIM_SUFFIX} to create DEPLOYMENT_URL = BASE_API_URL. To structure the request payload, review the model’s API specifications from the NVIDIA NIM documentation for the deployed model. | | Prediction API scripting code | | --- | | 1 2 3 4 5 6 7 8 9 10 11 12 13 | --- # Add a text generation NVIDIA NIM to a Playground URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/nvidia-nim-deploy.html > Add a deployed text generation NVIDIA NIM to a blueprint in the playground to access an array of comparison and evaluation tools. > [!NOTE] Premium > The use of NVIDIA Inference Microservices (NIM) in DataRobot requires access to premium features for GenAI experimentation and GPU inference. Contact your DataRobot representative or administrator for information on enabling the required features. In a Use Case, you can add NVIDIA Inference Microservices (NIM) to the playground for prompting, comparison, and evaluation. A [playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/index.html) is a Use Case asset for creating and interacting with LLM blueprints. LLM blueprints represent the full context for what is needed to generate a response from an LLM, captured in the LLM blueprint settings. Within the playground you compare LLM blueprint responses to determine which blueprint to use in production for solving a business problem. > [!NOTE] Text generation NVIDIA NIM support in the playground > The following text generation models aren't supported in the playground: > > llama-2-70b-chat > llama-3-swallow-70b-instruct-v0.1 > llama-3-taiwan-70b-instruct > llama3-70b-instruct > llama-3.1-nemotron-ultra-253b-v1 > llama-3.2-90b-vision-instruct > mixtral-8x22b-instruct-v01 > nemotron-3-super-120b-a12b To add a deployed text generation NVIDIA NIM to the playground: 1. InWorkbench, select a Use Case from theUse Case directory, and open or create a playground on thePlaygroundstab. 2. On theLLM blueprintstab within a playground, clickCreate LLM blueprintto add a new blueprint. Then, from the playground's blueprintConfigurationpanel, in theLLMdropdown, clickAdd deployed LLM: 3. In theAdd deployed LLMdialog box, enter a deployed LLMName, then select a DataRobot deployment in theDeployment namedropdown. Enter theChat model IDto set themodelparameter for requests from the playground to the deployed LLM, then clickValidate and add. Chat model ID valueTheChat model IDcan be set todatarobot-deployed-llm, allowing the value to populate dynamically. To hard code the value, review theChat model IDtable below, locate the NVIDIA NIM you're adding to the playground, and copy the value from theChat model IDcolumn. 4. After you add a custom LLM and validation is successful, back in the blueprint'sConfigurationpanel, in theLLMdropdown, clickDeployed LLM, and then select theValidation IDof the custom model you added: 5. Configure theVector databaseandPromptingsettings, and clickSave configurationto add the blueprint to the playground. **Chat model ID list** For NIM model deployments, the Chat model ID can be set to `datarobot-deployed-llm`, allowing the value to populate dynamically. To hard code the chat model ID value, review the table below and copy the value from the Chat model ID column. | Model name | Chat model ID | | --- | --- | | codellama-13b-instruct | codellama/codellama-13b-instruct | | codellama-34b-instruct | codellama/codellama-34b-instruct | | codellama-70b-instruct | codellama/codellama-70b-instruct | | deepseek-r1-distill-llama-8b | deepseek-ai/deepseek-r1-distill-llama-8b | | deepseek-r1-distill-qwen-7b | deepseek-ai/deepseek-r1-distill-qwen-7b | | deepseek-r1-distill-qwen-14b | deepseek-ai/deepseek-r1-distill-qwen-14b | | deepseek-r1-distill-qwen-32b | deepseek-ai/deepseek-r1-distill-qwen-32b | | gemma-2-2b-instruct | google/gemma-2-2b-instruct | | gemma-2-9b-it | google/gemma-2-9b-it | | gpt-oss-120b | openai/gpt-oss-120b | | gpt-oss-20b | openai/gpt-oss-20b | | llama-2-13b-chat | meta/llama-2-13b-chat | | llama-2-7b-chat | meta/llama-2-7b-chat | | llama-3-sqlcoder-8b | defog/llama-3-sqlcoder-8b | | llama-3.1-70b-instruct | meta/llama-3.1-70b-instruct | | llama-3.1-8b-instruct | meta/llama-3.1-8b-instruct | | llama-3.1-8b-instruct | meta/llama-3.1-8b-instruct | | llama-3.1-70b-instruct | meta/llama-3.1-70b-instruct | | llama-3.1-nemotron-nano-8b-v1 | nvidia/llama-3.1-nemotron-nano-8b-v1 | | llama-3.1-nemotron-70b-instruct | nvidia/llama-3.1-nemotron-70b-instruct | | llama-3.1-swallow-70b-instruct-v0.1 | tokyotech-llm/llama-3.1-swallow-70b-instruct-v0.1 | | llama-3.2-1b-instruct | meta/llama-3.2-1b-instruct | | llama-3.2-3b-instruct | meta/llama-3.2-3b-instruct | | llama-3.2-11b-vision-instruct | meta/llama-3.2-11b-vision-instruct | | llama-3.3-70b-instruct | meta/llama-3.3-70b-instruct | | llama-3.3-nemotron-super-49b-v1 | nvidia/llama-3.3-nemotron-super-49b-v1 | | llama-3.3-nemotron-super-49b-v1.5 | nvidia/llama-3-3-nemotron-super-49b-v1-5 | | llama3-8b-instruct | meta/llama3-8b-instruct | | mistral-7b-instruct-v0.3 | mistralai/mistral-7b-instruct-v0.3 | | mistral-nemo-12b-instruct | mistral-nemo-12b-instruct | | mistral-nemo-minitron-8b-8k-instruct | nv-mistralai/mistral-nemo-minitron-8b-8k-instruct | | mixtral-8x7b-instruct-v01 | mistralai/mixtral-8x7b-instruct-v0.1 | | nemotron-3-nano | nvidia/nemotron-3-nano | | nemotron-3-super-120b-a12b | nvidia/nemotron-3-super-120b-a12b | | phi-3-mini-4k-instruct | microsoft/phi-3-mini-4k-instruct | | qwen-2.5-7b-instruct | qwen/qwen-2.5-7b-instruct | | starcoder2-7b | bigcode/starcoder2-7b | --- # Use NVIDIA NeMo Guardrails with DataRobot moderation URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/nvidia-nim-evaluation-moderation.html > Connect NVIDIA NeMo Guardrails to deployed text generation models to guard against off-topic discussions, unsafe content, and jailbreaking attempts. > [!NOTE] Premium > The use of NVIDIA Inference Microservices (NIM) in DataRobot requires access to premium features for GenAI experimentation and GPU inference. NVIDIA NeMo Guardrails are a premium feature. Contact your DataRobot representative or administrator for information on enabling this feature. > > Additional feature flags: Enable Moderation Guardrails ( Premium), Enable Global Models in the Model Registry ( Premium), Enable Additional Custom Model Output in Prediction Responses DataRobot provides out-of-the-box guardrails and lets you customize your applications with simple rules, code, or models. Use NVIDIA Inference Microservices (NIM) to connect NVIDIA NeMo Guardrails to text generation models in DataRobot, allowing you to guard against off-topic discussions, unsafe content, and jailbreaking attempts. The following NVIDIA NeMo Guardrails are available as a NIM and can be implemented using the associated evaluation metric type: | Model name | Evaluation metric type | | --- | --- | | llama-3.1-nemoguard-8b-topic-control | Stay on topic for input / Stay on topic for output | | llama-3.1-nemoguard-8b-content-safety | Content safety | | nemoguard-jailbreak-detect | Jailbreak | In addition, DataRobot provides access to NeMo Evaluator metrics (LLM Judge, Context Relevance, Response Groundedness, Topic Adherence, Agent Goal Accuracy, Response Relevancy, Faithfulness) in the [Configure evaluation and moderation](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-configure-evaluation-moderation.html) panel of the NeMo metrics section. Those metrics require a NeMo evaluator workload deployment (created via the Workload API) and are listed in the NeMo metrics section of that panel. This page covers NVIDIA NeMo Guardrails (Stay on topic, Content safety, Jailbreak) via NIM deployments. ## Use a deployed NIM with NVIDIA NeMo guardrails To use a deployed `llama-3.1-nemoguard-8b-topic-control` NVIDIA NIM with the topic control evaluation metrics, register and deploy the NVIDIA NeMo Guardrail. Once you have created a custom model with the text generation target type, configure the topic control evaluation metric. To select and configure NVIDIA NeMo Guardrails for topic control: 1. In theWorkshop, open theAssembletab of a custom model with theText Generationtarget type andassemble a model, eithermanually from a custom model you created outside DataRobotorautomatically from a model built in a Use Case's LLM playground. When you assemble a text generation model with moderations, ensure you configure any requiredruntime parameters(for example, credentials) orresource settings(for example, public network access). Finally, set theBase environmentto a moderation-compatible environment, such as[GenAI] Python 3.12 with Moderations: Resource settingsDataRobot recommends creating the LLM custom model using larger resource bundles with more memory and CPU resources. 2. After you've configured the custom model's required settings, navigate to theEvaluation and moderationsection and clickConfigure: 3. In theConfigure evaluation and moderationpanel, locate the metrics tagged withNVIDIA NeMo guardrailorNVIDIAand select the metric you want to use. Evaluation metricRequiresDescription1Content safetyA deployed NIM modelllama-3.1-nemoguard-8b-content-safetyimported fromNVIDIA GPU Cloud (NGC) Catalog.Classify prompts and responses as safe or unsafe; return a list of any unsafe categories detected.2JailbreakA deployed NIM modelnemoguard-jailbreak-detectimported fromNVIDIA GPU Cloud (NGC) Catalog.Classify jailbreak attempts using NemoGuard JailbreakDetect.3Stay on topic for inputsNVIDIA NeMo guardrails configurationUse NVIDIA NeMo Guardrails to provide topic boundaries, ensuring prompts are topic-relevant and do not use blocked terms.4Stay on topic for outputNVIDIA NeMo guardrails configurationUse NVIDIA NeMo Guardrails to provide topic boundaries, ensuring responses are topic-relevant and do not use blocked terms. 4. On theConfigure evaluation and moderationpage, set the following fields based on the selected metric: Topic controlContent safetyJailbreakFieldDescriptionNameEnter a descriptive name for the metric you're configuring.Apply forStay on topic for input is applied to the prompt. Stay on topic for output is applied to the response.LLM typeSet the LLM type to NIM.NIM DeploymentSelect an NVIDIA NIM deployment. For more information, seeImport and deploy with NVIDIA NIM.CredentialsSelect a DataRobot API key from the list. Credentials are defined on theCredentials managementpage.Files(Optional) Configure the NeMo files. Next to a file, clickto modify the NeMo guardrails configuration files. In particular, updateprompts.ymlwith allowed and blocked topics andblocked_terms.txtwith the blocked terms, providing rules for NeMo guardrails to enforce. Theblocked_terms.txtfile is shared between the input and output topic control metrics; therefore, modifyingblocked_terms.txtin the input metric modifies it for the output metric and vice versa. Only two topic control metrics can exist in a custom model, one for input and one for output.FieldDescriptionNameEnter a descriptive name for the metric you're configuring.Apply forApply content safety to both the prompt and the response.Deployment nameIn the list, locate the name of thellama-3.1-nemoguard-8b-content-safetymodelregistered and deployed in DataRobotand click the deployment name.FieldDescriptionNameEnter a descriptive name for the metric you're configuring.Apply toApply jailbreak to the prompt.Deployment nameIn the list, locate the name of thenemoguard-jailbreak-detectmodelregistered and deployed in DataRobotand click the deployment name. 5. In theModerationsection, withConfigure and apply moderationenabled, for each evaluation metric, set the following: FieldDescriptionModeration methodSelectReportorReport and block.Moderation messageIf you selectReport and block, you can optionally modify the default message. 6. After configuring the required fields, clickAddto save the evaluation and return to the evaluation selection page. Then,select and configure another metric, or clickSave configuration. The guardrails you selected appear in theEvaluation and moderationsection of theAssembletab. After you add guardrails to a text generation custom model, you can [test](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-test-custom-model.html), [register](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-register-cus-models.html), and [deploy](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-deploy-models.html) the model to make predictions in production. After making [predictions](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-predictions/index.html), you can view the evaluation metrics on the [Custom metrics](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-custom-metrics.html) tab and prompts, responses, and feedback (if configured) on the [Data exploration](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-data-exploration.html) tab. --- # Use an embedding NVIDIA NIM to create a vector database URL: https://docs.datarobot.com/en/docs/agentic-ai/genai-integrations/nvidia-nim-vdb-embed.html > Add a deployed embedding NVIDIA NIM to a Use Case with a vector database to enrich prompts in the playground with relevant context before they are sent to the LLM. > [!NOTE] Premium > The use of NVIDIA Inference Microservices (NIM) in DataRobot requires access to premium features for GenAI experimentation and GPU inference. Contact your DataRobot representative or administrator for information on enabling the required features. The NVIDIA Inference Microservices (NIM) available through Registry include embedding models. A deployed embedding model can be added to a Use Case, creating a collection of unstructured text that is broken into chunks, with embeddings generated for each chunk. Both the chunks and embeddings are stored in the vector database and are available for retrieval. Vector databases can optionally be used to ground the LLM responses to specific information and can be assigned to an LLM blueprint to leverage during a RAG operation. The role of the vector database is to enrich the prompt with relevant context before it is sent to the LLM. Each embedding NVIDIA NIM available is listed below: - arctic-embed-l - llama-3.2-nv-embedqa-1b-v2 - nv-embedqa-e5-v5 - nv-embedqa-e5-v5-pb24h2 - nv-embedqa-mistral-7b-v2 - nvclip ## Create a vector database with a registered embedding NIM After you register an embedding NIM, you can add it to a vector database. DataRobot handles the deployment process automatically. To create a vector database with a registered embedding NVIDIA NIM: 1. On theRegistry > Modelstab, next to+ Register a model, clickand thenImport from NVIDIA NGC. 2. In theImport from NVIDIA NGCpanel, on theSelect NIMtab, click an embedding NIM in the gallery. Search the galleryTo direct your search for an embedding model, you canSearch, filter byPublisher, or clickSort byto order the gallery by date added or alphabetically (ascending or descending). 3. Review the model information from the NVIDIA NGC source, then clickNext. 4. On theRegister modeltab, configure the following fields and clickRegister: FieldDescriptionRegistered model name / Registered modelConfigure one of the following:Registered model name:When registering a new model, enter auniqueand descriptive name for the new registered model. If you choose a name that exists anywhere within your organization, a warning appears.Registered model:When saving as a version of an existing model, select the existing registered model you want to add a new version to.Registered version nameAutomatically populated with the model name and the wordversion. Change the version name or modify the default version name as necessary.Registered model versionAssigned automatically. This displays the expected version number of the version (e.g., V1, V2, V3) you create. This is alwaysV1when you selectRegister as a new model.Resource bundleRecommended automatically. If possible, DataRobot translates the GPU requirements for the selected model into a resource bundle. In some cases, DataRobot can't detect a compatible resource bundle. To identify a resource bundle with sufficient VRAM, review the documentation for that NIM.NVIDIA NGC API keySelect the credential associated with your NVIDIA NGC API key.Optional settingsRegistered version descriptionEnter a description of the business problem this model package solves, or, more generally, describe the model represented by this version.TagsClick+ Add tagand enter aKeyand aValuefor each key-value pair you want to tag the modelversionwith. Tags added when registering a new model are applied toV1. 5. After the registered model builds, navigate toWorkbenchand open a Use Case. 6. In a Use Case, on theVector databasestab, either: With an existing vector databasesWithout an existing vector databaseIf you have already added one or more vector databases to the Use Case, Click the+ Add vector databasebutton in the upper right.If you haven't added a vector database to the Use Case before, clickCreate vector databasein the center of the page. 7. On theCreate vector databasepanel, enter a descriptiveName. Then, in theData sourcedropdown, select from the data sources associated with the Use Case or clickAdd datato add new data from the Data Registry. 8. In theEmbedding modeldropdown, click the embedding NIM you registered. Then, configure thevector databaseText chunkingsettingsand clickCreate vector database. The selected embedding model is deployed toConsolewhen you create the vector database. If necessary, this process creates a newprediction environmentfor NIM embeddings. After creating a vector database, you can [manage](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#manage-vector-databases) and [version](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html) it, or [add it to an LLM in the playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#add-a-vector-database) to inform responses. ## Create a vector database with a deployed embedding NIM If you've already registered and deployed an embedding NIM, you can add it to a vector database as a deployed embedding model. To create a vector database with a registered and deployed embedding NVIDIA NIM: 1. In a Use Case, on theVector databasestile, either: With an existing vector databasesWithout an existing vector databaseIf you have already added one or more vector databases to the Use Case, Click the+ Add vector databasebutton in the upper right.If you haven't added a vector database to the Use Case before, clickCreate vector databasein the center of the page. 2. On theCreate vector databasepanel, enter a descriptiveName. Then, in theData sourcedropdown, select from the data sources associated with the Use Case or clickAdd datato add new data from the Data Registry. 3. In theEmbedding modeldropdown, clickAdd deployed embedding model. 4. On the next page, configure the following settings to add the NVIDIA NIM embedding model, then clickValidate and add: FieldDescriptionNameEnter a descriptive name for the embedding model you're creating.Deployment nameIn the list, locate the name of the NVIDIA NIM embedding modelregistered and deployed in DataRobotand click the deployment name.Prompt column nameEnterinputas the prompt column name.Response column nameEnterresultas the response column name. Validation processThe validation process can take a few minutes. A notification appears when the process starts and if it succeeds or fails. 5. After the validation of the deployed embedding model succeeds, open theEmbedding modelmenu, then, underDeployed embedding models, select the NVIDIA NIM embedding model. 6. Configure thevector databaseText chunkingsettings, then clickCreate vector database. After creating a vector database, you can [manage](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#manage-vector-databases) and [version](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html) it, or [add it to an LLM in the playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#add-a-vector-database) to inform responses. --- # Agentic AI URL: https://docs.datarobot.com/en/docs/agentic-ai/index.html > Create, connect, and test agentic workflows, integrate tools, and implement evaluation metrics. Create, connect, and test agentic workflows, integrate tools, and implement evaluation metrics. > [!TIP] New to the DataRobot CLI? > Agentic workflows use the [DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/index.html) ( `dr`) to set up templates, run local development, and deploy. If you haven't installed or used the CLI yet, see [Getting started with the DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html). - Build¶ Build and deploy agents from templates leveraging multi-agent frameworks. - Evaluate¶ Chat with agents, compare responses, and implement evaluation metrics. - Deploy¶ Register and deploy agentic artifacts and configure evaluation and moderation guardrails in Registry. - Monitor & govern¶ Monitor deployment details, usage stats, metrics, logs, and moderation events for agentic artifact deployments in Console. - DataRobot CLI¶ Use the DataRobot CLI to set up templates, run local development, and deploy. - Agent Assist¶ An interactive AI assistant optimized for the development of AI agents. - MCP¶ Integrate tools using MCP servers and connect IDEs and chat clients to your DataRobot MCP server. - Vector databases¶ Create vector databases, LLM blueprints, and GenAI deployments. - Prompt management¶ Create, manage, and share prompts that can be called into agentic workflows. - RAG workflows¶ Create LLM blueprints, moderations, tests, and deployments. - Code walkthroughs¶ Use code walkthroughs to implement GenAI. - NVIDIA AI Enterprise integration¶ Use NVIDIA NIM and NeMo to accelerate GenAI application development. - Feature considerations¶ Review LLM availability, GenAI capabilities, considerations, and troubleshooting tips. --- # Add a text generation NVIDIA NIM to a playground URL: https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/add-deployed-nvidia-nim.html > Add a deployed text generation NVIDIA NIM to a blueprint in the playground to access an array of comparison and evaluation tools. > [!NOTE] Premium > The use of NVIDIA Inference Microservices (NIM) in DataRobot requires access to premium features for GenAI experimentation and GPU inference. Contact your DataRobot representative or administrator for information on enabling the required features. In a Use Case, you can add NVIDIA Inference Microservices (NIM) to the playground for prompting, comparison, and evaluation. A [playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/index.html) is a Use Case asset for creating and interacting with LLM blueprints. LLM blueprints represent the full context for what is needed to generate a response from an LLM, captured in the LLM blueprint settings. Within the playground you compare LLM blueprint responses to determine which blueprint to use in production for solving a business problem. > [!NOTE] Text generation NVIDIA NIM support in the playground > The following text generation models aren't supported in the playground: > > llama-2-70b-chat > llama-3-swallow-70b-instruct-v0.1 > llama-3-taiwan-70b-instruct > llama3-70b-instruct > llama-3.1-nemotron-ultra-253b-v1 > llama-3.2-90b-vision-instruct > mixtral-8x22b-instruct-v01 > nemotron-3-super-120b-a12b To add a deployed text generation NVIDIA NIM to the playground: 1. InWorkbench, select a Use Case from theUse Case directory, and open or create a playground on thePlaygroundstab. 2. On theLLM blueprintstab within a playground, clickCreate LLM blueprintto add a new blueprint. Then, from the playground's blueprintConfigurationpanel, in theLLMdropdown, clickAdd deployed LLM: 3. In theAdd deployed LLMdialog box, enter a deployed LLMName, then select a DataRobot deployment in theDeployment namedropdown. Enter theChat model IDto set themodelparameter for requests from the playground to the deployed LLM, then clickValidate and add. Chat model ID valueTheChat model IDcan be set todatarobot-deployed-llm, allowing the value to populate dynamically. To hard code the value, review theChat model IDtable below, locate the NVIDIA NIM you're adding to the playground, and copy the value from theChat model IDcolumn. 4. After you add a custom LLM and validation is successful, back in the blueprint'sConfigurationpanel, in theLLMdropdown, clickDeployed LLM, and then select theValidation IDof the custom model you added: 5. Configure theVector databaseandPromptingsettings, and clickSave configurationto add the blueprint to the playground. **Chat model ID list** For NIM model deployments, the Chat model ID can be set to `datarobot-deployed-llm`, allowing the value to populate dynamically. To hard code the chat model ID value, review the table below and copy the value from the Chat model ID column. | Model name | Chat model ID | | --- | --- | | codellama-13b-instruct | codellama/codellama-13b-instruct | | codellama-34b-instruct | codellama/codellama-34b-instruct | | codellama-70b-instruct | codellama/codellama-70b-instruct | | deepseek-r1-distill-llama-8b | deepseek-ai/deepseek-r1-distill-llama-8b | | deepseek-r1-distill-qwen-7b | deepseek-ai/deepseek-r1-distill-qwen-7b | | deepseek-r1-distill-qwen-14b | deepseek-ai/deepseek-r1-distill-qwen-14b | | deepseek-r1-distill-qwen-32b | deepseek-ai/deepseek-r1-distill-qwen-32b | | gemma-2-2b-instruct | google/gemma-2-2b-instruct | | gemma-2-9b-it | google/gemma-2-9b-it | | gpt-oss-120b | openai/gpt-oss-120b | | gpt-oss-20b | openai/gpt-oss-20b | | llama-2-13b-chat | meta/llama-2-13b-chat | | llama-2-7b-chat | meta/llama-2-7b-chat | | llama-3-sqlcoder-8b | defog/llama-3-sqlcoder-8b | | llama-3.1-70b-instruct | meta/llama-3.1-70b-instruct | | llama-3.1-8b-instruct | meta/llama-3.1-8b-instruct | | llama-3.1-8b-instruct | meta/llama-3.1-8b-instruct | | llama-3.1-70b-instruct | meta/llama-3.1-70b-instruct | | llama-3.1-nemotron-nano-8b-v1 | nvidia/llama-3.1-nemotron-nano-8b-v1 | | llama-3.1-nemotron-70b-instruct | nvidia/llama-3.1-nemotron-70b-instruct | | llama-3.1-swallow-70b-instruct-v0.1 | tokyotech-llm/llama-3.1-swallow-70b-instruct-v0.1 | | llama-3.2-1b-instruct | meta/llama-3.2-1b-instruct | | llama-3.2-3b-instruct | meta/llama-3.2-3b-instruct | | llama-3.2-11b-vision-instruct | meta/llama-3.2-11b-vision-instruct | | llama-3.3-70b-instruct | meta/llama-3.3-70b-instruct | | llama-3.3-nemotron-super-49b-v1 | nvidia/llama-3.3-nemotron-super-49b-v1 | | llama-3.3-nemotron-super-49b-v1.5 | nvidia/llama-3-3-nemotron-super-49b-v1-5 | | llama3-8b-instruct | meta/llama3-8b-instruct | | mistral-7b-instruct-v0.3 | mistralai/mistral-7b-instruct-v0.3 | | mistral-nemo-12b-instruct | mistral-nemo-12b-instruct | | mistral-nemo-minitron-8b-8k-instruct | nv-mistralai/mistral-nemo-minitron-8b-8k-instruct | | mixtral-8x7b-instruct-v01 | mistralai/mixtral-8x7b-instruct-v0.1 | | nemotron-3-nano | nvidia/nemotron-3-nano | | nemotron-3-super-120b-a12b | nvidia/nemotron-3-super-120b-a12b | | phi-3-mini-4k-instruct | microsoft/phi-3-mini-4k-instruct | | qwen-2.5-7b-instruct | qwen/qwen-2.5-7b-instruct | | starcoder2-7b | bigcode/starcoder2-7b | --- # Build LLM blueprints URL: https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html > Configure, copy, and register LLM blueprints. LLM blueprints represent the full context for what is needed to generate a response from an LLM; the resulting output can then be compared within the playground. To create an LLM blueprint, select that action from either the LLM blueprints tab or, if it is your first blueprint, the playground welcome screen. Clicking the create button brings you to the configuration and chatting tools: | | Element | Description | | --- | --- | --- | | (1) | Playground summary | Displays a summary of the playground owner and creation date. | | (2) | Chat history | Displays a record of prompts sent to this LLM blueprint, as well as an option to start a new chat. | | (3) | Configuration panel | Provides access to the configuration selections available for creating an LLM blueprint. | | (4) | Prompt entry | Accepts prompts to begin chatting with the LLM blueprint; the configuration must be saved before the entry is activated. | You can also create an LLM blueprint by [copying an existing blueprint](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#copy-llm-blueprint). ## Select an LLM The configuration panel is where you define the LLM blueprint. From here: - Add an LLM. - Set the configuration . - Select or add a vector database . - Set the prompting strategy . DataRobot offers a variety of [preloaded LLMs](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html), with availability dependent on your cluster and account type. Alternatively, you can [add a deployed LLM](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#add-a-deployed-llm) to the playground, which, when validated, is added to the Use Case and available to all associated playgrounds. To add an LLM and begin configuring the LLM blueprint, click Select LLM. A scrollable list of available LLMs appears in a panel to the left. See the [reference documentation](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-reference.html) for brief descriptions of each LLM's characteristics. Use the filtering tools to limit the list so that it shows only LLMs that meet your criteria: - Use one of the multiselect filter options: ProviderFamilyRegionNote that if you have added a deployed LLM,DataRobotbecomes an entry in theProviderfilter. - Toggle to show or hide deprecated LLMs. LLMs marked with aDeprecatedbadge indicate that the end of support date for the LLM falls within 90 days. Retirement dates are set by the provider and are subject to change. While not listed in the selection panel, any LLM blueprints that were built using an LLM that has subsequently been retired are marked in theblueprints tabwith aRetiredbadge. See a list of upcoming deprecations and retired LLMs on theLLM availabilitypage. - Enter a string in the search bar to match all LLMs, from all providers, for that string. Once you select the LLM from the left panel, click Save: ### Add a deployed LLM To select a [custom deployed LLM in DataRobot](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html#create-and-deploy-a-generative-custom-model), click Create LLM blueprint to add a new blueprint to the playground. Then, from the playground's blueprint Configuration panel, click Select LLM and Add deployed LLM: In the Add deployed LLM dialog box, enter a deployed LLM Name, then select a DataRobot deployment in the Deployment name dropdown. If the selected deployment supports the [Bolt-on Governance API](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html), configure the following: **Bolt-on Governance API supported:** When adding a deployed LLM that implements the `chat` function, the playground uses the [Bolt-on Governance API](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#chat) as the preferred communication method. Enter the Chat model ID to set the `model` parameter for requests from the playground to the deployed LLM, then click Validate and add: > [!NOTE] Chat model ID > When using the Bolt-on Governance API with a deployed LLM blueprint, see [LLM availability](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html) for the recommended values of the `model` parameter. Alternatively, specify a reserved value, `datarobot-deployed-llm`, to let the LLM blueprint select the relevant model ID automatically when calling the LLM provider's services. [https://docs.datarobot.com/en/docs/images/add-deployed-llm-bp-fields-api.png](https://docs.datarobot.com/en/docs/images/add-deployed-llm-bp-fields-api.png) To disable the Bolt-on Governance API and use the Prediction API instead, delete the [chatfunction (or hook)](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#chat) from the custom model and redeploy the model. **Bolt-on Governance API not supported:** When adding a deployed LLM that doesn't support the [Bolt-on Governance API](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#chat), the playground uses the Prediction API as the preferred communication method. Enter the Prompt column name and Response column name defined when you [created the custom LLM in the workshop](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-genai-monitoring.html#create-and-deploy-a-generative-custom-model) (for example, `promptText` and `resultText`), then click Validate and add: [https://docs.datarobot.com/en/docs/images/add-deployed-llm-bp-fields.png](https://docs.datarobot.com/en/docs/images/add-deployed-llm-bp-fields.png) To enable the Bolt-on Governance API, modify the custom model code to use the [chatfunction (or hook)](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#chat) and redeploy the model. After you add a custom LLM and validation is successful, back in the blueprint's Configuration panel, in the LLM dropdown, click Deployed LLM, and then select the Validation ID of the custom model you added: Finally, you can configure the [Vector database](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#add-a-vector-database) and [Prompting](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#prompting) settings, and click Save configuration to add the blueprint to the playground. ## Set the configuration Selecting a base LLM exposes some or all of the following configuration options (listed alphabetically). Available options are dependent on the LLM selection. Note that the default value of the option can be dependent on the provider. | Setting | Description | | --- | --- | | Max completion tokens | The maximum number of tokens allowed in the completion. The combined count of this value and prompt tokens must be below the model’s maximum context size, where prompt token count is comprised of system prompt, user prompt, recent chat history, and vector database citations. | | Number of most likely tokens | An integer ranging from 0 to 20 that specifies the number of most likely tokens to consider when generating the response. A value of 0 means all tokens are considered. Note: The Return log probabilities setting must be enabled to use this parameter. | | Random seed | If specified, the system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed, and you should refer to the system_fingerprint response parameter to monitor changes in the backend. | | Reasoning effort | For OpenAI o-series only (not GPT-4o models) constrains the reasoning effort for reasoning models—either low, medium, or high. Reducing the reasoning effort can result in faster responses and fewer tokens "spent." | | Return log probabilities | If enabled, the log probabilities of each returned output token are returned in the response. Note that this setting is required in order to use the Number of most likely tokens parameter. | | Stop sequences | Up to four strings that, when generated by the model, will stop the generation process. This is useful for controlling the output format or preventing unwanted text from being included in the response. The triggering sequence will not be shown in the returned text. | | Temperature | The temperature controls the randomness of model output. Enter a value (range is LLM-dependent), where higher values return more diverse output and lower values return more deterministic results. A value of 0 may return repetitive results. Temperature is an alternative to Top P for controlling the token selection in the output (see the example below). | | Token frequency penalty | A penalty ranging from -2.0 to 2.0 that is applied to tokens based on their frequency in the context. Positive values increase the penalty, discouraging frequent tokens, while negative values decrease the penalty, allowing for more frequent use of those tokens. | | Token presence penalty | A penalty ranging from -2.0 to 2.0 that is applied to tokens that are already present in the context. Positive values increase the penalty, therefore discouraging repetition, while negative values decrease the penalty, allowing for more repetition. | | Token selection probability cutoff (Top P) | Top P sets a threshold that controls the selection of words included in the response based on a cumulative probability cutoff for token selection. For example, 0.2 considers only the top 20% probability mass. Higher numbers return more diverse options for outputs. Top P is an alternative to Temperature for controlling the token selection in the output (see "Temperature or Top P?" below). | **Temperature or Top P?** Consider prompting: “To make the perfect ice cream sundae, top 2 scoops of vanilla ice cream with… “. The desired responses for a suggested next word might be hot fudge, pineapple sauce, and bacon. To increase the probability of what is returned: - For bacon, set Temperature to the maximum value and leave top P at the default. Setting Top P with a high Temperature, increases the probability of fudge and pineapple and reduces the probability of bacon. - For hot fudge, set Temperature to 0. Each base LLM has default configuration settings. As a result, the only required selection before starting to chat is to choose the LLM. ### Add a vector database From the Vector database tab, you can optionally select a [vector database](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html). The selection identifies a database comprised of a collection of [chunks](https://docs.datarobot.com/en/docs/reference/glossary/index.html#chunking) of unstructured text and corresponding text embeddings for each chunk, indexed for easy retrieval. Vector databases are not required for prompting but are used for providing relevant data to the LLM to generate the response. Add a vector database to a playground to experiment with metrics and test responses. The following table describes the fields of the Vector database tab: The dropdown | Field | Description | | --- | --- | | Vector database | Lists all vector databases available in the Use Case (and therefore accessible for use by all of that Use Case's playgrounds). If you select the Add vector database option, the new vector database you add will become available to other LLM blueprints although you must change the LLM blueprint configuration to apply them. | | Vector database version | Select the version of the vector database that the LLM will use. The field is prepopulated with the version you were viewing when you created the playground. Click Vector database version to leave the playground and open the vector database details page. | | Information | Reports configuration information for the selected version. | | Retriever | Sets the method, neighbor chunk inclusion, and retrieval limits that the LLM uses to return chunks from the vector database. | | Retrieval mode | Sets the approach for chunk retrieval, either similarity (most semantically similar) or Maximum Marginal Relevance (relevant yet diverse). | #### Retriever methods The retriever you select defines how the LLM blueprint searches through, and retrieves, the most relevant chunks from the vector database. They inform which information is provided to the language model. Select one of the following methods: | Method | Description | | --- | --- | | Single-Lookup Retriever | Performs a single vector database lookup for each query and returns the most similar documents. | | Conversational Retriever (default) | Rewrites the query based on chat history, returning context-aware responses. In other words, this retriever functions similarly to the Single-Lookup Retriever with the addition of query rewrite as its first step. | | Multi-Step Retriever | Performs the following steps when returning results:Rewrites the query to be chat history-aware and retrieves documents for that query.Creates five new queries based on the result of step 1.Queries the vector database for each of the five new queries.Merges and deduplicates the results, creating one set of returned documents, which are then used for the query. | **Deep dive: Retrievers and context** It is important to understand the interaction between context state used in LLM prompting and retriever selection. The following table provides guidance and explanation for each retriever with both no context (each query is independent) and context aware (chat history is taken into account). Each retriever is described above. | Retriever | No context | Context-aware | | --- | --- | --- | | | Process to return a response | | | None | Prompts the LLM with a query. | Prompts the LLM with a query plus the query history. | | Single-Lookup Retriever | Searches the vector database based on the query.Retrieves relevant documents.Prompts the LLM with query plus documents. | Searches the vector database based on the query. Retrieves relevant documents. Prompts the LLM with query, documents, and history. | | Conversational Retriever | Because there is no history to get a standalone query, use the Single-Lookup Retriever instead. | Prompt the LLM with query plus history to get a rewritten standalone query. Search the vector database with the standalone query and retrieve documents.Prompt the LLM with the standalone query plus documents. | | Multi-Step Retriever | Because there is no history, this retriever is not recommended; however, you can potentially use the five new search queries to retrieve better-fitting documents for the final response. | Prompt the LLM with query plus history to get a rewritten standalone query.Search the vector database with the standalone query to retrieve documents.Prompt the LLM with the standalone query plus documents to get multiple new search queries that complement the initial query.Search the vector database for each new search query; merge and deduplicate retrieved documents.Prompt the LLM with the standalone query plus documents. | Use Add Neighbor Chunks to control whether to add neighboring chunks within the vector database to the chunks that the similarity search retrieves. When enabled, the retriever returns `i`, `i-1`, and `i+1` (for example, if the query retrieves chunk number 42, chunks 41 and 43 are also retrieved). Notice also that only the primary chunk has a similarity score. This is because the neighbor chunks are added, not calculated, as part of the response. Also known as context window expansion or context enrichment, this technique includes surrounding chunks adjacent to the retrieved chunk to provide more complete context. Some reasons to enable this include: - A single chunk may be cut off mid-sentence or may miss important context. - Related information might span multiple chunks. - The response might require context from surrounding or chunks. Enter a value to set the Retrieval limits, which control the number of returned documents. The value you set for Top K (nearest neighbors) instructs the LLM on how many relevant chunks to retrieve from the vector database. Chunk selection is based on [similarity scores](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html#rouge-scores). Consider: - Larger values provide more comprehensive coverage but also require more processing overhead and may include less relevant results. - Smaller values provide more focused results and faster processing, but may miss relevant information. Max tokens specifies: - The maximum size (in tokens) of each text chunk extracted from the dataset when building the vector database. - The length of the text that is used to create embeddings. - The size of the citations used in RAG operations. #### Retrieval mode Select a retrieval mode to set how matching chunks are selected from the vector database and returned for a query. Set either Similarity or Maximum Marginal Relevance. - Similarity returns the most semantically similar top N chunks, ranked by how close they are in vector space. These are the most similar chunks, regardless of whether they are repetitive or redundant. The Similarity method is faster but can return very similar or duplicate chunks, which can unnecessarily "spend" the context window. - Maximal Marginal Relevance (MMR) balances relevance and diversity, returning results that are both similar to the query and different from each other. If selected, setMaximal marginal relevance lambdato control the balance between relevance (1.0) and diversity (0.0). **Deep dive: Setting lambda** What setting lambda: | Value | Effect | | --- | --- | | 1.0 | The search only cares about relevance. You may get very similar or repetitive results. | | 0.0 | The search only cares about diversity. Results may be varied but may also be less relevant. | | 0.5 (default) | Returns a balanced mix of relevance and diversity. | When adjusting the value, turning lambda: - Up says “Give me the best matches, even if they repeat.” - Down says “Give me different perspectives, even if some are less perfect matches.” Use higher lambda values (0.7–0.9) when precision matters; use lower values (0.3–0.6) when exploration, summaries, or varied context are the goal. ### Set prompting strategy The prompting strategy is where you configure context (chat history) settings and optionally add a system prompt. #### Set context state There are two states of context. They control whether chat history is sent with the prompt to include relevant context for responses. | State | Description | | --- | --- | | Context-aware | When sending input, previous chat history is included with the prompt. This state is the default. | | No context | Sends each prompt as independent input, without history from the chat. | > [!NOTE] Note > Consider the context state and how it functions in conjunction with the selected [retriever method](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#retriever). See [context-aware chatting](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#context-aware-chatting) for more information. **Deep dive: How context is allocated** If some combination of the user prompt, vector database chunks, or conversation history are too long to fit into the context window, DataRobot must apply some form of truncation to work within the confines of the assigned context size. Context management is applied in the playground because chat history and citations often compete for the limited context window, which is particularly relevant for small context models. The calculation is as follows: 1. Modelcontext sizeis determined. If this information is not available (e.g., a BYO LLM with no context metadata available and the context size was not indicated in the BYO LLM configuration), DataRobot assumes the context to be 4096 tokens. 2. The number of user-requested output tokens is determined. 3. From the initially determined context size (step 1), DataRobot subtracts output tokens (step 2), the system prompt, and the user prompt. That value results in the available context budget that can be filled with either chat history or citations. 4. If there are only citations but no chat history to consider, DataRobot adds citations in the order of most relevant to least relevant, until the context window is filled. 5. If there are no citations but only chat history, DataRobot adds chat history in the order of newest to oldest messages until the context window is filled. 6. If there are both citations and chat history, DataRobot fills the available context using a heuristically chosen ratio of 4:1, citations:history. (This is based on the assumption that citations are 4x more important than history.) That is, it fills 80% of the context using the citation rules (from step 4) and 20% of the context using the history rules in (from step 5). #### Set system prompt The system prompt, an optional field, is a "universal" prompt prepended to all individual prompts for this LLM blueprint and can be up to 5 million characters. It instructs and formats the LLM response. The system prompt can impact the structure, tone, format, and content that is created during the generation of the response. For more detailed information, see [prompt management](https://docs.datarobot.com/en/docs/agentic-ai/prompt-mgmt/create-prompts.html). See an example of system prompt application in the [LLM blueprint comparison documentation](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/compare-llm.html#example-comparison). ## Working with LLM blueprints The left panel of the playground is like a file cabinet of the playground's assets—a list of configured blueprints and a record of the comparison chat history. You can also create a new LLM blueprint, the process described above, from this area. ### LLM blueprints tab The LLM blueprints tab lists all LLM blueprints configured within the playground. It is from this panel that you select LLM blueprints—up to three if you are doing a [comparison](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/compare-llm.html). As you select an LLM blueprint via the checkbox, it becomes available in the middle comparison panel. Click the star next to an LLM blueprint name to "favorite" it, which you can later filter on. #### Display controls Use the controls to modify the LLM blueprint listing: The Filter option controls which blueprints are listed in the panel, either by base LLM or status: The small number to the right of the Filter label indicates how many blueprints are displayed as a result of any (or no) applied filtering. Sort by controls the ordering of the blueprints. It is additive, meaning that it is applied on top of any filtering or grouping in place.Group by, also additive, arranges the display by the selected criteria. Labels indicate the group "name" with numbers to indicate the number of member blueprints. #### Actions for LLM blueprints The actions available for an LLM blueprint can be accessed from the Actions menu next to the name in the left-hand LLM blueprints tab or from LLM blueprint actions in a selected LLM blueprint. | Option | Description | | --- | --- | | Configure LLM blueprint | From the LLM blueprints tab only. Opens the configuration settings for the selected blueprint for further tuning. | | Edit LLM blueprint | Provides a modal for changing the LLM blueprint name. Changing the name saves the new name and all saved settings. If any settings have not been saved, they will revert to the last saved version. | | Copy to new LLM blueprint | Creates a new LLM blueprint from all saved settings of the selected blueprint. | | Send to the workshop | Sends the LLM blueprint to Registry where it is added to the workshop. From there it can be deployed as a custom model. | | Delete LLM blueprint | Deletes the LLM blueprint. | ### Chats tab The Chats tab provides access any previous prompts and subsequent made from the playground to the selected LLM blueprint. It also a place from which you can start a new chat. For full details, see either [single blueprint](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html) or [comparison](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/compare-llm.html) chatting. ### Copy LLM blueprint You can make a copy of an existing LLM blueprint to inherit the settings. Using this approach makes sense when you want to compare slightly different blueprints, duplicate a blueprint for which you are not the owner in a shared playground, or replace a soon-to-expire LLM. Make a copy in one of two ways: **From an existing blueprint:** In the left-hand panel, click the Actions menu and select Copy to new LLM blueprint to create a copy that inherits the settings of the parent blueprint. [https://docs.datarobot.com/en/docs/images/create-bp-7.png](https://docs.datarobot.com/en/docs/images/create-bp-7.png) The new LLM blueprint opens for further configuration. **From any open LLM blueprint:** Choose LLM blueprint actions and choose Copy to new LLM blueprint. [https://docs.datarobot.com/en/docs/images/create-bp-8.png](https://docs.datarobot.com/en/docs/images/create-bp-8.png) ### Change LLM blueprint configuration To change the configuration of an LLM blueprint, choose Configure LLM blueprint from the actions menu in the LLM blueprints tab. The LLM blueprint configuration and chat history display. Change any of the [configuration settings](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#set-the-configuration) and Save configuration When you make changes to an LLM blueprint, the chat history associated with it, if the configuration is context-aware, is also saved. All the prompts within a chat persist through LLM blueprint changes: - When you submit a prompt, the history included is everything within the most recent chat context. - If you switch the LLM blueprint to No context , each prompt is its own chat context. - If you switch back to Context-aware , that starts a new chat context within the chat. Note that chats in the configuration view are separate from chats in the Comparison view—the histories don't mingle. --- # Multiple LLM blueprint chat comparison URL: https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/compare-llm.html > The place to add LLM blueprints to the playground for comparison, submit prompts to these LLM blueprints, and evaluate the rendered responses. [https://docs.datarobot.com/en/docs/images/playground-compare-1.png](https://docs.datarobot.com/en/docs/images/playground-compare-1.png) The playground's LLM blueprint tab allows you to: - View all LLM blueprints in the playground. - Filter, group, and sort the LLM blueprint list. - View the playground's chat history . - Create and compare chats (LLM responses). To use the comparison: 1. Create two or more LLM blueprints in the playground. 2. From the LLM blueprints tab, select up to three LLM blueprints for comparison . 3. Send a prompt from the central prompting window. Each of the blueprints receives the prompt and responds, allowing you to compare responses. > [!NOTE] Note > You can only do comparison prompting with workflows that you created. To see the results of prompting another user’s LLM blueprint or agentic flow in a shared Use Case, copy the LLM blueprint or connect to the registered agentic flow. You can chat with the same settings applied. This is intentional behavior because prompting impacts chat history, which can impact the responses that are generated. However, you can provide response feedback on the creator's asset to assist development. #### Example comparison The following example compares three LLM blueprints, each with the same settings except using a different [system prompt](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#set-system-prompt) to influence the style of response. First test the system prompt, when configuring the LLM blueprint, for example: `Describe the novel Game of Thrones`. 1. Enter the system promptAnswer using emojis. 2. Enter the system promptAnswer in the style of a news headline. 3. Enter the system promptAnswer as a haiku. See also the [note on system prompts](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/compare-llm.html#consider-system-prompts). ## Compare LLM blueprints To compare multiple LLM blueprints: 1. From theLLM blueprinttab, check the box next to each blueprint you want to compare. 2. Send a prompt (Describe DataRobot). Each LLM blueprint responds in their configured style: 3. Try different prompts (Describe a fish taco, for example) to identify the LLM that best suits the business use case. ### Interpret results One obvious way to compare LLM blueprints is to read the results and see if the responses of one seem more on point. Another helpful measure is to review the evaluation metrics that are returned. Consider: - Which LLM blueprint has the lowest latency? Is that status consistent across prompt/response sets? - Which metrics are excluded from some LLM blueprints and why? - How do results change when you toggle context awareness ? - Do the LLM blueprints use the citations to inform the response effectively? - Do the they respect the system prompt such that the response has the requested tone, format, succinctness, etc.? ### Change selected LLM blueprints You can add blueprints to the comparison at any time, although the maximum allowed for comparison at one time is three. To add an LLM blueprint, select the checkbox to the left of its name. If three are already selected, remove a current selection first. The comparison panel retrieves the comparison history. Because responses have not been returned for the new LLM blueprint, DataRobot provides a button to initiate that action. Click Generate to include the new results. ### Consider system prompts Note that system prompts are not guaranteed to be followed completely, and that wording is very important. For example, consider the comparison using the prompt `Answer using emojis` (EmojiGPT) and `Answer using only emojis` (OnlyEmojiGPT): ### Chats tab A comparison chat groups together one or more comparison prompts, often across multiple blueprints. Use the Chats tab to access any previous comparison prompts made from the playground or start a new chat. In this way, you can select up to three LLM blueprints, query them, and then swap out for other LLM blueprints to send the same prompts and compare results. > [!NOTE] Note > In some cases, you will see a chat named Default chat. This entry contains any chats made in the playground before the new playground functionality was released in April, 2024. If no chats were initiated, the default chat is empty. If the playground was created after that date, the default chat isn't present but an New chat is available for prompting. Rename or delete chats from the entry name. --- # Deploy an LLM from the playground URL: https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/deploy-llm.html > LLM blueprints and all their associated settings are registered in Registry and can be deployed and monitored with Console. Use an LLM [playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-overview.html) in a [Use Case](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/usecases/index.html) to [create an LLM blueprint](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html). Set the blueprint configuration, specifying the base LLM and, optionally, a system prompt and vector database. After testing and tuning the responses, the blueprint is ready for registration and deployment. You can create a text generation custom model by sending the LLM blueprint to [Registry's workshop](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/index.html). The generated custom model automatically implements the [Bolt-on Governance API](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#chat), which is particularly useful for building conversational applications. Follow the steps below to add the LLM blueprint to the workshop: 1. In a Use Case, from thePlaygroundstab, click the playground containing the LLM you want to register as a blueprint. 2. In the playground,compare LLMsto determine which LLM blueprint to send to the workshop, then do either of the following: 3. In theSend to the workshopmodal, do the following, then clickSend to the workshop: 4. ClickSend to the workshop. In the lower-right corner of the LLM playground, notifications appear as the LLM is queued and registered. When notification of the registration's completion appears, clickGo to the workshop: The LLM blueprint opens in the Registry's workshop as a custom model with theText Generationtarget type: 5. On theAssembletab, in theRuntime Parameterssection, configure the key-value pairs required by the LLM, including the LLM service's credentials and other details. To add these values, click the edit iconnext to the available runtime parameters. Premium: DataRobot LLM gatewayIf your organization has access to the DataRobot LLM gateway, you don't need to configure any credentials. Confirm that theENABLE_LLM_GATEWAY_INFERENCEruntime parameter is present and set toTrue. If necessary, configure thePROMPT_COLUMN_NAME(the default column name ispromptText), and then skip to the next step. You can alsomake requests to the DataRobot LLM gateway. To configureCredentialtypeRuntime Parameters, first, add the credentials required for the LLM you're deploying to theCredentials Managementpage of the DataRobot platform: Microsoft-hosted LLMsAmazon-hosted LLMsGoogle-hosted LLMsForMicrosoft-hosted LLMs, use the following:Credential type: API Token (notAzure)Runtime Parameters:KeyDescriptionOPENAI_API_KEYSelect theAPI Tokencredential, created on theCredentials Managementpage, for the Azure OpenAI LLM API endpoint.OPENAI_API_BASEEnter the URL for the Azure OpenAI LLM API endpoint.OPENAI_API_DEPLOYMENT_IDEnter the name of the Azure OpenAI deployment of the LLM, chosen when deploying the LLM to your Azure environment. For more information, see the Azure OpenAI documentation on how toDeploy a model. The default deployment name suggested by DataRobot matches the ID of the LLM in Azure OpenAI (for example, gpt-35-turbo). Modify this parameter if your Azure OpenAI deployment is named differently.OPENAI_API_VERSIONEnter the Azure OpenAI API version to use for this operation, following the YYYY-MM-DD or YYYY-MM-DD-preview format (for example, 2023-05-15). For more information on the supported versions, see theAzure OpenAI API reference documentation.PROMPT_COLUMN_NAMEEnter the prompt column name from the input .csv file. The default column name is promptText.ForAmazon-hosted LLMs, use the following:Credential type: AWSRuntime Parameters:KeyDescriptionAWS_ACCOUNTSelect anAWScredential, created on theCredentials Managementpage, for the AWS account.AWS_REGIONEnter the AWS region of the AWS account. The default is us-west-1.PROMPT_COLUMN_NAMEEnter the prompt column name from the input .csv file. The default column name is promptText.ForGoogle-hosted LLMs, use the following:Credential type: Google Cloud Service AccountRuntime Parametersare:KeyDescriptionGOOGLE_SERVICE_ACCOUNTSelect aGoogle Cloud Service Accountcredential created on theCredentials Managementpage.GOOGLE_REGIONEnter the GCP region of the Google service account. The default is us-west-1.PROMPT_COLUMN_NAMEEnter the prompt column name from the input .csv file. The default column name is promptText. 6. In theSettingssection, ensureNetwork accessis set toPublic. 7. After you complete the custom model assembly configuration, you cantest the modelorcreate new versions. DataRobot recommends testing custom LLMs before deployment. 8. Next, clickRegister a model,provide the registered model or version details, then clickRegister modelagain to add the custom LLM to Registry. The registered model version opens on theRegistry > Modelstab. 9. From theModelstab, in the upper-right corner of the registered model version panel, clickDeployandconfigure the deployment settings. For more information on the deployment functionality available for generative models, seeMonitoring support for generative models. For more information on this process, see the [playground deployment considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#playground-deployment-considerations). --- # RAG workflows URL: https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/index.html > Create and compare LLM blueprints, configure metrics, and compare LLM blueprint responses before deployment. > [!NOTE] Availability information > DataRobot's GenAI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. Create and compare LLM blueprints, configure metrics, and compare LLM blueprint responses before deployment. See the [list of considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html) to keep in mind when working with DataRobot GenAI. | Topic | Description | | --- | --- | | Playground overview | Navigate through the GenAI playground. | | Create LLM blueprints | Create LLM blueprints and fine-tune results. | | Add a text generation NVIDIA NIM to an LLM blueprint | Premium feature. Add a deployed text generation NVIDIA NIM to a playground. | | Chatting with LLM blueprints | Use single and comparison chats. | | Compare LLM blueprints | Compare LLM blueprint chat results. | | Add evaluation metrics to an LLM blueprint | Configure evaluation and moderation guardrails for LLM blueprints. | | Deploy LLMs to production | Register LLMs for deployment. | --- # Use LLM evaluation tools URL: https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-eval-metrics.html > Configure evaluation and moderation guardrails for LLM blueprints in a playground. > [!NOTE] Premium > LLM evaluation tools are a premium feature. Contact your DataRobot representative or administrator for information on enabling this feature. The playground's LLM evaluation tools include evaluation metrics and datasets, aggregated metrics, compliance tests, and tracing. The LLM evaluation metric tools include: | LLM evaluation tool | Description | | --- | --- | | Evaluation metrics | Report an array of performance, safety, and operational metrics for prompts and responses in the playground and define moderation criteria and actions for any configured metrics. | | Evaluation datasets | Upload or generate the evaluation datasets used to evaluate an LLM blueprint through evaluation dataset metrics, aggregated metrics, and compliance tests. | | Aggregated metrics | Combine evaluation metrics across many prompts and responses to evaluate an LLM blueprint at a high level, as only so much can be learned from evaluating a single prompt or response. | | Compliance tests | Combine an evaluation metric and dataset to automate the detection of compliance issues with pre-configured or custom compliance testing. | | Tracing table | Trace the execution of LLM blueprints through a log of all components and prompting activity used in generating LLM responses in the playground. | ## Configure evaluation metrics With evaluation metrics, you can configure an array of performance, safety, and operational metrics. Configuring these metrics lets you define moderation methods to intervene when prompts and responses meet the moderation criteria you set. This functionality can help detect and block prompt injection and hateful, toxic, or inappropriate prompts and responses. It can also help identify hallucinations or low-confidence responses and safeguard against the sharing of personally identifiable information (PII). > [!TIP] Evaluation deployment metrics > Many evaluation metrics connect a playground-built LLM to a deployed guard model. These guard models make predictions on LLM prompts and responses and then report the predictions and statistics to the playground. If you intend to use any of the Evaluation Deployment type metrics—Custom Deployment, PII Detection, Prompt Injection, Emotions Classifier, and Toxicity—deploy the [required guard models from the Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-directory/nxt-global-models.html) to make predictions on the LLM's prompts or responses. Selecting and configuring evaluation metrics in an LLM playground depends on whether you have already configured LLM blueprints: **With LLM blueprints:** If you've added one or more LLM blueprints to the playground, with or without blueprints selected click the Evaluation tile on the side navigation bar: [https://docs.datarobot.com/en/docs/images/playground-eval-metrics-blueprint.png](https://docs.datarobot.com/en/docs/images/playground-eval-metrics-blueprint.png) **Without LLM blueprints:** If you haven't added any blueprints to the playground, in the Evaluate with metrics section, click Open configure evaluation to configure metrics before adding an LLM blueprint: [https://docs.datarobot.com/en/docs/images/playground-eval-metrics-no-blueprint.png](https://docs.datarobot.com/en/docs/images/playground-eval-metrics-no-blueprint.png) In both cases, the Evaluation and moderation page opens to the Metrics tab. Certain metrics are enabled by default. Note, however, that to report a metric value for Citations and ROUGE-1, you must first associate a vector database with the LLM blueprint. ### Create a new configuration To create a new evaluation metric configuration for the playground: 1. In the upper-right corner of theEvaluation and moderationpage, clickConfigure metrics: 2. In theConfigure evaluation and moderationpanel, click an evaluation metric and then configure the metric settings. The metrics, requirements, and settings are outlined in the tables below. Evaluation metricRequiresDescriptionCostLLM cost settingsCalculate the cost of generating the LLM response using a default or custom LLM, currency, input cost-per-token, and output cost-per-token values. The cost calculation also includes the cost of citations. For more information, seeCost metric settings.Custom DeploymentCustom deploymentUse an existing deployment to evaluate and moderate your LLM (supported target types: regression, binary classification,multiclass, text generation).Emotions ClassifierEmotions Classifier deploymentClassify prompt or response text by emotion.PII DetectionPresidio PII Detection deploymentDetect Personally Identifiable Information (PII) in text using the Microsoft Presidio library.Prompt InjectionPrompt Injection Classifier deploymentDetects input manipulations, such as overwriting or altering system prompts, intended to modify the model's output.ToxicityToxicity Classifier deploymentClassifies content toxicity to apply moderation techniques, safeguarding against dissemination of harmful content.ROUGE-1Vector databaseRecall-Oriented Understudy for Gisting Evaluationcalculates the similarity between the response generated from an LLM blueprint and the documents retrieved from the vector database.CitationsVector databaseReports the documents retrieved by an LLM when prompting a vector database.All tokensN/ATracks the number of tokens associated with the input to the LLM, output from the LLM, and/or retrieved text from the vector database.Prompt tokensN/ATracks the number of tokens associated with the input to the LLM.Response tokensN/ATracks the number of tokens associated with the output from the LLM.Document tokensN/ATracks the number of tokens associated with the retrieved text from the vector database.LatencyN/AReports the response latency of the LLM blueprint.CorrectnessLLM, evaluation dataset, vector databaseUses either a provided or synthetically generated set of prompts or prompt and response pairs to evaluate aggregated metrics against the provided reference dataset. The Correctness metric uses the LlamaIndexCorrectness Evaluator.FaithfulnessLLM, vector databaseMeasures if the LLM response matches the source to identify possible hallucinations. The Faithfulness metric uses the LlamaIndexFaithfulness Evaluator.Topic control metricsStay on topic for inputsNIM deployment ofllama-3.1-nemoguard-8b-topic-control, NVIDIA NeMo guardrails configurationUses NVIDIA NeMo Guardrails to provide topic boundaries, ensuring prompts are topic-relevant and do not use blocked terms.Stay on topic for outputNIM deployment ofllama-3.1-nemoguard-8b-topic-control, NVIDIA NeMo guardrails configurationUses NVIDIA NeMo Guardrails to provide topic boundaries, ensuring responses are topic-relevant and do not use blocked terms. Global models for evaluation metric deploymentsThe deployments required for PII detection, prompt injection detection, emotion classification, and toxicity classification are available asglobal models in Registry Multiclass custom deployment metric limitsMulticlasscustom deployment metrics can have:Up to10classes defined in theMatcheslist for moderation criteria.Up to100class names in the guard model. Depending on the evaluation metric (or evaluation metric type) selected, as well as whether you are using the LLM gateway, different configuration options are required: SettingDescriptionGeneral settingsNameEnter a unique name if adding multiple instances of the evaluation metric.Apply toSelect one or both ofPromptandResponse, depending on the evaluation metric. Note that when you selectPrompt, it's the user prompt, not the final LLM prompt, that is used for metric calculation. This field is only configurable for metrics that apply to both the prompt and the response.Custom Deployment, PII Detection, Prompt Injection, Emotions Classifier, and Toxicity settingsDeployment nameFor evaluation metrics calculated by a guard model deployment, select the custom model deployment.Custom Deployment settingsInput column nameThis name is defined by the custom model creator. Forglobal models created by DataRobot, the default input column name istext. If the guard model for the custom deployment has themoderations.input_column_namekey valuedefined, this field is populated automatically.Output column nameThis name is defined by the custom model creator, and needs to refer to the target column for the model. The target name is listed on the deployment'sOverviewtab (and often has_PREDICTIONappended to it). You can confirm the column names byexporting and viewing the CSV data from the custom deployment. If the guard model for the custom deployment has themoderations.output_column_namekey valuedefined, this field is populated automatically.Correctness and Faithfulness settingsLLMSelect an LLM for evaluation.Topic control metric settingsLLM TypeSelectAzure OpenAI,OpenAI, orNIM. For theAzure OpenAILLM type, additionally enter anOpenAI API base URLandOpenAI API Deployment; forNIMenter aNIM deployment(thellama-3.1-nemoguard-8b-topic-controltopic-control model). If you use the LLM gateway, the default experience, DataRobot-supplied credentials are provided. You can, however, clickChange credentialsto provide your own authentication.FilesFor theStay on topicevaluations, next to a file, clickto modify the NeMo guardrails configuration files. In particular, updateprompts.ymlwith allowed and blocked topics andblocked_terms.txtwith the blocked terms, providing rules for NeMo guardrails to enforce. Theblocked_terms.txtfile is shared between the input and output topic control metrics; therefore, modifyingblocked_terms.txtin the input metric modifies it for the output metric and vice versa. Only two topic control metrics can exist in a playground, one for input and one for output.Moderation settingsConfigure and apply moderationEnable this setting to expand theModerationsection and define the criteria that determines when moderation logic is applied. Cost metric settingsFor theCostmetric, in the row for eachLLMtype, define aCurrencyand theInputandOutputcost incurrency amount / tokens amountformat, then clickAdd:TheCostmetric doesn't include theModerationsection toConfigure and apply moderation. 3. In theModerationsection, withConfigure and apply moderationenabled, for each evaluation metric, set the following: SettingDescriptionModeration criteriaIf applicable, set the threshold settings evaluated to trigger moderation logic. For numeric metrics (int or float), you can useless than,greater than, orequals towith a value of your choice. For binary metrics (for example, Stay on topic for inputs), useequals to0 or 1. For the Emotions Classifier, selectMatchesorDoes not matchand define a list of classes (emotions) to trigger moderation logic.Moderation methodSelectReportorReport and block.Moderation messageIf you selectReport and block, you can optionally modify the default message. 4. After configuring the required fields, clickAddto save the evaluation and return to the evaluation selection page. The metrics you selected appear on theConfigure evaluation and moderationpanel, in theConfiguration summarysidebar. 5. Select and configure another metric, or clickSave configuration. The metrics appear on theEvaluation and moderationpage. If any issues occur during metric configuration, an error message appears below the metric to provide guidance on how to fix the issue. Metric configuration processingError message example ### Change credentials DataRobot provides credentials for [available LLMs](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html) using the LLM gateway. With Azure OpenAI and OpenAI LLM types, you can, however, use your own credentials for authentication. Before proceeding, define user-specified credentials on the [credentials management](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html#credentials-management) page. To change credentials for either Stay on topic for inputs or Stay on topic for output, choose the LLM type and click Change credentials. **LLM type: Azure OpenAI:** Provide the Azure OpenAI API deployment and the OpenAI API base URL. Then, from the dropdown, select the set of credentials to apply. [https://docs.datarobot.com/en/docs/images/change-metric-creds-azure.png](https://docs.datarobot.com/en/docs/images/change-metric-creds-azure.png) **LLM type: OpenAI:** From the dropdown, select the set of credentials to apply. [https://docs.datarobot.com/en/docs/images/change-metric-creds-openai.png](https://docs.datarobot.com/en/docs/images/change-metric-creds-openai.png) **LLM type: NIM:** Select the NIM deployment (for example, the topic-control model). Credentials are typically provided via the deployment configuration. To revert to DataRobot-provided credentials, click Revert credentials. ### Manage configured metrics To edit or remove a configured evaluation metric from the playground: 1. In the upper-right corner of theEvaluation and moderationpage, clickConfigure metrics: 2. In theConfigure evaluation and moderationpanel, in theConfiguration summarysidebar, click the edit iconor the delete icon: 3. If you click edit, you can re-configure the settings for that metric and clickUpdate: ### Copy a metric configuration To copy an evaluation metrics configuration to or from an LLM playground: 1. In the upper-right corner of theEvaluation and moderationpage, next toConfigure metrics, click, and then clickCopy configuration. 2. In theCopy evaluation and moderation configurationmodal, select one of the following options: From an existing playgroundTo an existing playgroundTo a new playgroundIf you selectFrom an existing playground, choose toAdd to existing configurationorReplace existing configurationand then select a playground toCopy from.If you selectTo an existing playground, choose toAdd to existing configurationorReplace existing configurationand then select a playground toCopy to.If you selectTo a new playground, enter aNew playground name. 3. Select if you want toInclude evaluation datasets, and then clickCopy configuration. > [!NOTE] Duplicate evaluation metrics > Selecting Add to existing configuration can result in duplicate metrics, except in the case of NeMo Stay on topic for inputs and Stay on topic for output. Only two topic control metrics can exist, one for input and one for output. ### View metrics in a chat The metrics you configure and add to the playground appear on the LLM responses in the playground. Click the down arrow to open the metric panel for more details. From this panel, click Citation to view the prompt, response, and a list of citations in the Citation dialog box. You can also provide positive or negative feedback for the response. In addition, if a response from the LLM is blocked by the configured moderation criteria and strategy, you can click Show response to view the blocked response: > [!TIP] Multiple moderation messages > If a response from the LLM is blocked by multiple configured moderations, the message for each triggered moderation appears, replacing the LLM response, in the chat. If you configure descriptive moderation messages, this can provide a complete list of reasons for blocking the LLM response. ## Add evaluation datasets To enable evaluation dataset metrics and aggregated metrics, add one or more evaluation datasets to the playground. The dataset must be a CSV file, in the Data Registry, and have at least one text or categorical column. > [!WARNING] When using evaluation datasets with an LLM that includes a vector database > Ensure that no column name exists in both the evaluation dataset and the vector database. If any column name exists in both, those columns are treated as metadata filters, and vector database results are excluded from prompts when you run evaluation dataset aggregation. This situation is most common when the vector database was built from a CSV source document. 1. To select and configure evaluation metrics in an LLM playground, do either of the following: 2. On theEvaluation and moderationpage, click theEvaluation datasetstab to view any existing datasets, then, clickAdd evaluation dataset, and select one of the following methods: Dataset addition methodDescriptionAdd evaluation datasetIn theAdd evaluation datasetpanel, select an existing dataset from theData Registrytable, or upload a new dataset:ClickUploadto register and select a new dataset from your local filesystem.ClickUpload from URL, then, enter theURLfor a hosted dataset and clickAdd.After you select a dataset, in theEvaluation dataset configurationsidebar, define thePrompt column nameandResponse (target) column name, and clickAdd evaluation dataset.Generate synthetic dataEnter aDataset name, select anLLM, setVector database,Vector databaseversion, and theLanguageto use when creating synthetic data. Then, clickGenerate data. For more information, seeGenerate synthetic datasets. 3. After you add an evaluation dataset, it appears on theEvaluation datasetstab of theEvaluation and moderationpage, where you can clickOpen datasetto view the data. You can also click theActions menutoEdit evaluation datasetorDelete evaluation dataset: **Q: How are synthetic datasets generated?** When you add evaluation dataset metrics, DataRobot can use a [vector database](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html) to generate synthetic datasets, composed of prompt and response pairs, to evaluate your LLM blueprint against. Synthetic datasets are generated by accessing the selected vector database, clustering the vectors, pulling a representative chunk from each cluster, and prompting the selected LLM to generate 100 question and answer pairs based on the document(s). When you configure the synthetic evaluation dataset settings and click Generate, two events occur sequentially: 1. A placeholder dataset is registered to the Data Registry with the required columns (questionandanswer), containing 65 rows and 2 columns of placeholder data (for example,Record for synthetic prompt answer 0,Record for synthetic prompt answer 1, etc.). 2. The selected LLM and vector database pair generates question and answer pairs, and is added to the Data Registry as a second version of the synthetic evaluation dataset created in step 1.The generation time depends on the selected LLM. To generate high-quality and diverse questions, DataRobot runs cosine similarity-based clustering. Similar chunks are grouped into the same cluster and each cluster generates a single question and answer pair. Therefore, if a vector database includes many similar chunks, they'll be grouped into a much smaller number of clusters. When this happens, the number of pairs generated is much lower than the number of chunks in the vector database. ## Add aggregated metrics When a playground includes more than one metric, you can begin creating aggregate metrics. Aggregation is the act of combining metrics across many prompts and/or responses, which helps to evaluate a blueprint at a high level (only so much can be learned from evaluating a single prompt/response). Aggregation provides a more comprehensive approach to evaluation. Aggregation either averages the raw scores, counts the boolean values, or surfaces the number of categories in a multiclass model. DataRobot does this by generating the metrics for each individual prompt/response and then aggregating using one of the methods listed, based on the metric. To configure aggregated metrics: 1. In a playground, clickConfigure aggregationbelow the prompt input: Aggregation job run limitOnly one aggregated metric job can run at a time. If an aggregation job is currently running, theConfigure aggregationbutton is disabled and the "Aggregation job in progress; try again when it completes" tooltip appears. 2. On theGenerate aggregated metricspanel, select metrics to calculate in aggregate and configure theAggregate bysettings. Then, enter a newChat name, select anEvaluation dataset(to generate prompts in the new chat), and select theLLM blueprintsfor which the metrics should be generated. These fields are pre-populated based on the current playground: Evaluation dataset selectionIf you select an evaluation dataset metric, likeCorrectness, you must use the evaluation dataset used to create that evaluation dataset metric. After you complete theMetrics selectionandConfigurationsections, clickGenerate metrics. This results in a new chat containing all associated prompts and responses: Aggregated metrics are run against an evaluation dataset, not individual prompts in a standard chat. Therefore, you can only view aggregated metrics in the generatedaggregated metrics chat, added to the LLM blueprint'sAll Chatslist (on the LLM's configuration page). Aggregation metric calculation for multiple blueprintsIf many LLM blueprints are included in the metric aggregation request, aggregated metrics are computed sequentially, blueprint-by-blueprint. 3. Once an aggregated chat is generated, you can explore the resulting aggregated metrics, scores, and related assets on theAggregated metricstab. You can filter byAggregation method,Evaluation dataset, andMetric: In addition, clickCurrent configurationto compare only those metrics calculated for the blueprint configuration currently defined in theLLMtab of theConfigurationsidebar. View related assetsFor each metric in the table, you can clickEvaluation datasetandAggregated chatto view the corresponding asset contributing to the aggregated metric. 4. Returning to the LLMBlueprints comparisonpage, you can now open theAggregated metricstab to view a leaderboard comparing LLM blueprint performance for the generated aggregated metrics: ## Configure compliance testing Combine an evaluation metric and an evaluation dataset to automate the detection of compliance issues through test prompt scenarios. ### Manage compliance testing from the Evaluation tab When you manage compliance testing on the Evaluation tab, you can view pre-defined compliance tests, create and manage custom tests, or modify pre-defined tests to suit your organization's testing requirements. To view all available compliance tests: 1. On the side navigation bar click theEvaluationtile. 2. Click theCompliance teststab. On theCompliance teststab, you can view all the compliance tests available, both DataRobot and custom (if present). The table contains columns for theTestname,Provider, andConfiguration(number of evaluations and evaluation datasets). #### View and customize DataRobot compliance tests Use the View option to review and, optionally: - Customize DataRobot pre-configured compliance tests, including changing the LLM for certain tests. - Manage custom compliance tests. In the table on the Compliance tests tab, click View to open and review any of the compliance tests in which DataRobot is the Provider: | Compliance test | Description | Assessing LLM | Based on | | --- | --- | --- | --- | | Bias Benchmark | Runs LLM question/answer sets that test for bias along eight social dimensions. | GPT-4o | AI Verify Foundation | | Jailbreak | Applies testing scenarios to evaluate whether built-in safeguards enforce LLM jailbreaking compliance standards. | Customizable | jailbreak_llms | | Completeness | Determines whether the LLM response is supplying enough information to comprehensively answer questions. | GPT-4o | Internal | | Personally Identifiable Information (PII) | Determines whether the LLM response contains PII included in the prompt. | Customizable | Internal | | Toxicity | Applies testing scenarios to evaluate whether built-in safeguards enforce toxicity compliance standards. For more information, see the explicit and offensive content warning. | Customizable | Hugging Face | | Japanese Bias Benchmark | Runs LLM question/answer sets in Japanese that test for bias along five social dimensions. | GPT-4o | AI Verify Foundation | > [!WARNING] Explicit and offensive content warning > The [public evaluation dataset for toxicity testing](https://huggingface.co/datasets/allenai/real-toxicity-prompts) contains explicit and offensive content. It is intended to be used exclusively for the purpose of eliminating such content from external models and applications. Any other use is strictly prohibited. **Bias tests** Bias testing is based on the following moonshot-data datasets from AI Verify Foundation: - bbq-lite-age-disamb.json - bbq-lite-disability-status-disamb.json - bbq-lite-gender-disamb.json - bbq-lite-nationality-disamb.json - bbq-lite-race-ethnicity-disamb.json - bbq-lite-religion-disamb.json - bbq-lite-sexual-orientation-disamb.json - bbq-lite-ses-disamb.json - JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language ModelsandJBBQ_data When viewing a compliance test from the list, you can review the individual evaluations run as part of the compliance testing process. For all tests, you can review the Name, Metric, LLM, Evaluation dataset, Pass threshold, and Number of prompts. If the test shows `-` in the LLM field, it uses GPT-4o. The following tests default to GPT-4o as the LLM but can be customized: - Jailbreak - Toxicity - PII Use a selected DataRobot test as the foundation for a custom test as follows: 1. SelectViewfor the test you want to modify. 2. ClickCustomize test. 3. From theCreate custom testmodal, modify any of the individual evaluations for the compliance test settings. NoteIn addition to the default metrics and evaluation datasets, you can select any evaluation metrics implemented by a deployed binary classification sidecar model and any evaluation datasets added to the Use Case. SettingDescriptionNameA descriptive name for the custom compliance test.DescriptionA description of the purpose of the compliance test (this is pre-populated when you modify an existing DataRobot test).Test pass thresholdThe minimum percentage (0-100%) of individual evaluations that must pass for the test as a whole to pass.Evaluations*NameThe name of the individual metric.MetricThe criteria to match against.LLMThe LLM used to assess the response. This field is enabled for Jailbreak, Toxicity, and PII compliance tests. All others use GPT-4o.Evaluation datasetThedatasetused for calculating metrics.Pass thresholdThe minimum percentage of responses that must pass for the evaluation to pass.Number of promptsThe number of rows from the dataset used to perform the evaluation.Add evaluationCreate additional evaluations.Copy from existing testCopy the individual evaluations from an existing compliance test. * Use the API-only process,expected_response_column, to validate a sidecar model with metrics you are introducing. It compares the LLM response with an expected response, similar to the pre-providedexact_matchmetric. 4. After you customize the compliance test settings, clickAdd. The new test appears in the table on theCompliance teststab. #### Create custom compliance tests To create a custom compliance test: 1. At the top or bottom of theCompliance teststab, clickCreate custom compliance test. Create compliance tests from anywhere in the Evaluations tabWhen theEvaluationtab is open, you can clickCreate custom compliance testfrom anywhere, not just theCompliance teststab. 2. In theCreate custom testpanel, configure the following settings: SettingDescriptionNameA descriptive name for the custom compliance test.DescriptionA description of the purpose of the compliance test (this is pre-populated when you modify an existing DataRobot test).Test pass thresholdThe minimum percentage (0-100%) of individual evaluations that must pass for the test as a whole to pass.Evaluations*NameThe name of the individual metric.MetricThe criteria to match against.LLMThe LLM used to assess the response. This field is enabled for Jailbreak, Toxicity, and PII compliance tests. All others use GPT-4o. You must set theMetricbefore setting this field.Evaluation datasetThedatasetused for calculating metrics.Pass thresholdThe minimum percentage of responses that must pass for the evaluation to pass.Number of promptsThe number of rows from the dataset used to perform the evaluation.Add evaluationCreate additional evaluations.Copy from existing testCopy the individual evaluations from an existing compliance test. 3. After you configure the compliance test settings, clickAdd. The new test appears in the table on theCompliance teststab. #### Manage custom compliance tests To manage custom compliance tests, locate tests with Custom as the Provider, and choose a management action: - Click the edit icon, then, in theEdit custom testpanel, update the compliance test configuration and clickSave. - Click the delete icon, then clickYes, delete testto remove the test from all playgrounds in the Use Case. ### Run compliance testing from the playground When you perform compliance testing on the Playground tile, you can run the pre-defined compliance tests without modification, create custom tests, or modify the pre-defined tests to suit your organization's testing requirements. To access compliance from the playground tests to run, modify, or create a test: 1. On thePlaygroundtile, in theLLM blueprintslist, click the LLM blueprint you want to test, or, select up to three blueprints for comparison. Access compliance tests from the blueprints comparison pageIf you have two or more LLM blueprints selected, you can click theCompliance teststab from theBlueprints comparisonpage to run compliance tests for multiple LLM blueprints and compare the results. For more information, seeCompare compliance test results 2. In the LLM blueprint, click theCompliance teststab to create or run tests. If you have not run tests before, you receive a message saying no compliance test results are available. If you have run a test before, test results are listed. In either case, clickRun testto open the test panel. 3. TheRun testpanel opens to a list of pre-configured DataRobot compliance tests and custom tests you've created. 4. When you select a compliance test from theAll testslist, you can view the individual evaluations run as part of the compliance testing process. For each test, you can review theName,Metric,Evaluation dataset,Pass threshold, andNumber of prompts. 5. Next, run an existing test, create and run a custom test, or manage custom tests. #### Run existing compliance tests To run an existing, configured compliance test: 1. On theRun testpanel, from theAll testslist, select an availableDataRobotorCustomtest. 2. After selecting a test, clickRun. 3. The test appears on theCompliance teststab with aRunning...status. Cancel a running testIf you need to cancel a test with theRunning...status, clickDelete test results. #### Create and run custom compliance tests To create and run a custom or modified compliance test: 1. On theRun testpanel, from theAll testslist: 2. On theCustom testpanel, configure the following settings: SettingDescriptionNameA descriptive name for the custom compliance test.DescriptionA description of the purpose of the compliance test (this is pre-populated when you modify an existing DataRobot test).Test pass thresholdThe minimum percentage (0-100%) of individual evaluations that must pass for the test as a whole to pass.EvaluationsThe individual evaluations for the compliance test, each consisting of aName,Metric,Evaluation dataset,Pass threshold, andNumber of prompts. In addition to the default metrics and evaluation datasets, you can select any evaluation metrics implemented by a deployed binary classification sidecar model and any evaluation datasets added to the Use Case.Click+ Add evaluationto create additional evaluations.ClickCopy from existing testto copy the individual evaluations from an existing compliance test.There is an API-only process to validate a sidecar model withexpected_response_columnto introduce metrics comparing the LLM response with and expected response, similar to the pre-providedexact_matchmetric. 3. After configuring a custom test, clickSave and run. 4. The test appears on theCompliance teststab with aRunning...status. Cancel a running testIf you need to cancel a test with theRunning...status, clickDelete test results. #### Manage compliance test runs From a running or completed test on the Compliance tests tab: - To delete a completed test run or cancel and delete a running test, click Delete test results . - To view the chat calculating the metric, click the chat name in the Corresponding chat column. - To view the evaluation dataset used to calculate the metric, click the dataset name in the Evaluation dataset column. #### Manage custom compliance tests To manage custom compliance tests, on the Run test panel, from the All tests list, select a custom test, then click Delete test or Edit test. You can't edit or delete pre-configured DataRobot tests. If you select Edit test, update the settings you [configured during compliance test creation](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-eval-metrics.html#create-and-run-custom-compliance-tests). ### Compare compliance test results To compare compliance test results, you can run compliance tests for up to three LLM blueprints at a time. On the Playground tile, in the LLM blueprints list, select up to three LLM blueprint to test, click the Compliance tests tab, and then click Run test. This opens the Run test panel, where you can select and run a test [as you would for a single blueprint](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-eval-metrics.html#run-existing-compliance-tests); however, you can also define the LLM blueprints to run it for. By default, the blueprints selected on the comparison tab are listed here: After the compliance tests run, you can compare them on the Blueprints comparison page. To delete a completed test run, or cancel an in-progress test run, click Delete test results. ## View the tracing table Tracing the execution of LLM blueprints is a powerful tool for understanding how most parts of the GenAI stack work. The Tracing tab provides a log of all components and prompting activity used in generating LLM responses in the playground. Insights from tracing provide full context of everything the LLM evaluated, including prompts, vector database chunks, and past interactions within the context window. For example: - DataRobot metadata: Reports the timestamp, Use Case, playground, vector database, and blueprint IDs, as well as creator name and base LLM. These help pinpoint the sources of trace records if you need to surface additional information from DataRobot objects interacting with the LLM blueprint. - LLM parameters: Shows the parameters used when calling out to an LLM, which is useful for potentially debugging settings like temperature and the system prompts. - Prompts and responses: Provide a history of chats; token count and user feedback provide additional detail. - Latency: Highlights issues orchestrating the parts of the LLM Blueprint. - Token usage: displays the breakdown of token usage to accurately calculate LLM cost. - Evaluations and moderations (if configured): Illustrates how evaluation and moderation metrics are scoring prompts or responses. To locate specific information in the Tracing table, click Filters and filter by User name, LLM, Vector database, LLM Blueprint name, Chat name, Evaluation dataset, and Evaluation status. > [!TIP] Send tracing data to the Data Registry > Click Upload to Data Registry to export data from the tracing table to the Data Registry. A warning appears on the tracing table when it includes results from running the toxicity test and the toxicity test results are excluded from the Data Registry upload. ## Send a metric and compliance test configuration to the workshop After creating an LLM blueprint, setting the blueprint configuration (including evaluations metrics and moderations), and testing and tuning the responses, send the LLM blueprint to the workshop: 1. In a Use Case, from thePlaygroundtile, click the playground containing the LLM you want to register as a blueprint. 2. In the playground,compare LLMsto determine which LLM blueprint to send to the workshop, then, do either of the following: 3. In theSend to the workshopmodal, select up totwelveevaluation metrics (and any configured moderations). Why can't I send all metrics to the workshop?Several metrics are supported by default after you register and deploy an LLM sent to the workshop from the playground, others are configurable using custom metrics. The following table lists the evaluation metrics you cannot select during this process and provides the alternative metric in Console:MetricConsole equivalentCitationsCitations are provided on theData exploration > Data qualitytab. If configured in the playground, citations are included in the transfer by default, without the need to select the option in theSend to the workshopmodal. The resulting custom model has theENABLE_CITATION_COLUMNSruntime parameter configured. After deploying that custom model, if theData explorationtab is enabled andassociation IDs are provided, citations are available for a model sent to the workshop.CostCost can be calculated on theMonitoring > Custom metricstab of a deployment.CorrectnessCorrectness is not available for deployed models.LatencyLatency is calculated on theMonitoring > Service healthtab andMonitoring > Custom metricstab.All TokensAll tokens can be calculated on theCustom metricstab, or you can add the prompt tokens and response tokens metrics separately.Document TokensDocument tokens are not available for deployed models. 4. Next, select anyCompliance teststo send. Then, clickSend to the workshop: Compliance tests sent to the workshop are included when you register the custom model andgenerate compliance documentation. Compliance tests in the workshopThe selected compliance test are linked to the custom model in the workshop by theLLM_TEST_SUITE_IDruntime parameter. If you modify the custom model code significantly in the workshop, set theLLM_TEST_SUITE_IDruntime parameter toNoneto avoid running compliance documentation intended for the original model on the modified model. 5. To complete the transfer of evaluation metrics,configure the custom model in the workshop. --- # Playground overview URL: https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-overview.html > The playground, an asset of a Use Case, is the space for creating and interacting with LLM blueprints. A playground, another type of Use Case asset, is the space for creating and interacting with LLM blueprints. LLM blueprints represent the full context for what is needed to generate a response from an LLM, captured in the LLM blueprint settings. Within the playground you compare LLM blueprint responses to determine which blueprint to use in production for solving a business problem. You can use playgrounds with or without a [vector database](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/index.html). Multiple playgrounds can exist in one Use Case and multiple LLM blueprints can live within a single playground. The suggested, simplified workflow for working with playgrounds is as follows: 1. Add a playground . 2. Set the LLM blueprint configuration , including the base LLM, prompting settings, and optionally, an associated vector database . 3. Chat to test and tune the LLM blueprint; view the tracing . 4. Build additional LLM blueprints . 5. Compare LLM blueprints side-by-side. 6. Add datasets and metrics to the LLM blueprint to help evaluate responses. ## Add a playground From the Playgrounds tab, the following options are available, depending on how many playgrounds already exist: - If you haven't added a playground yet,click theAdd Playgrounddropdown in the center of the page, then clickAdd RAG playground. This button is only available for the first playground added to a Use Case; use theAdd Playgrounddropdown for subsequent playgrounds. - If you've already added one or more playgrounds (RAG or agentic),click theAdd Playgrounddropdown in the upper-right corner of the page, then clickAdd RAG playground. **Playground naming** The playground is named, by default, `Playground `. You can change the name from the Use Case directory by choosing Edit playground info in the Actions menu. When you create a playground, the playground opens with two options for [creating and configuring](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#set-the-configuration) an LLM blueprint. From the playground you have access to all the controls for creating LLM blueprints, interacting with and fine-tuning them, and saving them for comparison and potential future deployment. ## Elements of a playground A playground has basic [navigation controls](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-overview.html#navigate-the-playground) to take you to specific components: And three major work areas: **Configuration panel:** Use this for setting up the LLM blueprint, including vector database selection and prompting strategies. [https://docs.datarobot.com/en/docs/images/create-bp-4.png](https://docs.datarobot.com/en/docs/images/create-bp-4.png) **Chat window:** Use this for sending prompts and receiving LLM responses. [https://docs.datarobot.com/en/docs/images/chatting-2.png](https://docs.datarobot.com/en/docs/images/chatting-2.png) **Comparison panel:** Use this for working with LLM blueprints and chats. [https://docs.datarobot.com/en/docs/images/playground-compare-5.png](https://docs.datarobot.com/en/docs/images/playground-compare-5.png) Once playgrounds are created, you can switch between them using the breadcrumbs dropdown: ## Navigate the playground On the far left, click the icons to access navigation components: | | Component | Description | | --- | --- | --- | | | Playground | Configure LLM blueprints, compare LLM blueprints, and chat. | | | Playground information | Display playground summary information. | | | LLM evaluation* | Configure evaluation and moderation guardrails for LLM blueprints in a playground. | | | Tracing* | Display an exportable log that traces all components used in LLM response generation. | * Available only if LLM assessment is enabled. ### Playground information The Playground information area provides access to the assets associated with the playground—vector databases, deployed LLMs, and deployed embedding models. It also provides basic playground metadata. Use the tabs, described in the table below, to view additional information. **Playground information:** [https://docs.datarobot.com/en/docs/images/playground-info-1.png](https://docs.datarobot.com/en/docs/images/playground-info-1.png) **Deployed LLMs:** [https://docs.datarobot.com/en/docs/images/playground-info-1b.png](https://docs.datarobot.com/en/docs/images/playground-info-1b.png) **Deployed embedding models:** [https://docs.datarobot.com/en/docs/images/playground-info-1c.png](https://docs.datarobot.com/en/docs/images/playground-info-1c.png) | Tab | Description | | --- | --- | | Vector databases | Lists each unique vector database used in the playground and a variety of configuration metadata. Entries that are grayed out are custom vector databases—not built in DataRobot—and so DataRobot is unable to report configuration specifics. Click an entry to view expanded metadata and listings of related assets, including associated LLM blueprints, deployments, custom models, and registered models. Versioning information is also displayed. | | Deployed LLMs | Lists each deployed LLM that is part of at least one LLM blueprint configuration in the playground. For each entry, metadata reports creation information, deployment name, and the prompt and response column names that were defined when assigning a deployed LLM to the blueprint. | | Deployed embedding models | Lists each deployed embedding model that is part of at least one LLM blueprint configuration in the playground. For each entry, metadata reports creation information, deployment name, and the prompt and response column names that were defined. | > [!NOTE] Note > Regardless of how many LLM blueprint configurations a vector database, deployed LLM, or deployed embedding model is used in, the listing displays a single entry. #### Playground information page tools Click on a playground name or description to make changes. When you add a description, it displays under the playground name at the top of the Playground information page. Click the Actions menu, located to the right of the playground name, to open a menu and delete the playground. Alternatively, you can delete a playground from the Use Case asset listing. Click Settings to expose options that control the display. - Set the columns you want to view by checking or unchecking boxes. - Reorder columns using the arrows to the right of the column name. - Set columns to appear on the far left by clicking the pin icon. --- # Single LLM blueprint chat URL: https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html > Describes using single and comparison chats in the playground. Chatting is the activity of sending prompts and receiving a response from the LLM. A chat is a collection of chat prompts. Once you have set the configuration for your LLM, send it prompts—and follow-up prompts—from the entry box in the lower part of the panel to determine whether further refinements are needed before considering your LLM blueprint for deployment. Chatting within the playground is a "conversation"—you can ask follow-up questions with subsequent prompts. Following is an example of asking the LLM to provide Python code for running DataRobot Autopilot: The results of the follow-up questions are dependent on whether [context awareness](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html#context-aware-chatting) is enabled (see continuation of the example). Use the playground to test and tune prompts until you are satisfied with the system prompt and settings. Then, click Save configuration in the bottom of the right-hand panel. ## Context-aware chatting When configuring an LLM blueprint, you set the history awareness in the [Prompting](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#set-system-prompt) tab. There are two states of context. They control whether chat history is sent with the prompt to include relevant context for responses. | State | Description | | --- | --- | | Context-aware | When sending input, previous chat history is included with the prompt. This state is the default. | | No context | Sends each prompt as independent input, without history from the chat. | > [!NOTE] Note > Consider the context state and how it functions in conjunction with the selected [retriever method](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#retriever). You can switch between one-time (no context) and context-aware within a chat. They each become independent sets of history context—going from context-aware, to no context, and back to aware clears the earlier history from the prompt. (This only happens once a new prompt is submitted.) Context state is reported in two ways: 1. A badge, which displays to the right of the LLM blueprint name in both configuration and comparison views, reports the current context state: 2. In the configuration view, dividers show the state of the context setting: Using the example of writing Python code to run Autopilot (above), you could then prompt to make a change to "that code." With context-aware enabled, the LLM responds knowing the code being referenced because it is "aware" of the previous conversation history: See the [prompting reference](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/prompting-reference.html) for information on crafting optimized prompts (including few-shot prompting). ## Single vs comparison chats Chatting with a single LLM blueprint is a good way to tune before starting prompt comparisons with other LLM blueprints.[Comparison](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html#comparison-llm-blueprint-chat) lets you compare responses between LLM blueprints to help decide which to move to production. > [!NOTE] Note > You can only do comparison prompting with workflows that you created. To see the results of prompting another user’s LLM blueprint or agentic flow in a shared Use Case, copy the LLM blueprint or connect to the registered agentic flow. You can chat with the same settings applied. This is intentional behavior because prompting impacts chat history, which can impact the responses that are generated. However, you can provide response feedback on the creator's asset to assist development. ### Single LLM blueprint chat When you first configure an [LLM blueprint](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html), part of the creation process includes chatting. Set the configuration, and save, to activate chatting: After seeing chat results, tune the configuration, if desired, and prompt again. Use the additional actions available within each chat result to retrieve more information and the prompt: | Option | Description | | --- | --- | | View configuration | Shows the configuration used by that prompt in the Configuration panel on the right. If you haven't changed configurations while chatting, no change is apparent. Using this tool allows you to recall previous settings and restore the LLM blueprint to those settings. | | Open tracing | Opens the tracing log, which shows all components and prompting activity used in generating LLM responses. | | Delete prompt and response | Removes both the prompt and response from the chat history. If deleted, they are no longer considered as context for future responses. | As you send prompts to the LLM, DataRobot maintains a record of those chats. You can either add to the context of an existing chat or start a new chat, which does not carry over any of the context from other chats in the history: Starting a new chat allows you to have multiple independent conversation threads with a single blueprint. In this way, you can evaluate the LLM blueprint based on different types of topics, without bringing in the history of the previous prompt response, which could "pollute" the answers. While you could also do this by switching context off, submitting a prompt, and then switching it back on, starting a new chat is a simpler solution. Click Start new chat to begin with a clean history; DataRobot will rename the chat from New chat to the words from your prompt once the prompt is submitted. ### Comparison LLM blueprint chat Once you are satisfied, you can compare responses with other LLM blueprints from the LLM blueprints tab. If you determine that further tuning is needed after having started a comparison, you can still modify the configuration of individual LLM blueprints: To compare LLM blueprint chats side-by-side, see the [LLM blueprint comparison](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/compare-llm.html) documentation. ## Response feedback Use the response feedback "thumbs" to rate the prompt answer. Responses are recorded in the [Tracing](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-eval-metrics.html#tracing), tab User feedback column. The response, as part of the exported feedback sent to the AI Catalog, can be used, for example, to train a predictive model. ## Citations A citation is [a metric](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/playground-eval-metrics.html#add-and-configure-metrics) and is on by default (as are Latency, Prompt Tokens, and Response Tokens). Citations provide a list of the top reference document chunks, based on relevance to the prompt, retrieved from the vector database. Be aware that the embedding model used to create the vector database in the first place can affect the quality of the citations retrieved. > [!NOTE] Note > Citations only appear when the LLM blueprint being queried has an associated vector database. While citations are one of the available metrics, you do not need the assessment functionality enabled to have citations returned. Use citations as a safety check to validate LLM responses. While they help to validate LLM responses, citations also allow you to validate proper and appropriate retrieval from the vector database—are you retrieving the chunks from your docs that you want to provide as context to the LLM? Additionally, if you enable the Faithfulness metric, which measures whether the LLM response matches the source, it relies on the citation output for its relevance. ### ROUGE scores ROUGE scores, also known as confidence scores, calculates the distance between the response generated from an LLM blueprint and the documents retrieved from the vector database. They indicate how close the response is to the provided context. ROUGE scores are computed using the factual consistency metric approach, where a score is computed using the facts retrieved from the [vector database](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html) and the generated text from the LLM blueprint. The similarity metric used is the [ROUGE-1](https://en.wikipedia.org/wiki/ROUGE_(metric%29) (Recall-Oriented Understudy for Gisting Evaluation) metric. DataRobot GenAI uses an improved version of ROUGE-1 based on insights from ["The limits of automatic summarization according to ROUGE"](https://aclanthology.org/E17-2007.pdf). The ROUGE scoring algorithm is not scaled; instead, DataRobot uses heuristic coefficients. The ROUGE score is reported in the prompt response: ### Similarity score The similarity score is based on the distance of the query embedding to the chunk embedding in the vector space; the larger the similarity score, the better. Do not use this score to compare results between vector databases, only to compare values for retrievals within a single vector databases. The score informs "this chunk is more similar than that chunk to the given query." The similarity score is reported in the citation: **Q: How is the similarity score calculated?** TLDR: The similarity score displayed in the citation is based on the [Hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) returned by the binary index during vector search, rescored to cosine similarities with the float embeddings of the user query, and then rounded to two decimals. Similarity scores are based on the distance scores that Facebook AI Similarity Search (FAISS) returns along the indices as a result of the vector search. These binary indices use the Hamming distance as their distance function to find the `top_k` nearest vectors for a given query vector. Instead of using the raw distance scores (which are integers because the [vectors in the binary indexes are binary quantized](https://huggingface.co/blog/embedding-quantization), DataRobot uses a rescoring method adopted from sentence transformers: 1. Retrieverescore_multiplier * top_kresults with the binary query embedding and the binary document embeddings (i.e., the list of the first k results of the binary retrieval). 2. Rescore that list of binary document embeddings with the initial (before they got quantized) float query embeddings. Rescoring is performing a dot product operation between float vectors, returning cosine similarity (multiply corresponding elements of the vectors, sum the results to produce a single scalar value). Applying this rescoring step preserves total retrieval performance, reduces memory and disk space usage, and improves the retrieval speed. 3. Finally, DataRobot rounds the scores to two digits because of floating point arithmetic precision issues caused by numpy, binary quantization, and the rescoring method. Because rescoring is performing a dot product with the embeddings, which leads to a cosine similarity, the higher the value, the better. ### Metadata filtering Use metadata filtering to limit the citations returned by the prompt query. When configured, the LLM blueprint only returns chunks that include the specified metadata column-value pair. You can add a filter for each metadata column, as needed. Each metadata column can be paired with a single value. > [!NOTE] Note > Vector databases created before the introduction of metadata filtering do not support this feature. To use filtering with them, create a version from the original and configure the LLM blueprint to use the new vector database instead. To create a metadata filter, click Filter metadata below the prompt entry box. All optional metadata column names appear in the dropdown, as well as the option `source`, which is content from the `document_file_path` column. If the vector database includes no optional metadata, only `source` is available for selection. Select a column name and enter a single value. The value must be an exact string match from the vector database (partial matches are not allowed). Use Add filter to add filters for different metadata columns > [!NOTE] Note > If the value entered for a source is not an exact match, while a response is returned there are no citations available. This is because there was not match on the filter. If the LLM blueprint configuration does not include a vector database, clicking Filter metadata displays the following: To enable filtering, add a vector database that includes a minimum of the required `document` and `document_file_path` (shown as `source` in filtering) columns. #### Metadata filtering example The following example, which uses the [DataRobot documentation](https://docs.datarobot.com/en/docs/get-started/how-to/genai-walk-basic.html#prerequisites) as the vector database, compares results to the same prompt ("how do I deploy a model?") with and without a metadata filter applied. The image on the left has no filtering. The image on the right set the value of `source` to `source: datarobot_english_documentation/datarobot_docs|en|mlops|deployment|deploy-methods|deploy-model.txt`. This is a value in the `document_file_path` column of the data source. Notice the differences, particularly in prompt tokens and ROUGE score. When you open citations for the filtered prompt, you can see the source is only the one path: --- # Create and manage prompts URL: https://docs.datarobot.com/en/docs/agentic-ai/prompt-mgmt/create-prompts.html > Create prompts, add variables, version, and share prompts with users or organizations for use in the agent experience. Prompts are a fundamental part of interacting with, and generating outputs from, LLMs and agents. Access the prompt management system from the Prompts tile within Registry for a centralized, version-controlled, and integrated system for prototyping, experimenting, deploying, and monitoring prompts as agent components. The Prompts tile initially lists all saved and shared prompts. Click an entry for configuration details. **Registry listing:** [https://docs.datarobot.com/en/docs/images/manage-prompts-2.png](https://docs.datarobot.com/en/docs/images/manage-prompts-2.png) **Prompt details:** [https://docs.datarobot.com/en/docs/images/prompt-3.png](https://docs.datarobot.com/en/docs/images/prompt-3.png) The system supports: - Creating prompts . - Versioning capabilities , for iterating on to prompts. - Managing prompts by comparing and sharing prompts with users or organizations. - Calling a prompt in an agent . **Details: Prompt management benefits** Managing prompts as simple text files creates significant risks and inefficiencies in building production-grade AI agents. DataRobot's prompt management system addresses the major issues, as described in the table below. | Problem | Prompt management system solution | | --- | --- | | Versioning conflicts | Versioning, which is essential for tracking changes, ensures reproducibility, facilitates rollbacks to stable versions, and creates a clear audit trail for collaboration, debugging, and compliance. | | Lack of consistency | Templates allow you to standardize and reuse prompt structures, separating static instructions from dynamic data. The result is faster iteration, consistency, and easier updates without changing application code. | | Compliance violations | Governance provides a controlled process for creating, testing, and deploying prompts. This ensures quality, security, and compliance through centralized registries and approval workflows, which is vital for enterprises. Governance also helps mitigate security risks, such as prompt injection attacks or data leakage, by enforcing policies on the types of data that can be included in a prompt. For large organizations, centralized management and oversight prevent the proliferation of inconsistent or low-quality prompts and provide a comprehensive view of how LLMs are being used across the company. | | Standalone testing | Integrated platforms enable experimentation and evaluation with data-driven prompt engineering. A/B, side-by-side testing and performance tracking (cost, latency, quality) help to identify the most effective prompts. | ## Create prompts To create a prompt, open Registry > Prompt. The prompt library opens with a Create prompt button available. **First prompt:** If you have not created any prompts, or none have been shared with you, the library shows an empty state. [https://docs.datarobot.com/en/docs/images/prompt-2.png](https://docs.datarobot.com/en/docs/images/prompt-2.png) **Populated prompt library:** If you have created prompts, or prompts have been shared with you, they are listed on the Prompt tile landing page. [https://docs.datarobot.com/en/docs/images/prompt-1.png](https://docs.datarobot.com/en/docs/images/prompt-1.png) Click Create prompt and complete the fields. All fields, except prompt text, support a maximum of 5000 characters. | Field | Description | | --- | --- | | Name | Enter a name for the prompt. The name must contain only letters, numbers, underscores (_), dashes (-), spaces, and brackets. | | Description | Enter a description for the prompt. The description is for developer information; it displays on the prompt library landing page and the individual prompt details page. | | Prompt text | Enter the text that will be included with each individual user prompt, up to 5 million characters. Prompt text can contain variables, as described below. Note that text entered in this field is counted as part of an LLM's max completion token limit. You can use a maximum of 100 variables in a single prompt. | | Variables | Enter a variable name marked with double braces {{ variable_name }}. The variable definition must be included with the agent files and must contain only letters, numbers, and underscores (_). Variable character limits are: Name maximum = 200 characters.Description maximum = 200 characters. | | Comments (optional) | Enter text to display on individual prompt details page. | If you are not using variables, click Create. Otherwise see the section on [adding variables](https://docs.datarobot.com/en/docs/agentic-ai/prompt-mgmt/create-prompts.html#add-variables). Also, see [prompt versioning](https://docs.datarobot.com/en/docs/agentic-ai/prompt-mgmt/create-prompts.html#create-prompt-versions) for details on modifying existing prompts. ### Add variables If you enter a variable in the Prompt text field, the Variables section expands to include a description field. The description is required; the content of this field displays on the individual prompt details page. It does not define the variable.[Variable must be defined](https://docs.datarobot.com/en/docs/agentic-ai/prompt-mgmt/prompt-in-agent.html) in the `MyAgent` class of the `agents.py` file. To include a variable: 1. Define the variable(s) inagents.py. Note that this step does not have to be completed first but must be completed before testing the prompt. 2. Enter the prompt text, indicating a variable using double brackets ({{ }}) around the variable name. With each variable you reference, an entry for that variable is included in theVariablessection. You can use the same variable multiple times in the prompt text. 3. ClickCreate. ### View defined prompts When you click the Prompts tile in Registry, the prompt library opens listing all prompts created by, or shared with, you. You can search for the prompt name or use the Created by filter to search by creator. Click an entry to see details of the newest version of the prompt. A list of all versions is displayed in the right panel. To view a different version, click a version in the panel or use the dropdown. The prompt displayed is read-only. To modify it, [create a new version](https://docs.datarobot.com/en/docs/agentic-ai/prompt-mgmt/create-prompts.html#create-prompt-versions). ## Create prompt versions Prompt versioning is vital for managing changes to prompts over time, as even small alterations can significantly impact an LLM's output. This practice ensures reproducibility, allowing teams to link specific model outputs to the exact prompt version that generated them, and facilitates quick rollbacks to stable versions if new changes degrade performance. To version prompts: 1. Click on the prompt you want to modify from the table list of prompts on thePromptstile in Registry. The display shows the most recent version of the prompt; it is read-only. 2. ClickCreate new version from currentin the upper right to open the canvas for creating a new version of the prompt. The existing prompt text, variables, and comments area are now editable. The name and description are not. 3. Modify the text and variables, as needed. Be sure that all description fields are complete. 4. ClickCreate. The new prompt is displayed in the versions panel. ## Manage prompts Once you have created prompts, or prompts have been shared with you, you can sort, compare, and share them. Access the prompt management system from the Prompts tile within Registry. ### Filter table listing You can filter the prompt table to list only those prompts created by certain users. To do so, select prompt creators from the Created by dropdown. ### Compare versions Use the Compare versions functionality to review changes made between versions. To compare: 1. Select the prompt from the table listing. The latest version of the prompt opens. 2. ClickCompare versionsto open the comparison modal. 3. Using the dropdowns, select the versions to compare. Color helps to identify changes in the prompt text; use the scrollbars to see complete entries. Changes in variable and comments are also indicated. ### Share prompts You can share prompts with a user, group, or organization. An owner can share, change roles, or create new versions from the existing prompt. To share the prompt, click the Action menu and choose Share prompt. The [standard sharing](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/sharing.html) modal appears. --- # Prompt management URL: https://docs.datarobot.com/en/docs/agentic-ai/prompt-mgmt/index.html > Create and version prompts, share prompts with other users, and employ prompts in an agentic playground. > [!NOTE] Premium > DataRobot's GenAI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. Effective prompt management is critical for developing production-grade AI agents. From within Registry, the prompt management system provides a centralized, version-controlled, and deeply integrated system to prototype, experiment, deploy, and monitor prompts as components of agents. Prompt governance establishes a controlled process for the creation, testing, approval, and deployment of prompts. A centralized prompt registry serves as a single source of truth, enabling quality assurance through integrated approval workflows that vet prompts for quality, bias, and adherence to company guidelines. | Topic | Description | | --- | --- | | Create and manage prompts | Create prompts, add variables, and version for prompt lineage. Also compare and share prompts with users or organizations. | | Use saved prompts in an agent | Configure variables, employ the saved prompt in agent code, and test the results in an agentic playground. | --- # Call prompts in an agent URL: https://docs.datarobot.com/en/docs/agentic-ai/prompt-mgmt/prompt-in-agent.html > Employ the saved prompt in agent code; test the results in an agentic playground. Prompt templates are reusable prompts created in DataRobot, often containing placeholder variables (like `{{ topic }}`) to be defined in the agent code. The prompt template itself is stored in DataRobot and can have multiple versions, allowing for prompt iteration and fine-tuning without changing the agent code. > [!NOTE] Token limits > Prompt templates count towards the LLM's maximum completion token limit. In the agent code, the framework gets the template through the DataRobot API and provides the variable values. These variables are then substituted into the template text to create the final prompt sent to the LLM. Each framework template handles initial input formatting differently, so the method for using DataRobot prompt templates varies. Before modifying the `myagent.py` file, create a prompt template in DataRobot and note the template ID. If needed, also note the version ID of the specific prompt template version to use. Then, in the `myagent.py` file, modify the appropriate method or property in the `MyAgent` class based on the framework. The examples below show modifications to the existing framework templates in this repository. Each example assumes a prompt template exists in DataRobot containing a `{{ topic }}` variable. For example, the prompt template might be: `Write an article about {{ topic }} in 1997.` The user prompt sent to the agent is combined with this template by substituting the user input into the `{{ topic }}` variable. **LangGraph:** LangGraph uses a `prompt_template` property that returns a `ChatPromptTemplate`. Add `import datarobot as dr` at the top of the file, then modify this property to use DataRobot prompt templates: ``` # Added to imports import datarobot as dr # Modified in MyAgent class @property def prompt_template(self) -> ChatPromptTemplate: prompt_template = dr.genai.PromptTemplate.get("PROMPT_TEMPLATE_ID") prompt_template_version = prompt_template.get_latest_version() # To use a specific version instead: # prompt_template_version = prompt_template.get_version("PROMPT_VERSION_ID") # Convert {{ variable }} format to {variable} format for LangGraph's ChatPromptTemplate # The {topic} variable is filled by the framework at runtime prompt_text = prompt_template_version.to_fstring() return ChatPromptTemplate.from_messages( [ ( "user", prompt_text, ), ] ) ``` Replace the prompt template ID ( `"PROMPT_TEMPLATE_ID"`) with the appropriate template ID from DataRobot. The example uses `get_latest_version()` to automatically use the latest version without redeployment. This example uses `to_fstring()` to convert the template's `{{ topic }}` variable to `{topic}` format, which LangGraph's `ChatPromptTemplate` replaces at runtime. If the variables in the prompt template change across versions (for example, if a new version uses `{{ subject }}` instead of `{{ topic }}`), update this code to handle all variables appropriately, otherwise the code may break when fetching a new version. > [!NOTE] Multi-agent workflows > The `prompt_template` property is used for the initial user input. In the [Agentic Starter](https://github.com/datarobot-community/datarobot-agent-application) and related templates, each graph node is built with LangChain's [create_agent](https://python.langchain.com/docs/modules/agents/) and a `system_prompt` argument (often wrapped with `make_system_prompt` from `datarobot_genai`). To ensure all agents follow the prompt template instructions, incorporate the formatted template text into each node's `system_prompt` (not the `prompt=` parameter used by some other LangGraph examples such as `langgraph.prebuilt.create_react_agent`). See [Customize agents](https://docs.datarobot.com/en/docs/agentic-ai/agentic-develop/agentic-development.html#modify-agent-prompts) ( Modify agent prompts, LangGraph tab) for the API used in DataRobot templates. **LlamaIndex:** LlamaIndex uses a `make_input_message` method that returns a string. Add `import datarobot as dr` at the top of the file, then modify this method to use DataRobot prompt templates: ``` # Added to imports import datarobot as dr # Modified in MyAgent class def make_input_message(self, completion_create_params: Any) -> str: user_prompt_content = extract_user_prompt_content(completion_create_params) prompt_template = dr.genai.PromptTemplate.get("PROMPT_TEMPLATE_ID") prompt_template_version = prompt_template.get_latest_version() # To use a specific version instead: # prompt_template_version = prompt_template.get_version("PROMPT_VERSION_ID") # Render the prompt template with variables (assumes {{ topic }} in the template) prompt_text = prompt_template_version.render(topic=user_prompt_content) return prompt_text ``` Replace the prompt template ID ( `"PROMPT_TEMPLATE_ID"`) with the appropriate template ID from DataRobot. The example uses `get_latest_version()` to automatically use the latest version without redeployment. This example assumes the prompt template contains a `{{ topic }}` variable. If the variables in the prompt template change across versions (for example, if a new version uses `{{ subject }}` instead of `{{ topic }}`), update this code to handle all variables appropriately, otherwise the code may break when fetching a new version. > [!NOTE] Multi-agent workflows > The `make_input_message` method affects only the initial input message. Each agent has its own `system_prompt` property. To ensure all agents follow the prompt template instructions, incorporate the formatted prompt template into each agent's `system_prompt` property. **CrewAI:** CrewAI uses agent properties ( `goal`, `backstory`) that can contain prompt templates. Add `import datarobot as dr` at the top of the file, then modify agent properties to use DataRobot prompt templates: ``` # Added to imports import datarobot as dr # Modified in MyAgent class @property def agent_planner(self) -> Agent: prompt_template = dr.genai.PromptTemplate.get("PROMPT_TEMPLATE_ID") prompt_template_version = prompt_template.get_latest_version() # To use a specific version instead: # prompt_template_version = prompt_template.get_version("PROMPT_VERSION_ID") # For properties that use {topic} (f-string format), use to_fstring() prompt_text = prompt_template_version.to_fstring() return Agent( role="Planner", goal=f"Plan engaging and factually accurate content on {{ topic }}. {prompt_text}", backstory=f"You're working on planning a blog article about the topic: {{ topic }}. {prompt_text} " "You collect information that helps the audience learn something and make informed decisions. " "Your work is the basis for the Content Writer to write an article on this topic.", # ... other properties ) ``` Replace the prompt template ID ( `"PROMPT_TEMPLATE_ID"`) with the appropriate template ID from DataRobot. The example uses `get_latest_version()` to automatically use the latest version without redeployment. This example modifies `agent_planner` to use the prompt template in its `goal` and `backstory` properties. Since these properties use `{topic}` (f-string format that CrewAI will fill at runtime), the example uses `to_fstring()` to convert `{{ topic }}` to `{topic}` format so CrewAI can replace it with the user's input. This example assumes the prompt template contains a `{{ topic }}` variable. If the variables in the prompt template change across versions (for example, if a new version uses `{{ subject }}` instead of `{{ topic }}`), update this code to handle all variables appropriately, otherwise the code may break when fetching a new version. > [!NOTE] Multi-agent workflows > Apply prompt templates to each agent's `goal` or `backstory` properties where you want the instructions to be followed. For properties that use `{topic}`, use `to_fstring()`. For plain text properties, use `render()`. --- # ACL hydration URL: https://docs.datarobot.com/en/docs/agentic-ai/vector-database/acl-hydration.html > Learn about fine-grained access control for vector database results based on original document permissions from external sources. > [!NOTE] Premium > DataRobot's RAG ACL management capabilities are a premium feature; contact your DataRobot representative for enablement information. This functionality is not available in the DataRobot trial experience. Access Control List (ACL) hydration enforces fine-grained authorization for vector database results based on original document permissions from external sources, such as Google Drive and SharePoint. When files are ingested from these sources, DataRobot captures and maintains their access control information, ensuring that users can only access vector database chunks for documents they have permission to view in the original source system. ## Overview ACL hydration enables organizations to maintain the same access control policies in DataRobot that exist in their source systems. This is particularly important for administrators who need to ensure that sensitive documents remain protected when used in vector databases for generative AI applications. For datasets, ACL hydration works as follows: 1. Ingest files from a supported external source. DataRobot registers them in the File Registry . 2. DataRobot captures and caches initial ACLs during ingestion. 3. DataRobot continuously monitors ACL changes in the source system using polling mechanisms: 4. When a user queries a vector database, DataRobot filters results based on the user's permissions in the original source system, ensuring only accessible chunks are returned. **What is the latency for ACL updates?** DataRobot targets low-latency ACL updates to ensure permissions are applied as quickly as possible. While the system is designed to support near real-time updates (targeting a few seconds in the long term), initial implementations may have latencies of approximately 1 minute, which is acceptable for most use cases. Updates that take 10 minutes or longer are not acceptable. See the [feature considerations](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/acl-hydration.html#feature-considerations) for more information. ### Key capabilities ACL hydration provides the following capabilities: | Capability | Description | | --- | --- | | Initial ACL capture | Automatically captures and stores access control information when files are first ingested from supported data sources. | | ACL change monitoring | Continuously polls source systems to detect and apply permission changes as they occur. | | Vector database result filtering | Filters vector database query results to return only chunks from documents the user has permission to access. | | Multi-user support | Supports multiple users with different permission levels, ensuring each user sees only the data they're authorized to access. | ## Supported data sources ACL hydration is currently supported for the following data sources: | Data source | ACL tracking method | | --- | --- | | Google Drive | Drive Activity API | | SharePoint | Delta Query | For information on the required APIs and permissions, see [ACL metadata hydration and enforcement](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/acl-hydrate-query.html). ## Setting up ACL hydration To enable ACL hydration for your organization, configure admin connections for the supported data sources. ### Prerequisites Before setting up ACL hydration, ensure you have: - An organization administrator account with appropriate permissions. - A connection to a supported data source . - Files ingested from the supported data sources (after the connection is configured). ### Configuration steps 1. As a DataRobot organization admin, go to theData connectionspage. 2. Select an existing connection of the supported data source type or create a new connection (using service accounts for admin connections is highly recommended). 3. Toggle onEnable access control list synchronization. DataRobot will use this connection to poll for ACL changes across all files in your organization. 4. Configure the ACL management details as needed: 5. Admin impersonation email: Set the administrator account to impersonate in the background. This field is mandatory for Google Drive. 6. Domain(optional): Overwrites the user's email domain with the specified string. For example, ifDomainis set tosharepoint.com, and the user email ismy-account@company.com, DataRobot will trackmy-account@sharepoint.comACLs. 7. ClickSave. There can be only one admin connection per data source type. Any admin user in the DataRobot organization can enable ACL synchronization for a connection they have permission to. Doing so overwrites the existing admin connection for that data source for the entire organization. Organization admin users do not see all admin connections, only those that have been shared with them explicitly or that they own. 8. Retrieve the connection ID to create the data source and begin ingesting files. Initial ACLs are automatically captured during ingestion. NoteFor Google Drive, extract both the Drive ID and Folder ID from the Google Drive URL. For new DataRobot users, an async job automatically refreshes external principals (user IDs) to minimize the no-ACL gap. ## How ACL hydration works ACL hydration captures the access control list, monitors the source system settings, and returns vector database results to each user based on those settings. When files are first ingested, DataRobot captures the current access control list for each file. ACL information is stored in the File Registry, including: ``` * The origin of each file (connector type and external file ID). * Original catalog ID and catalog version ID. * Initial ACLs with user and group permissions. ``` Permission information is preserved even if the catalog item is updated or files are modified. DataRobot continuously monitors for ACL changes in the source system using polling mechanism queries the source APIs at regular intervals, targeting low latency. For example: ``` * For Google Drive, DataRobot uses Drive Activity API to query for permission change events. * For SharePoint, DataRobot uses Delta Query to retrieve incremental changes. ``` When ACL changes are detected, DataRobot updates the stored ACL information in the File Registry. Changes are applied immediately and impact to future vector database queries. When a user queries a vector database, DataRobot checks the user's permissions for each file referenced in the vector database chunks. Only chunks from files the user has permission to access are included in the results. The system uses the stored ACL information ,combined with the user's external principal mapping, to determine access. ## User access levels ACL hydration respects the original source system permissions: | Access level | Behavior | | --- | --- | | Full access | Users with full access to a file in the source system can see all vector database chunks from that file. | | Restricted access | Users with restricted access see only the chunks they're authorized for, based on source system permissions. | | No access | Users without access to a file in the source system cannot see any vector database chunks from that file, even if they have access to the catalog item in DataRobot. | ### File access scenarios The following sample scenarios illustrate how ACL hydration works: Scenario 1: User who ingested files - A user who ingests files from Google Drive has access to all files they ingested. - This user can see all vector database chunks from those files in query results. Scenario 2: User with restricted access - A user who is a member of a specific group has access only to files shared with that group. - This user can see vector database chunks only from files they have permission to access in the source system. - Results from files in folders they don't have access to are filtered out. Scenario 3: Public files - Files marked as public or with "connectivity access" are accessible to all users. - All users can see vector database chunks from these public files. ## Integrate user identities with applications and agents DataRobot supports integration between applications, agents, vector databases, and ACL enforcement. This integration is built so that applications and agents automatically enforce ACL-filtered vector database results for the requesting user. For applications that call DataRobot agents and use ACL-enabled vector databases, you must propagate the requesting user's identity so that the agent can apply the correct ACL filtering. To do so: 1. Read the identity token from the incoming request. Your app receives requests (e.g., from a front end or API client). Read theX-DataRobot-Identity-Tokenheader from each request. 2. Pass theX-DataRobot-Identity-Tokenheader when calling the agent. When your app invokes a DataRobot agent (e.g., for chat or RAG), include the same header in the outbound call so that the agent can identify the user and filter vector database results by that user's permissions. The way you pass the header depends on how your app calls the agent (e.g., as `extra_headers` to an LLM client or as a headers object to a stream manager). Reading the header is the same in all cases: ``` # Example: read the identity token from the request (e.g., FastAPI/Starlette) identity_token = request.headers.get("X-DataRobot-Identity-Token") ``` Then, forward this value in the headers you send when calling the agent. If the identity token is not propagated, the agent cannot apply per-user ACL filtering and behavior may not match the intended access policy. ## Feature considerations - DataRobot supports ACL hydration with SharePoint and Google Drive. - Locally uploaded files and database connections do not support ACL hydration, as they do not have external access control systems to reference. - ACL hydration enforcement is not automatically applied to existing vector databases and ingested files. It is only applied to newly created vector databases with newly ingested files from supported sources. - DataRobot applies the latest retrieved ACLs regardless of the background synchronization health. If you need to confirm that identity and ACL sync are healthy, use the admin API monitoring described inMonitoring background synchronization. - DataRobot does not support cross-drive links for ACL hydration. For example, if you ingest files using a link pointing to another drive, the ingestion completes successfully but the files are considered inaccessible via ACL. ## Monitoring background synchronization Administrators can monitor the health of background synchronization using the Event Logs API. Two event types are relevant: ### User identity and membership synchronization Use the following to return a report on user identity and membership synchronization with the external source: `GET https://{DATAROBOT_URL}/api/v2/eventLogs/?event=External+principals+synchronized+for+a+connector` The request returns the following response fields: | Field | Description | | --- | --- | | timestamp | The time of the report, represented in UTC. | | context.connectorType | The data source (e.g., gdrive). | | context.mostOutdatedMinutes | The user identity synchronization latency, in minutes. | | context.mode | The job status, either: Create a vector databases, work with versions and related assets, and interact in the playground. > [!NOTE] Premium > DataRobot's GenAI capabilities are a premium feature; contact your DataRobot representative for enablement information. Try this functionality for yourself in a limited capacity in the DataRobot trial experience. A vector database is a collection of unstructured text that is broken into chunks, with embeddings generated for each chunk. Both the chunks and embeddings are stored in a database and are available for retrieval. Vector databases can optionally be used to ground the LLM responses to specific information and can be assigned to an LLM blueprint to leverage during a [RAG](https://docs.datarobot.com/en/docs/reference/glossary/index.html#retrieval-augmented-generation-rag) operation. The role of the vector database is to enrich the prompt with relevant context before it is sent to the LLM. The simplified workflow for working with vector databases is as follows: 1. Create a vector databaseobject. 2. Add anappropriatedata source from theData Registry. 3. Set the configuration,embeddings, andchunking. 4. Create the vector database and add it to an LLM blueprintin the playground. See the [considerations related to vector databases](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#vector-database-considerations) for guidance when working with DataRobot GenAI capabilities. Working with vector databases includes the following: | Topic | Description | | --- | --- | | Add data sources | Add internal and external data sources; actions from the Vector databases tab in the Use Case directory. | | Create a vector database | Create and configure a vector database. | | Versioning internal vector databases | Use versioning to modify DataRobot-hosted internal (FAISS-based) vector databases for tracking and fine-tuning. | | Versioning connected vector databases | Add data to an existing Pinecone-, Elasticsearch-, Milvus-, or PostgreSQL-based vector database connection. | | Register and deploy vector databases | Send vector databases from the playground to the workshop for modification and deployment, or deploy the current vector database version directly to Console. | | Use an embedding NVIDIA NIM to create a vector database | Premium feature. Add a registered or deployed embedding NVIDIA NIM to a Use Case with a vector database to enrich prompts in the playground with relevant context before they are sent to the LLM. | | ACL hydration | Premium feature. ACL (Access Control List) hydration enforces fine-grained authorization for vector database results based on original document permissions from external sources such as Google Drive and SharePoint. | --- # Update connected vector databases URL: https://docs.datarobot.com/en/docs/agentic-ai/vector-database/update-connected-vdbs.html > Add data to an existing Pinecone-, Elasticsearch-, Milvus-, or PostgreSQL-based vector database connection. > [!NOTE] Note > Versioning—creating a complete, new vector database—is not available for external, connected vector databases. Instead, you can add data to ("hydrate") a connected vector database without creating a new version. This page describes the ability to add data to a Pinecone, Elasticsearch, Milvus, or PostgreSQL connected vector database. See the section on [versioning a resident vector databases](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html) for details on creating child, related entities based on a single, DataRobot-hosted parent. ## Add data To add data, go to the Vector databases tile in the left panel and select the connected vector database you want to update. From the Actions menu, choose Add data. > [!NOTE] Note > Before adding data, take note of the number of chunks shown in the Number of chunks column. After adding data, you can compare the chunk count of the newly hydrated vector database with this value. > > [https://docs.datarobot.com/en/docs/images/connected-add-4.png](https://docs.datarobot.com/en/docs/images/connected-add-4.png) The Update vector database page opens, providing: - The Data source dropdown, which provides access to all Use Case vector databases and provides an option to Add data . - An optional field for attaching metadata . - A summary of the current vector database configuration. This is the same information provided in the Details section on the vector database listing. From the Data source dropdown, select Add data. The [Data Registry](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/dataprep/add-data-usecase.html) opens where you can select a dataset to append to the current vector database. Click to select the new dataset and then click Add dataset. You are returned to the Update vector database page; click Update in the upper right to save changes. When you receive a popup notification that data was successfully added, you can confirm the addition by: - Checking your collection at the provider site. - Comparing the number of chunks shown in the Number of chunks column to the number observed prior to adding data. ## Deploy the updated vector database After updating the vector database with new data, you can do the following. Note that you do not need to redeploy to pick up the changes. Because the deployment is simply a "pass-through" to the connected vector database index, existing deployments automatically have access to the added data. - Send it to the model workshop, where it is maintained as a standalone vector database custom model. - Deploy it, which sends it to the model workshop, registers it, and then deploys it. See the section on [registering and deploying](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs-register-deploy.html) vector databases for more information. --- # Vector database data sources URL: https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs-data.html > Learn to create and apply your own text data for use by an LLM in GenAI. Generative modeling in DataRobot supports three types of vector databases: - Resident , "in-house" built vector databases, with the Source listed in the application showing DataRobot. Supporting up to 10GB, they are stored in DataRobot and can be found in Vector databases tile for a Use Case. - Connected vector databases up to 100GB, which link out to an external provider. The Source listed in the application is the provider name and they are stored in the provider instance. - External , hosted in Registry workshop for validation and registration, and identified as Read-only connected in the Use Case directory listing. They have no size constraints, since they are hosted outside DataRobot, but must fit within the resource bundle memory of the vector database deployment. ## Dataset requirements When uploading datasets for use in creating a vector database, the supported formats are either `.zip` or `.csv`. Two columns are mandatory for the files— `document` and `document_file_path`. Additional metadata columns, up to 50, can be added for use in filtering during prompt queries. Note that for purposes of [metadata filtering](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html#metadata-filtering), `document_file_path` is displayed as `source`. For `.zip` files, DataRobot processes the file to create a `.csv` version that contains text columns ( `document`) with an associated reference ID ( `document_file_path`) column. All content in the text column is treated as strings. The reference ID column is created automatically when the `.zip` is uploaded. All files should be either in the root of the archive or in a single folder inside an archive. Using a folder tree hierarchy is not supported. **Data connector limitation for vector databases** Data ingested via data connectors (such as Google Drive or SharePoint) is stored in the [File Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/file-registry.html) and cannot be used as VDB metadata. All metadata files must be uploaded to Datasets storage in the Data Registry, as assets contained in Files storage are not validated by the EDA process. If a vector database requires metadata for filtering, access control, or downstream retrieval logic, ensure that the metadata is provided as a Dataset in the Data Registry rather than sourced from a data connector. See the [considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html#supported-dataset-types) for more information on supported file content. ## Internal vector databases Internal vector databases in DataRobot are optimized to maintain retrieval speed while ensuring an acceptable retrieval accuracy. Follow the steps below to prepare and upload the data. 1. Prepare the data as follows. Be sure to see theCSV-specific requirement detailsto ensure the data is in the correct format. 2. Upload the file. You can do this either: Once the data is available on DataRobot, you can [add it as a vector database](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html) for use in the playground. #### CSV-specific requirement details The mandatory columns for CSV are defined as follows: - document can contain any amount (up to file size limitations) of free-text content. - document_file_path is also required and requires a file format suffix (for example, file.txt ). Using a `.csv` file allows you to make use of the [no chunking](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#chunking-settings) option during vector database creation. DataRobot will then treat each row as a chunk and directly generate an embedding on each row. DataRobot vector databases only support using one text column from a CSV for the primary text content. If a CSV has multiple text columns, they must be concatenated into a single `document` column. You can add up to 50 other columns in the CSV as metadata columns. These columns can be used for [metadata filtering](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html#metadata-filtering), which limits the citations returned by the prompt query. If the CSV has multiple text columns, you can: - Combine (concatenate) the text columns into one document column. - Convert the CSV rows into individual PDF files (one PDF per row) and then upload the PDFs. For example: Consider a CSV file containing one column with large amounts of free text, `swag`, and a second column with an ID but no text, `InventoryID`. To create a vector database from the data: 1. Rename swag to document . 2. Rename InventoryID to document_file_path . 3. Add "fake" paths to the document_file_path column. For example, change 11223344 to /inventory/11223344.txt . In this way, the column is recognized as containing file paths. ### Export a vector database You can export a vector database, or a specific version of a database, to the Data Registry for re-use in a different Use Case. To export, open the Vector database tile of your Use Case. Click the Actions menu menu and select Export latest vector database version to Data Registry. When you export, you are notified that the job is submitted. Open the Data assets tile to see the dataset registering for use via the Data Registry. It is also saved to the AI Catalog. Once registered, you can preview the dataset or create a new vector database from this dataset. **Preview the dataset:** To preview before creating a new vector database from the export, from the Data assets tile choose Create vector database from the newly added vector database's Actions menu.[Select a provider](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#select-a-provider) and from the Data source dropdown choose Add data. [https://docs.datarobot.com/en/docs/images/vdb-export-3.png](https://docs.datarobot.com/en/docs/images/vdb-export-3.png) The Data Registry opens. Under DataRobot assets choose Datasets and click on the newly exported vector database. The Data preview shows that each [chunk](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#chunking-settings) from the vector database is now a dataset row. [https://docs.datarobot.com/en/docs/images/vdb-export-4.png](https://docs.datarobot.com/en/docs/images/vdb-export-4.png) **Create a new vector database:** From the Actions menu menu select Create vector database. A modal opens to [configure the database](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html). [https://docs.datarobot.com/en/docs/images/vdb-3a.png](https://docs.datarobot.com/en/docs/images/vdb-3a.png) You can download the dataset from the [Data Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-data-registry/nxt-explore-data.html), modify it on a chunk level, and then re-upload it, creating a new version or a new vector database. ## External (BYO) vector databases The external "bring-your-own" (BYO) vector database provides the ability to leverage your custom model deployments as vector databases for LLM blueprints, using your own models and data sources. This vector database type is identified as `Read-only connected` in the Use Case directory listing. Using an external vector database cannot be done via the UI; review the [notebook](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/chromadb-vdb.html) that walks through creating a ChromaDB external vector database using DataRobot’s Python client. Key features of external vector databases: - Custom model integration:Incorporate your own custom models as vector databases, enabling greater flexibility and customization. - Input and output format compatibility:External BYO vector databases must adhere to specified input and output formats to ensure seamless integration with LLM blueprints. - Validation and registration:Custom model deployments must be validated to ensure they meet the necessary requirements before being registered as an external vector database. - Seamless integration with LLM blueprints:Once registered, external vector databases can be used with LLM blueprints in the same way as local vector databases. - Error handling and updates:The feature provides error handling and update capabilities, allowing you to revalidate orcreate duplicatesof LLM blueprints to address any issues or changes in custom model deployments. ### Basic external workflow The basic workflow, which is covered in depth in [this notebook](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/chromadb-vdb.html), is as follows: 1. Create the vector database via the API. 2. Create a custom model deployment to bring the vector database into DataRobot. 3. Once the deployment is registered, link to it as part of vector database creation in your notebook. You can view all vector databases (and associated versions) for a Use Case from the Vector database tab within the Use Case. For external vector databases, you can see only the source type. Because these vector databases aren't managed by DataRobot, other data is not available for reporting.. --- # Use an embedding NVIDIA NIM to create a vector database URL: https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs-nvidia-nim-embed.html > Add a deployed embedding NVIDIA NIM to a Use Case with a vector database to enrich prompts in the playground with relevant context before they are sent to the LLM. > [!NOTE] Premium > The use of NVIDIA Inference Microservices (NIM) in DataRobot requires access to premium features for GenAI experimentation and GPU inference. Contact your DataRobot representative or administrator for information on enabling the required features. The NVIDIA Inference Microservices (NIM) available through Registry include embedding models. A deployed embedding model can be added to a Use Case, creating a collection of unstructured text that is broken into chunks, with embeddings generated for each chunk. Both the chunks and embeddings are stored in the vector database and are available for retrieval. Vector databases can optionally be used to ground the LLM responses to specific information and can be assigned to an LLM blueprint to leverage during a RAG operation. The role of the vector database is to enrich the prompt with relevant context before it is sent to the LLM. Each embedding NVIDIA NIM available is listed below: - arctic-embed-l - llama-3.2-nv-embedqa-1b-v2 - nv-embedqa-e5-v5 - nv-embedqa-e5-v5-pb24h2 - nv-embedqa-mistral-7b-v2 - nvclip ## Create a vector database with a registered embedding NIM After you register an embedding NIM, you can add it to a vector database. DataRobot handles the deployment process automatically. To create a vector database with a registered embedding NVIDIA NIM: 1. On theRegistry > Modelstab, next to+ Register a model, clickand thenImport from NVIDIA NGC. 2. In theImport from NVIDIA NGCpanel, on theSelect NIMtab, click an embedding NIM in the gallery. Search the galleryTo direct your search for an embedding model, you canSearch, filter byPublisher, or clickSort byto order the gallery by date added or alphabetically (ascending or descending). 3. Review the model information from the NVIDIA NGC source, then clickNext. 4. On theRegister modeltab, configure the following fields and clickRegister: FieldDescriptionRegistered model name / Registered modelConfigure one of the following:Registered model name:When registering a new model, enter auniqueand descriptive name for the new registered model. If you choose a name that exists anywhere within your organization, a warning appears.Registered model:When saving as a version of an existing model, select the existing registered model you want to add a new version to.Registered version nameAutomatically populated with the model name and the wordversion. Change the version name or modify the default version name as necessary.Registered model versionAssigned automatically. This displays the expected version number of the version (e.g., V1, V2, V3) you create. This is alwaysV1when you selectRegister as a new model.Resource bundleRecommended automatically. If possible, DataRobot translates the GPU requirements for the selected model into a resource bundle. In some cases, DataRobot can't detect a compatible resource bundle. To identify a resource bundle with sufficient VRAM, review the documentation for that NIM.NVIDIA NGC API keySelect the credential associated with your NVIDIA NGC API key.Optional settingsRegistered version descriptionEnter a description of the business problem this model package solves, or, more generally, describe the model represented by this version.TagsClick+ Add tagand enter aKeyand aValuefor each key-value pair you want to tag the modelversionwith. Tags added when registering a new model are applied toV1. 5. After the registered model builds, navigate toWorkbenchand open a Use Case. 6. In a Use Case, on theVector databasestab, either: With an existing vector databasesWithout an existing vector databaseIf you have already added one or more vector databases to the Use Case, Click the+ Add vector databasebutton in the upper right.If you haven't added a vector database to the Use Case before, clickCreate vector databasein the center of the page. 7. On theCreate vector databasepanel, enter a descriptiveName. Then, in theData sourcedropdown, select from the data sources associated with the Use Case or clickAdd datato add new data from the Data Registry. 8. In theEmbedding modeldropdown, click the embedding NIM you registered. Then, configure thevector databaseText chunkingsettingsand clickCreate vector database. The selected embedding model is deployed toConsolewhen you create the vector database. If necessary, this process creates a newprediction environmentfor NIM embeddings. After creating a vector database, you can [manage](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#manage-vector-databases) and [version](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html) it, or [add it to an LLM in the playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#add-a-vector-database) to inform responses. ## Create a vector database with a deployed embedding NIM If you've already registered and deployed an embedding NIM, you can add it to a vector database as a deployed embedding model. To create a vector database with a registered and deployed embedding NVIDIA NIM: 1. In a Use Case, on theVector databasestile, either: With an existing vector databasesWithout an existing vector databaseIf you have already added one or more vector databases to the Use Case, Click the+ Add vector databasebutton in the upper right.If you haven't added a vector database to the Use Case before, clickCreate vector databasein the center of the page. 2. On theCreate vector databasepanel, enter a descriptiveName. Then, in theData sourcedropdown, select from the data sources associated with the Use Case or clickAdd datato add new data from the Data Registry. 3. In theEmbedding modeldropdown, clickAdd deployed embedding model. 4. On the next page, configure the following settings to add the NVIDIA NIM embedding model, then clickValidate and add: FieldDescriptionNameEnter a descriptive name for the embedding model you're creating.Deployment nameIn the list, locate the name of the NVIDIA NIM embedding modelregistered and deployed in DataRobotand click the deployment name.Prompt column nameEnterinputas the prompt column name.Response column nameEnterresultas the response column name. Validation processThe validation process can take a few minutes. A notification appears when the process starts and if it succeeds or fails. 5. After the validation of the deployed embedding model succeeds, open theEmbedding modelmenu, then, underDeployed embedding models, select the NVIDIA NIM embedding model. 6. Configure thevector databaseText chunkingsettings, then clickCreate vector database. After creating a vector database, you can [manage](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#manage-vector-databases) and [version](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html) it, or [add it to an LLM in the playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#add-a-vector-database) to inform responses. --- # Register and deploy vector databases URL: https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs-register-deploy.html > Send vector databases from the playground to the workshop for modification and deployment, or deploy the current vector database version directly to Console. The Vector databases tab lists all vector databases associated with a Use Case, both [DataRobot-hosted and connected](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#select-a-provider). For DataRobot-hosted vector databases, entries include information on the versions derived from the parent; see the section on [versioning](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html) for detailed information on vector database versioning. Connected vector databases have only a single version, although you can [add data](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/update-connected-vdbs.html) to that version. **Vector databases tab:** For each vector database on the Vector databases tab, the Actions menu allows you to send the vector database to production in two ways, depending on the provider type. For example: [https://docs.datarobot.com/en/docs/images/vector-db-register-deploy.png](https://docs.datarobot.com/en/docs/images/vector-db-register-deploy.png) **Vector database details:** [https://docs.datarobot.com/en/docs/images/vector-db-register-deploy-2.png](https://docs.datarobot.com/en/docs/images/vector-db-register-deploy-2.png) | Method | Description | | --- | --- | | Send to the workshop | Send the vector database to the workshop for modification and deployment. | | Deploy this version (DataRobot-hosted) | Deploy the latest version of the vector database to the selected prediction environment. | | Deploy vector database (connected) | Deploy the vector database to the selected prediction environment. | ## Send to the workshop To send a vector database from the playground to the workshop, click Send to the workshop and provide the following information: **Resource bundle:** [https://docs.datarobot.com/en/docs/images/vector-db-send-to-workshop.png](https://docs.datarobot.com/en/docs/images/vector-db-send-to-workshop.png) **Memory:** [https://docs.datarobot.com/en/docs/images/vector-db-send-to-workshop-memory.png](https://docs.datarobot.com/en/docs/images/vector-db-send-to-workshop-memory.png) | Field | Description | | --- | --- | | Memory | Determines the maximum amount of memory that can be allocated for a custom inference model. If a model is allocated more than the configured maximum memory value, it is evicted by the system. If this occurs during testing, the test is marked as a failure. If this occurs when the model is deployed, the model is automatically launched again by Kubernetes. | | Bundle | Preview feature If enabled for your organization, selects a Resource bundle—instead of Memory—. Resource bundles allow you to choose from various CPU and GPU hardware platforms for building and testing custom models in the workshop. | | Replicas | Sets the number of replicas executed in parallel to balance workloads when a custom model is running. Increasing the number of replicas may not result in better performance, depending on the custom model's speed. | | Network access | Premium feature. Configures the egress traffic of the custom model: Public: The default setting. The custom model can access any fully qualified domain name (FQDN) in a public network to leverage third-party services.None: The custom model is isolated from the public network and cannot access third-party services. When public network access is enabled, your custom model can use the DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN environment variables. These environment variables are available for any custom model using a drop-in environment or a custom environment built on DRUM. | > [!NOTE] Premium feature: Network access > Every new custom model you create has public network access by default; however, when you create new versions of any custom model created before October 2023, those new versions remain isolated from public networks (access set to None) until you enable public access for a new version (access set to Public). From this point on, each subsequent version inherits the public access definition from the previous version. > [!NOTE] Preview feature: Resource bundles > Custom model resource bundles and GPU resource bundles are off by default. Contact your DataRobot representative or administrator for information on enabling this feature. > > Feature flag: Enable Resource Bundles, Enable Custom Model GPU Inference ( Premium feature) Next, click Send to the workshop. You are redirected to the workshop after the vector database is successfully created. ## Deploy the vector database version To deploy a vector database from the playground to Console, click Deploy this version (DataRobot-hosted) or Deploy vector database (connected) and provide the following information: **Resource bundle:** [https://docs.datarobot.com/en/docs/images/vector-db-deploy.png](https://docs.datarobot.com/en/docs/images/vector-db-deploy.png) **Memory:** [https://docs.datarobot.com/en/docs/images/vector-db-deploy-memory.png](https://docs.datarobot.com/en/docs/images/vector-db-deploy-memory.png) | Field | Description | | --- | --- | | Choose prediction environment | Determines the prediction environment for the deployed vector database. Verify that the correct prediction environment with Platform: DataRobot Serverless is selected. | | Memory | Determines the maximum amount of memory that can be allocated for a custom inference model. If a model is allocated more than the configured maximum memory value, it is evicted by the system. If this occurs during testing, the test is marked as a failure. If this occurs when the model is deployed, the model is automatically launched again by Kubernetes. | | Bundle | Preview feature If enabled for your organization, selects a Resource bundle—instead of Memory—. Resource bundles allow you to choose from various CPU and GPU hardware platforms for building and testing custom models in the workshop. | | Replicas | Sets the number of replicas executed in parallel to balance workloads when a custom model is running. Increasing the number of replicas may not result in better performance, depending on the custom model's speed. | | Network access | Premium feature. Configures the egress traffic of the custom model: Public: The default setting. The custom model can access any fully qualified domain name (FQDN) in a public network to leverage third-party services.None: The custom model is isolated from the public network and cannot access third-party services. When public network access is enabled, your custom model can use the DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN environment variables. These environment variables are available for any custom model using a drop-in environment or a custom environment built on DRUM. | Next, click Deploy. You are redirected to Console after the vector database is successfully deployed. > [!NOTE] What monitoring is available for vector database deployments? > DataRobot automatically generates custom metrics relevant to vector databases for deployments with the Vector Database deployment type; for example, Total Documents, Average Documents, Total Citation Tokens, Average Citation Tokens, and VDB Score Latency. Vector database deployments also support service health monitoring. Vector database deployments don't store prediction row-level data for data exploration. --- # Create a vector database URL: https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html > Configure the data source, embeddings, and chunking strategy for a vector database. > [!NOTE] GPU usage for Self-Managed users > When working with datasets over 1GB, Self-Managed users who do not have GPU usage configured on their cluster may experience serious delays. Email [DataRobot Support](mailto:support@datarobot.com), or visit the [Support site](https://support.datarobot.com/), for installation guidance. The basic steps for creating a vector database for use in a playground are to choose a provider, set the basic configuration, and set text chunking. **Provider:** Choose a [provider](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#select-a-provider) for the output of the vector database creation process, either DataRobot-resident or a connected source. [https://docs.datarobot.com/en/docs/images/vdb-3a1.png](https://docs.datarobot.com/en/docs/images/vdb-3a1.png) **Basic configuration:** Set the [basic configuration](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#set-configuration), including data source from the [Data Registry](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs-data.html) or [File Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/file-registry.html) and [embedding model](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#set-the-embedding-model). [https://docs.datarobot.com/en/docs/images/vdb-3e.png](https://docs.datarobot.com/en/docs/images/vdb-3e.png) **Text chunking:** Set [text chunking](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#chunking-settings). [https://docs.datarobot.com/en/docs/images/vdb-3f.png](https://docs.datarobot.com/en/docs/images/vdb-3f.png) Use the [Vector databasestile](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#manage-vector-databases) in the Use Case directory to manage built vector databases and deployed embedding models. You will ultimately store the newly created vector database (the output of the vector database creation process) within the Use Case. They are stored either as internal vector databases (FAISS) or on [connected provider instances](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#use-a-connected-vector-database). ## Add a vector database First, add a vector database from one of multiple points within the application. Each method opens the Create vector database modal and uses the same workflow from that point. **TheUse Case assetstile:** From within a Use Case, click the Add dropdown and expand Vector database: [https://docs.datarobot.com/en/docs/images/add-vdb-1.png](https://docs.datarobot.com/en/docs/images/add-vdb-1.png) Choose Create vector database to open the vector database creation modal. Choose Add deployed vector database to add a deployment containing a vector database that you previously registered and deployed . If there are not yet any assets associated with the Use Case, you can add a vector database from the tile landing page. [https://docs.datarobot.com/en/docs/images/add-vdb-2.png](https://docs.datarobot.com/en/docs/images/add-vdb-2.png) **TheData assetstile:** From the Data assets tile, open the Actions menu associated with a Use Case and select Create vector database. [https://docs.datarobot.com/en/docs/images/add-vdb-data-1.png](https://docs.datarobot.com/en/docs/images/add-vdb-data-1.png) The Actions menu is only available if the data is detected as eligible, which means: Processing of the dataset has finished. The data source has the mandatory document and document_file_path columns. There are no more than 50 metadata columns. **TheVector databasestile:** From the Vector databases tile, click the Add dropdown: [https://docs.datarobot.com/en/docs/images/add-vdb-3.png](https://docs.datarobot.com/en/docs/images/add-vdb-3.png) Choose Create vector database to open the vector database creation modal. Choose Add deployed vector database to add a deployment that contains a vector database that you previously registered and deployed . Choose Add external vector database to begin assembling a custom vector database in the Registry workshop , which can then be linked to the Use Case. If there are not yet any vector databases associated with the Use Case, the tile landing page will lead you to create one. [https://docs.datarobot.com/en/docs/images/vdb-3.png](https://docs.datarobot.com/en/docs/images/vdb-3.png) **The playground:** When in a playground, use the [Vector database](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#add-a-vector-database) tab in the configuration section of the LLM blueprint: [https://docs.datarobot.com/en/docs/images/vdb-3b.png](https://docs.datarobot.com/en/docs/images/vdb-3b.png) Once you've selected to add a vector database, start the creation process. ## Select a provider Select a vector database provider, either internal or connected (external) with credentials. This setting determines where the output of the vector database creation process lands. The input to the creation process is always either the [Data or File Registry](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#add-a-data-source). | Provider/type | Description | Max size | | --- | --- | --- | | DataRobot/resident | An internal FAISS-based vector database, hosted locally in the Data Registry. These vector databases can be versioned and do not require credentials. | 10GB | | BYO | A deployment containing a vector database that you previously registered and deployed. Use a notebook to bring-your-own vector database via a custom model deployment. | No constraints, but must fit within the resource bundle memory of the vector database deployment. | | Connected | An external vector database that allows you to use your own Pinecone, Elasticsearch, Milvus, or PostgreSQL instance, with credentials. This option allows you to choose where content is stored but still experiment with RAG pipelines built in DataRobot and leverage DataRobot's out-of-the-box embedding and chunking functionality. | 100GB | ### Use a resident vector database Using a resident vector database means using data that is accessible within the application, either via the Data Registry or a custom model. Internal vector databases in DataRobot are optimized to maintain retrieval speed while ensuring an acceptable retrieval accuracy. See the following for dataset requirements and specific retrieval methods: - SelectDataRobotforan internal vector databasestored in the Data Registry. ### Use an external vector database To use an [external](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs-data.html#external-byo-vector-databases) (BYO) vector database, use the Add > Add deployed vector database option that is available from the Vector database tile. Before adding a vector database this way, develop the vector database externally with the DataRobot Python client, assemble a custom model for it, and then deploy that custom model. See an [example using ChromaDB](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/chromadb-vdb.html). Note that this vector database type is identified as `Read-only connected` in the Use Case directory listing. ### Use a connected vector database DataRobot allows direct connection to external data sources for vector database creation. In this case, the data source is stored locally in the Data Registry, configuration settings are applied, and the created vector database is written back to the provider. The following connections are supported: - Pinecone - Elasticsearch - Milvus - PostgreSQL (pgvector) Select your provider in the Create vector database modal. To use a provider connection, select the provider and enter authentication information. If you choose to use saved credentials, for any of the connected providers, simply select the appropriately named credentials from the dropdown. Available credentials are those that are created and stored in the [credential management system](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html#credentials-management): If you choose to use new credentials, they must first be created in the provider instance. Once entered, they are stored in DataRobot's credential management system for reuse. #### Connect to Pinecone All connection requests to [Pinecone](https://docs.pinecone.io/guides/get-started/overview) must include an API key for connection authentication. If you do not have a Pinecone API key saved to the credential management system, click New credentials. Complete the field for the API key and, optionally, change the display name. In the API token (API key) field, paste the key you created in the [Pinecone console](https://docs.pinecone.io/reference/api/authentication). Once added, DataRobot saves the Pinecone API key in the [credential management system](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html#credentials-management) for reuse when working with Pinecone vector databases. Once the token is input, select a [cloud provider](https://docs.pinecone.io/guides/index-data/create-an-index#cloud-regions) for your Pinecone instance—AWS, Azure, or GCP— and assigned cloud region. After selection, the vector database [configuration options](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#set-configuration) become available. #### Connect to Elasticsearch All connection requests to the [Elasticsearch](https://www.elastic.co/docs/get-started) cluster must include a username and password (basic credentials) or an API key for connection authentication. If you do not have, or wish to add, an Elasticsearch API key saved to the credential management system, click New credentials. There are two types of credentials available for selection in the modal that appears. - Basic: Basic credentials consist of the username and password you use to access the Elasticsearch instance. Enter them here and they will be saved to DataRobot. - API key: In theAPI token (API key)field, paste the key you created, as described in theElasticsearch documentation. Once the credential type is selected, optionally change the display name and select a connection method, either Cloud ID (recommended by Elastic) or URL. See the Elasticsearch documentation for information on [finding your cloud ID](https://www.elastic.co/docs/deploy-manage/deploy/elastic-cloud/find-cloud-id). After selection, the vector database [configuration options](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#set-configuration) become available. #### Connect to Milvus Milvus, a leading open-source vector database project, is distributed under the Apache 2.0 license. All connection requests to [Milvus](https://milvus.io/docs/connect-to-milvus-server.md) must include a username and password (basic credentials) or an API token for connection authentication. If you do not have, or wish to add, a Milvus API key saved to the credential management system, click New credentials. There are two types of credentials available for selection in the modal that appears. - Basic: Basic credentials consist of the username and password you use to access the Milvus instance. Enter them here and they will be saved to DataRobot. - API token: In theAPI token (API key)field, paste the key you created, as described in theMilvus documentation. Once you've selected the credential type, optionally change the display name and [enter a URI](https://milvus.io/docs/connect-to-milvus-server.md) for the Milvus server address. > [!NOTE] Note > When you complete the configuration and create the vector database, it will be available on the Milvus site for your cluster, under the Collections tab. Open the collection and then the Data tab to see the vectors, text, and various metadata. > > [https://docs.datarobot.com/en/docs/images/milvus-collection.png](https://docs.datarobot.com/en/docs/images/milvus-collection.png) After selection, the vector database [configuration options](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#set-configuration) become available. #### Connect to PostgreSQL DataRobot can create vector databases by connecting to a PostgreSQL instance with the [pgvector](https://github.com/pgvector/pgvector) extension enabled, providing vector similarity search, ACID compliance, replication, point-in-time recovery, JOINs, and other PostgreSQL features. > [!NOTE] Note > If using PostgreSQL as the connection method when using a [BYO embedding model](https://docs.datarobot.com/en/docs/agentic-ai/genai-code/create-vdb-byo-embedding.html), the output dimension must be less than 2000. Making connection requests to the PostgreSQL cluster requires a username and password (basic credentials), plus some additional fields. | Field | Description | | --- | --- | | Host | The host name or IP address of the server. | | Database | The database name. Ask your PostgreSQL admin if unknown; often database name is the database with the same name as the user name used to connect to the server. | | Port | The port number the server is listening on. | | Schema | The schema, public by default, used to resolve unqualified object names over this connection. | | SSL mode | The level of protection to provide, either prefer, require (the default), or verify-full. Note that verify-full requires the Postgres server to be configured with a certificate signed by a public certificate authority or a DataRobot installation configured with a private certificate authority. | See the [PostgreSQL documentation](https://jdbc.postgresql.org/documentation/use/#connecting-to-the-database) for more information. > [!NOTE] Note > This PostgreSQL connection is separate from the [data connection added via the Data Registry](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/dataprep/add-data-usecase.html#data-connections), where you can also connect to PostgreSQL. While the fields are similar, a Data Registry connection is not used for vector database creation. Instead, you must configure it as part of the vector database creation flow. After selection, the vector database [configuration options](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#set-configuration) become available. ## Set configuration The following table describes the configuration settings used in vector database creation. | Field | Description | | --- | --- | | Name | The name the vector database is saved with. This name displays in the Use Case Vector databases tile and is selectable when configuring playgrounds. | | Data source | The dataset used as the knowledge source for the vector database. The list populates based on the entries in the Use Case's Vector databases tile, if any. If you started the vector database creation from the action menu on the Data assets tile, the field is prepopulated with that dataset. If there are no associated vector databases or none present are applicable, use the Add data option. | | Attach metadata | The name of the file, in the file registry, that is used for appending columns to the vector database to support filtering the citations returned by the prompt query. | | Distance metric (connected providers only) | The vector similarity metric to use in nearest neighbor search, ranking a vector's similarity against the query. | | Embedding model | The model that defines the type of embedding used for encoding data. | ### Add a data source If no data sources are available or you want to add new sources, choose Add data in the Data source dropdown. The Add data modal opens. Vector database creation supports ZIP and CSV dataset formats and specific [supported file types](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/genai-consider.html#supported-dataset-types) within the datasets. You can access a supported dataset from either the [File Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/file-registry.html) or the [Data Registry](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs-data.html). Vector databases allow ingest from remote sources that support unstructured data, which are outlined on the [Data Sources page](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/data-unstructured/index.html). When you use supported remote sources (such as Google Drive or SharePoint), access control information can be propagated and enforced so that VDB results respect the original document permissions—see [ACL hydration](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/acl-hydration.html) for details. | Registry type | Description | | --- | --- | | File | A "general purpose" storage system that can store any type of data. In contrast to the Data Registry, the File Registry does not do CSV conversion on files uploaded to it. In the UI, vector database creation is the only place where the File Registry is applicable, and it is only accessible via the Add data modal. While any file type can be stored there, regardless of registry type the same file types are supported for vector database creation. | | Data | In the Data Registry, a ZIP file is converted into a CSV with the content of each member file stored as row of the CSV. The file path for each file becomes the document_file_path column and the file content (text or base64-encoding of the file) becomes the document column. | > [!NOTE] Supported datastores > For a full list of supported connectors and drivers, see [Supported data stores](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/data-unstructured/index.html). Note that some connectors only support unstructured data, which when added, appear in the Files section under Data Registry. **Data connector limitation for vector databases** Data ingested via data connectors (such as Google Drive or SharePoint) is stored in the [File Registry](https://docs.datarobot.com/en/docs/workbench/nxt-registry/file-registry.html) and cannot be used as VDB metadata. All metadata files must be uploaded to Datasets storage in the Data Registry, as assets contained in Files storage are not validated by the EDA process. If a vector database requires metadata for filtering, access control, or downstream retrieval logic, ensure that the metadata is provided as a Dataset in the Data Registry rather than sourced from a data connector. Choose a dataset. Datasets from the Data Registry show a preview of the chunk ( `document`) and the file it was sourced from ( `document_file_path`). ### Attach metadata Optionally, you can select an additional file to define the metadata to attach to the chunks in the vector database. That file must reside in Datasets storage in the Data Registry. (It cannot reside in Files storage because the EDA process, which validates that the required columns are present, only runs on Datasets assets.) The file must contain, at minimum, the `document_file_path` column. A `document` column is optional. You can append up to 50 additional columns, which can be used for [filtering during prompt queries](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html#metadata-filtering). You can upload a single CSV with document and metadata information in a single file. Alternatively, you can upload data and metadata as separate files and achieve the same final result. > [!NOTE] Note > Vector databases created before the introduction of metadata filtering do not support this feature. To use filtering with them, create a version from the original and configure the LLM blueprint to use the new vector database instead. Either select an available file, or use the Add data modal to add a metadata file to the Data Registry. In this example, the file has one column, `document_file_path`, which defines the file or chunk, as well as a variety of other columns that define the metadata. Once you select a metadata file, you are prompted to choose whether, if you also have metadata in the dataset, DataRobot should keep both sets of metadata or overwrite with the new file. Whether to replace or merge the metadata only applies to if there are duplicate columns between the dataset and metadata dataset. Non-duplicate metadata columns from both are always maintained. **Duplicate metadata example** A dataset has `col1` and `col2.` The metadata file has `col2` and `col3`. The metadata going into the vector database is then `col1`, `col2`, and `col3`. Since `col2` is a duplicate, DataRobot either does the following, based on the settings: - Replace : DataRobot uses the values from the metadata file. - Keep both : DataRobot merges the two columns, with precedence going to the metadata file. ### Set the distance metric When using a connected provider, you can also set a distance vector, also known as a similarity metric (this setting is not available when DataRobot is selected as provider). These metrics measure how similar vectors are; selecting the appropriate metric can substantially boost the effectiveness of classification and clustering tasks. Consider choosing the same similarity metric that was used to train the embedding model. | Provider | Provider documentation | | --- | --- | | Pinecone | Similarity metrics | | Elasticsearch | Similarity parameter | | Milvus | Metric types: float | | PostgreSQL (pgvector) | Similarity metrics | ### Set the embedding model To encode your data, select the embedding model that best suits your Use Case. **DataRobot-provided:** Use one of the [DataRobot-provided](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#datarobot-provided) embeddings. **Deployed:** Use a [BYO](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#add-deployed) or [OpenAI API-compatible](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#add-openai-api-compatible) deployed embedding model. While adding a vector database, you can select an embedding model deployed as an unstructured custom model. In the Configuration section of the Create vector database panel, click the Embedding model dropdown and select Add embedding model. [https://docs.datarobot.com/en/docs/images/deployed-embedding.png](https://docs.datarobot.com/en/docs/images/deployed-embedding.png) #### DataRobot-provided DataRobot supports the following types of embeddings; see the full embedding descriptions [here](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html#embeddings-availability). | Embedding type | Description | | --- | --- | | cl-nagoya/sup-simcse-ja-base | A medium-sized language model for Japanese RAG. | | huggingface.co/intfloat/multilingual-e5-base | A medium-sized language model used for multilingual RAG performance across multiple languages. | | huggingface.co/intfloat/multilingual-e5-small | A smaller-sized language model used for multilingual RAG performance with faster performance than the multilingual-e5-base. | | intfloat/e5-base-v2 | A medium-sized language model used for medium-to-high RAG performance. With fewer parameters and a smaller architecture, it is faster than e5_large_v2. | | intfloat/e5-large-v2 | A large language model designed for optimal RAG performance. It is classified as slow due to its architecture and size. | | jinaai/jina-embedding-t-en-v1 | A tiny language model pre-trained on the English corpus and is the fastest, and default, embedding model offered by DataRobot. | | jinaai/jina-embedding-s-en-v2 | Part of the Jina Embeddings v2 family, this embedding model is the optimal choice for long-document embeddings (large chunk sizes, up to 8192). | | sentence-transformers/all-MiniLM-L6-v2 | A small language model fine-tuned on a 1B sentence-pairs dataset that is relatively fast and pre-trained on the English corpus. It is not recommended for RAG, however, as it was trained on old data. | | Add embedding model | Select a deployed embedding model to use during vector database creation, either deployed on DataRobot or an OpenAI API-compatible model deployed elsewhere. For models deployed on DataRobot, column names can be found on the deployment's overview page in Console. | The embedding models that DataRobot provides are based on the [SentenceBERT framework](https://github.com/UKPLab/sentence-transformers), providing an easy way to compute dense vector representations for sentences and paragraphs. The models are based on transformer networks (BERT, RoBERTA, T5) trained on a mixture of supervised and unsupervised data, and achieve state-of-the-art performance in various tasks. Text is embedded in a vector space such that similar text is grouped more closely and can efficiently be found using cosine similarity. #### Add deployed To add an embedding model deployed as an unstructured custom model, from the embedding model dropdown select Add embedding model > Add deployed. Then, configure the following settings: | Setting | Description | | --- | --- | | Name | Enter a descriptive name for the embedding model. | | Deployment name | Select the unstructured custom model deployment. | | Prompt column name | Enter the name of the column containing the user prompt, defined when you created the custom embedding model in the workshop (for example, promptText). | | Response (target) column name | Enter the name of the column containing the LLM response, defined when you created the custom embedding model in the workshop (for example, responseText or resultText). | | Chunks in request | Specify the number of chunks sent in a single request to the deployed embedding model. The minimum value is 1, the default is 64, and there is no maximum value. | After you configure the deployed embedding model settings, click Validate and add. The deployed embedding model is added to the Embedding model dropdown list. #### Add OpenAI API-compatible Adding a deployed OpenAI API-compatible embedding model connection directly from DataRobot allows you to access your own OpenAI API-compatible model deployment without creating a proxy custom model that redirects requests. To connect to an embedding model that supports the OpenAI API format, from the embedding model dropdown, select Add embedding model > Add OpenAI API-compatible. Then, configure access information. > [!NOTE] Considerations > Encoding format requirements > : Because DataRobot's embedding client expects > encoding_format="float" > when calling external OpenAI-compatible embedding endpoints, it is important to ensure that your provider accepts that format. Providers that do not support > float > encoding format are not compatible with this feature and will return a 400 error. > Rate limiting > : When using an OpenAI API-compatible embedding model, it is possible that third-party rate limiting may prevent vector database creation. | Setting | Description | | --- | --- | | Name | Enter a descriptive name that serves as an identifier for the embedding model. The name is shown in the embedding model dropdown whenever creating vector databases in this Use Case. | | Embedding model | Enter the model or deployment name provided by your hosting provider. If using a cloud provider, this may be a custom name you provided for the deployment (for example, my-embedding-model). If you are using OpenAI directly, it is the model name they provided (for example, text-embedding-ada-002). | | Endpoint URL | Enter the root address used for all API requests to the embedding model deployment. This path is not necessarily related to OpenAI, but might be if OpenAI is the provider. This parameter is essential for directing traffic to alternative endpoints, proxies, or self-hosted AI models that follow the OpenAI API format. | | Chunks in request | Specify the number of chunks sent in a single request to the deployed embedding model. The minimum value is 1, the default is 64, and there is no maximum value. | | Authentication | Select either a set of previously saved credentials or create new API token-type credentials. | Click Add to make the embedding model available. Or, optionally, click + Add parameter can pass additional parameters in the request body when calling an API—beyond the standard parameters exposed by the UI. Use these when DataRobot needs specific identifying information in the test request sent to validate the connection. Set key-value pairs and configure whether to send them during vector database creation, querying, or both. Once all fields are set as needed, click Create vector database. ### Chunking settings Text chunking is the process of splitting a text document into smaller text chunks that are then used to generate [embeddings](https://docs.datarobot.com/en/docs/reference/glossary/index.html#embedding). You can either: - Choose Text chunking and further configure how chunks are derived— method , separators , and other parameters . - Select No chunking . DataRobot will then treat each row as a chunk and directly generate an embedding on each row. **Chunking:** [https://docs.datarobot.com/en/docs/images/vdb-3d.png](https://docs.datarobot.com/en/docs/images/vdb-3d.png) **No chunking:** [https://docs.datarobot.com/en/docs/images/vdb-3c.png](https://docs.datarobot.com/en/docs/images/vdb-3c.png) #### Chunking method The chunking method sets how text from the data source is divided into smaller, more manageable pieces. It is used to improve the efficiency of nearest-neighbor searches so that when queried, the database first identifies the relevant chunks that are likely to contain the nearest neighbors, and then searches within those chunks rather than searching the entire dataset. | Method | Description | | --- | --- | | Recursive | Splits text until chunks are smaller than a specified max size, discards oversized chunks, and if necessary, splits text by individual characters to maintain the chunk size limit. | | Semantic | Splits larger text into smaller, meaningful units based on the semantic content instead of length (chunk size). It is a fully automatic method, meaning that when it is selected, no further chunking configuration is available—it creates chunks where sentences are semantically "closed.". See the deep dive below for more information. | **Deep dive: Chunking methods** Recursive text chunking works by recursively splitting text documents according to an ordered list of text separators until a text chunk has a length that is less than the specified maximum chunk size. If generated chunks have a length/size that is already less than the max chunk size, the subsequent separators are ignored. Otherwise, DataRobot applies, sequentially, the list of [separators](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#work-with-separators) until chunks have a length/size that is less than the max chunk size. In the end, if a generated chunk is larger than the specified length, it is discarded. In that case, DataRobot will use a "separate each character" strategy to split on each character and then merge consecutive split character chunks up to the point of the max chunk size limit. If no "split on character" is listed as a separator, long chunks are cut off. That is, some parts of the text will be missing for the generation of embeddings but the entire chunk will still be available for document retrieval. Semantic chunking is the process of breaking down a larger piece of text into smaller, meaningful units (or "chunks") based on the semantic content or meaning of the text, rather than just arbitrary character or word limits. Instead of splitting the text based solely on length, semantic chunking attempts to keep coherent ideas or topics intact within each chunk. This method is useful for tasks like natural language processing (NLP), where understanding the meaning and context of the text is important for tasks like information retrieval, summarization, or generating embeddings for machine learning models. For example, in a semantic chunking process, paragraphs might be kept together if they discuss the same topic, even if they exceed a specific size limit, ensuring that the chunks represent complete thoughts or concepts. That said, the DataRobot implementation of semantic chunking for out-of-the-box embedding models automatically detects the maximum supported chunk size of the selected embedding model and uses it as a safety cutoff. This forces a chunk to be cut if it would exceed the embedding model's maximum input length to ensure that all text is actually embedded. BYO embeddings, which use the default version of the algorithm, do not support the safety cutoff. Chunks can be of any length, even exceeding the embedding model's maximum input length, which can result in text being cut off (and therefore not embedded). The non-embedded text is still included in the citation returned from the vector database. #### Work with separators Separators are "rules" or search patterns ( [not regular expressions](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#use-regular-expressions) although they can be supported) for breaking up text by applying each separator, in order, to divide text into smaller components—they define the tokens by which the documents are split into chunks. Chunks will be large enough to group by topic, with size constraints determined by the model’s configuration. Recursive text chunking is the method applied to the chunking rules. Each vector database starts with four default rules, which define what to split text on: - Double new lines - New lines - Spaces While these rules use a word to identify them for easy understanding, on the backend they are interpreted as individual strings (i.e., `\n\n`, `\n`, `" "`, `""`). There may be cases where none of the separators are present in the document, or there is not enough content to split into the desired chunk size. If this happens, DataRobot applies a "next-best character" fallback rule, moving characters into the next chunk until the chunk fits the defined chunk size. Otherwise, the embedding model would just truncate the chunk if it exceeds the inherent context size. #### Add custom rule You can add up to five custom separators to apply as part of your chunking strategy. This provides a total of nine separators (when considered together with the four defaults). The following applies to custom separators: - Each separator can have a maximum of 20 characters. - There is no "translation logic" that allows use of words as a separator. For example, if you want to chunk on punctuation, you would need to add a separator for each type. - The order of separators matters. To reorder separators, simply click the cell and drag it to the desired location. - To delete separators, whether in fine-tuning your chunking strategy or to free space for additional separators, click the trashcan icon. You cannot delete the default separators. #### Use regular expressions Select Interpret separators as regular expressions to allow regular expressions in separators. It is important to understand that with this feature activated, all separators are treated as regex. This means, for example, that adding "." matches and splits on every character. If you instead want to split on "dots," you must escape the expression (i.e., "`\.`"). This rule applies to all separators, both custom and predefined (which are configured to act this way). ### Chunking parameters Chunking parameters further define the output of the vector database. The default values for chunking parameters are dependent on the embedding model. #### Chunk overlap Overlapping refers to the practice of allowing adjacent chunks to share some amount of data. The Chunk overlap parameter specifies the percentage of overlapping tokens between consecutive chunks. Overlap is useful for maintaining context continuity between chunks when processing the text with language models, at the cost of producing more chunks and increasing the size of the vector database. #### Retrieval limits The value you set for Top K (nearest neighbors) instructs the LLM on how many relevant chunks to retrieve from the vector database. Chunk selection is based on [similarity scores](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/rag-chatting.html#rouge-scores). Consider: - Larger values provide more comprehensive coverage but also require more processing overhead and may include less relevant results. - Smaller values provide more focused results and faster processing, but may miss relevant information. Max tokens specifies: - The maximum size (in tokens) of each text chunk extracted from the dataset when building the vector database. - The length of the text that is used to create embeddings. - The size of the citations used in RAG operations. ## Save the vector database Once the configuration is complete, click Create vector database to make the database [available in the playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#set-the-configuration). ## Manage vector databases The Vector databases tile lists all the vector databases and [deployed embedding models](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#add-deployed) and [OpenAI API-compatible](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#add-openai-api-compatible) associated with a Use Case. Vector database entries include information on the versions derived from the parent; see the section on [versioning](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html) for detailed information on vector database versioning. You can view all vector databases (and associated versions) for a Use Case from the Vector database tab within the Use Case. For external vector databases, you can see only the source type. Because these vector databases aren't managed by DataRobot, other data is not available for reporting. Click on any entry in the Vector databases tile listing to open a new modal where you can view an expanded view of that database's configuration and related items. You can: | | Description | | --- | --- | | (1) | Select a different vector database to explore from the dropdown in the breadcrumbs. | | (2) | Select a different version of the vector database to explore from the dropdown. When you click a version, the details and reported assets (Related items) update to those associated with the specific version. Learn more about versioning. | | (3) | Execute a variety of vector database actions. | | (4) | View the versioning history. When you click a version, the details and reported assets (Related items) update to those associated with the specific version. Learn more about versioning. | | (5) | View items associated with the vector database, such as the related Use Case and LLM blueprints and deployed and registered models that use the vector database. Click on an entity to open it in the corresponding Console tab. | | (6) | Create a playground that uses the selected database. | ### Vector database actions The actions dropdown allows you to apply an action to the vector database you are viewing. The actions available are slightly different depending on the vector database type and where you are accessing the menu from. - Use theActions menufrom the vector database listing accessed fromVector databasestile. - Open a vector database from theVector databasestile—a modal displays an expanded view of that database's configuration. A menu button,Vector database actions, provides access to a set of actions. From the Actions menu, you can: **DataRobot-hosted:** [https://docs.datarobot.com/en/docs/images/vdb-hosted-actions.png](https://docs.datarobot.com/en/docs/images/vdb-hosted-actions.png) Action Description Export latest vector database version to Data Registry Exports the most recent versions of the selected vector database to the Data Registry for re-use in a different Use Case. Create playground from latest version Opens a new playground with the vector database loaded into the LLM configuration. Create new vector database version Creates a new version of the vector database that is based on the version that is currently selected. Edit vector database info Provides an input box for changing the vector database name. Send to the workshop Sends the vector database to the workshop for modification and deployment. For more information, see register and deploy vector databases . Deploy this version Deploys the latest version of the vector database to the selected prediction environment. For more information, see register and deploy vector databases . Delete vector database and all versions Deletes the parent vector database and all versions. Because the vector databases used by deployments are snapshots, deleting a vector database in a Use Case does not affect the deployments using that vector database. The deployment uses an independent snapshot of the vector database. **Connected:** [https://docs.datarobot.com/en/docs/images/vdb-connected-actions.png](https://docs.datarobot.com/en/docs/images/vdb-connected-actions.png) Action Description Add data Appends a selected data source to the current vector database source. Data is added from within DataRobot and then written back to the provider. Create playground Opens a new playground with the vector database loaded into the LLM configuration. Edit vector database info Provides an input box for changing the vector database name. Send to the workshop Sends the vector database to the workshop for modification and deployment. For more information, see register and deploy vector databases . Deploy vector database Deploys the updated vector database to the selected prediction environment. For more information, see register and deploy vector databases . Delete vector database Deletes the vector database instance. Because the vector databases used by deployments are snapshots, deleting a vector database in a Use Case does not affect the deployments using that vector database. The deployment uses an independent snapshot of the vector database. From the Vector database actions dropdown menu, you can: **DataRobot-hosted actions dropdown:** [https://docs.datarobot.com/en/docs/images/vdb-hosted-menu.png](https://docs.datarobot.com/en/docs/images/vdb-hosted-menu.png) Action Description Create playground using this version Opens a new playground with the vector database loaded into the LLM configuration. Create new version from this version Creates a new version of the vector database that is based on the version that is currently selected. Export this version to Data Registry Exports the current vector database version to Data Registry for re-use in a different Use Case. Send to the workshop Sends the vector database to the workshop for modification and deployment. For more information, see register and deploy vector databases . Deploy this version Deploys the latest version of the vector database to the selected prediction environment. For more information, see register and deploy vector databases . Delete vector database Deletes the parent vector database and all versions. Because the vector databases used by deployments are snapshots, deleting a vector database in a Use Case does not affect the deployments using that vector database. The deployment uses an independent snapshot of the vector database. **Connected actions dropdown:** [https://docs.datarobot.com/en/docs/images/vdb-connected-menu.png](https://docs.datarobot.com/en/docs/images/vdb-connected-menu.png) Action Description Create playground using this version Opens a new playground with the vector database loaded into the LLM configuration. Add data Appends a selected data source to the current vector database source. Data is added from within DataRobot and then written back to the provider. Export this version to Data Registry Exports the latest vector database version to Data Registry. It can then be used in different Use Case playgrounds. Send to the workshop Sends the vector database to the workshop for modification and deployment. For more information, see register and deploy vector databases . Deploy vector database Deploys the vector database to the selected prediction environment. For more information, see register and deploy vector databases . Edit authentication Provides a modal where you can change the display name for saved credentials or add new authentication credentials, which are then stored in the credential management system . See the provider-specific credential information above . Delete vector database Deletes the vector database. Deleting a vector database in a Use Case does not affect the deployments using that vector database because the deployment uses an independent snapshot of the vector database. For additional information, see also: - Versioning DataRobot-hosted vector databases . - Updating connected vector databases . ### Vector database details The details section of the vector database expanded view reports information for the selected version, whether you selected the version from the dropdown or the right-hand panel. - Basic vector database metadata: ID, creator and creation date, data source name and size. - Chunking configuration settings: Embedding column and chunking method and settings. - Metadata columns: Names of columns from the data source, which can later be used for metadata filtering . Use this area to quickly compare versions to see how configuration changes impact chunking results. For example, notice how the size and number of chunks changes between the parent version that uses the DataRobot English language documentation: And with the addition of the Japanese language documentation: ## Next steps Once you've created a vector database, use it in a playground, manage its versions, or send it to the workshop to deploy. - Build LLM blueprints : Add your new vector database to an LLM blueprint in a playground to start building a RAG pipeline. - Versioning DataRobot-hosted vector databases : Create new versions of a resident vector database to track and fine-tune changes over time. - Register and deploy vector databases : Send a vector database to the workshop for modification, or deploy the current version directly to Console. --- # Update resident vector databases URL: https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html > Use versioning to modify DataRobot-hosted resident vector databases for tracking and fine-tuning. > [!NOTE] Note > Versioning—creating a complete, new vector database—is available for resident DataRobot-hosted (FAISS-based) vector databases. Although versioning is not available for external, connected vector databases, you can add data to ("hydrate") a connected vector database in place without creating a new version. This page describes the ability to version resident (FAISS-based) vector databases—creating child, related entities based on a single parent—brings a host of benefits to agentic solution building. For Pinecone, Elasticsearch, Milvus, or PostgreSQL (pgvector), see the section on [updating connected vector databases](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/update-connected-vdbs.html). Versioning uses metadata in the lineage process to help assess and compare results with previous versions. With versioning you can: - Update the data in your vector database, ensuring the most up-to-date data is available to ground LLM responses. - Create new versions, creating a full vector database lineage, but also select previous versions. This allows you to "update" older versions that are used by downstream assets and to roll back to previous versions, if needed. - Apply "tried and true" chunking and/or embedding parameters from existing vector databases to new data. - Use the dataset's metadata during retrieval, allowing you to more effectively search for chunks in dataset. Versions related to a single parent vector database are displayed in a collapsible right panel. Click any version to update the details and related items to reflect information for the selected version. ## Create a version You can create a vector database version from any parent or child version on which you are an owner—you do not need to have been the vector database creator. (You do have to be the creator to delete a vector database or version, as described in the [considerations](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html#sharing-and-permissions).) On the vector database details screen, use the selector to create a new version from the parent or the Vector database actions dropdown to create from a selected version. **From the parent (1):** From the dropdown that shows the displayed version, click Create new version to create a version based on the parent version: [https://docs.datarobot.com/en/docs/images/vdb-v3.png](https://docs.datarobot.com/en/docs/images/vdb-v3.png) **From selected (2):** From the Vector database actions dropdown, click Create new version from this version to modify the selected version to create a new version. [https://docs.datarobot.com/en/docs/images/vdb-v4.png](https://docs.datarobot.com/en/docs/images/vdb-v4.png) In either case, a version creation window opens with fields dependent on your update selection method—adding or replacing data. | Fields | Description | | --- | --- | | Update vector database version | Select the data and chunking settings for the new child version. | | Current vector database configuration | When adding data. Review the configuration of the vector database version from which the new version is created. | | Test chunking | When replacing data. Set whether to use chunking, and if so, the chunking configuration. | | Related items | When related items are connected to the source of the new version, manage which assets are updated to use the newly created version. | Choose how to update the vector database. You can either [add data](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html#update-vector-database-version-add) to the existing source data or [replace the data source](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-versions.html#update-vector-database-version-replace) with entirely new data. ### Select a data source Whether adding or appending data, you select the data source using the Select data dropdown. You can select any data that is associated with the Use Case. If the data you need is not already associated with the Use Case, use the Add data option to open the Data Registry and add a new registered dataset. Any data you add from the Data Registry is handled according to your selection and added to the Use Case, where it can be used in other vector databases. ### Update vector database version: Add Use the fields in this section to select the changes you want made for the new version. The new version is named, by default, `VX`. This name increments by one from the last version created in this vector database lineage. That does not mean that versions can only be built from the immediately previous version. For example, if you have `Parent-vdb`, `V1`, `V2`, and `V3`, and you create a new version from `V2`, that version will be named `V4`, regardless of its basis. If you click the Add data radio button, whichever data source you select is appended to the existing data in the vector database. #### Current vector database configuration When adding new data, the middle section of the window reports the configuration of the vector database this version is built from. This is the same information provided in the Details section on the vector database listing. Note that when selecting the Add data method, you cannot change the chunking configuration. Chunking of the new data uses the same chunking rules as those applied to the data you are appending to. The output reports: - Basic vector database metadata: ID, creator and creation date, data source name and size. - Chunking configuration settings: Embedding column and chunking method and settings. - Metadata columns: Names of columns from the data source, which can later be used for metadata filtering . ### Update vector database version: Replace Choose Replace data and change chunking to replace the data source completely and, optionally, modify the vector database configuration. You are prompted to select the replacement data. When replacing the data source entirely, both the embedding model and the chunking configuration can be changed. Fundamentally, this method rebuilds the vector database but provides you a starting point from an existing version. You may want to do this, for example, to test prompting strategies or to maintain deployed assets. After selecting the data source, configure the [chunking strategy](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#chunking-settings). ### Related items > [!NOTE] Note > The Related items information is only visible if the configuration has associated deployments, custom models, or registered models. The help text under Related items indicates the number of assets related to source vector database that you are creating a new version from ("There are 3 assets connected to this vector database.") Use the radio buttons to set the update method for all assets connected to the source of the new version, either manually or automatically. **Manual update:** When you select to update manually, after saving the new version you are taken to the [vector database details page](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#manage-vector-databases). From there, navigate to the LLM blueprint in the playground and manually [export](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/deploy-llm.html) it to the workshop. Then, register the custom model that uses the new vector database and do a [model replacement](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-overview/nxt-deployment-actions.html#replace-deployed-models) for the deployments that should use the new newly created model package. [https://docs.datarobot.com/en/docs/images/vdb-v10.png](https://docs.datarobot.com/en/docs/images/vdb-v10.png) **Automatic update:** When using the automatic update option, you are prompted to choose exactly which assets you would like DataRobot to update with the new vector database version. [https://docs.datarobot.com/en/docs/images/vdb-v11.png](https://docs.datarobot.com/en/docs/images/vdb-v11.png) Select either: Update all related LLM blueprints to swap the new version into each related LLM blueprint configuration. Update all related deployment assets to swap the new version into all related LLM blueprints, deployments, and custom model and registered model versions that are used by the deployment. ## Comparing versions Use the [Details](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html#vector-database-details) section to compare vector database versions and see the results of changes implemented. > [!NOTE] Check back soon > The documentation, like the application, is "continuous deployment." This section will soon be expanded to contain more descriptions, examples, and images. ## Create a playground Use the Vector database actions dropdown to create a playground from the selected version of a vector database. A new playground opens, ready for [configuration](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#set-the-configuration). The playground opens to the Vector database tab, with the version and chunking information preloaded. > [!NOTE] Note > Although this page was reached from within the vector database details page, you are creating a brand new playground. There is no LLM selected, so be sure to set the LLM blueprint in the LLM tab and also consider your prompting strategy. From the Vector database tab you can modify settings and even create a new vector database. See details on [creating vector databases in a playground](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html). Once the vector database is configured and saved you can send text queries. --- # Feature selection notebook URL: https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-adv-feat.html ## Setup ### Import libraries ``` import datarobot as dr ``` ``` from datarobot_bp_workshop import Workshop, Visualize ``` ``` with open('../api.token', 'r') as f: token = f.read() dr.Client(token=token, endpoint='https://app.datarobot.com/api/v2') ``` ## Initialize ``` w = Workshop() ``` ## Feature selection workflow Note that this section is project-specific. ``` w = Workshop(project_id='5eb9656901f6bb026828f14e') ``` ``` w.Features.Accident_Last2 ``` ``` Single Column Converter: 'Accident_Last2' (SCPICK) Input Summary: Categorical Data Output Method: TaskOutputMethod.TRANSFORM Task Parameters: column_name (cn) = '4163636964656e745f4c61737432' ``` ``` w.Feature('Insurance_Duration') ``` ``` Single Column Converter: 'Insurance_Duration' (SCPICK) Input Summary: Categorical Data Output Method: TaskOutputMethod.TRANSFORM Task Parameters: column_name (cn) = '496e737572616e63655f4475726174696f6e' ``` ``` pni = w.Tasks.PNI2(w.Features.Age) rdt = w.Tasks.RDT5(pni) binning = w.Tasks.BINNING(pni) keras = w.Tasks.KERASC(rdt, binning) keras.set_task_parameters_by_name(learning_rate=0.123) keras_blueprint = w.BlueprintGraph(keras, name='A blueprint I made with the Python API') ``` ``` source_code = keras_blueprint.to_source_code(to_stdout=True) ``` ``` w = Workshop(project_id='5eb9656901f6bb026828f14e') age = w.Features.Age pni2 = w.Tasks.PNI2(age) binning = w.Tasks.BINNING(pni2) rdt5 = w.Tasks.RDT5(pni2) kerasc = w.Tasks.KERASC(binning, rdt5) kerasc.set_task_parameters(learning_rate=0.123) kerasc_blueprint = w.BlueprintGraph(kerasc, name='A blueprint I made with the Python API') ``` ``` exec(compile(source_code, 'blueprint', 'exec'), locals()) ``` ``` kerasc_blueprint.show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABk0AAADECAIAAAC9YfG0AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd1gU1xoH4DOzjV6WKgI2EBRRiiAo2HvvRrHFWBI1iaYYa6Km2BKTqDHRq0axJTZU7J2iIgoKSLeh1KV32DJz/0AMIiwLLC6uv/e5z33iMnPmm3OmnPn2zFlK38iEAAAAAAAAAAAAvONoVQcAAAAAAAAAAACgBMhzAQAAAAAAAACAOkCeCwAAAAAAAAAA1AHyXAAAAAAAAAAAoA6Q5wIAAAAAAAAAAHWAPBcAAAAAAAAAAKgD5LkAAAAAAAAAAEAdIM8FAAAAAAAAAADqAHkuAAAAAAAAAABQB8hzAQAAAAAAAACAOkCeCwAAAAAAAAAA1AHyXAAAAIqi9Huv/PfCnfsnF9lz6ljSdPAPRy/evX94rnVzv9VquC46E/k0Nebsiu46qo6lGeB7Ljl6KfTBle/ceKoKob4tQun3Wnb4XEiYX52HZX01Xcmqon57BAAAANU09843AABAE9Ebuj1RJMrLuLvOg6/oOpqt3Lxd7Mx1eVQdC9Labbt5Odua6XDqWlL1KJqmKYricKh6x8ppM37z0Yv37v8xWkuxFfi9f49Nz8tKPjbVqJlWDNfcsbtT+xZ6fBXG90aL1FHPmq09enW1t9Cr87B8U9OVrCiu3fStR68E3omJT0xNTc0RpYmeJ8SFXj6166dFY5xN659sVP0eNULdF6V6VVfzP90AAACaAFfVAQAAAKgCZTpi2mBjmhBiNW5qn59CLhaqOiJVKbu3eWinzQ1blzZx7NPTuY00FWNjlOjNFmm6elZ9C9ItXPt7dzX575tXvpaBeVsD87Zdeo2a9UXMP0s++uafR+X1KE/le9RgilyUlF1dAAAA6gfjuQAA4H3EaTt+Ri9tihBCaJNh00aYYrgDgAqJA5a6mJubG5qYm1jbOfYaP/+XC4/LKD2HyVsPr+2j/16cnvW5KKG6AAAAaoU8FwAAvIf4LlOnuvDZvCu+x57LKJ3eMye0efdGfwCoEUlZabmUYVlGUpL7Ijrw0LoZg2YdeiajeK2nfDmh5XvQYa3fRem9ry4AAIBa4T4IAADvH51eMye25crST+5Y++vhWAnhu0z1cVJ4ki5CadkOXbzl36tRic9FKY8T757fv3aGh0ndUwHw3NfczxDlpfw97rWZgyiDDw5kZImybq1wrlaGwLLX3HX/XL735HlyxrPY+5d813/YzazqMhrth3363da9JwJCI58+T87OSE5JuH/71F/ff+jVstru8Fp29/liw/+OXL11/0nSi8z05Bexd4P+/dSVRyjzGadSRXlpV7/uUPlYrWk/5qsf/zpwMjA04smzF5mi9KyUxJjAY38sGtRWs6YdEwzb/VyUl/XyfxlHpzdqeFzF1g+eCgqNeJL0IkuUnvUi7v75ncuGtdPUsPSevup/JwNjnyRnpT9/9uDqv+tnuBtV68xw2k9Yd+D0tfsxj9LT0rJSH8fdPr17+ZgOujXEROnajVnyh39wxLMXyelPo+7471w73dWkxuDrbIvX0FZzT4kyRTnRm/q81hCaQ/9MyM1KPj2rRdWguS6r7qaLclNPzLGgCampRV7GILeeabMBS3adv/ngyfPkTPntVcPeKank+tWSfGzWtb8OREsIxXf2dNUkHPvFlzOzRNnhP3hWn4Wq16/R6XmZSf9OFv4XtbL2iGfiPu27v8/cin/yIuN5fNT1w799OqhdtZm/6nu+1KiRF6Xq1QUAAPD+wvxcAADwvqFMh08fYUpLE47sDS6IfbT/9mfre7YbP8N7c9jVYoUK4LabuGxJ5T80TNq4jpjvMnT80JWTPvwzqkRpUQp7LD+w50t3w8qEiFEbl8EfO/cb5rV4xLwjz6SEEEIZeM5dtqBXlYdhbWHLDj3GdugxesbUXXN9vruQLntZmlH/bzYtrbokz6SVvZAqZGratH63mV/MqbowEehbdOzp09FraJ+1IyZuf9iU8//UsHVNYRu30Uv+7jMjk2dmpvUqX2Fg6Tho9sZePdqMG7rmZiFb+TGt79BnSPe2lSkiXXNbj3FfuA/oYTV87JbIKpFzrUdtPbLtAxtBZYFmdp6j7TwJIYTIXg9JgbZ4HZN2MzhR5ukgdHZtw7keX1kct0M3Fx2K0J1cHPl70soqAzZ1drbmEGnszeCMmtpDQRxT9xHDKv/BV2Z7KVZy/WupLowoTcQSQnH1DXQo2eNbt1NkXVq3cPdszbmd+F8LcW3d3Y1oIo29eSeflVNaA/bIwO3LvfuW9TCu/DUJgZVjv5mOfSd9cHj+lK/9kiQvF1PC+dLoixKpVl2kWNG6AAAAUDcYzwUAAO8ZTuuJ03vrkvJ7Bw5FSgiTfHLvpTyGNh81fYixgqOQWHHyje2fj/e2b2Vp1rqL94wNF15IaNPea3ev8NZWUpB0iwm/7PzS3UCSdOGHab3trS3N23uO+fbMMwnXasQP6ye+NhqISB/tnOzSxqql0NTSslPvD749FltE6XWZs2vXPPtqI19kzw7M7e1o19rEzNLKwWvooiOPX0/ovEb6ePd0T3ub1iZmLS06eE/64XKKlDLsvuSnqVbVew/lZz+yNjUwfvk/swm+osY/Y1dsvV0rE3PL1m4Tvr8mYml9c1Mm9tgan/4urSxamLZxGbLszHMp0bCftXxy1ZCkcUeX+Qz1drBtbWpmYW7XfezqCy8klJ7758vHVBmqxbWdt/23D2wEbM69vz4b7mxrbWpp5zJ03g9+sQXVck31aotXQSTeDE6XEW77bl0NX22UbtnNw4pLCK3X1b3jf181anX1cORTsrSbwY/kNAepq55lyce/HuXRub15i7raqylKblAt1YFuYdWCJoSVFeQVsUQScT0giyHcDj27Vx1yRxl37daOQ2QpISEvqlSfEvbIbNwvfy/3MqaKHvp+OdrF1tqsVeeeH/58LU2mYTd5+57Pu1QbbKX4+fKmxl+U3qguAACA9xbyXAAA8H7hOU6Z5iIgpbcO+yUxhBA29+K/F7MYSq/f1PHWit0WpfH7Vn2/70Z8erG4vCgt6uwvM3w23SslvDZTPhtjppQpoPmu85YMMaFK763zmf3z+Zj0EnFZzuPr2+fP3ZkopfX7TB5hWTVStjQzOS23VMIw4qL0mAvbFwz7yPeJlOi4f/b1EMPX4mEKk2LjX2SXSGTiwoyEu9Hp8vIqbEn602fpeSUSmaQkM/7i7wuX++cylGa3YX3fxqz9FVvPL5VIxXlPA35d/EtwGUtY8YNju84+SM4Xy8SFybd3ffXDxUKWErj0dNetsmZh9PULofEpuSVimbQs+9G1Pz5d4Z/HUDrdvF0ElQtpes1d4KZNyZL2zpu87FDo09wycVnuk1C/nz//LUDyWiD1a4tXxBGBIQUMxe/q1bXyJTLKsHuPzjxCCOFae3Z/tRrfydtdh2LybwVFSmoqSVFMblxIaFxqXplE2e2lQMkNrCV5aJMB86bYcwkrjrhzv5QQUhZ6MSCPofhuA3sa/bdHmi4ejgKKyb9zO0rx6lNkj5znfTPMlJalHf980uf7bj3JLSsvTo/03zhl6q8R5USz87wlo17PQTXifFHCRenN6gIAAHhfIc8FAADvFY0eU8fbcNmiwOPnMl6OeSgOPHYmTUYJ3Hwm2jVsNvry2P27A0pYSqv7AC89JQTJcx4xtA2XLQvevz9eXOXzsvtXgkUMxXdw6SKodWVC2Jzrv/5xq5ylDQeM8FLWCDPC5gVeDROzFLedvc1bn/aAyQi9/URGaJ22Nqb/9V3YvLt3EiSE4ra0tpDTcmzhw4jHMkJpmZrqvcw48Lr072vGIZLoQzsD5b/s1uC2KLl1JaSEpfU8vJ1ejqrTdPd2E8ie3w3LkPE6eXkYVMTCbd/Dy4zDFt28FFJWY0EN1HTtVUPJjTxiq6B5GvrmNi79Z37re/F/H1hzWGny8d+PvmAIIaQ40P9aLkNpeQ7q9eonBXkdu3fVo9jiW9dCG159NewRt/PwIW24RJpweOu5zKpHSFnk7j+vFrGUXu8RveX9sGE96r8RFyV51QUAAPCewvxcAADwPtHrO2VECw6bf/X45axXD69lIUf9k2d83KrjhEmuv68OFcsroGZsfmTEU+ngToJ2tq25JKIBUxFVRenatrfgEEpzwJbHmVtqWEDDxMyAJqW1P8wyojt3nsp6dtCy69CG6x/VyHheYovT0/JZYmpgIO/xvonIsjOyGEJoPX09mpDKPWdysrIZQnjaujr0f5NqcU1cJs6dPaFPZxtLC1M9uijjRarMjEMIy+VyKEJYQig9W1szDmHyoiKfyn9VsOFtweYFXg4pG9y/RY+e7bm3oqWE79ynux7J8v/tL62tO8a69+qm9c/5YkJb9vRux2VLQy4HFij3ZbOma683Sm78EUv4/X+Lzf2t+oZKHp9eOWv5hZyXNVMU6Hcha6yPsdeIPgYn/HJZQjjWHh6WHLYs9GJgXiOq78090rPvYMUlTM6DsPhqpw+bH34vUTrURWDX0YZD7tV6cilc/w25KClUXQAAAO8njOcCAID3B2Uy5IMhQprJuXL0Sm6VZ0HxveOnHksJp/WYKV5ata8uB1tYUMQSQmnpaNWdU6DqWITS1tGRv4SGoI7RMUx+Xj5DCKWjq628HAdbXi5mCcXlqeJbMrFYTAghNP3a6BaxREoIIRzOq0+5bafsu+r/x+LxfZzaWxnrCPhaRlZ2jq0NXuvxUFra2oQQtqiwqI6RL41oCzbrxqVwCcu17d/HmkMI16FvH3O68HZAUOCNO6WUgXdfNwEhlGmfvp15rPju5QClZyearr2ql6yEI7ayZFYmLslNexwReOrvHxeMdOs5Z090ld92KA48fDpFRhsMmDzcjCaE0Oa9+zpyWXH4levZjaq+WvaILcyvPl0bIUxBfhFDCK2jqyO3H61Y/TfqolRHdQEAALyXMJ4LAADeG3TLMZN761CEMhp/8On4mpYwGzm533fX/RX+2bb/itY30KcrEie1r8tKpFKWEEqgoUmREjnLlRQXE0KYrANTOi661oDhZYRQuno6FCFsSXFp047tUPnIkdcDoIzHrV4z1IIrS7vx66r1B4PiU/PKKC1T9y8PnVjoUGWl4qIiQgitZ6jPIUTetE6NaQsm9dKFB993d3Ps39t8e5Ju396tOcWXLwcX5vAvhpYP7tmrfxdewGPvgW4CIr597or8n1psunpWRslKOGLFVxZ1mXCgzmRVeYjvoejpSxy9pkxsc2jLE2GfAa58Irlz4Urqf9WnlD2qOEIoXX29N3JZtJ6+Dl2xSOPfD2zgRUnB6gIAAHgfYTwXAAC8Lzjtxn7gLpA7vIk2HDBpWD1+4ezVeqYeHm05hC1JiH1W+0uCTF52LksI3bKVvNmkCGELHj3KkBHawKVr+4Z9IUXpd+7ShkvY8seJcuJpPLZcXM4SQgkE8uv1reE5eLrpUmzZlR/n/XQy/Fl2sVgmKy/MSEorfC0jwBbGxyXLCKXr6tGJV1tZFUs2pi2Y5+dOh0kIv+uQgRbthgy255bdvXAjj2Uzr1wME9OWgwY7Gvcc3EOLlN89fTZFXsqk6epZOSU3/ohVmDT24M6AIsJ3+miOp475kLHdNYj4nt+Z/2akUtIeVRwhtJ6Ti121PaL0nLvacglbFh8t7+dKFdOEFyUAAID3FfJcAADwnuA6jBvfmU/JUn3HWpkaGFf/n1GPn+6LWUq75+SRLet5d6SEfRbP9+RTTN7VM0FFhBBCWJawLCEUX4P33wMqkxITk8sQTrsBA9vJzQZI7l+6li4jXLspnw0xacADrqDD9I96a1Fsya3LwQX1X11xTJYomyGEbmPXpmEz+DcBlhDCyqSM3KEukoizF55JCbftlK8/aNV0bcG8OHfqnpgIuo37eOZwB2556LnLWSwhTOql82ESTpshUxaN66lLyu+cupAqd2RQ09Wzkkpu7BFbD2y63//8UmUc6w8WffXJpO6apDz01JkqWUJl7VHE2QtPpYTbfvKCQa8lmTQ6ffRxXx2KLQjwD2jMlGCEkKa8KAEAALy/cNMEAID3A9954jhbLpElnTwSXFrD32WJxw7eKWMpQbeJo9vKf0CmtMysWwo1eTTF1TLrMOiTP87snt6WS4rvbd149uUMO2xBdi7DEm6HMbMG2An5L5+TxSHHTiXLKF6Xz/78Zaq7tT6fprgaBuatW+hVSw2UBW//NTif4VhM2HbifwuHulgbanApmq9r3r7byGmD7aslZrh2079fNb2XnZk2n69j3mnoF/sOfNVVg0ieHPzdL6NJ32xiROFhL6SE28Zn2fweLbW5NF+3ZefBI9zNVdW/kMSGRZaylGa/L9fN9rYx1uRShOIK9I30qw+ZEYf9te6siKGFAzf4HV42pqu1Pp+mOAI98zYtDV5ftH5tUQ2T6n88pJQIus/50IVXHuJ/ueLtRCb53Ol7Eo6Nz8cD9EjJzWPn0uS/ANd09ayskhtVS/VUHPDnH3dLiY735590FZCS4BPnq2YJlbVH4rC/NpwTMRyLCVsPb57m0dqAz9cy6zTsq/0HFjtrkLLInRtPZTb25FLiRQkAAAAqIc8FAADvBQ2PCWOsOUSaeOxoeM3zBzEpp/4NLGYpfpdx4+3lPlNy280+eO/Ji5SczNTU6IB/vx/fUYctjPp7zuw/YyunemLzgvyD8hlKo/Nc31uXV3Z7+XJc2c1flu9/VE60O8347Uzk4+SczNT0RxE3Vnjwq21C9nTPJx//+SCf1bYbv3rvtfD49PSMnNTHcbf8fTd82q/a2A6Kb9V7/pbjQfFJyaJnkcG+Swdb85jMgNVz1gUXN6Su6kHyYPcfATkMbdxn5dmIp1mi5BcRVw5vW9CzeuLubWFFR9dvvVfIatiO/9nv1qMXqblZGVkp8deXOFd/PZFJP/7FzB8CRDKe5YAvd1wJTxSJMrJTHsUFfOtdrTHq1RbVMWn+hy7nszSHQ8pCTl5Mf5mQYVL9T94pJxwOh825ePC0qK6ESdPVs7JKblQt1ZPs8d51B5/JKIqi2LwrB8+8nsxV1h4x6ce+mLXuZjbR7fLhr6cfPEoWPY8K3rdkQEtuWcI/82f99qC8sfuhzIsSAAAAVEKeCwAA3gfavScMa8FhxRFHjsbUNmEVm3XxxNV8lnDtxo7tUuOsTWz+nX2/7Tl2LTwuObuwXMowkpLc1LibftuWTPAcvPRcapXZepjUQwt9lu67/jA5tygx4Yn01TYufTFs1Od/ng97llMiZVhGUlaQ9Twu7Krfvt//uvSiagGiqyuG9h615K9TIQnpBWIZIy0ryk6KvnV6/6kH1X5RTfrk2MZfDgfEpOaXS8TFWUnh53YuHdHb54/Ips5yEUKYpP1zRyzcfj4yOb9cJhMXZz67f+n4zeeqeyQve/Dr2CFz1/0TEPUit1Qik4nLCnPSkxIiQ676Hz4fXfVXAtj8e5sn9hwwf9PBq5FJ2YXlMlYmLslNfxJ589z+P7afefpfY9SjLd7A5lw6eDaTIWxx0Ilz6a/GHTGpZ08ElbJElnbq4FUFfveg6epZaSU3ppbqq+T2XzvvilnCZJw5fDG3WvUpbY/YvNBN43sP+/ov/7tPMovE4tLclOgbB378sNfARSeS5P16gWKUc1ECAACAaih9IxNVxwAAAAANQZnPOBm+qRcV9WPfgZtiGz0nNsA7gmf7yakrqz2p+98PGbE5uvEpJwAAAFAfGM8FAAAAAM0cZdjKzlKfz9UQ2vacu+PgCk+totvrP9+GJBcAAAC8rol//BkAAAAAoJEo4bANV7b2r/w9AVacdOKreTvia57WCgAAAN5jyHMBAAAAQPNGGwnKnmWWthPSRRmJ987v/33jvjsivKoLAAAAb8D8XAAAAAAAAAAAoA4wPxcAAAAAAAAAAKgD5LkAAAAAAAAAAEAdIM8FAAAAAAAAAADqAHkuAAAAAAAAAABQB8hzAQAAAAAAAACAOkCeCwAAAAAAAAAA1AHyXAAAAAAAAAAAoA6Q5wIAAAAAAAAAAHWAPBcAAAAAAAAAAKgD5LkAAAAAAAAAAEAdIM8FAAAAAAAAAADqAHkuAAAAAAAAAABQB8hzAQAAAAAAAACAOkCeCwAAAAAAAAAA1AHyXAAAAAAAAAAAoA6Q5wIAAAAAAAAAAHWAPBcAAAAAAAAAAKgD5LkAAAAAAAAAAEAdIM8FAAAAAAAAAADqAHkuAAAAAAAAAABQB8hzAQAAAAAAAACAOkCeCwAAAAAAAAAA1AHyXAAAAAAAAAAAoA6Q5wIAAAAAAAAAAHWAPBcAAAAAAAAAAKgD5LkAAAAAAAAAAEAdIM8FAAAAAAAAAADqAHkuAAAAAAAAAABQB8hzAQAAAAAAAACAOkCeCwAAAAAAAAAA1AHyXAAAAAAAAAAAoA6Q5wIAAAAAAAAAAHWAPBcAAAAAAAAAAKgD5LkAAAAAAAAAAEAdcFUdAAAAAAA0F/b2dmNGj1F1FADvtpi42FMnT6k6CgCA9xTyXAAAAADwkrGJiZe3l6qjAHjnnSLIcwEAqAbeWwQAAAAAAAAAAHWA8VwAAAAAUN2WbdtDQ++qOgqAd8wB379VHQIAwPsO47kAAAAAAAAAAEAdIM8FAAAAAAAAAADqAHkuAAAAAAAAAABQB8hzAQAAAAAAAACAOkCeCwAAAAAAAAAA1AHyXAAAAAAAAAAAoA6Q5wIAAAAAAAAAAHWAPBcAAAAAAAAAAKgD5LkAAAAAAAAAAEAdIM8FAAAAAAAAAADqAHkuAAAAAAAAAABQB8hzAQAAAAAAAACAOkCeCwAAAABAIRyz7vM37b9+Oywx5n5UkN/6IcaUqkNSEU6rsetOXzqx3J2r6kiqoi1G/nD68pk1XrxaFmieYQMAgDIhzwUAAAAATUHg8tm/4ffObRhopKRkkNILrCe+w+c7tn010qW1UIPL4euYmAvKi1iVRNJwSqtDXauOHa0MNKimbop6BUxpt7TrYGmgUfuibytsAABQGXyVAQAAAAD1ptHt070rhrUyExrqavKItKwoL+N5QnjQ2X2+Z6NyZRXLUBRFUTStvJSC0gusF0G3iZPt+OLEI18t3nL5SRHfxEKnqFw1oTSCauuwAd65gAEAQLWQ5wIAAACAeuOYtneysxK8/BdfS9+0jaNpG8ceo0Z7fzHlG/80hpDysN8nOv+uxG0qvcB6oc1s2ulT5UF7fj+bmMcSUp6eVKiqWBpOtXXYAO9cwAAAoGJ4bxEAAAAAGkYa8dvoDh06te3o2qXH4JEfrzsWV8KxGLRwoh1H1ZE1AUqgwafY0qys4nftXUVoFFqga2JmYqiF8QEAAO8G5LkAAAAAoGFYWXmZmGFZWVlB1ouo6we+23GrjCW6ejoUIYRwbD85mhgbvMG7YlJwyqjH3M07D164GhgZ8eBRzIOHN08fWDPZ2fDVC2l1LlDfAgkhhGhY9pnz/YEz1yMjIxMjQ+9d9Tvy5/dz3A1qfg1Os/XABRuOXgyKjgqPCvTbu9rH3aRqyo4itOHE/z14Gh/9ND766cO901tU7UvXGQ+vx+obj2P8Ftu/KpPSH7M9Pv7+3vFCqoYSwsKvHNg8y8POddw3m32v3gyNjw4Lv7RvvY9j1egpbZvhizf7Xb0VGxUWfuXQlgW9LHmVhTtN+vbX3f6XAqIiIx5FhYScWTNEWK0OK/bauv8nP/1zPuBh1P3o0GuX9q8e3YpDCKGMei/f5xcUcjchJjI+7Nq5Hd+MtdOu6/VBbtdlFx/FhfwxRLdqxUzaHfYk6u/pLWi5ZSoUcN1RUfz2o1f+fepaVNSDmJALxzYvGNxaU07EtVcgoYVd5/124l7Y7dDAG2Hh9yIubRpngacnAIDmDt9LAAAAAEDj0FwNbQPLjr0/nO0pYLKCguJkNS0k7DxgRK+Or3qf2sbtenywwqm95thpexKkiixQ3wIJ0bCfs3PXUnfDytmdtI0s2xtZtuWH79kTmlc9SI2O83buWuKu/zKTYda+1+Rl3Xt2+XLqMv/UmnaoAfHUrwSeoZXzmG92j6myBL9V10kr/xQWj/v4ZAZDCNFyXLj7f4ucdSti1rDqMuLTLU4Wn41aGZDL0qae46cNfVWarokpr7zojW0K7Ofs2L20m8HLveab2ThZ6ZQyhBAiNWjn0t6STwghRMesQ+/pmzqZSkZ95Z8lZ0CbNCrwdtb0cW7dHQXnb72cvUzb1auzQBYbHCRiiI6cMhULuM6oKG2n4eMr68vKddh85+5Oa6bO930kqSFeORVImU9Yv3VJLz1KWpydUUTpGBmY8srzmdr3HQAAmgV8IwEAAAAADcNz+ebC4/jop7ERsfcCLvuumdxWdGrVvDU3CmtNhLAF55cN7NK5czuHbl4+6y+nM9pdfHxcePVYoB4FctpN/fZLdwPxkzPfTRvs5NjZxrGb17cBJTUHx7GZtmqxm15Z7LElE/s6dHJ2Hjhn3bU0ymLI6iUD/huSxeQeme3Uxs6hjZ1Dm04zfdPeyHrUN/7a98jG0WvYinPJMpbJu7tt4XhPV6f2LgOmbg8rIAa9xvY1pQkhnPbTVy100hIFbJk1tIe9g2u38SuPPZFZjl7oY1M5ZIwtvPzd8K7OTjadPb0n/H6neqqH08Zn1Rfu+mUJJ1dOG+LS2amDW9/B0zdczGIJIWzhzU3TR3u6udp06NzBY/isXZFlRn3G9TaUP6Sr/H7grXwi9PR2rMxXabp4e+gwj4ODn8sUKLOOgBUpQfz03IZZI3p17OTiPPCjteeTpAaeX3813LSGuOVVIKXbbWA3XebhjjGenl179nV1dfcY83NwSd0NCAAAqoU8FwAAAAAoB6XZZtC8BR901q01FcLKCjNFBeUyRlqUcu/QT/sfSjlG9vYmtOILKF4gp+3QYQ58Wdxfi1f6hr7IF8tk4qLM7KJa0ly2I0c58CVRW79cezQio0Qizku6tfOr746ksYa9R/arI7dTnx1UuASZODfGb/vBaEJd5v8AACAASURBVBnFzYq5GZteJJEUp97cuftyPsuxam1NE8KxGzHcnltw5cevd15/nFcuLRNF+a3Zcr2IY9vd3fjlFllpbkpydolEVl6QmpRRfWoxTtvhIzoJJJFbPvv2YOjz3HJJWUFGwv2EzIr0HUUZOH7w057jN+/cjbzuu3qQBYdwzVoY17EvJXfOB+ZRFt79Olbk2gQufXsYso8vXHokU6RM+QErVELx3ROHrydklUrK85JC/l62+p8URttjgPeb76rKr0CWZQmhTOzd7Y01KELY8synyXmYmw0AoNnDe4sAAAAA0DCS8A0jJux5wRCK5msZWdj1GLdw+Uf9l/+ekzB0bXBpnavLUh8/K2YddHRqm/WpzgXkLs9tZduaw7y4daPGF9aq4beysaSZF3duPqvyimJxeND9ssmDW9lY0SRHoQgaF/+bBaQlpUpJBxMzA5qUMIQQIk5PFrGUqZYmRQjPqq0lTWsO2ho6aOvrq1lYtqBJVt3lv6yi0FvP33gxk9LruXLvrsmteC9DF1hbEUJkNF3n40PxzTPXs0eM7t/f/ufIaBm/84Bexmzcv2cTZY0osxFRlUbeiRJPG9iytQVNcl//E19eBVKFt/yuZvUZ1mu575Uvc5MePggLOH1gz4VE/AwBAEAzh/FcAAAAANBILCMuznwWfnLz0i2hEo5ZN6/2Cv3iIisWi1mKomsf/lXXAvKWp3lcmhCpVJG5tQhpYCaqHvEQwhKGEIGGhuLbYiRiKaF4PN6rVSQSKUsoiiaEsGwtKRdKoClQaBsUTVOE1FQMJew/c4w1J/fOtgXjPF2dbDp29fjUL12xqiwJPXchnbQZNLgzlwhcBw80k0WcOf9E1qgyGxFVRf3XdAzJr0A26+zyabN+2P3v1fDnrIVLn/FfbN69cahxPcIFAABVwHguAAAAAFASisfnU4TQNEURoupxL5L01CyGtu7qbkE/fFHX9OHipMfJDN2qW49WnKgnlYkTbRdvZw0ifv4kmVHG18NMYX4RS7e0s9GnHmQroXYkKc9SGEb/1OyBq67XMG+UAqlGScqzFIa2dve04kQ9ey1dRBubm/NJyeX9W6/EiQkhRJKdWVD+xiY4NT5MlN07evrZlDmDRrnsMRjez7Qs5Df/FzJCOAqVKY9iUb2G0u/ez4VPxElPUl41YmXYdVQgIWUvAvZvDthPCEfXftzaPasH9B7kTs6eq0/IAADwtmE8FwAAAAA0DMURaPJpQmiepq5RK8c+H/20dbEzj8m7f/eRQr8v2LSksZevpTEC589/+XqEg5kOX2DYym3sgA78GheWJZw+HSPmOX66edX4zmZaXL5+q+5zN62Z2ILKC/C/kqOUnJ3sSVRsASvw+njZFGczLQ7N0dA1MdRs+EAyWfzFy08Y4xGrN37Ur6O5Hp9DczQMLR16u7VS9KtsWfyFS49lvC6fb107tVtrQw0Oh6djbudkZ0Qz2SKRhGh2G+vjaqHDpQjN09HRqFqsWCJlKQOX3h6WWm8m1KQxJ048kLUY/uHSGQOF+deOX8hiCSF1llknhUqgOLrGRto8mubqtOg8bNkfq0eZkJxr/tfz2ephy69ATpu+4/p2bqnHpykOjystLCwnhGqSYX8AAKBMGM8FAAAAAA3D7bLIL3bRax+x4uQz6/64VqSiiF5Tdnfnz6f7/Ty6y/QtJ6ZX+bzGHJwscf/a33ru+tptwqajEza9/JCVpJxfveGictJchBQH7d8f2/9ThyE//DPkh/8+Fje0POnD3T/u6fPnnAFf7BrwxatPJfc3DpiyL6muEWwVJUTv/mFHz+3zO43+3nf09xWfscX+n/X87PK1I1cXeg/r++2hvt/+t7wsofI/kmPj8lh7++l/XjT9yu3zC9WGQ8menz4QMO+XAcN7yZJ2HQoqYAkhhM2WX2bdFCqB0huy/uqQ9f+tVJ50atWGy7lsDWHLqcDnRt0+WrOqe9WfymRyzl+6q3CwAACgGhjPBQAAAAD1Jst8FPU4LbuwTCJjWUZWXpSTnHDvwqFfF4yfsMg/pT5zLjUhJvPyEp9PNp64+yS7TCoty34SeupKTAlLGLamJFBpzF9zpszfevZeUk6pRFwsSgg8vG7qpKWnU5W3N+UPt8yd9+Pxu09zymSMTFpWmJWSGB54PuBRacMyaWzh3fVTpyzafiYkUVRQJpNJirOSIgPuvVA8c8YWhf0yY+pn28/de5ZdLJZJSnJexIQ9LuBRbM75lXO+3H39YWpBuUwmLS/OFSUnRNwJeZRfEWpJwG9f/HHlYXphSnJaDZtjcy4eOJsqY8sjjxyMKH/1ofwyFQhXfglMdsQl/8CIxNTcErFMJi3NeR55Yfe3EyetOp/xssWrhS2nAikqLfxGZFJOqZRhZKW5zyOv7Pzmo6/PZCpctQAAoBqUvpGJqmMAAAAAgGbBy9tr2dKlhJAt27aHhqrf0BXKZOLOoLVd76zqN/OosgZpAfzngO/fhJDgoOB169fXuTAAADQFvLcIAAAAAOqJNnUd0oUkRj9Nzcov4xq2dR351Sfd+Mzj+1EKDyACAACAdwryXAAAAACgngROH6zfMlSn6tzhrCz1zJ+HEprJi5UAAACgZMhzAQAAAIBaovh58ddD2zrZWpvrC0h5ftqTqKDT+7YdvCNSaI52AAAAePcgzwUAAAAAaonND9312fRdqg4DAAAA3p7X8lwV046qq5i42FMnT6k6CkLUvZ6Vwu+kX1xcvKqjIPb2dmNGj1F1FADqoJlMxztq9KiO9h1UHQVAzZrJvQ8A1AmeOwCaGvIMzUTVftRreS4vby9VxPP2nCLN4vhT+3puvKCbwaQZ9PWNTUzQWADK0SzSXKSjfQec1NBsNZN7HwCoE9z1AN4C5Bmag6r9KFq1oQAAAADAW+Do6KihoaHqKABAOXg8nqpDAABopmqYnys09O6WbdvffihN54Dv36oOoQbqV8+N5+7u9tnC+aqOogZbtm0PDb2r6igA3j2fLZzv7u6m6ihqMHX6h6oOAeClt3bvW7ZsmZ6ebkpKysOHD+Pi4uLjE5KTkxkG87EDvJMmTpzQ3tb2n3//jY2Nq3NhPHcANIXmmWeIKtQ6mG6i6ijeHkfdEh/zzGofYh56AAAAAPWXnZOlp69naWlpYdFi4MCBNE2XlZUlJCQ8fBidEB8fnxBfUFCo6hgB1ISzs/N3360SibJycrIzMzOzsrKzs7MzMzPFYrGyNmFkZOTatWtXN7fo6OhDhw4/ePBAWSUDALzrkOcCAAAAUH9pKWltWrchFEXTnIpPNDQ0HB0dO3TsyONyCSGZItHD6GixRKLSMAHUQW5eXklJaZs2rd26dhUaC3ncl+8YFhQU5mRni7Iys7OycnJyMkSinOzs7KxsUWZmWVlZvTZhZmZGURQhpIN9hx9//CEpKenY8eM3rt/AIE0AAOS5AAAAANRfhkgklUqrzelDUVRFkosQYmJq2svEhKYoQghLiI6ujgqiBFALz54+3bTp51f/1NHRERoJhYZCc3NzoVBoZCRs0aKFg4ODsbGxlpZWxTJiiSQnOzs9PT0nJzcnJzs7JycnJyc9LT0nJyc3N5dl2WqbMDM1rfgPmkMTQqytrL5YvPiDSZP++fffgBsBMpnsrewoAEBzhDwXAAAAgNoyNDAwMTU1NTUxb2FO05ScJRlGRtOc1NQ0C4sWFCFFhUVvLUgA9VZUVFRUVPQ86fmbf9LT1xMKhaYmpsbGRhX/YWRsZGNrY2ZiIqj84YjysrKMzMzKkV+i7Oyc7OxsoZFR1XIomiaEWLSw+GLx4pkzZhw7fvzCufMYngkA7yfkuQAAAADebTRNGxoampmZmZmamZqZmJiYmJqampiampuZ8fl8QgjDMEVFRRxOzR0/RiajOZzExEf//HOEL+AtW7r07YYP8P4qyC8oyC949vTZm3/i8/lCodDc3LxiLJjQSGhkKHRzcxMKhUKhsMbSKJoihAiFwjlz5kyZPPn0af8mDR4AoHlCngsAAADg3cDlcvX09IRCoXkLc3Mz8xYtzCtegzIzMxMIBIQQqVRaUFBQ8bpT6J07aWnp6enpObk5GekZLSxa/LFtW7UCZTIZh8NJfPRon69vxIMIQoiXt5cKdgwA3iAWi9PT09PT09/8U7t27bZs+b22FSmKogjR1dX18ZlS8QmHi4c+AHiP4JIHAAAA0LzwuDwjY6NX4zgq8lnm5uYmJiYcDocQIpFKsrNeTuXz6NGjinxWenq6SCSqbRZqUYao6j+lUhmXy3n06NG+fb4RERFvY68AQEl0dXXl/JVhGMKyNIcjFosrRnRqaAjeVmgAAKqHPBcAAACA6vXt09e7h5eJmYmpqZmhgUHFh8XFxZmiTFFmxvMXL+7dC8/MzBCJMkUZGXn5+fUtv6SkpKy0VENTs2IMV1RU5IEDB+Li4pW9HwDQ5ExMjBmGoWn61SdSqZTDoSmKLiwoiI9PiHr4MCY2JiE+4dSpk4SQ4qJi1QULAPC2Ic8FAAAAoHrW1tZPnjyJiY4JuB6QnpEhEolEmZnFRcqcDD4zK8vS0vL+/fADBw4lJibKX3jIoIEe7m5K3DoAKIuRkTFFUzKZlMPhsiybnJzy4MGD2NjY6JiYrMxMVUcHAKBiyHMBAAAAqN7efXuDg4KbdBPnz51/+DD68ZPHiixsa2vTpMEAQINpa2tHRUZFRT2MjY2Ji4svLS1VdUQAAM0I8lwAAAAA74VTp0+rOgQAUILdu3erOgQAgOYLeS4AAAAAeCk4KHhY0HBVRwEAAADQQHTdiwBA88JpNXbd6Usnlru//Ty1CjcNdeL1XXfrScyF5c5voXXe5rZAtXDWNxBl2HvV4TNB6/urOhAAAFA6Nb450hYj1p685L/Gi6eM0t52RXHMus/ftP/67bDEmPtRN7ZNsUK6Q2UogeDTQUZHuwv4qtg6Gh6UQuDy2b/h985tGGhEqTqU5kf5laNr1bGjlYEGpYLKVuGmoU5SmYyt+D/12laz18hzvLlfP3HWNwylYdHRsY2Jlvo9AgEAgBrfHCltyw4OVoYaStqzpqyoN3pQfIfPd2z7aqRLa6EGl8PXMaLFBSzHdvrBm6E3t461Rubj7aI4HFsjrpBb0fZUpy7CM5OMl1rTb+ecUUJra3h8fvTs5Xt3w+Jjoh5F3b0fdPbUrg0rZva31+coXAbH4cPtF676LrBTfBX1pz1ia1zcvTNfuxtUPxZ4PX8IehxzdmnnZlRdFEVRFP2WDtvmSKNVn9kb//a7fTcsMTr84a2L/ns2fDOhi2HFad3cKofWsx88b/3OI4G3Q+NjHkTfvnhm78blPp6WAlUH1jDvxu68hascW1RYxLLFBYVVN9tq9uGwJ3Ehu8aaKPUArGlb6oRjs/BExJPIIwvtavwyU9tz5flHceG7R+tW/LuR53iTXSK0B2wMehx3z3eSWQ03e57T8kuRT6J8P7Rsdv0+7RFb4+Ij/D9pp6KbHPokAPDeqenCy201clPgw6ioo4s93ngaeXtRvTtPQ68oKex352akuq54tR6UoNvEyXZ8ceKRT4d72Xfs4thv1YUCllAUTVE0p/k8iamewFznjxHGpyeaXvMxC5hiemG88d8D9D+1F1g28XdzlGLpJ9sOBvtGG04zbNS2lLArHBMbRxuLl4cxR8vAtLWBaevO3sM+/DjSd+XXP11JkdZdBm3QqqNti1w+Dr9qKE2HWZu3pM2YfeCxWNWxyFUe9vtE599VHYXKcFpN2HxsTU/jygso18iyU4+WttwHvscjCNu8KofS6zz751+X9DTnVp5ufKGlg6dlRyft+HMhyeXv2PCcd2d33sJVjklPEzGS/NRM2auPNN2mTe+iQVGCXh9+0PH01mgFLscN3pZaoY3NTGhK0GH24uHH5vulM6/9kWM7eckEKw4lMzQWckihrLEXwKa7RBTfPBeQM2K0+/ABFkcPJL++FwKXYUMt6fK75y+kMrWs/t5CnwQAgNNi0Jq9Pw42Stw/d+5vIXmq61C9M09Dr1NC2O/GzaiurniTbrxaD4o2s2mnT5UH7fn9bGIeS0i5KJsQQhL2Te6+r0njeOdwNLn2+pyXrxNSlLYGx0aDY2OmMbJ96fdXCgJLmmKb7MOInGERiixJ6evyWmszjXzbUVnf4kpj/pjQsWOnth1dHHsMHfPJD7uCU6QGXWb+umOFp27zPjeVxtjYeO3atf3699PS0lJeqayM1fP65rdl3fXfk2p8O4xNTNauXdO3X1/lNBbXacZ8LyMiuvbLvIE93O0cnB29R37wxS+/7L2R0dyeH+kWY9f/sbSXGZV9f//3nwzr42HfyaWz98hJizbt3ed/M7/5ZIUUo2a7owhKt8eSo+EhR5d313vjmsBkJSUXiF68KKv8gDYbOWtEy5KbB08+p2wmzO795ioN9sa2VG3hwgVTpkyxtLRUTnECYzN9SpxXwPGeO8dF47U/UQYDP57hWJ6Xx1BCoaGcKqUFuiZmJoZvvLZW2+dNoeTOuUsihu88bJh1tW+DNboN79+CLrtz5kq6Ki5Tb7MSAACgnmiTXsv2bhxp/vzop3N/vpnbhB0qBW4H7+jT0Dsadj01TVe8oT0oSqDBp9jSrKxi9XsGWLVy5dixY4xNTJRYZmJkdv+DGT0PZgw4mjXreuGZHJavp7moM1+j7lXfAUp7W4GRlItlLCsrL8pKenDt8I+zJ87cE1fGaz116TR7DiGEUEa9l+/zCwq5mxATGR927dyOb8baab922nPaf3Yq8ml89NP46MdBq7rzFFur2aBo2tXV5YvFi//55/CKlSu6d+/O5zV+9j5pxP6t57NbTdu4ZrRFbWNWeT1W33gc47fY/tUClP6Y7fHx9/eOF1a8NmfUY+7mnQcvXA2MjHjwKCYs/MqBzbM87FzHfbPZ9+rN0PjosPBL+9b7OFYdW0tp2wxfvNnv6q3YqLDwK4e2LOhlyass3GnSt7/u9r8UEBUZ8SgqJOTMmiFCju0nRxNjgzd4V9llTev+n/z0z/mAh1H3o0OvXdq/enSr5jLslqaIq6vrl198cfjwoRUrV3h292xUY2lZtjLiMM/PbNkdnJhVLJaKi0SP75z9+3/X0hhCCKlWOdWa48HDm6cPrJnsXO15WcOyz5zvD5y5HhkZmRgZeu+q35E/v5/z5vDnSrW31+uRdv/4q95CknV9+eSZ3x4IjEktLJeUF4oeh57fu/bXCy8fdzVbD1yw4ejFoOio8KhAv72rfdxN3tqxp1jlKG135GyO23XZxUdxIX8M0a1Sy0aTdoc9ifp7egtabp3XeI5QhNRylatXOSYdBg6zMzS0HzHA/s2bvDhgRdeBm8IrB21x2o2e2kNTdMF3w59HI6TCgT5DW1S75Nd1mMk5rqptS+UsLa18fKbs2PHX9u3bx44dY2xs3JjSaKGxEc2Izv25/1GLiZ+MsKhSb1zbD+YPFNzeviuknDY0NqAJeeMcJ7Sw67zfTtwLux0aeCMs/F7EpU3jLOjaP2/KS0TpvVOX0hhex1HDXn8HUNtzdD9jqujWyStZbP1utXWe9YTIPXJqq5za1fsyIvT8aOOfvueuBEVFRjyKCr178eC2L0c5GrzaikK7UM+ztWLXuvis/PNcQEjcw7Dwywe3zveq6X1RAIBmjhL2+Prv3ya1zjj9xewfr2X+92VIfXs+8m8uCt8OFHkakh9e3f3VBsRfl8aHTQh582bEcVzs/yju7p/DX3VQafMpe+Jjgzb0fPWWIMdx8elHsUEbe2kQUkdPuKYdrxYgbdxz5cWIqPv7P3J8Y3iAQl3xavvboKNCsR4UIYQitOHE/z2oqLGnD/dOb0FTRuP3RUXH/TlKr+46r70Pr2q2du0/+uijvX/v+eXnn4cOHaKnp1v3OnVhGSJhCcuSsnJZYkrJppvFiQwRmvCtKaJrrPmZt+HuUSYXJpvdmGzqN1yv4miieNx+Tvo7RptcmWx6ZrRwtaPAvMpZS2vwRrsZ/D3W9OoU0zOjhKs7842rVF7rTsLrPiZLLap8xOV4ddLfOtLk4mTTyxNN9g/QG/hqtyjuzGFmQVPNgqaaBYzTc6l/h6rJvkpl80O2bjg6aNd02yHD7HfERsuI1KCdS3vLivFnOmYdek/f1MlUMuor/yy5+daGraVSHA6nm5u7p4dHeXl5yO2QgMCg8PAwqbSBj4OylPPLvtRts+fDtb/Miv/wfzENGT1BCzsPGNGrY2Vj8wytnMd8s3tMlSX4rbpOWvmnsHjcxyczGEKIluPC3f9b5KxbcURpWHUZ8ekWJ4vPRq0MyGVpU8/x04a+Kk3XxJRXXvTGNgX2c3bsXtqt8sGCb2bjZKVT2txGNxEul6uExipNS86V0Vb9pg05nnAmqbSOpas1B9E2btfjgxVO7TXHTtuTULFlDfs5O3ctdTesfNtc28iyvZFlW374nj2heTW8Jyavvaoup+E5or8pLb6/e9Px57Xso0bHeTt3LXHXf9lwZu17TV7WvWeXL6cu809twBtq9T32FKgcZe6OvM1FBd7Omj7Orbuj4Pytl2OutV29OgtkscFBIqb+50gtl6z6lpMfd/l84oix5OyVuLoOU4HLxLEdqKd/HQ4pTIo9GDzv557jx9kc3ZpQ2Y51HmaKHlfNi7W11cyZM2bNmpWYmHD9RkDgjYC8/Pz6FkIZCA1pNk90x3dP4JSfPpzlfPqHsHJCCKH0+s2dYp/lP9MvfshsVkNopE0RcbXaoM0nrN+6pJceJS3OziiidIwMTHnl+Uytn5Nq3V/lXiLEYSfPPPb5uP2woQ47EiJfHjSUfs/hfQxJ7plTVyvaUrm3WjlHDlVbJchR78uIkdPgMX1fLc81bu00bG6X/oPcFk379kJjBtnKPSMove4rfbfOtH05b6/A2mmoNSGENO0bGwAASkbru322Z8tUu9yLX8/+7nxalVtKA3o+mrXfXGq9J9ZAoaehRnVa6h+/ApombFl8SGjm3AldnO15Z+5KCCFE07lrRx6t5eTclhMYKyOE0MZOTlZ0aeDN++V19oTreqyj9N0+2/XrJIv4/320YE9U9dfZFOiKv0lOl6OxPSjFKKsP/3ZV9C4oirK3t29v137+/PlRUVGXr1y5fet2aWldT58Kb+NVCsrIXHNMK15lPVBCLSKWEMLlzehr+KEJVVF1Ah1evy4GHbXz5oSU5xNC8fkL+xuMN3j5iwN8XV4fXUIIqfW9XQ73gz6Gn5jRLw9ODtXKmKOlvG/Qm/KrxtKIG3fyGbplB1ttQghbeHPT9NGebq42HTp38Bg+a1dkmVGfcb2rfEMtS9gyqnMbO4c2dg7tvL+/JSEKrdUscbgciqI0NDS8e3p9++2qQ4cPffrppx0dOlIN+aUJtihs26Jfwxin+b8udmv4W6BswfllA7t07mzj6DVsxblkGcvk3d22cLynq1N7lwFTt4cVEINeY/ua0oQQTvvpqxY6aYkCtswa2sPewbXb+JXHnsgsRy/0sam8mrCFl78b3tXZyaazp/eE3+9IqldAG59VX7jrlyWcXDltiEtnpw5ufQdP33CxWWYnldBYkrA924KzqFbjfva7dnDtx4PshXUmkCubo51DNy+f9ZfTGe0uPj4uFV8lcNpN/fZLdwPxkzPfTRvs5NjZxrGb17cBJbVWngLt9XJBq47tdWjZ08DglFpSVhybaasWu+mVxR5bMrGvQydn54Fz1l1LoyyGrF4yoOFnXT2OvTorpwl2p5bNld8PvJVPhJ7ejpWtqeni7aHDPA4Ofi5r6DlS/SpX/3LEBcHrx7m4j/vhZoH804nS854y3EIWceJorJSw2RePXM2i20+Y+OodvDoPM4WPq2aGoigOh0tRVHtbuzmzZx84eGDdup/69uurqampeCG0gdCAYgvzC0QX9h5LaTn+w4EV30dxrMfMHqATdfBASFFBfiFLGwqN3riLUrrdBnbTZR7uGOPp2bVnX1dXd48xPweX1Pp5zZR3iZDF+R+PEtOth4xyqpzogDLsN9JLn804d+JmxQ8JKPVWK+/IqV8l1FQhil9Gzi3p6+Dg2K6Th9ekb/53L5fXatQPS/rVY5fqd7ZyO320dLoNv+DB/kXj+zp0cu7S12fRrrtZze7LHQAAOSiBrc+2bbMdxSE/zFtx8rXMRUN6PnJuLvW8HdT5NKSMTkt94lewRCWE/cYDsjjydmgRbeLatW3FIjwHDxctitBturq8HESs7ezhwJM8DLlTRCvWE67tsY7Wd12we/ssmyTfj+dtDX2z41l3V7ymSqn/UVG/o4XJPTLbqaLG2nSa6ZtW7U7c+OdcVaMITdMURXXq1GnxokX//PvPmtWrvby9eA19OYmiKG0NToeWWl9317ahSV6W5PnLpmaD72SP/EfU+3DmxPPFETLS1l53ugmVnVK0xD+z3yHR6PMF5/NZ87baowwIIcSuo+5YA6ooq2Tt+cyBh0RDTuasjRbn1P64YtVeb7YZXZ5X+svlrOGHRf2PZM64Uhj4Kh3MSveezfA+kOF9IKPX8YLw+neomnRIvTQnJ5+laC0dLZoQQlEGjh/8tOf4zTt3I6/7rh5kwSFcsxbGdUTQsLWaDQ6HS1FEW0urf/++mzZuPLDfd+68ufUvRpywf/naa4U2035Y0eAcHysrzBQVlMtk4twYv+0Ho2UUNyvmZmx6kURSnHpz5+7L+SzHqrU1TQjHbsRwe27BlR+/3nn9cV65tEwU5bdmy/Uijm1398qaZ6W5KcnZJRJZeUFqUkb1d6A5bYeP6CSQRG757NuDoc9zyyVlBRkJ9xMym3ePv1pj7ff17T+gv2KrypKOLhr78ZbTMcVGruO+2XIs+PLeH6e5mcmdaqCiORhpUcq9Qz/tfyjlGNnbm9CEEE7bocMc+LK4vxav9A19kS+WycRFmdm1f5WgSHtVoLR1tSnC5Obk1dISHNuRoxz4kqitX649GpFRIhHnzhIwqgAAIABJREFUJd3a+dV3R9JYw94j6/OAWPPO1n3s1Vk5TbE7tW2u5M75wDzKwrtfx4rbnsClbw9D9vGFS49kjT9HXkaopHJqQBkPGDvAoOzW8XPPGUIIKb55/HQyaTl0rLdOxabrOswUP66arZr6AcYmCr3PKNDX16KZosJipvyB74H7/N7TJ9tyCNFwnz7FqeTqrmPPZKSksIilDIQ1vEvMsiwhlIm9u72xBkUIW575NDmPrf3zGinxEiF7fvrEvVLaYvhYj4oXDugWg8Z112aSzh27W9mbUOKtVv6RU69KqKlCFL+MFOXklEgZRlKY8uDMuvmrT2USYb9RvRs8TYr8/eLYDerfmi4P++3LjaeiMkok4oKUB/4HLj1S099pAAA1xbEdNsHTgOREXL1Vbf7whvVY5Nxc6n07kPs0pJROS73iV1QThF1y7/rdEk67bt3MaEIIx7abh3FBdHQK3ambuy5FCBF08XTTlkUH3sqkFOwJ19jVpIQen+/dMbdD8sFP5vxS8xxtdXbFa9SAo6LBnYc3NWHf+22jaZqmaS6H4+zstGzp0n/+Ofz1V1/Vq4T2TkYBU80CfUwvjDfe2Ud3uJCSFZZtjSx/2Ttk2fxiWa6UlcmYjEJZCcXr15rHEZf9cbP4dj4jZtjs7NLfI8tLaK6rGYemeD2tuLRMvCe48HI2U8qwRUWSq/HlSbXVHsXt14bHl0n2BhaczJDly9hyMfM0UyonL1ZfTToFLFco1KdYpqS4hKX0eq7cu2tyK97LE0pgbUUIkdG03AAatlYtvLy9znqfacCKSsHl8gghBoaGo0aOrPhEX1+/HuvLUk98t6Z7x1/Hr1kWELWquJHRyNKSUqWkg4mZAU1KGEIIEacni1jKVEuTIoRn1daSpjUHbQ0dtPX11SwsW9Akq+7yua1sW3OYF6G3njewj79s6VKytGGrKkFFYxkKDd2EXSs+aWVtHRp6V+5K4uTAnZ8H7lvnOnT6hzOm9O06ZcXu/l4/TFlw5HHdwy9lqY+fFbMOOjraFHlVe7duPFLs6wO+/Paq8pYOW1JcSgitb6BPE1FNTcNvZWNJMy/u3HxW5a/F4UH3yyYPbmVjRZMchSKSR/6xV9MKr1VOVU2yO1U3l3rzzPXsEaP797f/OTJaxu88oJcxG/fv2URZXXWuwDnyMkIllfMmuuWIsR6aBVePXKocRSmOOHoyccanfSb0E145lcPWeZgpflwp5uzZpr38FhYW1vYnmqYJITQhXd26vvqQy+XKeUNZV1+XZsXFxWJCmBcnD1yct3nKNM+/txjNGmn+4ug3V/JYQhUXFTN0K7035/ZnC2/5Xc3qM6zXct8rX+YmPXwQFnD6wJ4LicW1fV73Hb1xlwjCZFw4dnWxx7ABY/ptDPLPo9uOGOMmkEb7+T2sqACl3mrlHzlUwyuhakn1vYwQNv/W1XDx6P7W7VrSJK8hu1XHGcEzad2SZpLD71X/3hgA4B0ijTu08ULLmfN7rzjqa7144S/XMyp7UA3osci9udR6r5RzO5DzNFRHeA3qvyrr5qj0sNn8m9ful/V17eVhsP9EvlX37m1K7yzZnrtky+Cerponr4kdvT2ETILvjWQZv3/DO/a0Qf/ZMwiTd+3wwdvZtdza6uyKv6lhR4VSOg8Vmqbv/RbyDJLae60cLpcQoqGh0btP74pPjPkSmiKMYvXDMGypmEnLl0SklJ1MLH9WW++Sw7HSITRXY/VEjdWv/8VUh6ZpuqU2YYokkQrmKWhOaz3CFInDa+2/N1ZT5rk0u/Tupk8zSXGJxUQ4auYYa07unW2rNh4MeZxZyjXut+LkbyPlF0AJ+zdgrdrExcX5nTzZsHUVoaevt+CT+XIWkMlkHA5HJBKZmpoSQvLrOWUMm3Xt+2+Pu+4Yt3pF8KbXX8JlCUOIQEND8a+pGYlYSigej/dqFYlEyhKKepVKrwkl0BQotA2KpilCaitGAX4nT8bFxTV49Trp6+nNn193YxXkF+jp6xFCkp4/V6zg8vQwv41hp3Z0HLv2t5Ujen7+ad9ziy7V/co0KxaLWYqqmGuH5nFpQqRSRXOEireXLCXxaSlr18bDzWR7Ys1TQyq4zf82rtxjr8ZNVK2cqppgd6ptriT03IX00VMGDe68JTrGdfBAM1nE/vNPZMo4R15uTEnlvIFjM2Kck4DmDt1+d2i1TXqPHdTC/3BqnYeZsmNbt359/Veqh0kTJ+rq1jolJ8MwFEVJJJK8/DxTE1NCiPxp+HT1dCm2rKSMJYSwBQF/n0ga7jPzK1bYi/9g3eFIMSGELS0pYykNXV0eIdU6BGzW2eXTiu5PGOLRxcXZ0aVPG9fefezpsQvP1vZ5bp1716hLBCFsfsChs2lDfbw/GNri7DHTiWPtuSW3Dp182e2t7622jrNe/pFTe+XU54ZR78sIISzLsK9uS/W/cNW1XxSHJoTUcJ0CAHiXSEUh274/e/PjzX8tnP6nr8HXH33rnywlpCG9gjpuLg26HdT6NFRXeA247CvxObTBYddWXnbw9fDyHu59PPRPhXn3tJPc+yfwVq5n7sSePbsIAgr6eLcgj09ffSoj/EbclNii8Mthxr179vl2988ls748U9ObiXV3xd/YsYYeFQ3uQb2xX03S927qPAMhZMGCBTw5Hd3/t3ffcU3c/x/AP3eXsIckECIOtODEgQtHXcWqX2fde9S2dlirft21atW2Sqv1Z9Xq9+tqHVW/zrrqwlXQCm7RIEMUZISEhE2A5O5+fwQRIZCEFcTX8+EfPM4b77vP5+4+987nPseyNMPkZGfb2dsTQlRaoSlJrsj7qumPdKb+QMeT0lZpzVDUyzaYyb0dC0YIqrreclWW56Kcu8xcOLoerYs8fyacpb2lUiuSc3HvpsAn+YQQolUpM4p0h+V1Oh1P7OzsXqtdtGvZS5knRZkSHBRc3qWNc5NIyBcGput0WoFAmJGRcfXa1aCg4HBZ+OnTp8q1BT4teP03B/x+H79gTrIdRTJeTucy07N4ul4zb2fqvqoS6oo24XkCxzmf+KTfsisG3nw24S13bcLzBI5u6Ne1ARP2vDxdup48eVKlhSWRGP4ma7HCEovFixctMn/1XLrs+IZDIwcuaOntXZe5EGPe0lp5YgpHN+zo50E/elHGlYdh9KevkfIqKuefSzcz+vfpMn1Wv4tLz5V8jzQ/9mk8R3t2fteTCYt5WXD27Xu0syH5cTHx3Mtr18tNV3rdM09l7U7pcm8fPvl8wvT+H7TfVWdwH0nuzQ2nXrCkfOeIoatcxc81w4Rthw1pZvjqTll3GPZBo0Nb44xVMzPqlUmq9IwmhAweNLjkRJ7jeJ7nCX//3oOrf1+7cf3Gv+fM0ee5yubk5EDxeTkafaXWPjp44NbkJVPH8qlnF/wZrz9c+ZpcnqfsHR0oUvL45L64tnf9tb2EMI7NR67ataJv7/5+dmf+yjY8/bx5u2rqJeK1gG797/iTcV/6jR/VJbXeiIaU6uShvxQFp6xpt1qTz3qjNae0g2PanpSTbWu/VlYkP+F5AkcIMXbhMv9sZVo+jefoRt16e/0aFlnTRvIAADAHl3Z7yxdj5d/vXDV0/R4rwYeLj8fpytFiMX5zKc/toJSnIWPhlaO9aubNsWzlC1tg8AGZEMIlXz59e37XLn17eTn2bcPf+uF6ak5O4PWMkb38O/2Z8X4jPmLzhUi2Yi1hXht9eO5nh+bs2TRp6PcbkpI/+ulWZolDZ6wp/krBgSp/raiUFhSpqrZ3VecZCCGffWpg7COe4whF6Vg2NCQkMPDyvXt3//zzOKlQb5PScWxCNuGsNItPZPxT8sdiShiXRWgnqy7O1BNT3inl2IRsQjtYtXckERnF/o/X8TwhlG3FMlWVNsoKLRAyFCGMlb2rZ9v3xn2z49BvH7ew0T7fH7AnnCWcSqHQEtvOIyZ28HAQUIQWOjjYFImcV8iVPFO37+i+jR0EjI3Iq6OPB2N0qRqN1ekIIbm5ucFB11eu/G7SpMn//c822WNZqUlkU/CZNzas2v/C2cOj6O8RbExYeAZv3f3zrye0c7djaMbG0c2ltLc3TAk94vzFGM51yIqfPu7TUupkxdCMjUt9n96dPE09+GzEuQtPWWHb2ZtWTercyMWGYYQO0ma+zUqO2FwzVLSwrNp9unr+FP9WDV1sGIpibEWNOw2fMawpw+uUCrXZL7Howi9eTuKs283+ecEQH3cHK2sXz04j+rawKjJLvlbHU3Xa9+5S344xp7z41HNbdzzOoz2Gbji4ZcHwTl6udgKasXKUNO08+PN/D29BIk+elOULW3+1ftmoNu52Aitnz26frl05pi6Vdu1UoJovvulKr3vmqYTdMUYnO3bsPlt38LTFU/uJ0i8fPaf/lbE854ihqxwxfz2U47sLD9+9eXhJt5IvzL1k02H4wPq05saSnq0LhuEs+Oc79D/RnLDFsCFNGaPVrOLXAcvheZ7V6Xiej4qO2r5z5+RJU5d/++3lS5dzc039YK2dgx1FNLl5BZWESzyz93Iaxyae2H/l5QgVXG5OLk87OjmUuKwxjf1H+rep52RFU4xQoMvMzCOEoghV2nRzd8+ES0RJbPSxP25qGO9xG5b2E/Fxxw8GF/YTN3qrNe+sL7vmVNZBMIqy9xs14b0mYluB0KlBp6mrV46vT2eHXAxK5024cJl/trIRp06HawUtv9z84/QeXiIbhmZsnF1dSjybAAC8EfKeHvt6ytdnk6X/Ctj2jb+YKkerwMjNpdy3A4NPQ0bCK0971cybY1WEbfgBmRBCeGXgmVCNQ/ePVozuRO6cu6riSc6Nc0Hp7u//e/EgL/7J6XMxLCGErVhLmGdTgn6aOvdYrLDF9LXL/lVipFzjTXGm+IEqZ62oxMbDm9y+LcRzHMeyLKu7e/fe+v/7v3Fjx61evSY0NESrrcpf2njttRc6ztZmzrv274oYB4bQFOXsIOwiYQSEEF4b+Fyro4WTezmN8xDUYQhNUY62tE3pa7sap2MZ4bSeTsPcGWeG0DTl5iJ8x4YQQlQ5HEcx3b1tGggJw9CeEqG7+YVdWQUqaDnzaMTM12Jn08J2L533w40MnhCiunzo0sweg/yX7/df/moeNvLlH3HXLstmtW4zYt3lEYQQQrT3Vw+cvP1F2UvVRPrMiE6nu3kz5MqVK3fv3q3cCsdnhq7/4bj/f0bVLzIxO2jv3vD3v/IZ8P3BAd+/mlzqRzyN0T3a+cOu97ZO7zt3R9+5hVO1937qO2F3rElpG93jnd//t+eWGa2Gfbdn2HcFoWefmtVz1gVTnzOrwavC+ufm5StX790rZ2EJWvaZMGya58hpxVavidqz/byaNzubnHtr27qTfdYNaztl47EpRaYXZs7Z+PAnaXzz5lO2npfM7zT7nBnlpX2yddYiyZYfJrboMSOgx2uvbuoe0ydObtm7akPPHQs6jV57ePTalzuiTTi74sfzat7Apiu77pmportjHBt3ct+1z37uO7gXG7tjf9DLT82U4xwxfJXbYe56mOb9BjVzcaGG9G2+9kaowfpq13XIv9yp9PNHzyqKF7/s2PH7H833HTTYd8taY9Ws4tcBC9C/cRwTExN46VJwULBaXc4h5ezsbSk+L6fw5QI+/ezc7l5zi87Ca3LzeMreyaH4spS488crl3Ur+tEbTn32wq0ccR+D083vx2T0EmEIl3xq7/nZ3Ya7u/K5tw/ue/DqJOWN3aDNPOvLqjlxpRycyu/MRVk1+tfCXf9a+Go7aTcD1p3Wd2IztgvlOFvZyN0r1nXbvtiv/5Id/ZcUCaTcndABACxKF3dq2afukgPzRq1fHzP6k73mtgrKvrmUdq805XZg6GnISKOlHO1Vc2+ORvu+mx92KQ/IcRwhvCrw+KWFPYZ2aJ5zbfmlFJ4Qkn3z3KXUwaPbUZqbu08WvE/DRlW0JcwpL6+esaHRoXkDvv/uZtiMY/FmPlmEx7x+oOaUp1ZUXguKvKHtWz1e/34iTcvCwwMvXbpx/UZWVlZ1BhD5OPNwvTrjGjgENHjV/NUqMydfyEngybMnGdvrunzubvOlv82XRZYq7TSLkmUe8KgzSWw7r6/tvIJp/KW/lSvi+ISEvKg2whZezvu9nAkhhNP+ekp90MyRvCqhfw2rjA57mqTKzNWyPKfVZKS8eHTj7O9r5w7tP3HlxYSCdjevPrt0+rydVx4lZuSxrC4vO1URH/kg5GZ0uv4UY6N2z1rw25WolByW1eWoYu5FKynK6FI1Dcuyd+/cXbfu53HjxgcEBISEVEVWlU8P+mX1X68/v+Y92vjpZz8cvfVMnctyrC43MyUh6u7fZ69Fa8p3oPjMWwGTJszZcvpmlCIjl2W12SmxD6/dfmF69oLPuvPz1Emztvx1+7kqO5/V5qhfyO48zRDWnN+2Cwtr7LjxAT/+WJEUOBdz8sf1f5wNjUxMz2U5TqdJS4i4+eeWxaMmrLtRsoevKStUXlw48Yufjt2KUeXqdLmqmNATgbIcnnB8QbHnXNsw99fAR/LMhPikfDPLi00MXD5uxNTv9567+0yRkcuyrCYj+emDoCPbD1xP5YhG9p/pE2ZsOnM7Vq3R5mcrIv8+sGbS2MUnE1mDm670umeuCu6Ocbz6/L4ziSyf9/DQHw9ePbGW4xwxeJUzez1sxMWzUanpkWcCnxjOalCOvYb0FhHluaPXSvYaZl+cO3Enj67f/4MONkarWcWvA9WJ47jExMT9+w988vH0WbNmnzxxstxJLkKIvR1D8ZqcvDJqMa/RaAjl4ORY4kOgVNLdqw9j1Rodx7Ga1LiHgdsWfbzgtJKUMr0cZ4rRsjMoK3j/4ac6nksL3Hvytfcdjd1qzT3ry6g5pR2cyr9c8NkP/joWFKXM0ely0+MfnN/21fiZv0W9vMgb24XynK2a8O3Tx05bdyQ4Ijkjj2V1uZkpL2ShgUevmfnqOgBATZEr27Xw24sqx85z1n/hY21uq6DMm0vFbgcGnoaMXKLL0V419+ZYBWEbfkDWL5gRdPBMEstnBZ26UjCkWU7IiYsKlsu8euhsYuE2Kt4SJrnhu5b+dCO7Tq+5y4a4l8wdGGmKFztQ5aoVldiCMnrMaywdy8bEPN25a9eUKVMXLlx04fyFak5yEUJ4bf7WC+pVYbn30rgslrAcr87UhijYgtaVTnfgsnrBXc2tVDaLJRzHZ2vYqOS8swmGx8TltfnbA9Urw3IfZnA5LNHquCR1fmw+oQjh0nJWXc/+J43L5YlOx8UpdeVo01PO4lcDFem/hxUaemvj5i3l2fWaat+e3wghwUHBVToQspVQaGNrk5FhPNNYW49zxfn5dZo1cwYhZE1AQJW+5GxiYXXv0f3rxYsJIRs3bzH2vcUqRbmN2Ra0qmPIsj4fHq7Ez60CFFUl1WzWzBl+fp0IIYMMjZ9ViUQikSmJra8XL+7eozshZNKUaUZnfnPgElEU0+SLg3/Nqnvs0/cWBb0ZQ2VV270PAN5CeO4AqDrVk2cgJjd09ed7WKbdH3LDo1HXSq0dcyZKleT1dtQb9CJqTZev1eZX6TuxUHlqeGHRkg4D2pKox88SU9JzBS7vdBg6/4vOVtzTe2E1tCcjvIlqWTWrSO+tN04tKzsAAAAAKMNb1dCtFMhzAdQ41r7jAjYOdCj6kifPJp7euj+yPB+vBDAI1ezNhbIDAAAAACgN8lwANQ1llRZxJfQd3yYNpc7WJC89KSYs6OTuzX+EFB9VHKD8UM3eXCg7AAAAAIBSIc8FUNPw6aE7Zk3ZYekwoHZDNXtzoezKxkZtHd1kq6WjAAAAAAALQZ4LAAAAAKpE8+bNhg8bbukoAIyTPQk/8ecJS0cBAACVAHkuAAAAAKgSrm5u+o+cAtR8J8gbk+fy7+NvJbSShctexL3geXyDBADgNchzAQAAAFhely5daIpWKhXJyQp8WQkAyuDZ0HPUqJGEkByNRiaTPX706PFjWVRUVH5+vqVDAwCwPOS5AAAAACyvffv2PXv0YBiGEKLVaRXJCoVCqVQoFEplcrI8OVmhUCrVKhXLvpEf1ty4eUto6C1LRwFgwL49v1k6BLOpUlJ0LCtgGDtb2w4d2rfz9WUYhmXZZ8+ehYWFPXr8+IksPC093dJhAgBYBvJcAAAAAJa3ZcuW4KBgBwcHqVQqrSuVukvr1pWKRKJ3vN6pX7++jY2NfrasrCy5XC5PksuT5UlJcrU6Va1WvYiPz8vNreaAHRwcsrKyqnmjAEAIUapSBAyj/5silD4/zjCMt7d348aNhw8fTghRKhQPwsIsGSUAgIUgzwUAAABQU2RlZUVHR0dHRxebXpj/EolEYpFI6i719fUdMKCuvb194YL6/JcqVa1WqeXJcnmSPDExMScnp4pCXbZsWVZWxr59+589e1ZFmwAAg1QpqtL+i3mZ/3KVSN7v00f/t5OTc3WEBQBQMyDPBQAAAFDTlZH/EolFIheRVCqtW1cqdZf6tGwpEolcXFwoitIvqFar1Wq1XC5PSpLr81/6KRUMyaOeh0udlp07d7l16/a+ffuePn1awRUCgEH29vZiV7Gbq5vYVezm6uoqdnWXupe9CMuyDMNERkY2bdqUEJKRgXcYAeAtgjwXAAAAwJsqKysrKysrLjau2HQroVAkFkulUn3+SyQSS6VSX19fiURC0zQhJF+rVatUcrlcn/9S63uByeUKhYLjOKPbZRimjrOzPpXWvkM7P79OYWEPf/vt94iIyKrYTYBaz8nJUSQSSdwkIrFYLBZLJG5isavYVSxxcyt8bTkvLy9ZoVCr1KoUFcdx+nO5GH2GKzo6+uDBQ6GhIWfOnK7e/QAAsDzkuQAAAABqm3ytVp/DKjZdKBSKxWKpVKrvBVa3rlSf/3JzcyscAl+VopLLC0b+SkrSr6Z4/kssFhU+YwsYASGkZUuf9evXR0Q8+f33PQ8fPqyuHQV4kxTtgCkSicRikZ6Hh4ednZ1+Hn0OWq1Wq1Xqp0+jVfr+mElytVqdmprK87x+Nt/27UQuLkVXzupYmqHv3b23/8B+ZJwB4G2GPBcAAADA20JbSv5LIBCIXcUSN4nEXSJ1d3dzk0gkbi1aNncVuwqFQv2CKpVK8ZJAKCy2Bn2azMuryZo1qyMiInbv3vPgwYPq2SmAGqVkMkv/h0QiKeyZVdihUq1OjYuLO3v2nD6zLJfLTfy8Q4pSWZjnYlmWpumQkJC9+/bFxRXv3QkA8LZBngsAAADgbafT6ZLlycnyZPL699lomnZxcXF3d5dIJBKJm0QikUgkTZs1c3Gpw/E8TVHF1iMQMIQQb2/v1at/iIiIeBj2qNp2AaA66T8NIRKJRSIX/avBIpGLSCRyl0isi3wdVV0gNTo6uvADqUlJSdnZ2RUMIDk5uUmTJhzLEYq/cOHi4SNHkuXJFd4tAIDaAHkuAAAAADCM4ziVSqVSqWQyWdHp48ePHzt2DF2iV5eevm9X06ZNmzVrRgjhqyFQgCpgMJkllUoL3/Mlr3/q4fFjmf4PdapaqVBqNJqqiy0lRanVas+cOXPs2PGKf1YCAKA2MZDn8vb2njVzRvWH8rbBcS7J5fVRBmqOAf37dfHrZOkoAN483t7elg7BMFx+oeaosfe+sknc3AyOga2n0+lomqZpOi8vz9ramiLE0cGhOsMDMJebm2TkyBFiV1d3N4mL2MXN1a1OnTr6Ss5xXFpamkKhVKtVcbFxd+7cUanUyhSlKiVFrU7VarUWCTgw8NKhQ4czMjJNmRnPHQBvj4a2eROlSktHUX2cBGzJiQbyXCKRix8e6asejvMbpEmTGvqsDgDlg8svQAXV9fAo7M9CCOE5nuU5AcNwHJecnBwe/uTx48eycJmnp+fiRYsIIZmmDTkEYCnNmjetV99DLper1epnMc9uXP+n8COkKSkpOp3O0gEW9/x5rOkz47kD4O3hLGBbO+ZYOgoLw3uLAAAAAGAeqdS9cHyutLS0xzJZuOxJROST6Kjo/Pz8wtkaNmxouRgBzHA9+PrqNWssHQUAAFSC1/JcgwYNtlQcbxUc5zdFcFDwoCAUFkDtsSYggARYOgiANx9FUQkJCUFBweHh4RERESqVytIRAVQUz9fOceTw3AHw9sD5XqjUgRUAAAAAAErief6bb5bu3Lnzxo0bb1mSi/YYsurPC6dWdjc8AH9VoFx6LztwOijgfetq22RVYty7zVi798o/d6Jk98KCjgcMcC3+zU4AAICKQZ4LAAAAAMAUlH39Fj4NXGwqLTdj3X7W/+7e/uvHfuLSVknZeLRs3djNTkCZNn+NZuUz+7+b5w9t30hkI2CsHNyk1nlZtbMbFQAAWA7G5wIAAAAAi7PxfG/Sl1MG9WjV0NWOyktXPntyP/js/m1HHqRaIBHC+Ezb9PMkh1Mzpv0aYeBDTiay6TJ777LBjSUiR3srhs/PSVPGRT0MPn9k99HQpJeDmFEURVE0bXLW6vX5jcbpOGLrtZ/9S+sKpr31w6BxexI4c3aqIqw7jxnfzCo/6tD8f2+8GJNl5ebhkJVXXRsHAIC3BfJcAAAAAGBZjOfo9UdW9nRlCvI3AnH9Vu/WayK4v+foA2KBPBddx7Nlk7qpVhXrNMW4ebf29niZZLJxdG3g49rAp+uACSN2TP94Y0gGT0jenV/GtPvF9FUWm79y4qwutLu3lzOVF7TrlzNRaTwhefLYTEvHBAAAtQ/yXAAAAABgUQLfqTO6i4ni8s/LA47di03TWoka+HTq2UZzNbna+hpVFZ1s87iRW57k8Yyts9S7fb/p878c1PqjVdMuDvxFVv6uYqbKPPZF+2MFfwvaLzx1aJrj0U/fWxSkrfItE0JbO4rr2OgyU1NzdIQQQihrGyuK16SkZONdRQAAqDrIcwEAAACARdnV9xQzXNzpjTuDo1hCCMlXPA058zSk4L8p8bvTv5naq6VXAw9XJzshm5EYfnVEocZJAAAKOklEQVT/5v8+qDds4gf9OjevX4fJTnh0Yfe6gP1haYUZFNtG/T76YvrQbi097LnU2DuXj2z59WCoskhmyegMTNNZJx7OIoQQwikOTvb/7oY+O0TZdflqx/kfmjUU27BpcfcDD/y84eC90t+u5HT5WpbniS4nNf7hpV3z0t3a7JnSuFN7d1qWyDFNvjj416y6x4rknmhR2/EzPp/Yt907YmFO0pN/bqa7vxpQ18D8pcZpEsrZd8zsqf06+Xg3ktaxpTQpsedXTllxjur19frZA5rVd3ey5jUpT2+f37F+8/EIfX6qWHGQ3NTiB4EWdZy+fMln7zd1EVI8r82Mu7jqw0VHEwkhFKFdxmy/P0Y/n/bWt30/2pPElVkWBiNcGdriE7OrBAAAvB2Q5wIAAAAAi9IkxaeydIM+kwccjTwdqyn+37SoTd8hvVq+bLYKXRq0G75o5/Aic1h5dhy7dKsoe+TnfyZzhBCblp9t27HQz7kgQeTetNf4r7v1bDtv0tenElliygxloKwbtu1Y8Ler17vjvvFtajti8q5InUn7yulYjhBC0wa/BkU5dVu6Z9OHTQpGurdu6DuwISGEVNkoVrSk66jJAwuPraObRJiXxRPbOl7tm9a3IoQQ4uDeoveUta0k2g/mn0rhSxQHsS92EGjp6IBNC3s5UbpsVXIW5SCuIxHmpXOEMIZDMFIWBiOkTK8Sn/6ZXGlHCwAA3gT43iIAAAAAWJT2zq7NwSmU58h1xy//serz/s1FJX+K5TPOft2vbZs23q27D/rmr3iW59JubZ45qmsH36bt+07acieD1Ok1wl9CE0IY78nL/t3JKTf8yMIx/j6t2rXrN33N5STKY8CKhX1dKFNmIIQQwkZu/KBN42Y+jZv5ePUo0kmKz7q6Zkx3vw5NWnXuPjHgopyzbztxYnth2btIMVb2ovo+PcZ/v3x0Q4aNu3NXbuCVTEGrjxdP8bbKuL93zih/n1bt2vpPnLPjVkrZL2+WFqfp+MyL3w7u2M7Xu03XHqN/CdESPvP62inDunbq4N2iTYsugz/a8TBX/N7I3q+OTWFxePkUPwiUY+d+nR25R/8d3rVrx57+HTr4dRm+Ljjn5YJc6qFPfPXRNm714Z4kyqSyKBFh0RjKrhLmHw4AAHizIc8FAAAAAJbFxh6eM+LzjSdl2eIOIxdtPBJ88fcfJndyL5rt4tlMpSIjj2XzU2XHt/zxmKUEKbLr4fIsrTY78fq2nRfTeaZBo4Y0IUyToR/4WGnDNs1bdfhBco42Py32xrb53x5K4l16D+3jQhmfoWy8VvE0MiE9V6fNSri9f/XeRzpG3Ly5WymtakGrOSejIx7HyO49+uf86R1Lx/rYa8L3fbvzsYHuX0yz/u83ovPubJj304mw5BxtfkbC/VP7LkRX9ThevC41IV6Vo2XzMhJjk7N5QiiqTutxq3cdvR5y6+GVPSv6ezBE4F7X9dU+viwOTlfiIPA8Twjl1tyvuasNRQifp3wWX+rLgyaWRckIialVoioPHAAA1ETIcwEAAACAxeXH/71t9oj3e05cuvVidL57xwnf7Dz56xgvg2NssEmxiTpi7eZe52VTNl8er+ApWztbihArT+/6NPci5PrzIvmh7LtB93KJlad3A9r4DGZgE58+z+YpBwd7I+kxnud5nnAZt3fMGjJh7XWDiR+hR6N6NBd/93aSRYffp5x6Lv19z5Jx77Vu5O5kLbQVNWzgak0Rmi5twJPXDgKfeeP4pRTKvdeSPYH3b5w+snXlVwOalHp0KqssSq8Spq4BAABqC+S5AAAAAKCGyJPfOf7TzBG9Rq04Gce59Zz9lb+Dodk4bb6OUEKhsDB9otXqeEJRNCGEGOuQZXwGM/D5+fk8RdGlrVP3aMNQ72Y+jZu377/6Zhpx8GouIXml9G6iGJoQUvq6qgclev/D4Q2Z1JDNX47s2sHXu2XHLl8dl5fZp+y1g8CnnFky+aPvd/7v0t043qP9e6Pmrt/500DXUnaqsva11CpRSesHAIA3BvJcAAAAAFCjcOmy4xsOhbO0g7d33VJGLy9dfuzTeI5u0PldzyKL2rfv0c6G5MfFxHPGZyC8TqfjiZ2dXSVmSfKj9i355i+l07vz1n3S3NrwLHFP4zm6YbfeXkZG+3qpKuIktKtUakVygvduCnwiz9KyrEalzDBvIPzcF9f2rl/85dR+PXoNXH4hiRf17u9nuGOV8bIAAAAwD/JcAAAAAGBRVu0+XT1/in+rhi42DEUxtqLGnYbPGNaU4XVKhdrsVAcbefKkLF/Y+qv1y0a1cbcTWDl7dvt07coxdam0a6cC1bzxGQivkCt5pm7f0X0bOwgYG5FXRx8Ps/NtJXCKs6u+PZxg3W7GyuktrAxFHnHqdLhW0PLLzT9O7+ElsmFoxsbZ1aX0LFaVxMmpFAotse08YmIHDwcBRWihg4ONGd9oZxr7j/RvU8/JiqYYoUCXmZlHCEWV0nHLeFkAAACYx4x7FgAAAABApRO07DNh2DTPkdNen8xrovZsP6/mzf5dlo3au2pDzx0LOo1ee3j02pdr0yacXfHjeTVv0gxx1y7LZrVuM2Ld5RGEEEK091cPnLw9rvz7WLCN9OAfvzvRY8vwL5ZPPD/pt6jiLwOykbtXrOu2fbFf/yU7+i8p8h+ldKcqLc4KdYPiVZcPXZrZY5D/8v3+y4vGZtrilLjzxyuXdSvaI41Tn71wK9vw7EbLAgAAwDzozwUAAAAAlsTFnPxx/R9nQyMT03NZjtNp0hIibv65ZfGoCetuZJYr1aGR/Wf6hBmbztyOVWu0+dmKyL8PrJk0dvHJRNbEGdio3bMW/HYlKiWHZXU5qph70crKGemJT/t7089X0mx8P5k7UGxgjZrw7dPHTlt3JDgiOSOPZXW5mSkvZKGBR6/FaA2trkri5NVnl06ft/PKo8SMPJbV5WWnKuIjH4TcjE43pTAoKunu1Yexao2O41hNatzDwG2LPl5wWlnqskYLCwAAwByUs9jN0jEAAAAAQC3UvUf3rxcvJoRs3LwlNPSWpcMBMGDfnt8IIcFBwWsCAiwdCwAAVAL05wIAAAAAAAAAgNoAeS4AAAAAAAAAAKgNkOcCAAAAAAAAAIDaAHkuAAAAAAAAAACoDZDnAgAAAAAAAACA2gB5LgAAAAAAAAAAqA2Q5wIAAAAAAAAAgNoAeS4AAAAAAAAAAKgNkOcCAAAAAAAAAIDaAHkuAAAAAAAAAACoDZDnAgAAAAAAAACA2gB5LgAAAAAAAAAAqA2Q5wIAAAAAAAAAgNoAeS4AAAAAAAAAAKgNkOcCAAAAAAAAAIDaAHkuAAAAAAAAAACoDQSWDgAAAAAAarkB/ft18etk6SgAAACg9kOeCwAAAACqVpMm3pYOAQAAAN4KeG8RAAAAAAAAAABqA8pZ7GbpGAAAAAAAAAAAACoK/bkAAAAAAAAAAKA2QJ4LAAAAAAAAAABqA+S5AAAAAAAAAACgNvh/CwR1//nyIDoAAAAASUVORK5CYII= ) ``` w.set_project(project_id='605ab63ecd8a6669dfd64901') ``` ``` ``` ``` kerasc_blueprint.train(w.project.id) ``` ``` Name: 'A blueprint I made with the Python API' Input Data: Numeric Tasks: Single Column Converter: 'Age' | Missing Values Imputed | Binning of numerical variables | Smooth Ridit Transform | Keras Neural Network Classifier ``` ``` starred_models = w.project.get_models(search_params=dict(is_starred=True)) ``` ``` model_to_clone = starred_models[0].blueprint_id ``` ``` bp = w.clone(blueprint_id=model_to_clone, name='Now featuring selected columns!') ``` ``` bp.show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABm8AAAEMCAYAAAAiZyJjAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3gU1RrH8e/sbgohjYQOoUjvPXQQlCag0lRAUVRAUbGjYLlYruJVUVFBEVEELKB0kI40RSCU0HtvCZDedrM79w86BLIpkEV/n+fhXpI9O/POnDM4c9455xhBoYVMREREREREREREREREJO8ZTLHkdQwiIiIiIiIiIiIiIiJykZI3IiIiIiIiIiIiIiIiHkTJGxEREREREREREREREQ+i5I2IiIiIiIiIiIiIiIgHUfJGRERERERERERERETEgyh5IyIiIiIiIiIiIiIi4kGUvBEREREREREREREREfEgSt6IiIiIiIiIiIiIiIh4ECVvREREREREREREREREPIiSNyIiIiIiIiIiIiIiIh5EyRsREREREREREREREREPouSNiIiIiPwr+JRuzbOf/MLy9Ts4euwYp47uYcfqWbx7ZwGMvA7uJvKt9xyzI/dzbNscXmvin9fh3HRG4fa8O2U+azf8RP9S/5LHIe/GDJ6ygDUbF/GfBl55HQ3wL60HEREREZEs0F2yiIiIyL+d9+18tv0EMSfWM657GLbrlfWqx9trjhN76gjTHy58yyQ9bGUfYvy8SQx7qBU1S4WQ39uGzSeQoreVwc+RhHmT4rCW7c6IKfNZt+FL7vW7STu9kmHBYjEwDCtW41apwdxjyX8bDZvVoUIRf6w3+fDzrP5tRanRpDYViwXi7SFVnpf14HG8wnkv4jixp46xekjN6/8bLCIiIiL/GrovFBEREREADFtJunwynsNH72XYX/E3LaFxwxlBtH9lCG0LWUjdNomXX/qE2ZuOkWgEULJKWay7HTctFEuhGrRqUYey6cew3rS9Xi513Qjuqj4ij/b+7+YJ9S8iIiIiIrcGjbwRERERkQuMfNV4+ptPub/UP6hr2asubVuGYHFF8etbQ5m45hCxaemkp8ZwYMN69ib+Y9JUIiIiIiIi8g+h5I2IiIiInGOSlubAUrQjH45+iuo+eR1P7jACilEsyALOKA4fs+d1OCIiIiIiIiKZUvJGRERERM6xs/iD15h9wkVA+GDGvt6UwKyuReFViPCH/sN3s/9k577DnDy0k81Lf+LTZ9pR7rI1PiyE9Z9BVHQUZ7Z+SCvvKzeUj7tG7yLm1BFmPlrsqptWW903WHsiiphjU+lXPJNbWi8vvAG8ajB0xXFiT0Vd+HNm12e0vXTfPiVp2f99fl64jn2HjnDywHY2LPiB4X0bUuSqCYetVOzxPhNnLmHDtj2cOH6cU8f2suOvmXw7tAtVAq5z8nw68u2hqMtiOTmlD4XPfcUr/C02nIwi9uh3dLtqbRSD4AcmcvJUFKf+fI06l8blVYImvV/gg28ms/jPDew7eJjoE0c4vH0tK355hnpeYBR9mBnHoog9vpiXq1wywipfZbq89F++mjid5Ws2se/AYaKjTnDq6G62Lf+VL59rx235rnOebQWp13MIX01dxtZdB4g+eYyTB3ey9a95TP3mA157rBmZVdV5vmXa8NxnU1i1cSfHjx/lxIFtbFo2gx9HPEwt34zOZ1bqzQ3Z2Z6tIPV7DeXr6SvYtvsgJ4/sZuf6ZcwZ+w4P1fa/fH2oTOo/uzEYAZXoMvhLZq3cxIHDRzixfzN/zxrD233qUSg768pk5ZjcvvYzl+32f74NT5rBijWb2HfwMKeiTnDq8A42/D6GIR3Lkc+3JM37vME305ezfd8RTp04xIGNi/ll+MOEh2bQQHNwXWS5HQP8cyasFBEREZEc0po3IiIiInJB+uEpDBpYkYo/P06l/p/z0ZoOPDHrJC43vmsEN+DF78czpGnBSxYg9yGsxh08UqM19z/wEwN7vcy0gw7AxfFVK9ntbEy1kDrUK2tl6U7nxY3ZqtCwrj8GFqrXrYH3uOOkXvjQQuE6dShlhfTtq1h50p3o3Ig/pClDJ47jxfAClySLQilbtz1P1LmDjs2ep/OAyRxIvxhHULVWdGhy2yXrlwRQtEIjur0QTpumYXTqOpLItFwJz71jCL2TVz58lZZXJMO8CpWmcohBwnVOlRHUkEde6HfVd/EJonjVFvSu2oy7Wr1N5/tGseWKYzIC6/Hc9+N5rXlhbJf26OcvQIkKBShRoS63t/Rl7YSVZDb4yavS4/w0811aXdqR7lWQ0tUKUjJoK186Ly+f9Xq7vuxszwioy6DvJ/BGi0KXHH8+ipQKokiJILaPf4cJ7u0+2zHYSt3D55O/4IHyPpckVYpQqfG9VGp87scrzt11Y8jCMWXt2r9xrtmG84VQtsG9DP6uFQ9He1GkiN9liafgkjVo9/j/aNm0LN3ueotVCWbm28zkushaO3biPHdtOp1ZqCQRERER+UfTyBsRERERuYRJ7PJ3GDBiAymWknT/+FP6lHHjfR9LEbp9/B1DmxXESNzCDy/eS90KpShSuiYt+n7EkuNOfCv1ZNS4Z6l1rhM0ffcqVp5wgq0iDesXuKwz1VKiIY3CbICFwPrhVL0sBD/qN6qBt+Hk+KqV7HG3r9Oxmf82L0ZwwcIX/oRUfJYFdsBSjB4fj+HF8GAcB+fx7kO3U7lUSYpWbEyXN2dzwGEjrPO7DL/v0lFA6eyYMoTedzWnWoUyFC5SnKKVmtB12DwOOwwCw59laJdCZDjgIW0Oj5UqfFksRXr8QFRuvXTvPMDE/rdTo1IZChUpSVi1Ztz13GT2unOu0vfybZ/GVC5fhkJFSlC8SnPuf3chR9MNCjQZzHsPhl3+EGEU4p6PvuP1FoWxpu1h2n9606RaWQoVLkbR22rQePB84t0+Ln/aPv8yLUMNkiK/54n29ShdvDiFS1ejXruHeP7tX9l8aRImW/V2HdnZnlGQjh+M480WhbAm72TKm71pWqMchYuWpFSNFnR+4mW+W3dFxup69Z+dGGwVGDDqUx4o74N5Zh1fDepEnQqlKFyyEnXvGsC707YTn5UcZ1aOKRvX/g13vg2XK02hoiUp06AH7yyJwrQEUbSwi+2/vkXvO+tSungxCpetS4chszmUDr6VH2Voz7CM20qWrosstmPTicNhAiaOdKfG3oiIiIgIoOSNiIiIiFwllY2fPcV/lsVASGve+bwflTPpdPWuM4BXOhbG4jzOb8/ez7Pj/2RfTCppSSeInPU/ej34CZvSIF/NAQy+p+DZhIZ9E8tXx+MyvKnfrD4XZx4yKNCkKTW9zv5kK9WYJiUvuW31rk3zcH8MVxx/rogkN97l9643gMEdCmGkrOP93o/z0e/bOJFsJ/XMXpaOGkj/MbtJtwTRqmdnLoZikrB1KfPW7ORoTDJ2Zzqpp/ew5MtneG1WLC7Dn4bN65InSwe5Eji4fSeHTyfjcNpJOLmLtVtPuDfwwkzmxP4DnIhNxuF0kBy9k/mfPc3QWTG4jHw07Nj6sum9vGr149W7i2J1nWH+K/fx+JcL2XYyCYfLSWr8SfYePI3D3d5oawmqVPTHgoMNkz5l8rrDxNnTsSdFszdiPj9MXX9ZIih79XZt2dmercbjvNa1GFbXcX59phv9Ry1k6/EE7Ol24o/vYMW0hWxNcfP4sxlDvmb9eapBfgznQb4f0JMhP65hf0wq9tQY9q2ZxkfPfsqyLFwoWTmmbF37N9r5NhyXgiPdTuz+ZXzy/MesTDXBtLPx17HM2XiEOLsTe8IR/hr7Eu/OT8A0fKjbIpyA623Tnesii+0YM5mklLPJm5SkJCVvRERERARQ8kZEREREMuLYy7fPvcm8UyYBjV7h86eqXScJYaNmpw6UtUH6rp/4fG70VZ2PqZHfMnpxIqYRyO2dbyfIAEjmz0WrSTYtBDZqTm2v86XzEd68AT7OQ6yNOInTqzrNGgVf6PS1VWxKsyJWzMRVLFidSs55UafzXZS1maSunMCEnVfO65XKhkUriXIZeFerS63MsjFmAls27cWJgV/hwllfN8gTmbEsXxyB3TSwlatM+QsjoWxU73QX5W3gPPAzI6YcycrMXFdzxRB9yomJF3Xue4TGodbrFM7lesvW9mxU79yJCjaD9F0/MnJuVA473rMTgxe17mxNESs4tv7ImOVxOYwhK8eU3Wv/5nOdXMNf+5xg8ee28oUvfxA2Y1n79y4cGNhKlKL49ZrdFd/L8LrIUjsGzATizmVz4uISlbwREREREUDJGxERERG5BteRKbwwZAbHXX7Uf2EEz9W8Ru+3EUjlKmHYcBG/MYKdGa0tYsaxft1u0jHwqVSV8lY4O0XbQlanmliLNaVFxXM9n951aNUkEE79wadfrSDO9CG8ZUPOrltuoWSL5pSzmaSsXshy9+fjujYjgAoVi2PFIF+bkeyNvnwh+dhTUURPf5TiFjB8C1Ek+OIttK1QXXq9Noppi1ayecc+Th47wN4NyxjftzxWwLDZLlkD5FZmknTiOHEmWIKDL3bAGwFUrlYaKyaJEauJzGQ9m8x3E82ssb9xyAH56z3LzLUrmTriOe5vVBK/K89jDuotQ9nZnhFA1eqlseIifuN6drm5rk7uxhBIhQpFzsawOZL9OV0yJSvHlO1rPw84T3PylAuwEBgUeMWDsIszp07jAiz5A/B3+yn5GtdFVtoxgCuGE9F2TFcC0dHJSt6IiIiICKDkjYiIiIhck4sT04fy0m9HceWrzfMfP03NjKZPM/Lj7w9gkhAXT8ZLa7iIj0s82znqf7Fz1Dz1BwvWOzBtFbizVSmsgK1aa1oVtZDw1zJWLP+Dv1MMgpu3poEPYBSmVeuaeJl21i5cxpnc6OW8EL87fPE5l8Oy3daL8Ytn8eXz3WlVuyJhBf3x8fYjNKwSNcoE59KNtnFzpplyg5mWht0Ew+bFhYE3hj9BAQYGLuJOx+Rs1M3ZvXBmwSt07vMRs7bF4gooR+s+Q/l61hq2LR/HS62KXbLv7NXbNWVne4Y/gQEGxnXbfhZkKwY/8ucHMElMSMyFGLJwTDm49t0MJhfbvx37+WV6LFdnkOyOc5knq5Ws5JcyvC6y0o4BcBJ1IhqXK4pjJ3JcgyIiIiLyD+HG6rMiIiIi8q9lnmbuG0OZ0nQcD9R+hg8HnGDDVWWSSEwEMAi46o328ywEBvljAcykRBLP90+6jrFg3kbeadKAGnfeTtFRBwlofTtlrEksXLiShDPezF+TRvsWLbmzlhfL9janbQMfsP/F3EUnc95RDWfXm0gCcHFqYi+qPreETAeQGAXpNuwt7ipuw3n8Dz55YziTVuzkWGwqhl9hwl/8kalPV8toZ+6F5Egn3QQMH3zzGZDsoe/im2mkpgIY+OX3y6WOdjuHFn7IQwtHUrRue3o+9AiP9mhCWJVOvDaxAoHd2vHm6uTs1dv1ZKsdJJOcDGDgHxjgRsIuk3rMVgznrz8LgQWCsELO1oEys3BMObn2r7fZPGn/ub0PN9sxAE5OHovC6UrheFTOU6AiIiIi8s+gkTciIiIicl3m6Xm8PuQ3jrr8CH9uCPcWvKKL3kxg544jOLEQWLsulTJ6PcgIpE79CtgwSd25lb0X+iddHJo7kwgHeNfvQNvi5ejQvjK21LXM+yMW04xm0fwI7JaStGtfg4It2tPUD9LWzmTO0Vx6Q92MZ8+ekzixEFy3PhXdeb3JqxqNGwRgmKks+u8A3pu+ngOnk7A7naQlnOTg8YQMu4LNNDtp5zqlfXyunepwxZ4mxgQsJSjt9gIcecCM5eChWFxYCKpZi7K5GmoaJ9bP4JPnu9CgWX8m7XWAT0UeergF+SB79XY92dmeGc/u3VE4sRBUqw4VMvlOpvWfrRjOX38GAfUaUd0r869cf3tZOKYcXfvXdsu0f7dk0o4BcHHwq84ULn4f30d7aKJWRERERG46JW9EREREJBMmp35/i6EzTmD6FaZo4JW3kA42zZnH/nSwVezJU+0KXjUCw7f6YzzR2h/DjGfZrGXEXtI/6To8lxnr7ODTkG5PPEKnajbS1sxl4SkTcHFswe9EOKyU7dCL57q1IIA0/p4xj2O5NruQgw0LlnDCCbZKvRjUoZB7I0jMs//jTHe5/c6+61QUp12ApSyVrpPpcB3dxrYYF1jL0aZtOQ8eLm8nYvFyzrjAq3pvBrQIuiHTvKUemMOYGftwYuBXuMi5tUWyV2+mefYPhje+Xpd+Izvbc7Bx4WJOOsFWuTeDOhS67gNW5vWfvRg2zZnHgfSzU/m9/EDpHLaXrBxT9q/9a9fDrdT+sybjdiwiIiIikjElb0REREQkc2Y0s4b9l3mnM86Y2CO+4oO5Ubisxenx+U+MeKgRZYK98fYrQvWOLzFh4vPU8YXUyDH8b0b05ckO1zFm/baaFHxo0q8vdb3SWD1rISfP7cp1ZC4z1zmwlu/NE20CIXkVv849njtTpp2TunIUn6yMOxv/F1P55um7qFuqAL42A4t3AEUrNuTuh9pT+XwvsmM7EZEpmEY+7njxfR5vXp6C+WwYGNh8gggN8smw090VtZ6Iw+lgK0vvIQNpWiI/Nos3ASVq0r5zOEXP353bV/PrjCM4DS9qDRrNxw+GUyrIG4thwze4KGWKBXrMWjjxC79i7NY0TGspHvlqIu/2CKdcaD588hemUrMHGPpESwLdDtafpv2HMqBjPW4rmB8vw8DqE0zp8Pt4onNZrLiIOXjwQgIgy/UGmPGniXGZYKtCl0fbUCnE+8K5zM72UlZ8yccrYs9+58tpfDuoA7XDgvG1WfH2L0SF8Pa0qeaPgXv1n50Y7BFf8f6cKFyWENp+MI2fhnShfqkgvC0GVp9AipYtQXAWGkxWjim71/716uFWav8Zy1o7NgIa8sr0DRzZv45fn6qDX94GLyIiIiIe4p/yEpOIiIiI3GCuY7/y+kf30/y9ZgRc2XPqOsGvLzxKmdDxDGlSi76fzKTvJ5cWMEnZ9TNPPfopG9Ou2jLHZ/3Iwjeac0+wFTNlBdPnn7iYnHEdY9b0vxnWuDm+Vhen509iZlQuTy3k3M+4J5+g7I9f8WTtSnQf9j3dh11Rxr6W15YvYMdBF5hRTBn+OX0aDqZBhe58NK07H2Ww2avWK3Fs5Nsvl/HgR3dQsNXrzNn0+oWPzKTfGbBiLZNjTSCVVR8PZUKrb3m4fHUe/nQ2D3969fbTc3TQucQeySdPvk7tye/RrnhDnho9m6cyKOYyzcxHKHlV4a7+z/BUmef44KoPTVynl/Hx6FWknv9VVusNMGNXMGtFHG3aBFOz/w/82e4rOjZ6k9WO7G0P5wG+G/gEt/34NU/WrkiXN8fT5c1Lv+Bk7xedaLQ1Aodb9Z+NGFwn+O2FRygV/D1DW5SkzYtf0+bFDM6vu8upZOWYsnntX7cebqX2n5EstmOvej14tEkJ/C3Q6tEu1Byz4dx5EBEREZF/M428ERERERE3Odk3/k2+iEzLeD2X2DV82P12Or78FbPW7iM60Y49JYajW/9g4n/70rLtc0w9mHGPpHlmAZPmROPCJGnFVOaeuHRcjYtjc6ayIsUE53FmTFpM3A1YFsIVtZjX7rqdewZ/xYzVuzgRb8fpSic18TQHt/7JzAkz2Jh8sXzqxk/o2qE/7/+8jM2HY0hxOHHaU0k4c4KDuyJZvXgWP/2+lcTLhxlxcEJ/Oj89it8jjxCX5sRpTyL6wAYW/LaKQ5fMpGWeWsALHe/h2dG/E3HgDMnpLkyXg9T4UxzaEcHiaeP57KsFHPaA9c3TdoznwTZdefHreUQcOE2S3UFq3FG2LP2R4V/9wRkXmMlJJGVWb2Y0K3/5mQUb9nMyIY10l4v0tHhO7Ilg7rf/oUfbh/hm9+VtKKv1husYPz7dm1fHL2XLkRgSd+9iX3oOtge4opbw2l2t6DJkLHPW7SMq0Y4zPY2E6INs+WseCzfHn0tGulf/2YnBjFvHiPta0Gbgh0xaHMnB0wmkOU2c9mRiTuwjctVcJnw5itn73Wsw7h9TNq/9TOrhVmr/V8liO3as/43vVh8jKfEIy8ZPJ1KJGxEREREBjKDQQloRUUREREREbgALYf2msva9xrDsZWr3+IETevoQERERERG5PoMpmjZNRERERESyzwii6SOPUTHqL9ZtP8Sxk9HEplkILFqOum0e4bVXGuFLPPOnLiS3Z7sTERERERH5p1LyRkREREREss9WjbuffZkBJa0Zf27aOTh9KK9MPo4r4xIiIiIiIiJyBauvX/5heR2EiIiIiIjcomy+5Pf3wdvihU8+X3y9vLCYduKjDrB51WzGvf8Cz3y8jJOeuDaJiIiIiIiIJzLYpjVvREREREREREREREREPIXBFEtexyAiIiIiIiIiIiIiIiIXKXkjIiIiIiIiIiIiIiLiQZS8ERERERERERERERER8SBK3oiIiIiIiIiIiIiIiHgQJW9EREREREREREREREQ8iJI3IiIiIiIiIiIiIiIiHkTJGxEREREREREREREREQ+i5I2IiIiIiIiIiIiIiIgHUfJGRERERERERERERETEgyh5IyIiIiIiIiIiIiIi4kGUvBEREREREREREREREfEgSt6IiIiIiIiIiIiIiIh4ECVvREREREREREREREREPIiSNyIiIiIiIiIiIiIiIh5EyRsREREREREREREREREPouSNiIiIiIiIiIiIiIiIB1HyRkRERERERERERERExIMoeSMiIiIiIiIiIiIiIuJBlLwRERERERERERERERHxIEreiIiIiIiIiIiIiIiIeBAlb0RERERERERERERERDyIkjciIiIiIiIiIiIiIiIeRMkbERERERERERERERERD6LkjYiIiIiIiIiIiIiIiAdR8kZERERERERERERERMSDKHkjIiIiIiIiIiIiIiLiQZS8ERERERERERERERER8SBK3oiIiIiIiIiIiIiIiHgQJW9EREREREREREREREQ8iJI3IiIiIiIiIiIiIiIiHkTJGxEREREREREREREREQ+i5I2IiIiIiIiIiIiIiIgHUfJGRERERERERERERETEgyh5IyIiIiIiIiIiIiIi4kFseR2A/HPcc+89VK1cJa/DuCW8P3x4XocgIiIi2TTk1VfzOgQREY+k5xwRERGR3KPkjeSaqpWr0Kx5s7wO49agZxoREZFblu53RESuQc85IiIiIrlG06aJiIiIiIiIiIiIiIh4EI28kRviwT598zoEjzPo6YGEhzfI6zBEREQkl6xZs5aRX4zK6zBERPKUnnNEREREbgyNvBEREREREREREREREfEgSt6IiIiIiIiIiIiIiIh4ECVvREREREREREREREREPIiSNyIiIiIiIiIiIiIiIh5EyRsREREREREREREREREPouSNiIiIiIiIiIiIiIiIB1HyRkRERERERERERERExIMoeSMiIiIiIiIiIiIiIuJBlLwRERERERERERERERHxIEreiIiIiIiIiIiIiIiIeBAlb0RERERERERERERERDyIkjciIiIiIiIiIiIiIiIeRMkbERERERHxWNYiTRj44QSW/hXB7m0b2LxiGsM7FMTI68DcZBS4nTd+ms2K4Xfik9fBiIiIiIjILUPJGxERERERuQ4f6g76hfXr5vJB29CbmzTxrsazX3/BS3fXpUyILzarN/6FiuKTloh5M+PIAcO3OFVrlKWQn+2WSTiJiIiIiEjeU/JGPE7X0evZv3NrJn82MrlPiVxuwFaq9R3FvMU/8FQla65uWUREROSarOV5euom9kVO5ulKXtcpmJ/Gr//Onh3r+fbegKzuJEf3OYZhYBgWLDc5++DT8D56VvLGvnsyz3RqRuWqtal5+wP8b03azQ1ERERERETkJlPyRuQCC8Glq1KhWADeei1SREREbhZLQYoUsmD4VOHx5ztR9Bp36NYKPRncIwyrYaVAwRCyloLJyX1OGhGf3Uedeu15ef7pmzjixUKR8uUIMtJYNe4z5uyOIc3pIOHEQY4n3irjbkRERERERLLHltcBiFxp6pN1mXrhJxt1B89ict8AfuvfildWOPIwMhEREfmnGTjwSeLj41m+bDmHDh/OmyB8ClIkyMAeG4e1eX/61f2dd9alXl7GCKbtEw9TIy2WWO9AQkIKYHAwb+K9aQx8fL0xzBROnUq6ZaZJ+zey+AQQGuxLekIMMcnpeR2OXIPqSUREROTWopE3cssz8pen0/MjmLb4T7ZvjmD9oh8Z+VRLSp6bdcSvzvP8HrmZrT89xqUzkfhUfYLpGyNZ/1U3Sl766qq1IoNmRF6Yom3vijdocr0ZTEREROSWVaJEcXr27Mnor0YzevQounXrSqHChW9qDJaQgoRaXETNHc2EPcW478nOFL/iLt1W4QEGtvXhr1FjWZ1moUDB4As38kbo7QwdP40Vq9eya1skOyOWMPfrV+haKf/Va6xc8z7HIKj2/bz5ybfMWrCMzZGb2LN5Natnv0WHECsVnpzC7u0r+aD52ZuiLN9fXSlfGdo+9QFT5q9g6+b1bF4+je+H9Sa80JVfMsBSgPu+2Xhx+twt39On2KUnyEqN52exZ8daRne6dDo5C0V7jWPn9hV80MLnivIz2bN9Bf9r6ZuFeK53joxzdVmL3q+PZu6y1ezYEsH6hZP4fGAzimTlqcu3JK36vcPE2UuJjIxkd+Qa1i2exuTR79AvPPhcnV4/FvfahEFo0/6MGDOJeYuXE7lpI3u2RbB+0URGPNqISvW68cqIH1i8ag07t0awfsF4hveuQfAljcoSUp8Bn05lXcRfrFn+BxHr17FpwYd0u7IBX8rtugfyleLOJ9/j59+XsWXzBrauWcKCCcO4t7TVzTJeNB32B3u3TeP5ylfUZZdR7Ny5ge+7h7hxTi2Z1n1mzyQZn++NbFk1k4lv9aROgQyGxLlx/JnvN5v1JCIiIiJ5TiNv5NbmV4Onv/2G5+oEXOjA8A2rRednRlK7+CDueX0ZMRtGM3hMY3556kk+6Pcn94/aTppvdQa+24/qSUt5adg0jjjz9ChERETEA4SFhdHn4Yd59NFH2b9/PwsWLnIaHEkAACAASURBVGTFsuXExMbe0P0awSEUsJjERv3ND+OW0+u9vjxaZybvRpxb18UI5I7+vah8ahaPTNtJh8dNfENCyW+A3QTSgylXtyIlvc9t0L8IVW7vw4fVC+O45yVmnXJnzIqFwo2789BdVS95QAigUGEv0hKvLp2ck/sr36oMGDOWweFBF98kK1KRlj2H0KRFLV58cAizjmXl5szJztVriO7fg1p1KuM1ey1nx2rno079qnhZ/Khd5zasy7fjBLAUpHbtMCwpy1m1IS0L8VzvHJkYgU14/YfPeaSC74UEiU+p2txV6uzf3Vqlx7cy/caM5dXwApesL5Sf0JIVCS15G97rxzFuTSzOTGIhnzttwkJIzTZ0bnnpNrwoEFaHLq98S5crQvMuXZ/7Xx9NSFI3nph+EpelKD2Gf87gloEY6UmcPpmI4R9KcGEv0uJc1zi+LNS9T2X6ff0trza8mKjEuwjla4fhn+Jys0xWkhPXO6fG9c+3O88kZkbnG/IXLEfTB16jdsV8dH1oHLvOD4hx5/jd2a+RjXoSEREREY+gV23kFmalYp83eLq2H1HLRvLoXU2pXK0eDbu/zq/7nJS892l6l7cCqWwe8zqfbzKpMeBtnqoZQt2BbzOgcjxz332XmSeueGhx7mLkPTUpW6kaZStVo1zzd/hTs7WJiIj84xmGgc169o32smXK0u/xx5kwcQIjPv6I9u3b4+fnd0P2awkOIdgwSYiLJ2re9/x6tATd+7al4LnOe2upLjzexp/NkyayOjGeuAQTS4EQQs/dyZsJq/iwz700blCP8lVqUqVRJx4dG0lqaCu63V7g8tE3md3nmAks/E8n6tepTfmajWne4zP+zvA+KIv3VxdYKf/QGzzfIJDU7b8y+L7WVKtehzpt+/H+kuMYxTswbHAbLhuE4Iph8uO1L8Rctvoj/HD88u3bI/9iTaKFQvXqc9v5QQle1WhU1w8DC2Xr1704+iV/HRpV88KxZTV/J1qyHk+G58hG9cdepU95b+I3TuC57me3U6t1b54bu5ZTbvWRWyn34Ju8GB6Mfd9s/vNQe2rXqEn5Gg1p9uYykjPKwV2jvrLUJsx4fh/Sllo1a1K+RjM6vjaXI04TV+xavni6O43r1aZi3TY8OCqCeIJp2bU1hS1gBDSkbcMAXFu+pkvjxtRv0Zp69cJp1OUjVibntO6tlO39Bi+EB5G6azqvP9SBujVrU6VBa9r3+YD5p0w3y2TD9a6BDD9z95nk6vNdrlpDmvUezsITLvLX6k3vuueHy7h3/O7sN+v1JCIiIiKeQskbuXVZK9G5U2Vs8Yv478tjWLo3lrT0VKI2T+OtkUtJtFagSXjBs43cvotvXvucNY7KPDHyJ758tDynZ73LW/Oi0ftmIiIichUDLBYLhmFQoWJFnnpqID/9/BNvDRuW67vyCQrCz+IiMSEJV9pGfpi4Ae/b+9CzghXwJbxPL2onL2bsrwdwkkxCookRHHJx+irDILjGA7w37jdW/b2WyKU/MKxdcazYKFKsYNZu+M10Yo4e4XSyA2daPMcOniTpWn3g2bm/slbg7nuq4e3YzOcvvs2UTSdJdtiJPfgnY176D5OPmxS4/W7uyGgKqetJXsfStclYyzWk4bksjbVCQxoVjGfr1qNYqjckPODsNn1qNaZBfidbl/9JtJGNeDI6R5ZKtLuzDJa0CD598X/M2Hx2O/FHNzJr4gL2XBhIZKXy01PZc34KuJ1b2b9tJi9Vs4L1Nu7qWA1v5w6+ev51flhzmDi7E6c9kejTiRmv+XOt+spKmzCdJERHEZ/mxGmPYdu0UUza6sSwnWLbqu2cSHTgSDrGqjHfsjDOxBpWhlIWwDQxAaNQZcIrF8TXAMw0ovcfITajYLNS99bb6NS5Oj6OSEYOepNJaw4Rk+YgNf4kuzbsItqFe2Wy43rXwDXq3u1nkivOtys9kaPrfuS9CVtIt4ZSuXKhs+XcOn4395vVehIRERERj6Fp0+TW5R3GbSUtWPK14/M17fj8qgJOipcshoWTuID0vZN49dOWzHq9EYWjZzFw+BJO59EDy5w5s/NmxyIiInKZ6OioTMtYLGe7XS1Avfr1Lvw+LCwMLy8vHI6cDdENCArAYtpJSrIDLg5Pn8j8ASPo9VBjvhsZyqN3F+XwlFdYFGuCkURikgtL6UACDcAIpMXr3zO2Z2m8LuQXfCgVBuDEYrmxt/tZvr/yLk35khZch/9m1YErpkZLWs+KDan0bF+a8mEWOJOFQMw4Vi3ZQGrrerRsFMyEqXGENWlC2ZS/GTwqhsEj29OiXj6mL7FTo3kjQly7+OGPIzi978ydeLyKU6aEBdeR9aw7ns2sga00FcpYcR3+kz/25KBN5bRNOI9z8Fg6VClEkWALJJ87HvsJjkSZGIX9yGeAmfAn0xafolXHlgz9YREvxhxky8YIls2cyLh5u69O+mWl7pPOn4s1/HnoGlPo2dwoczO4/Uxy6hobcHJs7wGSzGr4+59bk8idY3Nzv0ZW60lEREREPIaSN3LrOvcW2bUZ+OTzuTgthBFMpeph+AFGaB1a1S7AgqVnMtnGjfH+8OF5sFcRERG5Utcu91KoUOFMy7lMFwYG6Q4HXt5nFxI5evRojhM3AAGBARhmKsmpZ+9KzPhlfDf1IJ16P8JLZggtvTfy/k+R2AHMFJJTTQzfAAK8wAi8k0e6lMIa8zdfvPE/Jq3eS3SKjYJ3vMb0T+/OcWyZyvL9VRZH1LjN5PTKpaxPa0p4q0YEzYigeYtKONb9zPI/Y2gccx8tWtTCZ1k8rZoXg70zWbzfCd65FI9hPTtiwrBkcoROdnzRlfJfZPBRPi9sFiA9nZykIoyQnLYJFw57OhheeHldejQOHOkmGMbZYzVPMWfoQyRu6EGHRrWoW6cGdVuVpd7trahs6crTc05d0Q6ycK4Ny9k1f8zr3Km7UwYwcQE++PreoLaX1WeSjDZht2M3DYzzCx25c2zu7jeTenpqzrWSSiIiIiKS15S8kVuX4ygHjrpwBc3g8bZvsPS6czZbKd3jHd6/O5SdU77lUPNH6PHuMNb2eJ7fLiyIa5Keno6JH35+N6pj4ayVK1be0O2LiIiIezq0b3fNz0yX6+x0Q4bBls1bWLhoEX+u+pPffvsVAJcrdyZfDQz0xzDTSE453xXrYMvPP7H2oaE8fL9JzO8vM/3I+X3ZSUk1MY38BPgbWAoUpag3JC+cwOeLdpxN8ODgdHQ8aZcfzQ24z3Hn/uoK9oPsPeLCUrohTUtb2bzvknL569K8ji/YD7HvSFYXmwfXySXMXvcSjRu1oWW5ANrUNFn731XEJCezaFU83Vq2psH0eO4sY7LziwXscuZiPPZDZ7dTpgm3l/uSzbuykdRznODYKReWUvUJL25hy+HstS9LQXfbRC5IPcyyCSNYNgGwBlC529uMG9aG29uF4zdnLkmXls3KuT53n28pFU7jMCubrxypA+6VwUVCXCKmpQSVygdhbDyd+y9uuf1MYr3WB9fcpjvH79az0HXqiTlz3Y9LRERERG4qrXkjty7nTuYv3IerYGeG/e8x7qhalEBvKxarLwVKVuP2BqUvZCe9Kj7CR682xyviM557+xMGvzKJvUGt+M/wXpS7kMI0iToRjWktRpsebSjrb8PqG0K5+tUonoVnLREREbl1maaJMz0d0zTZvWc334wdS+9eDzJkyFCWLF5Campqru/Tz98PgxRS0y52K7uOzWHCklhczmPM+HEpMRc+cpGanIppCSDQ34LrdBRRDsjXsCu96xXH32aAxQt/f98r3tLK/fsc9+6vruDcxcyZ27B71eCZEW/QvWYR/GzeBJVuQv8P3+K+Ygaxy2ax6Ew2utjNaBbNWUOKfzMeHdaDBkQw74/TmCTz57wVxBW5k+df7Ug5cwez5+07O7olt+Jx7mTW7O04bFV56osP6Ne8HCG+Z+9LgwoWwK18Wfp2Fi45jsunDs9+/DKdqxXB39uHAqUb0LVNFbzdPA3ut4kcspaldbfW1CwRiLfFwOplIz0hgTTAMDIYZ5OVc+3cybwFe3F61eLZz9/mwYZlKOBrxerlT9FKtakUanGvDE72bd5OvOlDsyeG0KtOEfysFqy+ARQqkC93xoFl4ZkkK9t05/jd2m8m9SQiIiIinksjb+QWls6Wb//LuFaj6dfmBca2eeGyTx0b/kebXuM5aK3I4+88SV3nav7z2iR2O0z4+zNeHhvOLwMGMfyRv+g5dg/pODm0bAnbBtWgZtePWNL1/IY28t5dD/HNodx5u1ZEREQ8j8vpxGK1sn//fhYvWcKK5Ss4ffr0Tdm3X/5850beXPJLM47fX2hGuReuLG2SkpqGaeQn0B/M/UuYvPhpmndszZs/tubNy8o62XXJ369/n5PFoL3cvb+6kpPdE97m0xZjeblBDz6c0oMPLzk2x9HfGfbBfLKTuwGT04umsXhwc+6uV5nkZW+y+NTZDSWtnsfimE70qGOQsno8My+MZMiteJzsGj+Mj5p8w6vh7Rg6th1DryiR+aiXVNaO+YiZd3zEvbX6MHJqnys+v/psZsQ87W6byBkjtCGPvfUGTbyu+MB1ht8XrL181M25fbt/rtPZ+u27fN1iFAOr38s7P9zLOxeKJjFrUAsGLUh1q0zSiglM2H4nz1TrwLs/d+Ddy2Ky58KZcPOZJEuPEu4dvzv7PZRJPYmIiIiI59LIG7mlmQlrGf5gL54bNZvVu6OIT3XidCRx6mAky9Ydxo6FsO6DGVjTyd+fv8tPFxb8TGXT1+/w3V4v6g4YTI8SZy8F5+7xDHr5O5buPkWy00l68mn2bdhDtF5LExER+UdyuUyOnzjOTz//TL9+/XnmmUFMnzb9piVuAPL7WTHMFJLT3MlYmKSkpIDhT2CABcwz/P56P178dilbjsWT5nSSnpZETNQRdm36m9V74i5ME5V79zlZu7+6Sso2vurXi4Gfz2HdwTOkOOwkRe1i+U/v8+D9rzLzWlOuucGMX8HPc47jNBNZMWspp84ffPLfzFgYhdOVwB+Tf+fYpR3puRVPyna+6Xc/fT/6lZU7TxKf5sSZnkrCqcNsW7OI35btI7PJ1FzRCxnc+0n+N3Ut+06nkp6eyul9a5ixaBvJ5tm1lzI/Ce63iZwwjOOs/yOSg2dSSHe5cKbEcChyEWNeeYyXZ0dnvI8snGszMYKPH36QQaPmsu7AaZLsThzJZzi8LYK98V4YbpYhbQsj+w/gv7+tZf+ZVJwuJ+mpCZw6upv1y39n2Z6UHJ+PzJ9JsrFNd47fjf1mVk8iIiIi4rmMoNBCebFeu/wDDXn1VZo1bwbAg3365nE0nmfQ0wMJD28AQMeOnfI4GhEREQEICQnhzJkzWfrOnDmzAVizZi0jvxh1I8ISuYRBofvGsOLt+vz9xh08MuVM7q/bIpIDes4RERERuQEMpmjaNBERERH518pq4kbkRrIUrkeHWrB7636OnYoj1VaA2+rdzUtPNsTbtZcNm3Nn1IyIiIiIiHg+JW9EREREREQ8gE/tBxg+8i78r5zJznRybPZoftyV/SnlRERERETk1qLkjYiIiIiISJ4z8I7dydI1t1G7QimKBvlAWhzH921mxczxfDHpb6KytOi9iIiIiIjcypS8ERERERERyXMmcWvGMqjP2LwOREREREREPICSNyIiIiIiIiLyr3TPvfdQtXKVvA5DRET+xd4fPjyvQ8gy/fcz923bsZ0Z02dc9jslb0RERERERETklhcYFMjbb73FtKnTWLlqFU5n5utEVa1chWbNm92E6ERERK7h1svd6L+fN8gMLk/eWPIoDhERERERERGRXBMcFEyFChUY/Mpgvv/+O+7tci9+fn55HZaIiIhItmjkjYiIiIiIiIjk2IQJP5CYmMiZM2c4cyaGhMQEEhMu/pyYlMCZ02c4ffo0Docj1/cfHBx84e8hISE82rcvD/fpw7z585g6dTrRUVHX/f6DffrmekwiIiIZGfT0QMLDG+R1GLni1d2l8zqEW9rwCgev+ZmSNyIiIiIiIiKSY1OnTic0NJigwGACgwIpXboUQUFBBAUH4WXzuqxsQkICsXGxxMfFExsbR0xMDPHxcRf+HhsXS0JcAmdiY0hKTHRr/5cmbwCsVitWq5WOHe6ic6fORKyL4MeffmTnzl25dswiIiIiN4qSNyIiIiIiIiKSY9OmTb3mZ/nz56dAgQIEBgUSFBRESHABgoKDCAwMokBwMGXKlCYwKIjgoGACAwMu+67D4SA+Pp7YuFhizsQQHxdPXHwsZ87EEBcfT3xsHDGxsZQsUYL09HRstsu7Oqznfq5dpzb1G9Rn586d/PzzZNauXZP7J0FEREQklyh5I7nGYrm4hJK3tzd2uz0PoxERERERERFPkZSURFJSEhxxr7y/vz8hoSFn/z8khJCQEPzz++Mf4E9ogRAqVaqMv78/oaGh5M+fHwCH3Y5pXnub55M6FcpX4M033yA6OprEhIScHpqIiIjIDaHkjWSJv78/xYoVO/enKMWKFqNkWEmKFStOcHDQhXJK3IiIiIiIiEh2JSYmkujmdGm+vr6EFCjAI3370rBhw0zLW6xnXzwsXKgghQsXuvB7v/x+JCclZy9gERERkVym5I1cJTAwgFKlylCs+NnkTNFiRSlZvATFihfDz88PANM0SXemYzEsWK3WPI5YRERERERE/q1SU1M5dvw4NqsVq+36z6emaeJyubBaraSkppGSnExIaAgAKckpNyNcEREREbcoeSNXKRkWxgcfvI/L5cLpdGGzWjAumRINwDCMyxacNE2Tw4cOU6p0qZsdroiIiIiIiAghBUMwrvylCQ5nOl42Gw67nR07d7JuXQTbtm9j185dvPzSSzRr3uxs0evNuSYiIiJykyl5I1fZtnUbGzZupEb1Gnh5Zd5ETNNkxvTpFCxYSMkbERERERERyRPBQQUAcDqdWK1WHA4HO3bsYMOGDURGRrJr126cTmceRykiIiLiHiVvJEPjvx/PJ5+MyLScy+ViyZKljP12HK++8spNiExERERERETkaj4+3mzfvp31688ma3bu2Ikj3ZHXYYmIiIhki5I3kqHdu3ezPmI9tevUvuaaNi6ni+UrlvPZZ59dNbx80NMDb0aYt5Ty5cvndQgiIiKSi8qXL697HhH51/Ok55xHHulLWlpaXochIiIikiuUvJEMFSxUiLS0NCxXrHVzntPpJGLdOkaM+ASXy3XV5+HhDW50iCIiIiJ5KiSkgO55REQ8iBI3IiIi8k+i5I1cJiQkhPvuu4/2Hdpx5nQM23dsp2KFithsF5uK0+lky5atvPfecM0XLCIiIiIiIiIiIiKSy5S8EQACgwLp1rUrd999N/Hx8Ywb9x3z5v5O0eLFGPXllxfKOdPT2bN3L2+9/fZVcwe/P3w4DL/ZkYuIiIjcXB07dsrrEEREREREROQfLuM5seRfIyAggF69evHt2LG0aduGSZN+pF+//sycMRO7w8Ghg4dYuWIlznQnTqeTAwcP8tprr5OWmprXoYuIiIiIiIhINlmLNGHghxNY+lcEu7dtYPMfX9ArTN1Euc0ocDtv/DSbFcPvxOeG7cVK6a7vM3PBVIaG58J72kYQLV4bx9iBDQg1cr657LBWG8T8bZGseiMc77wJ4RaTy21AAF8q3PsWP47sSVmd0n8Uw8eHZ9qFMqWJj8f/+6L/Kv9L5cuXj+7du/Pt2G/o3Lkz06ZN57FHH+fXX3/FbrdfVnbipElYrBaOHj3K0KGvkZKSkkdRi4iIiIiIiEiOeVfj2a+/4KW761ImxBeb1Rv/UAv2eBNrhT5MWrWGVZ93pZR6jTLgQ91Bv7B+3Vw+aBtKZrkNw7c4VWuUpZCfLdOyOREQVpWqYcH4Gjndi0FA02d5p3d9bgu1YM/8CzeAlapt76QcJ1k0f2MOY8hafeWtnMWae23gRrmV6gLAQXpAGNXbPMs794dhzetwJNcYVisVQm2E2Ixz7dCgeq0QZt9fkFdLWTyqbSpv+C/j4+tL506d6NGjO1arlTlz5jB58hSSkpKu+Z0jR44wccJE5s2bR2Ji4k2MVkREREREROTW0HX0ej5undnYCgdr/9uRB344iivX9mylWt/P+fhBf2YN7MuXOzNfm9an4X30rOSNffdkXnp+JAv3JeAVGgTxJhQxsBgGFuv57qusb/9m8W30LBPe6ETZwiEE5PfGatpJjo3m0O5IVs7/lfG/reH4Dcg+GIaBYViweFIPX26xVeDhF7pQ4vRcnvp8DQkm5O/8OREftWD3p125d/ReMm8BBvnLtefF99/g4cprea7hs8zKynvA1ip0aFsGTv7M3I1XV6Bvw2f4/rWOlC4SQoGAfHiRTmpiLCcP7WL9ijmM/2EOm2MuRnkr1deNjvWqa8aZSkJsFId2bmHNyoX8Nm0pO+Ju3DV+K9UFONn/03DG3PsLzw0cyB2zhrIg3szroG5pPkX9GdHAl7B8Fvy9DKymSYrdxfE4B+sPpzJtTxpH0vMuPoOsj3SpUCWY1ysZLFoWw4SY3I9JyZt/CR8fH9q1b8d9PXqQL18+Zs+ezeQpv5LkZjLm519+ucERioiIiIiIiEjWWQguXZUKxWLwdqtD1EKR8uUIMtJYMe4z5uyOxQTSok6f/XjXeHo2GZ+D7d881kLlqVG++CXTkfkSUDCMagXDqNa4A726jqXfYyP5O1c7XNOI+Ow+6nyWi5v0IPmb9eGhKgbbRo5lUWwWz5s1gLKN2tGtSzd6tK9BYS8D0rIeg61aW9qVhuOTFrA+g+SbtXBFalcKu6TevfELKkzZGoUpW6Mp99zbnBd6vcKs4y5urfq68bFedc1Y/QguXIbgwmWo2bwjfZ+I5IfXX+a9RUfJ/T70W6kuzknfzcQxi+j7aTse7zKKReMP52Li/d/Hms9G5SDrxanKDIP8vlbK+1opX8SXuyum8M6ieJYn3+zITLZsOkPHTVn9nkFQgBdl8rtu2PRrSt78w3l5eXHHHXfQu3cv8vv5MW/BAqb8MpmY2Ni8Dk1ERERERETkH2Pqk3WZeuEnG3UHz2Jy3wB+69+KV1Y48jCyKxn4+HpjmCmcOpXErf8eeTrbvniAbqN2kGZayRdUlPJ129LvpafoWONR3u67kLs+2+bGaJGrWXwCCA32JT0hhpjkPHwdPIfcPg4jiNZd76SgPYIvpu/L8jmzFLmb978aQkNvcJzYTKSrGjVDsxqtjert7qA0x/lh/iaufeWks+nT7jzw9R7SDB8CChSidI2W9HluEN0qt+Pp+8Yx97Pt2ar3f750tn3Zk+5fbicVb/IXKEr5mk3o2PsRHmpai0c++Rqjf0/e/ivhH/DvQ06ZxC77jbkn2/FA105UmDgaDxp8mGtatGhB/Xr1WLZ8GRs2bMTlurEpqt2Rp3lyczp2wMfbSlhBH7rW8ueukHw8VzOVNavtaLX1szR76T+UzWajffv2jB37DU88MYA1a9bweL/+jPl6jBI3IiIiIiIiInnMyF+eTs+PYNriP9m+OYL1i35k5FMtKel19nO/Os/ze+Rmtv70GJW8Ln7Pp+oTTN8YyfqvulHy0kUYrBUZNCOS/Tu3sn/nVvaueIMmXlyDAZYC3PfNxgvl92/5nj7FLBih3Rm/eSs7Rt9DINndPpCvDG2f+oAp81ewdfN6Ni+fxvfDehNe6NKgDUKb9mfEmEnMW7ycyE0b2bNtI1tWzWTiWz2pU8C9oT6udDsOp4npSic55giRi8fx4huTOeyyUbZBXYpc0vuV2XkHsITUZ8CnU1kX8Rdrlv9BxPp1bFrwId2KWwArFZ6cwu7tK/mg+eUnwBJSi96v/5+9+w6PongDOP7du0vvBUIvEnoLIKEI0kGqCNKLWLAgIhYQFRRRFBQVQUEBUek/UEGK9EBo0nvvhNDTe3K3t78/EiABktwllwTC+3kenwdze7Mzs7Mze/vuzkzn3+CdnDy6j/3r5zN1SJMM+wc7nhq7mXPHl/JOlYx14fHcNE6dOsDvz3vfWXNB8WnOR38sZevOPZw+fphT+4L495cP6FbZJdt1GbIuxwM4B9KmkSumI5sIumH9zVvzjV1s3n2UVdNG0LnLO/wZkoMbwIbqdGhTGq4G8e+hrIKeGmpyEilmDU1NIibsMkc2zePTX3aQpIGbu2ta/TzoeOWg3eW4Pe9j/4Z5fPdSQyrX684H381h4/bdnDq2j/3r/mBCv5p43tndg9tWbtpAZszGZFJUDU1NJi7sEgeDFjL+lZ4Mmn2SJLty9B81gPTNM+vzxsCTH67l7Mmd/NTeLeOOFB96/bqP80d+Y2BxQy7Kl5NjVobWb3zJotXBHD1ygGO7g1g3dyxdy94tmCX9AUkH2bA9Cp1/C1qVLZwr3zg5OdGqdSvGjRvHgoULeOP116lWtSpKHq2fpJnBqIGmQVKyypkrCXyzPZ4zZvAuYk8ZBdx8nRjW1Itfny3Cmj5+bO5TlKWd3GnmmJqGYmegVYAHv3QtwoY+RVnZ1ZuxNR0odk/3pnO0o2t9T37rVpSNfYuy8llvxtayx/eeopWr4c2mfkUYVeKeDwx6mtTwYGqXIqztU5T1PYswt407bdM3dcXAoI5+bO2f+l9wd3fq2ijqIm/eFDI6nY7GTzVm0KAX8PXxZePGjSxYsJDw8PCCzpoQQgghhBBCCCEAnGsy9NeZDK/jduepWsfSten81hQCSgzj2dHBRB6YzsgZjfjfm28wcfAOek07QbJjDYZ8MZga8Zt4f+xSQh/WJ8Adq/HajFmMDPS4+9SwXyWa9fmQxk/X5r3+H7Liqgro8K7Vhs7NqmW4QeXiW4Gnen9MQCUnug2YzekcvPRiNqmp0xvpdHfzYEm9K8XoMWEqI5u5o5jiCb8Rh+Lqg2dRO5KjzZDJsuWKe2NGz5nKoIqOd244O5QJoEOZ1H/nYPawVCZPaeDL1QAAIABJREFUKtStRKnbc/K4+lG1+UC+qVEU47PvsyIsk3cjdNmV436GynWo7WIm9OAhrufkwXv1LD+/3Dtt/34E5iAJu5ptaVsKQueuI8vYTXo6A44unpSq1pwXX2mEgzmMrVtPZvHWjZXtLlft2Q6v0nV47oNfee6eXNiXfZJeo6fjHd+d15fdyHw6rpy2AWtp0eycOpEl7WYxsGJ7Olb5hRPHVIvOmyNb/iNsYHfqN66Jw+odd9u7Sz2a1HJAPbGNrTe1B9+Itqh8Vh4zhyoM/uVXRjXwvHvM7P3wDyiNa2JaTVvSH2gAyRzefwxj90DqBbjB+cL5ULyqquj1etxcXWnf/hk6de5EVGQkwVu3sm3bNo4fO563GVDIEIz0KebEc2Xt0h1vBW9nSDECBjteaOnFi0WUO8fOwdWOVrU9qeYSxeCdyUQDir09Q1t78ryncidtezc7WqQFXrJdEk1voHcLL97wSzeO6BXK+upxzqeXMeXNm0JCp9PRpGkTfv55OiPef58Tx0/y+utvMHXqjxK4EUIIIYQQQgghHhp6Kg0cw9AAZ24GT+GlDk9RpXo9Gjw/mj/Pq5TqOpR+/nogiSMzRjP1kEbN18bxZi1v6g4Zx2tVYvj3iy9Yfu/ddfU0U56tRfnK1SlfuToVmn7OjqxufpsjWfxKwJ3ty9cYxJxrWdyxtzh9Pf4DxvBOfXeSTvzJyJ4tqV6jDnXaDuaroGsoJdozdmQbMjwor8Ww+sO21K5ViwrVG9Ck3wTWXzfjUrsf/epm9XpPRoreHhfvUlRv2ocvPulBGb1KyL79aYEIy+pdcWtA2wZumI/+wnONGvHk0y2pVy+Qhs9NYlum6zAYqPHyKAb62xNzcC7Dn08tc+2W/Rg+aw9huZiBSIvdzjcDu9Kofj38q9aiasNOvDTrMEk+Leje3CvTNy+sL4eCa7nyFNOrXDwfUkDrethRu10rSnGV9WuOZjFlWuq2dT9Yw7lTx7hw4hAn9gazfs5n9HniJv+MeY3PNlsw5ZdF7S537dm/ZhM6fvwvoaqGOWoPPw59nkb1AqhUtw39p+0jBk+adWtJ0Szu0Oa0DeRI4iE274rGrCtJ1YouWHreJB/Ywo5o8G7UlJrpoitOdZvS0NXMuW3bCMkkmmZV+Sw8ZuX7jeHdQA+STi9j9ID21K0VQNX6LXlm4ETWhmkWlyttp8ReuMgNsx3lnihty9p+aOkNqQfR08uLjh3a883XXzNr5kz69u1LiRIlbLYfJW3Nm6olnRnR2AV/HUSFGQm5c/JqbNsVTpdFN2m+8BY9V8dzSIUnqrgxsIhC+JU4Rq64RasFN+m6OobV0RrFnnDhWc/Ub1eu5kY3T4W4sATGrb5F2wU3ab8sgnHHUoiwIOZZupI7r/jpSI5K5Nv1YXRaeJPWi2/xwoZYtqSf100z8fuqGzSdl/pfs79i2G+jTlSCN484RVEIDGzAlB9+4IORI7lw/gJvvP4G3377LdevXy/o7AkhhBBCCCGEECI9fWU6d6qCIWYD40fMYNO5KJJNSdw8spTPpmwiTl+RxoG+qTdsUk4z8+Op7DZW4fUpC/npJX/CV3zBZ2tuPbyLZusr0uXZ6tgbjzD1vXEsOXSDBGMKUZd2MOP9T1l8TcOreRdapb/branE3rpJTLKK2RTHlb0L+HLuUUx6H6pUKZLNzSsDNYYv5+ypY5w/foCj/61l5azR9KruQuKJeXz667HUhdctrXdNQwOUIlUIrOKLowJoydy6EEpUZjf79JVp17ocuuR9TH7va/45klrmmCsHWTFvHWdz84aUouBZszdfzv6L7bv2cHjTHMa2K4EeA37FfTOvG6vLocPH1xudOYHw8ISCWevEribtW5eA0I2sOZKzdaIUp/K0e+1Netdyyz6oYUm7y2V7VlMiOb50GvOPqSiGMI5vP8H1OCPG+Ktsn/Er66M19KXLUSarRp7TNpAjJiIiotEUHc6uzugsPW8SdrF6SxRKiaa0qnY76OFA3ZZP4aWdY826s5m/CWVN+Sw6Zk/QqXMNHIyHmTLsE+bvDiEy2UhSzA1OHzjNLTPW9cOAFhFGhJZ6jjxuDIbUoFjxEsXp1bsXM2fOYMaMXyhVqlSO06wU4ENwfz+29CvKmud9mdHCjU7eCmpsElMPJ99d70bTiI5XiTRpqKqZG7EqCYodrcrZoU9J4qft8fwXbSbFrBEensgPh5NJ0Bmo56dHp9jxdGkDOjWF2dtiWR9uJtGsERdnZOOpZC5l18kpBlqVt8NeNfL7lhiW3VCJVjWSU8xcuGWyKPhjCzJt2iMsICCAl156kfLly7Nj+w4mTJxIaGhoQWdLCCGEEEIIIYQQmbEvzROldOic2jF1dzum3reBSolSxdGROo2S6dx8Rk1uxorRDSl6awVDJgQR/jCvIm5fFv9SOsyXd7H94j23a+P3s/VAEn2eKYt/aR1EZJaIytVzF4nXquPqasW6HrcDFlose2d/zMifNnEhIa2yLKx3JXYHSzeG0aJjMz6as4H3Ii9x9OA+gpfPY/aaM8Q/qO7tSlCupA5z6H72ZvX2krUUd54e/Tuz+pTF7k4lOFCmdGp+dbrMb+tpOSiHvaM9CimkZDuXUN6wq92WNiXg8u/rOZTtlERG9k/sTI/ZlzGjoLN3xqdEZZ7qPpSPXm7NRz9EcLrDOLYlWpODB7Q7W7Rn9RqXrpqgahH8PHWQkNZGUq4TelNDKeqMU6avUOW8DeSMAW9vDxTNTEJ8App9bQv7q4NsX7mJ8M5dad26CpMOH0O1r0WbZr5oJ//HqjMqD5xyMNfle8AxM5SlYjk95su72ZHZ6z5W9sNaSgpGDewd7O9PywZWrVqZJ+laQzVn33cZ9KnHsETJkhmnOLMzEW60vi2azRqJKWauRRs5dCWJZWeSuZhd3Favp7Qr6AyOjO3pyNgHbFLUVYdOp6OkC5jjjByOtzproNNTzh3McSnsj83B921EgjePoICAAAYNegF/f3/27NnD5Mk/cP78+YLOlhBCCCGEEEIIIbKTFmDInIKDk8PdG2OKJ5VrlMYZUHzq0CLAi3WbIgrmzQiL2GYSJy0lhRRNQdFll56Jo5O70XX6OVTsqThwGos/bECFKkUhOV0tWVrvWhirPhpA3IEetG9Ym7p1alK3RXnqNW9BFV03hq6KfMBX9alP6Cs6i0qvYQYccHTMemvFuzWDniuDPnIXP475mvk7z3Er0YBvq49ZNrlLNjvJrhxh99VHSlIKGvbY58396WzYU7ddS0pwmV/XHcW65SQ0zCnx3Lq4n2XfjcK15lo+D2xAk0p6th2yLhf3tztbtGczxhQTKHbY2aVPz4jRpIGiZPr2TK7aQE441aZ5Aw905kucPBOPZkV/lbD7X9Zc70rfds9Qa8oxjtd7hrZ+KofmruZ8JjEUW5TvvmOm6NApgJZFzq3shxV7e+wUSEnOm8jmVxMm5Em6lgqoXZu2bdtmv6EGqllFp9MRGxuHm3vq4jHWBm5OHwxn8FFTzt4g1ch2/HPQKyjpzqucvZ12d52cghxvJXjzCKlWvRovDBxAjRo1OXjwIMOHv8PZs2cLOltCCCGEEEIIIYSwlPEKF6+YMXv8wyttx7Ap03VUAPSU7fE5X3Xx4dSSXwlpOogeX4xlT493+Ovq7buhGiaTCQ1nnJ1tuvpFztJPucS5UDO6sg14qqyeI+nv2rrUpWkdR0gJ4XyoGdvP5p/CmXkf8XHtRUzp+B6TXjlI319Opi6ebk29J10meO53BM8F9G5U6T6O2WPb0LxdIM6r1j5gtyGpZS7XmOYVfuLI6SwXGyI2Og5NV5LK/h4oB8MzvTGo8y1GMXtIWD+XqRtOpi2ubST8VszdBeEz0KNPf6cvy3L8S8aH0c2Eh0Vg1lXCx8cZhej8vWFpX5sOrYvB5d9ZfTQXK4ErdtjbK4AOnaKQ69uuBdqebdAGrKF40HDoSHqU1GE6vZZVJ1TAmvNmL0uWX6Tv4HY8W3c2np1aUTRpJ5NXXM50yjTry2eBtHNdVyaQRqX1HLn3jal021jWD4Pi7Yu3knqO5IVtW7flSbqWcnF2yTJ4YzKZMBgMXL12lU2bNrMpaBODBr1Ak6ZN8jGXacwqV+LBbJ/IqH9i+C+z7kKxIyQOdO72NPRQOJnpvJdZ70fnak9dNzgV86CNNEyaBig45VGURda8eQRUq1aNL78czzdff43JpPLOO+/y8cejJXAjhBBCCCGEEEI8atRTrF1/HrNvZ8Z+/TKtqhXD3V6PTu+IV6nqNK9f9s6TtnaVBjFpVFPs9v3A8HHfM/KD+ZzzaMGnE/pS4c6NIo2b12+h6YvTpkcbyrsa0Dt6U+HJ6pR4wCxF1rMyffU0y5cfJ8WuJm99N4bna/nhbLDHo2xjXv3mM3oWV4gKXsGGvFowwHyT1eM+ZckVB+oM+YzBVdNeI7G03vXladm9JbVKumOvU9DbGTDFxpIMKEom72Gop1ix8gRGQzXe/HEig5tWwNsxNW0PXy8yxrxUzh85QYzmQJPXP6RvHT+c9Tr0jm4U8XLKkL45/CY3jeDUoBv96pXA1aCAzg5XV8f7nsZOMZrQFE/qNm9IKWd9DsqhEXfxIjdUPeWeKJPvNwwd6rajtR9cWr8ey2I3CnoHJ+x1gM4OJzcfytZswctfTuWdOnaYow6w52wugkC3FXB7zlUbyILOYIdeAfT2uPiWpXaL3nw8azG/vVwVR+NFFkyYwwkVq/orMHH87785qBan04ujeKGtN9FBf7EmLPO6saZ8FlNPsWbdOVS72rw9dRz9G5TDy1GP3s6VYpUDqOyjs7JcCm7lyuGnM3Lx/OOzXIVqSg16hUdEsOrffxn21tsMHvwqCxYs4Nr1awWXMc1I8GUTZidHhj/lwlPeelz1oFMUPFztaFhUn3rsNCMbLhox6ewY0Myd3iUMeKZt5+akw9GC/WwOMaHq7XjxaXe6+unx0INOp1DEy44n0hIITzBjVvQ08XektB3o9TrKFrXDz0bPUsibNw+xKlUq06tXLwIDAzl+/DijRn3IkSNHCjpbQgghhBBCCCGEyDETR38dz+wW0xnc5l1mtXk3w6fGA1/Tpu8fXNJX4pXP36CuupNPP57PGaMGu35gxKxA/vfaMCYM+o8+s85iQiUkOIjjw2pSq9skgrrdTuggX3YYwMyQ3K7BYm36KmfmjmPy07MYUb8H3yzpwTd3PtMwXlnN2Ilr83SxZy16GxM//4em057jjU/6sbb/b5xRLav3EJ8GvPzZGBrb3ZOoOYLV6/bw4KUTVE7/MZZJjWcyKrAdH81qx0f3bJH+LYL4rXOZe6I1b1VvzxeL2vNFhi3vTsukhQexeONQmnZsyScLWvLJvftM9+/QEyeJ0qpQZeB01hZ9n/rjPa0uh+n0fg7GD6Bd7Vr46Y5wNcOhNVBj+HLODr+/7KF/DKLFl/utnOosPQfqtWuBHyHMWHvcwnQM1B6+lBP35Qe0lFBWfvUTQXE5zlA6Bduec9UG3l7Dg18oMVBt6F+cGnrf3lCjjvDH6PcYvyMm7Z0lC/urtLaihixnXvBrfNumE83US8xasJWYrGYvs7h81jBx7Ncv+OXpaQyp0ZXP53Tl8zs7jGfFsKcZti7JinI5ULNuNezUc+w/9MDXLwoFRadDNanoDXpiYmLYtGkzwZuDOXX6VEFn7T6nj8WypKQnvUu7MqG0a4bPjLdiGbAugSsaXDgZw8ziXrzu58ibLR158550spsE78zxWBaW8KS/jxPvtXHivTufaGzccouxIRpXriRzppYdVSt4sKCCR+rHZiM/rYhgkQ3WypE3bx5C5cqV5cNRo5g0aRJubu589PHHjBgxUgI3QgghhBBCCCFEIaDF7mFC/74Mn7aSnWduEpOkohrjCbt0mOC9l0lBR+nnRzKklsquqV+w8M6i20kc+uVzfjtnR93XUqc3AlDP/MGwEb+x6UwYCaqKKSGc8wfOckuxzaO/VqefeJyfB/dlyNRV7L0UQaIxhfibp9my8Cv69xrF8quZTaJkKxpRW6by7aYoHANe4d0OPihYUu+gKNfYv/kwlyISMZnNqImRhBzewIwPXmbEyluZT8KVeIKZg3vx4qQ/2XbqBjHJKqopidiwyxzfvYG/gs9zZzK15KNMefU1xv+1hwsRSahmFVNSLGFXzrB/y2qCzyam7keLYPXowbz36yaOXo0hWVUxJccTeTOU04d2sfPs3anNEoIn8+5PGzh6PZYrodcw5aQc8bvZuDMOQ80WtCiaj7cMHerSoWURuLiB1cezbxvqrbMcOXeN8NgkjKqGZlZJjosg9PRe1iz4njef78HwFVcynarLagXZnnPRBh50Y/reujMbE4kJu8zRHav5/Zt36dKuH5+tv5IhgGbJeZM+v2vnreKqqpF8eDHzD2Uz+ZkV5bOq2uL28e0L/Rk27V/2XgwnPkXFmBDB5eP7OBdjZ3F/AIBjbVo/5YV2LpigB03BVkgkJSayMWgjH330Mf369WfGjBkPZeAGQDOmMH1dBOOOJHEgykycCqpZIyLWyK6b6t2+1mRiYVAEI/YnsidSJU4Fs1kjPlHlzI1kVl8xZRks1owpzNwQwWdHkjgcYyZBBaPJzLWIFC6lpL7BaI5KYNz2eP6LMpOkgclkJuSWCVtNsKd4+BR5eNe4e8yUKVuGfn368lSTpzh95jSLFi5m9+5dBZ0tIYQQQgghhBCiUPpw1Kg7c/b3H/hiAedGiILn2nI8QT915Nrk7nT75ZztAiBZcGwylk0zuxE7ow/tvz+WL/sUwjIKnu0msmFyay583ZXev4XYrH0OGzqEwMD6AHTs2MlGqeaMu4c7iQmJGI1ZrdeVUfrxc9SZsnmVtcfChIqXgNS1j76aMOHuBwpLZNq0h0CZ0qXp0bMnzZs343LIZSZMnMj2bdvRNImrCSGEEEIIIYQQQoj8EbflD+ad7Miw/q/Q6n8fsc7aRb6t5khgu2YU1S7y57qTErgRDxdDRfoPbo1nxDpm/X250LbPmOjCOx3co06CNwXIz68oPXv2pG3btoSGhvL95Mls3rQZszm389EKIYQQQgghhBBCCGEl0xl+/3Ypz8/ozqihy/hv/C5i8zJ+4/QkHVr4op3/mzUnC+utcfFo0lO+9wcMrm5k1/hpbIiWh+xF/pPgTQEoUrQovXv1pE2bNoTdCuOnn6axbt06CdoIIYQQQgghhBBCiAKkEbP9B8bML8uASA0HhTwN3jjXb0dLH41zSzYgsRvxcLHDLj6U4xs2MGaR7aZLE8IaErzJR76+vnTr3o0OHdoTGRHFtGnTWb9+Paoqp78QQgghhBBCCCGEeAhoUQSPf4ngfNhVwpYxBFYdkw97EsJaSZxe+il9lhZ0PsTj7L7gzYejRhVEPh45x0+e4J9l/1i0rYe7B926P0eXLl2IiY5m9uzfWL1qNUaT5YtA2Yoc38Jv6bKlnDx5qqCzYZUqVSrzXNfnCjobQggh8pk111OPM7l+E0LkVIZFb4UQQgghxCPlvuBNk6ZNCiIfj6R/yPpmg7u7G927d6dz584kJiUxf/4Clv/zDynG/A/a3CbHt/Dbun0bPGLBG98iRaRtCiHEYyq76ykh129CiFyQ2I0QQgghxCNLpk3LA05OTnTs2JGePXugmlQWLFjI8uXLSUlJKeisCSGEEEIIIYQQQgghhBDiIZdp8Gb37j1M+XFafublkTBvzm+Zfubo6EinTp3o2eN5VLPGsmX/sGzZMhISEvIxh5aR41u4BAbWZ9jQIQWdDZuY8uM0du/eU9DZEEIIkYeyup4SmZPrNyGEJYYNHUJgYP2CzkaBeOed4Vy4eJHDhw5z8eJFzGZzQWdJCCGEECLH5M0bG3BwdKRdu7b06tULB3t7Vq1axeLFS4iPjy/orAkhhBBCCCGEEI+FqlWr0qpVKxRFIT4hgcOHDnHw4EEOHz5CSEhIQWdPCCGEEMIqErzJBTuDHa1at6J//344OTmxcuVKliz5k7i4uILOmhBCCCGEEEII8ViJjIyiZMmSALg4O9OgQQMCGzRAr9MRFx/P4YOHOHLsKMePHefs2bMFnFshhBBCiKxJ8CaHihUrxuzffsXFxYU1a9ey5H+LiYyKKuhsCSGEEEIIIYQQj6WIiAg0s4aiUwDQ6XR3PnN1caFhw4Y0bNQQnU5HVFQUB/YfoKhf0YLKrhBCCCFEliR4k0NPVHiCFctXsGTJEgnaCCGEEEIIIYQQBSwyKhLVbMKgs3vg5zr93WCOp6cnzVs0R1GUO3/z8HAnOjomz/MphBBCCGEJCd7k0L49e5kxc2ZBZ0MIIYQQQgghhCjU7Ax2eHh64OnpiZenJ+6eHni4u+Pt7Y2Hhyfu7m54e3tTvFgxQMk2PQBVVdHr9cREx+Du4Q4ggRshhBBCPFQkeJNDySkpBZ0FIYQQQgghhBDikeTu4Y6nhyfu7u54eqUFZdxTAzTeXl64e7jj4eGBl5cXzs7OGb6bkpJCdFQUEVGRREdFExUVxYULFyhWrDiNn2qc5X5VkwmdXs+B/QdYuGghz3V9jiZNm+RlUYUQQgghckSCN0IIIYQQQgghhMgVezs7XN3ccHVzxdvLG28fb1xdXXF1ccXHxxtv79T/9/b2xtfXF4Mh4+2IFKORiPBwIiIiiIuLIyQkhPDwCOLi44iLiyMiPIKIyAjiYuOIjIxE07T78tCwYQOaZhKIMasqJlVlzdo1/P33Mm7dvJkn9SCEEEIIYSsSvBFCCCGEEEIIIcR97O3t8fZOC8S4uOHt7ZUahHFzxc013f+7uuLl5ZVh/ZgUo5G42NjUwEtEBBERkVy7fj1dECaeuPhYIsIjCAsLw2Qy5Tq/UfesR6uZzaAoxMbEsGLlKpYvX05cXFyu9yOEEEIIkR8keCOEEEIIIYQQQjwG7O3tU9+GueftGB9vb7y9fXB1dUkN1nh74+7uft/bMbcDMXFxqW/DXL9+nWPHjhMXfztAE3HnLZmCCJJERUUDYNbM6BQdl0JCWLx4Cdu2bUNV1Wy/P2zokLzOohBCCAGAv79/QWfBZvoVu1XQWSi0JHgjhBBCCCGEEEIUQp9+OgZ3d0+8vLzw9PTAwcEhw+dJSUlEREYSFRVFTHQMERERnDt3jpjoGKKjo4mIjCQ6JpqYqGiiY2IeOFXZwyQqKgqz2cyB/ftZ8udfHDlyxKrvBwbWz6OcCSGEEIVXTbeEgs5CoSXBGyGEEEIIIYQQohCKi0/g6pVrREZFpQVooomOjiEiKpLoqChSUlIKOos2lZSUxBtvDCE0NLSgsyKEEEIIkWsSvBFCCCGEEEIIIQqhbyd9W9BZyHfWBm6+mjABJuRRZoQQQohCSsbP/KEr6AwIIdLTU7bbVyxf9zcfBUps9VGieDVnzMKVbJ3QGofsN7ehwt1mCq5ebUPv15gh38xl03/7OHP8AEe2LmVCe1+U7L8qbKqwnCc6SnT5guXrV/JZE7tsti0sZRbCFh6e8yE/xzVr9vWoj7cFKed19/C0SyGEEEII8XCS4I0QVnGg7rD/sX/vv0xs65MnN2DdSlejWmlPHBW5vfsoURxLUK1meYo4G9LaRd63ldsKc5spyHrNNfvqvP3Lj7zfpS7lvB0x6O1xLVIMh+Q4Hu7Z4gvC49S35qasCi4lK1O1lCeOFnzx4SmzEAXv/vOhYMaT+8e1h2NfthlvH6Ex2oZyc0ylnxZCCCGEEFnJs+CNY8O3WbJqPXv37OPU8SOcPbKHA1tX8c+siXw8qDVVPPQ5TFlP9RensWbjHN6snNM0RI7o3KnyzGtMmLGYLf/t5tTxgxz7by0rf/+aj/o1opRVj5o9usdRURQURYdOfmM9GvSeVO/4OhNmLCZ4xy5OHtnHwS3L+d/3I+jfsASOebhraSt541GpV4cGPelT2Z6UM4t5q1MTqlQLoFbz3ny9OzkP9yp966PgcSqreMjp3KncbjBf/vI/Nu/YxanjhzixJ4j1i35iwltdqeP7aPUj1nokzkV9WV5ZuI/zJ3cyq1uRfA2I3F8/2Y8x+V6nen+G/n2I84cXM7RyVm8kutBo9GrOntzPr13d8ilzQgghhBBC5E6evZ+tL+JPTf8Sd18d1zvjWbQcnkXLUatpR158/TBzRo/gyw1XMFmVsg7PstWoWDwS+4f5h1Yho7jX4pVJ3zPy6WIY0tW7vXcpqjcqRbUAF079u5PQZEufJ39Uj2My+37oSZ0fCjofwhKK55MMnTyJtxsWQZ+unTn4VSCwQwUCn3mevks+5fXP1xBitPXepa3kjUelXnX4+VfAQ0lm6+wfWHUmCg1Ivn6J2Dzer/StD7vHqaziYaa41+KVb75nRLNi2KXvL9z98K/jh3+tqhj3ruZAmFpgecxbj8a56FR/AANrO6IoDjR7sTfVlk/lmHU/nnLoQfWT3RhTAHWq88WviA7FoSqvvNOJP4cs5br5/s30Ffswskdp9IqKl683emIprC1bCCGEEEIUHnk8ua6J4z/14fmfTpCEPS5exfCv1ZiO/QYx4KnaDPr+F5RX+zDuv1iZQsZKvkWKMOytoWwODmbnfztJSEjIu53pitNtwk+MauaFFnaAudNnsCjoAOdupWDvVYqq9ZryTJXrbI+WoyigRfMW1A6oTXDwZg4dOozZ/IBf0PlBX5q+305heCN31Gs7mf3TLJZsOsSlaDMuJarStPMghr/Siqo9v2RGzHW6TTpIHp5F4rGj4OBoj6IlEhYWL2OceGTpHNzw8XTEFBtJZEK+3DEu1Nzc3Bg1ahSbgzezY8d/xMfFFUxGbl/bNU+7tpuWdm0XruLqV4py/rVo1syd7Qfy8k1BkS2dH11e6kzJhO3M31iK3l168ErzP3h3Q4yMK7c5+OLnoZASFY2+6asMrruaz/cmZdxG8aTt6y9QMzmKKHt3vL29ULhUMPkVQgghhBDCCnm+MqLZmEyKqqFgqOdhAAAgAElEQVSRTFzYJQ4GXeLgpn/ZOGI2s1+qTP9RA1jcbRonVFB8mvPhd2/TvnIp/Nwd0BLDOLd3LbO++5Glp+65+aWvxLB/DjPs9n5uLmJAy8/ZYbQynUeUXqdQr1496tWrh8lkYveePQQFBbFvz15SjLZ9hcC58eu839wbwoL4qM87LA65e/Mm+eY5dq8+x+7Vd7e32XF08afjq0N4uVNDqhR1IPHGKbYt/YWvZwQTmr6IjqVoMeA1Xn62CbXK+OBEEtG3rnD+9FHW//Yts3ZH3d2nUznavvQGg7s0ploJF8yRl9gX9CfTflrE7lu3n79T8AjoydsvtKV+dX/KFfPESUkk7NJaPhs4jrO9FvHvsOL8/WoLPtiaLiNOZWg96HVe6fIUNUq5oyRGcuXUFqaN/pxll9THol0CODs70aZNa9q0aU1sbCybNm8ieHMwp06dRtPyr5Suzd7g7cYeaDfWMKLvSP65evf5ypRLB1j+40G2HviIxb/0oVL/4fRY9BJ/hJrJ+viPZXWEhs67Nn2GvE6/NnV4wseOhGsn+W9nNH4ZJqLUU/GNe9uKgs9Tg/n4hWZUq1CaEr7uONtBUmQIBzcs5NvJizgQebeObN1msj+nrMsfkG27t+ycS5Vf9QpY12/cy+IyKaDzoufMg/S8/SfjHj5t8xJzrj0gqGnLPkT6VivPEwNPfriKRS94sPadNry5Ot37UYoPvWat46vAw4xt+zJzU562YH/Wl9Wqcij2VOo6mt/eb8qTT3ijj7/O8R0rmDVlNmsuJmZbWkvagM77SQZ/8hGvta6El52CphmJDVnPuEEf8NfVAgrKFwKKAgEBtQkIqM1bQ99k3979BG3axK5du0hJScm3fNy9ttvM6L7DWXTp7jmXfPk04ZdPs2/TA/KfB+OIJWlmNS6vUZrlsA+4fzxRPLry67bxtLB/wOYpOxnd6hXm39Qs7kctG9cyp6/Qlf5POXFz6RwmzqpI1Q7v0bZfB4oHLeLe09CafeV8vL39UWZjzL3f0SzuW+dcM1s+PmUoiy8+OjM3/53OirrvM+CNzvw6eEmG+jFU7M2Qtg789+004oa/T2Nfz4xzh9v8WiWtiDkojxBCCCGEEOnlefDmgbRodk6dyJJ2sxhYsT0dq/zCiWMqmDypULcSpW7/YHL1o2rzgXxToyjGZ99nRZiFtyptlc4jwmAw0KB+II0aNiQ5OZmd/+0keMtW9u/fh8mU+6dkG3VuTVFdCgd+/Ya/QixIzxb171yTob/OZHgdtzs/rhxL16bzW1MIKDGMZ0cHE6kBjlUYPGMWowK90s2t7YJPqUr4lHoC+/2zmb07KnVaBMdqvDZjFiMDPe7+YPOrRLM+H9L46dq81/9DVlxVAR1FGz3PgA7V0p0gbhQpakdyZg/IOlRh8C+/MqpBuh+D9n74B5TGNdFsu3p5RJhMJgwGA25ubnRo354unbsQGRHJlm1b2bhhI+fOncvzPDTu0BwfJZk9s75LO6730ojc8ROTN3RgyjMBdGpVgrl/hGLO8vhrKO6NGT1nKoMqOt6Zd96hTAAdyqT+O+tnlHV412pD52bVMnS+Lr4VeKr3xwRUcqLbgNmcvn2a2bLNWHROWZm/7Nq9xecc+Vuv1vQb97KiTFbJrz5E+tZMmDiy5T/CBnanfuOaOKzecbfNudSjSS0H1BPb2HrTDK6W7C8HZbWmHIoLAZ2ev/v/9qWp13EIdRoH8Fn/Icw5m8VdQUvagFKMHhOmMrKZO4opnvAbcSiuPngWtSM5WgI3tqLXG6hf/0nqB9bHpKrs3rWLDRuCbHb9lpXb13aHfvuaJZcsvIucF+OIhWlmNS7jlM/XVxb2o7kb1wAcqNuzG1WVC/y8cCexl04wf9trTHr6ebr7L2Hq6btjjTX7yn2+rGFF32rp+HQPxdMbL51G1M1dzJm9hb5fvshLdZbzxb7k2wWm1at9qRK2gkFLT9H+FQ1Hbx9cFEjRyLtrlRyWRwghhBBCiPQsfO4rDyQeYvOuaMy6klSt6AKAFrudbwZ2pVH9evhXrUXVhp14adZhknxa0L25V8YFOtXTTHm2FuUrV6d85epUaJr6RLHV6RQSeoMeRVFwdHSk6dNN+OSTMSxYuIC33nqLatWroSg5L3W1Sq7o1Ats2XbFormhc38c9VQaOIahAc7cDJ7CSx2eokr1ejR4fjR/nlcp1XUo/fz1gJ4K/T/hvUBPUs6v5NMBzxBQsxb+NRvQ5JNgEjI+ho7/gDG8U9+dpBN/MrJnS6rXqEOdtoP5KugaSon2jB3ZBi8lQ0FY/2knnqwTgH+tRjTt8QO7Hnh/Q0/5fmN4N9CDpNPLGD2gPXVrBVC1fkueGTiRtWk3DR7HdglgMKQuHuvl7UXHDh2YMuUHZs2cSd++fSlRvHie7beyvws69Sxbt18n01uNWjQ7th7BpOjxr/IEGZbefeDxN1Dj5VEM9Lcn5uBchj+f2o5qt+zH8Fl7CLP0nqYWw+oP21K7Vi0qVG9Ak34TWH/djEvtfvSre3exXdu1GUvPKWvyl12711lxzuVnvVrTb9xfj1b3I+ZIFr8ScKePK19j0APeusmDPkT6VqvPk+QDW9gRDd6NmlIz3V1np7pNaehq5ty2bYSoVu7P4rJam24KF/6dyEudm1GtRl3qtH2ZcasvYfJsxIj3O1E000Jb1gYUtwa0beCG+egvPNeoEU8+3ZJ69QJp+Nwktsn8kjZx+zRSdDoURcHOYKBhwwZ8+ukYFi1axPvvvUdAQECurt+yUrWSKzrzBYK3hlq47kfejCPWpfng88mW11da9DJeqln9bp9dvR3DV4RiNCdwbMFs1obpLMxz7sc1xb0pfTuVQD30N0tOmEALZ+3ijYTpKtGjZ10c72xpzb5sNN5m8TvsXpb1rVa2hXR0nt54Khqx0THcXPM7f14pyfMvtsU37cDryzzHK21cOTJ/HjvjYoiO1dB5eeOjA+vGMGvqLuflEUIIIYQQIr2CC95gIiIiGk3R4ezqnJoRRcGzZm++nP0X23ft4fCmOYxtVwI9BvyK+1qeWVul84jS6w0oCrg4O9O6dSu++fpr5s6Zw6uvvZqj9NxcFDBHEhFl4S+63Na/vjKdO1XBELOB8SNmsOlcFMmmJG4eWcpnUzYRp69I40BfdPon6NCxOvbqSX5+ZzRzdl8mOkVFTYnjVnjcPVMIVaTLs9WxNx5h6nvjWHLoBgnGFKIu7WDG+5+y+JqGV/MutEp/h1EzEXkllPAEI2pyDFcv3SD+QTd29U/QqXMNHIyHmTLsE+bvDiEy2UhSzA1OHzjNrdvV9pi3S0h9SwygeIni9Ordi5mzZvLjj1N58sknbb4vN2cFzNGEZ/mUuEZ8ZCRJmg4nF5eMryI+6PjrKtOudTl0yfuY/N7X/HMktR3FXDnIinnrOGvpCxeaSuytm8Qkq5hNcVzZu4Av5x7FpPehSpUid9uCrdqMpeeUNfnLrt0rVpxz+nysV2v6jfvqMQf9iEXHJ5/6EOlbs5awi9VbolBKNKVVtds31Ryo2/IpvLRzrFl3NvUmtzX7s7SsVqcbz56/F7LpdBiJxmSiLu3ktw/HsuiKGZeGbWjqmUkbtLQNaBoaoBSpQmAVXxwVQEvm1oVQouRJ8Tyj16eOQk5OjjRt2oTx479g3vx5Ob5+y4q7iwJqJOGRFl7b5ck4Ym2amZxPeXV9pStC609/5utOvlxY9B4vTtxOmGJpP5rbcU3Bt0032ngmseOvfwlJO0zx2/9ieSiU7NCNpq53j43F+7LVeGsNS/pWa9tCOg4eHjjrzMTFxmNOPsiceQewbz6QPhX1gCOBA/sSkLCRWX9eRCWB2DgNxdMbTwXrxjAr6zmn5RFCCCGEECK9gpk2LW3X3t4eKJqZhPgENMWdp0f/zqw+ZbG7c8/BgTKlAVR0Oguzaqt0stGkaRNWNV1pk7Ty0u2b5V7eXjzbpcudv3t4uFucRnwioPPA00MHN7P5VWeL+rcvzROldOic2jF1dzum3reBSolSxdEZfKlYTo/58g42ZzVFDIB9WfxL6TBf3sX2i/eUIX4/Ww8k0eeZsviX1kFE9lnMwFA2LR+72RGSSf3kU7sE+HDUKBhls+SsZumc/QZ96g/48uXKU758+Tt/d3e3vG1mJTZBA50HPh46CMus3Sq4eHnhqGgkxCeQ7SQ1diUoV1KHOXQ/ex+0ZkmOqVw9d5F4rTquri6pTwnbss1Yek4RZnn+smv31pxzCflYr3fybUG/ca+86kfyqw+RvjUb8WxfuYnwzl1p3boKkw4fQ7WvRZtmvmgn/8eqM2re9eW2SDfxMLuOpDCgbUnKldBB5AO2sbANKLE7WLoxjBYdm/HRnA28F3mJowf3Ebx8HrPXnMk8AJUDj8r1VH4z2KW+neLp4ZHh+s3Ly9Mm6Wd3bWff7lsO/dCGkOm96fDDcdS8GEdynSZ5eE66Uf/tH5ncowy3/v2Ql7/YkhowdrIwz3ZFcjeu6UrSuVtDnGI2snhd2N2gecohliw7wwtvtaBHK282/BOBZs21SZ5dx2TFgr7V4rZw4563qXW4ebih01KIj08BzFxeNo+1r31H3wGN+G2KDy91KcblJR+wIUoDJZ64eDO6su64K4BdHl2r2KJtCyGEEEIIQUEGb5xq07yBBzrzJU6eiQfvZxn0XBn0kbv4cczXzN95jluJBnxbfcyyyV2yTy+N4t3aJulk5+TJkyxdtsxm6VnL08ODN954w6JtVVVFr9dz48YN/Pz8AIiOjrF4X2cuJKJVLk/D+kWYdiaLKaiwUf2nPfGbxV5wcHJA0dlh0AEmkwVTfuThpGSKLnVNCC3zXOdXuwRYumwZJ0+etGma1gioXZs2bdpku52GhtmsoVMUrl65QsmSpQCIibG8bWbl1NkEtCoVaNLIj+nnrj643SruNGpSA4Nm4uypc9m3I0Wf9pagzuYtSktJIUVTUNIWGLFpm7H0nLIif9m3eytqKB/rFav6jXvlUT+SX32I9K3ZStj9L2uud6Vvu2eoNeUYx+s9Q1s/lUNzV3NeBcUnb/py25TjdjvP4jyytA1oYaz6aABxB3rQvmFt6tapSd0W5anXvAVVdN14c5XtbjgW9PVUfnNycmL4229btK1JVTHo9YSHh+Pj4wNAZGSUTfJx9mLqtV0jC67tgLwZR2yQZt5cXxko230CP71aDdOeybz28SpCb3eGluY5l+Oa3r8z3QMc0Bk6MG1PhwdsodG0WzuKr1jIVWv2lYfjbVay61tz3hYU3NzdULQkEpLSptKMCea3vy/Rqd8g3te8aWZ/kK8WHiYFQEskIUlDcXTDzQ7Q8uhaxQZtWwghhBBCCCio4I3iQcOhI+lRUofp9FpWnVDR+RejmD0krJ/L1A0nUy+wMRJ+K+aehTM1TCYTGs44O99/yavztTSd3Am7Fca2rdtsmKJ1/PyKZhm8MZmMGAx2xMTEsDl4M1u3buPE8ROsXLnC6n39t3EnMe1a0XDwMNquH82aW5n/xLe8/rM4jsYrXLxixuzxD6+0HcOmzObXN9ThapgZXZknCSyh4+jlLG49pFziXKgZXdkGPFVWz5Hz6W5JutSlaR1HSAnhfKgZq2cTTMuvrkwgjUrrOXLvk3vkpF3q0efw7Dx58mSBtk0Pt6zfnDGZTBgMBq5dvcamTZsJCgrCv6J/6htDNvTf6k2Ed3yW+oPfpXPQB/xz3yLyCl6N3mR4G0+UpH2s2phJgCe9lJDUdlSuMc0r/MSR01a+uWEFm7YZS88prJh/Pbt2b805l4/1ivG65f3GvfKqH7FpHyJ9a6761qS9LFl+kb6D2/Fs3dl4dmpF0aSdTF5xGRXQ59E1hi2uXRSPxrSqaw8plzh/JX19pyuzxX0BkHSZ4LnfETwX0LtRpfs4Zo9tQ/N2gbDq3xyV80EK+noqv7m7u0EWwRtVNaHTGUhITGDrlq1sDNqY4+u3rOzYsJOYtq1oMHg47TZ8zOrs3qzOw3EkN2na/vpKwa3+2/w8phmeIX/y+vDZHEvMQZ711XIxrtlRu2tnKmeTT4d6XXm23GKmX7JiDM31eJv177BMZdO3WtU3ZaDg7u6KoiWTkHg7XGLk6KKF7BnwES/00ohcPYJlobfHsRQSkzQ0xQU3VwXC8+haxeq2nfPrfiGEEEIIUbjl+VS7OoMdegXQ2+PiW5baLXrz8azF/PZyVRyNF1kwYQ4nVDCH3+SmEZwadKNfvRK4GhTQ2eHq6nhPhEnj5vVbaPritOnRhvKuBvSO3lR4sjol9NakU/ioptRJn5KSkti2dTufffY5/fsP4JefZ3D82HG0LJ5ezkrkmunMOpaMrkQXJi+axojn6lPB1xmDTo+9W1EqNejE6+88R1Wr6j+L48gp1q4/j9m3M2O/fplW1Yrhbq9Hp3fEq1R1mtcvm5qW6QTrg65hdqjD29+OoHN1P1ztHfAqW59ubapin6FyTrN8+XFS7Gry1ndjeL6WH84GezzKNubVbz6jZ3GFqOAVbIjIQR2pp1iz7hyqXW3enjqO/g3K4eWoR2/nSrHKAVT20VnVLlOMJjTFk7rNG1LKuXAsZmpKa5vhEREsW7aM1197g8GDX2XBggVcv349T/YZu/lnftgRjVLsGb5Z8DOjutWngq8TdgYHPEvVosMb37NkWl8qGoycmT+ZxZbcxFdPsWLlCYyGarz540QGN62At2Nq2/Tw9cKa+xjZsWmbUS08p6yRXbv3PGP5OZeP9WpVv3FfmfOoH7FpHyJ9a+76VhPH//6bg2pxOr04ihfaehMd9BdrwlLzn1fXGFanq+hx8/XBxU6HzuBK8Vod+fCnsTxbBCKCVrApWntwmS3tC/Tladm9JbVKumOvU9DbGTDFxpIMKPKouM2pJhVN00hKSmLrlm2MG/c5ffv0ZerUqbm6fsvK3Wu7Tkz+38+M6t6AikVcsNPrMDh5U7qYe8a3AvJoHMltmrbuAxTv5oyZ+AKVtaP8+O4EgsLvqXtL85ybcc2xHs91KIUucQcfPV2T8pWr3/NfAF1+PovZripdO1dCb82+cj3eZv07LHNZ9605bwsKzq7OKCSSlHz3WJmvrmJuUBRm9Sr/LNhE5J2PzCQlJKHp3HB31Vk3hllZz5aWpzBe9wshhBBCCNvJ43iGgWpD/+LU0Hv/rqFGHeGP0e8xfkdM6mvl4UEs3jiUph1b8smClnySYXuV0+n+HRIcxPFhNanVbRJB3dL+bDzIlx0GMPOypekUDrd/0JuMRnbu3EXQps0cOLAfo9GGT64bTzJ92AcUnTaeflWbMmRCU4bcu43pGLp/lnPigm2O46xfxzO7xXQGt3mXWW3ezZidA1/Tpu8fXDInsWfGJJa3mkTX2gOZ8vfAezOVYd9n5o5j8tOzGFG/B98s6cE3dz7TMF5ZzdiJa8nJ/UUwcezXL/jl6WkMqdGVz+d05fM7ScezYtjTDFtveb2EnjhJlFaFKgOns7bo+1R/e01OMlWg9Ho9JtWEQW8gOiaGoKAggjcHc+bMmfzLhBrC/PffxnvyJIY1aMxrXzXmtXu30eI5ueQTXpt8AMse8lQ5/cdYJjWeyajAdnw0qx0f3bOFrd7w06zoE+9tM/XfXnNPeUwcteicsiaHFrR7i8+5/KtXsKbfuFde9SO27UOkb31wvWR/nqRtGbKcecGv8W2bTjRTLzFrwVZi0vJv+XlpHavTVdxpP2Ej7SdkSIXkS/8wZuL6tBuVDy6zJX1BiE8DXv5sDI3t7tmvOYLV6/bksJQiA01DQ0NVzezes4egoCD27dlLii2v37KSdm3nN208fas25rUvHzBGZpg4MW/Gkdymmbuxcv196dnVbU+HEnoUpSbv/L2PdzIkd5XfX2jPOIvynPNxzblRZ57xU4he+xerbz6o8EaO/72Ugy+9T0DHTgRM+4Z9Fu8rt+NtNr/DQrL4ZhZ9a87bgg5nF6e0N2/S/VmLZvW7Tajw7r3bayQmJaMpLri7ppbH8jHMmrqztDyF47pfCCGEEELknTx780a9dZYj564RHpuEUdUwGxOJCbvM0R2r+f2bd+nSrh+frb9y9xaQFsHq0YN579dNHL0aQ7KqYkqOJ/JmKKcP7WLn2eg7cwerZ/5g2Ijf2HQmjARVxZQQzvkDZ7mlKFal86gzmUzs37efSZO+pVefvkyYOJHdu3fZNnCTRr26gU96d+OFL+ayZv8FbsYkoaoqiTE3OHdoK3/OXMj2SLPNjqMWu4cJ/fsyfNpKdp65SUySimqMJ+zSYYL3Xk6bFgPMt9Yzst8bfP33Hs6HJ2EyJRF+fjf/bDhOggZmLd2vvMTj/Dy4L0OmrmLvpQgSjSnE3zzNloVf0b/XKJbfN62W5bS4fXz7Qn+GTfuXvRfDiU9RMSZEcPn4Ps7F2KFYUS8JwZN596cNHL0ey5XQaznOU0FKSEhg4/qNjBr1If379WfWzFn5G7hJo0XuYcpLXXluxHT+2naC0MgEUkzJxN66wN51cxn78nM8N2YNIdacMoknmDm4Fy9O+pNtp24Qk6yimpKIDbvM8d0b+Cv4PDY5A3PRZlIelJyF55RVWcyu3VtzzuVXvWJlv3GvPOpHbNmHSN+a8/MkdacRrJ23iquqRvLhxcw/lJzhszy5xrA4XTPhh9axYsshzlyNJCFFRTUlEhFymDW/fkLPXmNYfePusXlQmS1pA4pyjf2bD3MpIhGT2YyaGEnI4Q3M+OBlRqy8lZMSinTMZjOHDh3ku+8n06dPX8Z/MZ7/dvyXf4GbNOrVDYzp3Y0Xxs+7c21nUo0kxoZz+fQBNq9cxKL/7q6HkyfjSG7TzIs+wFZ5zsm4prjRrHNzvLnFmr+CicqkQ1Evr+GffcnoSrXj2XqO1u0rl+Ntlr/Dsqy4LPpWa+o1Q3054OKsR9ESSUi2pPfVSExMBMUVd7e0n8F5dK1iaXkKw3W/EEIIIYTIO4qHT5EMV7qrVq0EYPfuPUz5cVqBZOphNm/ObwBs27qNryZMyGbrvGNvZ4eDoyOxsbFWfe/xOL4KRXrOYOu4J9k1phWDlkQUmoBdZgID6zNsaOr7UF9NmFCwa964e5CQmGBVELFJ0yZ31ryZ8uM0du+WJ7tFfnv8+g3rSR0J23lYrqfym8FgwMXZheiYaKu+93hcvwkhbGXY0CEEBtYHoGPHTgWcGyGEEEIIkSMKSwr7MjCFVorRmO9PaD6MdEXr0b42nDl2gath0SQZvHiiXhfef6MB9uZzHDhSeN60elRYe0NKiPwm/Ub2pI6EyBsmk0nGSSGEEEIIIYQQFpHgjXikOQT0ZsKUDrjeO1ODpnJ15XQWnM75dD1CiMJJ+o3sSR0JIYQQQgghhBBCFCwJ3ohHmIJ91Ck27X6CgIplKObhAMnRXDt/hK3L/+DH+bt44DqvQojHmPQb2ZM6EkIIIYQQQgghhChoErwRjzCN6N2zGDZwVkFnRAjxyJB+I3tSR0IIIYQQQgghhBAFTVfQGRBCCCGEEEIIIYQQQgghhBB3SfBGCCGEEEIIIYQQQgghhBDiISLBGyGEEEIIIYQQQgghhBBCiIeIBG+EEEIIIYQQQgghhBBCCCEeIhK8EUIIIYQQQgghhBBCCCGEeIhI8EYIIYQQQgghhBBCCCGEEOIhIsEbIYQQQgghhBBCCCGEEEKIh4gEb3KoVu1a1KpVq6CzIYQQQgghhBBCCCGEEEKIQkaCNzmlwVdffcm4cZ9RqVKlgs6NEEIIIYQQQgghhBBCCCEKCQne5NDhw4cZMXIkDg72fP/9d4wf/wX+/v4FnS0hhBBCCCGEEEIIIYQQQjziJHiTC8ePHeeDDz7k449H4+LiwuTJ3/Ppp5/wxBNPFHTWhBBCCCHEQ8bOYEeNGjXxcPco6KwIIYQQQgghhHjIGQo6A4XBwYMHGT78IAEBAbz00ov88MNkdmzfwdx58wgNDS3o7AkhhBBCiIeAalb5fNxY7B0ciE9I4EroZS5cuMjly1e4fDmEy6GXuXXzFmazuaCzKoQQQgghhBCigGUavPH392fY0CH5mZdH3sGDB3n77eHUrx/IwAH9mT59Gju27+CPOXO4evVqQWcvAzm+hYuXl1dBZ8Fm2rdrS8PA+gWdDSGEEMLmzGYzl0OvUKHCE7g4O1OpUmUqVPAHDfQGPQAmk4lr169x4fxFQkJSAzq3yfWbEMISMp23EEIIIUThkGnwxtvbi0C5gWo1TdPYvXsXe/fuofFTjRk4YADTp09jy5atzJ8/n+vXrxd0FgE5vuLhVbGi/NgUQghReJ05e5py5cqg16dehuv1+gyfGwwGSpcqTakSJTGZG2JnsLvzmVy/CSGEEEIIIcTjQ9a8ySNms5ltW7fx+utv8M2kSVStVoWff57OW28NxcfHp6CzJ4QQQgghCsDFCxcBJdvtlP+3d+dxVdX5H8ff514uILIIiALqReWipmmWiWVqptniUrllZTpZUzM1jq2albZPOU3TNNYvJ7Mm10xLza1yqUxtEpfccuNqcRVcUAHZ4S6/P1BCxUAF7lVfz3/ywffccz7n++D7vXQ+5/P9mkyy+FnkcrmqPSYAAAAAgO8xwiKjPN4O4lLg5+enLtd30b33DFZEZISWL1+u6dNn6OjRo94ODQAAANUsODhYTZo00XUdO6rPbX0qPN7tcstkNmnd2nX6z3/e1/4D+2sgSgAAAACATzA0m+RNDbNYLOrevbsGD75HtYOC9NWSJZr96SxlZGZ6OzQAAACcJ7PZrAYNGqhJ48Zq0rSJGjdpoiZxcaobFSVJys7OUUhI8Bk/7/GU/GmekpKiCRPe19atW2okbgAAAACADyF54z0BAQG6+ZabdefAgapVq5YWLlyo2bM/U05OjrdDAwAAQCXUDg5WXJxVNptNcVarrFar4uPjFRAQIJfLpfT0dDkcDiUn22W375bDkaKDBw9q6tQpCg8PP+18LpdLOQa+4dcAAB2mSURBVDm5mjJlipYsWSK32+2FuwIAAAAAeB3JG+8LCAxUn969NXDgAJnNZi1atEizZs1Wbm6ut0MDAACAjlfTNGwgq9UqayOrEhJsslqtio6OliTl5OSUJGnsdjlSHHLsdciebFdRUVG553vpxRd19dXtJKNk7xunyyl5pHnz5mnmzE+Vn59fY/cGAAAAAPBBJG98R2BgoHr37q07Bw6Qy+3RggULNHfuXP7nHQAAoAYFBwfLeko1jc1mk7+/v5xOp9LS0uRIcSjF4Sitpjlw4MBZXWPo0KEa0L+fZBgymUz6/vvv9dF/P9bh9PRquisAAAAAwAWF5I3vCQkJUZ8+fdS37x0qdhZrzudzNX/+/DO+uQkAAICz5+fnp9gGsSdV09hsNkVEREg6uZrGfqKiJiVFRcXF533tLl266OmnR2lX8i795733tXPXzvM+JwAAAADgIkLyxneFhoWqf79+uu2225SXn6+5c+Zq/hdfVMkDAwAAgEtJ2WqaBFuCrNZGimscJ4ufpbSaxm7frZSUFDkce5WcvEsZGRnVFk/9+vXUvHkLrVy5Uh4Pf4oDAAAAAE5B8sb3hYWGqV//vrrt9tuVlZGhmZ/O0tKlS+VyubwdGgAAgE85UU1TdsmzhGbNFF6njiTp6NGjcjhOLHlWUk2T8muKip28HAMAAAAA8CEkby4cdaOi1K9fX/XseauOHsnQrFkkcQAAwKUrIiKiZMmzOKsSbAmy2eLVsGFDmUwmFRcXa//+/SdV0+zauUOZWVneDhsAAAAAgIqRvLnw1KsXpUGDBqlHjx5K25+mWbNm67tvv5Pb7fZ2aAAAAFXOYrEoJjZGNputtKKmSdMmCgsNk1RSTWO320+qqNm3dx9/GwEAAAAALlwkby5c9aPr686BA3XTTTdp3959mv7JDK1etZp10wEAwAUrIiJCthN70sTFnVRNk5+fr9TU1JI9aezJcqQ4tOeXPTqWdczbYQMAAAAAULVI3lz4rI0aaeCdd6pr1+vlcOzVJzM/qXQSJygoSHl5eTUQJQAAwG+CgoIUGxsra5y1tJqmadN4hYaGSKKaBgAAAABwiSN5c/GIaxyne+66W9d1uk67diVr5sxPlZS05ozHh4aG6N/jx+vll17WL7/8UoORAgCAS8mJahqbLV5xx/eoOVFNk5eXp7S0tNJqGrvdrj2796igoMDbYQMAAAAA4D0kby4+jZs01t2D7lKnzp20bft2TZs2TZs2bjrtuGHD7tOAAQOUnZOjkU+O1N59e2s+WAAAcNGoXbu24hrHydqoJEGTYLMpvmlTBQQGSvqtmiY52S7HXoccDof2Ovay5CsAAAAAAKcieXPxatGiuQYNGqTExERt27ZNkydP1datWyRJYaFh+njyx/L3t8jlcik7O1tPPvmUDhw44OWoAQCArzOZTKpXr56s1riTqmkaNWokwzCUm5urlJSUk5Y82717jwqppgEAAAAAoHJI3lz8WrZsqSFD7lWbNm20ceNGTZ48RV27dlXvXj1l9vOTJLlcLmVlZemJJ59S+qFDXo4YAAD4itrBwYqLs55cTRMfr4CAALlcLqWnp8vhcCg52S67fbccjhQdPHiQahoAAAAAAM4HyZtLR7t2V2nwvYPVLKGZ3G63zGbzSe0up0uHD6fryadGKiMjw0tRAgAAbzCbzWrQsIGs1pJETUKCTVarVdHR0ZKknJyckiSN3S5HikOOvQ7Zk+0qKirycuQAAAAAAFyESN5cel54fqzaXX31ackbSXI5nUrbv18jR45Sdna2F6IDAADVLTg4WNY4q2w2W8mSZ9aSf/v7+8vlcik1NVWOlBNLnpVU07C0KgAAAAAANYjkzaUlql49fThposxmvzMe43S69Ouvv2j06GeUn59fg9EBAICq5Ofnp9gGsSdV09hsNkVEREg6QzXNrmQVFRd7OXIAAAAAAC5xJG8uLY8+OkLduneXXzlVN2W5XC7Z7XY98+xzbC4MAMAFoGw1TYItQVZrI1nj4uRvscjpdCotLU12+26lpKTI4dir5ORdLJMKAAAAAICvInlz6YiJjtHED96XyWSq1PEut1ubNm7Uyy+/omLewAUAwCecqKYpu+RZQrNmCq9TR5J09OhRORwnljwrqahJ+TVFxU6+ywEAAAAAuGCQvLl03H7bbeo3oL8iIyJkGIYkyeVyyuOR/PzMkozTPuNyu5WZkant27bVcLQoa+68udqxY6e3w4APeGb0aG+HAOAsnO/8HRERUbLkWZy1tJomLi5OFotFxcXF2r9//0nVNLt27lBmVlYV3gEAAAAAAPAKkjeXHovFonr16ikmJkYxMdGKiY5RgwYNFNuwgepFRcnPr2Q/HLfLJRlGpSt1UH1eHzdOq1au8nYY8AGLFi30dggAzkJl52+LxaKY2BjZbLbSipomTZsoLDRM0unVNHa7Xfv27pPb7a7uWwAAAAAAAN5gaPaZd67HRam4uFipqalKTU09rc0wDEXWrauY6GjFxMYoJjpGd9450AtRAgBwcYqIiJCtTBWNzRavhg0bymQyKT8/X6mpqXI49mpNUpIcKQ7t+WWPjmUd83bYAAAAAACghpG8QSmPx6PD6ek6nJ6uLVu2SFJp8iYpaa3Gv/ueN8O7pCQmtteI4Y94Owz4KMYj4LvKzt/R0dHq1r1baTVN06bxCg0NkVRSTWO325WUlKTZn31GNQ0AAAAAADgJyRsAAIBqMOy++5SXl6e0tLTj1TSfyG63a8/uPSooKPB2eAAAAAAAwIeRvAEAAKgGEyZM0MKFi7wdBgAAAAAAuACxGz0AAEA1yMzK8nYIAAAAAADgAkXyBgAAAAAAAAAAwIeQvAEAAAAAAAAAAPAhJG8AAAAAAAAAAAB8CMkbAAAAAAAAAAAAH0LyBgAAAAAAAAAAwIeQvAEAAAAAAAAAAPAhJG8AAAAAAAAAAAB8CMkbAAAAAAAAAAAAH0LyBgAAAAAAAAAAwIeQvAEAoNqYFdfvdc1fMkfPJvp5OxiUwwjvqrGfLNTKcTd6OxQAAAAAAIBSJG+AcxKgq0Z8qg3rFuvvN0XK8HY4QI2rjjFwcY6rkEYt1bJRHQUaF8sdXVyMwFi1bN1EUUEk1wAAAAAAgO8geYMqV7vPO9qxY50WjkxUnXKfVVrU5dWV2r1tkUa3Mdd0eFXGMAwZhkkmnsfiQmW2aficTdqzeZaGN7f8zoG1de2YL2XfsUEf3hFS+tPqGAPeGVe11eONldq9Y52mDKp/5i9GS1s9u2Sz9myZomENL/yvz9p93tGOnZu04OF4+fZMbFarYe/pq+VT9Jfmvh0pAAAAAABAVbnwnz7BNxm11Or+tzR+cLz8vR1LtSjU+n/fqSvb3aKRXx+Rx9vhAOfCVFf1o0wyAi7THx/vregzfCOYE+7WqIGNZDbMCq8bcfxBf3WMAW+Nq1ytXrxCRz2BSuzdQ7Fn6IeAq3qpZ0OTCjd8qa/S3DUWHUyqE9dSCTEh8idZDgAAAAAALhEkb1BNPHJ5QtXp6bf1TMewi2b5I6AqWK1WvfDCWF1//fUKCAz0XiABdVU/zFBR5jGZOz+kB68qJxajjm768x/UujBTmW5DERHhlRrPpoAQRdWPUvgZlqKqqL2m5a1ZrCWH3PK/spd6Wcur7ghUh943KsZUoDULl+mAD+dufK1vAQAAAAAAcPZI3qCaOLVp6jv68kichrzxku6IrWipG4uue/E77d42V4+3KHusobC+72nnzp/08YCI4w+NDUVe95DemjhdXy3/Xps3bZR923ptWDZNb91/jZq366+n35qi5auTtPPn9dqwZLLGDW592hJuRm2bej/+luYu/0Hbt6zXhmUzNP4v16uhpcy12w7S8//6UAuWrNCWzZtk3/Kjflz4km6NMCvh4dlK3r5Kf+98ynJTtay68eHXNPPLFdq65Sf9nPSNlkx9UXfEsdwPSpjMJiUmdtCoUSP16Scz9PSoUUpMTJTF8ntLl1VDHBF1FWly69DiCZpqj9GdD/c5rerEL+EuPXJTgP733iT9WGhSeN06x784yh8Dpoir9ae352jd+v8p6fvvtH7DOm1a8g/1P37i328v75zljfeN2rp6vqa9dLeuDC8nlRTYUDc8+IqmLfxWmzdvVvLmJK1bPlezJryiBxPrlJ98yl+nL5bsl9vSUrf3KmcZsdrX6o7udWXk/KB5yw7LI8mI7KpnJ8/Vyh/Xate2zdq5/hstfv9p9Wteu4IE19nMd8dbKpyvKu77ip3v3Goo4toH9MaEKVq8bOXxOTNJa7+ernefvF2t65SN4+z7QOZmGvHFZv2y82f9svNn7V45Vh0tle+fkj66QoPHTNDiFT9qx9b12rB0ut55pJPq89cQAAAAAADwMbyWi2rjSv1SzzwZoiYfDdPL/7xfO4d9oG0FVXFmkyLa9FCf61uW+QW2KLzRler79Ifqe8rR/nFXa9CYCYrI7a8/zzsotyQFtdbwDz/QY1eGlGYwAxtdoT5/Ha+2sSN0+5gVyvCYVO/aARrSs+x1QhRVz6LCnDOEFtBCD77/oUZ3qPNbZtS/vmxtGyk434df1YfXWPz9dV2njurcpbMKCwv1v//9T99/v0rr16+Ty+Wq1msbdSIUbvIo89AaTfnoe93z2jDdf+V8vbq+8PgBoer+0D1qcXiB7pu7U7f+0aPAiEjVNqSi8tY0M0Vr4Lh3NOr6UBnOXB05mCMjOFJ16llUmOWuuL3cnVfKG+9S7brxuu6u59S2WS31G/KRdjmPNwS20IMTJ2l0YniZfXNqK7JhM0U2bCr/DR/po6RMnd6zRVo/b6F2D/6zmvXqqVbv79LmE+eUobAuvXVDuJSx8Astzzh+8846ir+qmRqeWBsyuL4u6zpU/7i8nopvf0oLDlfRwm+Vma+Mivq2Ms53bjUpsu0t6tut7Of9VLdxW/V66ArdeHN7PTbkeX11sIrnwkrN55IR2lFjpryj+xICS5NCAda26mkt+Xdh1UYFAAAAAABwXnjXFNXIo5z17+qxf62Xu+0j+tfj7RVSleuneY7py2du0hVt2sjWupN6PbdY+1weuTPX6t3hA3Rtu7ZqdlUP3fveeh1THV3fr5vqmSTJrGZDx2p42yAdWjFe9/e8Ti1atVOHAWP02R6XGt4xXINtZR4ie7K19IXeuvrKtrK1uVadB/5ba4rLC8isJoPH6onEMBXsmqcxQ27VVW3a6rL23XTL0L/r66p6kIuLjtnsJ8MwFBgYqM6dO+mFF8Zq6rSpeuhPD6llq5YyjOpZeNBUJ0J1DI+ys47p0Fcf67PUBhow7CbVPX45s7Wv/tgjWFumT9OPOceUle2RKTxCkWf45jBCOuimDiFyb31ffa+9Vld36aZ27RJ1Td83tSqv4vbfVWa8x7fqoE6Dx2npAbdqXzFYg686UV5hVvy9z+vJxDoq2rNQLwy5RW1bt5GtdQd1en6F8ioYgq4dC/T5liKZGt+q29uW2a3LCFf32zopzHNQi+esVvaJkLJX6x9D79C17dvJdlkbXXZNb90/abMKIm9Q/66VW16uYpWbr86rb091znPrb59fPKqbWrVqrfjLr1GnQU/rg3UZssTdrldHdVd5xVKV4tql8be3UZPmrdSkeSvFd35FPxRXdj730+UPjNZQm7+ObZyqxwZ0U6vLr9QV3QbrsUlrdZjcOgAAAAAA8DEkb1DNirRr6rN6+Zts2Ya8queq7IGmJI9L2emHdKzQJVdRhrbNfU/Tf3bJ8Dusbau360BOsYpz07R64odamuWRuVFjWU2SzM3Vp3cL+R1bpr+NnKhvd2eq0FmgQ1vm6qXx3yrHnKCOiXV/GxwepzJS9+lIXrFchceUlnJQueU9BDY3Ve8+lyugeLPGj3he05McyigsVsGxg9r10y6l83AQleDnV5KICAsNVe+evfSPN97Qx5M/1rBh91X5tQLCwhRkcisnO1fuwo2aMu0n+XcdqrsTzJIClTj0HrXNW65Jn/0ql/KUneORUSfitCUIS3k8JcuJRbVQYou6CjQkeQqV/ss+ZXoq0f57yox3tzNHqetm6LWpW+U0R6pFi6iS8Wpuqp69WsnftUP/eXyMpiTtVVaRS66iHKUfyVGF6VOXQ/PnrFO+KVa9+12joOM/NsXcrP4da8udslifrS1TPmgYqtP6Lr320edavWatNn87RS/eHCuz/FQ/pm7VfMFWdr46n7491bnOrWU+n3P0qPKcbrmLs5W6caFef+RFfZEuRXS/XV3DqjAZWdn+MTfXzTc2lqlwvd5+8g19seWg8oqLdCx1oxZMWyJ79Ra5AQAAAAAAnDWWTUP1c6VpzgsvqWPLf2nAS89oxZaxyq2W6+xXSppTuixK9euYpLzj2ZKiA9p3yCOjXpBqGZIsjdS0oUmmWjfrnaSb9c7pJ1JswxiZdPjsru8Xp4TGZrn3JukHR9U9CXxm9GhpdJWdDhcQs19JBVjdyEgNGDCg9OchISFVcv6QsBCZPEXKzS2S5NbeedP09Z/e0j1DrtV/x0fq/tuitXf201qW6ZGMXOXkumWKC1XoGZ69e7J/0Nzlh3VDr+v17JRlejIjRVs3rteK+dP00VfJyq2o/aySDC6l7f5VuZ5WCg4+vsdM6Rj8Qd/Zyy2Pq4BbB7/6TMsfv0a9evRV9zdWakGmSU379FX7AKd+njtXW08spWaEqsuYjzXp7jhZSvsjQNZGJbGZTFX09epfufnKqNK+PfUSlZxbf4cn6wct31CkO260Kr6BSco8j3jKqmT/mCxRatzAJPe+DVq3n0w6AAAAAADwfSRvUCM8h7/RK89/rnbv99eLz63SP/LLOUZuSQEKDDzXt7LdKi5ySoZFFkvZcxSr2OmRDKPkTfjjb6ifmaGAWgFnXyFkmEr22PBU7fJoc+fN044dO6r0nPCuqHpR+uP9D1TqWKfTKT8/P+1P26+Y2BhJUnZ2dgWfqpyQ0BAZngLlFZT8znqOrdB/56So9+D79JQnQtf7b9Trn2xWkSR58pVX4JERGKIQi6TyciOew1r07BDl/DRQt15zha66srWuuqGJ2nW9QS1M/TR8UUXtGWcVv6eoSEUeQ8aJzW1MFvmZJDmd5expU8lzZq3QjEX71XNwZ93VM0aLPqunO/u1kF/eD5ox79fS8xoRN+q+vlaZM9bo3bFvaPqPu5We76e63Z/TvLdvq/g6lZ3vKjtfVaLvz31mquTc+vs3Io/bU/Lf0p+c75yvyvePYS6J0TBVXfUnAAAAAABANSJ5gxriUeaqt/TcJ4n6+O6ReuxgkAwdK9PuVnZWjjymBmpuC5Ox8ch5PGisQHGqfk11yx32hf5401h9e8b9IMrbPL3i85qsibq2kVlbfq2a6psdO3Zo1cpVVXIu+IbGTRpL95+5/UTCJjMrSytWrNCqVau07edtWrRoYZXGERoaLMNTqLz8E6OtWFtnfqK1Q57VHwZ5lPHlSM3bd6JKoUj5BR55jNoKCTakM42bgr1aMfUtrZgqyRyiFv1f1kcv9lDXmxMVtGixcn+3/evzu6HiA0o77JbJerUSY03auvdcKiwKtPbTudpx11+UePcAXZPRQP2sho7Mn6XFh36blUx1oxXtL+Utnap3lu0oSXCpWEfSj51h43uzzKXfuGcx31V6vlLFfX+WPVGlarVW4uX+UlHJ/Ug6iznfI6fTKY+CFBR0Suqlsv1jbqnd+9wyNe6orvH/py27zqUyCwAAAAAAoOaw5w1qjidbP7z9smbsDVNsbOApbz+7tGfLdh3zBKjTn5/RPVfWV5DZJHNgiKLCa1Xtm9Kunfp66R656/bRi288oO4toxXqb5bJHKjwhq3UtX3cuWU1XTv11ZLdclmu0KPvvKx7OzRWeKBZZkuwopu3VfMz7fIOSHK5StbjysvP18rvV+q558bo3sH3auL7E7Xt523Vcs2g4CAZyldB4W+Pzd1pizT1m0y5XWn6Ysa3yihtcqsgr0AeU4hCg8/wu2xuom79u6lNg1D5mwyZLX5yZmerUJJhSEZF7ed7Q87tWvrNfrkDrtSj/xypPq3qK9g/QOFx7dWvx2Xyr+RpXPY5mv5jvsy2u/T2mJsU4XFo7sxVKlvv5D5ySIeKpVod+mlwu1gF+xmSyaLg4MDT5o+iYqc8Rh1d1fUaNQwy66zmu8rOV9Xdt2fDqK3EAffohoRI1fKzKLRRe/3htZd0d0OTctcs1cosz9n1gTw6dCBdHnOMegzsoSbBfjIHRij+6laKVSX7x7VTCxZuV7FfS/3l3b/rwc7xiggsOS6sbrhOzQkBAAAAAAB4G5U3qFGe7CS99be56vafAWp4Slvuyqmauv1G/bXVrXp15q169aTWoiqMwqmtH/5NH90wQQ/2eEKTejxxUmvxT2+oxz2TlXLWL+079fOHr+r9Lu/pkcvv0CtT7tArJ5o8uVowootGLCn4vRPgEuN2u2UymVRUVKTVq1br2+++08aNG+Vy1czu6UG1ax2vvCnzQ0+Wvnyik+KfOPVoj/ILCuUxais0uPzzGZEd9MBLY9XRckqD+6i+XLJWeZHdf7f9/CtDCrR24pua3/1N3XHFUI2fM/SUdme5nzqN+6AWTP1aj3bsq/p1PSpYN1PTNp08B3mOfKNZy4erc69uen5GNz1/UqtLu8r8e9/2Hcr0tFCLoRP0db2n1P7Rr85ivqvcfOWooO9rtOrG8FfjW0bpo1tGnRxK5o8a9+ZCnShgqnwfuORY8Y22jWitNv3e1Df9jv+4eKNe6zlEkyo1n7u0a/KLerPjBxqdeLOenXSznj0l7PIrpgAAAAAAALyDUgDUMI+yVv5bry0+pNNyI4VbNf6hP+lvn6/VL0cL5HK75CzI1uHUZG34/kutsOdX2VJqnuy1GnfvPXrsvYX6MfmQjhW45CrO1eGUzVqxbu85p4o8Oev1zz/cqxHvLda6X48ot8il4ryj2rttvXYfs7DXAko5ncVam5Sk18eN06BBd+nNf/5T69evr7HEjSTVDjLL8OQrr7AyI8uj/Px8yQhWaEj5Xx2GsV8bvtuslKP5crrdcuVnyLF5mSY+/YBGLkyXKmivivHtTl+qUYMf1htz1mrPkQI5nQU6sidJXyzbpjyP5PZULiubs2qGZu92yuPO1LKp83XaCmyeo/pyzIN68sNvtTXtmApdLjkLc5VxaJ92bVqjH+1ZpfeTt+JtPfF/y7T1QLZS9+0vmV/OYr6rzHxVUd9X2zKU5fHkatPiOVqZnK48p1MFWfu06euJ+uvdw/Xf5DLLlZ1FH7iSJ2vEyP/q2+TDynO55Mw7oj0/2ZVuGJWfz/O364MHB2nYm59p1c6DOlbokstZoOzDe7UtaZk+X7GnJnsJAAAAAADgdxlhkVE1+kwHF5YTe2wkJa3V+Hff83I0l47ExPYaMfwRSdLr48ax581FJiAwUH5+fsrNyTmrzzEez5WhqDsnauXLV2vN2O66b/bRmk1mXDLMSnh4phaPiNGch27Q0ysvzX1lmL8BAAAAAMB5MzSbZdMAoIYVFhSwRFM1MdVrp1uvkJJ//kVph7NU4Beupu1u01MPd5C/e7d+2pJF4gYAAAAAAAA+j+QNAOCiEdD2Lo0b31PBp65R6HEpbeEEzdhVc8vSAQAAAAAAAOeK5A0A4CJhyD9zp75Naqq2CVZFhwVIhVnav2eLVs6frHenr9Ghym15AwAAAAAAAHgVyRsAwEXCo6ykSRoxdJK3A7lEuZQ8YaASJng7DgAAAAAAgAufydsBAAAAAAAAAAAA4DckbwAAAAAAAAAAAHwIyRsAAAAAAAAAAAAfQvIGAAAAAAAAAADAh5C8AQAAAAAAAAAA8CEkbwAAAAAAAAAAAHwIyRsAAAAAAAAAAAAfQvIGAAAAAAAAAADAh5C8AQAAAAAAAAAA8CEkbwAAAAAAAAAAAHwIyRsAAAAAAAAAAAAfQvIGAAAAAAAAAADAh/h5OwBcGGw2m0YMf8TbYVwywsPDvR0CfBjjEfBdzN8AAAAAAKAqkLxBpUREhCsxsb23wwAgxiMAAAAAAABwsWPZNAAAAAAAAAAAAB9ihEVGebwdBAAAAAAAAAAAACQZmk3lDQAAAAAAAAAAgA8heQMAAAAAAAAAAOBDSN4AAAAAAAAAAAD4kP8HLYo4O86EahAAAAAASUVORK5CYII= ) ``` bp.delete() ``` ``` Blueprint deleted. ``` --- # Pass features into a task URL: https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-adv.html > Learn about how features of the same data type may need to be processed differently than others in the Blueprint Workshop. Certain features of the same data type may need to be processed differently than others. For example, suppose you are working on solving a problem with a dataset containing text features. One of which lends itself well to using word-grams for preprocessing, while the other uses char-grams. When using Composable ML in DataRobot, you can pass one or more specific features to another task. **Blueprint Workshop:** When using project-specific functionality, DataRobot recommends running the following code. ``` w.set_project(project_id="") # or # w = Workshop(project_id="") ``` In this example, select the Age feature, perform missing value imputation, and pass it to the Keras neural network classifier. Note that similar to other pieces of functionality, you may auto-complete feature names with `w.Features.` to complete available features. ``` features = w.FeatureSelection(w.Features.Age) pni = w.Tasks.PNI2(features) keras = w.Tasks.KERASC(pni) keras_blueprint = w.BlueprintGraph(keras) ``` You may link a blueprint to a specific project if desired, ensuring the blueprint is validated based on the linked project, for example, to confirm that the selected features exist in the dataset associated with the project. ``` # Make sure it is saved at least once, or pass `user_blueprint_id` to `link_to_project` keras_blueprint.save() keras_blueprint.link_to_project(project_id="") ``` **DataRobot UI:** To only pass a desired column into a task, add the Task Single Column Converter or Multiple Column Converter. Then, pick the column name from the original dataset as the parameter column_name or column_names. The following task(s) will only receive the selected column(s). [https://docs.datarobot.com/en/docs/images/bpw-20.png](https://docs.datarobot.com/en/docs/images/bpw-20.png) Click Update and then Save Blueprint to see the new task referencing the chosen column. Here's an example of a blueprint performing specific preprocessing on certain columns. Notice how each column name is observable. [https://docs.datarobot.com/en/docs/images/bpw-21.png](https://docs.datarobot.com/en/docs/images/bpw-21.png) Continuing with this example, you can also pass all columns to another task. To do so, add a new connection from Numeric Variables to the desired task. You may link a blueprint to a specific project if desired, ensuring the blueprint is validated based on the linked project; for example, to confirm that the selected features exist in the dataset associated with the project [https://docs.datarobot.com/en/docs/images/bpw-22.png](https://docs.datarobot.com/en/docs/images/bpw-22.png) Features may also be excluded instead, which is particularly useful when a particular feature should be processed one way, and everything else, processed another way. **Blueprint Workshop:** ``` without_insurance_type = w.FeatureSelection(w.Features.Insurance_Type, exclude=True) only_insurance_type = w.FeatureSelection(w.Features.Insurance_Type) one_hot = w.Tasks.PDM3(without_insurance_type) ordinal = w.Tasks.ORDCAT2(only_insurance_type) keras = w.Tasks.KERASC(one_hot, ordinal) keras_blueprint = w.BlueprintGraph(keras) ``` **DataRobot UI:** To process certain features in different ways, add the Task Multiple Column Converter. This task lets you select columns. You can give it a list with several columns that you want to include and the rest will be dropped (using the parameter column_names). Alternatively, you can instead provide a list of several columns that you would like to use. Next, create an edge from the categorical data to the modeler, insert the alternative processing task, then add a second Multiple Column Converter and pick the same column name and change method to be exclude. Now, one column is processed using one task, and all others are processed with a different task. [https://docs.datarobot.com/en/docs/images/bpw-23.png](https://docs.datarobot.com/en/docs/images/bpw-23.png) [https://docs.datarobot.com/en/docs/images/bpw-24.png](https://docs.datarobot.com/en/docs/images/bpw-24.png) --- # Custom task creation notebook URL: https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-custom-task.html ## Setup ### Import libraries ``` import datarobot as dr from datarobot.models.execution_environment import ExecutionEnvironment from datarobot_bp_workshop import Workshop, Visualize from datarobot_bp_workshop.magic import * ``` ### Connect to DataRobot Read more about different options for [connecting to DataRobot from the Python client](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html). ``` with open('../api.token', 'r') as f: token = f.read() dr.Client(token=token, endpoint='https://app.datarobot.com/api/v2') ``` ## Initialize the Blueprint Workshop ``` w = Workshop() ``` Get the Sklearn Drop-In Environment ``` environments = ExecutionEnvironment.list() ``` ``` scikit_env = environments[7] ``` ``` customr = w.create_custom_task( environment_id=scikit_env.id, name="My Custom Ridge Regressor w/ Imputation", target_type="Regression", description="Impute values and perform ridge regression." ) # customr = w.get_custom_task('608e5bc8b66a4934d58d0d4e') ``` ## Iterate on a custom model ``` %%update_custom {customr.id} import pickle import numpy as np import pandas as pd from typing import List, Optional from sklearn.compose import make_column_selector, ColumnTransformer from sklearn.impute import SimpleImputer from sklearn.linear_model import Ridge from sklearn.pipeline import Pipeline def fit( X: pd.DataFrame, y: pd.Series, output_dir: str, class_order: Optional[List[str]] = None, row_weights: Optional[np.ndarray] = None, **kwargs, ): numeric_transformer = ColumnTransformer( transformers=[ ( "imputer", SimpleImputer(strategy="median", add_indicator=True), make_column_selector(dtype_include=np.number), ) ] ) pipeline = Pipeline(steps=[("numeric", numeric_transformer), ("model", Ridge())]) pipeline.fit(X, y) with open("{}/artifact.pkl".format(output_dir), "wb") as fp: pickle.dump(pipeline, fp) ``` ``` '608ef74c5dda651931052422' ``` ``` custom_ridge = w.CustomTask(customr.id)(w.TaskInputs.NUM) custom_ridge_bp = w.BlueprintGraph(custom_ridge, name="My Inline Custom Blueprint") ``` ``` custom_ridge_bp.show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAxEAAAB8CAIAAACZh6ooAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3dZ1wURxsA8Nndu+OoB0cVQUBRQERABFFBBUXsvYMaY4s19q5RYxJbNFFj7/21FxQVBGlKEVCQJooFkN7huLa77weKgHccVQh5/j8/JMvu7HMzszvP7s7tYRx1TQQAAAAAAGqFt3QAAAAAAAD/ApAzAQAAAADIBjkTAAAAAIBskDMBAAAAAMgGORMAAAAAgGyQMwEAAAAAyAY5EwAAAACAbJAzAQAAAADIBjkTAAAAAIBskDMBAAAAAMgGORMAAAAAgGyQMwHQimBaQ3ZcfxwWeWVeh6/HJttmmUfUhy+xDzb2UWrB2EBdSGzB1gn6FQD11dqPagAaidlr++uMzLxUz5+7MGpZTc56Y3BaRn7a4+XGRGN3yRrwd1x6fnbKDXd1rJ6b4oodezlYd9ZWIqpuieE4jmEYQWD1La9pyOn2mrp6z+VHz+PefshMT0l7F/XS63/Hts0bbq7W6MqqhjCasO/645eR/4xRaNJyG4lhMuPA/574BcfEv01NSc3J/JKRnPg23N/rf4d/X1izDiS3YOvUiH7VSlsKgGZW2ygCQBtAaOlo4AiTs17w86Czix8V0JJWwrTGrpjZhYkhUkNbA0fvyO8dZa34L/cN67avZfaNqdrM3Xt008jOClWGVVUdY2sdY2unSdN7zbaefaugyfaGa1o49bM2En9p2lSssfB2Ni797TS/XmES8hwtA46WgantwPHzlrzYO3vWrud5EntWa9aYftVKWwqAZgb3mUDbhqvraDExMv3Fi/zhi92NJJ/hmd1mLen7LvAln8a5mhowClTAVOzXXb3z26jO8lR+zO2di8b3tujSTldPz6Sn44RF2848jbh/z7ewpYP8foQ+q3u0a6ejpqmj2cHEov/4+TtvxRTSDM0+a45vH6La+m8rAQAaC3Im0Lbh2jpaOKJzgo6eCe86Z7YtW8I6ygPnuWk/PXUyOo/GmBqaMPqVwTjOm4/9bK2MhEk3Fju7zN/5v4C4tPxSobA453P0s+v7V08d+PPdf9/dlUYQlhbzRRRNUyJeXnJMwP/2Lhg+73IyiXDtIWPs5Vo6OgBAs4OcCbRtuIamOo6ogtzYW5eeqU6YPYxbMyPC24+bN4K8f+nJx9wCCuEamlwCIYQI0+VeWdmZORE7ejOrb8Dqvz8mPT/r0/+mflOWNPKmY1f9dvTiHf/Q10kfk7My07NTE2P9b/yzzLWjvOytMZ2Zd79k5qc9XW1WcQ+svgXK6fWf98dVr5dJn1MyPsZFPjm/c1Yv7dqfzBNdZq+b2oGBhLGH5y6/mSSUuiLTbltkRmZ+6pnx1aa2YKpTLmZkZ2Y/32hdZUdsQ5dlf18PepWQlpaa/jH2td/dy/tmWtbIZOWGn/qcmZ9d/i/j+gytyopmatpN/+WMx/OEpOSMzwnRvlf+WuLaqcaUmrLKuXQ3IPR10qfk7Mz07OT4SM/j64d3kmfrOc7YfOKOf1xSSnb654+vnv5v50w79QafBun8lyExJEIYm8OpLWeqVxUhVIf2YncZvuSXg2dv+YVGffickpORkvo28sXdo7/OcmjPqrHv9n3cVuw6ce3p88ikT8lZ6SnJcWEB/1tiw2ySfiW9pQBoi2A+E2jb2GpcBRzRhfkF2Y8vP9h+cu4kgztHP1JfV2Cau8/qk3H796CSArNCCmFq6moYQgiR75+/SCUtDdvZ9TYkXiR+neHE6Gxnp44jcVxQiOTJUd/COL1+WDG3f9XBTI6j27WfW1eHYU7bR046/EZQv09VrwIxbt8NF0+vtFOrSA3UjXoM+cl64HCH5SPnX/solrwLhvm4SRYsjC72OXI8gl+/8KRhmsy5cm+HU2WOwtQwMNfQ48T8U7f5Y5iq7cqz59b31aiYXi2nbzHwBwvnyVOuLJy2+vYnUflq31aOPNfIdsyaM04zs5ja2l9nZqnqWbjO2d2/r9H4YduCihpyx0y5h605gRD5JfE9rwGbS1SX9sJUe89bv6jqZ1TktjfrO86s75iZ7ifnuf3yKL28TjH1QWv3rKu6JlPTwJSLFVU5Br7uuqk7KgBtDNxnAm0arqquiiFaVFTMp0v8L9/Ntpnpbl31vpFC39luxok3rkWK6JKiYhrhHK5q2UW36LWvXzaFGGb9+mhWuXjGNHr26kQgMjU4OLmeU8XF70/N6G1qbKip3V7XzHHyDq9UMabWZ83v7voNPA7rUiDebuKfx1faqYo+PdoxfYBpBz2dLr3HbvH4KGLoj9yxc1I7KbvGte3sOjIQLYp47JvTRM/flAYvX91fHSuJOvvTEBsDXV0tA3Mb1+nLt9+IrpG3CR7M7qClqlH+T3vi+UwaIVx7/J9nNjhoYMVvzq8c06NzB22D7v1m7fVJI9kmUw+f/tmyxv2VssrpZKCpo2doO/FXn0wa5+hoUXE3trkN6mGg207LqMfQ9R6fxYht+uOGqfVrApylqGFgNWTOnltH3PUIKuvpgVORosZWT3nR9Wkv8bvjU3sY6bfnaunpdRswZcuNuGJMxXLuyZPzTWvcHCU/Xpw3wMLEUFNbT9/cYdiya+9r6bx176gSWwqAtgtyJtCm4WrqXBzRvOIShJAw7NrNJIOJMxwrH5BgXNfpo9VfXb3xlkR0SXEJTWMMNXVOWYrED33sl09hLNvB/aq8M0C+h72FHEYVhLyIru8oSfPSP3xMz+eJSBEvK+Hx34s33M+jMPlew50b+ESjDgWybOavGaqJlb78w23OXs/YdJ6Qn/ve9/DCeccTxTjHaepIPcnnAELfQJ9AiMp+9y63iYZBor1ZFyUciSIv/XXtZXKBUCwsyXof/vj8rYjCOuyBZT1/7XAtnEy7+fPkn889T8rjC0rSo+7vnua+/7UAyXefv2a0RrVaLKucglKRWJj/wW//8j8D+TSiha9unHzwKqVASAqLUl6cXLXjcRGNyfXoZ6dchxCGHHqbl52Zn52Z++XDu/AnV3fOtOXyYi4sHjnn0scm+qpl/dqLLs1KScsrFVGUsDg99tHhRcNnn08SIyW7pauHqlWrDaroU1xCcg5PRAqLMt6GxaTXFm+Td1QA2grImUCbhnO4HLwsHUIIid7cvBarPtrdtXwiEt5+3PRB7BfXb3+mEEJkcTGPRriqmmr5YVHif98nj8IUerv251SMFcyufXqqYHTJc5/Qxj6wovP9n4YLaYzRydS4SR6SSyiQaT1ymBGD5gdeuJBQdUYSP9I7MJPCWOY9LCXPw8HkFdkYQjSvhNdUdw6ovKxskkZM60k/9Fav75cTGd1HDDViIPHbKwcfZlWNiB916sjTYhpTGTByAEf6iE5lhL5IIhGu1NFY6+tZj84PC3krQhijfQfdhn1dElPsOm711sVO7ZpmmkOD26sMneu7/5/nAhpXcxnpoNgkEaFm6KgA/GtBzgTaNEyZo4whmldSSiGEEPnu1o0Itov7uPY4QojoMnG6Pel340EahVBFfoBz1FQqDoti/9uPsilMyWGkU/mX6YgO9vZ6BM0Pfeyf3+hcgi5JTyugEa6qWstg36gCMeXOXXQJhMm7HHif9XWubn52ZtadH3VxhLE1tVUlngRofqmARgiTV5BvqlsLdNb9kzc/i5Cizc/3wgJv7Vs22V5PoY6FYyqmZvoMRBW+Ck+o8SCPLoh4mShGmJxJ19peR0rmZGRTCOEqHJUqH5jKzc6hEMIVlZVknwuFjxZ3USt7DqWlq2NsZT9izpYrUQXyhkNWn7+92b4JcpSGt1flB8oMCflAIkzBxMyoyfKbJu+oAPxbQc4E2jJMiaPMwBAt4JWWLaA+3735AvVyn9SZQCybqZO78nyuPS6fr0PzS0tphKmoqFQODCX+V+6lkriqy9QR2jhCCNcZ4GzBoIUR3k0yyYcWCIQ0whjMphrdahaIKSrJ+FUMtpzk+xZkWkoaiRCuYWSkUte8RtZ6dO6TtSNn7L0fm08pd3KeseHY/dBY/9Or6nKPpvyD0EUFhd/MXaYKC4ophHCl2vMeoVCIEEI4Xi2xEorECCFEEPW7zUSJ+flf4oPvHVgydvrJJDEmZzJj7hCO7I8ho4oa3l5fIyvIL6AQwpSUFZsuv2nyjgrAvxTkTKAtw5Q5KgghupQvKE9xqPQH1wOE3SZPsVHt6zauQ4HXTa+KVwzR/FI+jTBFVZWv46cg+PzlGCFScJg2yYhAmLqTiw0LiSIfeX+R9LWjVofmlZQghKjsi1O0NL7O1f36T3f4Px8lfhIqNfJVOokwprWTg4ykiRaJxTRCmBxb9i0p4WevPdP7dbdwnbvtQlCygFA1G7Hx4rUt9pUzzKRkonRJcTEqa89vzlm4CkcJL1ulIY3SyNy3OOS+TxqJMHljE32pGUVdq6jh7VUJU1ZRKnuoWtq807Fhsjf4L4KcCbRluJKKCo4Q4vP4Fad4OufRTb8Sw/GLts0ZpZXjedOvqHLtUh4fIQxXVlX+OrCJ4y4d9ytGLKvZc3sr6Qwd14eNhC9veyT/K1ImRBe+e5dBIly1R89af21PAlGEx4MUEuGqQ+e7d2HWtiaVn5NHI4S3N6jrnCBBesTd/cvH2jrMu/RehOS6TJ/Zr+ztP7RAKKARwuTk5KrnFnRRQnwKiXAVqx4mNT4IpmLdszMD0fyEmNq+C9ZsMAaTgSFEk2JSahpR1ypqRHtVRMPpbmnEQLTgfaK0t0g0CaktBUCbBjkTaMswJSUFDCFawK/MmRCd633Tu0BrtJuLSrrnjcCvr9Upu8+EsOrPeOj02ydufyGJDlOWrVowuY88EoTe9UhtrpSJphFNI4Sx2MwmGYpEkU980knEMJm2dKhm/UoUhh49GFBIY/K2q45udKzlvY9UamxsHoWITi6DO9VroOd/fHD8bhKJMAUt7bKJMlR2Zg6FEG5kUvNXbkSvHzz6IEaMLlMXuVb7fhy72+yfnJUwutDvvl/jp5jVG6E3dtZQbRxRhbFRn0gkpQXrXEWNaC+EEEJyZjNmD1DAaN5zr8Bm/VUb6S0FQFsGORNoyzAFRQUM0eLS0qrvBSh4dtM7j0Lkl4d3g6t8+Y0W8EsRQphijZkgJX5H/gkrRUqOPy/oKYd4gbc8m+/BHF2Yk0fRiGE29kcXEy6r8XkTP/Dw/sACitCdeOjWicXDenRQYzMwnKWs06XXqOlDTGsbwKlPF1evfZBGIkXrRVf8buyYPchCX1WOwHCmAreDuePExX9cOjbHlEBIGHzjbgqJMS2XHvnT3a4Dh4VjDLaqjmG7Gg/1lPrO2zB/uE1HDUUmhhFyqgZ2k34aaUQgKu/Tp7J0h8qMCE8WI4aR2/qFfdsrMnCWcvvuQ0ba6eBIGH5018NMitCdePDKvun2hqosloJ2t+GrLlxcbs1G/Kjju+9mNXfKxJJXkmcRGMIIliK3vYn9yLm7b3ocHKmN04KYs6ee8RCS1oJ1raJ6thfDZMavm2f0N9FWZLGUdLoNW3Hu4qqebCRKuvT37YxmrY1aWgqANgym9IG2DJNXVMQQzedXf31x8cOFZtyFNVemSyvuM9WYPUu+P/vHpbm35hgRiMrzvuTRjKMRnR9wP6DAxUW1+7zzz12PDrffEtzIdyWSH04v+Mno8tEFViYTtp6dsLXKn4RhG/2fxH+SngCKP12ZP178x7E97t30+s37s9+8miuIIsL2nIl/xw/6c8MFp1MzjbvN/Mtj5l/Vy6j8L6bZsHlLFhku21Xt7zSV4/fnkaDy3FX06tQ/fu57B2o4bXrwelP5GiWe8wPCruWn31jxo6H6ufV9LGftvzdr/9cSSt9eXfTjX6+a/RXVLOc9EV/21FxKUwWvz62cuTey7CNIacG6VRGqZ3thLP0BCw8MqNqVaTLLb+vcPwJLGvtpZaitpWCqE2izIGcCbZmcogKOIZrPr9N8WJJfKqQRU1FZqeb9Hd6Lo8fDZvxuz8jwuPK4WX+WlvpyebGbwoZVMwZa6Se+TWqKKSlU5tONwwY8cp8/Z5xzr66Gmkq4iFeQ8SnhdfDDVzJ/8EPw7voK12fnR7pNGubiaNOlvaaaEiEuyc/4nBgTGeLreevGBxIhRGc/WTF8dMSyn2cO7WWmpyqPk4LigswvnxLj3rwKeFL+vnQ6K/B/VzsP7mNprKuuyESi4uzkxAi/u6cOn3n6qTIxpD5dmDeSv3LTT6P6mLRTwvi5X95G+Ad9JhBCiM4P3TNhQKDbooWTB9ub6XGIkqyk1753zh444fm2uFkbJT3iaZCFdYd2Glw1FUU5BkaJ+MX5WakfEqLC/B/cvOEZnfO1oaS0YJ2qqKyAureXOOnGvtuiXkP7W3XSlBcXpCWEPv7fob8vPs9oopeS11optbQUAG0VxlHXbOkYAGjtmJ0X3PXe2huL/HXoyH0x32FAAqA2mM7MOxF7+mPRvzkP3hPXElPfAfhPgofPAEiEqRmY6HFYDDa3c795xy5t7K1Q/GLnz4cgYQIAgP8qeDYHgCQYd/gu74ODKr5ITQs/3Vo1/1i137MAAADwnwI5EwCS4Opy/I9ZpZ24eHFG4kvPC3/vPheSCc9AAADgPwzmMwEAAAAAyAbzmQAAAAAAZIOcCQAAAABANsiZAAAAAABkg5wJAAAAAEA2yJkAAAAAAGSDnAkAAAAAQDbImQAAAAAAZIOcCQAAAABANsiZAAAAAABkg5wJAAAAAEA2yJkAAAAAAGSDnAkAAAAAQDbImQAAAAAAZIOcCQAAAABANsiZAAAAAABkg5wJAAAAAEA2yJkAAAAAAGSDnAkAAAAAQDbImQAAAAAAZIOcCQAAAABANsiZAAAAAABkg5wJAAAAAEA2yJkAAAAAAGSDnAkAAAAAQDbImQAAAAAAZIOcCQAAAABANsiZAAAAAABkg5wJAAAAAEA2yJkAAAAAAGSDnAkAAAAAQDbImQAAAAAAZIOcCQAAAABANsiZAAAAAABkg5wJAAAAAEA2RtX/Wb9uXUvF8R3ExsfdvXO3paNAqK3Xc5O4fed2fHxCS0eBTE1Nxo4Z29JRAAD+i2DMaiWqjkfVciYHR4eWiOf7uYtaRf9r8/XceAFBgagV5EwamprQWACAlgJjVmtQdTyCZ3MAAAAAALIxvl0UGhp24NDh7x9K87l4/kxLhyBB26vnxrOzs126eGFLRyHBgUOHQ0PDWjoKAMB/Quscs6KLFC6la7Z0FN+PhTLPTSerxkK4zwQAAAAAIBvkTAAAAAAAskHOBAAAAAAgG+RMAAAAAACyQc4EAAAAACAb5EwAAAAAALJBzgQAAAAAIBvkTAAAAAAAskHOBAAAAAAgG+RMAAAAAACyQc4EAAAAACAb5EwAAAAAALJBzgQA+NfA1AZsvuIRsHOQXEtHAqoizJc+jo0K2mzHaulIQJuHycktcVW/3keuRTob5EygScj1WPq/iJcPdw1Wx1o6FPCvJbsXYWzdrhZGmgoM6GatCdF18KBOKMP78SthS4fSbOAU11pgBNFZncFlYBhCCGHdLLkekzXWdcC/T7s0Qc7Etv/5+gOvl2HhCbHR76LDIgMe3D25a+MPg0w5RJ3LIMxnHX709Pwik7pv0vYpjjwYH//SY7Wdas2+wOy3I+B97IN13VtRdWEYhmH4d+q2rRBhvPjW66Soa4tNmJL+rNh7k+e7+IhTY5QbUjiuYjpk/s7j1/xfhCbEvop58djj7O4Nbr31Gn6zpQWOuGonipjwqKBHHmd3r5tq167K1WIL9qK6hAckI8yGDjZEGb4Pq6dMiiMPxie8vr+gUwudpxrWyaVu9V8/xdWBnI7SPyM17k3S8nHT9pum9WiCxhkXzhJTOT1G8+4Xq1sq09lM9dwYtelqjdpXE3wUQtPYwli3/OxNKKhqGapqGXZ3HD7rp6jzm1b/7p0qll0GrmrQtXO7PBZ0xxowefMf9x1Imznn4vvWfQEnCP97kvXfLR1FC8I1tDVxTM5szvIRNxbeTqeq/ZHoPHXNRH0CI9U0uAQqIutTMKbSfc7e/Wv66VTeWmFx9cx763W1Ukx4GJwioBsW7vc/4qqdKBBbWUPfXEPfvPfQaeNOzp19IKSQbtleVIfwgGQM88GuBijt0pOI1nWSalgnl7bVf/4UVweEPMOUQ5RfZWCYIpswZhPG2uxRXUp/9S705zXHPuk3r3OHv67LmhhHmWmoSDXyIqipns2JY/+Z2LVrt45de1j0HTZ2wY6TgaliVcsf9h/b2Fv5P5IIaWhobN++feCggQoKCk1XKk3SKg5r/1rfh/Mfqcbvo6t51/Xr1/fp04fFlHhbqP7kNLQ5mDC/kHCcN7cHu9qfMNXBP820EOTnUxiXq1bPE3i7cTv/WddfG8uJvPDrguFO9qbdenR3HDV52Z6z5+4HFdR1IMfllDW1NdUUmvlyTzZx7KEJZl27dTSzNLd3Hb3wT48koZLFj9tnmbWOW6atNDxpzdc6mpXRzXWgAUrzevxa1KJxgAbYvXvX8OHDOSqcJiwzMSpn0KWMfpcyXK5n/+hb5JFLs1Tkl3VnsWVv+i/QZPOZKJFASNI0KSjO/vTK58pvcyb9cDqezzR0XzfdlEAIIUx9wIZztwOCw97GRiWE+zw8tnaciWK18YPosvRu1IeEmA8JMe8DNvdh1m2rVgPDcRubHiuWL7969crGTRubaDwWv75w0DPHYPrubWN0pZ23mX23Pnsfe3u5aeUKGGfs4YSEyLMTuGVPfNX7ztt3/NKjp/5Rr1+9iw2P8L6470d7E5vxa/edfxoUmhATHvHk3E43i6oPATFF4xHL991++jwuOjzC+/KBRf31mBWFW03esv/U/Sd+0VGv30UHB3tsG8olOi+4nhgXuMuxykeW7zBowe9XPf3eREfGhPo8ubB1jEHrGBkRYjKZDg59N27ccOXqleXLl1tbW+F4o44FnKuhjlOZD49ceNdu0oKRulUKY3SesnCw3IvDJ4MFuJqGKo4YPTc8eRcffMBV8etKhNkKj6j3wb8PlK9WrEKfn1YN4KJs3w1Tf9hy0T/2S5FAJCjKfB/qeXb7/kfplMymRzi35/y/br0MfxHq/yw84uXrJ3vGVwYn6YhD8oaDF+26/jggJjoi2v/22a1udppfC29AR6qBEgtFJE1TYl5eStTT0ys3X0umGEa2PbRxhJCEXoRzLd02HXnoFxz/JjzC69LBhQ7aNRqKrec099eLHr5RUVGJUaEvn96+duTXuVUeaEvvyfUNrw6l1RaMxAMHq71Mac1XW7PW2oLSYijrEBbL77+LDzsyovIJMq4z7XRCXMCufnJV1rn3Li5gd/+KEZBhPsxFH33xeSgjZapv58G4vWfvPnL+oXdAdNTrd9GhYY8vHVo52kK1svlld36EGjqsSNhK4imu7gfLqzdB9y5um2pdz4um5ta1a9eFCxdcvHRhx45fnZ2d5OXlZW8jC00hEY1oGvEFZGIqb09QSSKFuJqsDhhS1pBf6qh2arTmo6naz6Zq3R6hUtaPMCZjoBXn2BhN76laHmO4Wy3kdKoc5jibOcZW9cw4rafTtDxGc7d2Z2lUqUXDblxfN811ulUWMQiHbpyDozQfT9XymqR5wUVlcGWPxhg/DNcOcNcOcNf2G6/So/5n/Wa7QKELgg/uuu56ckbnocNNj8XFkEis2qlHF72y+2JK2mYDZuzppiUavep+dq3Xyg3bqkURBNHL1q63vb1AIAh+EeznHxARES4W1+ERpSRkquf6lcpGp2dt//PHhFknYvkNKAPndncZ2b9rRWMz1fStx649NbbKGiyDnpM3HeGWjP/pTgaFEFKwWHzqxDJr5bIexda3HLnkgJXu0tGb/PJoXKv3hOnDKktT1tRiCoq/2aec6dxjp9b1qjjDsbSNrfSVSqlv1mthbDbbyan/wIEDeaW8AP+Apz5P42LjaLrevQtT5arhdH5myPnT/tN+n/Wj9b0d4QKEEMJUBs6bZpp9/4fbCUPn0GyuuiImfhPwInf6ONve3VmPX5Q9zcC1e9gZ4oLAkIhq7cvuPXKQFi6MPLXn5ucG9R9cZ+LOg2v6q2DikpyMYkxJXVWLKSigEJKSvLK7zj9+co0dp7zVtLv0n7q+Tz/Lle7r738hG9KRZKHEJIUQkpKxYip9Np0/+ENndtnpUK6D1bAOCCEk+Bqw6dzjJ9fZqVXMMlFU1+uirteRFXH6dGg+iWrvyfUPr/bSZAQj8cChaysTk9J8UptVdgtKjqEcmRAcmjVvoqW1KdMjTIQQQvLWPbsycQUr646EfxyJEMI1rKz08VL/oMjyRmBaDB6sh1IuPJF1l6m+nQdXtxoy1rlyfYaGodXweZaDXG2XTd/yqE6dS7omGVbqd7AgRY1OfadstOoiP2766bcNHA2aC47jlpaWlpaWy5bRryIjn/n7BwUGCQQC2VvWBYYq0xl1HfmxBsyKOsG4CkgoQojBnOmsNksTK6tJOSXmQEvVror5c4MFBQhhLNbiQaoTVMtmfCOWMtNJGSGEpD4HJhhTnNQWaFccswRmoEEoNF2FN+f35kpfPwspoPD2Zp0VEUJ0UdCeGWN629oYm3U3sx/x48kovrrT+AFVsm7y7YHR3Y1MzI1MzDs5/vpchOq0VatEMAgMw9hstmM/hy1bNl++cnnJkiVdzbtiWAMCp4vDDy3bH05ZLdy/3LbhTzrpQs/1gy27dze2cBi+8WEKSVP5YYcWT+htY9Wlh4v74fBCpNp/nLMWjhAiuszYvNhKIdPvwI/D+pqa2/SasOlGEqk3ZrGbccVYSxd5/TKip7WVcffejhP/Dql5uiSM3DavsOPw397ZNH1oj+5WZrbOQ2bsetwqM12CYGAYUlRQGDTIec/u3RfOn583f16nTp3qVQiuylXF6KKCwsxHZ2+ktp8wa3DZlRDRYewcF6XoSxeDiwsLimhcjauOI/5LH788pOnYz7LiklXJ2rYbQxwTEl7taRuh37WLEk5+8A9MrdcUqEqYcq/BvZSpN8uhNawAABVsSURBVMfG9u7ds5+zjY2d/di9gZWzCmoecYTx9M3LbVX4cTfWTHI272ZtPXjuHz5pmO7QrWtcvh5y9ehI0gMjWIpcPXPHqTu2TOxAkJ/DI9IlDIKMbrPXzTBmFb66sGyCs3k3a0tnt2Unw7K/rkl0ct+y0k5VmOTxy/QhVhbdjS16OWzx432twzr05HqEV3tpMoMpq70aB05tZUprPunNWscWlHrwCqNehBbjmjY9O5ZVD9PcvocChnCjnhV32hSt7c2ZojfBIeWZFtPSdaAe+uL16E2dHszVt/PQhQ/XOJubW3TqZu8wee2Jl3lMg9E71gysxwDQsGFF0lbV1e9g6WTey8Ftp1c6pWjp5tajieYDNCkcx3EcJwjCyspqxfLlly9fWr1qlZ1dL4Jo4MMBDMMU2YRZe4XVfRSNcZSfLfpcfizQgSE5o65mDriSNcmz5DWJOpoqz9DEclKL19zPGng5c4xnoWcBrdNRcbQqQgiZdFUep4oVZ/O2e2YNvpw59E7u9hhhrvSRRL+LyhxtXJBf+qdX9ogrmYOuZc30LvKvvBalxWcfZDhezHC8mNH/ZmFE/XPvZn3XgDg3t4DGcAUlBRwhhGGqFlN+P30zKCQsyvf8VlddAjG022nIiKBhW7UaNcbjixfOz5s/r/7FCN9e2LDdp8h4+o6NDc4XabIoK7NQQJLCvNjbhy/FkBgjOzYoLr1YJCr5EnT8lFcBTegbdsARIkxGjjBlFHr/tvq47/t8gZifGX172wHfYqJzH7uKmqfFeakpOTwRKSj88imjpEYPJjqOGNlNThR1YOmWS6Gf8wQifmHG28i3Wa3uNlM1DAYTIaTGVRs+bOiBA3+fPHHCwcGhjtvKcTgKOFVcVEIJXp2/GMkaMGNqZwIhtt2MaVa8pydvfCQRr6iYxlS5qhhCpaEPn+Vi7ZxcupVdcbEs7KzkyXf+gWnVaghTVFbEEJWXm9/QiqNpGiFM09TOVIONIUQLsj6k5Es73RCdR402Z4miD67cfv11Bk8kzP/0/PiqX66l0WoDRn0dpurekSRgdFt2711CTFJs5JsXjz1Obppsrlgad/GXUzESrgMJE9dBhrgg/K+Vu+9GZ/BEwsLUV/cvPnlXmT8SHYcNN2eR8UeXbzofmlwgJElhcVZOcZWUqQ49ue7h1V6azGDKa6/6gYPXWqa05pO2vK4tKP3g5b30DeMRnXr1Kn9W2steozAmJhXv1stOGUMIyVn2tlUkY/yflx/LTIuhg3RRytNH0XWby1TfzkOTxbm5PDFFiYpSX3n8sXDr3SzEHTh6QCOndzZ+WKnnwUKJi1NfXv79whsxoW5qqtmo4JsZwWBUveC/cvXKkiVL6lVCFyt1P3dtfzetRxM0jjspj+BiZBH/YJSgPG+h6YISMk9MkySVUUTyMOZAQyYh5P8TVPKigBJSdE5O6d9RAh7OsNEmcIzZT5+Bk8LTgUVeOVQpRRcXi54mCD5JO4lhjIFGTBYpOutfeCeDLCBpgZD6kCWuJceqr2adPMjgcjkYTfFKeDSm0m/T2ZNTDZjlfUmugz5CiMTxWgNo2FZSODg6PHD0aMCGTaJsPFZVUxs9alTZEg6nPtPuyC+3ftnWp+v+CdvW+0VvLmlkNGTapy9iZKaprYojHoUQQsL0lEwa01KQxxBi6nfUw3F514Ohrgerb6ar1w5H2bLLZxh0NiSo5NDnnxt2fwStX7cOrWvYpnVS+xFU1ljtdNu1021XtkRLS8ZpTpmjjNPCkhIhQlTynYuP5++bNr33mQPqP47SSb6+1jufRlhJcQmFG6ioYAih0hf3vNLGTHId0n1PZISI6NzHjkt/uv0sqXp10bySUoRwjioHR5kNqUm66Pntp9lOw/tvOO+9Mu/Tm1fhfvcunn6UWDPHLcMyMNbDqeSQoI9V9lUSERDJnzrEwFgfR7nfbFJ7R6otMppGCKOLXp7euOYf3w8178YghBBi6hq2x6mUiJdpUlLG8m72/Nk7KQM2q/aeLP0Bj8Twai+NoSEjmPpHiElrPmnLG9CCNT94QZBPJN/Zpr+96oVbBfp9+hiVhqw5nLfmwJB+NvJ3fIQWjvZc6u35ZyllO2BaDnbRRclnvV434NlH/TsPXfD8aYRwzKAOndrjKL/+eyzTJMNKQ6qa/PL+YwltrqSk+O3fJGrZMYsgGAghRQWFIUNcy5aoMuvRzBRFlwqptALR61T+nUTBR2mHBUHoKyGcwd46ib21+l+0lHAcx9srIqpYFFXHMQ8nDFUQVSyMKKp7pPXTnDmTvOWAXhyc+hSfWIK4o38Y24HICzm0efel4PdZpQyNgRvv/DWq9gIw7qAGbCVNfHz87Tt3GrZtXahwVBYtWFjLCiRJEgSRmZmppaWFECooKKhX+XS2z69bbtocG791Y+Ce0up/QhRCcmx23a+9KJFQjDAmk1m5iUgkphGG4ah8wJAEk5OXq9M+yt5iUv9ZQZVu37kTHx/f4M1lMjA0nDZlSi0rlDVWVna2poYGQigzM6v2ApVVlDGaz+PTCCG60O/MrU8j3H5YRXP7s179cSVKiBCiS3l8GmMrKzMREiH+y9t3Pkz6afCwnvsiQvX6OHZAKef84mrkRWRq4odS2sTI3lbzcKKkh1cym57OfrBhenHkxKH2lj2sLXo4GdkMcDLFxy1+ILHvNeDavdaOJIH4zV/jxhx5TyJW5xmHr63v1clUC0l7XQJG4AghTPoLcXAmA0dILJaaTta7J9caXu2lyQymARFKbT4py30aP2eBzgn0jRD0tXOy59wNd+xnInp51f95Xu+8Sf36Wcr5FTo5tkPv7z39UPYpWT1cnXVR8qknbxo0XaS+nQchRNMUXXlmqf95D6EmG1YaUtW0UCiksVp6dA3NPWYhhNatXVvLjBGxWMxgMAoKCzkqKgihfFGdEoa3r3LmvhHX9d44LfUKVo7AsIrOUOdbgOUfpvlmgTRbzoRx7BevmdgeF799/CCOxI11dFiI53XhoHe8ECGERDlZhVUmmNFisZhGCgoK1RoP16h9q/rJzsoODAhs6NayaWppoQUSlovFIgaDWVhY+MzvWUBAYFxsnIfH/Qbtgc4P3Lfxit3ZqauXZShgqLBiOVVUUEzj7U2MOdirnCboK6LUj6kUxbk7Z/BmXwlv1KjD421R6sdUCu9g11ufiP7YkBsk8fHxzdpYRUVFSFLOVHaOyMnN9fXx8fZ6amBksH5dne53qagoYbSAV1pW/aI3V6+ETd8wczKd57n6TkrZ2UNYyqdpTFFZCUM8Gonjrl97NWftoLG9/07u4GCCJZ95/O3Aw3vxNLjQdaD93KWDvTY9kvBosw5Nz0/2u7DP7wJChLLp+O2nt7oMcLVTePBEwhEn/PQ+hcINevU1IKIr73gp9nC0ZiPh56QUqkkf5QsTL27YaHn1wPCVe+e8mnYsXsJxLfz8PoXCDfsM6PRP9FtJV6mi9C/ZFN6hp50u/iZZ0ilaRk+uZ3i1l8awlhGMRDIjlNx8D0skLn/0ofEtSGX4eLxc1dvepX8nZZfudNhvQXk8nndQ4fj+zrZ3CgcZ0gmHnrwtT5kshw3SQclnPRuWMjWAvIVdNxYSpn5MpRBCsjp/w4YVyVvV9F0OluYesxBCaO3ab5eRpBjHCYFA8OLFC3//wPDwl/fu3W2uACgytQRRrNJ1dwtffNuPMObnYoSrsOw5WLzUWQU1S8OVWD2UUUJhjb/RYppGCJNvXNbTZCdBnMEkMIQIlqKGgaXTlI0nr52ZbcYWfby883wciaiczEwRku81zs1GV4mBIZyppMSuEjmdmZ5FE+1cJroYKTEINrdTT3NdQuZWrRopFiOE+Hx+YEDQtm2/urtPP3b0eGxMbAO+k/UVXfT8r+2Xkzm6ulWvrcik6LhCWs7hp/XTrLUVCJxgK2uqyXg2UmvoCY+9kiiNkVt3zx7YVUeFReAEW03PfICtQV0rn0x49OQ9ybT8+eB2916GamyCYCrpmFiZqLfSeWhisQghVFhY+NDz4eo1a2bOmHnmzNnklOS6l6CgpIChUn7FPQnqy4MLPvkU+eXuZd+K72dRfB6fxpVVlMoqgfrscf1ZibrrpEljnLsRH70eSBh46LxHR07GCHDdUX9dPbx6rG0nDQUGTrCUtbr0GvHT8rFmhKymJ4ycxzt3b6/CwjGCyRAXFQkQwjCESTzi0Nt792KFTIsl+zZP6K6twGBxDPrM27NtUjss3+++dxPOCCivj0zP7b9cT5WzXrhtrpmk98yRCfc94kSMrosO7Zrr2InLJnCCzdFQ+zqWieO8fNIoOeuf/1w90lxbiSWnZmA7zqVKWY3pyd+GV3tpMoORqPYypTWftOVkU7QgneX9ILRUyeHHrRNtUfijZzk04j1/FFCgPWj5uuGd6HiPR+U5glwP10Ha6JOXVzOmTJii3YRpTp3V5RlMFX3bmb9vm6qHl4R4BRTQdTjvNWxYkbxVTU1S1a0MTVE0TYvF4siIVzt37ZoyZerevX+GhoaQZAOnWNRtryK/ZDElz17WV7Evl1AiEI5hHCWmvRbBQAjRIu+PIjHOnN5fZYouQ5VAOIYpy+NSX/VEi559FpMEc1Y/lTHaBIdAOI5pqjE7shFCKIdHURjhYMzWZyKCwA20mNr1HyabKgNhdF18M2FxtdjJ/Ohzm1b+9ryQRgjl+Fx7uthxuPOWy85bvq5Dvq34j89+PrFLLbqP2+szDiGEkOjV78Omn0iufavWqCwlEovFwcEhvr6+ERERIlFTvumNLgrd99tt56MT9KosLAm4cCFu0BLzoTuuDt3xdXGDX8orfnPqt9NOR+a6rDjpsqJyqShyt8u0c5/qdAktjjm141i/wwu7jfn1/Jhfy0Mvub+039InDXlZQjOhKArDMD6f7x8Q8Mz32Zs3byiqgbOtFRTlMVrAq3xmShd4rnDotKLqKnQpX0BjiipKFf+f433hwc+DJi1aRBPx/zyIlXheEsUfWbpW6/BvbmaOC3c6Vnv0K47B7947XGvTY+q9Zm+rePFS+WfO9XwSVoLIUolH3IXtf/U7udp24p7rE/dUhClK9dy663FzjAJ0QeCuX+86Hh67YIvbY/cziTVrgHx7buvePifW2bluOOm6ocofKu4K8MOO7703cO8YyxkHbs2o8vfKMbxRPfmb8GovTWYwEtVW5mcpzcdTHyitWROboAXpHO/bT9c4jrIx5flteZpNI4RKgh89zRsx0RorDT53r/zOsZyNq5M2+nz8cWwz3mXCWIZD1pwesqZyAZUfvHOvRyaNkOzzXsOGFSlbfa4RWZNUdetAI4qmaJqOjIz08fENDgkR8L/rWfptTNH19qpT9JV26itVLhRlFU1/wkul0Yf4whPt1H7SZi9yZi+qspW04S0xtuiKrqq7uvxKF/mV5cvop/5ZWz/TqamCxO5Ms06cy504CCFEif65n3u1njOfmuC6n8x6F/0+LaeILyJpSlRamJ385rnn2T0rRrm6bfOq+OEUOtdz09yVp3zffCkUkKRYUJKXmfL2dUjwu/LvVpOJ55auPuObmM0jSTEvJynyXRaGydyqtSFJMiI8Yu/eP6dMmbpz586QkJCmTZgQQgjRBQF///4ws9oJX/DmwLz5v90M+5DLJylSzC/KTk2M8Pf0e1fasIqii8J2uk9bdtgjODGzkE+SopLsT1F+L5PrnoXRxeF/znRfevjhy485JUJSxMtNjg1/X8hs/ISLpiIUCoMCg7Zv3zF16rQDfx+IiopqcMKEEFJUIDC6lFfbL5nQpaWlCFNSUa486EqDL92Io1hy4sjrd99Lu5Qjv3hvmTJu5o4LjyI+ZBbySZIsLcx4/zrgxokrQXlU7U2PYWkRz6I+5ZaKKYoszfsc5X187ezVHlm0tCOuNPbo3GkLDz54+Sm3VCQsyXzrf+UP98nr7n1ppgtNOt//4J+++WyrOSuGSfrp09K4E3Mnz9p7IzAho1BAkmJ+UXZybKj3Tb+ksuOKyvJa47Zg962wpBy+WMzPSQq96x3LoxFFlzdl43pyzfBqL01mMJL3Ib1Mac2HpDdrk7QgXRhw9UEaSRcH3Pctfz0IL+SuVyZJFT275vml7NPI9RjmrIk+entKTvabCF3y+uGtgMQsnljML0h5/fj4kqmLzyRWnFRlnfcaNqxI3upb3/tgaRY0TcfGxf5z6LCbm/svv2z18/P7zgkTQogWCY88yd0ezY/Mp4pJRFJ0bpEoJJMsb2ax+IpP7uqI0rA8sphEFEWXlJKJGQLPVMnvPKRFwhPeudui+VGFFI9EIjGVliv8JEQYQlQ+b3tQyYt8ik8jsZj6nCWW+aWIb2Ec9a9fCHrwwAMhFBoaduDQ4YZ89Nbq4vkzCKHAgMA/du5svr2wmEy2PLuwUHbW2lbrufHs7GyXLl6IEPpj585mfZCvqKgoJkmZZwcHR4ey+UwHDh0ODQ1r+jgYVhsenp/2fnP/hXebYiYawDQnHQ/Y3jNk88Afrrf41X6rCqaJsR22+p4YV3R86tD9Mc2TIxCdF1x9uLTdrXlOawP+cz/K8n3GLIQQl8vNzZWdOZSNWdFFCpfSW/WLEpqWhTLPTScLVR+P/i2zg/4FhCKRsOnvKoFmUVLS2Nc1NAKmoqlF5mWLFPT6zl45SS/vyS6ftjaifi+4ls1QS5QY8+FLdgGfodbRZtSqBb1Y1PvI6Ba4Fd2qgmlmbDvX/lr0xxtP4v9NN1XAN+qSMIGqIGcC4PvC243/6+GWnkyEEKLJbN9f9j8ranNj6nciZzVl54FhSlWfnNDkF48jl9+2wFDeqoJpXvI9hzlp0Em3HkHKBP5jIGcC4PvCOHJ0Pk+shvKSQu6f/OPAg2QYdxoIY+Un+IZ2tOrcQYcjhwQFaUnRAffOHboUktnwmWltI5jmpWDr6qxOv7/uDSkT+K+BnAmA74uMO+o+4GhLR9Em0AWhJ5fOONnSYZRpVcE0L57/Zjuzzc28EzLxyMTOR5p5JwDUUyt9Xw4AAAAAQKsCORMAAAAAgGyQMwEAAAAAyAY5EwAAAACAbJAzAQAAAADIBjkTAAAAAIBskDMBAAAAAMgGORMAAAAAgGyQMwEAAAAAyAY5EwAAAACAbJAzAQAAAADIBjkTAAAAAIBsEn6j19jYeOnihd8/lP8aqOdvqamptXQIkg11HWxvZ9vSUQAAQIvpIC9w08lq6Si+HxUG+e1CCTkTl6tmB8ND84N6/hfp3Nm4pUMAAICWxGGQFsq8lo6ihcGzOQAAAAAA2TCOumZLxwAAAAAA0NrBfSYAAAAAANkgZwIAAAAAkA1yJgAAAAAA2f4P1+tpX+Gjg+MAAAAASUVORK5CYII= ) ``` custom_ridge_bp.save() ``` ``` Name: 'My Inline Custom Blueprint' Input Data: Numeric Tasks: My Custom Ridge Regressor w/ Imputation ``` ``` custom_ridge_bp.train('5eb9656901f6bb026828f14e') ``` ``` Name: 'My Inline Custom Blueprint' Input Data: Numeric Tasks: My Custom Ridge Regressor w/ Imputation ``` --- # Blueprint Workshop overview URL: https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-overview.html > An overview of common usage with the Blueprint Workshop. > [!NOTE] Note > If you are using JupyterLab, "Contextual Help" is supported and assists you with general use of the Blueprint Workshop. You can drag the tab to be side-by-side with your notebook to provide instant documentation on the focus of your text cursor. ## Initialization Before proceeding, ensure you have initialized the workshop and [completed the required setup](https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-setup.html). > [!NOTE] Note > The command will fail unless you have correctly followed the instructions in `configuration`. Use the following code to initialize the workshop. This is necessary for any Python code examples to work correctly. ``` from datarobot_bp_workshop import Workshop w = Workshop() ``` All of the following examples assume you have initialized the workshop as above. ## Understanding blueprints It's important to understand what a "blueprint" is within DataRobot. A blueprint represents the high-level end-to-end procedure for fitting the model, including any preprocessing steps, algorithms, and post-processing. **Blueprint Workshop:** In the Blueprint Workshop, blueprints are represented with a `BlueprintGraph`, which can be created by constructing a DAG via Tasks. ``` pni = w.Tasks.PNI2(w.TaskInputs.NUM) rdt = w.Tasks.RDT5(pni) binning = w.Tasks.BINNING(pni) keras = w.Tasks.KERASC(rdt, binning) keras_blueprint = w.BlueprintGraph(keras, name='A blueprint I made with the Python API') ``` You can save a created blueprint for later use in either the Blueprint Workshop, or the DataRobot UI. ``` keras_blueprint.save() ``` You can also visualize the blueprint ``` keras_blueprint.show() ``` [https://docs.datarobot.com/en/docs/images/bpw-1.png](https://docs.datarobot.com/en/docs/images/bpw-1.png) **DataRobot UI:** In the UI, blueprints are represented graphically with nodes and edges. Both may be selected and provide contextual buttons for performing actions on nodes and edges such as removing, modifying, or adding them. [https://docs.datarobot.com/en/docs/images/bpw-2.png](https://docs.datarobot.com/en/docs/images/bpw-2.png) Each blueprint has a few key components: - The incoming data ("Data"), separated into type (categorical, numeric, text, image, geospatial, etc.). - The tasks performing transformations to the data, for example, "Missing Values Imputed." - The model(s) making predictions or possibly supplying stacked predictions to a subsequent model. - Post-processing steps, such as "Calibration." - The data sent as the final predictions, ("Prediction"). Each blueprint also has nodes and edges (i.e., connections). A node will take in data, perform an operation, and output the data in its new form. An edge is a representation of the flow of data. The image below is a representation of two edges that are received by a single node; the two sets will be stacked horizontally. The column count of the incoming data will be the sum of the two sets of columns, and the row count will remain the same. If two edges are output by a single node, it means that the two copies of the output data are being sent to other nodes. ## Understanding tasks The following sections outline what tasks are and how they are used in DataRobot. ### Types of tasks There are two types of tasks available in DataRobot: - Theestimatorclass predicts a new value(s) (y) by using the input data (x). The final task in any blueprint must be an estimator. Examples of estimator tasks are LogisticRegression, LightGBM regressor, and Calibrate. - Thetransformclass transforms the input data (x) in some way. Examples of transforms are One-hot encoding and Matrix n-gram. These class types share some similarities: - Both class types have afit()method which is used to train them; they learn some characteristics of the data. For example, a binning task requiresfit()to define the bins based on training data, and then applies those bins to all incoming data in the future. - Both transform and estimator can be used for data preprocessing inside a blueprint. For example, Auto-Tuned N-Gram is an estimator and the next task gets its predictions as an input. ### How tasks work together in a blueprint Data is passed through a blueprint sequentially, task by task, left to right. During training: - Once data is passed to an estimator, DataRobot first fits it on the received data, then uses the trained estimator to predict on the same data, then passes the predictions further. To reduce overfit, DataRobot passes stacked predictions when the estimator is not the final step in a blueprint. - Once data is passed to a transform, DataRobot first fits it on the received data, then uses it to transform the training data, and passes the result to the next task. When the trained blueprint is used to make predictions, data is passed through the same steps. However, `fit()` is skipped for data. ## Constructing blueprints Tasks are at the core of the blueprint construction process. Understanding how to add, remove, and modify them in a blueprint is vital to successfully constructing a blueprint. **Blueprint Workshop:** Defining tasks to be used in a blueprint in Python requires knowing the task code to construct it. Fortunately, you can [search tasks](https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-walkthru.html) by name, description, or category, and leverage autocomplete (type `w.Tasks.`, where you press the Tab key at ``) to get started with construction. Once you know the task code, you can instantiate it. ``` binning = w.Tasks.BINNING() ``` **DataRobot UI:** If you will be working with the UI, you will need to start with a blueprint from the Leaderboard, so it is recommended you first read [how to modify an existing blueprint](https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-overview.html#modify-an-existing-blueprint), and then return here. The blueprint editor allows you to add, remove, and modify tasks, their hyperparameters, and their connections. To modify a task, click a node, then on the associated pencil icon, and edit the task or parameters as desired. Click "Update" when satisfied. ( `More details `) To add a task, select the node to be its input or output, and click the associated plus sign button. Once the empty node and task dialog appear, choose the task and configure as desired. ( `More details `) To remove a node, select it and click the trash can icon. [https://docs.datarobot.com/en/docs/images/bpw-5.png](https://docs.datarobot.com/en/docs/images/bpw-5.png) ## Pass data between tasks As mentioned in [Understanding Blueprints](https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-overview.html#understanding-blueprints), data is passed from task to task, determined by the structure of the blueprint. **Blueprint Workshop:** In the following code, you enter a numeric input into a task to perform binning on the numeric input for the blueprint (determined by project and feature list). ``` binning = w.Tasks.BINNING(w.TaskInputs.NUM) ``` Now that you have the `binning` task defined, pass its output to an. ``` kerasc = w.Tasks.KERASC(binning) ``` And now we can save, visualize, or train what you've just created by turning it into a `BlueprintGraph`, as shown in the [example notebook](https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-walkthru.html). ``` keras_bp = w.BlueprintGraph(kerasc) ``` You may pass multiple inputs to a task at construction time, or add more later. The following code will append to the existing input of `kerasc`. ``` impute_missing = w.Tasks.NDC(w.TaskInputs.NUM) kerasc(impute_missing) ``` You may replace the input instead by passing `replace_inputs=True`. ``` kerasc(impute_missing, replace_inputs=True) ``` The `BlueprintGraph` will reflect these changes, as shown by calling `.show()`, but to save the changes, you will need to call `.save()`. ``` keras_bp.save().show() ``` **DataRobot UI:** To add a connection, select the starting node and drag the blue knob to the output point. [https://docs.datarobot.com/en/docs/images/bpw-6.png](https://docs.datarobot.com/en/docs/images/bpw-6.png) To remove a connection, select the edge which you'd like to remove and click the trash can icon. If the trash can icon does not appear, deleting the connection is not permitted. You must ensure the blueprint is still valid even when you remove the connection. [https://docs.datarobot.com/en/docs/images/bpw-7.png](https://docs.datarobot.com/en/docs/images/bpw-7.png) ## Modify a task Modifying a task in the Blueprint Workshop means modifying only the task's parameters; however, in the UI, it can mean modifying the parameters or which task to use in the focused node. This is because you need to edit a task in order to substitute it for another. **Blueprint Workshop:** Use a different task where the substitution is required and save the blueprint. **DataRobot UI:** To modify an existing task, click on the node and then the pencil icon to open the task dialog. [https://docs.datarobot.com/en/docs/images/bpw-8.png](https://docs.datarobot.com/en/docs/images/bpw-8.png) Click on the name of the task for a prompt to choose a new task. [https://docs.datarobot.com/en/docs/images/bpw-9.png](https://docs.datarobot.com/en/docs/images/bpw-9.png) Now you may search and find the specific task you'd like to use. Click Open documentation after selecting a task for details on how the task works. [https://docs.datarobot.com/en/docs/images/bpw-10.png](https://docs.datarobot.com/en/docs/images/bpw-10.png) ## Configure task parameters Tasks have parameters you can configure to modify their behavior. These include, for example, the learning rate in a stochastic gradient descent algorithm, the loss function in a linear regressor, the number of trees in XGBoost, and the max cardinality of a one-hot encoding. **Blueprint Workshop:** The following method is the best way to modify the parameters of a task. DataRobot also recommends [viewing the documentation](https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-walkthru.html) for a task when working with one. If you're using JupyterLab, "Contextual Help" is supported. You can call it using `help(w.Tasks.BINNING)`. You can continue working with the blueprint from the previous step by modifying the `binning` task. Specifically, you can raise the maximum number of bins and lower the number of samples needed to define a bin. ``` binning.set_task_parameters_by_name(max_bins=100, minimum_support=10) ``` There are a number of other ways to work with task parameters, both in terms of retrieving the current values, and modifying them. It's worth understanding that each parameter has both a "name" and "key". The "key" is the source of truth, but it is a highly condensed representation, often one or two characters. So it's often much easier to work with them by name when possible. **DataRobot UI:** You can modify task parameters as well. Parameters display under the header and are dependent on the task type. [https://docs.datarobot.com/en/docs/images/bpw-11.png](https://docs.datarobot.com/en/docs/images/bpw-11.png) Acceptable values are displayed for each parameter as a single value or multiple values in a comma-delimited list. [https://docs.datarobot.com/en/docs/images/bpw-12.png](https://docs.datarobot.com/en/docs/images/bpw-12.png) If the selected value is not valid, DataRobot returns an error: [https://docs.datarobot.com/en/docs/images/bpw-13.png](https://docs.datarobot.com/en/docs/images/bpw-13.png) ## Add or remove data types A project's data is organized into a number of different input data types. When constructing a blueprint, these types may be specifically referenced. When input data of a particular type is passed to a blueprint, only the input types referenced in the blueprint will be used. Similarly, any input types referenced in the blueprint which do not exist in a project will simply not be executed. **Blueprint Workshop:** You can add input data types to tasks in the Blueprint Workshop just like you would [add any other input(s) to a task](https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-overview.html#pass-data-between-tasks). For the sake of demonstration, this code example adds a numeric input and then subsequently adds a date input. However, you can also add them both at once. ``` ndc = w.Tasks.NDC(w.TaskInputs.NUM) ndc(w.TaskInputs.DATE) ``` **DataRobot UI:** If the current blueprint does not have all of the input variable types that you'd like to use, you can add more. Select the Data node, then click the pencil icon to modify the input data types available for the current blueprint. [https://docs.datarobot.com/en/docs/images/bpw-14.png](https://docs.datarobot.com/en/docs/images/bpw-14.png) In this modal, you can select any valid input data, even data not currently visible in the blueprint. You can then drag the connection handle in order to denote data passing to the target task. [https://docs.datarobot.com/en/docs/images/bpw-15.png](https://docs.datarobot.com/en/docs/images/bpw-15.png) ## Modify an existing blueprint Consider that you have run Autopilot and have a Leaderboard of models, each with their performance measured. If you identify a model that you would like to use as the basis for further exploration in the blueprint workshop, you can use the instruction below. **Blueprint Workshop:** First, set the `project_id` of the `Workshop` for the project that contains the model you want to use. ``` w.set_project(project_id=project_id) ``` Retrieve the `blueprint_id` associated with the model you want to use. You can do so in multiple ways: If you are working with DataRobot's Python client, you can retrieve the desired blueprint by searching a [project's menu](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest/entities/blueprint.html#list-blueprints): ``` menu = w.project.get_blueprints() blueprint_id = menu[0].id ``` To visualize the blueprints and find the one you would like to clone, [DataRobot provides an example workflow](https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-walkthru.html). You can also search a project's [Leaderboard](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest/entities/blueprint.html#get-a-blueprint). ``` models = w.project.get_models() blueprint_id = models[0].blueprint_id ``` By navigating to the Leaderboard in the UI, you can obtain a specific model ID via the URL bar, which can be used to [directly retrieve a model](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest/entities/model.html#retrieve-a-known-model), which has a `blueprint_id` field. Once the `blueprint_id` is obtained, it may be used to clone a blueprint. ``` bp = w.clone(blueprint_id=blueprint_id) ``` The source code to create the blueprint from scratch can be retrieved with a simple command: ``` bp.to_source_code() ``` The command may output: ``` keras = w.Tasks.KERASC(...) keras_blueprint = w.BlueprintGraph(keras) ``` This is the code necessary to execute to create the exact same blueprint from scratch. This is useful as it can be modified to create a desired blueprint similar to the cloned blueprint. To make modifications to a blueprint and save in place (as opposed to making another copy), omit the final line which creates a new `BlueprintGraph` (the call to ( `w.BlueprintGraph`)) and instead call the blueprint you'd like to overwrite on the new final task: ``` keras = w.Tasks.KERASC(...) # keras_blueprint = w.BlueprintGraph(...) bp(keras).save() ``` **DataRobot UI:** Navigate to"Describe > Blueprint"and choose Copy and Edit. [https://docs.datarobot.com/en/docs/images/bpw-16.png](https://docs.datarobot.com/en/docs/images/bpw-16.png) DataRobot opens the blueprint editor, where you can directly modify the blueprint to, for example, incorporate new preprocessing or to stack with other models. Alternatively, you can save it to be used with other projects. ## Validation Blueprints have built-in validation and guardrails in both the DataRobot UI and Blueprint Workshop, allowing you to focus on your goals without worrying about remembering the requirements and the ways each task impacts the data from a specification standpoint. This means that the properties of each task are considered for you and you don't need to remember when a task only allows certain data types, requires a certain type of sparsity, handles missing values through imputation, imposes requirements on column count, or anything else. Blueprints automatically validate these properties and requirements, presenting you with warnings or errors if your edits introduce any issues. Furthermore, other structural checks will be performed to ensure that the blueprint is properly connected, contains no cycles, and can be executed. **Blueprint Workshop:** In the Blueprint Workshop, call `.save()` and the blueprint is automatically validated. ``` pni = w.Tasks.PNI2(w.TaskInputs.CAT) binning = w.Tasks.BINNING(pni) keras = w.Tasks.KERASC(binning) invalid_keras_blueprint = w.BlueprintGraph(keras).save() invalid_keras_blueprint.show(vertical=True) ``` [https://docs.datarobot.com/en/docs/images/bpw-17.png](https://docs.datarobot.com/en/docs/images/bpw-17.png) **DataRobot UI:** [https://docs.datarobot.com/en/docs/images/bpw-18.png](https://docs.datarobot.com/en/docs/images/bpw-18.png) ## Constraints Every blueprint is required to have no cycles; the flow of data must be in one direction and never pass through the same node more than once. If a cycle is introduced, DataRobot throws an error in the same fashion as validation, indicating which nodes caused the issue. ## Train a blueprint **Blueprint Workshop:** Use the `keras_bp` from previous examples. We can retrieve our `project_id` by navigating to a project in the UI and copying it from the URL bar `.../project//...`, or by calling `.id` on a DataRobot `Project`, if using the DR Python client. ``` keras_bp.train(project_id=project_id) ``` If the `project_id` is set on the `Workshop`, you may omit the argument to the `train` method. ``` w.set_project(project_id=project_id) keras_bp.train() ``` **DataRobot UI:** Ensure your model is up-to-date, check for any warnings or errors, and click Train. Select any necessary settings. ## Search for a blueprint **Blueprint Workshop:** You may search blueprints by optionally specifying a portion of a title or description of a blueprint, and may specify one or more tags which you have created and tagged blueprints with. By default, the search results will be a python generator, and the actual blueprint data will not be requested until yielded in the generator. You may provide the flag `as_list=True` in order to retrieve all of the blueprints as a list immediately (note this will be slower, but all data will be delivered at once). You may provide the flag `show=True` in order to visualize each blueprint returned which will automatically retrieve all data ( `as_list=True`). ``` shown_bps = w.search_blueprints("Linear Regression", show=True) # bp_generator = w.search_blueprints("Linear Regression") # bps = w.search_blueprints(tag=["deployed"], as_list=True) ``` **DataRobot UI:** Searching for blueprints is done through the AI Catalog, and works just like searching for a dataset. You may filter by a specific tag, or search based on the title or description. ## Share blueprints Building a collection of blueprints for use by many individuals or an entire organization is a fantastic way to ensure maximum benefit and impact for your organization. **Blueprint Workshop:** Sharing a blueprint with other individuals requires calling `share` on the blueprint, and specifying the role to assign (Consumer by default, if omitted). The assigned role can be: Consumer: The user can view and train the blueprint. Editor: The user can view, train, and edit the blueprint. Owner: The user can view, train, edit, delete, and manage permissions, which includes revoking access from any other owners (including you). ``` from datarobot_bp_workshop.utils import Roles keras_bp.share(["", ""],role=Roles.CONSUMER) # keras_bp.share(\[\"\","\"\], role=Roles.EDITOR) # keras_bp.share(\[\"\", \"\"\], role=Roles.OWNER) ``` There are also similar methods to allow for sharing with a group or organization, which will require, respectively, a `group_id` or `organization_id`. ``` from datarobot_bp_workshop.utils import Roles keras_bp.share_with_group([""], role=Roles.CONSUMER) keras_bp.share_with_org([""], role=Roles.CONSUMER) ``` **DataRobot UI:** In the UI, sharing a blueprint is just like sharing a dataset. Navigate to the AI Catalog, search for the blueprint to be shared, and select it. Next, click "Share" and specify an individual(s), group(s), or organization(s), and choose the role you would like to assign. --- # Blueprint Workshop setup URL: https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-setup.html > Learn about the first-time setup necessary for using the Blueprint Workshop. This page explains how to configure the Blueprint Workshop, including installing the required libraries, enabling your DataRobot account, and establishing a connection to DataRobot by providing credentials. ## Installation Run the following steps to install the Blueprint Workshop. 1. mkvirtualenv -p python3.7 blueprint-workshop 2. sudo apt-get install graphviz or brew install graphviz 3. pip install datarobot-bp-workshop ## Connect to DataRobot Use the following sections to connect to DataRobot in order to use the Blueprint Workshop. Each authentication method below specifies credentials for DataRobot, as well as the location of the DataRobot deployment. DataRobot currently supports configuration using a configuration file, by setting environment variables, or within the code itself. ### Credentials Specify an API token and an endpoint in order to use the client. You can manage your API tokens in the DataRobot application by selecting your profile and navigating to API keys and tools. The order of precedence is as follows. Note that the first available option will be used. 1. Set an endpoint and API key in code using datarobot.Client . 2. Set up a config file as specified directly using datarobot.Client . 3. Set up a config file as specified by the environment variable DATAROBOT_CONFIG_FILE . 4. Configure the environment variables DATAROBOT_ENDPOINT and DATAROBOT_API_TOKEN . 5. Search for a config file in the home directory of the current user, at ~/.config/datarobot/drconfig.yaml . For more information, read about the different options for [connecting to DataRobot from the Python client](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html). > [!NOTE] Note > If you access DataRobot at `https://app.datarobot.com`, the correct endpoint to specify would be `https://app.datarobot.com/api/v2`. If you have a local installation, update the endpoint accordingly to point at the installation of DataRobot available on your local network. ### Set credentials in code To set credentials explicitly in code: ``` import datarobot as dr dr.Client(token='your_token', endpoint='https://app.datarobot.com/api/v2') ``` You can also point to a YAML config file to use: ``` import datarobot as dr dr.Client(config_path='/home/user/my_datarobot_config.yaml') ``` ### Use a configuration file You can use a configuration file to specify the client setup. The following is an example configuration file that should be saved as `~/.config/datarobot/drconfig.yaml`: ``` token: yourtoken endpoint: https://app.datarobot.com/api/v2 ``` You can specify a different location for the DataRobot configuration file by setting the `DATAROBOT_CONFIG_FILE` environment variable. Note that if you specify a file path, you should use an absolute path so that the API client will work when run from any location. ### Set credentials using environment variables Set up an endpoint by setting environment variables in the UNIX shell: ``` export DATAROBOT_ENDPOINT='https://app.datarobot.com/api/v2' export DATAROBOT_API_TOKEN=your_token ``` --- # Blueprint Workshop walkthrough notebook URL: https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/bp-walkthru.html ``` import datarobot as dr ``` ``` from datarobot_bp_workshop import Workshop, Visualize ``` ``` with open('../api.token', 'r') as f: token = f.read() dr.Client(token=token, endpoint='https://app.datarobot.com/api/v2') ``` ## Initialize the workshop ``` w = Workshop() ``` ## Construct a blueprint ``` w.Task('PNI2') ``` ``` Missing Values Imputed (quick median) (PNI2) Input Summary: (None) Output Method: TaskOutputMethod.TRANSFORM ``` ``` w.Tasks.PNI2() ``` ``` Missing Values Imputed (quick median) (PNI2) Input Summary: (None) Output Method: TaskOutputMethod.TRANSFORM ``` ``` pni = w.Tasks.PNI2(w.TaskInputs.NUM) rdt = w.Tasks.RDT5(pni) binning = w.Tasks.BINNING(pni) keras = w.Tasks.KERASC(rdt, binning) keras.set_task_parameters_by_name(learning_rate=0.123) keras_blueprint = w.BlueprintGraph(keras, name='A blueprint I made with the Python API').save() ``` ``` user_blueprint_id = keras_blueprint.user_blueprint_id ``` ## Visualize a blueprint ``` keras_blueprint.show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABa8AAADECAYAAACY9t2uAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3gU5d7G8e/sbjYBQnoBpPdeAoTeFJQqHaWIFVTE3hW7R1GPFSuvoDQ5ikrvTaoQekLvnRDSgCSk7c77B4ghJJBAQjZwf64rSjLlqTM785tnnzG8/QNNRERERERERERERERchcEUS0HnQUREREREREREREQkMwWvRURERERERERERMTlKHgtIiIiIiIiIiIiIi5HwWsRERERERERERERcTkKXouIiIiIiIiIiIiIy1HwWkRERERERERERERcjoLXIiIiIiIiIiIiIuJyFLwWEREREREREREREZej4LWIiIiIiIiIiIiIuBwFr0VERERERERERETE5Sh4LSIiIiIiIiIiIiIuR8FrERERyXOGd1tG/DqPtZum8Ux1a863C+rI+1Pms27TZIaWvTkvUzwaPsOs8AMc3z6b15t7FnR2bl72Zrw0ZQFhmxfxVmO3gs7NVV1vvzC82/Dq5Dms2TA1V8fc9SqodF2d6kVEREQkb9ycd4UiIiJyzbw6f8ueqCjiT67jw6b2a9tJkXI0bhVCtRLFcTNyvpmlWEWatGxAlWBPrLnYrlAxLFgsBoZhxWpcfyGtFfrw2ZT5rN/0DT2KXsMO7G35ckck8dFH+X2QPzdNtdtKUKd5faqW9MJeGAqVTb/IcfsWKU/TNo2oXsorV8dcdgoq3Wtiq8bgUVNYtHwt23ft4fjx48RGnSDq8G52hi1k+o8f8EzPBgTlwTOMQlUveSxXnw3X0yY36zlJREREromtoDMgIiIiLsQIott9HQmwAJSh96B2fLBmPmcLOl83keT1n9G59md5tj9LYB3atW5AhfTjaHxn4ZVdvyio9i1U/cpSkobtW9Eo8NJxOfaiPpSo6EOJivVo0/0hntv+P156+GX+tzfl2pMqTPWSl3L72XAD20RERERubhp5LSIiIhdZK/bh/jbFLox0sxDY5T66BWncm4gUBqkseyWEEiVK4BtYgsCy1ajTpg/DPp3HvmQDr1r9GTX5Xdp565yWW9f+2aA2ERERkeuj4LWIiIhcYCdk0CBC7Cbxi8bz+2EHhmdbHuhb4dYaYSgihVZa8jlS0p2YppO0pDiObFvOLx/ez10P/cJBh4Fb+QE83/c23QTlyvV9NqhNRERE5HroGkFERETO82zDA/0qYnNEMu2Hd/l88g7SsBMyaCD1r3Hqa4yiVOn8LF/9upiIPYeJOraPPevmMuHd+2kamLvZy9xC32HTySjij/1E78vmmjXwuXciJ6OjiF79Og2y2rV7adoM/ZD/LVzP/sNHOXlwB5sWjGfkg00Izry+R1W6PPkWo37+k2Vh4Rw4fJSYk0c5tnsTf0//nvcebMltWdWJ2200H/gcH/3fbyxevYn9h45wKvIoR3asY8WvT9LQDYwS9zP9eBTxJxbzYo0MoZ8i1en5wn/4fuI0lodtYf/BI5yKiiT62B62L/+db565i4pFrlBB7l0YcziK+Oh/f05OGUyeD5z/J5+TprMibAv7Dx0hOiqS6CM72TR3NK92qUQRj9K0GvwG/zdtOTv2HyU68jAHNy/m15H3E+qf1eWnlap9P2TijCVs2r6XyBMniD6+j51/z2DMaz2pUTz7QhjFq9HzpW+YuXILB48cJfJABGtnjubdwQ0JvFLZc9MfLmOhzNDpRJ2KInbbJ7S7rC8UofN3u4mLPsqMh0pedsFtC3mDdZFRxB3/kyGlzi/Ntl9czG8O29cSTIeXfmTuqs3sP3yUUzntP9nJ73Svqx1yyiR6yfdM3JYGhp0GzRpSBCvVn13IqegoYja+T7Ms515uw+fbIok/dYhf+/tdOvdyfteLWyCh973FT7NWs2v/EU4e3kXE0sl88eRdVMpqru3rPX9cSX58NmTZJiIiIiKX05zXIiIiAhgEdR1MtyAL6bt/4+eVZ9ixdwJ/PzWS1pX6cH+rz9iwODH3u7VVot+rL2X4gweBFRrSbVgInft0ZsQ9D/JdRFKelSI7hl8LXps4ludDfTMEEv2pENKRxxrcQZeWz9Lt0d84mH5hfZ9mDH31CdpkCswU87uNGi16UaNFD+4f9CNDB77FvEjHv+n4t+flT165bDu3wHJU9zM467xCHr2b8MBzQy7bFndvStVszcCaLenc7l269fuWrQU4PWy2+SziR4XGPXjpp3bcf8qN4OCilwT7fErX4a5HPqZNiwr07vwOq86aGZZa8K7Vjk7NK2YYyVmcElWa0vu5UDq0KEPXXl8RnqnctrLdGfXb19xb2T1DWsFUa9aDas0u/OrgMrntD5dzcmLVSvY4mlHLrwENK1hZuitDQrYaNAnxxMBC7ZA62MeeIDlDWYMaNKCsFdJ3rGLlySt0imthDSK0W5cMf7DfmP5zDelefzvkgjOKE1EmYGDz9sHTcLBv9d8cc9SjfMlQmpW38veeSzuLrUro+Yct6TtYtfY0ZtZ7vrJrqRefxjz/8zhebRGQ4cW17pSpcwcP1Lmde+6dzLABLzL1UNq/2+Tb+SOfPhsgizaBxGuqZBEREbmZaeS1iIiIgLU8/Qa3pTgprJ/4C+Fp4Dw6jZ8XxOO0lKD74E4EXMsIXjOVo399y9N9WlG9XGmCy9ej1f0fMe9IGpagtrw75nVaFcvz0lzKUpK+n47m+VAf0g7N4/372lK9bGlKVG1GzzdncTDNRplu7zOy3+UjZEnfy+j+IVQocxt+QaUpXbst9775OzsSDLzqDeHHHx+lelYjNh0HmTi0LXWqlScwuDRlarWk8zO/sS+LQOpl0vcxZnAzqlcuT2DwbZSq0Yp73l/IsXQD3+Yv8cGgMllfwKXM5uGyQfgE/PsT3Hc8UfkVDPonn5XKEViiNOUb9+W9JVGYFm9KBDnZ8fs7DGwfQrlSJQmqEEKnV2dxOB08qj/Ea/0zlyGdnVNeZWDnVtSqUp6g4FKUqNacXm/P40iagVfo07zWM/DSka+2Kjz67RfcW9kdM3Y93z/VlQZVyhJUuhohnR/l/ak7OJNVXPh6+kPGHO9ZxcpIB9iq0qSR7yV5s9zWhKZlbIAFr0ah1LxkuEhRGjWtg91wcGLVSvbmpE9AztvXcZQ/XuxO07pVKVEyF/3nRqebR+2QY5aSlClpAUwcZ+JJMCFty1KWRTvBVoPWzTP1LwwCGjWhkhUcx9aw5kimhsq3egmm96c/8VrLAIyErYx/vgchVcoSXK4urR/8L0tOOPCo1p9vxz5NvaxGPV/r+SM7+fXZAFm2iYiIiEhmCl6LiIgIbnUGcF+IO5xbzeSph3ACmHHM/3U+0U4DrzsG0afsNVw2pO9i3BvvMe6vXUQmppKScIKI2Z9y/8BPWH8O3CoM4KmeweTnq7rsDR/lpU6BGOfW8+HAR/jv3O1EJqWSHLuPpd8OY+joPaRbvGnXvxulMxfRPMepoyeIO5eG05lKQuR25n37BF0eHs/+dPAMfYoXO/lenn/nWQ7t2MWRmCTSHKmcPbmbddsisxoEfDkzicgDB4mMTyLNkUbSqV3M/3I4r82Mw2kUoUmX2/N+KpBr8U8+T58jLT2V+APL+PzZT1mZbIKZyubff2T25qOcTnWQevYof//4Au/PP4tpuBPSOpTil+6Ms9uWMi9sF8fikkh1pJMcs5cl3zzJ6zPjcRqeNGkVgnuGLYq0HMoTjYthOA7x86P9efWXMA7EJZOaHMf+sKn89+kvWJbGZa6rP2SUuoXla87gNOw0atkow5QHBr7NW1D3wkMNW9lmNM+4I3t9WoV6YjhPs3pFOFlk8fo449i5Joydx+NJTruB/SeX6eZZO+SIhcAOjzKgug3MVLas3cQ5gOQw5i+Lx2nYaXxna/wvqZcihDStg7vh5PTav4m41obKbb00eJSXuwRhcZzgj6fv4elxq9kfl0xKYiThMz9mwKDP2ZICReo+ykvdAy4/9+Tx+SPfPhuyaxMRERGRTBS8FhERueV50GJQHyrbTBKW/8Gck/8Of0tc/juzTjgw3BszsF+1PHtxY8qOCYxZloRpFKV5h5Z45dF+L+dGg26dqWAzSV45gQm7UjMtT2bTopVEOQ3stUKo557lTjIxiV36Od+sTsG0+NKhW0vye/A4ZjzLF28g1TSwVapOZRed+M15Moy/9zvA4knFykGXXmia8axbu5s0DGy3laVUTjqTeZatW/bhwKBoUBBeF4NubtRrfzvBVkjb9gujl+d0Soe87A9JrF60hiTTglfTVtS/OAK/CKGtGuPuOMy6DSdxuNWmZVOfi0FGW9UWtAy2YiasYsGa5Gz2nccKqv9km25+HJeZWNzw8C5B5ZD2PPDmeOb/372UtZqkH/2DL6ccOR+EJZHlM5cQ5zQo2uwu2nhniOq61aR5Iy8MM5HVS8LI05bKtl5s1O3aiQo2SN89mVFzTl3Wr5PDx/Dd4gRMw4u23drinZNA9DW3fx5/NuSoTUREREQu5aK3PiIiInLDeN3OgG4lsZqnWfzHQqIzRkuS1zBl5lHuf6wcNfveQ8Mv3yYsc5zpWpinCd9ygPSOtXGvVIXyNtiSF/PaZmYUp0rVUlgxKNLhK/ad+ir7dT0CCfaxwLkchFCcUaxdewBH6xoUrVaDCraZRORH/i8ySYw8wWkTgnx8chawKgiOGE5GOwELXt5eWCBDQMpJbHQMTsCtWHE8LVwyH7UtMIR+Qx+hb7u6VC5diiAvCwknj3DcEYwVMG228/P/moDhRZUqwVhxEh8RzoGcTr2Rp/3BJH75QtYkd6R9yRa0rmpj9bZ0sDegXXMviJ7JF98XZdQPvQht04Si/5tLIhZKt25FJZvJuTULWX7mRs2TUFD9J5t08+u4xE77L3YQ90XWeUnaN4MRD73GvNh/6z1h+VTmRfdiYEBLurXz4c+pcZiAtWxTmpa2YiaHMX95/LXNd52t7OrFi+o1ymDDSezmDezK6pxinmbj+j2kdw7BvVpNKlth/VXPPdfY/nny2ZD7NhERERHJSCOvRUREbmkGgZ3upZOfBWfsIqYsissUpEll/R/T2ZcO1vI9GdCyaB6la3L2TML5OGRRT4rmKphm5HyaEaMYnp45XdkD9xyP8HRyOv40TsDwLE6xGxAMNFNSSDXBsLm58OiDVFIvBLAslsvHYqamXYiyWa2XjNS0VRzAuMUz+ebZPrSrX5UyAZ6424viX6Yadcr7XH7BahSlWDEAk4SzCTkfsZnH/cGM/osFG9MwbVVo364sVsBW63balbBw9u9lrFj+F2vPGfi0up3G7oARRLvb6+JmprJu4TJuZLyuoPpPlunm23GZIV3TQWpSHCf2bWH59J/4zxN307j1EMZuy/SC2MTlTJ5xDIfFhw79uxJsAbBQou3t1LGZpG5cxNKYvG+oK9eLydnTZ7Lp107OnD7f5y2eFx4CXWt6V5T3nw05bhMRERGRDFz33kdERETyn+U2evZvi6cBhn8fJh3oc4WVg7m7/x28tXQmp687lmPB28cbC2AmnM3Ri7rMtHTSTcBwx6OIAUk52SiJxEQAJ9ETB1DzmSXkxcBxMCju5YkBmEmJnCvQQYOFacRiFnk1Auj99jt0LmXDceIvPn9jJJNW7OJ4fDJG0SBCn/+FP4fXyrSbRBISACx4+XpjhZzNHZ3X/cF5nAXzNvNe88bUad+WEt8eovjtbSlvTWThwpWcjbUzPyyFjq3b0L6eG8v2teLOxu6Q+jdzFp3MYdC9oNo3H9PNt+MylUXP1KPvxJhc5D6FNeN/Ydvgl6jTcgD9KvzCV/v9aNehIXbSWDtvEccvaaj8rJd/+rVB8QvfXLicBS9vz/PnzsQEEvJrro08+2y4ljYRERER+ZdGXouIiNzCrJV6cW+oew5HMlvw7XAPXQLyYJixJYimTStixSRp9w4O5mDKDWd8DHEmYLmNcjmaMBkwz7B370kcWPAJaUTVvHpsb3hTt14FbJik7NuTo/znFzMllZQLQX13d1edT+QK3GrRrHFxDDOZRf95lA+mbeRgTCKpDgcpZ09y6MTZy4Ne5ll27TyKA4PiDZtS2y2rHWchz/uDk8NzZrAhDeyNOnFnqUp06lgdW/I65v0Vj2meYtH8DaRaSnNXxzoEtO5Ii6KQsm4Gs4/lMHRdQO2br+nm13F5jdJ3TGL0sgSw1+fhIc3wLNGJXs09IHU9U2ddOhdz/tbLP/3aglf9EKplVS+GFw0aVcGGSfKubezL6ZQ5uVRgnw0iIiIimSh4LSIicsuyUat3H+raDRzHx9OrTBA+AVn/+Lf4gE2pJkax1vS/+7brvIAw8Gv3LMOa2TGc8SyetYKEC0tM8/wPhh0Pt0sDIc5j29ke5wRrJTrcWSmHXx9LY9OCJUQ6wFZtAE91Csz5lCNX4F5jMA+3LYphJrF64UrO5ME+r5UzOooYJ2CpQLUKefVKzRvMPP8fR7ozh6Mz09gyex4H089POfLiveUKrD84j8xh+vpUcG9C78ceoGstGylhc1gYbQJOji+Yy4Y0KxU6DeCZ3q0pTgprp8/LNJr3CvsvoPbN33Tz57i8ZmYkU/9vKscdVsre+wwvPH4PzYtASth0ZmV6yJDf9bJl9jwOpIOtan+euCvgsnrxqP0wj93uiWGeYdnMZcTny3DmgvpsEBEREbmcri9ERERuVfYG9OtdBRsODk37jZXnsl/Vsed3Jq1NxjTcadKvBxVzGrMxihJc9jb8irhhMWwUDa7BXY9/w6wxg6log8T1o/h49r9zqZpnYohzmmCrQc+HOlDNz/5v8CZ1Db9PP4rDcKPeU9/x6aBQynrbsRg2PHxKUL6kV5YBsOSV3/L5ytM4raXo+/Wf/N/wzoSU9cXDZmCxF6dE1SbcfV9HqmcV/bRVY/B7bzC4TTWCi9mxe5agdufnGDfxBRp5QNr+SXw59WSBfh3eGbWRDUfSwVaBga8Oo8VtxbBZ7BS/rS4du4VSwtWv9tJ2sCH8HKZRhDue/5BHWlUmoIgNAwObuzf+3lmP/kzd8D0fzo7CafHjzo+mMvnVnjQq643dYmB196JEhdvwyWLD6+oPWXEeZ+YfaziHO82HPEiIWwprZi7k5IWYp/PoHGasT8NaeSCPdfCCpFX8PudEjufpLqj2ze9087wdrlPisu/4Zt058GzF0483wp0kVv4597KHDPldL6kbvuejOVHn62XUZD67rynlfezYiwZTu8sLTJj4LA08IDl8NB9PP5U/554b8dkgIiIikkOufjsjIiIi+cSjaV96lrVC+h5+n7LxynPOOo8x/dflJJoG9nq96VM9hxEKWyUembSe/UeOEXvqOMe3LePX9/pQ09PkbMRPDHnkO3ZkmKzYjF/BzBWncRoe1B06ntULR9Dk4pQQyaz69DUm7E2BYrW5/4tZhO87Suyp40Tu3cJfrzfFnlUeHAcY+/hjfLf5NGaxavR5+2eWbNxFZORJYo/vY+fqmYz/6EnuuC2LyyLDTpm2w/jqjxXsOnSUqIPhrBz/Ch3LuuE8tYy3h3zIysScVUW+SdvMmG+WEeu0ENBuBLO3HCA66ihHtixi8tdP0NrLxb/Kb0YxZeQo1p818ajSh/9OXc3eI8eJiz5J9LFdLH2pAVnOCuKM5I/nHuD9ZVE43ErT4fkfWLRxD1FRJ4k5tpedy96kVVYd4nr6Q5acnJj5CwtPm1isVkhew7T5kf8Gp53HmTltLSlYsVpNYudPYkZULkKOBdW++Z1unrfDdXLs4+cPJ3HQYWAYBmb8IibNyuLBVH7XizOS3597iA9XxUDxejz4+Qw27z1K1OEIVo57iQ632Uje/T+GPfQFm1OuL6ns3JDPBhEREZEcUvBaRETkllSMtn27UNJqkrrlN6Zsv9qkzSbR8/9k8WkTbNXo1ate1gHFf9Y+vZZxX4zl9yUb2Xk0hrMp6TidaSTFHWfnqql8/VJfmnV8hTnHM03Y6jzOL8MH8sq4pWw9GkfCnt3sz5A1M3oBz3XpztPfzWXDwViS0p2YzjSSz0RzeOcGFk8dx5ffL+BI5t1GLeb1zm3p/tL3TF+zm8gzqTic6SQnxHBo22pmTJjO5qQsCpK+n98//pTJy7Zz/HQKaamJRB/ayJzRr9Ct7UC+CS/oyDWAk0MThtJt+LfMDT/K6RQHjtRETh3cxII/VnG4EMSSkjd/Tq9OQ/nwf8uIOBLHuTQHjtRkzsZGcmh3OGsWz2Ty3G2XvdjTPL2ez/q1psOwT5i0OJxDMWdJcZg4UpOIi9xP+Ko5TPjmW2YduLRDXHN/yIYZu4BJs0/hxCRxxZ/Micw4XNfJ8dl/suKcCY4TTJ+0OJcvPC2o9s3/dPO6Ha5X0t/fM3pdKiZOTs6azPy4rBoq/+vFjA/jkz5t6fLi98xct59TCamknovj2La/mPifB2lz5zP8eShHryi9Bvn72SAiIiKSW4a3f6Be/CwiIiKSgVHifqZt/IQ2RgT/uf1OPtmRT29FExGX4VblcaYveptmxibe69SNz7blV4BYRERERHLEYIpGXouIiIiIyC3GwLdcNUp727F5+FGl9VB+mPQ6zYom8PfIp/lagWsRERERl3CDXoEiIiIiIiLiIgw/uny0iFHtM7wQ1Ezl0J8v8OgPu648z7OIiIiI3DAKXouIiIiIyK3F4o978kFOnauEnyWBk3vWM3fCl3w8bi1RmiVIRERExGVozmsRERERERERERERcS2a81pEREREREREREREXJGC1yIiIiIiIiIiIiLichS8FhERERERERERERGXo+C1iIiIiIiIiIiIiLgcBa9FRERERERERERExOUoeC0iIiIiIiIiIiIiLkfBaxERERERERERERFxOQpei4iIiIiIiIiIiIjLUfBaRERERERERERERFyOgtciIiIiIiIiIiIi4nIUvBYRERERERERERERl6PgtYiIiIiIiIiIiIi4HAWvRURERERERERERMTlKHgtIiIiIiIiIiIiIi5HwWsRERERERERERERcTkKXouIiIiIiIiIiIiIy1HwWkRERERERERERERcjoLXIiIiIiIiIiIiIuJyFLwWEREREREREREREZej4LWIiIiIiIiIiIiIuBwFr0VERERERERERETE5Sh4LSIiIiIiIiIiIiIuR8FrEREREREREREREXE5Cl6LiIiIiIiIiIiIiMtR8FpEREREREREREREXI6C1yIiIiIiIiIiIiLichS8FhERERERERERERGXo+C1iIiIiIiIiIiIiLgcBa9FRERERERERERExOUoeC0iIiIiIiIiIiIiLkfBaxERERERERERERFxOQpei4iIiIiIiIiIiIjLUfBaRERERERERERERFyOgtciIiIiIiIiIiIi4nJsBZ0BEREREZHuPbpTs3qNgs6GiAgA23fuYPq06QWdDRERkVuegtciIiIiUuBqVq9By1YtCzobIiIXTUfBaxERkYKmaUNERERERERERERExOVo5LWIiIiIuJRBgx8s6CyIyC1q4vifCjoLIiIikoFGXouIiIiIiIiIiIiIy1HwWkRERERERERERERcjoLXIiIiIiIiIiIiIuJyFLwWEREREREREREREZej4LWIiIiIiIiIiIiIuBwFr0VERERERERERETE5Sh4LSIiIiIiIiIiIiIuR8FrEREREREREREREXE5Cl6LiIiIiIiIiIiIiMtR8FpEREREREREREREXI6C1yIiIiIiIiIiIiLichS8FhERERERERERERGXo+C1iIiIiIhkyRrcnGGfTGDp3xvYs30TESumMrJTAEZBZ+ymYaVcrw+ZseBPXgu1FXRmroOFUne/z4yFs3inpdtV1r1ZyiwiIiI3goLXIiIiInKLcyfkqV/ZuH4OH93pn8+B2RuZ1nWy1+LpH77mhbtDKO/ngc1qxzOwBO4pCZgFnbcbJv/bq3iZmtQs44OHUdC94XrKalDstmrUKO2DRw42dJ0yi4iIiKvTo24RERERuel4NHmSn1/vQrlgP3yLF8GNdJIT4jl5eDcbV8xm3PjZRMQ5Lq5vGAaGYcFyA2JpNzKt6+HepB/9q9lJ3fMbLzz7FQv3J2APLIVnQkpBZ+2GKiztlRdupbKKiIhI4aDgtYiIiIjcdKxBValfrQzuF/9ip6h3EBXqBFGhTgu692jFcwNeZuYJJ5DChi/70eDLG5GzG5nW9bAQXLkS3kYKK8Z+yew98ZhASuQhzhZ01m6owtJeeeFWKquIiIgUFpo2RERERERuUuls+aIHNWrUpmLNhtRr0ZG7H/uQ33cmYS11F8P7VcNa0Fl0WQbuHnYM8xzR0Ym30DQhUlhY3IsTGByIb1GNxxIREbmZKXgtIiIiIjcpE0dKMqlOE9ORzJnoI0QsnchbP6wm2YTiXp4X5vW1UuXxKezZsZKPWv3zsjkD/xZD+Wz0JOYtXk74ls3s3b6ZratmMPGd/jTwzTivQm7Wvd60LvAoTbsh7zFx1lLCw8PZEx7G+sVT+e279xgS6nPl+YqLlOfOJz5iyvwVbIvYSMTyqfz89kBCAzOH8g2w+NLv/zZzYNe28z9bf2Zwycy3ELnJvxst3v6Lfdun8mx16yX78O75Lbt2beLnPn4X8p/VfjewcdFEPnuoKdUa9ublz8azeFUYu7ZtYOOCcYwcWAefTIU3ilWm67OfMXXxanZEbGDjol/46ok2lHbLkHb9e3jz8zHMXLCMiPAt7I1Yw5pZ79DJL6v2+qcey9L+8Q/439xlbI3YxLawJSyY8DY9yp0vl+HfltfGTWXFmnXs3h7Org1LmPPDy/SqViwX80nbaPTqfPbuXMM3nYpnKpg/94zZwP6Inxhc0pLD9HJf1lyVw7BTtccIfpq+hIiIzWxfM4/fP3uCjuWL5Ki0V28rsPg14tEv/mT9hr8JW/4XGzauZ8uCT+hdSre2IiIiNyM9phYRERGRm5/FhkcxH0rXbMuDjzTD3RnNihU7cWS/AX51O9CtTc1LLpiLBVSixb2vU79qEXrdN0SfC5sAACAASURBVJbd6bld93rTAjyqM2T0j7wS6pthbuJi+Jeuin/pitg3jmVsWHzWZfOoyaOjf+SlUO9/R7EEV6VN/1dp3roezw96lZnHs6+VPMn/de3XDd8yDej58hh6ZlrbXq4R94z4Dr/E3jw27SROgKJ1GD7m/3imQfGL5fUoU49uT35F/VJP0X3EMuJMC0HN+nBf54zpFCcwyI2UhGyy5l6dIT+M4ZUmPv/Woz2YyvXL4HnOef73dB8qhVSltP3Ccs9garQdzCe1g0jr/gIzo3Mynj2diOV/Ez24N42b18F97mouzjherCEt67rj2LGSFVFO8MxJetdQ1tyUwyhG/a59/v3dXoaGXYbRoHl93hk0jPF707Ivak7ayihB35GjeKmNF0Z6IjEnEzA8/fEJciPltDMH9SkiIiKFjR5Pi4iIiMhNyo2Ql+exb9c2DuzYwo71y1g4/h36V4xi+huP8s5fZ68+HYZ5hrmv3km9unWpVKsJLQeOZGGkk2L1BjIwxO3a173mtKxUGvQmz4f6kLp/Fm/d15H6depSuU4TWr65jKQrFshK5fve4NnGXiTv+J2X+t1OrdoNaHDnED5ccgKjVCfefqkDlwyUdsbx2yP1qVCt1vmf2g8w/kQ2QcLrLX8O6qVynZZ0eX0ORx0mzvh1fD28D80a1qdqSAcGfbuBM/jQptftBFnOl7fq4DcYXr8oUcu+4qHOLaheqyFN+ozg9/0OSvcYzsDKGUZ/m2dZ+FZXGjWoT+W6zWjV90vWZhlrtVJh4Bs8F+pN8u5pjLivEyF161Oj8e10HPwR8y8Ec82zq/hkcA+aNW5I5Rp1qdG0Kw/9GE6yfzt6t/XN8ejrlE3LWX0a/Jq1ok6GpwNFQlrR1NPJvpUrOezIZXo5Lmtu95vKgTkf8VC3NtSsHUKDOx/m3bmHSPdpxosvdCUo20LnrK2M4k24s0lxnFt/oGezZjRqfTsNG4bStOd/WZmUwwoVERGRQkXBaxERERG5pRhFKnDXo09wb93iVw8gmg7OnoriTIoDZ3oCx9b/wgcTtpJu9ad69cBLL6Zzs+61pmWtSOcutbA7dvL9syMYH3aE06kOHKkJnIpJuHIw3lqFu7vXwp4Wwajn32XKlpMkpaUSf2g1o194i99OmPi2vZs7spqmJCeut/w52K8jNY7tU79l0jYHhi2a7at2EJmQRlricVaNHsPC0ybWMuUpawGs1ejWtTq2M4v4z4ujWbovnpT0ZKIipvLOV0tJsFaheWjAv/ky04k7dpSYpDQcKWc4fugkiVlVqLUiXbvVxj0tnK+eepNJYYeJS0kj+cxJdm/azal/YvuGgU+de/lg7B+sWruO8KXjefuuUlixEVwyIOf1kbSWucvjMUq14o6a/wTb3Qm5vQW+5j7mLdh7fpR9btLLaVlzvd9E1v05maW7ozmXlkL8oTX89Orb/O+Yk2JNO9Aq85wuF+s0h21lmpiAEVid0OoBeBiAmcKpA0eJ18TsIiIiNyVNGyIiIiIiN6k0Nn7Ujb5jj+DEwGIvin+parToPZzXHm7Pa1/Gsrvzu6w8l5t9Oji+7yCJZi08Pa82d3Fu1s3h9rZyVClvxXlkNX9daQqGrNjLUbm0BeeRtaw6mGlqkMSNrNiUTP+O5ahcxgKxuc5szvKfJ7s9waHj6VAjkGAfCyRdiBanRnI0ysQIKkoRA3ArQ8XSFixF7mJU2F2MyiJ/pUqXxEJ07tK/2AZhrD6czRQrhhetR/zMj/3L4Xax4O6ULXM+XYslN7dhiayatZSYbj1o3746/w3fhsNelw5tAjB3/srsPY48Ti+Py3EunLURqdx3522UL2WBuCzWseesrYyzq5m6OJp2Xdrw2vhFPB93iK2bN7BsxkTGztuTfQBeRERECi2NvBYRERGRW4CJMzWRUwc3Mu2zV/gqLA1rcBNaVs38ksIc7Ck1lVTTwLBcPRybm3VztL3FDZsFSE+/wnzd2cmz8HGOZVV+EyfgjofHtebHSVpqOhhuuLll3EcaaekmGMb5m5wLo3SzZ+BexD33tWJYzs81bma/d8OvPQ/0LIs1bi1fP9GbZg3rU7lmI5o+OZXI3DccSWFzmBcJFe7qSF0buDfsyJ3BDrbMmst+R96nl7fl+Kf9LdnXdU7byoxm9mv38dD7Y/h18UYOm6UIadeH5z4bw8edA3JeMBERESk0NPJaRERERG4thht2+/lgmsUw4OozX7uOtEiORzuxlG1EaCkLW4/k4iV1qYfYd9SJpVwTWpSzErE/Q/SxWAitGnhA6mH2H3WSf2NcnJw9nYBpuY1qlb0xNsfkX+2nHePgMSdO7+k8cucbLM12TuRcPsC4sF9L2VCalbESkXkUO2AJKEEJOyQtnMCoRTtJPb8hMafO/PvCxUx5sF7pzix5PVNmHGTAkLvoHjIWn653EJS8hi9mHsEBWHOdXs7kvhyXM7ybc0eIHVIPsf9Yxr6Vocw5bisg+QjLJnzGsgmAtTjVe7/L2Lc70PauUJg955rKKSIiIq5LI69FRERE5CZlYHUvgt0CWNwoUtyfcnXa8fAHo3i2gRvO+E2s25te0JnMnfQdLFxyAqd7A57+9EW61QrG0+6Ob7nG9OpQA/uVtnXsZsaM7aS61eHJz96gT91gitrseJdrztBP3qFfSYP4ZTNZFJufwXwH+yN2cMZ0p+VjrzKgQTBFrRasHsUJ9C2St2PDHbuYv3A/zoBuvP3xw9xRswRedisWqwe+pWvRtnG5axvJ49jFvAX7cLjV4+lR7zKoSXl8PaxY3TwpUa0+1fwtOGOiiEqDIk16MbBhKTxtBljc8PT0uCzN1LR0TMOHkLZNKV00u0B6Otv//JPNjpJ0ffAV7r/Tj9NL/mDehZdD5ia93Mj1fg0rxQP8KeZmwWLzpGTdLrz6zdt0D4TYJTNZetrMusw5bStrBW7vfTt1b/PCbjGwutlIP3uWFMC48V8sEBERkRtAI69FRERE5CZlo94zU9nxzOVLzNSjzPrwG5Yk3PhcXZ9k1o3+LzPu+C896g3mqz8HZ1p+pWC8gz0T3uWL1j/yYuO+fDKlL59cXGaSdmwub380n3yNXQOJKyYwYUd7nqzViff/14n3L1mamocppbN1zH8Y2+47hnR4jh87PHfJ0rRNH9NhwDgO5WLw+j/73TbmfX5o/S3DavfgvfE9eO+fRWYiM59qzVMLl/Db4uG06nI7b/5yO29esr2D3Rn+fXTHTuLN6lQf/B3zg16g8dPzyGrgsePwDCYue5RPO3SljeMQP/6ygjMX2sqMyWl6uZPr/RpedBq5mE4jL9kLKYem88ZHC4kzsy9zTtrqsH8THn7nDZq7ZUrXGcvcBeuusZQiIiLiyjTyWkRERERuOo5Te4nYd4KYs8mkOUxMp4OUhFiO7l7PvF8+54k+fXlm5rFrmDe64DlPLeSlgY/z8Z/r2B+TTHp6MjH7w5i+aDtJJjjNK0Rjz23n+yEDGDZqNusPxXIuLZXEqN0sn/whg+55hRnHb0CNpGzlq6GP8p8/1nEgNhmH00F68lmij+1h4/K5LNt7Ls+mEjHPrmPkoAE88+0s1uyJ4kyyA0daItGHwlm2/sg1h8rNhA18ev8gnvp2DusPxpCY6iAtKZYj2zew74wbhhnL3BFDeH7MUrYeP0OKw0F6SiJxUUfZvWUta/aevljGpGVf8Nw3i9gaeZZjR09knyczlvkTZ3PcYZIS/huTtqRcsiyn6eWuoDndr5OYLQuYuXwLe47HkZTqwJF+jtjD4cwb8yb97nmDuSf/7ZdZlTknbWUYJ9j4VziHYs+R7nTiOBfH4fBFjH75YV6cdepaSigiIiIuzvD2DyxEk/yJiIiIyM3o1VdeoWWrlgAMGvxgAeemMDII7DeaFe82Yu0bd/DAlNjCNJO3iMuYOP4nAFauWMmHI0deZW0RERHJVwZTNG2IiIiIiEghYglqSKd6sGfbAY5HnybZ5kvFhnfzwuNNsDv3sSniGkfZioiIiIi4GAWvRUREREQKEff69zLyq854Zn5Bneng+Kzv+GV3YZwMRURERETkcgpei4iIiIgUGgb2+F0sDatI/SplKeHtDimnObE/ghUzxvH1pLVE5foFhCIiIiIirknBaxERERHJM0FBgTRs2IhtW7dy+MiRgs7OTcjkdNiPPDX4x4LOiIiIiIhIvrsseP3qK68URD4Kne07dzB92vSCzkauqX0L3tRpU9m5c1dBZyNXqlevRs8ePQs6GyIikkMF+ZIxm9XG8OFPAHD2bALhEeFsDY8gPCKcw4eP4HRqWLCISH7R/Z6IFFaF8SW53Xt0p2b1GgWdjZtKVvHWy4LX/7zlXa5uOoUveK32LXgrVq2EQha8DggMVN8RESlMCvDaPyYu7uK/ixf3pFnTpjRt0gSr1UrSuXNsjYhgS3g4WyO2sn//fgWzRUTykK7ZRaTQKnyxa2pWr6Hzbj7IHG/VtCEiIiIikmdSkpNJSU3F3W4HwGKxXFxWtEgRGjduTKOGDbFYraSmprFr107CwyPw8fEpqCyLiLgku5sbqWlpBZ0NERGRApVt8DosbB1fff3tjcxLoTBx/E8FnYU8ofa9sUJDG/PU8GEFnY088dXX3xIWtq6gsyEiIpk8NXwYoaGNCzobAJyOjycoKCjLZYZhYFitANjtbtSuVZvatWtjGMbFdaxWKw6H44bkVUTEVfXt149aNWvy62+/sWXLllxtq/s9ESkMXOn69Xq9sqdcQWehUBtZ5VC2yzTyWkRERERyzc3NDW8fHwL8/fD29iHA3x9vHx/8A/xwOHI+FYjTdGIxLMTHx18cfa3AtYgUtJq1ajJs2OPExsZxKvoUsTExxMTEEhMTQ2Ji4g3Jg7+/H/Xq16Ve/Xrs27ePXyZPJmxtmKZbEhGRW4qC1yIiIiJykYeHB4EBgXj7eOPv74+Prw9+vn74+fnh6+uLn68vvr6+eHl7XbJdQkICsbGxxMXFkZSUgOk0MSxGNqlAeno6VquVTRs3MemXX+jdq5fmDBQRl+FId1C+fHkahIQQ4O+P/cJUSAApKSlEnYoiLjaO6OgYYmNjiI2JJSr6FHGxsZw6FU18fPx1P4gLCg4Czp9HK1aowBsjRhAVFcWvv/7GokWLSE9Pv679i4iIFAYKXouIiIjcAjw9PfHz9zv/f7/zwWh/Pz/8/Pzx8/O9+HsxT89LtvsnKH3+J459+/YSExt7/u8xscTGxRJ9KpqkpKSL2wwdMoRy5cpjs1x+qelwOHA6HCxespQ//viD48eP53vZRURya9euXXw48t+3h9ntdvz8/ChRogR+/n7nH+r5++Hv60fNmjXx8/MjKCjoknn+M54/IyMjiYn591waGxtzcVl2AgP/nX7JuLDfoMAghg9/ggEDB/D7778zf958UlJS8qEGREREXIOC1yIiIiI3mSefHI6fny8+Pr74+/vj7e2NzfbvZV9qaurFoEl8fDyHjxwhImIrMTHRxMfFE3NhBPXp06ev6evpsXFxmBl+N00T0zQ5dy6JadNmMGvWTM6cOZsHJRURuTFSU1OJjIwkMjIy23Xc3Nzw8/PF3z8Af39//Pz9CAwIwM/PjzJlylCvfj38/QOwu7ld3CYlOZmo6FPExsRemJYk+uIo7gB//8sTMcDAwM/PlyGPPMKggQOYNm0GM2bMICEhIT+KLiIiUqAUvBYRERG5yZQsWZKY2FiOHTtGdEwsp+NOEx0Tzen484Hp/J6vNTYuFpvVgul0YlgsxETH8NuUKSxcuJDU1NR8TVtEpKCkpaVx8mQUJ09GXXG9K43irlWrVpajuDMzMDAsBsWKeXJv/3vp27cP8+bPy+siiYiIFDgFr0VERERuMq+99nqBph8XG4thWDh46CC//vobK1eu1AvGREQuyMko7tKly/DDD9/laH9WiwWr3c7d3e6++Dd3d/frzqeIiIgrUPBaRERERPLU4cNHeP31EWzevLmgsyIiUih5Z3opbnbSHelYDAsWi4X406fx8fYGuOKobRERkcJEwWsRERERyVMxMTHExMQUdDZERAqtgIAATNPEMIxL/u5Id2CxWjAMg+joaCLCI9i6bRvbd2zn8KHDzJ49C4Bz584VRLZFRETynILXIiIiIuJSnho+rKCzICJSoPz8/AGDdIcDm9WKw+Fg/4EDbNm8hW3btrNjx3bOntWLb0VE5Oan4LWIiIiIuJTQ0MYFnQURkQJVtKgHmzZtYNvW7Wzdto3du3frhbciInJLUvBaRERERERExIVMmvRLQWdBRETEJSh4LSIiIiIF7sORI2FkQedCRERERERciV5BLCJyU7JSrteHzFjwJ6+F6jmlKzJ82/LG5FmsGNke96uu7E3r18fy47DG+BtXW/lGs1Cq27tMWzCTd1q6FXRmciVXbZAnPKjS4x1++ao/FXRYioiIyDXRdf7V3ajrU9dsC2twc4Z9MoGlf29gz/ZNRPz1NQPKKPwnlzLc3XnyLn+mNHfHXtCZuQr1XhEA3Al56lc2rp/DR3f643KxIckgP9rq5mz/4mVqUrOMDx7GzVKim4vhUYqadSoQWNR2lT5nULzF07w3sBEV/S3k32yX13ocGBQrXYNaZXzxKGRd7fI2yO9zQRrpxctQu8PTvHdPGax5vn8RERG5Feg6/2pu3PVpwbVFNtet9lo8/cPXvHB3COX9PLBZ7Xj6W0g9Y2KtMphJq8JYNaoXZRUNvOUZVitV/G342YwL/cegdj0/Zt0TwCtlLS4VF8m37urR9GmmzF7I+nUb2LU9gr0R69i0YjbTf/yI1x9oT3Xva71ls1LrwW+Zt3g8T1TTbd+NUKzbKHbuXM+sF0PxybL3utH6/RXs2z6bV+oW3jYxDAPDsGBxpSP0ZmWtzPA/t7A//DeGV7vSk/BiNBsxl707NzKmR/GLf82PtiqY9i9Gh49XsG/nesbfE5z9CdmtPq8tCGd/xHgeLF34rzKKdRvFzl1bmPl4JRcP3rnI542tCvc/15PbYubw0agwzpr5l5TOg/ldBw4OTB7J6O3uNB02jDu8buGKFhGRQiH76zYb5e7+hOVbI4iY8ixNs75RdBm3yj1tRvlbZhe5Ts5PFi+qd3yUkaN/Y/nfYezavpltf89n1s8f89rAZpS+MV/bu6Ksrlvdm/SjfzU7qXt+48muLalesx517niDeWdMMAwshoHF6trH683OvYQn33QLYEa/IJYMDGbZgCDm9Qngpw7ePFndndIFPIjfIPfB4io1fBjXw5f7fPMjR/k457U1sDJ1Kpf692u41qL4BJXHJ6g8dVt14cHHwhk/4kU+WHSM9Fzt2YJPuZpUKRmHXcfbjWMUodZDn/HVift5ZOK+fBz5V1BS2PBlPxp8WdD5uEVYAggOtGC41+CRZ7vy+7CpRDovX81apT8v9S2D1XDgG+CHlbM48qWtCqr9E1k1Zxmx3XoQ2rUDpaZM5GgW9eAe0oXOpS2krJvLvONZrCD5xDU+b4q1HMx9NQy2f/Uji+LzMXKt8yA3pA7S9zBx9CIe/OIuHun5LYvGHUFHtYiIFC5WSt71Dj//pyP+eyYwdOgXrMnXa5Q8ctPf02Yh38rsGtfJ+cXwqssj//2cl1qXwJahfHa/0tRqVpqa9Yuxa84ajqYUXB6zvm61EFy5Et5GCivGfsnsPfGYQEpUzPnFu8fRv/m4AsirZGQtYqO6t/XfqToMg2IeVip7WKkc7MHdVc/x3qIzLE+60Tkz2bolli5bcrudgXdxN8oXc+bb9CP5PIQvne3f9KVmzdpUrBlCnRad6fn4+/y48hjpPvV44PMfeL1ZcZcail5YBAQE8O6773JH+zsoWrToDUjRxGF60fLlL3i1ubfarBALbdyYF198gdDQUGy2Anqk5x5AsLdBavwZrK2GMiTE4/J1DB/ufOx+6qTEE+808PPzzVG/s7gXJzA4EN+iWZftastvtKS1c1gQ5cTeoAtdymY1asGDJl3bU9KSzNpZi7IM8rsKV6vbm4Lhze292hOQuoEp0/bjKOj8SB4wiV/2B3NO2mjQqytVbtLBSiIicrOyENjmVX7++G5KHJ7Ck0P/y6q4ggtc5+7681a8p70Vy3ydLCXpNfIbXmkTjBGziQnvPU6Xdk2pXjuEuq3u5p5nPuHncTNZdTrv+n3e3cMauHvYMcxzREcnUggeKbmU0NBQXnjxeUIbN74hsZI94TG0n3SS1pNO0mFKNA8tPcusWBO7VxGeqWsniyjJLSvfv3/uTEsh1WFiOlJIiD7E5iWT+c8j/Xhg7E6S3coz6JX7qH7hxs3wb8tr46ayYs06dm8PZ9eGJcz54WV6VSt2+UnWWpWnpodzYNc2Duzaxr4Vb9Dc7Rr2U0gZFgsNG4bw3LPP8r//Teb1Ea/TvHlz7G759TKCdLZMGMXcmHLc9/E79Ch1tbttN1q8/Rf7tk/l2eoZ1zXw7vktu3Zt4uc+fhfn1fFvMZTPRk9i3uLlhG/ZzN7tG9i4aCKfPdSUag178/Jn41m8Koxd2zawccE4Rg6sc9lXn4xilen67GdMXbyaHREb2LjoF756og2l3TKkXf8e3vx8DDMXLCMifAt7I9awZtY7dPKzUuXxKezZsZKPWmWqwyJlaf/4B/xv7jK2RmxiW9gSFkx4mx7lCmfEwe7hTtu2bXnrrTeZPHkyTw4fTu3adbBYbtx0FBa/APwtTqLmfMeEvSXp93g3SmVK3lblXobd6c7f3/7ImhQLvgE+F05YWbeVxa8Rj37xJ+s3/E3Y8r/YsHE9WxZ8Qu8LO77y8qz2mVW/3MzWVTOY+E5/GvhmcTbxKE27Ie8xcdZSwsPD2RMexvrFU/ntu/cYEuqT9fnn3HqmLziB060m3btkMY1GsWb0uCMAI2E10xZFY3I957jcHJcXllz1uLp63V/d9Z4DDPyaPczH341nzqIVF47tMNbNn8TXz3enjk/GfOS+Dq74eZOD+jlfR/UYOOI75ixbw86tG9i4cBKjhrUk+GpVVDSUDs08SY9YypKTmZ5c2EvR8sE3+WnqIjZt3sLerevZvHwW03/6jHd6Vb3Ql3JT3rw8D1oIaD2C+Vsi2DThYepk+Yz1Rp37L+QoR22QdR3k7JjLxTkjeTOLVsVjqdyOOwrpZ4mIiNyKDPxavMhPX9xD+ZMzeO6R/7Dk1KXXJ9d3T2bk+Dr32q4/c3tPm9My5e7eNy/Kn3P5VeYLsrxOtlLn2Zns3bmO77oWz7CyhRIDxrJrxwo+ap1xzg0rdZ6dwd4dK/i4zYVwXZHy3PnER0yZv4JtERuJWD6Vn98eSGhgpvrNti6zKlVOrk+haPPHeKGtH0Qv5bX+D/DmxOVsP36WlLQUzkbtI2zuz7z7+bwrDirKq36c+3vY8/WCxZd+/7f5Yrsc2Pozg0taMPz7MC5iGzu/645Xxi2u87i9WXh4eNCubTveevstJk/+heHDn6B27dr5FisxnZBmgmlCcoqDPceS+GRVInuc4Bdop6wBxQOK8FQrX8Z0D2Re/2D+6h/E1K5e/HOoGG427qjvzQ89AlnUP4hZPfx4u447JTJl2eLhRo/GPvzUK4jFA4KY1d2Pt+vaCcjUfOVr+7F0YCCvlMq0wGalZW1vRt0dyPz+QSzsF8iEDl7cmfEQN2w80CWYFYPO/yzr7UVIHlVdwQyPM0+zZtRHTLnrRwZX6USX6j+wY5sD0n2oFFKV0v+MM/cMpkbbwXxSO4i07i8wMzqHz43yaj+FhNVqpUnjUJo1bUpKSgpr/l7DsuUr2LhxA+npuZuU5Uocx+by6vPFqTD2Qd799CF2Pfh/bE/Oiz1b8KvbgW5tambokG74lmlAz5fH0DPT2vZyjbhnxHf4JfbmsWknz3/dumgdho/5P55pUPziExmPMvXo9uRX1C/1FN1HLCPOtBDUrA/3dc6YTnECg9xIScgma+7VGfLDGF5p4vPvkx57MJXrl8HznAsPgc2hokWL0L7DHXTs1JHTZ87w119/sXLlSrZv256v6Ro+fvhaTOKj1jJ+7HIGfPAgDzWYwfsbLnzvyvDijqEDqB49kwem7qLTIyYefv4UMyA1q8PXUoK+I0fxUhsvjPREYk4mYHj64xPkRspp59WXZznzclb9EooFVKLFva9Tv2oRet03lt3/HGIe1Rky+kdeCfXNMOdYMfxLV8W/dEXsG8cyNiw+i5GzqWyYNot9Ax+japfO1PphN+EXD1sD79ZdaecLcbOms/ifUS036hyXk+PKuFrd5sT1ngMs+NfvSM/bM25vI6B8fboMrUf7uxrzzH1vMi9z8Pd65ei8A4ZXc0aMH8UDVTwuXqy6l61P5/9v777Do6j2P46/ZzeNFJJsEkA6EqRJ7woICHIVRcGGUqzoVRG9oogN0Wvh2n5eRLwiehUUvYqAFBHpUqQI0jsooaf3tmV+fyTEECHZTTYF/Lyeh4ckOztzzsyZM+d858yZ+nk/F/e0oU/TdrQJcnFs67azG8gBzRj54Uc83cXGH1PW+RBasxGtazaiadqPvD57v3dGapdYDxZtkRiEdhrNtP+7ndr7PuK+Rz5hxzkfeauour9sxwBw85zzpM7IYfuWXdhv7kyHtiFwOLmkFIiIiFQyC6GdRvPJpGE0TVrMU/e/yKKTRVoaZe6TmVDNjWtuiW378/O4T+tmW8OT/Vjm/Huo4vPsZN/6jcQ9cCtt2jXDd8Em7ABUo13HFvhaAmnb7lKsP+3Ja6taImnbth6WrJ9Y+2sOBLTgwanTGNs59I9WZs3LuOqOZ7iiZxvGDHuG+SecJezLomlyt30aQLcb+lLDksuvH7/JtzGljKe403Yslz5sKXjjvL0IBQYG0q9fX6699toKjZVgcNYNjoha1RjUwLfQfjewBUKuHfDx5a4+4dwTZRQcO/9gX65uE0aLoGRGrs8hBTD8/BjVN4xbwoyCdfuF+NI7P/Bc4nRCVh+G9A7n/xBEAgAAIABJREFUoZqWP85Jq0GDSCuB3gs5Fqvy3vyVtY2VG1JwWerQvEkQAGbaWt4ccRPdOnUgunlrmne9nnunbSc7ojc39yoyZYBzP5NubE2jpi1p1LQljXv8k3V5NaJn67lIWH2sGIZBQEAAPXp2Z/z4F5j55UweffRRWrRsgeGVN9+apG+ezOP/txlX24f5v390IsSbO9NMZdEz19CmdWuiW3VnwHPfc8xp4krexORRt9CtQ1sua9+PYVM2k0oYVw3uQw0LgJXLRrzAqLaBxK6axL3XXUmzlh3ocsvzzDrspO5NoxgaXahiN9NY8uL1dGzXlujW3ehx67/ZYD9Xgqw0GvoCT3QOJXv/XJ4ffi3tW7eleac+/G3Ev1h8kdwE8fHJu50aWr061183gDffeIPPpn/GPffcTd26dctlm5YwG2GGSVpKKrE/fMqs43W45Z5rCu76WesP4v5+wez44nPWp6eSkmZiCbcRcZ4aywjpwjVdQnDt/JBB3brRsWcfOnToTNdBb7Ems+TPi1WoXDZu2YXuQyey5JSLoDZDGdr+zK1oK42HjWdM5zByDy/gxeF/o22r1kS36kL38avILKGoOPfO59sduVgaXsuNbQvNEmWEc/XA7oSap/l+9lrSziSpQuo4986rMu3bokpdB/zx/e/H9qFly1Y0vrwr3W9/mo9+ScK3wY28MvZqzjVY3i3nvN64W+/4cPl94xgR7Ufq1hk8fksfWl7ejjZ9hvL4tE3EF9u/Mghu2IhaVie/H44pNC+ylSYjJjCmSziOmB957YEb6dS2NY1btKfdze+zzasNCE/rQQuhHR7h4yn3En1kOn9/8D02ppZwApR73V+WY5CfRE/OObfqDJO0337ntMuXhpfWc/NYiIiIVBYD/yZDmTz5flrlrueVB59j7p+Cet7pk7lzzS1b+9OTPq2HefJEGfJfio2VX57PE5fJ3f4zG9MtRHXoyKVnFvdtSdf2gRhYaNSx/R9PvwW1o2tLX+w717Mh3UL08Bf4R6fqZO+Zxdjb8tpt7a4ZyevLT2LUvpYJY/ud3a4vsX/vQfvUWo8WlwVjcf7GT2uOl3ogiDfKcZnKuSuJr+9vW3BcGl1+N9NPnqvRW16xlItD4VjJgGuvK7dYiZE/53XzOoE8dUUQ0RZIjrcTU1BMTdZsSGDgV7H0+jKO2xZlsM0JlzYLYUSUQcLxdMbOj+PqmbHctCiVRSkmtS4N4sawvG83bRHC4DCD9PhMXl4UxzUzY7l2biIv78ol0Y2wVr3LqnN/TQs5yVm8vSSe67+Mpe/Xcdy1NI2fCt8IMx18uvA0PT7P+3fVt6ls8dL4scoLXuMgMTEF07AQGByYlxDDIKzVEF775FvWbtjE9hXTmdC/NlZ8qHlJpPuJ9dZ6LlBWqw+GAUGBgfTt24c333iDz2dM54EHH/DC2nPZP+NZXl6eRvTwV3jOmzcDTCdpcbGk5jhx5iaxe84UvtjlxPCJZ/faPZxKt2PPOMHaqR+zJMXEWq8h9S2AtSk3XN8Mn9SlvPrUVFYcSibHkU3sjjm8NGkF6dYmXNG50HE3HSQdP0ZCph1nTionjpwm41wnrPVSrr/hcvzt25k0ejxfbIwhKcdOdupp9v+6n7gLf+D1n1h98i5MkRERDBo0iA8//A9Tp35I9+7dvbod/9BQAi0u0tMycOVsZfrnv+LXawR3NLECAXQecSdtM5cxbdbvOMkkLd3ECLOd5y3ZgGnmTacR1YzOzSIJMAAzh7jfjpFsuvF5cQqVS5cjneO/zOS1GTtxWCNo1iwqr1xZL+W6AS3xc+7lP/94nukbj5KS68SZm05cQnrJc405Y5g3+xeyLLW5fnBXzjzBZrmkPzdfEYTryPfM2lToqlARdZy751VZ9m1Rpa0DCn0/PTGRTIcLlz2N41sX8PrDE/guDmxX30ivUC/ebXN3/1ib0r9vQyw5m3l3zBt8t+M0mfZcUo9vZf7nP3Kw2BaxhYhIGxZXJgkJmX+UI+tlDBzYAj/Hbt4fNZaPVh0kPsuJy5lDakIyWd68r+ZRPWhg6/oYn374AM2PfcFDI992bw7M8q77y3QMzmTNg3POnToDMBPjSTTzjrGIiEjVZqXJgFvpFgaJ25ax7lxvqfNWn8yda26Z259u9mk9zZMnypL/UqngPGf+wopNmVgbd6FLfpTa2qQLXSNT2bXrOJbLu9A5P4Lu36YbnYKc7PppHXFGEwbe2BI/+w7eG/My32zLa7clH1nH1Cdf5OuTJuG9BnJ14eh1sf17D9unRhAhQQa4kkhMLkOH3xvl2Jv9rPMpr1jKRcjHN2/sc9FYSf369Uu9zsvaRrBqWE1+GlqDH26JZGrvEK63GTjTsnlvew4FEQDTJCXDSZLDxOl0cTrNSabhy9UNfbHmZvP+2gx+TnGR6zJJSMji39tzyLT40KGmFYvhS896PlicuXyyJo0lCS6yXCbp6XaW7cvhSEnHz/Dh6ka++DntfPpTKnNPO0lxmuTkuvgtzuFW8NsbKvGtWj7YbKEYpovMjExMozo9n/+UaXc0wLegHvKnfj0AJxaLm0n11npK0L1Hdxb2WOCVdZWnM3eKwsLDuXHgwIK/h4aGln6lzhPMfvElrmjxf9zy0jOs2vECGWVN6Dm3c5IjJxzQPIqaYRbIzL945J7iWKyJUSOQagbgW49L61qwVOvPexv7896fV0TtupdgId6z7fs0oElDK66jG1kX473XpD0zbhyM89rqyo3VmhfIrlOnDnXq1AHABGzh4WVed0hoCBYzl4yMXMDF0bmfs/jBd7hzeDf+OymCewfW4ug3T7M02QQjg/QMF5YG1al+ntijmbaOOcvi6T3gKp6dvpQxSUfYuXUzq+Z9zic/HCCjpM89qnCdnDj0OxlmS4KD8+csKygr61h5sDS3nl2c/mEWy/7RlQH9BnH1G6uZn2zh0hsG0cnfwa45c9h5ZnBLBdVx+Ll3Xhle3bdFN+FmHVAMM2Udy7bkclPf+jSuYwFvzc7g5v6x+EbRsI4F17Et/HLO0Q4lbCbAD4Nccgs/y+VXn8Z1LXnl7VA5D3XwpB60hNH3/rvAlczyL7/g54RSNvi9XfeX8RiU/Zw7R50BmLm52E3w8y+vd3KLiIh4i4O9M9/ghzp383Cv5/hmen3+MeptVpwu1DZwt21UXJ/MzWtuiW1/d9qf7vRp3c5TohsbdEN5t/MrMs9mCmuX/0p2nw5c1TWMGbNTqHfFFTTK2sDYKUmMnfQ3enaoxtzlubTq0RWbaz/TVx7D6deX6LoWXEc3sPb3Im3PjC2s/jWbO/7WgOh6Ftza7Z62T81MMrIASyhhoRaILUUcwFvluDz7WWd447z1ooULq36MDc6OlRRWw89ObK7n76FzuUyycl2cTLGz7Xg2cw/k8HtJ3TyrlXrBYPEJYMJtAUw4xyI1gi1YLBbqBIEr3c720gTuLFYaVgdXei5b0kpevLxUXvC6Wht6dQnF4jrC3gMZYLuRuwfVx5q0gckvvMEX6w8Rl+VD5NXPMffdgSWvL59h6+uV9ZRk7969zJk712vr81T10Oo88tDDbi3rdDqxWq3ExsZSo0YNAFJSUsq0fTN+Of8c/y0dPryZCc+t4c2scyyDC/AnIKC0ox1d2HMdYPji61t4HXbsDhMM46w7ludn4F/N3/MR4oYlb+5i07u3kubMncvevXu9uk5PNGvWjEE33eTWsk6XC4thcPr0aWrVqoUBJCYllTkNIdVDMMxsMrPz9q2Zuor/zj7C9UPv5knTxlV+W3n9y+15cy+ZWWRmmxgBIYT4AueqxM14Fj47nPRfb+Xarm1o364V7Xs3okOv3jSzDGbUwpI+9yxPZm4uuaaBcWZya4svPhbA4Sj9o2Upq5i58CTXDe3BkOsuYeGsGtw2uBk+meuYOff3gvWWtY5z+7x097xyY9+X/gxysw4oPiOYLjPv/4K/lLVuwv39Y1jznyyylOopldzsXEz88Csc3zTzcoDT5da+LVN+PakHzXS2LNlMZK+e9B7/MW9l3suYBaV53NLLdX8Zj4E32hV/qjPIm3vO14DcnBJnmRMREal0jtj1TP7nQtb+/R3+M2oEH0wP46n7xjP/WP4ICy/0ydy+5nqp/Vlin9aDPHmjfVkRsQxv5rmELZGwZgVbcq6kc++uhH63mR49m2L/5St+WpdEt6Tb6NmzDf6rUund4xI4NI9lvznBz8uTvHraPnUe58BvWZhNG9G1UxRTDpzC06EP3izH3uzDnlN5xVJK6fWJEytoS+fWvHlzbrrxRreWdTqdWCwWMjIyCA4OBvA4cL1/awIjdzo8LmMAmJRYz/lbDYxCfebSPbnxxzzZlTnIvnKC10YoXUeN5dY6Fhz7F7NwjxNLdC1q+UHmkhm8t3Rv/oThdhLiUou8SMnE4XBgEkhg4J9PIUuku+spm/i4eNasXuPFNXomqkYNeOj8nzscdnx8fElNTWXlqpWsXr2GPbv3sGDBfC+lwCR5zTs892VnPr3jKR4/HYhBaqHPXaSlpGNa6tA0OhRja0L5FXT7cX4/7sIV+h33X/MCK847/5OH85Hlr9dSvzPd6lnZUfTObynt3bu3UstOSewOB74+Ppw8cZLlK1awcuVKLm18ad6IcS+pXj0Yw8whs2B+Azs7v/qSTcOf5a7bTZIWPcXcY2eq8Fyysk1MI4iQYAPOd3yzj7JqxjusmgFYQ2h288t8MqEfvfp3JnDh92QU+/nismXIfooT8S4s9TvSubaFnUdLc/nJZtP/5rB3yCN0vuMWuibVYXB9g4R5X/N97B9nj+d1nBVrQU3vwXnp9nlFyfvewz3hVdVa0flyP8jNyw/gQd1UzPXG3f1jbcGhYy4sDa+gV+P32bHfk5HSLhLiE3FZLiMiIhCDlLy02o/x23EXlnrt6VDLws7jxZW3MtbFntSDpp2D3zzBg18/zvT3hjHwlXc5efpe3tiUVj71f4Ucg/JrVxi2SGxG3jEWERG5ILiS+WXKQ9x+6hU+fnkg70z3w+fuccyJcXilT+bRNdcr7c8S+rQe5Mkbfd+ytfPd5a08+xQblwFwnV7Ogl+epFvXflzVOIR+rU02vbqWpMxMlq5N5ear+tBpbip9G5rsm/wj+51A7pG8dluDLlzZwMqOw4XankHt6dEuAHJjOHzsXC8NP1d2PW2fZvLzsvWk9r+ariNHc82S5/nBrflC/zgWXi3H5dmHhfKLpZRSZcdJLIYFioldnytWcteIEXTv4d1pVt3icnI8A1x+WYz7LpWfz/feI8OXmHSwVPeja6jBXk/nnMnfjiXYj/YhsC/1XAuZOEwTMKhWTlHmcp/+2eLji9UArH4ERTagTe8hPDfta/57X3MC7L8zc+J09jjBlRBLrB2qdRnM0A61CfYxwOJLcHBAkQi7SeypOEzrJfS7tR+Ngn2wBtho3LElta2erOfi43Tkldbs7GzWrF7LSy/9k2HDhvPhf6aye9duTC+PIMZMY927LzPzaCi1awcUuRvn5PCOPaSa/nT/+zPc2a4mgVYL1oAQosKreffOnXMfi5ccxhV5AxPeuI+rW9Siup8VizWA8Lot6dWpQemOvXMfP/x4CKdvGx5772WGdWlIeIAVq28wtZq2pen53h54ATpTdpKTk/n+++95auxY7h85kpkzZ3LixAmvby8wOBCDLLJz/iiTrhMLmbE8GZfzBN/NXFHoDdYusjOzMS0hVA8+zz63NqLPzX1oXac6fhYDq68PjrQ0cgDDAKOkz8uaIcceliw/icu/HY+9/RQ3tKxJsJ8/4Q06Mbhfc9ydFMB5cDZfrM/CGj2Ed5+/BpsZw5yv1lD46RxP6rhcuwPTCKN9r67UDbTi0Xnp7nlV3vvWE0YQnW+5k95NIqjm40v1ep2467WXuKOuhYwNS1idYnq2D4q73uDm/nHuY/6CPdh9WvDI5H8xskdjbAF5y4VGhnOetn7B9tN//53TTisNL63/xwXbeYAly37H5d+Bx99+kutbRBHo40tI7TYMHHYNTc5qW5axLva0HjSdxK9+g7uemM0R3+aMfPMF/hZVTnWlu2W0TMegvNoVBiENG1LTYuf3w8dKvRYREZGKl8Oh2c8w4plFnK71NyZOfY4+EYZX+mRuX3O92f4srk/rdp680/ctWzu/ovNcfFwmb5E4li7cSFZwd+6dcCud2MwPKxMwyWTdD6tJqdmXf4wbQGNzLwt+OJw3Gtq5n3nzdpPr24pH33mBW1rXJNDHj9AGV/DAmy9x2yUGyavms9STiXY9ap+aJP3wAdN25WCpPZB3v5rCU4M60TgyEB+LFb+QGlzW5Xr+/o9BNM/PZ9Fj4bVyXBH9rPKKpVxECmIlSUkVEitxm2ln1VEHrmoBPH5lEFfarARbwWIYhAb70rWGNe/YmXaW/m7HYfFl+FXVGVLbh7D85UKqWQhwYzsrYxw4rb7c07M6N9W0EmoFi8UgKtyXS/NXkJDpwmVY6R4dQD1fsFotNKjhS00vBQTKuRz60GLUt+wbVfTvJs7kHXz2/BheXZead8crYTlfLxtFjwF9GD+zD+PPWt7J/kI/x6xazu7RrWg9+C2WD87/s30rr103nI+Ouruei8OZgLTD4WD9+g2sWLGCLVu2YLdXzCtfzbSNvPPqHPr85xaKvms1Y/UMZuzpy6Mtr+WVr67llbM+9eZj0g52fvwqn/T+gJH9nmBavyfO+tT+6xv0u/Mzjng8GNbBro9f4cOeU3j48pv45/Sb+OeZj8wM5o/uyegfs4tbQZV2ZjqZ9IwMVq1YycpVq9izZ4/3b3KcQ2BQtfyR14X+aKaw6InuNH6i6NImWdk5mEYQ1YPPvT4jogv3vfQCVxR9SseVyKIfN5EZcXWxn5d9ZHA2m6a+xbyr3+KmNiOYNHtEkc/Pdxu0aHpOM3/GYh67YhA1I02yf/mKz7edfa6YHtSVx/bsJdlsRrMRH7C4xpN0euwHD85L986rmBL2fYWOujb8aPi3sXzyt7FnJyV5PRPfWsCZAezu74PirzfT3Kp3nOz/bAJvXfER4zr359lp/Xm2SLKLG73r2L+FrRnD6d+mNTUtOzjhArCz4+N/8UXfSQxvdxfvzbnrT98rvM6y1cXu1INFrzcu4pa/xsPvNuTrMdfyyj/Xs+Ph2Rzz+ktu3a37y3YM3D/nPOFPq/Yt8HUeYsu2cw5fEBERqcIcxMx/gQdq1uDLMbfwzjuHufX+GWXuk7l7zS2p7e9p+/P8fVr3+5ne6PuWtZ1f3IOS3s9zCXGZGBdgkrB0DsvG9mBgh2ZkrhrPsvi8BnnG+h9YlnQ9t7YzyFr/GfMKnu5zcmDGy7zbcxpPdbqVN7+5lTf/SDX244uY8K/FpXhJnAftU/tePhj9NDWmvMrQ5j14eGIP/jRhq2MXlu/msefwOY7F494px+Xfh4Xyi6Vc2JxOF1arhfT0dFauWMmqVavYs3dvhcRKPLF/Vxrf1AljSL1gJtY7O1hij0tj+I+ZHDfht72pfHRJOH+vGcAjfQJ4pMh6SqqhDuxO48vaYQyLqMaYftUYU/CJybKf4pgQY3L8eA4HWvvSvHEoMxvnv2PPZef9+Yl85YW5sstt6Kgz7iA7Dp0kIS0bu9PEZc8iNf4oO9ct4tM3n2Bg/6G8tOT4HyEdM5FFz49kzMcr2HkilRynE0dOBkmxx9i/bQPrD6YUPNbhPPAZo5/6LysOxJPpdOLITODwrweJMwyP1nOhczqdbNm8hbfeepshQ+5g4sSJbNiwocIC13lMUlb/m9e+j/3zPD05O5n0wIO8+u0mfkvMxuly4shOI/74Abb8tIhVB7O8dizMtE1MHHYnj09ZwPoDsaRmO3HaM4g/sp1VvxwtdajcTN/M23cNY/SU7/nl9wQycp3YMxM5unszh1J9K3ZUqRdlZWWzcuUqxo8fz5133MmUDz5g9+5yGJ1/HkGBVgwzi8wcd7ZnkpWVBUYw1UPOXWUZxkm2rNzOkcQsHC4XzqwkYrYvZerT9/HUgjgo4XNv5NoVt4SxQx/ijdmbOJyQjcORTcLhjXy3dDeZJrhM96746Wtm8s0hB6YrmaUz5vGnGUg8qOMyV73LE+8vZeepNI4fO5l3HnhwXrpzXpW07yu0vjUz2Pb9bFYfiCPT4SA75RjbFk/l0TtG8d8DhepFD/ZBcdcbt+udrD18NPJ27nlrFmv2nSY1x4nTkU1a/FF2b1zKt6sOn3MqdwAyNrJsfTo+rXrTu8Yf5d9MWcvLw+9jwpfr2B+bTq4jh5TjO/jh21X8VnRmjzLWxaWrB7PZ88nzvLEug7CrnuCFG2qWS4OjQo5BebQrAtrQ98pwzEOrWO6lKalEREQqVja7PxnLi0sSCOnyOO881BL/svbJ3Lzmer/9ef4+rdttDW/0fcvazq/gPBcblzmzrtTVfLXwJE4zndXzVxBfkIENfLckFqcrjZVfL8ofoJEvazf/GXknD7+3kF+OJJJlzyUjdj8/ffk6w24fx7wTpW07ud8+dZ5Yyvghg7nrlRn8sOU3YlOzcTqdZKWe5tC21cz66EvWJuUl+k/HwkvluCL6sFB+sZQLVV6sZCXjx4/njjvu5IP//IfdFTTIz1OmPZcPfkzk5R3Z/JrsIt0JTpdJYpqdDbHOP/o3DgdfLk/kqS1ZbEpyku7Me0lkRpaTA6dzWHTcUexwO9Oey0dLE3lpRzbbU11kOsHucHEyMZcjuXlPAbiSM3l5bQY/J7vINsHhcBET5/DW62wxQiOizjoCZ97uuXHjJiZNnuKlzVw8Pp/+XyBvLp7KnEzez9eXgGoBpKZ6dgtDx7dydO7cidGj8u7Xvj5xYqXO5RQSEkJOdja5Htzk6N6je8Gc15MmT2Hjxk3llbyLjEHUbVNZ/XJHNrxwNXd/k3jR3DyrWqw0eegrvh99CbMf6M3TqyvyBl75C+7zKsvfH8DJd29m8IeHin0BouWSO/liyXO0Wz6GtqN/4MJ9NuRiZhDW/18sfbcvv71xE0P+G1PqF70WNXrUw3Tu3AmAAQOu99JaRUTkQqL+nohcSKpS+7U0sZJnxo0rmPN63IEG5ZW0v4SJTY4A54i3Gnxz8Uza+xeTa7d7HLgWAUhLS/OoMhb3WGp0YEC/DlxW20awnxWfwEgu63EPrz7UBT/X7/y64+J56kMqVvpPn/H5Xmg57H6uDrtQn/eQAj5NGDayL2GJPzJt9lGvBa5FRERERKT0FCupuv7qc6+LiHiFf9shTJx0HcFFY4umkxMLPmDmfoWopJQcB/j07TncMvVmxo2ay8+vbiBNd0IuUFYaDXmakS3tbHh1CktTdCBFRERERESKo+C1iEiZGfgl72PFxktp26Q+tUL9ISeFk4d3sHreZ0z+YgOxf7GXXIg3maSu/TcvfNGA4Ukm/gYKXl+wfPHNOMbupUt54SvvTRciIiIiIiJysVLwWkSkzExSNk5j9IhplZ2QvygnBz64lSYfVHY6ypGZzKpX72VVCYu5Ts7kjstnVkiSpDSy2T/nRe6YU9npEBERERERuTAoeC0iIiIifwnNmjVl0E2DKjsZIlJGc+bOYe/efZWdDBEREakACl6LiIiIyF9CZFRUwRvhReTCtXrtGrjIg9edO3fGZrOxe/dujh49imlqzjAREflrUvBaREREREREpAqpVbMmD/79QQAyMjPZtWsXO7ZvZ8+ePRw4cBCHw1HJKRQREakYCl6LiIiIyF/OpMlT2LhxU2UnQ0Tc1LlzJ0aPeriyk1Fh4hMSCn4OCgykU8eOtG/fHh+rFYfDwYEDB9i2bRu7d+9hz549ZGZmVmJqRUREyo+C1yIiIiIiIiJVSEJC/Fm/G4aBj9UKgI+PD82bN+eyJk2w+vhgmibHjh9jy5ZfKyOpIiIi5UrBaxEREREREZEqJCEhscRlrD553XnDMKhXtx716tYr+MxmCy+3tImIiFQkBa9FRERELjIDrruO+IQEUlKSiY9PICUlBbvdXtnJEhGRIgIDA4mMjMRms2GLsFEjMopwm40aNaIwTRPDMEpch2m6AIPEhAQiIiMBSExMKueUi4iIVAwFr0VEREQuMiPuGkFwcPBZf0tNSSUpKYnEpKS8/xMTSUxKJDkxiYTERJKTk4mPjyc7O7uSUu253r16k5aexubNmzFNs7KTIyJSwGKxEB4eTmRkRF4wOjIKW4SNiIhIIiJshNts1IyKwj8goOA7uXY7CQnxJCYkEh8fT2ZGBkFF6vLCTJcLDIPjx0/wv6+/ZtXKVcyb911FZE9ERKTCKHgtIiIicpG5/fYh+Pr4ElI9JG80ny2C4JAgbOF5I/siwm20aNEcm81GVFQU1vx5VCEveJKelpYX3M7/l5Bw5uckEhMTSMwPdrtcrkrMJbTv2J4+vXtz9Ngxvv7f//jpp9U4HI5KTZOIXPz8/PwKRkrn1bE2Imw2atWsVfC3onVrenr6WXXqgQMHSMj//dTJU+esV9+fPPmcwWun04nVauVIzFG+nf0tK1esrPT6WEREpLwoeC0iIiJyEbI77AWBEjhY7LLBwcF5AZf84HZwcDAR+UHvWrVq0aJFCyIiIggKCjrre4WDMWcC2wmFgt7p6enEno4tt9HcUfmPx9etU5snnniCe+65h1nffsviHxZfUCPIRaTqKFwf1qpVKy8wXShIXatWrbOebLHb7aQVuuEXExPDhg0bSUxKJDEh7wmX0taDp2NjadCwIWdmDnE4HfhYfTh48CBfffU1Gzdu8Fa2RUREqiwFr0VERET+4tLT00lPTyfmSEyxy51rtGFwUHBBYKd+/XrYbDbCwsKwWCwF38u120lMyBuxnZiQSEKhoM6Z/9PT0vMD7e6LzA9eG0betmw2GyPvu4+7Rozgh8WLmfXNLI/XKSIXJz9fX2wRER6Nlj5Td506dYrExCRiYmJKHC3tTfHxcZimC5fTxGK1sHH9Rr763/84dOjF1L3HAAAQ1UlEQVRQuWxPRESkKlLwWkRERETckpuby6lTpzh16lSxy5U0ZUmT6GhsnT2fsiQ9I43EhERiY2NxuVyEhYX/aduGxYK/vz/XX3cd1w+4jp9+Ws1XX/2PY8eOeX1/iEjVUNJo6TP/znBntHRcbBxZWVmVmCtISEjAZZqsWL6cb2bN4vjx45WaHhERkcpw3uB1dHQ0o0c9XJFpkQqk41uxwsP/3Lm+UF3b/xq6du5U2ckQEZEioqOjKzsJBdydssRisRAaGkp4eDgRNhth4WFEREQSGhZKZEQE9evXp23btthsNvz8/Aq+53A4SElJpVq1gPOu2+qT18zt0aMHvXr1YvMvm9m9Z7fX8igilefmQYMZdNMgoqIiCQsLO+smWGpKKolJScTFx5GUkMiBAwfzAtKJCcTFx5OUmEhySkolpt59P/+8nmXLlhMfH+/xd9XfE5ELQVVqv5bV0FpxlZ2Ei9Z5g9c2WzidFaC6aOn4Smk1aXLxXFxERKRyuVwukpKSSEpK4vDhw8UuGxQURITNRmhYGJERkdSpV4c7hgwpcRs++UHsdh3a0bFTR0zA8EbiRaTSOE0n+/fuZ82aeBITE4mPj8+bkighnly7vbKT5zUxMcVP5VQc9fdERCpWq5DMyk7CRUvThoiIiIhIlZeRkUFGRgYcPQpAs2ZN3QpeA7hcJqaZ9/OZwHWD+vXZuHFTOaRURMrb3LnfsWb1mspOhoiIiFSAPwWvBwy4vjLSIRVEx1dKY83qNQxYrbIjIiJVh80Wcd7PXE4XpgFWi4WcnFz27d3D5i2/4uPrw/BhwwA4UoYRjSIiVZX6eyIiFef1iRNhYmWn4uKnkdciIiIicsEJDwvD5XJhsVhwuVxA3hza6enpbN+2ne07trN9xw5ijsRg5g+77t6je2Um+QJkofYNE5jyaDu2vTyYF9dUnekYjPBePD/lSa458i59xy0lp7ITVAVYa17Bg08+xK3dW1A31EJ2wu8sfG0kzyyKx6zsxImIiIiUkoLXIiIiInLBCbeFY7FYSE5OZuvWrezYsZNdO3dx9NjRyk7aRcQgqG5zWtYLZ1+5TxTuT/vR05k2IoQlzw5n3I8JxQZcjYDatGjViKg4n/ypYDz7/kXHryWPfTiZUc39C6bGCY6qhX9O+l9rP4iIiMhFR8FrEREREbngrFv3M0uXLuPUqVOVnJIAGvQexiMjBtDj8vpEBhrkpMTx296trFk0k6mztpFUZaOHVlre8x5vDwtm/sP38P4+p1fXHtD1MWa8cD2NatgICfLDauaSmRxHzIHtrFk8i8++3cjJ3D+WNwwDw7BgKWWg/M/f9yR/IQz+YBVv9/EvYSt2Nr06gCHTj+MqXTLLhX+X27ijqR+5B77myX9MYsnhdPyiahOcrjHpIiIicmFT8FpERERELjiHDx+u7CQAVhrc+g6zXupJpPWPiKtPRF0uv7IOTXy2Mv3bbVTdoa8Wwhq0oMklSfiVw8hqa1Q0raJr80c4OICQyHq0jKxHy27XcufgaYy8bxIbUk0gh83/vo12/y7t1s71/fLNX9VhoWZ0Y0KNHFZ/8m8WHkjGBHJOHSGtspMmIiIiUkYKXouIiIiIlIZPW+56uDsRxLL87fFMnP0rR5Lt+Nnq0bJTT1pnreR0VRqeWykc7J48hJun7CXHtFIttBbR7a9h5JOPMKDVvbx8zxKu+/duvDvmuzTSmP1Qe2YX/O5D+7Hz+fqeEL59oDdPr64a831b/EOICAvAkZZEUqYj/68G/gF+GGYW8fEZVfdeiYiIiEgpKHgtIiIiIlIagXVpEGHFFbOASR+v4UB+BDY39hAbFh5iQ8GCBhFXjuS5u66iReN61I6sTqCvk9QTe1g5czIfbqvDTUNv5JouzagbZiXj+E5+/OwtJs7cQXLhSGS1hlxz70OMHHgFLWoH4Uo6wubls5jy/ldsjCsS/vVkWetljP5uO6Pzf3XFfsXwPv9k3Zl4rRFI10ensfjVptSPCMCZHMPWpV/y9rtf8asbc6K4HLnYnSYmDjKTjrF92SeMSYmi9fQRNOrUnpqW3ZxwWWny0Fd8P/oSZhcJFltsbbjj4b8ztF87Lo3wJfPkXn5en0JNy1mZOO/3S8yfxwxC297GY3ddQ6eW0TSsFUY1I4v4I4t5acQEfjCu4pl3HuPapnWpWd0fMyueQ78sZto7k5mz70xw+VxlArKTzr1vLbaOjBz/LA/2vYxwXwPTtJMWs4SX736ab0/krQ9LOLd9tJXbznzJvokX+93L9JMuN8tDcfl6iY3N7y97GRYRERHxkILXIiIiIiKlkXWSY0lOLPWuZvi137J/wRGyzrmgBVvrftxwVYtCjW9fwuu1Y9DTHzOoyNJ+DTpy+/MfYMu4mb/PPZ03t3JACx6cOo2xnUMpiNnWvIyr7niGK3q2YcywZ5h/Ij8I6cmy7jD8qd+m4x+/RzbmyiHP0fayagwe/gn7Hef/6vm4HM68fFksWIpZzqh+Bc9Pf4+7mwQUvIjQv35brquf93PlzOhsoUa3Wxh+XeHjGUJUDV9y0k2oFkbj9pdR1y//o+CaNO81gjcvr4H9xieZH29y7jIBQefat5Za3DrxPcZeVR3DkUHC6XSM4AjCaviSk+ICrMUn1+3yUFy+jFKX4QfmnvZk54qIiIicpbi2ooiIiIiInI99M59MXkO80YCb35rD8i9e5u/9m2E73/AQM5VFz1xDm9atiW7VnQHPfc8xp4kreROTR91Ctw5tuax9P4ZN2UwqYVw1uA81LABWooe/wD86VSd7zyzG3taHlpe3o901I3l9+UmM2tcyYWw/wg1Pl83n3M+kG1vTqGlLGjVtSeMeRUYlm+msfP02unfuQJPLu9B96ESWnHIR1GYoQ9v7ur27DKsfQba6tOxxB6+Mv5X6Vicxm7dw6rxTq/hw+X3jGBHtR+rWGTx+S15e2vQZyuPTNhHv7pQsJeWvtMw0lrx4PR3btSW6dTd63PpvNtjBTFvLmyNuolunDkQ3b03zrtdz77TtZEf05uZe4Zw1/XahMtG45bn3rRHShWu6hODa+SGDunWjY88+dOjQma6D3mJNZqF1uZL4+v62BflsdPndTD9peF4ezpOvoul1twyLiIiIlIWC1yIiIiIipeLkyDePM/jvk5i3O4OIDjfz9KRZrFnyKa8O70TNokFs00laXCypOU6cuUnsnjOFL3Y5MXzi2b12D6fS7dgzTrB26scsSTGx1mtIfQtgbcLAG1viZ9/Be2Ne5pttp8m055J8ZB1Tn3yRr0+ahPcayNXhhmfLusu0E3toP8dTsnHY0zn+y0xem7EThzWCZs2iSuhQ+HD54/M4uG8Xh3f/ys6fF7Ng2vPc3jKIrD2f8+LHuzjvwG1rU/r3bYglZzPvjnmD73bk5SX1+Fbmf/4jByt7omzTQdLxYyRk2nHmpHLiyGkyTMAwCGs1hNc++Za1GzaxfcV0JvSvjRUfal4Sefb+KlQmXI7z7FvTxASMqGZ0bhZJgAGYOcT9dqzkKTlKUx7Ol68i6XW3DIuIiIiUhYLXIiIiIiKllsuxn6by2OC+9Bz6PB8sOUhuzY7c+dzHzHv/NhoXN0mf8yRHTjjAP4qaYYWa5bmnOBZrYlQLpJoB+DUguq4F19ENrP29SMQ2Ywurf83OW6aexbNlS83JiUO/k2EaBAcH4XYY3DQxTRNcqfwybTQ33Pkma4uLvvrWpmEdC65jW/jl5AXy5kujOj2f/5Tpzw6hd6uG1Kzuj281G/XrReJvgMVS0qyNf963Zto65iyLx6h5Fc9OX8rWdQuY9cFLPHptE4JK2vnlXR7cKMMiIiIiZaHgtYiIiIhImeVwavMc3hg1mKtumcC8GBdRPR/j0T7BxXzHhT3XAYYvvr6Fo5B27A4TDCO/se7BKGmPli09MzeXXNPAsJS0PQc73x1IdNOWNGrWnv6vrSeZYBo3qwE5JQwbNqx5+TcsFZSrsjNsfbl7UH2sSRuY/MjNdOvQlugWHen66BxOuTlS/E/71oxn4bPDufeVj/nfsi3EmLVp3/sWnnjnY964LrKEfVPee67kMiwiIiJSFgpei4iIiIh4jYuU3XN49+s9OC3BREdfUtLr9EqWe4RDx1xY6nXhygZF1hbUnh7tAiA3hsPHXJ4ti4nD4cAkkMDAiggy5nLg82d57vs4ql85hrfub4Z/sYvH5OWl/hX0auz+3Np/qOj8gSWyFrX8IHPNDN5bupdT6XacziwS4lLL9nLJ7KOsmvEO4x65i2t6XMV143/kpGmjV//OFDu22aPyICIiIlL1KHgtIiIiIlIafu144LUnGdHncuqHB2A1DKzVbDTqNIiHb7oMq+kgLjaRMocFnfuZN283ub6tePSdF7ildU0CffwIbXAFD7z5ErddYpC8aj5LE03PlsUk9lQcpvUS+t3aj0bBPlgDbDTu2JLaZY64n4crlkUvv8g3x/1p9/BLjGzuV0y+9zF/wR7sPi14ZPK/GNmjMbYAKxZrAKGR4ZQcj674/LkSYom1Q7UugxnaoTbBPgZYfAkODqCkCUPOy9qIPjf3oXWd6vhZDKy+PjjS0sghb2BzsbvBo/IgIiIiUvWUug0lIiIiIvJX5tPiau686R4a3HzPOT41yTownY8WJ2KWebyIkwMzXubdntN4qtOtvPnNrbxZaDv244uY8K/F5MUfPVs2ZtVydo9uRevBb7F8cP5i9q28dt1wPoopY7LPw0xZw7/++R09pgziofFDWTzsvxw455QaTvZ/NoG3rviIcZ378+y0/jxbZIniRzOXlD/vjzY2E5bz9bJR9BjQh/Ez+zC+SHr2l2KdRkQX7nvpBa4oOvjclciiHzeRUey3PSkPIiIiIlWPRl6LiIiIiJSC6/A8/vXOFyzauJ8TKdk4XS4cWckc37eeuVPGccudb7EuzUtRwazd/GfknTz83kJ+OZJIlj2XjNj9/PTl6wy7fRzzTjhLtazzwGeMfuq/rDgQT6bTiSMzgcO/HiSuXOcqNkn+6T3eXpFMQNv7eeK6iPOPHs7aw0cjb+eet2axZt9pUnOcOB3ZpMUfZffGpXy76jD2YrZU4fkzE1n0/EjGfLyCnSdSyXE6ceRkkBR7jP3bNrD+YAqelgjDOMmWlds5kpiFw+XCmZVEzPalTH36Pp5aEFfy+jwpOyIiIiJVjBEaEaX77CIiIiJy0eveozvPjBsHwKTJU9i4cVMlp0hE3NW5cydGj3oYgNcnTmTN6jWVnCIREREpdwbfaOS1iIiIiIiIiIiIiFQ5Cl6LiIiIiIiIiIiISJWj4LWIiIiIiIiIiIiIVDkKXouIiIiIiIiIiIhIlaPgtYiIiIiIiIiIiIhUOQpei4iIiIiIiIiIiEiVo+C1iIiIiIiIiIiIiFQ5Cl6LiIiIiIiIiIiISJWj4LWIiIiIiIiIiIiIVDkKXouIiIiIiIiIiIhIlaPgtYiIiIiIiIiIiIhUOQpei4iIiIiIiIiIiEiVo+C1iIiIiIiIiIiIiFQ5Cl6LiIiIiIiIiIiISJWj4LWIiIiIiIiIiIiIVDkKXouIiIiIiIiIiIhIleNT2QkQEREREalo1/a/hq6dO1V2MkTETeHh4ZWdBBEREakECl6LiIiIyF9OkybRlZ0EEREREREpgaYNEREREREREREREZEqxwiNiDIrOxEiIiIiIiIiIiIiIgUMvtHIaxERERERERERERGpchS8FhEREREREREREZEqR8FrEREREREREREREaly/h/qnAAQB/Xf8QAAAABJRU5ErkJggg== ) ## Inspect tasks ``` pni ``` ``` Missing Values Imputed (quick median) (PNI2) Input Summary: Numeric Data Output Method: TaskOutputMethod.TRANSFORM ``` ``` rdt ``` ``` Smooth Ridit Transform (RDT5) Input Summary: Missing Values Imputed (quick median) (PNI2) Output Method: TaskOutputMethod.TRANSFORM ``` ``` binning ``` ``` Binning of numerical variables (BINNING) Input Summary: Missing Values Imputed (quick median) (PNI2) Output Method: TaskOutputMethod.TRANSFORM ``` ``` keras ``` ``` Keras Neural Network Classifier (KERASC) Input Summary: Smooth Ridit Transform (RDT5) | Binning of numerical variables (BINNING) Output Method: TaskOutputMethod.PREDICT Task Parameters: learning_rate (learning_rate) = 0.123 ``` ``` keras.task_parameters.learning_rate ``` ``` 0.123 ``` ``` keras.task_parameters.batch_size = 32 ``` ``` keras ``` ``` Keras Neural Network Classifier (KERASC) Input Summary: Smooth Ridit Transform (RDT5) | Binning of numerical variables (BINNING) Output Method: TaskOutputMethod.PREDICT Task Parameters: batch_size (batch_size) = 32 learning_rate (learning_rate) = 0.123 ``` ``` keras_blueprint ``` ``` Name: 'A blueprint I made with the Python API' Input Data: Numeric Tasks: Missing Values Imputed (quick median) | Smooth Ridit Transform | Binning of numerical variables | Keras Neural Network Classifier ``` ## Validation Intentionally provide the wrong input data type to test validation. ``` pni = w.Tasks.PNI2(w.TaskInputs.CAT) rdt = w.Tasks.RDT5(pni) binning = w.Tasks.BINNING(pni) keras = w.Tasks.KERASC(rdt, binning) keras.set_task_parameters_by_name(learning_rate=0.123) invalid_keras_blueprint = w.BlueprintGraph(keras) ``` ``` invalid_keras_blueprint.save('A blueprint with warnings (PythonAPI)', user_blueprint_id=user_blueprint_id).show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABs8AAADECAYAAADQ87/fAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3hUxQLG4d/Z3RRSSIXQew81QOhVAWkqXUBQVFCxN1TsvVcUFNGrVBWki3SlSgu99xogBNJDssnuuX8QFSFl0wzi9z4P97lmz56dmTN7splvZ8bwCyphIiIiIiIiIiIiIiIiIvJfZzDdUtRlEBEREREREREREREREblWKDwTERERERERERERERERyaDwTERERERERERERERERCSDwjMRERERERERERERERGRDArPRERERERERERERERERDIoPBMRERERERERERERERHJoPBMREREREREREREREREJIPCMxEREREREREREREREZEMCs9EREREREREREREREREMig8ExEREREREREREREREcmg8ExERERERESuM1bK9vmMnyeMpHmwtagLIwXKwLfOAD6cPYmHQz2KujAiIiIicp1SeCYiIiIi1yzDrz3P/7CQ9Vtm82gt1wfAjZI38fr0RWzcMo0RFa7Pj7yejR9l/vYjRO7+meda+hR1cQpUfutm+LXj2WkLWBcxK1f95r/oeu1H1oq3M+advrTseQ93NPXHKOoCZUF9NS88qXnrvQxq1ZnRnz9GmGdRl0dERERErkfX50iCiIiIiBS54t3GciAqitizG3mruXveTlKsIk3bhFGzlC9uuRj9tnhXoVnrRlQP8cF6rY6a55dhwWIxMAwrViP/lbRW7suH0xexacvn3OpVAOXLjyzq5nIZi1Wiebsm1CpTPFf95j+pgPvRNcESQt9Xn6W9v0nkj88yeuF5TABbTYaOmc7SlevZve8AkZGRXIg6TdTx/ezdsIQ5E97k0V6NKOmW/yL8F/pqru7x+Wl79/Z8sucMsdEnmXF7EAYX2fTB44zZlYZH6EjevbcWtkKpoYiIiIj8lyk8ExEREZGCZ5Sk55CbCLYA1vL0ub0DvkVdputMyqYP6Va3EqVr38SraxLyfT5LiXp0aNuIaiW9KOr5L1nV7Voq4/WioPvRtcAz/AFGdQmEuGW8+eYSYsyMByylaXxjG5rUqUyZID+83G1YLFbcvfwpVaUB7W69h5fH/8KmpR9zW7X8LQd43ffV3N7jC7rtU7fyySvTOOH0JGzkk9wS/C9LHkVERETkmqfwTEREREQKnLVKX+5o552xVJqFEt2H0LOkBjdFpJAZQdx8321UtqVzZMqn/HTamclBdlY8E0apUqUIKFGKEhVqUq9dX0Z+sJBDKQbFQwcyZtqrdPDTPSsreb/HF1zbJ674nC82pmAE3sT9g6pdnyGliIiIiBQZhWciIiIiUsDcCbv9dsLcTWKXTmTGcQeGT3vu7FdZg5siUqgsZW9hyI1+GGm7mDYlgpQsjktLuUhquhPTdJKWHMOJXSuZ+tYddLlrKkcdBm6VBvFEv7L6gzlT+bvHF1jbO48xfdJvJOJOw0G30bAAltsUEREREfmD/hYQERERkYLl0447+1fB5jjD7C9f5aNpe0jDnbDbB9Mwj1ufYXhRvdtjfPrDMnYcOE7UqUMc2PgLk169g+YlcrfbjVv4K2w5G0Xsqf/R56q9iAz8b5vM2egootc+R6PMTu1RjnYj3uL7JZs4fPwkZ4/uYcviibw9rBkhVx7vWYPuD73EmG9nsmLDdo4cP8n5syc5tX8Lv8/5gteGtaZsZm3iVpaWgx/nna9+ZNnaLRw+doJzZ05yYs9GVv3wEI3dwCh1B3Mio4g9vYynal82ZF2sFr2efIMvJs9m5YZtHD56gnNRZ4g+dYDdK2fw+aNdqFIsmwby6M7Xx6OIjf7r39npQ8l6UomF8iPmEHUuigu73qPDVfUpRrdx+4mJPsncu0pf9QeILewFNp6JIiZyJsPLXHo0y7rltoyWEDqNmsAva7Zy+PhJzrnaBoVYN7BSo99bTJ67nC27D3Lm9GmiIw+x9/e5fD26F7V9M2loF/pDXq97ofUjWzCNBz7LFzNXsGv/Uc6djeTssX3s+n0hM796h+fubk2ZKxrMs1InHv1kOmu27uP06VOcObqbbSvmMPXDO2jgmc1r/clC6c7daephkL77Z+YfdLjypMuYRC//gsm70sBwp1GLxhTDSq3HlnAuOorzm1+nRaZ7crXjo11niD13jB8GBvK3K1jYfdWtBOFDXuJ/89ey7/AJzh7fx45fp/HxQ12omtlea/m9rlA49/hM296F5yydz+pkE1ulm+hRVzufiYiIiEjB0adLERERESlABiV7DKVnSQvp+3/k29Xx7Dk4id8ffpu2VftyR5sPiViWlPvT2qrS/9lRl/3AkxKVG9NzZBjd+nbj+QHDGLcjucBqkRUjsBWjJ3/DE+EBlwUlQVQOu4n7Gt1A99aP0fPeHzmannG8fwtGPPsA7a4YUPYOLEvtVr2p3epW7rh9AiMGv8TCM38N9BtBN/L0e89c9Ty3EhWpFWiQkNlKdH88168Zdz4+/Krn4uFHmTptGVynNd06vErP/mPZmZqHRriKk9NrVnPA0YLQwEY0rmzl132XhRa22jQL88HAQt2werh/c/qy2UAWSjZqRAUrpO9Zw+qz2VQsL6wlCe/Z/bIfuOeyDQqjbhb8QjvQtWWVy2bp+FKqenP6PB5Op1bl6dH7U7ZfVi5X+kNBX/f8nM8o3phHv/2O59qUxHZ5SOQdQNnqAZStHkb7dp5snLSaSHtGXWrew7S5r9Mh6LJEzS2YiqHBlPPbxecu5WDehLdqiIfh4Oia1RzKbXYG4IzidJQJGNj8/PExHBxa+zunHA2oVDqcFpWs/H7g7ye2VQ8nPMgC6XtYsz4OM/MzZy8PfdXwb8oT337Hs62Csf7Zzh6Ur3cDd9bryIDbpjFy0FPMOpb213Py3U8K6R4PmbQ9JOXQmGbMWlbsSKNr84q0bFEOy5ajFPBdRERERET+ozTzTEREREQKjrUS/Ye2x5dUNk2eyvY0cJ6czbeLY3FaSnHL0K4E52UbIdPOyd/G8kjfNtSqWI6QSg1oc8c7LDyRhqVke179+jnaeBd4bf7OUpp+H4zniXB/0o4t5PUh7alVoRylarSg14vzOZpmo3zP13m7/9UzkEg/yPiBYVQuX5bAkuUoV7c9t704gz2JBsUbDGfChHupldmMFsdRJo9oT72alSgRUo7yoa3p9uiProUC6Yf4emgLalWrRImQspSp3YYBry/hVLpBQMtRvHl7+cz/GEj9mbsrlMQ/+K9/If0mEpXNIHb6gTWsPuMAWw2aNQn428wbS9lmNC9vAywUbxJOnb99fc+LJs3r4W44OL1mNS5PFHK1jI6T/PTULTSvX4NSpXPRBoVat3T2Tn+Wwd3aEFq9EiVDylCqZkt6v7yQE2kGxcMfYXSvEmT6NnGlP+T1umfZALk8n1GCW97/H8+3LYk19SCzXhpMy9DKlChZmlJV6tFi1CLir+pLPnR+7CnaBRkkbf+W+25qTMUyZShZMZTGXYbw2Ksz2JHuQllt1WlYtxiGaWf3tv2k5fyMq1lKU760BTBxxMeSaELatl9ZEe0EW23atrzy2hgEN2lGVSs4Tq1j3YkrOnFh9VVLCH0++B+jWwdjJO5k4hO3Ela9AiEV69N22PssP+3As+ZAxn7zCA0ymw2W135SWPd4yLTtc+Q8y44dUThwo07DOnjk8aVFRERERK6k8ExERERECoxbvUEMCfOAi2uZNuvYpRkAZgyLflhEtNOg+A2307dCHj6Cpu/juxde47vf9nEmyU5q4ml2/PwBdwx+j00Xwa3yIB7uFZJ54FBA3Bvfy6iuJTAubuKtwffw/i+7OZNsJ+XCIX4dO5IR4w+QbvGjw8CelLuyiuZFzp08TczFNJxOO4lndrNw7AN0v3sih9PBJ/xhnuoacHX5nQkc27OPE+eTSXPYSTi7n427zuBSxmQmc+bIUc7EJpPmSCP53D4WffIgo+fF4DSK0ax7x2yWYswl+zZWrovHabjTpHWTy5ZbMwho2Yr6GcGgrUILWl7eOO4NaRPug+GMY+2q7XkLO7LjjGHvug3sjYwlJS2PbVDgdTNJ2PUrCzfs41RMMnZHOinnD7L884d4bl4sTsOHZm3CMg8BXOkPBX3dc3k+twbDeebmUlidF1j0dH/u+XwJu88mkeZ0kBJ/lkPHzpN2ZShiLUvtGj5YSGPLlI/5cdMJ4uzp2JPOcShiERNnbs4kcMuERwUqlbaC8xzHTl7MRSX/YKFEp3sZVMsGpp1t67dwESBlA4tWxOI03GnauS1Bf2u/YoQ1r4eH4SRu/e/syGsnzmVfdW90L093L4nFcZqfHhnAI9+t5XBMCqlJZ9g+710G3f4R21KhWP17GXVL8NX3ljz2k0K7x2fV9jlycPL4KRwYeJSvRCmNcIiIiIhIAdFHSxEREREpIJ60ur0v1WwmiSt/YsHZv0a7k1bOYP5pB4ZHUwb3r0kmu1jlSeqeSXy9IhnT8KJlp9YUL6DzXs2NRj27UdlmkrJ6EpP22a94PIUtS1cT5TRwDw2jgUvTH0wu/PoRn69NxbQE0Klnawp78hxmLCuXRWA3DWxVa1GtwBZxT2bt0nUkmxaKN29Dwz9n0RUjvE1TPBzH2RhxFodbXVo39/9zIN9WoxWtQ6yYiWtYvC4li3MXsFy3wT9UNzOBndsO4cDAq2RJihdkElzQ1z3L89mo26Mb1WzgOPo9H04/6VrQ64zhXLQDEzca9b+TFkF5u0NYigcQYDPAGcP5GBcX77O44elXimphN3LnixNZ9NVtVLCapJ/8iU+mn8hYAjCJlfOWE+M08GrRhXZ+l6eFdWjZpDiGmcTa5Rso0F6cTTvX79GVyjZI3z+NMQvOXbVUZMr2rxm3LBHTKE77nu3xc6U/5dhPCvge71Lb58RJzPkYnIDFP5AAjXCIiIiISAHRR0sRERERKRjFOzKoZ2msZhzLflpC9OWjuSnrmD7vJA7DjTr9BtA4s2XE8sKMY/u2I6Rj4FG1OpUKa0dfw5fqNcpgxaBYp085dC6K2Oi//zs3+y7KWMDwLEGIv4sfs51RrF9/5FJgUrM2lQt9R2KTpDOniTPB4u/v2oC6i+eNXbmEdSkm1tKtaFsjoyLujejQsjhE/8bHX6wizvQgvF0zvACwUK5tG6raTC6uW8JKl6YWFUxZc9cGBV83W4kwBj03lllLV7Nj72HORh7l0JYVfDesGlbAsNku28OqKOqcx/MZvtQKrYgVk8SIdWy/MmPO8nTnmDfhJ46ngXfjR5i7cTUzP3yUAc3L4ZWbsrp74g5g2km1Z9ef3Lnx4z3EREcRG3WKM4e2s2nxVD5+uDNVikHyobmMun00Cy/8dY7ElbNYGO3E8GlNzw5/haTWCs1pXs6KmbKBRStj87bfWZayaufi1KpdHhtO4rdGsC+zJS3NODZvOnDp3lizDtVcSrNy6CcFco/PfdvnxJ56abqf4e6Be2FOPxYRERGR/xSFZyIiIiJSAAxKdL2NroEWnBeWMn1pzBWDyHY2/TSHQ+lgrdSLQa29Cuh1TRLiEzEBw8sndwPtGK4v82h44+Pj6sGeeLi88Y6TuNg4nIDh44v3PzDwa6amYjfBsLlRkFmdGf0bizenYdqqc2OHClgBW2hHOpSykPD7Clat/I31Fw3823SkqQdglKRDx/q4mXY2LllBLsbK81/WXLZBQdbNVmUQ3y2bx+eP9aVDwxqUD/bBw92LoPI1qVfJv9D+QCvo657p+Qwf/HwNDJzEnY9xbdbZpbNxYfHT9Bz6PvN2x+L0rUrHoaP5ct4Gdq/8hic7lHatzPYU7ACGOx4upiim6cCeHMPpQ9tYOed/vPHAzTRtO5xvdiX//cCklUybewqHxZ9OA3sQYgGwUKp9R+rZTOybl/Lr+YLvxJm38x/3I5OEuPgsZmg5iY9LvDQjy8cXHxc7Vtb9pODv8S63fQ7cPS5NBzXtl8ouIiIiIlIQCv27rSIiIiLyH2ApS6+B7fExwAjqy5QjfbM5OISbB97AS7/OIy7fA50W/Pz9sABmYgKJLpzPTEsn3QQMDzyLGZDsypOSSUoCcBI9eRB1Hl2Oq5NqsmfgW9wHAzCTk7hYpAO/+XxxZySLF27ltZZNqXdje0qNPYZvx/ZUsiaxZMlqEi64s2hDKje1bceNDdxYcagNnZt6gP13Fiw96+ISbUXUQAVVNyOYPi+/QrcyNhynf+OjF95myqp9RMamYHiVJPyJqcx8MLRo6lgQzFRSUgAMvLy9crkHoZ3jS95jyJJPKRV2EwOH3Mld/VpSvnYPnptcneJ9uvDiuuxDFWd8DDHpJngEEBRggSzjOztLH21Av8nnc9GjUlk3cSq7ho6iXutB9K88lU8PB9KhU2PcSWP9wqVE/q0TF2JfNZNITAQw8PUrnkXgaqG4n8+le2NSIokurmKZpQK7x+el7bMtGAFBAVgAZ+wFXF2tU0REREQkJ5p5JiIiIiL5Zq3am9vCPVwcLLcQ0GkA3YMLYJqVpSTNm1fBikny/j0czWz5sis4Y88TYwKWslQs4+LeSmY8Bw+exYEF/7Am1Cior6AZftRvUBkbJqmHDrhU/sJiptpJzQgVPTzycm2cHF8wl4g0cG/Slc5lqtL1plrYUjay8LdYTPMcSxdFYLeUo8tN9QhuexOtvCB141x+PuVidJbvMuZVAdXNLZQWTX0xzBSWvnEvb87ezNHzSdgdDlITznLsdEJRxYMFw4zl2PFYnFjwq9+AynnauiyVM5vn8NFjvWjaegRTDqWBRw2G3NGWYjk+9ThHTzvAUoKK5XI8OtfS90xh/IpEcG/I3cNb4FOqK71beoJ9E7Pm/32PrkLtq2YC+/aexIGF4g3DqJnZ/cgoTqMm1bFhkrJvF4dcnwaYqSK7x+fISrkKZbFiknriKGcUnomIiIhIAVF4JiIiIiL5ZCO0T1/quxs4IifSu3xJ/IMz/xfU6k222E0M77YMvLlsPj+MGgR2eIyRLdwxnLEsm7+KxIxHTPPSPwx3PN3+PoDrPLWb3TFOsFalU+eqLi7FkMaWxcs54wBbzUE83LVELmfVZM6j9lDubu+FYSazdslq4gvgnHnljI7ivBOwVKZm3lIPnCcWMGeTHTya0ee+O+kRaiN1wwKWRJuAk8jFvxCRZqVy10E82qctvqSyfs7CK2bsFG4Z86rA6mZe+h9HuvPfHZRlyk7EspVccIJb3cHc29YvX++TlKM/M37O4Ut7ApYMyXmvtvQDbN15EdNwp06DGrjl47UzZZ5h1leziHRYqXDbozx5/wBaFoPUDXOYf0UAXLh9NY1tPy/kSDrYagzkgS7BV7WzZ927ua+jD4YZz4p5K4jNV2crqnu8Cywh1KtXEitp7N66m9TCfj0RERER+c9QeCYiIiIi+ePeiP59qmPDwbHZP7L6YtaHOg7MYMr6FEzDg2b9b6WKq2PKhhchFcoSWMwNi2HDK6Q2Xe7/nPlfD6WKDZI2jeHdn//ag8eMP0+M0wRbbXrd1Ymage5/DS7b1zFjzkkchhsNHh7HB7eHU8HPHYthw9O/FJVKF890wD9l9Vg+Wh2H01qGfp/N5KsHuxFWIQBPm4HF3ZdSNZpx85CbqJVZGmerydDXXmBou5qEeLvj7lOKut0e57vJT9LEE9IOT+GTWWeLNExxRm0m4kQ62Coz+NmRtCrrjc3ijm/Z+tzUM5xSrvzl4Ixk3k/ruIgHLYcPI8wtlXXzlnA2I1dwnlzA3E1pWKsN5r5OxSF5DTMWnHZxycYCKmNeFUTd0vYQsf0iplGMG554i3vaVCO4mA0DA5uHH0F+rs7suXbFL/mCCbtSMa0VuPOLybzeL5yqQcXw8C5Jzda3Mfq+dhS/qpI+tBoxmnu7N6ZKsDduhoHVw5+K4f25r2dlrDiJOXbMhQAoiQ1rtpJqWinXqjVVCyFfTVoxjs83XgSfNjxyfxM8SGb1zF+uCkkLu6/aI77gnQVRl+5HY6bx4ZDmVPJ3x90rhLrdn2TS5Mdo5Akp28fz7pxz+bu3/BP3+DwyAlrSrp4bpB9j7e8nXb6XiIiIiIjkROGZiIiIiOSLZ/N+9KpghfQDzJi+Ofu9wJynmPPDSpJMA/cGfehby8WRVVtV7pmyicMnTnHhXCSRu1bww2t9qeNjkrDjfwy/Zxx70v463IxdxbxVcTgNT+qPmMjaJc/T7M9pKCms+WA0kw6mgndd7vh4PtsPneTCuUjOHNzGb881xz2zMjiO8M399zFuaxymd036vvwtyzfv48yZs1yIPMTetfOY+M5D3FA2k4/Yhjvl24/k059Wse/YSaKObmf1xGe4qYIbznMreHn4W6xOcq0pCk3aVr7+fAUXnBaCOzzPz9uOEB11khPbljLtswdoe3XikQknp+dNZUmcicVqhZR1zF505q8BbWck82avJxUrVqvJhUVTmBuVi2H9AiljXhVA3cwopr89hk0JJp7V+/L+rLUcPBFJTPRZok/t49dRjQp+ttQ/zb6dj+5/nkWR6RhBzXhg3Hwi9h3j7LGdrJ/9KY92KIubAZjmX4GOW226jXiId777hc17j3Du3FnOn9rPtgUfM7C6G+b5lXwwbg0pOb64k9OLf2ZjqomtTne6VyuE5MZxiG/fmsJRh4FhGJixS5kyP5Pgu7D7qvMMMx6/i7fWnAffBgz7aC5bD54k6vgOVn83ik5lbaTs/56Rd33M1nxOx/pH7vF5YhB8Qw9aexmkH13I/J1FuO6tiIiIiFx3FJ6JiIiISD54075fd0pbTezbfmT67pwGL02iF81kWZwJtpr07t0g27DAjFvPdx9/w4zlm9l78jwJqek4nWkkx0Syd80sPhvVjxY3PcOCyCs29HFGMvXBwTzz3a/sPBlD4oH9HL6saGb0Yh7vfguPjPuFiKMXSE53YjrTSImP5vjeCJbN+o5PvljMiStPG7WM57q155ZRXzBn3X7OxNtxONNJSTzPsV1rmTtpDluTM6lI+mFmvPsB01bsJjIulTR7EtHHNrNg/DP0bD+Yz7cXdXIG4OTYpBH0fHAsv2w/SVyqA4c9iXNHt7D4pzUcd3V7uAuLmfLzOZyYJK2ayYK/bULkJPLnmay6aILjNHOmLCMuV1NiCqaMeVUQdUvZ+hG9u47gre9XsONEDBfTHDjsKSRcOMOx/dtZt2we037ZReK/eE3H1L3fcXun3jzx5UIijp4nyZ5GStwpdv46lbe/+I0LTjCTk0j6c6roOVb/8D2LtxzhbEIq6U4n6anxnDkYwYKvX6Jf5yF8dSAt29f8g/PUHCYti8N0C2Xg4MZ4FkL9kn//gvEb7Zg4OTt/GotiMrtYhd9XzdgNvNe3Pd2f+oJ5Gw9zLtGO/WIMp3b9xuQ3htGu86PMPOZau2WtcO/x+WKpSN+h7fHBzrap37M1v1UVEREREbmM4RdU4l/8Z5mIiIiIyLXLKHUHsze/RztjB2907Mx7exw5P0nkumWh/PCZbHyzBax4iob9JnKmEP4a9Wz+Kmvm3EflhMU81GYoU04X7GJ+btXvZ87Sl2lhbOG1rj35cJdSm6Lg0+E91n5/B+Vi5zK81XB+itbQhoiIiIgUEIPpmnkmIiIiIiIiBcPwo9WwxxnWvQX1qpQlyNsdq82TgHKh3DDsHb57ujmexLNi5hJys2JnbqRsGMu7Cy+A/w08++yNBORrhUSDgIo1Kefnjs0zkOptR/DllOdo4ZXI728/wmcKzoqGRwMefmkg5S0pbB73PnMUnImIiIhIActsO3MRERERERGR3LOFcvMjT3FvuSzWJTTtHJs9mqd/PE3Bzge7jPMMM158iwGt3qHDgLd5Y0EEDyw8f/W+ZK4wAun+zlLG3OjBnxmcaefYzCe598t92e//JYWkGE2e+IiHQ91I3fURT3+xF+12JiIiIiIFTeGZiIiIiIiIFAzjHGunTaVc68bUrV6eED9v3I00EqJPsH/bWhb++A0T5u0mrtCSs0scxybz8DPhjO+6i0kRsXkLzgAsQXikHOXcxaoEWhI5e2ATv0z6hHe/W0+UVmEtIinsmzOeac16cGT0R0SkFHV5REREROR6pD3PREREREREREREREREREB7nomIiIiIiIiIiIiIiIhcTuGZiIiIiIiIiIiIiIiISAaFZyIiIiIiIiIiIiIiIiIZFJ6JiIiIiIiIiIiIiIiIZFB4JiIiIiIiIiIiIiIiIpJB4ZmIiIiIiIiIiIiIiIhIBoVnIiIiIiIiIiIiIiIiIhkUnomIiIiIiIiIiIiIiIhkUHgmIiIiIiIiIiIiIiIikkHhmYiIiIiIiIiIiIiIiEgGhWciIiIiIiIiIiIiIiIiGRSeiYiIiIiIiIiIiIiIiGRQeCYiIiIiIiIiIiIiIiKSQeGZiIiIiIiIiIiIiIiISAaFZyIiIiIiIiIiIiIiIiIZFJ6JiIiIiIiIiIiIiIiIZFB4JiIiIiIiIiIiIiIiIpJB4ZmIiIiIiIiIiIiIiIhIBoVnIiIiIiIiIiIiIiIiIhkUnomIiIiIiIiIiIiIiIhkUHgmIiIiIiIiIiIiIiIikkHhmYiIiIiIiIiIiIiIiEgGhWciIiIiIiIiIiIiIiIiGRSeiYiIiIiIiIiIiIiIiGRQeCYiIiIiIiIiIiIiIiKSQeGZiIiIiIiIiIiIiIiISAaFZyIiIiIiIiIiIiIiIiIZFJ6JiIiIiIiIiIiIiIiIZFB4JiIiIiIiIiIiIiIiIpJB4ZmIiIiIiIiIiIiIiIhIBoVnIiIiIiIiIiIiIiIiIhkUnomIiIiIiIiIiIiIiIhkUHgmIiIiIiIiIiIiIiIikkHhmYiIiIiIiIiIiIiIiEgGW1EXQEREREREROR6VqtWTXrd2quoiyEi+TRr9iz27t1X1MUQERGRf4DCMxEREREREZFCFFyiBK3btC7qYohIPq1asxoUnomIiPwnaNlGERERERERERERERERkQyaeSYiIiIiIiLyD/n0s7Fs2LCxqIshIi4KD2/Kww+OLOpiiFbKehIAACAASURBVIiIyD9MM89EREREREREREREREREMig8ExEREREREREREREREcmg8ExEREREREREREREREQkg/Y8E5EiV6akk/C6aUVdDBERERHJp9nLPYq6CCIiIiIiIvmm8ExEilx43TS+fTW+qIshIiIiIvnkv7xEURdBREREREQk37Rso4iIiIiIiIiIiIiIiEgGzTwTkWvKhDlBbNlXrKiLISIiIiIuuvvm84TVuljUxRARERERESkwCs9E5JqyZV8xFqwpXtTFEBEREREXdW8VDyg8ExERERGR64eWbRQRERERERERERERERHJoPBMREREREREREREREREJIPCMxEREREREREREREREZEMCs9EREREREREREREREREMig8ExEREREREZFriIUyPV9l9uJ5vNLaragL8zdGQHtemDafVW/fiEdRF+YaYQ1pycj3JvHr7xEc2L2FHatm8XbXYIyiLpiIiIhIPig8ExEREREREZFriIF3udqElg/As9ATGA/CHv6BzZsW8E7noBwDH8OzDHXqVaaEly3j2Nw9/7rjHsojX37GkzeHUSnQE5vVHZ8SpfBITcQs6rKJiIiI5IOtqAsgIiIiIiIiIrnlScUOt/PA0O60qVuBYC+D1LhzHNm7ldW/TGX8jG3EXLPphZXQYWP44HYf5o0cxuf7HAV6ds/mjzDphR5ULhmIr7c7VtNOcuw5jh/YzupFM/jupw2ctv91vGEYGIYFSx6Tr6ufn5v6+dJ73Ao+6JjTPLY0Nr7RndsmnsKZt2IWCo9m/RlY0x37gR958rFPWXI4EfcSZfBJTC3qoomIiIjki8IzERERERERkX8VKxX7fciMV9oSbP0r8bEFlaNuq7JUt21l4k/buHan/ljwr1iH6qVjcC+EqVrWEtWoV63MZcsqeuIbXJ7Q4PKEtujKoN4TGH73p6yPN4FUIj7pT6NP8vpqmT2/cOt37bAQUq0qfkYqq775hJ8PxGICqWeOkVDURRMRERHJJ4VnIiIiIiIiIv8mtobcMbI1QUSx/IMXeXvmFo7FpuEeWJ7Qpm2pf/E3zl5L05OKRDq7P7uNPmP3kmpaKeZXimphnRn+5AN0r3cXrw5bQrdPdlOwc97yIoGZ94cx88//thE2ah4/DvPlpxEdeHpVWhGW7S8WD1+C/D1JT4ghJjk946cGHp7uGOZFoqOTrt2sVkRERCQPFJ6JiIiIiIiI/Jt4laNikBXn8fl8+vVqDmQkQPaoQ6z/+RDr/zzQIKjVcJ67ox11qpanTHBxvNwcxEfu4bepn/HltrLcOvgWOjerRTl/K0mndrL4u/d5e+oOYi9PQopVovNd9zP85pbUKeONM+YYEctnMPbz79lw7or4KTfHWmvw8JztPJzxn86o7xnS8TXW/pEXGV40f2gCi96oSYUgTxyxx9m6dBoffPw9W1xYk9KZbifNYWKSTnLMSbYv+4Yn4kpQf+JQKjcNI8Sym0inler3f8+Ch0sz84qwyhLYgIEj72Nwp0ZUCXIj+fRefl8XR8jfdo/P+vk51i/XDPwa9ueROzrTNLQalUr5U8y4SPSxRbwy9GUWGu149sNH6FqzHCHFPTAvRnNo0yImfPgZs/b9EW5l1icgJSbztrUENmH4i6O598YaBLgZmGYaCceX8OqdT/NT5KXzYQmg/1db6f/Hk9I28lKnu5h42ulif8iuXq+wofY9+e/DIiIiIrmk8ExERETyyUrF3q8z5r6arHu+P29uSM/5KfKPMgLa8/zYJ+l87GNufGYp2e5CYvjRdvRHDI0Zx9PjNnL+mhp4slCm58uMfagR217tzUurr41v47siV9egQHhS/dZneaXjXp57fBpH9LYUub5cPM3JGAeW8jcwpOtP7J9/jIuZHmghsH4nerarc9kf/24ElG9Er6e/ptcVR7tXbMKA58cRmNSH+2afvbS3lmcd7h0/gVHhfvyZGYXUoN3AZ2nZtgFP3P4s8yIzQpDcHOsKw4MKDZr89d/BVWl123M0rFGM3kO+YX8e7m3OdMelelksWLI5zijekucnjuHO6p78sfKiR4WGdKtw6f8XzY5eFkq26MuQbpdfT19KlHQjNdGEYv5UDatBOfeMh3xCqN1+KO/VLUnaLU8yL9ok8z4B3pm1raUU/d4ew6h2xTHSkzh/NhHDJwj/km6kxjkBa/bFdbk/ZFcvI899eMTss7lpXBEREZG/ye6zooiIyH+cB2EP/8DmTQt4p3MQBbNlRWGcs+j5lq9DnfL+eBrXS42uL4ZnGerUq0wJL1sOfc7At9UjvDa4CVWCLNgLrUR5fR8YeJerTWj5ADz/ZV3t6mtQ2PeCNNJ9y1O30yO8NqB8TsObIvJvkxbBN5+tJtqoSJ/3Z7F8yqvc16UWgVl9PdaM55dnO9Ogfn2q1WtN9+cWcNJh4ozdyGcP9qVF44bUCOvE7WMjiMefdr07UtICYKXakBd4rGlxUvbMYFT/joTWbUSjzsN5a/lpjDJdeXlUJwKM3B6bwbGfT2+pT+WaoVSuGUrVNlfMyjIT+e2t/rQOb0z1us1oPfhtlpxx4t1gMIPD3FxuLsPqjndgOULbDOT1F/tRwergeMRmzmS5tKWNunc/w9Bq7sRvncSjfS/VpUHHwTw6YSPRri6JmVP98spMYMlLPWjSqCHV6regTb9PWJ8GZsIa3ht6Ky2aNqZa7frUbt6DuyZsJyWoA33aB/z9d81lfaJqaOZta/g2o3MzX5w7v6RXixY0aduRxo3Dad7rfVYnX3YuZww/3tPwz3pWrnsnE08bue8PWdTryvK62odFRERE8kPhmYiIXN+s1Xhw5jYOb/+RB2tmN8jiTYvnf+Hg3s18favvnz81DAPDsGApwJHtwjhnzrzp9O4qDu3dxMQBIVl/AHBryOjF2zm8YyLDyv37PyZ49xzD3n3bmHd/1Ws8PLASOmwsC5dN5IGaRVhSW3XueLwXZc8v4J0xG0goxFlnRfM+uLYUbhs4ODLtbcbv9qD5yJHcUPw/3NAi1yUHx6Y/Su/7PmXu7iSCGvfh6U9nsHrJt7wxpCkhV4ZopoOEc1HEpzpw2GPYPWssU3Y5MGzR7F6zhzOJaaQlRbJm/NcsiTOxlq9EBQtgrc7Nt4TinraDMU+8yvRtZ0lOsxN7bC3jn3yJH0+bBLS/mRsCjNwd6yozjahD+zkVl0J6WiKnNk3lzUk7SbcGUatWiRwGNGzUfXQuB/ft4vDuLez8fRHzJzzPgFBvLu6ZzEtf7yLLiWvWmnS5sRKW1Ag+fuJd5uy4VJf4U1uZN3kxB4t6ozQznZhTJzmfnIYjNZ7IY2dJMgHDwL/ebbz5zU+sWb+R7b9O5OUuZbBiI6R08N/b67I+4UzPom1NExMwStQivFbwpS+umKmcO3Iy5yUR89IfsqrXFeV1tQ+LiIiI5Me/f1RMREQkO5ZgQkpYMDxqc89jPSiVxW8+a/WBjOpXHqthJSA4MCNoSSXik/40anwTTy06X0CboBfGOV2RxJoFK7hgehLeoxNlsmgHj7DudCtnIXXzLyyMdPVr1ZJ/Fvwr1qF6aV/cizDj8G49lCG1DXZPncDSQt0opKjeB9eSf6AN0g8wefxS4gK7cE+vcvrgL3LdsXNy5Xge6X0jbQc/z7glB7GHNGHQc18z9/P+VM1ukwbHaY5FpoNHCUL8L7s72M9wMsrEKOZFMQNwr0i1chacJ9az5ugViVHSZlZtSbl0THlL7o7NMweRh46SZBr4+Hi7PmvXNDFNE5zxbJrwMD0Hvcea7H7PuZWhUlkLzpOb2XT6X/J5yChO2+e/ZeLo2+hQrxIhxT1wKxZIhfLBeBhgseS0a8fVbWsmrGXWsmiMkHaMnriUrWvnM2PcKzzUtTreOTV+YfcHF/qwiIiISH7ob2gREbm+eQQT4mdgj43H2mYEw8M8rz7G8KfzfXdQLzWWWKdBYGCAS4MxFg9fSoSUIMAr88GInB7/pyWvX8DiKCfujbrTvUJms5s8adbjRkpbUlg/f2k2SxkVvWutba8Lhh8de99IsD2C6bMPU9RfqpeCYBK74icWnLXRqHcPql/b0y9FJM9SORMxi3cf7E27vi8z97iTEm0f4aGOPtk8x0maPR0MN9zcLv/Uk0ZaugmGkTFYkJtvdPwz3/4w7XbspoGR47TddHZ+fDPVaoZSuVYYXd5cRyw+VK1VElJz+MqCYb1Uf8Pyr1li2wi8kTt7VcAas57PHuhDi8YNqVanCc0fmsUZF3+pX9W2ZjQ/jx7CXa9/zQ/LNnPcLENYh748/uHXvNstOMeloAtXzn1YREREJD8UnomIyHXNEhhMkMVJ1IJxTDpYmv7397xq1pWt+m2M7OzB72MnsC7VQkCwf8YvSCvV75/OgT2reaeN22XnbMK9H89kU8TvbFj5GxGbN7Ft8Xv0yThx9o9ndk6DoFYj+HD8FBYuW8n2bVs5uHsrO9fMZfIrA2mU2fJGnuXoMPw1Js//le3bt3Ng+wY2LZvFj+NeY3i4f+bDFRc3MWfxaZxudbileybLGHq34NYbgjES1zJ7afSlZXqC2jP6u1msWreR/bu3sy9iOQu+fJreNXP6trcbrV7+jUO7Z/FYrctfycCv11j27dvCt30D/3YOw7saPR77kFnL1rJnRwSbl07l0wfaUe6y1TZzavucZdbWEWxeOpkP72pOzcZ9ePrDiSxbs4F9uyLYvPg73h5cD3/jr+cHtribd8dNZMHSVezYvo2DOzawcdEUPnviFupd/s3nPLQB1ho8PGc7R/bt4si+XRxa9QIt3Vxvn0tt1IDBz49jwYp17N0ZweYlUxgzsjUhOTWRVzidWviQvuNXlp+9Ijl1L0PrYS/yv1lL2bJ1Gwd3bmLryvnM+d+HvNK7RkZfyk19M39vAVCsAjfe/ybf/7KCnTu2sGvDchZPeplbK2aV/FgIbvs8i7btYMuku6mX6RfN83vdM85SoNcg8zZw7T2Xi3tGylaWronFUq0DN2TZhiJyfXASt3sWH/+4B4fFh2rVSud/yWL7MQ6ddGIp34xWV95DvMNo08gT7Mc5fNKZu2MxSU9Px8QLL69/IuSwc2DyaJ5bcI7irZ7g/Xtq4ZHt4ccv1aVCS9pXdX1vtb/80/UDS3ApSrlD8upJjFm6lzOJaTgcFzl/Lp7U/Jw45QQrJn3IMw/cQec27ej24mJOm4G07xJOtnO7ctUfRERERK49Cs9EROS6ZvgHEmAxiY1az8RvVuJoPoy7Gl02XGIU54YRg6gVPY8vZu3jfKKJZ2BQ1kvRWErR7+0xjOpaE38jmfNnzxKTbOBT0o3UOGfOj2d+UgLrd6Jnu4bULBeEr6cbVqsb3sFVaXXbc0waO4wal0+w8qzF8PHfM+GJ3rSqXhJfDys2D2+CytWgaceb6d4gIItf8HYiZs/nkMNGje7dCP3bpC0Dv7Y96BAAMb/OYVlMxjey0/2pGlaDcgFeuFmtuPuEULv9UN775lV6BBfgYJBXPR78ejKf3NeFhuX88HT3JKB8A3o+9ClTX253aTP5PLXtlTJra08Cyjei19Nfs3Dqq9zXvTFVgr1xt3kSULEJA54fx7u3/LFPnIWghjfRq2NjapcPxMfDhtXdm+BKDek+4g2m//gqN+WYUhVS+wBG8ZY8P/EbXhvSltqlfPFw8ySgQkO69W9H5RxGUG01G9HA28nJrdv+PuvQsxbDv/qBb58eQPs6pfEvZsPqVgy/kMrUb9mFAe2rkJdhxUx51GL4l9P48tFbaFYlGG93d7z8QqjWsDw+FzO7xgZ+TR9mwkcDKLPvK+594Bt2JGd24vxed/6RawC4+J7LzT0jle2bd5FmrUrjhr5ZvKiI/Ou4N2LEm08ytGNdKgR4YjUMrMUCqdy0FyNvrYHVTOdc1AXyHUs49jN37m7sbvV46MMX6Fs/BC+bO34VWzLivVfoX9ogdsU8ll4wc3csJlFnzmFaS9OpXycq+9iwegZStUkoZQor53dG8curLzH9lAeNRr7C8Nru2dR7H/Pm7yHNVocHPnuH4W2qEuhpxWL1xC84gJzzsH++fs7zUUSlQbFmvRncuAw+NgMsbvj4eJLnefrWynTs05H6ZYvjbjGwutlIT0gglUsTu7Jthlz1BxEREZFrj8IzERG5rln8A/E3TBLi4ola+C0zTpWl77DO/DEGba3Qi3s6+bBjymTWJcYTl2BiCQgkKIvfkIZvMzo388W580t6tWhBk7Ydadw4nOa93md1cs6PZ8uM55dnO9Ogfn2qhjaj9eC3WXLGiXeDwQwO+yOesFL19hd5Itwf++H5vDTkJhrWq0+1es1o/eIKknMYf3DsncdPO+xYKnXlloaXDRoZAdxwc2v8zLMsmLmGhD+KlLCG94beSoumjalWuz61m/fgrgnbSQnqQJ/2ri1vmTMrNYa+wIMNvYha8Sl3dWtFrdDGNOv7PDMOOyh364MMrmbNX9te6bK2rlavNd2fW8BJh4kzdiOfPdiXFo0bUiOsE7ePjSAef9r17khJy9+fv2BUR0JD61G1bnNaD3iarzbF4FbxFl4fdQOZTRZ0iWM/n95Sn8o1Q6lcM5SqbV5jbZpr7QM26t79DEOruRO/dRKP9u1IaN1GNOg4mEcnbCQ62xFUA59KlSlldXD08PHLBlutVB/6Mk80CyD9+GLeHHELTRvWp2qdMBr1+Zxt6XmsZ6asVB78Ao+H+5GyfzbPD+lKWP2G1G7akZuGvsOi6Cs7twW/xg/w9di7qHZsIvfdO4YN8Tm8AfJ83f+Ja5BRxNy851y6Z5gkHDnKWacblaqUd/FaiEh26tSuTWBgYJGWwVbnBgbdOoxXxv3AinURHNy7k4NbV7F88iv0re5BysEf+GrRhQLYT9HBgUmv8vGmeDxr9+O96cvZtWsLWxd/xbM3lMaMXMjL7yziUv6Ru2OPr1jO7lQLFXu/z/KIbRzctoql346me9nCG6Yw41bzzmtziHQL5f4XB2eznK2D/d+9zPsbYnGr0IXRE+YSsW07h3ZHsHnGSOrnmEb98/Uzzy/nx2XRGCEdeXHqEnbs2smRPVvZMmEAZfMY2BlBzbj7lTHMWf47+/bs5OC21Sz5uA+VjBh+W7yRpGyfnZv+ICIiInLtUXgmIiLXNQ8/P7wsThITknCmbmXi5C24tx/KwOpWwJPwoYNomLyMCTOO4iCZhEQTwz/wquXa/mSal5YzLFGL8FrBeBqAmcq5IyeJNV14PDumg4RzUcSnOnCmJ3Jq01TenLSTdGsQtWqVuPRL21qFbt1DcXfs5YvHnmfihhPE2R047ImcO5+Y8yCZ4zhzZ27ioqUMPXo3/3O5HUvpLvRp6Y3z2AJmbEz563jDwL/ebbz5zU+sWb+R7b9O5OUuZbBiI6R0cMF8kLDWpGePWtjil/LGU+P59VAsqekpRO2YxSuf/kqitTotw4Ox5Kdtr3RZWzvsMeyeNZYpuxwYtmh2r9nDmcQ00pIiWTP+a5bEmVjLV6KC5e/PT7xwgeR0J860BE5tnc9bI19mzjkIvOEW2vsV4Kw8V9vHWpMuN1bCkhrBx0+8y5wdZ0lOsxN/aivzJi/mYLb7nVgICg7E4kzm/Pnkv/qRtQY331wH9/TdfP7gKL5acZDoiw6cjlTiz8dysSAHvKxV6NGzLh5p2/n04ReZsuE4MalppMSfZf+W/Zz7W/BkENj8Eb79cgS1T07h/uEfsCbGhcLk9br/I9fgj6rl4j3nyj0DMC9Ec8G8dI1FJP+69+jOpEkT+XL8l9xzz900btwYD49sFwEscM7Dc3nnwyn8smE/kXEpOJxO0i/GcmrfOmaPfYa+g95nbUIB3aQv7uaL4YMYOeZnNh27wMU0O0lR+1k57S1uH/AMcyMdeTrWceA7Hn7qf/x6IJpkh4P05PMc3nKQc4W6V5VJ7MoxfPBrLJ4N7+HxbkFZfxHo4h6+Gj6AYe/PYPW+s5d+d6SnkBB9gt0blvLTisOkZfNK/3j9zAv88vxwnvj6V3ZGxpPqcJCemkRM1En2b1vPuoNxuQ5TDeM0m3/bzrELF0l3OnFcjOH49qWMf/punpp/Lufz5abviIiIiFxj8jx7X0RE5N/A188Xi2knKckOODkxezKL7v2QQUNa8L9Pg7jr5lKcmP40S2NNMJJITHJiqVic4lmMa5gJa5m1LJoO3dsxeuJSnog5xs6tEayYO5lvFh4gKafHczVq4SDy0FGSzFB8fDL2O7JVpHolK84Ta/ntYHZDNllxcnbhDJY91pzunXpxw7urmBdroUrPXjT1SGfXrFns/GM2kVGcts9/y4SBFflrH3YPKpS/VDaLpYA+RriXp0o5C5ZiXRizoQtjrjrAQZlypTEKtG2vfInTHItMh9olCPG3QHJGUmM/w8koE6OkF8VyGOsy49aybLOdW2+sQNWyFojNR3ku52L7WNxKUKmsBefJzWw6nfuFutw93TGwY7df/sMKVC1nudTfDuWlv+XCn317A2uP5zCYZvHnxnvuAGcsy6dN4ffzeVyYzNXr7vbPXIP8v+cyuWcApt1OmgnuHtksUSYiLktISMThdFKubFlK9exJr169SHc42LN7D5s2bWLLli0cOXIEp7Pw9nJyxu/nlwlv8suEnI50cGBcP6qPu/LndpY+1YwqT115+GE+792Az688/OJRFn02ikWfuVA4l4+1c2zRB9y16INMH8283JC25mXCa7+c7ZmT5j1ErXlZPOiMYtYDrZnlwmuReoqVE15iZbbtnHUbZ1e/rKWz+d2uVHs3N691iZl8gNnvPsjsTJ+b8zmuatuzK/jgwRVkXYPsywO42B+yO08B9WERERGRXFJ4JiIi1zXf4r4YZgrJKZeSFTN+Bf+beYweg+/kSTOQdu5beWvaduwA5kWSU0wMT1983SDTrxOb0fw8egiJW/rRtXkDwhrVI6xDZRq370AtS28e/Dmnx2NyVX7TbsduGhiWjGFwixs2C5CeTl6/q2vGrWDqz6fpNrgNt3Urzc8zStK/dy1syWuZOvvon+c1Am/kzl4VsMas57MX3mXKukOcu2gj+IbnmP3xzTm/Dk7AA0/PnFInM4dvLht4FPPAcKHt856fOUmzp4Phhpvb5eVNIy3dBMNwYZadiem8tI+L+edPXGyDbE/rYvsY1ktlNCx5Wk7TnmLHxB33y/MV81INcDhdatt81dewYDEuvWbOL5TI5iURBLdvS4cXv+b95Lt4Yv6pPLwnXLzu/9A1yO97DjK5ZwCGuztuBthT7dk8U0RclZCQgNPhwGqxYLNd+pPaZrVSr25dateuxbBhd5KclMTWrduI2LyZzZsjirjEIiIiIiKSWwrPRETkula8uA+GmUryn+vLpbHz+2lsHDKaOwaYxPzyFLNP/vHNcDsXU0xMwxtfHwOy2kcr5QQrJn3IikmA1ZdafV7lm5c70b5LOF4/LyAp28cX5a9CaWeIjHZiqdCE8DIWdp7Iy7faU9j4wyz23vYA4QP70jymLL0rGJyf+yMLov6KCCzBpSjlDslLJjFm6d5LASNpnD8XT2qm57Vi/fOThZOEuERMS1lqVvPD2Ho+6/Ah7RRHTzlx+s3hns4v8Gt2+5fl1Pa5bIkCVawe4XXdwX6pPoDrbYBJeno6Jl54eV0Ru7jaPtY6HDrpxFKpJe2rfs6O/bmZKebkfPQFnJYaBAV5YZCxtFPaSY6ccmIpH0bjUhZ2nsquv+Ximmcmo56WCuG0KG9lx9FsojAzjYPTH+feHx9l4pjbufn1jzl99i7e3ZhQAPv7ZF22wr0GeXnPucYIDCbQuHSNRST/EhITMDJbes/gzzDNy9ub5i2a0aJlCwzDIDYu7s/DbG5uVz9XRERERESuKdrzTERErmtePl4YXCQl9a8hdef/2bvv8Ciq7oHj39nZ3bTdlN00kA6hN0WKSJGmIoKIgqIUFbEiqAiCigUbYnlVEBuiwiv64k86gtKrEASkhY70BJJsym6STbbM748ECBDIBlIInM/z8JDszs6cmXtnkpwz986JBUxblorXc4I505dz9lFJXpyZTjSdmWDTRX5EqtXpeF9HGt8QjFGnoBr0uO12sgFFAaWw9690h9y7WLwsHq/fjQz7eATdG0RhMvoRVrU5vbrUw9dJ2Tz7Z/LT+izUWg/y6Wu3Y9GOMOuXNdjzLeNNPsUpFwS07MXDzSpi0iugM2Ay+V9w902Oy42mhHLTba2oFKgCHg5u30W65kebp0bz0I1RBKo6VH8zEWEB5x4Hzx7+WHwQb3h33hw/iE71owk2quhUf8IqNeC25lVzt1fSx7YolCBa3P8QHWKsBOgNBFduzsD33qJvJR0ZGxazOk0r2jFA41RCIppagS69u1DdpEf1t1Dz5gZUxMfj49nDvPm7cOnr8+zEDxjctiYW/9zlQsLDOL8mx3nbdxw6xEmPSrUaVc7+gujZx+Klh/D6NeP5j1/i7voRBOoNmCs2oUe/24lR86+jKPtbAM8eFv15AI+hCcMmjKVfy2qE+auoBhPRdZpSx3reOal5SFo9noEvzuSwoR6DPxzDnREl9Kutr330itqgaOec7xTM1aoRpXNx6OCxy16LENejoKAgoqOjiYmJoVmzm2jfvj13392Nhg0boqpqoZ/X6VQURUEDQkNCzrzu71+6z0cTQgghhBBCFJ2MPBNCCHFNCwwKyBt5lu9FLY2FL7ah5ovnL62R5cxGU4IINhW8PsXakkFvjaH1+TeNe20s/HMjmdZOl3z/ykdGOdn4zUfM7fQRPZsM4POZA857313gpy7gPcm8aX8wrPW9RIVrOP/+hf9uPXdKNy15GTOWDqFtt468Pr0jr5/zroe9+b4+tms3qVpd6g74kj8iX6L5sEVkrJ7GtF2dea5BV975pSvvnPP5/Ntys+O7d5nS4UsGd3mRyV3ObRjXlvF0eehHjhRy7Et11JlipNqdI5ly58hzQ0ldHBmi+gAAIABJREFUz7iP5nN6AJ/vx8DDkZXLiBvaiMa9PmJZr7yXXf/w3l39mezD8Tns9bD3xzf5qPW3jGpxB69MvoNXzgv7UqOX3Hs3809Gf+5o0pgo3XZOeAFcbP/uA37q/Dn9bxzIhFkDL/hc/nX6vr8FRsDO797h63aTeKZhT96e2pO3T7+lZTBvaDuG/nn+SC4vicve45lPqzFjeFfeeXs925+ZybFif8yQb330StvA93OuKPxodFN9DJ4DbN6afllrEKK8CwoKwmw2YzabCQ42YzKZMZtNmM3BmE0mzGYzJrMJk9lMsNmMOdiM2WRGpzu3IO/1ekm323G7cgoeeXYej8eDqqqcjI9n+86ddOncGQCH3VEi+ymEEEIIIYQoPhcUz0aPGlUWcZQ7cbt3MWf2nLIOo8ikfa99s2bPYvfuPWUdhhBXjaBAFUXLIjPbtyc2ZWVlgWIi2KwDLszAK0o8m1ds44ZmMdwQ6oeSncbxfZtYNO0LPp+fCJGXfl+j8DvVC+NNXMzIh59m79An6N2+EVVCIO3INtYcNNGlU228mm+VA8ea6fx6oDvP1rSzZNpcLpgBUrOx8LXBDE8YxqCuzYiJCkJ1O7GnpZAYf4T1+9POTJGXufJTXvzCxMjeLfA7Fp9bJsnewedPPEn6i0N4uEMjqoQa0HIySU1O4MiBOFbuzzr7bDD7Rsb1e4idjw2ib5cW1K9sJUh1knLiAP/8fZQcH459iUzXdzFaBlsX/kF6TFuaVQ9Dl5HAnvW/882n3/D7wXwFniIcA8++Hxk6Ipg3nutByxphGLNTObJjP4mK4tPxASBrF98OfoA9/Z9gcI+2NK4WTpDqIjM1kaMH9/DPyoMFPsoPgIxYlq530K1tBzpE/sxPCbkdQktby9j+gzjw3LM83Kkx1SwGsk7u5a/1Nur0bE/F/Osowv4WeFgdm/h4YD92D3qCgXe1pF7FUIzuNBL+3cGBdANKgdE72TXlNcbfMoM327/ImO5reXrOyQLO3itTKm1QhHPOZ/5N6HxrGNqBGSy71FSYQpQDRqMRk8mUW+gymTAFmTGZg/K+NmG1WrBYLLnf5/0zm80YCpgmMcflwmG343A4cDgc2JJt7EvYh8PuwJGR+5rDnoEjI28Zu4PU1FS8Xi/16tXlo48+umic7rznoe3cuYNZs+aycWMst7a59UzxTAghrkWSbxJClFfvjxtX1iEU2T0976F+3XplHcY1paB6jxJijTjnb/AFC+aXalDl1ZrVa8rliSXte+17f9w41qxeU9ZhFEnPjtn8MDb3bvhnx1fi97XBZRyREOWRQkSfb1g99mY2jOnEI7/aSreYdN1QiXn6F34fWoGZT3Tg5dVFe6bV1c7U8V2WfdGN+E/vo9fXB7hUqUVX4SF+WvwqNy4bTtOhi3CWWpTCdwqhd3zAkk878+/4njz4/ZFLtqkQl+uLkce469bc3+VC20QUunxJF8GSU2w+FcEu1w033MA333x9zmuapqFpGi6Xi+XLVzBr1iyOHTs7VWqbtm3OJJY/nziJ2NiNl719IUTpatGiOUOHPAPk3qy6bs06EpOTSbHZcLt9nPXhOiD5JiFEedWt291lHUKRjR41ijZt25R1GNeUC+o9Cr/KtI1CCCFEOaOLbEbXJrBv57+cSErDqQ+jRrMevPR0S4zeA2zZfhmjU4QAHKt+5L+7uzG03+N0+t8r/JkqPalc08fQb3BnQm1/MnnmUSmciRKV4w0nzXMTd99d8bKnQ7Tb07HbHbnFMLuD+BPx2B2O3NfT7dgddux2B+n2dBx2BxkZpTph7xl2x9knhJ6emvHUyVP8NnMmS5cuxemU2wmEuFb1vKcn9/a8F8gtmqempmKz2UhOTiIpyYbNZiMxMZGUFBuJSUkkJ9vIcJS/qVoNegMu97V1k5gQQghRVBctnsXGbuTziZNKM5Zy4b9Tvy/rEIqFtO+1Jf+dcEKIa59f0wcZ9/ldmM5/3Irm4cT8L5m+V1Lk4jK59/HDx7O4/5v7GDVkNn+9uwG71M/KKZXqD77M4AYuNrw7iSVp0pCiZDk89dib+TaDHr/86RDLC4fdgablnlPbtm1l1qzZbN685cxrQohr17gPPuCfLf8QHR2NxWLFYgnDYrFgtVqoWLECDRs2wGq1EhQUdOYzp0fI2mw2EuITSE6xYUu2Ycv7PyEhgaSkpKtqFFu37t24pdUtzJjxPzZt2lykz0q+SQhRHgwd8gwtWjQv6zCKxah9Vcs6hHJtXMzhi74nI8+EEEKIckXBmLqH5bE1aBpThegQP8hOI/7gdlbP/ZGJP23gVPnJP4qrjkb62s8Y81NV+qdo+ClI8azcMmDIOEbckiWM+UWmaxQlz2JYy60hLX2atrG883q9/Pbbb/z552KOHz9e1uEIIUqZw+Fg//79wP6LLmM0GrFYLFjypqGNjorGYrVgDbMQU6sWlhYWIiMjzxmJ63A4sNlsef9SsNmSiY9POPO1zWYjJSWlVAr14VYLDRrUZ+zYsRw+dJjpP09n3bq/ytWNDkIIIcSVkuKZEEIIUa5opMVOZuiAyWUdyHXKw74vexPzZVnHUYK0VFa++xgrC1nMGz+dvg2nl0pI4nI42TvrDfrOKus4xPXj+kqofv/9D2UdghDiKpaTk0NCQgIJCQkXXcZgMGA2m3OLbOeNYouOjqZWrZqEh4cTGBh4dr0uF7bk3ELa6Wc8lsQotvDwCNA0UBSqVKnM6NGjSUpM4rdZM1n0+0JyXDKloxBCiGufFM+EEEIIIYQQQgghhChFLpfrzEgzX0axRUfnjl6zhFmKNIotISGB5ORzR7Sd3W7BoiKjUPLWd/p/a7iVwYMH81DfvsydO485c+aU2bMnhRBCiNIgxTMhhBBCCCGEEEIIIa5Cvo5is1jCsFrDsVqtWKwWIsLDsVgsVK5cmSZNm2C1hmM0GM58Jtvp5GRiIik2G8nJNpKTk7Al2ziVlEhE5IVT8CqKggKYzWYefPAB7r23J7NmzWbevHnY7faS2HUhhBCiTEnxTAghhBBCCCGEEEKIcsrlcnHy5ClOnjx1yeUuNYqtQYMGWCy5o9gKe66aqqoEBgby4IMP0Lt3bxb9sbA4d0cIIYS4KkjxTAghhBBCCCGEEEKIa5wvo9isYRam/neqT+tTVRVVVenRvceZ1/z9/K44TiGEEOJqoCt8ESGEEEIIIYQQQgghxLXOHBrs03Iejxev1wuAI9+0jTq9WiJxCSGEEKVNRp4JIYQQQgghhBBCCCGIsIYX+Lrb7UJV9SiKQnJSMtu2b2PHjp3E7Yrj6JGjzJ8/D4DMjMzSDFcIIYQoMVI8E0IIIYQQQgghSknXO26nVYvmZR2GEMJHYWFhZR1CqbJYrQB43B5UvYrX6+Xw4cNs27qNHTt3sisujpTU1DKOUgghhCh5UjwTQgghhBBCCCFKSUxMrbIOQQghLiowMIhtW7exfccO4uLi2L17N06ns6zDEkIIIUqdFM+EEEIIIYQQQgghhBDMmjWTWbNmlnUYQgghRJmT4pkQQgghhBBCCFGC1qxeQ7fVd5d1GEIIIYQQQggf6co6ACFEfipVe73P3D9n8koLqW0LIYQQQgghhBBCiKuB5KwKp6Ni97HM/nMeb7UxlOB2rs62UKNa88yH01j+1yb2xW1h+4qJPFRZyg/iXIqfH8/dYeXX1n4YyzqYQkjvFaJI/Lhp6P/Y/PfvfHC7FaUEtmCuXJ/6lUPxV0pi7eLqpHHTQ/+yefp+PrjFXSL9SgghhBBCCCGEEOJKSM6qMApBlerRoHIY/iV8iMquLS6SGzU2YNjXE3mpx01Us/ijV42YrDpy0jXUmAH8tDaWtRN6UUWqEdc9RVWJseqx6JW8/qPQsImF+Q+EM6qK7qrKi5ZYd/VvNYxfFyzm742b2BO3nf3bN7Jl9QLmTP6AVx/pTN0Q9TLXrNLg0UksWjqVZ+tc7jrEZdEFU/fOJxn3zQxW/RXLnrh/2PnXH8z/YTyvPHwLlfyKsrLy246KoqAoOnRX05kszgjtcJRds+NY9Xhm0eal1WUxYuIuDs44Ss+gkoru4hQ0FAXpV0IIIYQQQgghrntB3Sewe89W5j1dk3OzRnqq9viQVTu2s/3XF2gVenX/ER3UfQK7d//N/BEtKDhUA+3eWc2BuAWMaly+8mMXU7L7XH7ziT4r1vxrySgoN+rXsg996xjJ2TeD5+5uQ936TWjUaQyL0jVQFHSKgk69us/Xa51ftIkvuoczt08kyx6OYuVDkSy6P5zvu4TwXF0/KpXxIEaFoherYuqF8mPPMPqHlUREJfjMMzWiFo1qVeTM+awGEhpZjdDIajRu241Hn9rG1NdG8N6S47iLtGYdoVXrE1MhBaOcb6VGCW7M4x/9h5HtotHnO+5GSyUa3FKJ+k2D2PP7eo5laz6usby2YzabPuvDjZ+VdRziYuyH/Tii2alWNRurEshJH7ukEphN3UgNzwl/djlLNsYCts6m6TW4cXppb1cIIYQQQgghhCgvVCrc8RY/vHsn1n3TeOKJT1mf6mseqgwpATR47BM+jx/I4/89QE5Zx1MaSmyfy2s+0Te+51/LLsaCc6M6omrVJETJZvWUz1iwLxUNyD6VnPv23h/p2/rHMohV5KcG6Kkbop6dKlFRCPJXqeWvUivKnx61s3h7STqrMks7Mo0dW21021rUzymEmA1UC/KW2PSPJVxPdBP3RV/u/2IXTowEhUVTq3Fruj38CP1vbcIj//ka5Ym+jP3LTjn4UXdVCY+IYOhzQ1ixciXr/1pPZmYJ9mpdBXqN+4JR7cPQkrYw7ctv+GXZFg4k5mAMq0S9Zm25s24Ca9OkFQU0aNCAHt27s2LVSv6O/RuX21Wq2/cc92d3FtSq4iRGhZP5qvO6SBs/T0rgpiOR3DUinH2es++pVbKJ0YPjkD+HPReu91qkM3qwmjXcGSopzrzfyhQPtw48zIRO8H8fVeW9rdfonVxCCCGEEEIIIcoRHRHtR/PD+B5EH/mVZ5/4iLUpZZeH0vmZsYb647ankJJZ2LAADY8WTJuXP2X0wX6MXZd2HeRBr8d9vkJlkH8trB/73s8V/PyNKFoWSUkZ0tZF1KJFC9q1b8uqFavYvGULbnfRhhoV1b5tyTy93U0O4GdUqRzuR68mJu6yBPB8Yyex63Mo9XEFV6kSH4zndWWT49HQyMaRdJh/lh3mn+W/s3TEFKY8Vod+o/ozo9ckdnlAsd7G6E+G0bVOJaKC/dCykjjw9x9M/mQis/acd+KptRk6ZxtDT2/n1C/07/g261xFXE85peoUmjVrRrNmzXC73cRu3MiyZcvYtPFvclzFW6wIbP0UL91mgaRlvNL3BWYcOXsCZ586QOzCA8QuPLt8sbVjUC26PfEMg+5uRd1IP7JO7mHNrK8Z/81KjuXfRf9KdOj/JIPuaUPjKlYCcJKWeJyDe3ew+PuPmRybenabAdW4/bGnGdyjNfUrBuFNOcymZf/HpC9+ITbxdMVEIaRpH4YNvJ3mDWpRLTqUACWLpMN/8NaAsex/4Bd+H1qBmU904OXV+QIJqELnR57i8R630rBSMEpWCsf3rGLSa28z+7DnuuiXAEajgTZt29CmbRucTidr1qxl+fLlbNu2Da/XW/IBuP3ZcUyhRy0n9SNgTfzZt6Kb22miB301O50rhLPv2Nn3wqs6iVYVthzwJwdQQu2MHp5I12o5RAV50bL1HIgLZvK0SGYd0p1pr5A6KQzrkU7zmtlUs3oIUBSS4kN467UKxFZP4tUeDupXyqFiqIdAPTjtRv5Zb+Hjn8LYkn52+zF9DvJ7Xxcz367Ny5tzC1nWpr5/HgC/HDp0S2JQBweNoz0EoJCWYuTgYX8Wz4li8g4VDdCFZDL4iQSebOUkTA+apmBPMDN2TCV+szm5va2TsGDofouT97aWwRyWQgghhBBCCCHEGQqWW0fw/acPUO3kXJ5//F2WJZ6bXyg8h3SpXM+bLFLa+5Sz0VluZvDrr/Bk59qEGRQ0zYX9yGLGPvIyv524WM7DzdZpX5Bwx3P0H/8W2/oMZ9aJwu/aLXyfDNz65mKm9klhYq/7+c/ufHmte78gdtwt/PVqJx79PxtaMe2/70pqn/MUmE98D/uQ2cx6MprFL3Xk6fn2vHd1RD80mZVjYpj9ZGdeXnV62JZKoxdmMeuJMGY+1YWRK53Fkze8YK90hLd7hZ8mPEDktk8Z8OR3bC9gDERR868FHj8fc4+F9eNLv68Q83RBuVEFdGH0+fYf+pwOyLWRN7o8xrScXvyw4i1arnmFFk/P4XQ660rP24W2ayOb6u/vT4fbOtDhtg5kZmaycuVKVqxYSVxcXInkUjUvuDTQAGe2h33HM/nQoRDTzUStCCNVlBzirQE8Ws+fJhY9NwTq8Ecjxe7k0yXprHSCYtDTsUEQfaoZqRmg4Mxy8/eBDL7amU1CvpB1/gZ6NArinspGqvhDVoabzSe9hJ83crRaQwvfN1H5Y3kS407ka1e9Spu6Jh6oYaR2kILOo5GQks209en8efoUV/Q80i2KR/K+9WZl8cKsdDYXw6Erm5kstTTWT/iAX++YzICYrnSr+zW7dnrAHUrNm2pT6fQ4O1MU9W4bwIcNI3Hd8xLzknw8IYprPeWEXq+nZfMW3NKqFdnZ2az/az0rV61m8+ZNxVKpvqV7ZyJ1OWz57kN+O+LD+orj+Ac2Ysh33/L8jeYzc536V25C9+c+p2nFodzz2kpSNMC/LoO/mcyoFmH55tkNwlqpNtZKNTBunsKU2FQ8AP71efKbyYxsEXJ2/tSo2rTvO5rW7ZowvN9o5p3wADoib7mf/nfVz3eCmImINJDtuEi8fnUZ/PV3jGoZenbdxihqNa2MKctbfMelnPH396dDh/Z06tSJzKxMVq9azdJlS9kVtwtNK6H99RjYtk+Pp3Y2TWp4IT6vRXQuOrfOxJChkhro5M6W2XxzzC+3b6BRr5YT1WNg6z49XkBxe6hZz0klQ956A93Uu9nGh7XcuIZVYl5q7suRTVLo38aZr69oRFg0sjPBUjud7s2c51xog0KzufXOeJpW9dLrVSt7L/G7Y5E+7+dk8JjDjGroyXcuaFijnFijsjHuCmfKDhWPzkXvYUcZ2cyD4tGRnKxDCfQQaiG3f3v9WbzWn+6dYMF6/8tpASGEEEIIIYQQopjoCGk+lCmf96NOyh+MePwNFsaf94e0TzmkS+V6NAjwIWeji6b3uAmMbB+M4s4g+aQDxWQlNNJAdtqls7Se4wsZPdxM9SmPMvbjx9jz6LfEXWpoh695sSIcxyve/yIq/X32sGd9LIlP9KbJjXUxzN9IblkngBtvro9BF0jTG2ugrtqVmwvShdO0aWV0WatYuyW7hPKGCiHNhzL5Pw9Qcc+3DHp2SoGFM/Avev61IL7kHgvrx4X282Kaoag4zttrUGBgIF26dKZr166kpaezYsUK1qxZQ9zOuJLdsJL7zLHTrNEB3FvVkO+4K1gCIccF6A0M7BjGoxHKmbbzMxno1CSU+kGpDF6fTRqgGI0M6RzK/aHKmXUbzQY6mHO/LnQ6V1XPgx3CeDpKd/acVBWqhqsEluzgvDPK7jFwWVtZsSGNfr1uoF5MEOxMR7Ov5cMBPXn1wFESHS4MIVW45fH3mDioA/fdFsb8/7OdvdPBs5fPz7mr4qwirecaoepzL1z+/v60bdeG9rfdVmzFivq1Teg8B1i15ji+zGZ35e2oUnvQGIY0DeTUys955YP/se6wk5B6XRnxwRju6zmEh39Yw8R9ULPf6wxvEUrOwfm8/+ZE5vxzAgcBRN87nj/fuvWcddbqP4YXmgfj3PV/vPnGJBbEpWCseDN9Rr3FiA5deXPkMta8sOjsD2LNzuI3+zJ67jFSPQFERQeQ5oKKF+yxSvWHx/BiixCce2fz3ttf8/vWeLL8LFSpGUJK3i8Y12O/BFDV3MtMUGAgnbt04s477yDFlsKqNatZumQpBw4cKOYtKuza6092Nzv1a2WjXxuAG9BFpdO9DuyeFc3ylid4qm06MbMj2O0B1Bwa1PCiOE1sPpR7OdcyTXz4Wk1ePWokMQsMJhe39DrOxJ527rvZzfwl+rPtpaks/qoao1cYSfV6ibJ6SXPn9RVNZeGEGoxabcDh8VKhTipvDE+gS20bD9ez8MaOQibp9vHzNbvFM7yhh5xjIbz/VQRz9hhx4CW643H+fPrsb29KYAa3N/Tg3R/O/a9HsjUDUDQiKrpxOXO3t2ZKDW6aUqyNIoQQQgghhBBCFJGCX8zDTHzkPhrlrGfsk68y+4KigkrtAb7lkICL5no0d+E5G8wtub2lGe+Or7n/0S/Ymu4BxY+IahG4Cn2aioZj00Se/09jfn35Gf7zwj/cP24j9gITQUXcp6K4gv0ves6qBPf5YnnhbX8R63iAu5rdTA11I3s8gKEBrW4KREFH9ZtvIkq3ixNeIOhGWjUw4Nqxng0OHbUGFXfeUEdIs2f5btJj1Do8lSefnEBs+kWOolq5yPnXAo+4D7nHwvqxEnwF/dybwozzZ+oCFOsFO1ws5+21Sq/PvZM/JDiYbl3v4p4ePUhKTmbF8uUsXryEY8eOFbIG3yiKQqCfjipWP3o0CaKWDlKSXBzRIBoAjTUbbIz/10O6phAeqGD3QI2GZgZEKCQfd/Dh5iw22TXMYf482drMnTWCuGd3NlNToU59M71CFRxJmfxnYwZrUjTUQD23xJgYUt+IqZD4KtcO5vEoHdmpWUzamMHyJC9OVaFiiI60/IV4zc0PvyfzXUqxHJZzlF3xDDc2WxqaYibQFIiOdLyKQmijBxn5aivqV62AxZBBfJIXFT1RFcLRYfPt4lFc6ymnzilWdD63WHE5zEEKeFOwpfo41vFKj79ah+5310WfvoR3R3zD8ry5fE9tn8Vbn7fhjk870bpFOJMOBnNXtwYYPbv57IXXmLrn9FXTQWKy47zpIWPocU8DjK7tjB8+ll8P5EaQeXgd37z0BlXnf0Xf23rQKewP/s+W9xnNTcrxYyRnugAXJw6nU+DdFWoN7u7eED/XNj4Y+jo//Zu3d9kn2bvlZPEdl2uAPq9vhlnC6HZX7sU//kQ8Jw4twemdjb+ueC7+GXsD2eu106B2FhG6AOK9UOPWdJrq/PhsVTCL3ck8+WA6PWqEs3ufghKYxY03aLj3BbA1320PoTE2Rj6eQf2KLix6HfEpCioQFeFGh/5se2mQcspIslMBVE7E5+snGthT9KTnAOg4HmfhvQVpdBjopG51N7odBi55ZvnyeTWbu9o6MXr9+eyjikw9dLogpyMxVXfuuaApaIASlk2L6i727DTg1BQSjxvO37IQQgghhBBCCFGGVGK69SYGL0nLl7LuWHYBi/iYQ9qXlLt8gbkeQFd4zsarabl/T0fUpUXdcPZsPIlTyybxX19zGTnsnfYKY5v/wvj+7/Dq+gcZvbyAKY583ifbhZ8tzBXs/+XlrEp5nzP/ZvnGTLq3bUnLqK/Zc8KLGtOSVuHp7NyZQd2GLWlhns7sNA2/JrfQPMjDzlXrSFRieKhY84YKllbD+OGBftQ++hNPDf740s/oU4KKnn8tcD1X3o+VK+7nPiiu8/Y6oDfk5lLDrVbuvfde7r//fo4fP47HfflZ5NpNraxseuHrLruTCduyzz7vTNNIy/CQ4tYAjZN2QDHQqZoBNcfJF2sz+Csvj5qcnMVn24y0a+tHsyiV/6bpaFdZj86Tw5Q1dhafPu0dLpbuyaZ7PSMNLhWkoqdTdQNGj4uvVqUz+3STezT+TSyFxwLlKcPimR6LJQRF85KZkYmmBNPutR+Y3LcqhjMDMfyoUhnAg07nY6jFtZ5CtGnbhgVt5xfLukqSXn+2WHFPjx5nXg8JCfZ5HRlZgC6E0BAdnCrkxCyO42+sTI1KOnQBdzAh9g4mXLCAh4qVKqDThxNTTcV7dB0r9hdyu4GxKrUq6fAe3cDaQ+ftQ8ZmVm9x0vfOqtSqrIOi/u6hr5oXRyzrjlzk+JRSvwQYPWoUjCq21ZWY030zumIFKlTszyZ7P4LVrYSF/wQkXdG6PacC2XgKmlTPorEB4l1O7mnvxLsvknknFI6tDWZnn1P06JjJ5/uC8NbMoqFB4d+4QE55AcVDuycOMfnOnHzt5aFKNICSb1rEy3PiqJEMzYkpwMvlrOqCz6vZxFTU8CaYWHHk0mvUMoOYFaunQ1s7r7xjZ3i6kR17gli50sKUtX5kXIvDH4UQQgghhBBClENudk8fz6IbHuGZ217l16lVeGHIxyw/mS/34msO6VJ5Bh9zNpp9HbOWJtGhW3tembqE4SmH2fHPJlbO/S9TFu3z7e9pzwlmvvEWrev/h/vfGs3K7WPIOH8Zn/fpMopnBSnpnFVp7rOWxtplW3B2bEb7VqFMm5lG5datqZ61gZGTUhj5+Z20axbA7GU5NGrbCot3L1NXHMNj7Fy8eUNdKJ0fHwjeVJb9/BN/JReS7Ncyi5Z/LUhx9ePi6OeFKY7zthgtWHD15/gBVDW3OHvDDTec83qk0cWpnKLfFO/1amTleIlPc7H1uJPZ+7I5VNiIPlWlsgl0en/e7OPPmwUsEmnSodPpuCEIvA4X2y444X2gU6kWDF5HDpvthS9eUsqueBbQhNtahqDzHmb3vgyw3MMj91ZBTdnAxDHj+Wn9ARKz9IR3epXZn/YofH15FEvnYllPYXbv3s2s2bOLbX1FFRoSwtNPP+3Tsh6PB1VVOXnyJFFRUQCkpfleod/3bxZaneq0ah7BpH0JlxwlUyzHP+8Oh0tsBb8APxSdAb0OcLt9uPvlCqsdl1y1LreYcolpMUurXwLMmj2b3bt3F+s6i6J6tWo8+OCDPi17um+mJB/ixhvmEmFYTEqSEfC9uFvwiv1Zt1PPoA5ZNKuusUJN5Z4bFDZ+G8xRD3hPhDB7dyJjWqfS7r9GvQf+AAAgAElEQVSBHKmbhVXTs2hr7jPQlBA7j3TMQU0PYuIXUfy03Y9Ep0Z4ywRmj0i7stgAzaUjRwNFd3m/cVzweQX0CuCh8HNB07NgQjUcu1Pp2jiTm+pkcVPzFJrdbKeuUoMhq/XX5PShQgghhBBCCCHKH/ep9Ux8ewFrn/qEr4YM4MupoYwY9DrzjuVN3+hrDulSS/ias9GSWPBKfxxbetO1VRNuurERN3WoTrPbOlBX14shC5J8+ntaS1rG26//RrOv7+PNV9fwYdb5C/i+TxpewA9//8vPe5VGzqo497mQLZG8Zjmbs2+lRYdWhMzZRNt2dXD9/Qur1qVwS0of2rVrgt/KdDq0rQAH5rL0Xw8YizlvqDnYvHgT4be1o8Pr3/FR5mMMn3+J6Rg9x4uUfy1IcfbjS79fDHPjFVt7F4/3x40rpS0VrF69evS85x6flvV4POh0OjIyMjCZcic+LGrhbO8/yQze4S5yHwNAo9DrnJ+qoChnn4emu+TSF3P2OWllmacsm+KZEkKrISPpfYMO994/WLDLg65WNNFGyFw8jQlLduc9MM5FcmI65w7K1nC73WgEEhh44SmkC/d1PVcmKTGJNavXFOMaiyYqKvKSxTO324VebyA9PZ0VK1ewevUadsXtYv78eUXe1l9L15N+RydaDR7K7YtfY9Elhkb6fvwv0Y6u4xw67sUbMofHbx/D8ovNp6u/kRNJXnRVbqZFRR07jl7ilM85zIFjXnRVW3JrVZXtB/P9uAq6ibY3+kPOEQ4e81LkUzovXl2VFtxSWWX7+XeocDn9UkW9zLNz9+7dZdo3MxwFDL/Px+12o9frSbbZWL5sGUsWL6VZ7f3cPfZ0QbdSMUShsGVrIFmd7LRq4qR9VDoVnSY+Wps3RaLXwIKlJoY/Z6fPLU5WNcxGcQazbn9uX9SFuog2QOZfFiZs8M9rL4XkFLVYryPFxq3nRCroojNpEQk7EgpZPtvIyvmRrJwPqB7qdopnylPp3NY6k8DVwRfeASaEEEIIIYQQQpQVbyp/T3qaBxLe4buxPfhkqhH9I6OYdcTtew6poMdw5ClSzsZ5lJXTPmHlNEA1U/e+sUx5swu33dGCwAW/+/j3tEbqmk949ecW/NB3BM+fDEQh303uRdgne5oDTXcDdWqFoPyTfFlJ5tLJWRXXPusvmRcG8J5cxvy/X+KWVl1oX9NMl8YaG99dS0pmJkvWpnNf+440n51O52oaeyb+yV4PxZ831Fzs//VFnpzxPFMn9KPHO58Sf/Ixxm+0X6SNMouUfz3rbFsUaz++5Pt/+BBXIYrhvC1OZZlHBdApOrhE7czldmPQ64k/Ec+y5ctZsWIFAwcMoE3bNqUX5GleD8czwGvMYtScdP46/zGUpykGjjhAF2ykVYjC7tQiXp3ytqMzGbnJDHsKHAek4dY0QCGghKpcl1f4K8oG9AZUBVCNBIVXpUmHB3l18gy+H1QPf9chpo+byi4PeJNPccoFAS178XCzipj0CugMmEz+51X4NE4lJKKpFejSuwvVTXpUfws1b25ARbUo67n2eNy5vdXpdLJm9Vreeutt+vXrz9dffUPczji0S4yMupSURV8yeWc2uoo9+PSXSYy4tzk1wwPR61SM5khqt7ybp164l3pFOv6XaEf28Mfig3jDu/Pm+EF0qh9NsFFFp/oTVqkBtzWvmrsu9y4WL4vH63cjwz4eQfcGUZiMfoRVbU6vLvUwnnNw9jJ3bhw5hkY898kY7m8cRaDeSEjV1jzx4Vv0qaCQunIeS2yXcYw8e1j05wE8hiYMmzCWfi2rEeavohpMRNdpSh2rrkj9MsflRlNCuem2VlQKLJ0fEiXN7c4d85uWns7vC39nxMiRDBwwkO+//4Gjx46WyDbTt5n4O0ejbuuTPNvaTeLaUJbku9Amrg/lj3Qvbe9O4P7aGlk7TMTm/TbhTTXktlejVB6u78KkAjoNU6D36ryOeAJYHGvEa8xk2PAEutd0YTJohFXIoNctznPPBTWHjp3tNI70YNSBqgd3po5sBRRFQ1E83PrIQTZPO8grTa71p/AJIYQQQgghhCgfsjkwczQDRi/kZPSdjPvmVTpaFfD4mEO6BJ9zNmp1Ot7XkcY3BGPUKagGPW67nWxAUYo455FmZ92nY5l+NISKFf3P/azP++Th4PZdpGt+tHlqNA/dGEWgqkP1NxMRFuBzPKWWsyqWfb50Xjh3kUSWLIgly9SGx97sTXM2sWhFMhqZrFu0mrSozrwwqhs1td3MX3QwdzRYSeQNNQ9Jq8cz8MWZHDbUY/CHY7gz4mKpeK1I+Ve4sC2KrR8XZz+/mGI4b691p/P8qSkp/P57bi718cGDmT59OidOnCi7wDQXK4+68Qb48/ytQdxqUTGpoFMUQkwGWkWquW2nuVhyyIVbZ6B/+2AerKgnNG85c4AOfx+2s+KIG49q4NF2wfSMUglRQadTiAgzUCNvBcmZXryKSpta/lQ2gKrqqBppIKqYhi2WcD/UU3/Ib+wZcv7rGp7U7fz42nDeXZeeW3FPXsaMpUNo260jr0/vyOvnLO9hb76vj6xcRtzQRjTu9RHLeuW97PqH9+7qz7dHfV3PteF0QcztcrF+/QaWLV/Bli2bcbkKm6C0CFy7+XLoy0ROepeH67XlmXFteeb8Zdw70c2Zy65/i6cdJ3/3LlM6fMngLi8yucuL54azZTxdHvqRw14nG7/5iLmdPqJnkwF8PnPA+UGds+1908byabvJjGjemw9/7c2HZ97TcB1fyJsf/MHl1M7Azc7v3uHrdpN4pmFP3p7ak7fPrDqDeUPbMXSx78fl2K7dpGp1qTvgS/6IfIkGwxZdTlBlzuv1oigKTqeTVStXsXzFCnbu3InXWzoPddTSTPwRp6PdjZk09vrx1eIgzrmRJdPEzyuM9OyZRSNNx7INQZy+CUJLMzEjVk/btnZef99+XnspV+F1RGHjb5HMbXmcnrVtfP7J+RNwn/2JoYRkMOjpeFqff/X36ln4VxAZOie3t3USFgzdb3Hy3tagEo9eCCGEEEIIIYQonJsj88bwRFQkPw+/n08+OUjvx6exw6cc0sXXqvmYk1SsLRn01hhanz9DmtfGwj83FnkWF80eyyfvzqLjV/efNweP2+d9ylg9jWm7OvNcg66880tX3jlnyRzf4ihCTvb8nFXzYYu46KChEtnnQvLCR7yARvKSWSwd2ZYezeqSufJ1liblJnwy1i9iacrd9L5RIWv9j8w9M3tUSeUNvSQue49nPq3GjOFdeeft9Wx/ZibHCuqPRcm/HiygLZ4vnn6cae1UrP28YL738euJx+NFVXU4HA5WLF/BypUr2bV792UPiCkpe3fa+fWGUB6sbGJcZdM577kS7fT/M5PjGvy7O51vK4TxVJQ/z3b059nz1lPYFWpfnJ2fK4bSzxrA8C4BDD/zjsbSVYm8eUTj+PFs9jU2UK9mCNNrhuS+7XXxxTwbvxTDs9JKbOSZJ3E/2w/Ek2x34vJoeF1ZpCcdZce6hfzw4Yv0uONh3lp8/Gx5Q7Ox8LXBDP9uOTtOpJPt8eDOziDl1DH2bt3A+v1pZ4a1evb9yNAR37N8XxKZHg/uzGQObtlPoqIUaT3lndvtZvOmzXz00cc80Pchxn3wAbGxG4q3cJbHc2IJrz/Yi4HvTGPR5n85le7E4/GQlX6SA1tX83/f/szaFG+xtaNm38i4fg/x/KT5rN93inSnB48rg6TD21j599EzJ5c3cTEjH36a8TM3cjDZidvtJPlgLHOWxJGpgVfLd5XNiuOrwQ/xzIQF/H3YRpYrh4xTe1n18/v0e2AUc09c/igbzbGJjwf2Y+ik3/n7UDIZOR5cmTaOxm3iQLoBpQjHJXPlp7z4xRJ2JNg5fiz+smMqSznZ2axds5axY9+hb9+H+HzCBLZv315qhTMAND3LNwSQrYFrfyi/Hjj/lgOFf5aEEucGLSeQxZvzPetL07NwYlWGzzKzI1El2wPuHB0pNiN79wax/qh61V1HvCnBjBxdhfFLAzmYqsPt0ZF8LIg5G/xzz4W85RTFwOa/AzicpsPtBU+2ypG9Zr75rCojVunRvP4sXutPisOfBesLvQ9ECCGEEEIIIYQoRU7ipozkjcXJmFs+zydPN8DPxxzSRfmYs1GUeDav2MZhWxZurxdPVgpHti3hm5cHMWJ+4mXkCTTSVn/Ge7+fuuDZQ77mxcjewedPPMm7v23kX5sTj9eD22kn6fg+Nq9ayMr9WYXHdQU5K9/Kc8W7z5fMC59eV/pqflkQj0dzsHrecpLO7MAG5iw+hcdrZ8WMhZzIH0SJ5Q2d7JryGuPXZRDa/kXGdI+6aELe5/wrBbRFMfVjir2fF8znPn6dyMpysmLFCl5//XX69n2IL7/6irhdu666whmA5srhyz9tjN3uZEuqF4cHPF4Nm93FhlMezlQm3G5+XmZjxOYsNqZ4cHjA69XIyPKw72Q2C4+7udisj6e38+0SG29td7It3UumB1xuL/G2HA7n5A4V8KZmMnZtBn+lenFq4HZ7OZLo5vxhBZdLCbFGnNMCCxbMByA2diOfT5xUTJu5dvx36vdA7lyoZfkwQaPBgJ+/P3Z70Uqo10f7KkT0+YbVY29mw5hOPPKr7aordBS3Fi2aM3RI7v0o748bV6Zz9QYFBeH2eMh2On3+TM+O2fyQ98yzZ8dX4ve1wSUV3nUn4vYjrH4mgw1fxPDIYv01fy4IIYQQovR9MfIYd92a+7tcaJuIMo5GCCFEWbg+8k1CiGvF0CHP0KJFcwC6dbu7TGMxm81kO53kFGFAzOhRo84882zUvqolFdp1YVzMYaCAeo/Cr9f79KHlVo7LVaQT6lqli2xG1yawb+e/nEhKw6kPo0azHrz0dEuM3gNs2X7tjDQsLzIyimcAtyganSWTrrVh3wEjJ1JVnKqHGvXTeKlPBkbNjy37r77RckIIIYQQQgghhBBCXM+KOjhGlB4pnolyza/pg4z7/C5M58/Ip3k4Mf9Lpu+9/KkYhShP/OrYGPdyegHngsKJVeFMP1xMT8oUQgghhBBCCCGEEEKIa5wUz0Q5pmBM3cPy2Bo0jalCdIgfZKcRf3A7q+f+yMSfNnDqOnuwpLh+Ge3+LN+RQ9MqOUSbvOBSiT8WwOoVVib+HiTnghBCCCGEEEIIIYQQQvhIimeiHNNIi53M0AGTyzoQIcpc2o5whr4WXtZhCCGEEEIIIYQQQgghRLknxTMhhBBCCCGEENete3reQ/269co6DCGEACBu9y7mzJ5T1mEIIYQQ1z0pngkhhBBCCCGEuG7Vr1uPNm3blHUYQghxxhzKrnh2662tCQwMJC5uF8ePHy+zOIQQQoiyJsUzIYQQQgghhBBCCCEElStXoX//fgDY7Q527NzO9u07iNsZx8GDB/F4PGUcoRBCCFE6pHgmhBBCCCGEEEIA/QY8WtYhCCGuU/+d+n1ZhwBAcnISXk1DpyiYzSZatWxFy+Yt0KkqLpeL3Xt2s33bdnbs3Mnu3XvIdjrLOmQhhBCiREjxTAghhBBCCCGEEEIIQXJyMjpFOfO9oigoqgqAwWCgUYNG1Ktbj4f0erxeL4cOH2LbP9vKKlwhhBCixEjxTAghhBBCCCGEEEIIQVJy8qUXUECvz00n6nQ6alSvQY3qNc68HRoaUpLhCSGEEKVGimdCCCGEEEIIIYQQQlxHAgMDCQ8Px2q1YLFaiQiPwGIJIzoqyud1eDUvCgqnTp0iKu9zqalpJRWyEEIIUaqkeCaEEEIIIYQQQgghxDVAVVVCQ0OJiAgnzGIhwhqOxWrBag0nPNxKmCWMyPAI/Pz9z3wmx+UiOTkJW7KNxMREPG43qv7iKUOvx4uiUzh+7Dgzfv2VlStWMnfunNLYPSGEEKLUSPFMCCGEEEIIIcQ1ITAwEKfTidfrLetQhBCi2BmNRiwWCxarJfd/iwWrxUJ0VPSZ1yIiIlDznlEG4HA4sNls2Gw2kpKS2bt3L8l53yfEJ2Cz2UhNTT3nutmgQQMiIiIu2L7b40Gvquzbv49ffpnBxo2xaJpWKvsuhBBClDYpngkhhBBCCCGEuCa0bNGSRx4dyNy5c/njjz9xOBxlHZIQQhSJ1Wqhe/e7CQuzEBEejsVqxWK1EBkRgX++0WIut4vkpOTc0WLJSezds5dTSYmk2FJISk4iOTkZW5INl9tV5BgSE5POKZ55PB5UVSVu506mTZ1G3K5dxbKvQgghxNVMimdCCCGEEEIIIa4JZrMJq9XKwEceoX+/fixZupS5c+dx5MiRsg5NCCF8Uq9+fWrGxGBLTiYhIXdk2P79+wodLVacTp06Sb369fC63Siqypq1a5jxvxkcOnS4RLYnhBBCXI2keCaEENcoJdjOa6+c4vb4CDp/Fkx2WQckSoxqzeDJAYn0vtFJJbOG02bi/VcrMz2h+Lcl/UqUtYq3neCrB7PY8nUN3tiilHU41z25Jlx9qnY6zoTe2ayfWJ33dpTeOeLw1ON49sM880wOdruddLsdu92Ow+7AYbdjtzuw29OxOxx4PJ4Si8NkNuN2uzEYDKhGI126dKFr167s2bOHmbNmsW7tOpnSUQhxVVu3dh3vvvdemcaQlJSMx+3mz8V/8tv/zSQhoQT+sBBCCCGuchctntWqVYuhQ54pzVhEKZL2vbaEhYWVdQjiKqQYXdSPySYiBSS9fC3QuOmhQ0y+28PiCdUY9ZceDcCQxbAxRxhSXTvTzqYQjRwHqFVsTH37FNV2RdN3fChHiiFXKP1KFO4ifbWYBEU6qRflZpd0wHxK9phfilwTiqqk20rDHOWkfpSHf0q7QTQdGnqqVo3GZArGbDZhNpsxGo0XLJqRkYHD7iDdnp5bVHPYcdjtpKfnFtcKKrql2+0+Fb3MwWYU5ezO6/W5f/LG1KrFqJdHkZScyPx5C1i0aJFM6SiEuCpdDQX+lStXMnv2bFJSUor8Wck3CSHKg1q1apV1CMXm4ejEsg7hmnXR4pnFEkaLFs1LMxZRiqR9hYCg9kfZ9KKDfT/VoOcMP0ruHuii0Ghwz1E+7uZl3ntV+eJQWaQiSyqGq2HfyjcFDUUBXb5D59cohb5VNXKOhPHSh5EsPq7DEOKBDMCqoQN0urKKuHj5Nz7FtCfSqG7xYA7wonp02O16jhwKIHZLML8tM7P7svKgpdc3g9ofZdMLGeyfVZl+U4NIvSBrrtFuyF6+76Ty7cs1Gbe3fJ4nBfXVq1v5vz6Vi2Ou81D3lhQe6ZRO61o5RJk13HY9/x4OZN1foUxdEsSxHF9XVn7brFy01WUw6XdSVz+KVi9HnPO60WjEbDbnFdOC874++31w3vdVq1YpetEt//95RbeqVaqgFvCDT6eqAIRbwxk4cAAPP/wQy5YtZ87s2SVzQIQQohw7ePDgZX9W8k1CCFG6GpkzyzqEa5ZM2yiEEFeZ0ApOYiJUjGWYVCupGK6GfSu/FDZNr8GN0899NapyDiGKwurZkSw4oqIB2ba8H++HrfQdaC31SEuKGpZNo8ou/E6/oPMSaskh1JJD45vSeLR3AFMnVuK9DQbcRVx3qfZNxUuDnsf4PKkajy/ww+daQblRcF+92pXv69PVf8yVoCweH36MkTe50Oc7xsYQFw0ap1G/jpc9a4pSPCuvbXb1t1Vxy8nJITk5meTk5CJ9zmg0YjKZMJlNuf8HmTGZg/K+zn3dbDITGRlBjRrVMZlMWC0WVFVFucRdI4qioCgKRqORzl06c8cdt5OaknqluymEEEIIIYS4xlxQPOvW7e6yiEOUEmlfIYTwnc7owWrWcGeopDivzuysn58XRdORlKortWnailvRjrNC3P+qc////HFqGkHBLmrVzqDbXUn0b5rFIyOOoIytxtht6lV9PDyahzaPHmP0sWqM3Xp1xyrEFdO56DXsKKOaudFSA5k2I5xfYgM4kKLDGJxDvfoO7qxmYK3MoCfyycnJwWazYbPZivS5iRMnUL16dZ+W1asqmqYRZjk7Bbpeb8DtdhVpm0IIISTfJIQQpen9ceNgXFlHce2TkWdCCJGPtWkSr/ZwUL9SDhVDPQTqwWk38s96Cx//FMaW9LPLWhonMaq7g4ZVcqhs8RCgKKQk+rPhr1C+nhXKdvvpJTVufWofU+9QmfhiDf7z79niQEjHo8QOc/DXhNo8uiRfAl2XzdBPdzE071uvLYz+gyuw7iLDaXQhWfTtk8jDrTKpEaqRmRjAX9tVos678VoJtTN6eCJdq+UQFeRFy9ZzIC6YydMimXXovOLLJWIo0nouCLag9UaT88h+frnby+/jYxi6Ll/gOicvfvYvz4aG8MTjFdlSx9fjnrfPAdl0uy+RQe0yqGvRyEr2Y82yCMb/ZuLYRY9nJoOfSODJVk7C9KBpCvYEM2PHVOK3xKK0fRFj8Muh8z2JPN4+g4ZRHpRsleOHzEyaWIHZ8RDT5yC/93Ux8+3avLw5X5FJ56HP67voc/p7dxBvPFmVae5Ufph8gpZbbqDFuyGkFzEeX/vV+XztH4Ud54vxuhVyPKCh4Eg18k+skX82BrN04GGm9HTS7zEbM16MYJe3GPv8ZfSjSxwhti6IIKH1Kfq/EM+2EZWYdckpyot2DbngOmaA9FMBrFgYwdd7XfS8K5XbGzmpZIaMU/78OTeKcQsDzplC0pf9DamTwrAe6TSvmU01a+55kBQfwluvRbP/jn8L7quF9HHfD6FG7Q7xfD8gg5sruVGzDMRtDWHydCuLTpzuoCV77b34/ldgYVrxnfe+tse51we4edABfunu5o+PavPsmnwnreLmgTf28X7DQN58qipTk0r+mhDYJImXbnZDqplXRlViRvzZtsi2+RG7xo/YNflCLO3z1i+HDt2SGNTBQeNoDwEopKUYOXjYn8Vzopi8I18f8cvh9p6JDL7NQf0IL950PzbFhjLpf2HEpuTrYyVwflzRz97riMlkvuT7bo8bVZc7hePhw0f4+++N1K1Tl4aNGua+L4Wzq4Ya1ZonX3qa3m3qUylEhzP5EAveG8zohUnS34uFStVe7zDhqTqsf60P78UW+Reaq4SOij3G8tVzTdny1r28seZS5/C1ss9CCCGEKA1SPBNCiHwstdPp3sx5zsUxKDSbW++Mp2lVL71etbI37+Fo1rrp3Nsi/7Ia4RUz6XZfJp1bZ/L8qxVZVLQZii6LEpTBa+8c4ZEqGqdTcH4VMrmrQu7X2fkXdnuoWc9JJUPe94Fu6t1s48NablzDKjHP11mLims9Z/eCHZuDsHVLpXnjLIzrgs5MZaezZtKiokb2lkA250B4UY67fxZD3jzM83W9nM7t+kdn0b3vUZpGVuKeiWZSzs++6Fz0HnaUkc08KB4dyck6lEAPoRbIzhsVUaS29zUGo5PBYw4zqpHnzHIY3NSqk4PpnEa8Qj7GU6R+dT5f+ocPx7lINJX1P0fxa+vDDKiaTrfq4ew6oBRPX72cflQIz6kQRn/sofqbyYwdnsSeN8KJK6Z2Lug6Fhadyb2PHube85Y1VsjkgcFHsGTV5Knlerzg8/5GNkmhf5tzz4MIi0Z25kVGDxZnH1e8NG2Xr/EMOTRrm8iNTTJ565UqTD1a8iNFL77/FO95f1n9T2H7piCS7k6heZNM/NaYzp6zAZm0qa3h+dfE6hTf13/51wQvt7RPJ1KnsGVWFL/F+9A2pXne+uW1QUNPvmeQaVijnFijsjHuCmfKDjX3uah+Tp58/TAjG+ZrL6uT9l0TaN0si+Gv3MC8vEJ4iZwfxf6z99pkMpnO+V7TNDSvF52qYk9PZ9u27WzesoXY2Ngzo9pGjxpVFqHm8eOmoVOZPMDM4lf6M+rP5BIsDJXmtq6QsQHDvp7IkHp+Z645poho/LId/9/efYdHUa0PHP/ObEtvhISEXqSHKqGJNAFRkCJyVRC799pQQVBUFDtYuIpiRVARfl5RUToiQmgiHUIvAQIhIZTUDZtt8/sjAdJIZpMNRd/P8/Dca3b2nPeUObs7Z86cqzdmr6v89gqs2ZSmNQPZplzpJyxUpKwK/tUb0aRGCHt0FOPqKbMQQgghrnYyeSaEEEVpBhZ/VI/nV5vIdrmJapTOK6NT6NXwLMOahPHKTqXQsYs+qM+Y1UZsipuo+lnce18KDzZN5437AvhrcpDHF9UBcFuYUmSlxCWCpfmgFEbU1MjcF8YrX4az7LABY5iN7rek8tIAKwXvv9ZyAnj3pfq8eMzMqXNgCnDQcXASHw/M4vbrnSz43Xjxh2opMXiUjt6y7QokLjOdQW2yaGn0Z2P+jaABjXNoblDYFe9HhgbhoLPeNRr2T+GJRhqpmyN4YUYo604oBNfLZMxTydze/RTDfg3k48TCYSh+Vno3d+E+GM6QlyPYbgUUjarRThy2gpXg3Rjq3pLCqOYubEdDeOuLcBbtN3HO5KJWTSdlbsXiNvBD0RUMgBJSrOV0xuNZvyqWi47+gd569kSuHyvjDQzvaadJLTccMnihz2s0HOR5P9Ije3cET39nY879p/jvcF+GTPcny1tXxS6MY0asmptGXZP5/LFMonP8+eTjSGbtsHBGcxI7OIlPhuTQtWcmEXFhpLg9PG80A8s+q8O4lWbS3W4iq7jJcEJ0CSFVqI8XK5/C4bURvP5DMOuTDFiq5jBoeDLPd7Yy5t4MlrwZQmqljr3n4yip/BoNB3rrvC/fOAaQuzeAddlp3NbCSowhgE35N374Nsmmg6/Coa3+JLo0Gg6u5DHB4KBpbTeq28KqrWZceqr1Mp639W9NZnRzF/bjwbz9WVV+3WcmGzfVeiTx26OFZ/Ib3JrMM81c2A6HMuGTcBYmGDFXtTL0gWTGtMtgwn2BrHmvwOe+l8+PCn32/kOoqoqPjwVN01AUBbvdzvYdO9i8aRNbt27j+PHjlyUOn/ZP8vWLt1I7MozQQF9MOLFlp3MycT9bVi/km28XEt0SxugAAB+uSURBVJ928WzI24dNLTCBW3kuZ14VYWk/lLsambEf+IFnn5nCsoRszFWjCcj25h1FV79rpb284Z9UViGEEEJcG2TyTAjxt9L4zgQW3GXDcP4PbgtTn63He4c8+BWmQVaakUw7gErS7jDeWphB93ttNK7rRN1pyludkX9sdqaBHFf+sfuCeftNlfCPjzGwfQbdAoKYm3WpjLxAzaVPBzuq3Y8P3q/Gryfz/57qy/wFgdzZ30rrIm8Jue4sYx+y0jTaQZhRJTlNwQBEVnWiYtR1YdOb6VyQ68+iTUYGd8+mVwONjXsVQCOmeQ6++RddPap3ay79b7RhtAby5uRwVuRfA009EMKr/5dNnzFZdGru4JPEAu0JoClogBKaS2xdB/t2mbBpCqeSTBTizRiS3PTreg6L05dJk6KYlZTfX+1G9u/14ke1qjOe4y6P+1VRZfUPt9569tDZTAOa4sbPz42KAbeOWErtq3rrrGg/0kVh/8JoXmt+mHf6neDFHfUYt7GM59/pdWEcUwADu1dEMKtPFmNrGNi9zYcUG4CJtT9WYdnNOQysZqeWAikelPd8PmmpZs7Y8vI5kWwoOR5Drnf7uKaycXkYK47mpXMu2Z8ZU6Koc91R7mmRRZfAEH7KLCMNbyip/KrNe+e9p+1RkM2fxVsMDLgxi571I9i0P29MbdPOSigWZv1pwXVZxgQ3gb4auA2c9eDz8LKct0lubuliw+z24cP3ovn2yPnvCiqniu4jqdq4rbsNs9OXd96vxpz81Y05yQF8MTmK2h8lcle7dHoGBfFjRv57KuH88Ppn79+Mr58f+/ftZ8vWLWzduo29e/ficl3+WjFENKRVo5pYLvzFjF9wBHVjIqgb05kBA7sw6u7nmJ/sBnLZ/OFQWn94OSK7nHlVhEpkg/oEK7msnv4hCw+kowG5KUepzK/VV59rpb284Z9UViGEEEJcK2TyTAghdDhxzIxVsxHg66bMtWDZ/izfozKwg536ERqV+ivfZKdOhIY71Y9Npe6ZBCgubnzkCNNutmO6UAgXtaoBKPrv8vRWOsWo/BkXRHL3s/TpfI539/rhUG10au5COxHCyjJuFi9W78ft1IvUUC2ZfDRrNx+V8J7oSCcqhSc9tBx/5m4w0r1LFi+8kcXoTDM79/kTFxfG9LUWrKXc1l/uGAwOrovWcKf4s07PI83Ky6QzHpNDf78qSmf/qEg9lyYsyIWiQc45Fc0bfVVvnVGeyTPAZeLnT6Lo9N4xhjyaQtyBaKzlSUdHPkdPKVDPSWQgcH51n8PE8bMKSpgbXwUw6i+vRwy5ld/Hc33564DKPZ3s1KmqweWYPCuJ3j6j57z3oP8Vp7J2VSBnbszgpvY23tvvi8uUQ6/rHWiHQ1mYqFyeMQEFq00B1UVIAHC2rMMv43l7oQ0CWJlYRsImOw0i89prbVKRY8/5s3qvyl2d844hw8M+ruf8qLTP3r8Xa3Y2o0aPvtJh5HOy/YMh3Pn5QXIVC4GhVakd05URT4/k9sZ9eGLodBZ9uOcfP+FZMgWLjxlFO8fp09Z//IpKcfVRLYFUCfHBmZVGWo7smyaEEEL8XXnp9mYhhLg67P2+Hg0GNKXu+X+D6nu26uwSNIeKXQNF1ffzXcu/in7hyU0AmoaPucKhFHN+IC+rlEpwFvf1sGPI9Ofjt+rR8a4mNBjUmA4TQ0jx4MqNt9IpiW1XCL+cUKjeMZPrzWCobqVLlMbxTYHs0TEzUWK9l8JiKWEyVDOy8KM6PPBlOP/b4Eei5qBNuzRGjT7KOzc4y548LU8MCpftwqfeOtHbr4rS3T8qWM8lB59DtxgXqmZmb6IKXuqr5epHHtDSA3n9k1COh2Yw4eFMIkpIzBtjiMMBKBqmgrdOKQoOF6BcbPNKKe9l6uOKeiE7oHLH3tJ487yvSHvk7AxmyRmNup0yaWEAS5NMeocpbF8VRILLs/TLOybgMnMgSUFTc+nQzFHmjw9vfcbobQOjAri4shMYOvpCZX72isqi4cq1YXdraC4bmaePEb/iO175fB02DQKDAvLPJwPXPTqHA3vWMKnL+YlwhSqdH2HyF7NYsnwVO7Zv4+DubexcO4/vXr2L1qEFO4wnx1Y0r3w+Nej+8Ot8t2AFO3bs4MCODWxaPpcfPn2dh2NDSh8nfOvQ+/FJzFm6ml3xW4hfNZevJwwjtmrRlZkKqKEM/XIbh/ftyvu382tGRBUdRTyJ30TnCSs5tHsuzzQ2FEojeNAn7Nu3la+HhOXHX1K6m9ny+3dMfqADjdreznOTv2X52g3s27WZLb99w8RhMYQUKbzi34B+z0xm7vJ17InfzJbfZzPl8a4X9y5EIbjVv3j5v18x/7c44nds52D8etYveJW+YSW11/l6rMVNj77F94vj2Bm/lV0b/uC3mRMYWDuvXEqVbrzwzVxWr9/I/t072Lf5DxZ9/hyDG/l7MI4buX7cUg7uXc/UvkUezqtU4V9fbSYhfgYjolSd+XleVo/KoZhpOPAlZvz6B/Hx29i9fgk/Tn6cm+v46ipt2W0Fatj1/PuDn9m0+U82rFrJ5i2b2P7bu9weLZfWhBBCiL8jWXkmhBDeZjlHbAM3OMwcSc17TFZWtgFNddCopgtlX+l7kjhdCpqm4eejIy+HmUMnFdTobLrVrEr80Uv/HFZDHFQzQc6fYXz0lw92ABTOpBkountEaTF4ko7HZXP5MGepLw/dn8mgFhEci8qmkWJmxjofyryns2i95/+vOyCYh/4TzQpP9tHKNRO3IIK4BYDBReOeyUz/TybdOuXgtzrI+zGoNo6kKqjVrHSsphF/opJmGTyIR2+/KvZWT/pHGfXs0QosxUWHO09yRwQ4jwaxMEFBreWFPl+RfuSB9K2RvLjYytc3n+Tps1qxC0KejCEVUlnlzU+3Mvu4EmClZ5O88y+hssfe0njzvK9oe+T6MWelhbsHZzKgSTghXbOIsPvzQVz+3mOXYUwAlT83BJDZKZMOt6fSe311lqSVcrQ3Pqv0lsuQw4l0UKvlEBsBO1NKKcb5z9soK52jNeKPF6gDXytdGhfsex7ScX6U57PXcImnRIorQDXi4x9CjabduP+hjljcp1m9em8pk7YqYS160b9r00I/2P3D69P5zhdp1dCXwfdMZ7/T02Mrmhfg05iHv5jG87GhBSZ9/alSoyFVatTDvGU60zekl1w2n6b8+4tpjI0NvjiRHtmQrneNo9ONLRk9fBzzT3g6G1zR8nuSronQmq0Z9NxXDCpytLn29fzrpU8Js97Of345mbca3S+GJ776kqdbB14or0/NlvR/cgqtokcy4KU40jSViI5DuOeWgvkEUjXCRG42JbM05uHPv+L59iEX69EcSYNWNQk4l38XlzOE+m0aUuP8zSMBkTTpNoJ3m0fgGPAs80/r+SbhJH7Vn5wecTvtOsVgWbzu4njj35YbWlhw7VnD6lQ3BOjJrxxl9aQcij+t+g25+N/mmrS99TFad2rFq8Mf49uDjksXVU9bKdW4Y+JHjO0ahOK0cuZkNkpAFUIiTORmlOv5A0IIIYS4ysntMUIIURGKm9ibztK9lhNfg0ZQNSv3jkzmrkiwxgeyOhtAIeGAD5mamxvuSOHuxg78DGCwuKgaVHTFgELqaSOawU6vXlnU9dMwWFzUb3qO6JIugrktzF/lg8Ng4/FxSTzcJpcwC6iqRnCoC78CibvTTaQ6wDcmnWFNHQQYAFUjwM9d5E6K0mPQn06xytJVtsTVoay0OenTO42BsecwJAWx8GDR23h11LvbwtI/zbhDMpjwzGl61nMQZMqrm9DIc3RrZi85XoOdHjdl0SLChVkFgxGcOSq5CihKgUkNb8bgtrBknRmX8RxPPZ/M8Bg7oRYNg9FNtTo5NAoptWL18yAevf2qWBZ6+4feei6BatAwKIBBwz/ETst2abz4SgIzBtnwcZqZPT2MPW4v9XmlnP3IU5rKullRzD7pIrpq8XFB/xhSQeU9b3Sk69U+rkBgiBN/I6gGN1ENMxj3wgkGhMLZDcH5e1xV4thbRlm9dt5XuD0Udi8PYZvbQb8BKdzb0UnGhlCWpHsea3nHBIC0teFMO6SiVs3gg0mJjOlhpX6IG6MKZn8nDWMy+M/wdJqol/m8dfmybIMZtzmHp0an0L++gwCTRmiUlcEdbRRasOj2Yd5KH+zGczz5bApDGjrxM2gER1l55JlkhoZD+qYQfs/Aczr6gif1YncqaIqLNtdbqeEDKC4635fAlpkJvNBSlqldPibaPLeEQ/t2cXjPdvZsimPZt69yV71Ufh3/b15dmVX2jRBaJovH9aZlixbUb9aeG4ZNZFmKG/+WwxjWpuherB4cW+68DNQf/jKjY0OwJyzglXtuplVMCxrEtOeGl+PIKbVABhrcM55n2gVh2/MjY4f2oFnz1rTu/TBv/5GMEt2XCWN7UWihmDuNHx5qRd1GzfL+Nb+Pb5MvMUlR0fLrqJcGMTdw64uLOO7ScKdv5OMnhtCxbSsatunF8E82k0kIXQf3IELNK2/DEeN5opUfqXFTeOCWzjRu1pb2Q17ixwQXNQY+wbAGBT5gtCyWvdKP61u3okGLjnS540P+KnGux0DdYeMZFRuMbf8vvHRPX9q0aEWTdj24ecQkluZPJmlZa3l3xEA6tmtLgyYtaNKhHw9M24GtSndu7xaq+7tD7tZVrMuAsI5diCkw2Pi26UKHADeH1qwh0eVhfrrL6mm6dg4vmsQD/bvStHkbWvd+kNcWH8UZ0pExz/YrcWX/+TrV01ZKYHt6tw/EvfNzBnXsyPU39qBt21g6DHqPNTk6K1QIIYQQ1xRZeSaEEBWhaNTpfJLpnU8W+rM7y5+J3wSTmn8Rwbo1jJmHs3iyfiZvTMrkjcKJFPqvxM2B7L77HC16HuePnvl/dPry1hN1+TK5WADsnxfFey2P8nzzTF54JZMXihxx/g5RLSOAHzYY6dIli5ffzuLlounojSFFfzpF6Smblh7IzFUmbuqTyuMa7P1fMLuLXifRVe8KO3+pxvR2x3i4QyrTOqQWOtaxN5Je46pwtEjaSrCVBx9NplPRT0i3kcV/+l9cDeXVGBR2/RLF520TeaxBOq+/kc7r5w/SVOZPasTIP70xRaI/Hr39qii9/Ux3PRfPgaZ3JrDvzuKvuLJ8+ebjGry53ZB3QdJLfX6arjrTaDYsgV+H2lk39Tru/c3z1WGa1Z/J04Lp8WI6NYo0tydjSMWU77zRk27ZfRz9dai46PvUAfo+VfjPuckhjJ8RRFqlj72ll9Wb531F28OVEsx3m0/zfocMurrMTFscQOaFiq38MQEApw+fTqpOxItJDKubzWNPZfNYsUB9UVeGsCfpcp63Cht/imBe+yQGNjzLlMlFN2Qr3EcOLIjig7ZHGdMsjXffTePdgmmmBjNhRhBny7UsVEdfWK+/Xo4f9iFds9G4fyJLw2rQ7n0DvbvYCA2C/h1tvLXdvzxBCi9RfOvS59+Ps/PAeGbsKGMCTXORdSqVzFwXkE3Sptm8NbMv3cc0pXHjqqgbTlzcb9OTY8ubl6Eet9zaDLNrLx8+8xLf7js/45HNqTPZpZfFcB23DWiG2RHPO6NfY86hvIncnKPr+OLZV6i94DPu6nYbPUOX8mNZeyNWtK7KnW4au+d+wqx/9WZsvdPsXruHlByAE6z94iuW3dWagTXrUEuFFKUR/fs1xpj5O2+O+YIVGXm1kxo/l1en3ECfD3rSKTacTw6czs/HSVrScc7kOAAHJ45mAiXcvWGoR7/+zbE4djBp5MvMOpw/IZ57kv1bC3wnVRRCYu5k7IsdaFo7ijCTleTTbgwYiYwKR+WsvsfV5vzF4lXpDOjXhZ5N32PTDhdgoU2PzoRqh5j128G8dDzJT29ZPU7Xysaf/48V+/P65bmj65kxbgJ1YqZxT4dedAn5hZ9KWvls0NdWn87X0AClamNiG4ezb+NJbFoupw6XsTGzEEIIIa5ZsvJMCCEqQlPZviaE1YlGclwKtmwT29eF8+TzNZmRWOCim92XKa/V4s3f/TicoeJygzNX5XSqhS2bg4g7ply44OBKrMLI/1ZhRaKRHDc4bUYS9vpw6lIx5Prw5Wv1uP/bENYcMZFpV3C5FLLSzezeGchPmy04ADQjiz+uzei5gew8ZSDXBU67StpZM/v3+7P+mEFfDB6kU5S+sqmsXxTKHreGxenHnBWW4j/udda7ZvVn4ot1efp/waxPNJKZq+ByqpxO9iVutzn/sVeFKYqJLZt8OZqh4nSDK9dA4v5AvviwNmNWFbiY7+UYtBw/3h9fl5H/C2LTCSNWh4LDZuBYgh+HrKWvxPKE7jrR26+KZaCvf+iu5wJcaRbij5k4Y1VxuMDtVMlMN7NzexBff12D2x6ty6vrTRcf8emlPq+3zgwGQFPIzlHL/VjFjC0RvLXGWPwCnwdjSEWV57zRla6OPq6nDs/sC2L+Zl8OnDKQ41BwuVTOJvuyZG40Q8dGs/hMgYMrc+wtraxePO8r3B6akaULgznhgtwDoczaX3g0qfQx4Xz9ngrk5efqc++XYSzZYyHVmvf+c1Yjh/YH8ONPoazN5LKft+60IMaOq8U7y/1ISFdxulTOHPfn1798yNEofC7m+vDZq3V57P+C2ZRs4JxTwXrWwqrF1Rg+Npp55eks59uhrL7gQb3kbI5g1P8C2XnaQFKqCbvbh2VrfUjL9mHh+oo+l1To52DLpJup36gZdRs1p35MLLF97uGZLzZgrXETL3z4DJ31bcVUgIsTh45g1RQCAsrau8qTY3W+31ib6+oYcB9bx8rSHoFXEnNtGtRQcR/7i7VHinyzs25h9VZb3jE1vXWJoqLlv1SyyRw94QRLVSJDCsRqT+F4qobi64evAphrUq+Gihrch4827Ly4b9u+XcR/eDOBikp0jSjPL8hcaIMNrEu8xPSXEsSNL33Nty/cSfeYOkQGWTD5hlGrZjgWBVTVk3uoraxdsIIzSi1uuqlx3hSXuQW9uoaj7V3CwgMuL+fn5XKc28Ff8XYwV6fOpfYl09lWStY65i4/jRLZlRe+/Z1t6xbw46ev8mTf6/C/THsXCyGEEOLykpVnQoh/LGtcTRrHFf7bgR/qcd0PxY91bIsidlBU8Rc0hX3Lo3huS9m/mFzpfnz1UR2+KvNIhaPrInlgXWSZR15gN7Hq52hW/Vz6YZrNwi9f1+SXrysWg/50PEv3PFeSP3+eUqh7LJgFJV2M9KDeNauFebOrM2+2vgjdZwJ4/+0A3i8zYe/HoOVYWDC7BgsucVxJ/fNSfRZASw/h3iHFn4enu0509qti6evpH3rruQDbjgjueCLC+7EAZfZ5HXUWUcWJ6raw88ilL4WVNO4UzsjI4skNqT+5+Et6x5CS+4TC7/9tTL3/Fk3UwtRRTZhaNAwd5S2t713qtdL7uKarDs9uD2fU9vBLB1ZEZY29pZUfvHfe602rtHjO7ahGl8HVKhxreceEC3JNrFlQjTULSj/scp63AM7T/nw2xZ/PCvytau9Ebo7NJStLLTKBZmbp99VZ+n3paXr//AD01ovbyOrva7K6QIxrptejzfQy3icqkYbbbuXUkS38Mvl5AmKW8npse25oaGDNdg9TstuxawqKquP7hwfH6nq/asKoAk6nvlVLhVz+2YWSyq/hBiz4+JQ3HjcOuxMUEyZTwTQcOJwaKErehJimlXFTi4LF1+J5rShq3l5z2qVTV8Ju4r5BtTCk/cXH499h1vpDnDpnJLzni/zywW2e5kjOhkUsSRnI3X1upsWUXexuezO9I11sn7mYBBcoVbybn3fLcb791UvXtd620k6z8IV7yN56B307tKRN6xjadK9L227daawO5vGFpz0onRBCCCGuBTJ5JoQQ4qoQFOrElWnE4WOn86CTDI008NuMwHI+AkuIy8xg4/qGbpwJgSz16BF/4gKpQ3EFqGE59G0IBw6ZOZFuwGZwUa9pBs8OtWLWLGw9eOnV1EKUi2LCbM67mK8qClxLPcyRwonTbtRa1xMbrbLzmAcPQrQf5dBxN2rt9nSubSA+ocD0m38burT2AXsiCcfdVN4DctxkZWSjqdVp1CAYZduZyqt9RxJHkty4g3/lod7jWXHJPbE83FgzP121ViwdaxqIL7qKD1DDq1HNDDnLZvLR73vzV9o6OHMq8xKP2DVgKO3KkG0Tc+Yd4e6H+zCgzXRC+vUkwraeD+YfwwUYPM5PH8/LUZwS3ImebcxgP0pCUsG+VaDMutsKsB0jbuZk4mYChkAa3/4a0yf0olufWFi4qFzlFEIIIcTVSx7bKIQQ4spTHdw+9gA7f97NvtkHmT4kB8emCP67SS5aimuDGuog+JyZRfNDSPD8dnyB1KG4MiyNzjLx+SMsnbaf+B/3cOB/+1n6ykl6hUPy6nBmH5VncYnyUjBYfDGrgGrCN7AKtWO68+BbH/FMaxPu9K1sPOgsM5WrinMPy/5Ixm1pzVPvj6F/s0gCzBZCa7djcK8mmEt7r2s/8+btxm6K4cnJ4xnSIhI/o5ng2p145N1XGRqlkB43n98r9a4pFwnxe8jULNzwn3Hc3ToSP4OKwSeQqqG+3l0b59rH0mUJuMP7M+GdB+nZtBpBZgOqwYfQGs3o1q52+e5kdu1jyW+HcJla8tRHrzG8fR1CfQwYTAFUa9SKRlVU3GdSSXWAb/vBDGsbTYBRAdVEQIBPsTztDieaEkKbbh2o4XepiTwnu3/+mW2uKPrd/zz39g4j44+fWHI6r608yc8THqerGAgMr4K/SUU1BhDV4lbGTZ3AgKpw9o/5F/YyK1ZmvW1lqEuP23vQonoQZlXBYDLizMoiF1Dko0IIIYT4W5KVZ0IIIa4CLiyagRyXCzLN/BUXztuzgzkmF9DFNcJ9Oohxo4KudBjXNKlDcSWYs3xYsdNOq1p2qgW4wWEg+bgvq1dW4eNF/qR6sLBGiMKMtHx6LnueLv6KZj/Ogren8kf25Y+qYmxs/OI95vV8j4EtRzDl5xFFXi9tMtDFgZmv8cGN0xjT7g7enXMH7154TcORtJgJk5ZW+hMHrKtnMnPPTTzZrC9vfN+XNwq9Wt5dPUviZOdXbzK9+6c83GsU03qNKvSqY+s79Lr7G456PMY42fXVG3x+4yc81nwgr387kNfPv6RZmT/yRkYu+4Mflj9Bl1t78PLsHrxc6P0u9hf4/8f37CVda0zjEZ+yNOJZ2j21hJIWXrkS5/Fd3L95v1c/urqOMm32ajLz20o7ozc/z3icrhJE34nL6TuxUCrkHv2V8ZOWkaZdusx62iqxSnsefHU8nUxF8nWfZfFvG8tZSiGEEEJczWTyTAghyqms/W6EB9w+fPZCw0L7zVzK1VDvV0MMQgghKi5jZzgjX9K/h54QerhOHST+UBPqRoQS5GfBqLix52Rw6kQCOzetZt73P7B0XybX4tys+9Qyxg57lP0jH+GOrjHUCoaMxB2sSQigV8+GuLVSSnVuN589fDeHH3yMh27rSLPoANxpR9i8/EemTv2eDacuw11TuTuZ8si/yRz1BMO6x1ArxIRmzyH9TAqJh3YTd/Cc1556oGVtZOLwu9n1wIPc1SuWpjWr4G+wkXbiENs2HSv3VJ2WvZn37x3O3gcf4d5b2tMkOgSzM4OUwzs5lGlC0c6y+KWHGZ3yFA/2bct1kf4YnDayMtI4lZzI+oMZF8qYE/cBo6YGMPaOWCzHky8dk3aWpd8tZHSPO6m64wdmbc8t9Jre/DwrqN503ZzZ/hvzVzlo3qAW1cODsKh2Mk4cYMOyOXz65S/sSLvYL0sqs562UpRktqzcQfW211E9xIKSm0HSgc0smTmVKSVu0iyEEEKIa50SXKWqPBFLCHFFDeyRy9evZQLw+Ds1WLRWVh4IIYQQQlwrpo49zi2d877LhdxQ9QpH47lxzz/PDV1uAGD4iPuvcDTXIoWqQ79g9WvX89f4ntw356w8dluIcvju2xkArFm9hrcnTizjaCGEEEJUKoU5svJMCCGEEEIIIYQQZVIj2tK3JRzYdZgTpzOwGUOp1/Y2nn20PWb3IbbGl3OVkRBCCCGEEFcZmTwTQgghhBBCCCFEmSyt7mTilFsIUIq8oLk4seBTZu+XDWuFEEIIIcTfg0yeCSGEEEIIIYQQogwK5vR9rNhQj1bX1aJasAVyM0hOiGf1vG/4eNZfpF6LG7kJIYQQQghRApk8E0IIIYQQQgghRBk0MjZMY+SIaVc6ECGEEEIIISqdeqUDEEIIIYQQQgghhBBCCCGEEOJqIZNnQgghhBBCCCGEEEIIIYQQQuSTyTMhhBBCCCGEEEIIIYQQQggh8snkmRBCCCGEEEIIIYQQQgghhBD5ZPJMCCGEEEIIIYQQQgghhBBCiHzGKx2AEEIUNHXs8SsdghBCCCGEEEIIIYQQQoh/MFl5JoQQQgghhBBCCCGEEEIIIUQ+WXkmhLjikk6q/PKH5UqHIYQQQgghhBBCCCGEEELI5JkQ4srbuMvEfS+brnQYQgghhBBCCCGEEEIIIYQ8tlEIIYQQQgghhBBCCCGEEEKI82TyTAghhBBCCCGEEEIIIYQQQoh8MnkmhBBCCCGEEEIIIYQQQgghRD7Z80wIIYQQQgghhABGPvHYlQ5BCCGEEEIIcRWQyTMhhBBCCCGEEAKIjW13pUMQQgghhBBCXAXksY1CCCGEEEIIIYQQQgghhBBC5FOCq1TVrnQQQgghhBBCCCGEEEIIIYQQQlxxCnNk5ZkQQgghhBBCCCGEEEIIIYQQ+WTyTAghhBBCCCGEEEIIIYQQQoh8MnkmhBBCCCGEEEIIIYQQQgghRL7/B7rld/vWJi+TAAAAAElFTkSuQmCC ) ``` binning.set_task_parameters_by_name(max_bins=-22) ``` ``` Binning of numerical variables (BINNING) Input Summary: Missing Values Imputed (quick median) (PNI2) Output Method: TaskOutputMethod.TRANSFORM Task Parameters: max_bins (b) = -22 ``` ``` invalid_keras_blueprint.save('A blueprint with warnings (PythonAPI)', user_blueprint_id=user_blueprint_id).show() ``` ``` Binning of numerical variables (BINNING) Invalid value(s) supplied max_bins (b) = -22 - Must be a 'intgrid' parameter defined by: [2, 500] Failed to save: parameter validation failed. ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABrQAAADECAIAAABROFbnAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd1gUVxcH4Dsz24ClLVWlqSAIogiKDQt2RY29l2hiTSyJsUYTNc0YNcaaGDWx+9l7RQ2KDUQRFEREBem9LGXbzPcHoojssjRB+b1PnieyO+XcO7Nlzt65hzI0MSMAAAAAAAAAAABQ99A1HQAAAAAAAAAAAADUDCQHAQAAAAAAAAAA6igkBwEAAAAAAAAAAOooJAcBAAAAAAAAAADqKCQHAQAAAAAAAAAA6igkBwEAAAAAAAAAAOooJAcBAAAAAAAAAADqKCQHAQAAAAAAAAAA6igkBwEAAAAAAAAAAOooJAcBAAAAAAAAAADqKCQHAQAAAKBuYhoM2Xhm24y2pkxNR/IRo/SdR6w9vnuWi7CmIwEAAIDSITkIAADwUaEMuyz53/k794/PcSoj30GZ9/7x0IXA+/un2NT27wMijzmnQ57Hh535tr24pmNRq7xBUoadF+0/ezvoWJlH6gNV+48aYzt2w69D2/f/fEJrI+q97/2jPwGKiBwHTh3doefiTV+5i2o6FgAAAChNbb8YAAAAqMsM+m6OTE7OTAr8pa1A23V0bFt3dHe01OeXle2g9Rq18WrpYCFm3n9epLwomqYpimIYqtyxMg2Hrj104e79TQN1qyOyYt4Jsoxd69i17dzKqb5BmUfqQ1WJo/Y+0BZDVyzqYsTFH1y0+HwaRwjhOY7fcMj32p2wiMj4+Pj05ITkmCePAy6d2PbznEEtzfnl3sMHdAKU/VZTrs4RdPkjPDEzNfbwWBOK5N9d8/WGRwqhy4xVU5141d8WAAAAKC8kBwEAAGoryrz/uN6mNCGM9ZCx3vo1HU4NKri7tm8zu3pNe6+4kVPedWkzV+9OLe3Ndat7dNa7Qb63XddOlTlq74HI84v5vSQk6/LPP1/K4AghhND1PLp3bOXcsL6Joa6AR9OMQNfIslGLzgM/X7b13F3fdSPty3dj7AdzAmjzVlOZzpEF/7F8/0tW5D7jm09Ma0EqFAAAAN6G5CAAAEAtxTQaOqGzHkUIIbSZz7j+5rioBqgilMmAaSMb8pTP964/ksC+/Zzcb6G7paWlsZmlmY2ja+ehM9acjyqgDFxGbdi/wtvwI3wZluetpoKdI/Xb9GdgASXpPX20fW1PlQIAANQ9SA4CAADUTgL3sWPdBVym767DMSpK3OXTYQ1xUQ1QJegGn4zrbkgpHu3fG1TwzrOKgnyZkuU4VpGX8fLRtX2/TOg1ad8LFcW3Gz13WIOP7ttz+d5qKtg5bPSh3f9JicBt9Ei38t+gDQAAANXqo/t6AwAA8HEQd/50eCOeKvH4Xyt+3x+uIAL3sWPctJ54kFC6Dn2/Wv+/y6GRMclxUZGB53avmNDWrOz5vviey+8nJWfG/TPkrVnSKKORe5JSk1NvftuyxDaEVp2n/HLg0t1nMbFJL8LvX9y1cmIbi+LLiJr4zPx+w79H/QJCnsfEpiXFxj25f+vEnz9M9GpQojn8Bu3HfP3r3wcv37z/LPplSmLsy/DA6/+b6cEnlOWEE/HJmQmX5zUtylroOA365qc/9xy/FvDg2YuXKcmJqXGRYdcOb5rTq5FOaQ0T+myPSc5MffVf0qHxpY2Ooq2nnEhOSU5/9Jv3W7Hp9N3yJCM19uSkesW/OfHclwYmJmfEH51cnyaktCC12TVt0WP+tnM3gp/FxKZobkJVBEkI02TYL3tOXrkf9jQxISE1PurxrZPbFw9qqv92d6g/Ftr3fBUcNZ6px6hFfx71e/TkRUpSfFJ0xKNb54/+/eu3n3nVL9ZIkV2POX8cuhEckZAQl/gi7IHfiX1rJ7TQVPuCrtfTp7WQUoadOf1UpWG5IlzqlT/3PFIQStCynYcOYZy+upSSmpx278d2Jefa6/z7o8TMlOj/jZK86dCqOgH4Zp7jvv/n9M2IZy+TYiJCr+5fN7NX4xKzGZa3h0ml32pKdo76xXxP++dxPLve/Zph4kEAAIDaBZ/NAAAAtRBl3m98f3Na+eTgv/7Z4U9335q1slPjoRM6rg26nKvVBniNhy+aX/SHyKyhR/8Z7n2H9l0yYuKW0Lwqi1LSYfGeHXM9jYsSNSYN3XtPa9nNx+ur/lMPvlASQghl1G7Koi86F8s16EkaNO0wuGmHgRPGbpsy5vvzia+yM5RJ9wW/LSy+JN/M1klC5ZS46bNwYcM2n349ufjCRGhY37nTGGevvt4r+g/f/FBWgQaxCTf8I1XtXCQtPRoyVyOK0ka8pm3cxRShm7m7CnYkFA00o81btrRhiDL8hn9SaSFqiTH37O9T9IdAiyZUMkja0MW7T/tGRek6fUuHtkO+9uzRwbrf4PUhRXvUcCwq0/PlWpcy8Jjz785vO5rzXqfS9IwbOBg3cHDv0lkUuNs/Xk4IIXzHz/ef/NHbpOgc5JvauphaGT7apCnpp+fZwU1IqV7c8I/SJjdICGGTE5I5QiieoZGYUkXdvBWnamFXz7OdHXMr8s0meA6eniY0UYbfuJPFabdhLU8Ayqj13H93LupgWlQ+SGjt2u1T164jRu6fMXresWjFq8XKfXQq/VZTsnNIrpqWcxk3/UIVfdratm9nRd9/UYnXDAAAAFQxjBwEAACofRi74eO76BPZ3T37QhSEjT3+78VMlrb8ZHwfbWfz5+Sx/22ePbSjk62VhV2LjhN+Pf9SQZt3WbH92456VRQkXW/Ymq1zPY0U0ed/HNfFycbKskm7Qd+dfqHgWff/ceXwt8avEeXTraPcG1o3kJhbWTXrMvK7w+FSyqDF5G3bpjqVGHulerFnShdXRzszCytrF6++cw5qSt8oo7aPb+dkb2dm0aB+044jfrwUp6SM28//eax1ya84sjOf2Zgbmb76z2LYruTSUhjKyBv+iSrCa9KmlfHrnqYbtGlrzSOENmjl6fzmd1XdVm1dBZQq4YZ/GYPPNO9aFXtk3idtmzexrFdWE6omSOXjQ4vG9O3o4mBnblHf0rH94GXnXyooA8/ZiweZlTy5NBwL7Xu+lAZosS5l9snqf5Z0MmdkT499P6a9S0Mz83qWjVzbzb+Q/daBE/f8al5nEyo35N9pvT1s69c3t3Xx6DXuqxWHQ5XqA+A5uDXToTh52IMnijKjLUTXs65HE8KpsjOlHFE8uOqXyhJe007ti3caZdqqTWOGqOJu335Z7Jyo/AlAWwxZ889iL1NK+nDX3IHuDjYWts07TVx9JUElchy1ecfsFiUG+ml/dCr/VvNO56jFJoWGJqsI39nNuXyFXQAAAKCaITkIAABQ6/BdR49zF5L8m/uPRbOEEC7jwv8upLKUQbexQ220++xWRuxc+sPO/yISc+UyaULomTUTxvx2N5/wG46eNciiSkoqCDymzu9jRuXf/WXM56vPhSXmyQvSo65unjFla6SSNvQe1d+qeKRcfkpsQka+gmXl0sSw85u/8Pls1zMlEXvOmtfH+K142Jzo8IiXaXkKlTwn6Ungo0RNmTcuL/H5i8TMPIVKkZcSceGPLxefymApnTY+XStYvkX+4NrtbJYStPJqVXSDJGXcvkNzPiGE8GzatX/dKoFbR08xxWbdvB6ibYKpVGzG49sBj+MzCxRaN6FSQXI5j66eD4iIy8iTq5QFaU+vbJr57alMlhK36eheMmWj4VhUpue1WJffYvLCAZYMm35hwfDPN10KS8pVsKqC7KSo6DRF8fQT06BpEzFNFPf3rjt492WWXCnPTYkKurDr6L1sDVkqoY1dPYawKdGx+WXFWog26zF1tBOPcPIHd+7nE0IKAi74ZbKUoHXPTiZv2qvj3tZVSLFZd26Fan9OaHECCFpOXeBjTqsSjsweMXvnzWcZBbLcxJBTq0aP/f2BjOg0nzq/RA1grY9OFbzVvNs5aqliY+JUhBJa21niEgQAAKA2wSczAABAbSPqMHaoPY+TXjtyNulVhiP32uHTCSpK2HrMcMeKlSWRhe/e7pfHUbrte3gZVEGQ/Jb9+zbkcQX+u3dHyIs9XnDf1z+ZpQQu7i00jQ7i0q/+vummjKONe/T3qqqxjITLvHY5SM5RvMZO9hWcOiXvpu/tPI42aNuxqGyCjmfH1kJVTGBQkorfzKutUWF2hdekg5cFw0lvXLz9bkGLStCqCVUaJJfz8EGUilC65uYGFU4bV6bnS1mX16xfX3seUb04sPZQrKbsMJuRkqriCL/l8E/bmWj7yqANjI15FGEz0jI03ttK80WGlvbu3T/9bteFv0faMJwy9sgfh16yhBCSe+3UlQyW0m3Xq/PrEr185/atDCgu9+aVgIqfE6X1RvN+fRryiPLJ/g1nU4rnPAtCtm+5LOUogy79u2iqoqz26FTirUZT56jzqsdpI4kxLkEAAABqE3wyAwAA1DIGXUf3r8dwWZePXEp9nQkouH3oVKyK4jsPG+FRjloBxXBZIQ+eKwklbOxgV/k5hyl9hyb1GULp9FgflfKm2EJmanLK8Un1aUKJzCyMNH7NYJPv3HmuIpSuY9OGVTYHMpebmJDFEdrISFOuROMWMq9dul3AMfU6dGrCI4QQQUvv9gYk9b91f17P4oSendvoEkIIbdWpY2Mel3/70jVNQ9QqEoAWTahUkDwz99Hfbj7m6x/6+FlS/Iuo+347J9ozhFA8HlPxMaWV6fl31qX0nVxsGcJJg26HyDWvmnJq25EYBdHzmH0y0P/o2jkj2lrplhmAQCQghHBymbzUYyfovi48IzU5MzkuMSrk7sV962b1bKRD8qJOzh+7+Hz6q1Wk146dT2UpsVd/71epWMambVsrhisIuHAtsxLnxLu9YeDU1JpH2OzgoIgS90pzWffuRioJJXR0tteUyVNzdCryVqNV56gjlykIIZRAKKiS0csAAABQRZAcBAAAqFUosz4j+0hoNt33kG9GsUtt+d0jJ6KUhLEbNNpLV/3qGnA52VKOEEpXXHb2hFBlLELpicWalxAJy5hXjM3KzGIJocT6elWXKeBkMjlHKB6/wvlGLvW/i/cUHM+hu7cNQwjPpau3JZ1zy+/6tf/u5FNGHbu2FhJCmXt3bc7n5IGX/MrKh5Q/AC2aUOEgeY1G77x8atNXQ73dmlibioUCXRNrR1c7zXncKgtb23UpsaE+RRE2Ky2jrHohXPrFBf3Hrz4VlsnqN+46fvFfpwLCru34xruepjDkBXJSdo6K41TyvIyEqAfXTvzz0xcDWneavONRsWI+udf2n4xT0UY9RvWzoAkhtGWXrq48Tn7P92papc6Jd3qj8LXG5WRlvzMuj83OkrKE0GJ9scZDWNrRqdRbTRmdo4ZAyCcasrIAAABQQ1CtGAAAoDahGwwa1UVMEcpk6N7nQ0tbwmLAqG7fXz2ldTHUN5s2NDKkCeGkORqKBnAKpZIjhBKKdCiSp2G5vNxcQgibume085wrmkd3qUHpG4gpQri83PzqzRSUc+ts/MXzwT+0b+3avYvl5mj9rl3smNxLl/xz0gUXAmS9O3Xu3oLvF9WxZ2shkd8666u5UHG1NaxiQVKmQ5Yt71ufp0r47/elK/dej4jPLKB0zT3n7jv6pUt1hVoBnKyggBBC6eppkccm8phLv427tN7SvfeocZ9OGtbeumm/b/c4GAzp9d3t0tNVbHZGhpIjQmMTY5qQd9OPct85LYbtKTPDJ7u9a9+j8fNdvUYPb7hv/TOJdw8PAVHcOe8b/+acqIoTgMuVSgkhlL6hwTsJQNrAUEwXLlLe6r8VfKvRsnNK32Vhj7OZ6Zrv5wYAAID3DCMHAQAAahGm8eCRnkKNCRHauMcIn3JUEn29nnnbto0YwuU9CX+hvpArm5mWwRFCN7Ctr3EGNy776dMkFaGN3Fs1qdhPjZRh8xYNeYSTRUVqiKfyOJlcxhFCCYWa+/UNNubsySAFEbTq07N+4z69nXgFgef/y+S4FN8LQXLaqldvV9NOvTvoElngyTNxmpIc5d+19ioUJN+lXWt9iivw/Wnqz8fvvUjLlatUspyk6ISc2jWQi8uMjslkCW3YvEVDbScSlCXeO/H7V4Nae03ZG6UgwibjJnTSUbtszIsEFaHNbK3ULqINZfjerX5SInD7bHI7sWWfwe1FRH732Ok38+5VzQnA5UQ8jlUR2sDN3bHEa40yaNnKgUe4gohHmop6l6Ya32rU79PKpgFDONnLF4lIDgIAANQmSA4CAADUHjyXIUObCyhV/K7B1uZGpiX/M+nw8305R+l1GjWgQTk/wimJ91cz2gkoNvPy6etSQgghHEc4jhBKIOK/uf5n48LCMljCNO7Rs7HGpJ/i/sUriSrCcxw9q49ZBfIHwqbjP+uiS3F5Ny/5Z5d/de2xqclpLCF0Q0et80zsy7Mn7sqJsM2QaZ/2c+HJAs5eSuUIYeMvngtSMA37jJ4zpJM+kd05cT5ec/2F8u9aexUMkiOEcColW7uygSXJgy5fS2cJv9mYqZ3KN4dhwYszW088UxFK19xC7ZrKyOCH+RwlcG7RhF+ZMLnEY38fi1cxNiPnfDN9RHsdIgs4cbpYvriKTgDFgzPnnysJr8moL3q9lasTNftsWlcxxWX7nfIr5zSH1fdWox5t4epqzhBFWHCYrKq2CQAAAFUByUEAAIBaQ9By+BAHHlFFHz/on1/K86rIw3vvFHCUsM3wgY00ZxsoXQubBhIdPk3xdC2a9pq+6fT28Y14JPfuhlVnXs0vxmWnZbAc4TUdNKmHo6Ro9jX57cMnYlUUv8WsLWvGetoYCmiKJzKytKtXspJtgf/m3/2zWKb+sI1H//6yr7uNsYhH0QJ9yyZtBozr7VQis8hzHP/D0vGdHS30BAKxZbO+X+/c800rEVE82/vHsaRqTVSxyfeCXioJr+GYRTM6NNDj0QL9Bs179/e01PAliI0/deR2PhG2nzzRnS+7fepS4X25bOzZk3cVjP2YaT0MSN6Nw2cTNI9/qsiuy9Gw8gepCA8KyeconW5zf/m8o72pDo8iFE9oaGJY9SMbKyn70p/bHsk4xubTP/f8OMyzsYmOUM/c0Wvk4mmd3z4RxR2mLJ7q49HIVI9PUYzQyNZz+LT+DRnCZkRHq8+X5QbcCJZxjFUHr8aVS9vm+m3ZFJhPxB1nT28lJHn+R88VT8VW1QkgD/rz17PJLFN/2Ib9a8e1tTMSCHQtmvl8s3vPVy1FpCBk66oTKeV7EVXhW43WKOP2nV35RBl981YsBg4CAADUKkgOAgAA1BaitsMG2TBEGXn40L3SZ/Fj407871ouRwlaDBnqpPGSndf48713n72MS0+Jj3/k978fhjqLuZzQfyZ/viVc8WoRLvP6qetZLCVqPmXXzUtL2rwaQ1VwY83i3U9lRK/ZhHWnQ6Ji01PiE58++O/btiUrl6qe75g+bUtwFqfnOHTZv1fuRSQmJqXHRz2+eWrXrzO7lRhvRAmsu8xYf+R6RHRs8osQ/10Le9vw2RS/ZZN/8c+tSF+VgyJ4+ya/dJY29V5y5sHz1OTYlw9892/8olPJbGdxbMKpfZeyOJphSMHt4xeKboJk408dvyMjDMNw6Rf2nkwuKyFTkV1rr/xBcsmHVm64m8OJHIauPnbz6cv4jNSk1LiIq/NbVmoAXXWQh/w+fcmFeCVl0uaLLaeDIqKToh/eOb5+jncDPkUIx71qFb9p3ykzf9157t7j5ykpSWlxTx6cXTfKgc+lXVuz5UaB2q2zCRfPBMo4nrOPj8Yqv2VTRf37y94XKoqiKC7Td+/ptzPdVXUCsImHv570y400ot9i4u8ng5/GJseE+u+c36MBr+DJgRmT1gWXcyReVb7VaIsy7dbPS5dSvjh/+mF1ziMAAAAA5YfkIAAAQC2h12WYTz2Gkz84eChM3cUzl3rh6OUsjvAcBw9uUWpCh8u6s3PdjsNX7j2OTcuRKVlWkZcR//jGsY3zh7XrvfBsfLGZydj4fV+OWbjz6sPYDGnkk2fK1/u4+LXPJ7O3nAt6kZ6nZDlWUZCdGvM46PKxnX/8efFl8Q0kX/62b5dP5v954vaTxGy5ilUWSNOiH908uftEcIlSEMpnh1et2e8XFp8lU8hzU6Pvnd26sH+XMZtCqjs1SAhho3dP6f/l5nMhsVkylUqem/Li/sUjN2I0z6mYfnHvmRSWcLnXj559M0EaG3/m6PV8jqgSTuy9rEVNmIrsWnsVCLIg+PfBfab8csAv9GVGvkKlkhfkpCdGPwm5ffnU/nOPNFSqef9kj3eO7TF47l/ng16k5coVBVlxD6/uW/nnf+ks4fJyc18Nf03x/9+Bi/efJ+XIlCyrlGUnPg06u/37YT3H/R2p0LBxNu7E7stZHN9l1BgPUeXizLv159ZAOUfYpNP7L2SU6MEqOwG4zIDfhnbxmffnqcBnKVK5PD8j7tF/e36a2LnnnKPRmlpamqp5qykf2nbo+C5iIn+w70BweeMFAACAakYZmpjVdAwAAADw0aIsJxy/91tnKvSnrj1/Cy9n0QSAt9DWk48G/tyO+M1zG7YrsXKpTFHbFTdOTGuYc3Fmx/F7y7hBXBO+w/QTvsvaUfd/6NN/7SPkvUon9v7t5oEJVpknJ3eYfCS1NiWhAQAAACMHAQAAAKDWoQw7TPx6ok8710YNTPQEDE9kbOXSbeKvOxe0FZFsv6OXyrylu0wFAZtXnU8nRt0WLepuXL7bfCljW0crQwFPJHHoNOWvvd+205XeWjl7IzKD6ghbzPp+lDVdcG/L6hPIDAIAANQ+GusQAgAAAAC8fzyXAbPnTbV65/5bTh59fPGCg5UY6fcam3j4u19GdPjVe8TKn84GfXE+TdusFSXx+dV3Q/eiKi6cPProN1P/iih98j4gOq3m/j7LhS979PuCPx9jukEAAIBaCMlBAAAAAKhlqJSb+/dZeXk0c7C2MNQTUIqc1JdPHtw8f3DHtlNhWVVU7FYVvWfWQs+tfR7tDlJf2fhdtImw4EVKfmMJLU2KvHtu9x+rdt5Jxh3zahVEnNi6v02/54t/D1JfIwYAAABqEOYcBAAAAAAAAAAAqKMw5yAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdheQgAAAAAAAAAABAHYXkIAAAAAAAAAAAQB2F5CAAAAAAAAAAAEAdxavpAAAAAAAA6qJFCxfWdAgAVSzscfiJ4ydqOgoAACgfJAcBAAAAAGqAV0evmg4BoOqdIEgOAgB8YHBbMQAAAAAAAAAAQB2FkYMAAAAAADUmICBw/cbNNR0FQGXt2fVPTYcAAAAVhJGDAAAAAAAAAAAAdRSSgwAAAAAAAAAAAHUUkoMAAAAAAAAAAAB1FOYcBKh29c1Zz2aKmo4CAADgY3P8irCmQwAAAAD44CE5CFDtPJsp/l2RXdNRAAAAfGyMrpjVdAgAAAAAHzzcVgwAAAAAAAAAAFBHYeQgwPuz7YTJ/Qidmo4CAADgw/bZgDR3p/yajgIAAADgI4HkIMD7cz9C5+wNg5qOAgAA4MPm0yGbECQHAQAAAKoGbisGAAAAAAAAAACoo5AcBAAAAAAAAAAAqKOQHAQAAAAAAAAAAKijkBwEAAAAAAAAAACoo5AcBAAAAACA6kDX77/i+MVTy734722XlHGXpftPX1/ZXfjedlmdGIv2M37bffVWUGTY/dDrx1b2MaVqOiQAAPj4IDkIAAAAAADVgdKzaupibSyqsoSW0H3W/+7dPftrTxN1m6RE9Z1dG5rp8ijtlq/VBC6z/9r4zQB3O4mIxwjEZpZCmZSr6aAAAODjw6vpAAAAAAAAoLxEtt5jvxjv07GZjakuJctKef442P/cvq2HH2TUQPaIcZm4Yc1Y8akZEzdFqCq8FVHb2buX9mtoLtHXEzCcPC8zJSYyxP/C4Z1HAhLkr5ahKIqiaFrrVN/by5cZp/7gLX5ruqobdKgI/Mln5K44tjyNqgxhm+GjHAXyyIPffLX+0jOpwKy+WCp7XzsHAIA6BMlBAAAAAIAPC2M7bO3h5Z1MmVdJL56JVbMODRx4wbuOPCA1kBykjWydHeplCCo3PI8xs3e1r1+UmRPpm1q7mFq7tOszevC2yZ+tv5PNESIL+mN4yz+032SJ5asmzveFtrBvbEjJru/440xkJkeILDE6p6ZjAgCAjxKSgwAAAAAAHxSe24QZXiYk+cqa71YevR+dqRBIrF1ad2qe/1/SexvVVl2UYRtHDtn8WMYxOoaW9u49J3/zhY/rpBUTL/X9I6zigxK1lXN0uvvRV//muc8/dXCi/pEp3guuK6p9z4TQQn0TI5EyJyMjT0kIIYQSigQUl5+amotbiQEAoFohOQgAAAAA8EHRtbI1YdiY0+u3+0eqCCFEnhx150zUnVdPUyYdJn87obNzY+v6pga6fFV2fPh/+zb+9aDBwDGf9GzjZGXE5MY9vLhz9cp9oZmv0046dj0nTZ88oL1zfT02IzroyuHNmw4EpBRLx5W5ANNk1omQWYQQQtjkA+O6/nCzMBLm6fsAACAASURBVKVG6badue3CT442JiJVZkyw7/416w7cV3/zM6uUK1QcR5R5GbEhl3fMzTJrvmt8w9buFnRYPMs4TD9wdla9o8USdrSkxagZ08b0aNnIhJ+X8PjW7SyLN9Oql7K82ji1Qhm6DZ89oWdrF3s7SyMdKj81+sLy8cvOU50XrZ3dx9HKwkDI5adG3b2wbe3GYxGFSb0Sh4MUZJTsBFrSavJ3i6d2b2LMpzhOkRNzacWnC47EE0IoQhsP/zt4eOFyisDve0zalcBqPBalRrg8oOnn5T4lAACgzkByEAAAyouxHfzjhmmOt5cM/zlAWdPBfEgo4y5LNn/TM3pd94W+pc8aRRl2Wvz7+IwtC7YEpr2/yzO6fv9lm2e2fLBi8Pf+72N0jAZld1FFiBwGLlre9fG3X+9/jhMWPg75CbEZKtq627g+R56cjs4v+TQtad6jf2fnoi/6fGPrloMWbB9UbAmBbasRS7ZIcodMO57EEkJEzlO3bpvvafgqq2bRpPOoRe07tZg7dtGpeBXRZgENKKFNi1av/m3auMPIb92a6Awet+OJdq9HVqliCSE0XWohRcqg/ZJdGz51eFXyRGjj1teGEEKqbWY+2rzd0HF9X/etvpk5XybliI5RY/cmVgJCCCFii6Zdxv/WzFzxyTenUrl3DgfRK9EJtOWwlRvmdzaglLlpSVJKbGJkzpdlsYQwpYdQxrEoNUJK+1NiyvGkKustAAD4QKBaMQBAdahkecTaXl1R39rZ2dpIRNXO6Gqvt2tolvK8fofZP4xp1ciElpe+QLlofxZVeTnRiqueMqMKpb51sx6zfxhhreZSG+BDowjasdE/lbIdsvrYlb0rpvVykrz7iz+XfW5RzxbNm9u7evl8ezZWxbGZgRu/HNrOw62Je4+xm4OyiVHnwV3NaUIIYz9u6VetDQrCD88f3tWlWcuWPSf/ciWBqt9n2fwexpQ2CxBCCFE9Wf9J84aOLg0dXRp3LDYcj5P+98twL08Ph2ZtvMasvJTI6rUYM8adr7mJFCPQk1i5dBz143fDbBhVTNC9xFLumOY1+2zheHtBdvDuOUO7ujRr2aLrmDnbAlM131utLk7tcTmXvu/XqqWbffN2HYf9cUdBuJwbv40f2K61h33T5k3b9pu0LaTAxHtIlzd98/pwNHYp2QmUfpuebfTZh38NateuVaeuHh6ebQet9s8rWpHNOPi5W2G0DZt9uiuB0upYvBNh8Rg0nxLl7w4AAPjgITkIAKAdxv7Low+ehRz80rHU6xm9dkvOPX18b/tA/cK/y1tOsYRKrq6eXo9V16Me3901wqKUDwC+2+KLIc9Cd020qnWfDnr9NzyOeHBqeuMaSu4wLhM3n7+86wvH6tw/z2HC14MapJ39dUNATlUMG6y2s+j9qaImqJ7vX7k1TNh2xoxuBh9ydwC8oYo+NGfwtPUnw3JNPIYsWH/Y/9K/P41rbVE8RcipclKSs2UqlTwj7NjmvY9UFC817EZ4olShyI2/sXX7pSyOsbazoQlhHAZ84iJQhG6Yu+LQg6Q8hTwz+ubWb74/mMAZdxnQzZgqewHNOEVy1JO4rAKlQhp3d9/Pux8qGRMnJzM1nzS8ZnNOPo149Czs/sNbF05vWzLCRS8/fM/32x+VMtCQcezV3Y6WBa2bu+pEaFKeQp4dF3xqz8Wn1T03IafMiItNy1OoZNnx0Um5HCEUZeQ68ucdR27cCQy5umtZr/oM4VnUM33TxqLDwSrf6QSO4wihzJw8nUxFFCGcLOV5rNp7e7U8Fu9GSLQ9Jaqz4wAAoJaqdZd/AAC1FG1qYUZTwqaff9XP8p33TsZh1Pxh1gzFGJtKGEJelUf06D3vQsXuDa3k6hrk3jjrl86JPPv1qP9OK4TuPn2taNm9c+fjP/gJ7ataYYFL/WotcKnnNX5cUyps3zbfqpnwqfrOovem6pqgjNyz1TdL0uvzQbUv8w1QQfLYa1tnD+7eacySLZeeyi1ajf52+8lNwxuXOmmQKiE6XkmEZhZGRa8AeWJsMkfp6OpQhAhs7a1o9uWdGy+KJdVy712/X0AEtvbWdNkLlIMqPupFLkeJxXplvKFyHMdxhM2+u21W/9G/3Sj1jZFf364Bzcbeu5tQox9blEGnJf/uWjzS29XOwkDI15HYWJsKKULT6mZweqsTuJybxy6nUhadF+/yDb55+vCW5TP7OKjtnao6FupPCW23AAAAHxF8QwYA0I7Q1MKQkmdmMx2nTHYXvfUUZdRz2gRXWWYmS0kkmkZQ0EJ9MwszY92SVwvqHq8OeXfOXkxmBS19fGxKDIITtenXvR5dcOe0b2m3blW799kJtRFl2HVwd1N50KHjz6q/HGcdxGX6HTmbxGs5uJ8Dbi2Gj4osMejYqi8Hdx667GQMa9Zp9syu4tIWYxVyJaH4fP7rzyiFQskRiqIJIaTM3z2q8ocRTi6XcxSldkiw8uG6AfaOLg2d3Hv9fDuTiBs7mROZmh8IKIYmhKjf1vtBSbp/OsiGybiz8Ysh7Tzc7J1btZ15LFHjW/lbncClnlk8btKP2/93+V4MV9/de+jXa7ev6muqbgKKKopa7SlRRdsHAIAPCZKDAABaoSWmJjSbfHbL7qf1hk/vX3zYHc9h5Iyewlubt92W0camhT/BMw7TD0WG+//akV+0equp647eDboVcO2/oHt3H1z8bUh9Wv3jJVanTDpMWbt17/nL10IeBD8NC3544+Se5aNalshDiqy8J/+w5/TVkJCQyJCAu5ePHdzyw2RPo5Jf8/PvnriYwPKdP/F5+xZdvXYDu5lS0pvHfVM5QiiTLot3Hrt+O/BJWEhE0JWzfy0Y7KhuHAO/w7L/osKOfeX0enuU4aDNERH3/x0qeb0KpWff76u1xy7fDA8Nuue7b/0Xna2K7s9W1znqleiQoHu+e9ZOauvoMWTB2l2XbwREPAq6d3HnyjGuRY2nJO0+W7Vl11nf66EhD56GBgRe2Ltx7ieur4dLaNeEwgKXzyMePY94FHV9aXt+Ge0ihNCSFmOWbDnrd/vxw6B7l/ZumOFV2u3chBBCdD17tBMrQ69eSSqWmhXU95r43T/HfO8HP3j68G7wtdMn/lm7fHATRquYS56EhBCiY9N9+s8Hzvk9DL3/KODKxd3LBtq+myqjTTstufAg9P7uz1xLjiApb89XsotKNkHjaVnWy6Qg2PdGJm3v3a2UJgN86NissGPrDoaraLG9fb1yn+Ly6KhYlrZu06H4q0PPvWNLEZHHPItly16AcEqlkiO6urpVmFqSR+5Z/O3ZFIMOc1d/7iQsfZGYqFiWtmnfpXEZMxgWqY44CW1qaSkgef67N/g+TpQqVKr8tJTs8lVEKXjpt3vtwi8m9OzYue93FxM4SZdenqUP4Sv7WAAAAJQbkoMAAFqhjCTGNJeZfGfXjmuqthMntSy6TqEMuk0Z7ZR66s9jEWlSTiQxKSWFVliIsI+jEZWXlpSUkUeJCwsRqnv8nfUlzXv07+zmaGWiL+IzDL+w0OHuzRObvB5mJ3KavPXAtrmDOziY6wsZnlDPxKpJ664DfFoYv/NGLw86fjpKxWvi09flzSg9yrBTP29jknH1xOUMjhBClEaN3ZtYGevyGUZQWHhxx4p+6sYxlEnX9cvte/6Y1svNylAkEBlbt+g/c/2+ZZ2NKfWdo0mJDhEVVl08v2/FNB+PRqZ6Ap7I2LbViCVbVn1SmGiiTdx6D+rq0dRaIhbyGIGeqZ2bz5SfDh1c0Vttrq7S7XpVQ3PHD+M6NbXUF/JFxjZufYd3bqjmqp3n2LKFHhsb/ODNsE2R0+S///fvghFdnOsZ6fAYvo6hRcPm7XuN6NJIy4vgkoROk//a/9ecT9o0MtUTCHQNLezdrMX5JbqaMmw9a9vvI+pH/D31ix2heSU2Ud6er8ouIkTzaVnmy0QWcu+Rgmns4aZfsf4DqEUELaf8/M34rs1sjEUMRTE6koatB80Y2IThlCnJ6eXOD6menDwZJue7zly7dGhzC12ewNC2/ZTflg+vR2X6nfJN58pegHDJiSkcU6/HsB4NxTxGJGncyqV+5fPwbPK5Fd8fihO2nLF8clNBaZFHnDodruA5f7Hx18kdG0tEDM2IDE2N1af+qiVONi05WUF02gwe41FfzKMIzReLReUYBs807Dqka/MGBgKaYvg8ZU6OjBCKUjNEsOxjAQAAUG5IDgIAaIU2khhRXE5WdvL5fw/HNRg6sWdhRoKxGfR5D3Ho3j23pdlZORxtLDF5551VXSHCMgoUlqC+0CEhTOOx3831NJI/O/39uN5urs3tXdt4feeXp+YaQfX41JFQOW3X5xO3omstyrjbAC9DLuns0Rs5hXsrs/BiOTBNxi/90k032W/9pL4dnFw82gxdcviZymrgl2PsmfJ1QmkdokUhzlfLn53f1cXFtXGztl4jFvx9N4Nv+8mP88ueSf+NkgUuNbWrnDU0KbFdQ0tG9eJZTNHzjMP4ZXPbGCtjLv485ZPWbs0bO7u3HLLpQSkT8muJaThm6deehgVPji8Z18e9uVvT1l17j//1Qmrxs4Q29Phi++ZJ9tG7pk3dEJCt5gQqTwnUqusiQrQ5LTW9TLic5y+SWL5dI+sKdyKANoRCYf169ap1FzznbqMHTly+5X9+t4OePn74NPj6lT3LhzoIC57+7+8LFcgPqSJ3r1h3N1vUdNhvh648enQ/+OLfi7rV4+LPL/u1cHNlLxDjdyVMRtsOXn0l6MHTB9d9/13s06AKLjS4LP9ffzgRz3eZ/t2Y0uYEUD3ZuWx1QCbfptfibSeDHoREhQXdOzyjudrMXLXEyaVdOXg5lbLo+t2+S6GPHj4PD76/bUQDrXOOlEmbz5ZvOHHlVkT4w6cP/C+tG2JHZfx3MTBXTRPKOhYAAADlhuQgAIBWhIaGujQrzcllZcG79twXdBk/yoEhROQ5frRb3uVth1+oSF6OlKOMJO/cx6u+EGG5ChRqKHTINOrr4yJQPf7zqyW7Al5myVUquTQlTar2GkEVc/Lo3Xy6fr/BbQvvWqLr9RrSXo+NPns4sODVMmUWXtQe49i/nxMv2/eneVuvRmXKlAXJoceWr78qZRzae5rS5eqE0jqk7EKcRctL09PzlCyryIkLPv3LjGUnUoik2yddDCs6HFJzu8pXQ5M2MZXQbF5aWlFGl2kyYICzQBm26cv5f/s9Tc1XsSpZdlpmfoUv/JhG/fo3EypC1s/6bm9ATIZMUZCd9OT+k5Q3mThK0nb2v39NaRq7d/rkNTcy1O+pHCVQq7CLCmOsRD1QQrj01HSONjGVVLQTAbSiIxL9ve3vXbt2zpw508urg75+1Q9WZZ+d/HXt3nMBT+KzClQsq8zPjIu4fXzzwqGjV9+sWLHz/LA/J4+eseHM3ej0fIU8N/nJtf2/jB2x8GS8SssFVJE7Z83752pkap5KpcxLe3b/aUrVzF7HZV7bsOZqpsjt86/7mpSyxfzwvyePmLj6sH9EUrZMpVIW5KS+DAvwPeL3TFHa5qolTi793JLJc7dffRifLVOplLLcjOTYJw/u3H6apc3BoKiEe/+FRKfnK1lWlZ8RE+K7dcFn806nqF23zIMFAABQTnV13ncAgHLSN9SnOXlurpwQ9uXxPRemrh09rt0/600mDbB8eWiBbyZHqFxpLkvbGhi8c5VRWIjQ26fz4l2+czOiHwYH+Z3cs+N8ZK66x8u+mCgsdOjyqtojz9bBjmFf3vzvaamXQu9ik84fvvxVW58eg7qtun4qk27Uf1BrofLRsWMPCwemUQadlvy7bZRt0TTlQhtrQohKfeFFjQTWjaxoWqfXhoBeG95uRX2relTFO6H4lhKi45WkqZmFEU3yWEKKqi6a6+qouejjsm5evicf2N2mcQOaZFakWZrbRfPNylVDUyASUEQul7/+26axFc2+vPlflJbHtCyvTpKAmzFqrh5po+6fTyBs5pX9e2+laX1Xouae51dlF5X/tHz7ZUIIJ5crOCIQlnZzIkDVyZFKOY4zMTHp0aN77969OI579vx5YGDg/XvBERGPFYoqeFGz2U/Obfv53DZ1z6sitwxz2FL8EbnvvDaN5hVf5NmmwS02FV8k/8WFjfMvbFS/1zIWkEdfWDPpwpoSj74TCVHcWObZdFmpm8g9NdPp1DuPssnHvvA6pn6DRBZ3bdv310rvjVK6otQ4S6O8t6qP/aoyN0gIIVxe5PFVXx4vubDaVd7qhCS/NV/6lRZQ6fsiRPOxKHWtCp0SAABQZyA5CACgFX0DfYoryCvgCCFctt8/R6P7jfn0G07SWRD8y/4QOSGEy88r4CiRvj6fkBKXflzqmcXjpPeH9Wnbwr2lq7t3Q48u3k704C/PqHs8o8x43ip0SPN5NCFKpfZjBrgsv31nEvqO6Tiyb70zh82HD3bi5d3cd/xF4RbeFF5cumrv7aiUfJ5pt2+PrxugdmuEJUQoEqnLw3HqykwKdYSU+s4pT3pQcyFONWGxHCGvYiujCWo2oKld5ayhKS+Qc0QgeJ224jiWEKJiNXRC+WKmaJp63dxSNye9dynItEsn7++2r86bNPd0nHank8aer9IuKu9pSd4pikoJBHyKyGVyDasAVJ5KpZLJZCKRiGEYQghFUY0bNbK1sRk5YoRCoXj8OPzu3XvBwcFRUVE1HSkAAAAAIUgOAgBoycBATHGyvFd3dSoeHtgfOG7xhBFcxrl5x18VB5TnF3Acpacvpsi7U+YVvPTbvdZvNyGMvtOQFTuW9ejSy1P3zNnc0h+/UL7gFInxqSxt08qzPv3wpZZjvgoC/3fs8cgvPEcNbZvRYLANlXby4NnkV5mcV4UXL+3e4PtYTgghitIKLzLMq88QNidLytENHO0NqeC0UpJBirgXcSxreOLznkuvljqZoLrO0a4lFaTj6tlMQORxL+JYQkgZTSi1wKXmdjHOUbEsbde+S+NNoU/KHCjEpqWms3QTExNdimRxhBBF7PM4lrZ297CkH8aVekzL6vYSFHEv4ljaxrOdNRP6orS8H6d4eujrqQfn7NowdsCP6xKSJq0KrNjdiSV3WkVdpOVpqQklMZVQbFpqenlWAqiIvNxckUhU/BEej0cI4fP5zZq5uji7TJz4aWZWVuFTfB6+kAMAAEBNwpyDAABa0RXrUiS/QPYqW8LGn9l9JZNVxZ/Yd7Vocja2IK+Ao/UNxO+8taorRFiuAoUaKMMvXUlghS1nr5nX38VCLBAa27Ye3KPUyo5vqJ4e3Xs7n7EfuW5JTwkXc+yAf07RU2UWXpQrlBxl5N6lrZUuQ4jqWWh4Nif0mrZodEsLXYZmRPpmxsVu51VFXLj0jDXtv2zVZ92cLQ0EDM2IjK1curS25WnonPJ2QpkoPc+ho70dTHR4fAPr1hN+Xj7Kis69c+l6Fld2E0otcEk0tqt8NTQ56YsXSSrGrlHRHImqyEuXX7BCjzlrvunnbKbL4+vXbzFgbM9i8/GXGfPbVBHnL0ap+C1mb1gxto2dsYhh+GJLRzfH4gV0OFXq9VUTvj4azW86+belvc0q/SVB86EvZ5nRytYDJZS+nZ0FrXjxLLay7QJ4h0gkMjM3b9yocQu3Fp06dVKxan+noSiKZhhCiJGhYeEjEhOTwjGGAAAAADUCP1QCAGhFV0+H4mR5+UV/c1nnvvZq/HXxRbj8AhlH6RmIS65LmbT5bPnS9vxiD7Hp5y4G5pl0K/Xx8o+YKwjcuvpkt9UDW4xff3R8scc1lrZlk07tvjC7/SALU67g7oE9D97ca8mlXTl4+cuOPl2/29f1uzcrqJ4U/SM2/HEm5+Q0fssF829azz6fe3337vDuM136/Higz49vln+9QeXD7T/t8N4yucfX23q86TLF/VU9Ru+MUdM5VT9skBLY9Z6/o/f8N/vJvL1y9enC4ZJlNUEV43clbJZr88GrrwwujD74577jtqlvVzSrerJz2er2fy/07LV4W6/FxQIpdbCb8sm94NxxvVo0t6BD41lCiCJ0+697u68f13LChmMTSGmrlxVzyT082v7jX502z2g28IddA38ofIzLPTWr06yLxUftsSlXfp6xzu7g3D4//nA7dMbRWK2nHyx1pxoOfXm7qKzTskxCV3dnvirq3oPs8jcE6iiBQKCvr69voK8vLvy/gb6BvoGBgb5Y/Ppxsb5YX1+fz3/zLsZxnEymaVSrUqnk8XgRERGOjo6EkKSkJJUKpSQAAACgxryVHFy0cGFNxfEehD0OP3H8RE1HQcjH3s+1yrHjxx4/jqjpKOAjoafLUFx+nkzTFHD5+fmEEhvo04S8lVApLETYwMOhgZGQkmXFRQad371p/ekUYl764xwp9xASNuXS/DHTn8yaMqyzq40hyYoJ8X8m7tGtCctpSu1I/fcdiur/ReMc390n37odubDwYuLsz/p4OFjoMcqCnKyMlISY14UX8/zWfb1JPH+YpzA2QU4IkT1cP2Vq9tdfjvF2tTHic/K8zLTEmKgwv6evbsPmcgJXjh39aNJno3p4Olub6DEFGfFRwXdfytV3TmVvaH0Xl/vg3IVsh44eDY3p3MSI22e3rtt69nU1y7KaoIrcOWuewfczB7RpZCyQZcY8fJpCURraRcirGpoR46ZMHtCxuZ2pHqPIy0x5+SwiuNQamrkBl29LfTp6e5vv35vIEkK4rBsrxn0WNfOLMd2a20n4+UlPbt1OdxzYuf7rVcqKuWQHSIPWTBj7+LMpE/q2aVrfSKDMSnz+MCqbT5WcI7MgfMeSVe0OLuv89dL+N6afSKpMerAqu6is07IMohbdOxhzUQevlHpXNdQxAoFAIpFITCRiPX2xvp5YLBbriQsTfWKxnriIRPJWbWu5QiHNyZEWSUpKfhoVlZ6Wnp6RLs3JlebmSKVSaY40Kytr7tyvO3Xs9O6MmiqVihDO/7r/oSOHXzx/cebM6ffYaAAom5OT46CBg2o6CoCPXC25Tsfr/ZeVK1//mzI0MXv9x8f9BcX/un/xltegj7ufa5VfVq70v+5f01GQgV1l/67IJoR8scrq7A2Dmg4H6gjKbPjW6yta3Vna7dND6VWfaPvAMA7TD5ydVe/oFO8F16uo+G81EHf96comn4R1Qwb/FVVq+oquN3rvpW9bXpnrNut8wfuO7kNHGfX61Xdd9+erBo78R13BZvhgbJof27dDNiHEyMus+OMCgUAsFov1xRJjicREoi7lZ2RkRNNv7povkfKTSnNzpDnSHKk0V5qenp6enl6Y8svOzlYqNQ7HLmb69Om9e/fiFU0myLEsoaicnJxTp06fPnM6O+vV8NXC74QBAYHrN26ufLcA1Kw9u/4htemaq2K8OnphJAdAdasl1+l4vfv49Hv9b9xWDADwMaDNPfq0IJGPnsenZhXwjBt5DPhmehsBG3U/VLtBVVALSK/t3PPYZ9bYz7v9b/HFTBy3KsVzGDu5u1H6xW1HXyIz+HEoYOvHFExbscJUrC82MDAwEOvrid+a00Eul+fk5ORk5+RIc3JyctIz0qOjo7Ozs3Ok0tePS3OkOTk5CkXV/2aQnZ1dWKxbpVIxDBMZFXXs6LGbN29qn14E+EB5tPL45ZefU1PTUlJS0tLSUlJS01JT09PTMjIzazo0wuPx8BoEAChVKcnBj+/Xy8JfsWqbj6+faw9Pz9azvpxR01EAvFdCt5Er1/cVF7+DjVPFn96y7wkyIR8OZeS/a44N3Tpk4ZfHb/10p9KlguE1puHIBZNdFHd+2uyLbPnHgiKsiuikp6VHR8fk5GRnZ+XkSLOzi1J+2dnZcrm6+TffhxxpDp/PV6lU/v7+x4+fePJE67kxAT5wKcnJ8fHxEhOTRo0amZhI9PX1Cx9XKBVpqWlpaWnJKclpqelpqanJqSnpaempqakZGRmFyfTq9v3338XGxR45ciw1JaXMhddv3BwQEPgeogKoO2rtdfreRLPQHN2ajuL9GWOZ4qqfV+JBjBwEAPgIUILMiKsBjdwcbCwNhUSWlfAs9PrJnRv33kmuVDUJeM+47Bt/LN1rOy6DE1IEycGqw+fnxob5+i49gBuKPx5COrGp7rx1f5iVvWhNSIxP2L9//9mz59LT02s6FoD3Kibm5YYNG1//yefxTUxNJCYSiURiaWEpMZGYGEucnZtKJBJzc/PCG/yVSmV2dnZ6kYSExPSM9PS09MTExJSUlCos12Nna+fu7u7T1+fKlSsHDx6Kj4+vqi0DAHzokBwEAPgIcFkB22aN31bTYdRaqsgtwxy21HQU2uAy/X6a5KfmSTZh36hm+95rPB+JgifHvh91rKajgLokIDAwIBBjjgCIQqlITExMTEx89yk+n6+vry+RSCzrWUokEhOJxNLC0sbGxs3N7XXekBAilUoTExNL5A3TM9KTEpM0lwUvgaIoQyMDQgjDMN5dvbt37x50N2jP3r2RkZFV0lIAgA8akoMAAAAAAADwXikUisKU39OnT0s8xePxDAwMJIU1xSXG9epZSiQmlpaWbm5uZmZmDMMULiaVvqoalJiYmJb26h/pGenJSckFBSULdxkZGTHMq4tfHsMjhLi5t2zVulVISMiuXbvCwx9Xc3MBAGo1JAcBAAAAAACgtlAqlYVZP0JK5g0ZhjE2NjY3M5OYmpiYmJibmklMJDY2Nh7u7sYSyesS4RmZmenpaWmpqSkpqWlpaSmpqSKhsMSmeAxDCHFxdlm9enVERMSBAwcDAu68h9YBANRCSA4CAAAAAADAB0ClUqWmpqamppb6rFgslphIJMYSS0tLiURiYiJp0KCBq6urubm5SCQqdRWGxxBCHOztv/9+aXR0dMjD0GqMHgCgtkJyEAAAAAAAAD54UqlUKpXGRMe8+9SwYcPGjh3L4zGlrkgzDCHExsbG1ta28BGKoqovTgCA2oau6QAAAAAAAAAAqpGuri4hnLpnWRXLcRxFr1UgfwAAIABJREFUUa8nK9QXi99XaAAANQ8jBwEAAAAAAOBjZmpmytBvjYxRKBQ8Ho+iqNTU1NCQ0IePHoWFh9na2i5csIAQkp2TU0ORAgDUACQHAQAAAABqjL29/awvZ9R0FAAfOQsLC4qmVSoVwzBKpTIy8mloaGh4WFhYeLhUKn29mI2NTQ0GCQBQU5AcBAAAAACoMRKJsadn65qOAuAjRxHq9q3bDx89Cg8Pe/o0SqlU1nREAAC1CJKDAAAAAAAA8DGbN29eTYcAAFB7ITkIAAAAAFADfHz61XQIAAAAAKhWDB8SxnbwLycvHl3siaQ2AAAAAADAh+IjvpSj6/dfcfziqeVe/KrY2vvuKMai/Yzfdl+9FRQZdj/0v42jrZEjqjGUUDizl8mh9kJBTewdBx7eA6H7rP/du3v2154mVOU2pG/t7GxtJKIquRkoL8599PN7+57+2k6JrgcAAAAAgPL6eC/lKD2rpi7WxqIqall1dtQ7F+YCl9l/bfxmgLudRMRjBGITWp7NMQ7j994IuLFhsA3SRe8XxTAOJjwJr/DYU81aSE6PMF1oQ7+f10wVHG1R29mHzly6GxgUERb6NDTw/vUzJ7b9+u2n3Z0MGa23wbhM3Hz+8q4vHLVfpc6gDZx6T1259eC1WwERYcGPbl04/e+qxWPaWQnLXLMW9SpFURRFv6eT+iNi5P0y/HjYtc/zyv7ZiM6ftzH82cGXA/WqJRKKcBRFcAQBAAAAAAghev03PI54cGp642LXWjzbAb9dexgaeuirtkY1871Zr/+Gx4/vnp7n+c7++Z1+vB4VdmZh85q/NnxXFYVdi65/y1Dxa/zKKnFhLmwzfJSjQB55cGY/LyfnFq7dlp7P5ghF0RRFM7j2e0NoKd7U3/TkcPMrYyz8RpufH2r6Tw/DmU5Cq2oe30lpl7NzaGq0c6DxOONK7asKmsKY2bva1391GjO6RuZ2RuZ2zTv6TJwWsmvJvJ9947QoBEUb2To71MsQ4PR7G2XQ/PPVv8/vZMkr6hmBxMqlnZWzm17E2duxMk7j2rWnV2VBfwxv+UdNR/EByokWxnA5drYyE0o3SePRpnRlTuacKl4UXlAdgVBB+xq13FcdWwYAAAAA+Agw9Xot//en3iaRu6dMWXc7U/OVWnWidFwmrV2fMOHzPVHyGgui/Kog7Npz/atJWdf41brzEhfmtIV9Y0NKdn3HH2ciMzlCZMlphBDyZOeo9jurNY4PDqPDczJkXt3tS1F6IsZexNhbiAY0yf/BN/taXnXsk3v4IN3ngTZLUob6fDs9tpI3I1fVOFFl2KZhzs7NGjm7u3boO2j6j9v845RGLT79/a9v2+nX7tdmlTE1M1uxYnnXbl11dXWrZot0vcErNy3sbEGl3d/9w3Qf77ZOzdybdxwwYs5v/+48dSOr5j5voIiLi8uihQvbtW/H51XJBBMlqeJEj/MJz6bAodivX7R5+v8Oh0WuTS3+IGMjc+AR6QtRtKo6AqkutEBlZqI0FhWdzJSqw6fP7u1+trjFB9UMAAAAAKjTaLPOi/5dNcAy5tDMKatvZFTjlRot1DezMDPW1TDKh1NxBl4L1i1qb/hBXYl/oGGXU/Vc46s7K8o6WyihSEBx+ampuR9ZcqFRo0ZLly7t1KmTQFCV0/dFhqR135vUaW9Sj0Opk67mnE7nBAY6c5oLRFW4j5pTZYMgWYVMruI4IpOmRgdfiQ6+evbyvB07JjmOXTju4ODN4SpCmXRZtHZ2H0crCwMhl58adffCtrUbj0UUOwuZJrNOhMwq3FrygXFdf7ip0GKtWoOhKQ8PDw8PD6VSGRAYeOXKlaDAu3KFosIb1G0/7ZsuEpJ6ZfGorw7GvBp/KUuOCjgXFXDu1TIV7FU9e58pMz7r19bJXJifFOF/7K9VW/1iX0cqsvIeN/WzT7ya25jokIKslLhnTx5e+mfNtoCi37907HpOmj55QHvn+npsRnTQlcObNx0ISCnM5lCGbsNnT+jZ2sXeztJIh8pPjb6wfPyKpyMOnJ1V7+gU7wXXi3ajY9P902mfD+jQzMqAys+Ii7i2eckPx6NVH9ARJ4QIBHyvjl5eHb0KCgr8/W9cvXo1JCSEZdkq24FS9DCWGmBf4GxG/BNePWbZOqcFj/DscrrXM42MffWgqW2BJUPdjxLJCaGMchbNTeljJ7fQYzkZLyrMYNtu82Mv6MI+NHTMmD0gu3VjmZ2JSoeiUhMMly+pF9Aw9dsBUmcreX0jlS6PFOQIgm9L1uw1vp/9avsOw5+dHaU4+kOTBfcoQoiJWxnLE0KIUO7tk/qZt7S5pUqHUFkZgmfRoksnLLY9ZDhCaMO8yVMSp7YtMOYRjqNyEvVXLLU6kl7Qs2OBsQHp367g5wfVc4M0AAAAAEBVoiQd5v2zboRd0sk5n/90JeXNtYD6y65SL5qWnac6a7gUoiWtJn/3//buPC6K8g0A+DMze8HustyXKAqIeN94onhr3qameZaZZaalad5XZaZl5lWZWWlaP628T7wQREQFFeRUlBu5Fva+Zub3B6CgsLssICjP99Mfscy888z7vjPyPvvOvMvnDPS14xIsq5enBq2f+fm/mS8OPQx39+/MHvLxtE3r7k1cdCSz0i/dKw+P22tt0L6J0h3jxn8fXzrKG7szYmOP6ysGvPNPAWtR/KZUP2wAeHH8u0E+7+iROa5Bn/X/8KS8uCJd394TvKr50TkDP79aPE+PavvpkSPv2/33waAlwRpLRrvlAiQd+yw/sP0t53tbp8/5Nbr8nDJzxvjPn6/RAXJlvaKSz4nmHz43MCeAtJv4y52JxQfT31wz6N39unG/X1nXLXS5/4fHZCbqvOKecKagjofvXA63e/du3bt30+l0YWFhVy5fibpzx2Aw46FWo1gG9CywABotnZSh2qwgmg8X+TjxmhC6LAerd1oK2ttzGlmTAmClcs3WC7JgDRBcTv/WwolNed5WhEZtuPVQ+dN9bXbpVUsKuKPaCkc35jURgFppiHzCOJZJjTdtY/9be+rc5byNmaX1yaF6+4ne8uL5CgmSZrOl2v3hsvPF/ZrgzBzuMhMAABi1+tMjssgqpiVq7Qlptih8+zeHh+yZ3nzYcL+f4+7TYLD17uTrUZy3Fbm0DJy+uY2zfvRnJ/KM9hvL9qpTHA6nW1f/Ht27a7Xa8OvhwVdDIiNvW9ARe4wc6Ezqon7d/G9q5ftaUD/Wbef9+ssnHcXFs0YFjduP/HhbB/f5o1cGS1kAgd/s3XuW+tuVvoZA6ODh6+DhxYvcuzeikAYAQas5u/cs8ZeUTDp18e07eVnPPu0XTV12IpMGIJ17jJ/2RqvSjiV2cuZqFS/EwPeb/fOvS7vZlhTCc/Hp0FikZiw8o3pAIBD069d3wIABKrUq5GrIxUsX42LjWLbaMdPce0kc2lfb3ouBLBIAgNQP7KniKqlCa83Qbtrd6XwaAIBt6aOhaO7dJA4DQBho75Yaj+K5jNaGll0KNvsY9As8ThQCADi3l07rrSltINbJntWqwN5XNrLz0w9BaKvtNTSrgyczboVDYkX/Lpvenq+ZvSplaRu6tCOxDi4aBxctL85xbwxFk/oJC9KWdKYJmszPJwlr2tYetAoARhB0TTByAJwKfz2+fUEIIYQQQq83UtJ1/t5tU1tIzy1+b82ZrDJ/OhsbdlU4aGLBqvKhEOk6YeP2JX1tCIMy/4mCEDnYOnO1RRWP/umMM8sWiZvtfWf9d+8mvPNLbIXvHTI+KjR11lWO3wy1EzadEB6R+/6E9h39uCdv6gEArDp2acUlrTt09KKuxtEAQDp26NCYVF+9FqWt9miXkHSdv+f7t9wTfpn10d7o5582FZg1xn+OkQFyZb2i0t5i0dsYLejJ9QaPxwsI6B0YGKhWa8KvX79w8WJNzuYh4Gkqz8HVaqwnt7QeCHtr0OkBONwZ/e3ecSKKq44v4g5ob9tKWDg7XFsEQPB48wbajrctWXqGJ+b2EwMAVPpYPcWZ1M/uQxeypHNShKcjZV3dhOcztfn6RPXdKzeKpo5r1LK5EO7LWPm1zdPHrHiYlqvQcyVNery3Ycesfm8G2p38pzSrTCdue/bVRAnTe9VLFIcCAIFAENCnd9/AQMsSRq18RST98GpohpEHLKteq5TvrFXzOljnBG9b/s3/wlI0kpbDFn+z6s0x86b8HrojCbynrl7kb6tLPvn12h3H7mQqwMp17Kbz63o93d1n2qpPu9po4v5Zu2bXqVgpz73LxKXrFvcbtnbJpdBPz5bckVl50NrJy46nF9JWLq5WRXpwL189zaasWugv0SQe3fDFz6fvZqn59k28JdI81qwzqq8oigMAQmvrgYMGDB06RFogvRoacvHCxYcPH1ajVCIuUaAdLm/lo+VcszIAkC6ykS0g/ojr5W6ZHwTImh91iqcBKF1rL4bQiCIfEwDAqkSbV3qvSOPlqoEr0vcYl7FjjPzNLoaTFzgldchSQT81XXaFV8gwLg5MkQHcAYClzmz3WhrCVdCMW4vCNYuyB/kWTGlpvyamknn9Rrf3Hp61qA2tS5d8/ZPTsQSeAhjX/hnnPyz5l5OwVg5uQzMPHMevdr6rBCBYJ3eDXgPAUqF7vTrtrUaFIYQQQggh9JIQ/OZTdsx8s60ufP2cFUfLpXso3+nGhl0AFQyaWEOlQyEQdxvcTczE/Dz+nZ13ZTQQfKemTvpK33TGKm7v+OT7doc/n/v9p3fGb7wpf34oZUZ4JlUlfvOGcjUR9otZhXvXIxRvvdG5ixd1M4EG4Lbu3smaALJZl04uZFwmAyDs2L01Vx8TfkNB+syqzmiXlHT+6Ndd7/qk7JszZ3uE7IWTphqbM8Z/oZqr3CsIm6r0FkZ6qOwTfgCEQxXr/IUKqVeKx+lWVoKAPr379e9XWFQUHBwcGhoaez/WsgIJgrDmk00c+KPaC31IkObpU1lwBQBgQ28UbHpEy1jC0ZqQ0+DVRjzdicjPUGyOVN+Ws2I7wZye4qFewtHx2n2F0KKVeJwtochTfX9TGSplKWtOj+aiea14okqO29jX5j0XUluo3nVTeTmP0VCEu4QseppDZw2/n87/VWrZOQHUbnIQDAUFRSwhthZZkyBjCMK27aQlK7q38nSz5yqz8hgKOC5ujiQUGLswLNur3niWMBr4LGFk5r5iIQGMtKDQaFa7qvVDtRg5wo8ju/DV4t2Xi1gAyIk+sm5b7yFbB/T0d9yVbPPG8NY8Ov6HT1fuSyi+phW5+WUy/1TzUaNb8/TRmxatP/yQBgBVStjuz9Z4nvxpcuCoAXbn/ikAAADWIM1Iz1fpAfSZKbLnv6CgvEaMbMPX3/tm/uoDj2gAAO2TxKgnFp5R/cOhOABgZ283/I03Ro8alZWZlfn4goY5KiDTTe77ImWidSIjb+2rdiKtshjw6iXrQPJ/uGoTZMifM0k2yssxPokgrNUdG7GGJKu7pd8y2DYvWPKespW73p5DZkkJCsDFyUACp6QOWZDm8PI1BACVmVXaOizIpRyZDgDIjFj7DaeK+s3Q+DUzkDHcirugke0p7RsBGh4j+OFb932Pi3OLZG4h+awjsQQLQNhp/ZvpE+5zNSyRm1ErL21ECCGEEEKo1lDNh09oDkze5Ythzy0kYXzYlZQH8OKgCYCsdCjEsCwLQDj5+fs5Jtx8omG1uY+MDy50ifuXr+/696ZpX64In7TscvmHuUyEV2DW2VclfrOHcrUQturW5ZuqkQHdurn8nJDJUM27dXeU3b+v9GvTzV988GgRy2/fo6uQvn81LJdo/rblo13CvvuC39+a6pt24IPZ31X83klCaNYY//m9qtwriCr3lspZ1pPrJQ6HCwC2EknJOD0r+8HDByb3Ksu3g0Nwh3Kf6OWa7fe0JQk6li1S0lIDC8A+kQMQ3AFNuZROs/Oa8roOACA/X/3DPV6fAH5nF+rPIrJPYw5J6/aGyoOKu7lCfzFBO7Ilr3WFxyY4A5pxebT+p6uyo8V1TLOPcmvuhWa1nBzk2NtLCJZRKVUsYdNn5e97JntySyYh8Zs0BgCaJI0GYNlelegd0PtUwEkLdqwRHE5Jwmj0qFHFn0gkNsZ3UaoBSImthIScSu6lFtQPr7GXB0laDdkeMWR7uV/Q7h5uJMexeVOKSQu78qCSbD/P08eDZNJuXHtcJiRlZEiUZvJQT5/GJJjz7wjHs3lTikmLCEt94bxqtMUBYNnSpbDUsl1rQHGju7q7ublPuy2fakPdtXM8AJBXpULoHOubOdC+mbodF7L0mtF9NUyS84lMIv2azf2JOaP6q7YlCRlvdRsu8SjWOocBIOg+7z/eM1RXWod0E1cAIMgqvtc3M42nZDUiK8bM/cptT2mbu7NMtuhKasV7syrhkQhOvwD58i/li2S8mARhcLD93mv8evpqSYQQQgghhCpgiD+46WyjmXMDVxze1+TTed9dflI6wDE+7KpwRGB0KMTKw45czOs3vO/yfRcWSVNi7twOPv7n3rNJxv5+pjP/W7OuZ6vvx69bFhy9Sln2VybCMy85WJX4q6DGw2aLrl2K0vTv3Le77f7/ihr37NlMfWPJLumSbUP7dLY6eknXNqC7PZO470o6zRto+WiXtB343gxgCi/9deB6fiUpG1Zleoz/HMt6hQW9pTIW9GQz1Idxupubq5uba/EnTtwqPJ3LMKxax2QV6e9maI4maR9XNlOSohqLgOQI1k4UrC3/G2cRSZJkIyEwCv09ZYU7v4CkmtoAo9BFys2PtGpqMzlo1T6wm4RkUuKTlGA/eubYJpT0xo5Vmw6EP8xVcxwHrDi6dZTxAgj7gRbsVZn4+PgjR49atq85bCWSDz/80MgGNE1TFPXkyRMXFxcAKCoykVNPeqRmWzTr3tVpV1J2hXcXS+qn0qeaCb4VnyC5HBLAYKj8RlUTK0cRJEkAVBRIzbY4ABw5ejQ+Pr4asZrQrGnTSZMmGdmguNGl+Y87NjruxA2S5vEATCSFXyhCEHafM6ufunMz9gpVOLoRcfMXmzQamEzJ0fjcVT0L+/xpneqndmA5Z+/yaQBCIp/ZX0fJhDt2uhyI5udqWMdu2UcXF1X11Fg9qWOBIM39B6Tc9gRwCAAaKu1ILOfU9qaK+MJh7VSdWqg7dZV27iL3I7zmhXAwPYgQQgghhF4VhpzwHV+cuvbBlp/mTf9xn+3iWatPpBsATA27KvyF8aEQm3dq+TRF1IRh3dt36ti2U79mnQP7+ZHj5p0y9j4/Nu/SF6v/7fzzm2tXhG5Wl/2FifBYYAD4AkEVRn81OJSzOOzKyssPvRyp7eXfr7vk2O2APi30t/6+GibtIZ3Yp097frCsX4AbPDx+8RENvGqMdllFZNBtx8A+/Vb/+q3q3UUnK3pwmM4wOcZ//sQs7RWVfF71h04trHMTanuc7u7mPmPGdCMb0AYDxeHkFxQ42NsDQK7erMxY4p382TEGc2frsVDZtcmnCIIoeREhaWZpUPJqwtobL9dacpCQdJ+3ZEIj0pB47lQcTfq4uvJAFbR/+4V4HQCAPj9XVmbiNWswGFiwtrYu17tIR+N7VU1ebl5oSKile5vm4uJcYXLQYNBzOFyZTHYl+EpISGhcbNzJkyfMKfD6xXDZkAHdZ88fHLTybEXzRU3VT0W1qs94nMEwkmPvDV51+cWXDnA6ZuYxZJMu/u5kTFpFfV6X8jCdIT279fKkopNLb3fCTgEdBaBLTU5nzOrb+ozHGQzZxL9HYyr6cbl7pnktXvystlni4+NrtdGVihcXWwEAMBgMHA4nv6Dg8qVLF4IudvZ9MGJ9cS7Yo+oHIaLuWqsHyLu31/R1kblrRN9e4zIAwHBPXRQt+lg+sYfmahstobEJe0AAAGmrd+WC6rr99hsCHQAAkS+lLL5qLGTgZBYC6aryd4aY7Eq20fKCTzoHnwSgaL8BWXs/kAX2VFmH2Jj5xQlCCCGEEEL1AlN4a9eHb2V/+ev6UVv28Tgzlx5JNZgYdlW0LoTpoZAmLXj/luD9AJTY7831e9cOChzib33qtNG/n9nC0C0r/vL/ffLiT55YE1A6PcVUePIiBUs2auEjIe7km5mMqNGhnGVhcyrMKgAA8+TSyVuf9eg+qK+3eFA79uZX16Qq1YVrsjf79u96VDawKZuw43wiXb3RLqt/cHjhnEOf7Ns+ddSXW7OevLvpxVcmgsrkGL9USUVZ3isq/vxc5UesRNV7sjlqe5zewrcFzKjg8+LkTKFUGhwScuniJVc312VLa20GI0NnKIHhqZcek11/cWIiwU1VAGnD6y4h4gvNuMgYOkMJpIjXSQwJz08zYw0sC0BYVS+9Z3aa0mRBHC5FAFA8oaNn+36TVuw59NuslgL944Mb98XRwOTn5OjBqtu4KZ3dRRwCSK5IJCgTOZuTnctSboMmDGom4lACe+8urd0pk3vVa7TBAAAajSY05Nq6dV9MnTrt5592x96PNX9BEunZH/fc15Luo7b+vWvx2K7ejtYckuKJnX27jfjg07EtTddPRbUKCeeCkhnHkWs3zRrQytWGR5GUwM6jdWBXTw4AGOKCLmUx/I4Lvls8srWLiMe38+w6blBL3rOzSjx+PFbHbfvxllXj27lYc3gSz57vb1430Y0oDD5xwcz3zNIJZ88/pLntF2xfP7VbUzsBRXFFri06tHAgTba4Tm9gCdtOgd09rC28DdUqg0EPAEUy2ekzpxcvWTJj+ozffvs9LT2tmsXK7olu6Vi/nk8+6mnIvWZ7ofRekBtue07GBIzIHu/LqmNEEVoAAKaQm6MHq7aFU1rpRRQAyYqsmZd91dBWQRE8hqdasCh7pLdexGXt3JTjemiedSRK13+gvJ0zzSOB4oBBRWoJIAiWIOheM5Mj9ycvb/+qvGESIYQQQggh7cP/lk1fduaJ69CNu1f0dyCANjrsqoiJoRDVrP+b/ds1suGRBMXlGORyLQBBmPFgFysP27r+YJrE3b3MPEAT4dHJ0XEylt/7g2Vvd3SxpkhKIHayszJ+rBoeylkSdsVZBQAANvfCqQi1qPe7ayd0hdtnr+SzoAo7G1LkMvDTpcO92fiTZ5NpqPZol6XzQjbNWPhfCrfl7M2rhjq9mGxhTY7xn6soC3uFxb3lRVXvyfUQTRtYFlQq1YULlxYvWTJ12vTdP+9+8KBqLxysMlYfnGZgrASf9BL2sqdEFJAEIRFxuztTHABg9Rce6w0kd1pfm0nuHFsKSIIQW5GCyku7kmqgKe47fWzGuFASCkiScLLjegkAAPJVDENQvX0EjblAUaSnM9el6o1dUw3KaTXv34R55WKnC6P/WLnoqzAZCwD5lw5dnBcwvP/qg/1XP9uGTiz9n9TgS7Hz27Yb9+2lcQAAoL+z4Y1pv6QZ36s+Ks79GfT68PAbly5fiYqK1OstXa1HH//j/M+dd301pWXA3I0Bc8v+ynCfPHY87pEltbrn16/29vtx9qCFewYtfHaoqE2D3v4jhdHc3P3t8QHfjmk/fdt/ZWfhPk1000n712/ts2dx1wmbD0/YXHrS+owza785Z/Zywob7v375c59dc9uM+WLfmC9KylCemN9nfpCJM0qPiy9k/fym/3jO+bPWC86aebzaxjAMQRAajeZq8NXLV67cv3+/xhZHBwAAtkh0Lpbs01HVjuH/FCR89oWNSvTXFd6YMeq2LHnphrD4+wa2SHQoghMQIF/9tbxMHRIv96ohbv7rfLxbxhjfgm1byr6Zo+QWRUiUsz7M6ln29sNwzlwXKknN4ACNnQ2M7KHZcFf4MiNGCCGEEEKoGgypJ1a97+L816LxW7YkT3hvf4yxYVcF+7NGh8yEQ7dZ61b1LLuGH1Nw5vxNcx67YeURW7460v+n8WUeYjIYD08Zsn9/3MCPWw/78u9hXz7bSweVMx7/c0O5rgvOVrrSsuVhV5JVSGUA2PwLRy4uCRjV2U8VvPpiHgsAyvCzF6UjJnQk1OF/HC95oK36o10m99KGuVubHlo07MsvwqPn/pf+XFubHOMnl6+oTyzpFSqHARb3lheY6Cr1Gc3QFElpNJprYdeuXL5y587dmh2nm5R4X364ke2kxqKNjZ8tQazPlU87r8pg4VG87Bc3uw9cBB/1F3xUZq/KLrOkWPlf7rZTHawWDbJaVPIZe/Fq7tpUNiNDm9SO29JbctBbAgDA6HeeKPi7im8nrIGZg3Tug+iHWflyjZ5mGb1alpcWE3bm980LRw2Zsi4ooySrxBacWTl70a+XYzJlWpo2aJXSnPTEuzfCHxQVX2J00h/zF/92OSlPRdMGVX5y1INcgjC5V31jMBgib0d+++13b01+e+M330RE3LA8MwgAAHTmhdWTxs34cv/ZyEc5Mg1N02rZk4d3Q/755a9rUsayWmXlNzdOffuTXSfDk3JkGprWK/NS7gXfSivugkxu0JIpH27672ZyvsZg0OQnRxy7EKtigWFLryJ17E+z3567/dStlAK1XqfMSbz619dT31p6PLMKU71Yxe3vZkydv+v0rcf5Sh2tVxWkxd5+KOMSps5IFbx14c4LMdnyjPSs6lRsDdJptddCr61f/+XkyW9v2749Ojq65u84LOfyDSstC/oHtocflv0KgLhzwTbWAKzOOiiy9G19LOfMDs9FR8QxuZSWBoOOlBbwEhOF4WnUy7xqGKnNkmVNNl20Ti4kDTSZny48dkOgYqG4agiCG3nLKqWINDBAa6nURPHuHzwXX+WwjCDomkCqEJwKr/QrE4QQQgghhOolTezeJWuC8sXdPtnyYWu+0WFXBYwOhQgiK/LKvZQCtYFhaLU09d6F3Z/PWnwy17y/8NmikB82nM4pO0oxPioEbcy29+d89e/NRwUamqENGnleRlLk1TPBD9SVHrEqQzljWcZqhF1xVqF4R1nI36eyaFYRcuJyyWsaVTeOBeXQjPzKoTOZT49RA6NdTdxmu7RGAAAgAElEQVTelZvClLZ9F64a6fJiwsXEGP+5irKoV0C1esvzTHSV+kqv198Ij/jqqw2TJ03e8t33kZFRLzkzCACsXvfj+YL10ZqoQkZBA82wBXL9jRy6JElkMPx1qWBxpPqmlFbQwDCsUk0nPdGeyTBUuDwKq9f9cqFgXbTmnoxR0aA3MFkFuhQdEABMoWr9NeX1QkbDgsHApOYaLFhUiJA4OD394dSpkwAQEXFz245dlpx6ffXnvt8AIDQk9OuNG2vvKDwuly8QyOWm07OvVD0TThN3h6zvcmPVgJmHzZ4aWNf8/bvOnzcXAL7euLFW32UgFAoNNK3VaIxvNqa/9vf1MgD4aJPH6WtVXJDkdeE0ODVkrvLGzuYzg3DVEYQQQtWyc0n6G71kAGDb28nkxgghZL7eAb2L30G2bceuiIibdR0OQq+VlzZOt7KyIghCpTIxP/Xp9X4g2ylabl178dQ3U1xz24pVADB8+IinH75Cz4nXdzq9Xle9eYL1AenceVh7SLr/KDOvSMOx8+o86rMPu/GYh1HR9XS2Zt1SKnH9jIqR9qphvpD0kJdZSGko2qtV0WcTlTyWH/XgpU5gRAghhBBCCCHUoKjVatMbofIwOYjK4XeYtHHbG6Kyj66ydObJHw8m4gIRqAr4LQo2fi4r35GIzKuOB1MsXu8eIYQQQgghhBBCNQ+Tg6gsgleYcDnCq0PzJq4SPmiLspKjQ47/sePAjZz6/apRVN/w5ILLMboOTXSuIgb0VFa6VcgVhx2nhdiREEIIIYQQQgihegWTg6gstihiz/zpe+o6DPTKK4pxnL/Ssa6jQAghhBBCCCGEkAmYHEQIIYQQQshyo8eMbuXXsq6jQOjVduTokfj4hLqOAiGEGihMDiKEEEIIIWS5Vn4tewf0rusoEHq1hVwLhdpMDk6fPv3BgwdxcXFSqbT2joIQQq8oTA4ihBBCCCGEEHqd9QsMfOutiQCQm5cbfS/6/v3Y2LjYtNQ0lmXrOjSEEKp7mBxECCGEEEKoBkyd/k5dh4DQK8bfv+v8eXNfwoFy8/KcXZwBwMnRKTCwb2BgIEmSGo0mNjY2Ojo6NjYuMTFRp9O9hEgQQqgewuQgQgghhBBCCKHX2ZMnT1q29CNJEgBIkir+UCAQdOzUsV379hyKYhjm0aNH+QX5dRomQgjVDUwOIoQQQgghhBB6neXl5dE0XZwcLIsAgkNRAECSpLe3t7e3d/HnIqHwZYeIEEJ1B5ODCCGEEEIIIYReEwRB2NnZOTo62jvYOzs6OTg62jvYt/TzezEzWBZtMBAkGRcX17p1awBQKJUvK16EEKp7mBxECCGEEEIIIfQqIUnSzs7O2dnJwcHB3sHB2cnJ3sHBydHR0dHJ3t6OwykZ5xYWFRXk5+fl5WZmZrq5uVVYFG0wAAEXLlw8+Pfffn4tipODCCHUoGByECGEEEIIIYRQvcPhcGxsbOzt7e3tHezt7dzcXIv/x9XV1cnJiaJKXh2oUCgKCgoKCgpSU1Ojou4UFBRkZ2cXSAtyc3LVanXxNj4+Pp07d36ufJphaIPhzNmz/xz+p6CgAADAr8VLPD+EEKovMDmIEEIIIYRQBSQ2kvfnzD5+7ERCYkJdx4LQa65x48b9Avs5ONo7ODo6OzrZO9g7Ojra2dkRBAEADMNIpdLc3Lz8/PzUlNRbtyIL8vNy8/PycnOl0kK9Xm+y/Py8vLI/MgyjVquPHj12/PhxhUJRW2eFEEKvCEwOIoQQQgghVAGJrSQwMDAwMPBB0oP//vvvWliYwWCo66AQej1NnTIFABQKRXZ2dkFBQXJy8rVrYQXSgoL8guzs7NzcXJqmq1N+YVERTRtIkmJZViaTHTp8+OzZc1qNpobCRwihVxsmBxFCtY6wka9cnjM4y2ngDzbaug6mIaAclHOm507oqPEQs5oC0dcrGh/MrlaB2IKvPffAzJ8mqaN+9loTRdR1LPUdXg5V4jkgY/sEbfiOZhtiarhrGUAs1fXq0oWQy+RyuUwml9fG3B+RWFT8P14+Xks+X1JUVHTs2LEzZ8/KimQ1fiyEGrjtO3acP3eeYZhaKp9l2YICKQD791+HLl66aM5kQ4QQajgqSA76+PjMnzf35YfS0GA91x47O7u6DgGVQ/D0rZprnaSAWYdawHZ6+/GeEXTQ9qZLr3NYAOCqF6xKndeMLa5tkYTVKYBqUrDvi5ymca6TN9mmVv2vbmzBeuyFDmARobOmpYsh7vVp4Jqplgo1jMuhpiqQFbtoWrnQd2qhsrSMW5J65bp13KefMAyjkCtkcplcrpDLZQq5Qi6Xy+RyefHPcplMVvyB4uk7yEyyEYuL/4ckSACQSCRTpkyZMuXtkKuh//z776NHj2r8vBBqsBQKRe1lBott2rw5Pi7ezKMMGzK4u3/XWo0HoYam3o7Te0tk7YQNaIHyJlYVfMFdQXLQ3t7OH++DtQ/rGVWfsG/a7YWKpANeYw7xq/WghbnY1qPTvhvOnNjgufNxLY2Lq3+IlxBk/UIASxBAlp4rv610sierS7X7bLNzUAbJldCgBHBgSQCSrNNAKyFol7P//aJm9rTYiqFoUi7npD62ioiy+feSON7caUA13OjCvmm3P1U+ONJ46j5hYbm8CNtnXuJvA6hfPvfemFhfetdzHaCO1Lvrri6rhaT9ekhnDpD19NG5iFmDnPMoxTrsuu2+C8J0nfE961E11o9+ZYyQTOwp6e3cv5FIJBKJRSKRSCQUi8RCkUgkEoocHOzt7e1d3VxFIpFIJLKxsXm6dGkxnV5fkJ9fUFCgUCgUCqVcIVfIFQqlQqFQKORKhVKuUCgUcoXYxoZhGLLM3bN4AYSAgIDAfoEJCQn/HTlyPez6yz55hJBFYu/Hmr9x8+Y+tRcJQqhe8awoWdbQ4GPFCKEqsHXTNHeieLU5XKz+IV5CkPUJcfugV8eDz352aayTEETIUedTqRQLoC3gAACkOEye4VBXIRpH2WnbNtbzi38gGVt7na29rl2noncmWO3b4bHhBtec93vVfKMTTOsx6dvymr53im8in1PHnu8AdaWeXXd1Vi2EUP3eovQlnfSc0qrgSfSt2xW1asEkhJpMDtafaqwv/coknU5XvESpyS2FIpFELBaLbURikY3YRiwWiW3EIrFYLBbZ2IgbebiLxTY2YrFIJCq7F00b6PLJwWIUhwKA5s2bL1u6ND8vXy6X1+BJIYQQQgi9ZOWSg8OHj6irOBoUrGeEXnskj3YQswYlJdW87FE+n88QLJlXSNbso5TVYUZtELH/azb+fwINywpt9D6+yuFv5E3roJ65OJVY33T9PapOzoVm6d7vpC9Lb7r+bt0EgF49pH7cgrSlnQ1sofX+Q45/R1g9lJI8G13LVoqhTbnXcDHMOqVUKJQKBWRlGd+MJEmxSCy2EYvFIrHYZuiwoZ06djSyMQA4ONo7OJZ8+yKR2BTh6wgRepWFhoQOD8HBGkINAl7vZeHMQYReHw4d8laMUrTy0Lnb0tYc0Mh5d8LtvztgF1U6TrFvl7d0pKJNE11je9qKIKS5ghvXbX8+YhtdMuOB7fVB0r4h1I6FXt8/KkniSPqnRSxQXN/u+86F0vwIqZ2/NW4+AAAwBXbTZruFvTCzi5SoJ0/MndJd5WXLqnKtrkdTLmVmXRC28mWLcoc11bkIGVbLeRhrs2e/85HHZZJZFR3C9F7lIniuBFfdzAd/j2BOb2o+P6w0FFKz8IdHH9lK3n/PPaqF8ZoBACCstMPfzJ3VR+lnz6rz+aGXnDb9K0qv4NxVs9/PntNdY8cBliXk2eL1qzz+zTVZ+WYcgq8bODr3vb7KNi40oaUyHot37XA7mgXNJyafnqz/7wvfzyNLU28kPXF13MTi/zcI18zx3G8o/H1PZreoRv5fSWRmHMt4C5ZlvF0qq40XMQZCRwMLhKKQdyeCd+emzcUZKXvHaKa+W3BooVMcY2m3Ma/VKjyzu6ecsnvmTPs0695ijyMVxWzOJVPuquSCLMfqyhmnnxP1Y94oHNxW4yEGZY7g/HGXjWesnj6/bCRmSQvpglGyrt7apg60FUHkZUnWrXR9MOTR8x2gkq5i6oxZ335Zv01XdvEwUGpu7F3JnoMOZzNJM8+0siYoq6L43c4UWdjzjddVmesCusx6+PdIw7lvfT8KLe3HhOGtNUlft7Fe+4Hnvrwauxys2+d91sUAheLlSz0OZZVUlLaAHxHKjwgtPXJt9GS+rt/wvFn9FO1caSsgiqS85BRB0DGXPTGlTcPXDR6TOztQ0cqJYWT82xG2u/5nFyEtbcrq9auq3ZzrPYZhimRFRbKi4h/9/bsSRMXtzQJLG2gOh6NWa1UqlYODPQBgZvBFlEvPOZ99OKF3Kw8Jqcl/fGrD7GVn8l7R7lE9lOe4L7d/0CJ85cQNEfVnzWvSfdT6nz7uELVu7JrQChfoqJ9hI4QQqmGYHETo9WHvKxvZWfP0qhbaansNzergyYxb4ZBIAwA4+MnG+j/dgHV0Vw1/UzWwp+qTFe5n82ssDEKoXPll6swmJQti8N1Ub7gBADx7kYOB9m6p8Sh+iby1oWWXgs0+Bv0CjxOFRsu1bK/SoGIihQXDC7u2U/PChMUP9pEOKn93VhtlHakDR5M1I1DPW5vyiR9TPEYUuKpHTk7r4OwxeodYWnaIQ+onLEhb0pkmaDI/nySsaVt70CoAzKl844fgaWavSlnali4ZpHINPi10IotfjmH0WKZbsCwj7VJ5bZjGUuF/uRzumTLdUza8mWPcQ8KSDmBmq1WCzpEs+45utjZ//aK8hDWOsRbV9nNXpZ2rauw7KWPLbMBzU701O9Ve7f3BZQ5jKmbn9tJpvZ/1Iid7Vqt6YTKmxV2FYDr0Ka1Nrq5zQG7H9qp1y5vsS6ux2a8VxV+Nnm9u+xLRt4V5I6Rd26v4oaU1YaXq7cvSj0Qh0hq8HJgefWXOJBF1xOXfrMorrcZ7Ml8ze1XK0jZ06fsBWQcXjYOLlhfnuDeGogGAr5mzOmVJm9I6dND0HZbds7N60fJGJ3Ira5eq9Ktq3ZzrO4lEQnHKJQdpmi6eMJjyOOXW7VtRUXdiYmIWf/ZZ74DeNXFAfqf5+/ZMFwctn7b0fH5NZNBqvMAq4rVe8POOeS35JWtkObnytYpXLTNYY3UobtyqVWPxHaK2HymoUsCEsFGLlh62RhakellhI4QQqkuYHETo9cJSZ7Z7LQ3hKmjGrUXhmkXZg3wLprS0XxNDPN3g9FbvxSEcDcG4ectnzMye1arwy5miG1tszEmXAAAw/G1lZg+9GEGbsdnTG7OyBPs1vzgGPaI49pp+b+SsHK0UP91CJdq80ntFGi9XDVyRvse4jB1j5G92MZy8ULomZkWHML2X8SDvi4NlhWM7ydtzhDcNAAAiP1UbirgfbV3EgqOJmmF9R2bPa8Hm3HZe/ptdWCYh8ZItXpD1Zr/cKcfEO1KfHYSwVg5uQzMPHMevdr6rBCBYJ3eDXvOsdSw+RLM3she2oTUptht2O55O5Kq5dJPGBmllY2+GOlR2vg8AYVuuLo0ey3QLmtkuYLw2TNJaX4mmpg7QtWzCwEOq6t2G9R1rVqsZoYh1/uRPzeF3cr+fajV+r1Bu2aCw5KrkKFmmRd+sn+fK3FXCXTtcDtzj57MG/3EZu8ar+g6QOQfbZzNm9DSWCvqp6bIrvEKGcXFgigzgXv5oVesq5eIkHl1z/uKQJDyD4jupxk7NWtpLuXhG0dmvbHNq5uZQYfys7xjLer65VyUAaONFYQrpqHbKtpToFg0AYNVS0d2KeBglTKVZ33E1dDlQ+laeDMnwr0bxjKwQVeM92Xt41qI2tC5d8vVPTscSeApgXPtnnP/wWRreZ3jWp61pzSO7tbscTyVzeE7Kie9mLe5atHamOPTb0jt/NfpV1W7OrxqJREIAUby2KUmST3Ke3LgRERV55969uxqN+bezEoJuH/++Yrini72d2IoLBo2i8ElqYmTIqT/2nYqWlvQagiAIgqzBpWBqvMAq4XebOLkFT5d06LNPtwUlK3hO7iLFq/fS97qtQwu8cgEjhBCqc5gcRKj+8puUfHKyhir+geHv/Mzr24em/tBjQS7lyHQAQGbE2m84VdRvhsavmYGM4TKlGyhklIoGADIjQfL1V6TjjrQx3YoCRTZHauR16qR2SHcdqbPe+p3rsScAAJBjdeKkeNJIZdmXNtk2L1jynrKVu96eQ2ZJCQrAxclAAsf4msuW7VVCKzx9izOun2KQD3szngBg27ZRWTH8q1E80zWj1I7so+EoxV9tcbysAADISbJd95diyGJ5zzb6XamldQsALMECEHZa/2b6hPtcDUvkZnCfxWDxITKYEX3VfIPVN9+4HcggAAB0nMR4S2/gpNFjpdPmtGBZlbULY7w2zFAgo1iCsbZmSKCYqnYA46dZttWMIRJPua9v82jTiMwV97yW3bRoveeSq5IAoGIvOx8YIl/iQcXeEWRrAIB77R+HoKGqMa66JgRkm4q5uDRpDi9fQwBQmVnU88eitJZ3FZa8edH+cgoBAOos4W/b3Jo2T5nWTh4gtv23Bh+UfC5+UmNhzzenrp7SCM9EUqP7yAd4O99KJADYTl2VdsA/cJ1P1+TlwIitWGCoAlP30prsyRnMGwEaHiP44Vv3fSULHJO5ZV85SmpG9dPwDFabvnM9nEYAgCpLtHuLm+f21MldCwfY2PxTVFG7PMdUv6rWzbl+E1oLlUrF7cioyMjIqKg7ebkVv2LATJSzb4cWjUvWXwKetcS5WVvnZm17jR4TsPDtz09kMQDa2z9M7PhDDUReqsYLrBLSxcdbQmhD9v5wKqmQBdBmp7yCS7fUbR1a4JULGCGEUN3D5CBCr7PMNJ6S1YismEqn+SmEF+PIMd113s4s1Mgf7FxdU2eWybG+VdkAiqD7vP94z1AdtyQmuokrABAmvt+2bK9yyOvBNln9Cob0Um+Ot9aTmp5taDbT9kp6xVuXq5l0nZcLS/Jl2w/Ebi+/mbuLgYRnaSZWJTwSwekXIF/+pXyRjBeTIAwOtt97ja+saP5MFQ5B6Zu7s0y2MMzIs4rm4xo9FldvogXLMtouVaqNCtnb0AQLKjXJWtABjJ8mmJkcBKC5/+1y6/lt2vgPs4OT3JXmxl5paSm5BHgZXMQAxbOO9Nz0AoKwZ6wIAI6JmE2XT2lrrKtorW4kkdN66po6sVB7b1Ez3kxGer6p9i3/GXntqji/T9HAbppvE61ormpQFz37yO5UKlGTlwMQSg0BJG0rAqhs7dwa78klVSS6klpJEVydjwvLZAuvZZTZQC0MiScn99L5uLBQZEZXMdKvauDmXK99vXFjZmZm8czBGmK4u3X8pJ8faAm+2M7Js23f6Z/Mf9NvyLyJe0//EPcapFPLI/gCHsGq8/LMv/Gj1wHJFzvYCgxyqVSFrylECKFXg0WTIBBCL0X8314+o1s1K/5vrLfpaYMvYPWkjgXC6HvhWQYAoOTBMgBgWQHPonBLFd9WKl2VViKf2V9HyYQ7Nnj1mNzSZ6xf94222abGQ5bt9RzNfdujmUSjHrIuPKAaKQPc2PRb4rjKR3zP10xF+PzyiVeWc2p703d/cfxfhHUqq+/UVbpwUcqm3oZKk7NmHoKAmh1pGz8d4y1Ylol2qWJtvBCNKrAtTbK8+FQSLOoA5raaKWyh+Itddul2RWtny5zL72nBJaPXAxAs9+l3cwShpwGIkmqvbsw12lWKl2EoLq9Gbg4VsrjnV6muVDGSs/lss56ydhTwW8oG2xN3r9ok06bLMf9yAJqXlEGwpLZ7a31lf11ZdiszXkUcAoCG2k0qVd4QNXJzrs/S09NrNDMIACyt1egYlqU1sry06Mt/rvk5TMOC2EZEAABQzT88nBQX+k1AcYKbcOj1/pbdB85evHrv7p0HsXdirh3/c93kjnZP28PkBlUtEAAABB79Zn/x58nL9+7dS7oXcevikUM/fjHb37biXmDVdPBH3xw+F3I/OjL66pHf107xdyo7+ZQA0m7iL3ceJdx/lHD/Uczv093KXh8m4+H2WnvlYeyRT/2elklIxu5KSIj6fbw9UUEJtyMv/Lnl3e4tOr/5+ZZ9F69FJNy/HXn+j41T2paNnhD6jPh0y5GLYXHRtyMvHNz2Ud+Sl2YCIenw1urvfz1xPjj63t0H0eHhJ9cNs3+uDovPusnADzf8fSY4JjrqfsSl8/vXjvGkAIBwCFz+x5GQ8JuJsfcSbl86/fPn41oITd1AOF2WnXsQH75zWJnXFRAOb/16Ozn6t+lupNEyzQrYdFQEz3fMyt+OXYqOvhMbfvafLR8NbWplJOLKKxBI+y5ztv536/b1iKtXbkfeunt+85vuON5ECKFXAM4cRKhh46v9fRjQ8x7nEACsXEGxpL5FY5pIqPhdUQaaYFnWWlB5gXrewycE6a4IbOwUnVLB38Okrd6VC6rr9ttvCHQAAES+lCr7/qEKD2FyL7OCpAWHz1m9945sbDvnNDdFC4L3W5ig0m+0y9aMnvc4h2BEkvc+cL9s8h1TWl7wSefgkwAU7Tcga+8HssCeKusQm2odgtQ8ziFIV2UPVzY6s9q5H1PHMt6C5bY12S6V1Ibp+XcE3X3SkwnOYEixOZVMkE2q3m2q1GqmFEa5rDij/H3ok08K2LKVYvKSqZrqx6zn1VRXIUTKAS0Z0POSa+rmUHnAlvT8qtaV1vrwFf7b42SjWzra9pU764Rbg3m0GQGYfzkAkNcjRLKesu5v5gwOb3RWWtEWFtwAjUdIqTILgXRV+TtDTHZFQRXfk92UvdzZ6PTSU7BSBvg9bVwzVN6vzLw5Uy88qYyA5AiEth6tAt95rwefyQsJia8op0ratxs0sm+rZ0uNOXr3mrSig6/VuGl7Ew3mbFDVAgEEfrN371nqb1eaERY6ePg6eHjxIvfujSh8PkhBqzm79yzxl5Skf1x8+05e1rNP+0VTl53INCdJXNX4TZbAtWvcceznv5Zb98mzy1srf7RXvvnB0ScMAFi3nffrL590FJes8NO4/ciPt3Vwnz96ZbCUJZ17jJ/2xtPSxE7O3ArW0eL7zf7516XdbEvOmufi06GxSM0AABhsvTv5ehR/jyJyaRk4fXMbZ/3oz04YW5/ZEH31et70N7v2bMs/E1Zy+Qg7927Hp+NCQ3IYEBkp07yATUZFCDuMGF9aX407D5/bsWeHdVPn7ntQ0frFRiqQcJ2wcfuSvjaEQZn/REGIHGydudqimk2vI4QQqhX4TQ5CDQzB+A8s6NfEYEWxNq7KGfOzJruAMlocogAAIjlJIGOZ3hOy3/bTW1NA8Wknm7JzcIicPA5L6QYNkjezZik+7d1K7f7ckI/hn7gq0FOaj5ZlzO6ktecDSbISO9q6tBSmkJujB6u2hVNa6UUUAMmKrBmOqUOY2qvcGRoJMjXE7orGMGSwdIy/msqwOfWg7MlVXjMM/9x1HmNbtPbTvAFeehsukCRr56IObK17PgZK13+gvJ0zzSOB4oBBRWoJIIjSdJLFh2D4Z8N4NEe9YGnW1LY6Oz5LcRjXpqoWtmAJU8cy3oLlSjLeLsZrozySYikCgGKFtrr2XaUr1iT/NlYjMPAO7rWPYyzqNoTZrWYOlgw74HbwCe3uVO6KMHXJVJH5Pa3yEizvKgSIbQ1CDpAU4+ZbtGx55mg7KIiQXK6pm4MFp2zkdKpcV0TsRds7jH7E6OwZPQxFEXZnC80KwPzLAQCk1xz3PCRJp6Kt36Qu7q/0tmU4JPCEBt+2RR9MLWxJ1kJPpq2CIngMT7VgUfZIb72Iy9q5Kcf10Dyb4skIjl8R6Djqjz/LHu9rsKZYiZvy/U+zJjpC4S3bC0VmtJHRhjB5RjoDwRJ0py5KDwEAQfeamRy5P3l5+9dobmGVcTt9fvZhwv1HcXfjbgUH7Vs32Svn2Ko5665UvuIRKzuzbHD7du28W3frPWVjUDYjbD9lSiduFTaoQoGU99TVi/xtdckn10wb2qFtO5+23XqvDlZVHBzlM23Vp11tNHH/LJnYv3Wbjh0Hz/76UhbhPmztkkHPJv8x0kPvdWjWonWzFq2btZm5L+uFVFFV46/8jHza9h6+4nQ6zTKFN3fMG9+jcwffToOm7rotA9u+4/o7kwBA+U5fNa+DdU7wtnff6OXXunO38Sv/SaY9xsyb4lN6z2LlQWtGdOnYwaddj4AJP9x4Pj9GNZuyaqG/RJN4dOW0YZ3adWjZtf/Q6d+cy2MBgJVf2zx9TI+unX1atmvZfcS7e+5pHPq9GWhn/N8FbdTVsCKw7xHQtvTiseoU0F3EPAwNTaXNKNNEwOaUoHt0+pt3R/Zt1aZTx8Gz1p9JMdj2WPzZCOcK4jZWgYS42+BuYibm57E9enTp079zZ//uY78NVZluQIQQQnUOZw4i1MAQbNNeT/b2evL0A0Yu3PiHpHg1UmWU/f5H8o+9ZV9+I/uyzD5P/y/1tjj2bXW7AemXBgAAgMFqw7xmv2SVO0Dicbdv26csbSNbvka2vMwvir8MZ4tEhyI4AQHy1V/LV5fdy/ghsk3sVZaRINlC8f6r3IFDcj5iIf5/ktiyIxRjNUPEHHXd2zVtdvecPd1znm6gj3cZtMwhpUwhhEQ568OsnmXvrAznzHWhsrqHIO4fdfu5c+pcn8Ivviz8ovh3LHnimxbzr1uQiTJxLOMtWJbx1jRRG+VLajUpOWFSuY9oudUfOzy+ukuxAGBRt9lj7DTZ1lOSj03Uhe1sPuO8WZP+WKVwyx5J/xWFHmWq3OQlU0Xm9jQjJVTeVcDEKRP0sAVJwxY8+0CbZbvqt5LVbGvi5mDBKRvr+VWtKzpb8uftvO+6F/WleXvOiGSsWQGYfzkAABgEP37TyHlFxpRmirkLFHPLHd6KvGIbl1HjPZm4+a/z8W4ZY3wLtm0p+6bDZ02TdNJta2/o5uIAAA46SURBVOeUxa2lmzdLNz/dPUey9jebAnPnu1beEOEmzij9kaCQ1fiNTD1n79H1O2pwgMbOBkb20Gy4KzTz2K89wqrZkDkfxSSt+u1eJflBlpbn5si0NIAi49bBDfuH9Vvcys/PiYzIZMzcwPwCKa83hrfm0fE/fLpyX0JxkkmRm6+oJDfYfNTo1jx99KZF6w8/pAFAlRK2+7M1nid/mhw4aoDduX8qe/lmVU+wCiVIY4/sOvDW4CVeebHX4rJVAJB5bfevQZM7jmnctAkJ2USLkSP8OLILXy3efbmIBYCc6CPrtvUesnVAT3/HXUl5AACsQZqRnq/SA+gzU2QA5b/ooLxGjGzD19/7Zv7qA49oAADtk8So0n/ZCcK27aQlK7q38nSz5yqz8hgKOC5ujiQUGMuIq26cuVo4ekTAgFbf3rpHA/A79e9lxz48cP4BbU6ZxgM2qwTlzf/+upyoBwB1Svhvy9Y2bbtnWvdBAbZH/31uEjRlrAJ/PMGyAISTn7+fY8LNJxpWm/uoklc7I4QQqmdw5iBCDQxL3g21DUnlqGhCo+DeDXP8eGnj356+yV5ntW19k68uWD8qImkGDFoyL4cfedsmOI0oHhjQqQ7zv3e4nMpRMWDQcJLjBRW8pl8r+GW91zv7bEMfc2U6gqYJeSEvNkb8722+HgBYzpkdnouOiGNyKS0NBh0pLeAlJgrD0yhjhzC1V1lGgyTDT9vFMSzfYH34Mr/cX+pGa4ZVCjeuaPbJ/yThqRyZlqANZF6WVXAsT1f+0ATBjbxllVJEGhigtVRqonj3D56Lr5YmYqpxCFZl/d2qZvP/Z3Mrk6PUE3oNlZZs/VBZ8Sw8k0ycjvEWLFeQsXYxURtP20vKj07j5itJPQ2MgZQV8mLu2vz+u8eoD5utC+cazDgQVNLoxk+TogBYQqEy+krO8ooinTeEcsoNVk1dMlVlZk8zVkLlXcXIKecn2Jy4bZWUS6n0BE2TBVlWZ4+4T1zifibf3DM16+ZQ9VM2cjpVriuWc+6UJJMGbZLdgUTCzACqcDkU10OuePXn3jN+sT8bx89REjRNqJWch4mif/61uyarlZ7MSG2WLGuy6aJ1ciFpoMn8dOGxGwIVC886qlbw07pmc/+S3Mqi1AZCWcC/esZ16hL341VZd7fShjB1Rqrbzgv/J47JozJyuDpGEHRNIFUIToVX9fnz14k+8puh3i1aN2vRxrutv/+QaZ/ujlB6DFz+w6e9jL3k7Sk68+FjJUuIRJW9yc7kBka353g2b0oxaWFXKnye9Dk8Tx8Pkkm7ce1xmX9OlZEhURrgefo0tmyUUdX4XywgKyXTAHwnl9KnfkGXnZ7DElbWVgQAr7GXB0lKhmyPiCl5DWLC/egfhooJ0t3DzayIS6ooIiz1hXQfYdNn5e/7lk/q17apiw2fa2XfpLEjnwCSNDkbQ3nt5OV8osnAgX4UAPDaDerryMafPZVEV6PMakSlvncjWge8Rk1ffF2g0Qok5GFHLuYRLn2X77twJ+zkPz+u+3hYc0sbEiGE0EuFMwcReoUpgxv7BT/7MemQV/ND5TbQ33HzH+tW7iOWSLjo9nlkpX+p0YXWv25v+mulxyRSwlzeDXMxEZmOe/U/96v/VfxLVsM/+nvjo79X7RCm9jI3SDpDeD2XaJYmOfncwNhUzbBK/vGDjY4fNHZgJl/03dei7yotolqHYFX8kwc9Tr7w2+fa/cVuAABsoe2M8eWeLDVxOkZbsFw5RtrFeG2U0txznjDPuVoHAqi021R+ms4OBpLhxzyuYDD43JVVpjjOmS2+3lvKfWb8knmhOYgL3/t5fV92f/7OhS13mhdzhY374oeVdBXWyCkX3HVceNexkpMojbTaN4cK4wdLe77xHSs8lvqea8A416oGYP7lUELLDT3pGnqy4l/WeE8GAEOe8Kdtwp9Kf3QanDrUXyuXk2Xyg7xzfzc693fFu1evXwEYPyOGE/J345DSQ4fu9eq0t5ItGxyW0SlzH0ce3bJU1PbcF/7devtSoXfN2E2n07GEkQWhTW5gbHuSyyEBDAbzHvyulazPc/GzwADwBQLzj8XodQYguFzu0130egMLBEECAMtW8s0Nwbfim7fuE0kSABUVQ9gPnDm2CSW9sWPVpgPhD3PVHMcBK45uHWVOqaqI02ezx7w9ZGi7bfdjOw8d7ELf3X8mmQbCwfIyqxFVcf1X1IeMVyCbd2r5NEXUhGHd23fq2LZTv2adA/v5keM+OpVnfsAIIYTqBCYHEUINgo2dgZZx9AJdr7FPJrpQ538Tm/08HXodUZouvowhWXzO9HOvr4sGeMoNA2mvGuYLSQ95mYWUhqK9WhV9NlHJY/lRDyqYWI3qI4LL4xEAJEkQla9N/bLoszPzGLJJF393MibN1EO9upSH6Qzp2a2XJxWdXJpOFHYK6CgAXWpyOlMTjygx8iIFSzZq4SMh7uTXxLpPGY8zGEZy7L3Bqy5X8C48M16Vqs94nMGQTfx7NKaiH5dLopKOrq48UAXt334hXgcAoM/Plb3wCgKKqnD4pbl1+Pjjt2cPGd1pr+2IAc6a8K0n0mgAyqwyjTEvqnIISc8BnXigS0nOeNqIpWGbqEAATVrw/i3B+wEosd+b6/euHRQ4xB9Ona5KyAghhOoAPlaMEGoASP2bS5Ji/otNOPhg73iV/pbz97dw2NygkXZ6iZp3+oRtcoNZF6EBnnIDwW9RsHHp43N7EqP/iUv6X+K5NU8GOUJWiONB08srozpBUHwrHglAcq3EDp5t+83asP3TjlymMOrmA7NW561dhrigS1kMv+OC7xaPbO0i4vHtPLuOG9SSV+HGdOLx47E6btuPt6wa387FmsOTePZ8f/O6iW5EYfCJCzXzFRydHB0nY/m9P1j2dkcXa4qkBGInOyvLOzedcC4omXEcuXbTrAGtXG14FEkJ7DxaB3b1NHfGBJ1w9vxDmtt+wfb1U7s1tRNQFFfk2qJDCweSyc/J0YNVt3FTOruLOASQXJFIULZYnd7AEradArt7WL+YhTTE/vffHdptxDtLZwy2L7r079k8FgBMlmmSWSUQlNjRQcglSY7Ird3wZTvXjnaCgksnit8qWC5s4xVINev/Zv92jWx4JEFxOQa5XAtA4K0IIYReBThzECHUENB8llLRNMh4N4Idvz4oScP8SMPG5NksW2hT11G8VA3wlBsInlxwOUbXoYnOVcSAnspKtwq54rDjtDDH3KUc0EvGaf/JkbhPyn3E6tJPfr3zkqKOIipHc3P3t8cHfDum/fRt/00v83mFiUs6af/6rX32LO46YfPhCaXL3bD6jDNrvzlXU9PzlSH798cN/Lj1sC//HlZmNSTzX8T6HEPMr1/t7ffj7EEL9wxa+PRTfdSmQW//Yd66T4b7v375c59dc9uM+WLfmNL1eZQn5veZH3Tp0MV5AcP7rz7Yv8z6PHTp+jx0elx8IevnN/3Hc86fdV1w9rmJd3Tq8T+D53w3aERfOmXPwZDiRZPYfONlmmZWCYTNsI0Xh218tpM25diqb4KkbAVhG6nAVIdus9at6ll2oWmm4Mz5m2YHixBCqM5gchChBqSyF369/hjBT8t9f6r89y+hZhpu5SOEalNRjOP8lSbeF4nqCTr3QfTDls2c7Wys+RyC0amKcjOTY26FHP/70LkEWT1J5zK5QUumfJg4//0Jfds2kUBR6r3QZNGgAb4MW1GA6tifZr/9aNbc90b1aO0uYqSPb1/8Z+fOvyNya+4rOG3MtvfnyBbOm9KvbRNbLqtTFeZnpz6MDX6gtnDdJ/nNjVPfvv/urMmD/Fs1dhBSGmnmwzu30qqw7pPi9nczpsbPen/GG91autvyDEXZj2IeyrgEW3Bm5exF2QtmDevc3EVIGTTyImluVmr4g6KS9XmCty7cKVoywZ+fnlXB4diCc3+eWtR/ktO9Qwfuap9+aLxMM8I1XgKTf/f8iav6Nj5NGjna8EldUWZSRNDhH385ek9a0uLPhW2kAgkiK/LKvUadmzey5RPaooyk22f379z2/DueEUII1UeExMGprmNA6DU3pr/29/UyAPhok8fpazhzByGEEKqWnUvS3+glAwDb3vXi79hlS5f2DugNAFOnv1PXsdQ4wmni7pD1XW6sGjDzML6tF9U8f/+u8+fNBYCvN24MDQmt63AQQqiBwpmDCCGEEEIIIQAA0rnzsPaQdP9RZl6RhmPn1XnUZx924zEPo6LNnqqGEEIIoVcNJgcRQgghhBBCAAD8DpM2bntDVHYRCZbOPPnjwUR8WS9CCCH02sLkIEIIIYQQQggACF5hwuUIrw7Nm7hK+KAtykqODjn+x44DN3CJG4QQQug1hslBhBBCCCGEEACwRRF75k/fU9dhIIQQQuilIus6AIQQQgghhBBCCCGEUN3A5CBCCCGEEEIIIYQQQg0UJgcRQgghhBBCCCGEEGqgMDmIEEIIIYQQQgghhFADhclBhBBCCCGEEEIIIYQaKFytGKGXZ+eS9LoOASGEEEIIIYQQQugZnDmIEEIIIYQQQgghhFADhTMHEap1GU/Io5f4dR0FQgghhBBCCCGE0PMwOYhQrbt5nztzNbeuo0AIIYQQQgghhBB6Hj5WjBBCCCGEEEIIIYRQA4XJQYQQQgghhBBCCCGEGihMDiKEEEIIIYQQQggh1EDhOwcRQgghhBCqAfPnza3rEBB6xdjZ2dV1CAghhDA5iBBCCCGEUE3w9+9a1yEghBBCCFUZPlaMEEIIIYQQQgghhFADRUgcnOo6BoQQQgghhBBCCCGEUB3AmYMIIYQQQgghhBBCCDVQmBxECCGEEEIIIYQQQqiBwuQgQgghhBBCCCGEEEIN1P8BOTgdiF6cXDcAAAAASUVORK5CYII= ) ``` keras.validate_task_parameters() ``` ``` Keras Neural Network Classifier (KERASC) All parameters valid! ``` ``` ``` ## Update to the original valid blueprint ``` pni = w.Tasks.PNI2(w.TaskInputs.NUM) rdt = w.Tasks.RDT5(pni) binning = w.Tasks.BINNING(pni) keras = w.Tasks.KERASC(rdt, binning) keras.set_task_parameters_by_name(learning_rate=0.123) keras_blueprint = w.BlueprintGraph(keras) blueprint_graph = keras_blueprint.save('A blueprint I made with the Python API', user_blueprint_id=user_blueprint_id) ``` ## Get help with tasks ``` help(w.Tasks.PNI2) ``` ``` Help on PNI2 in module datarobot_bp_workshop.factories object: class PNI2(datarobot_bp_workshop.friendly_repr.FriendlyRepr) | Missing Values Imputed (quick median) | | Impute missing values on numeric variables with their median and create indicator variables to mark imputed records | | Parameters | ---------- | output_method: string, one of (TaskOutputMethod.TRANSFORM). | task_parameters: dict, which may contain: | | scale_small (s): select, (Default=0) | Possible Values: [False, True] | | threshold (t): int, (Default=10) | Possible Values: [1, 99999] | | Method resolution order: | PNI2 | datarobot_bp_workshop.friendly_repr.FriendlyRepr | builtins.object | | Methods defined here: | | __call__(zelf, *inputs, output_method=None, task_parameters=None, output_method_parameters=None, x_transformations=None, y_transformations=None, freeze=False, version=None) | | __friendly_repr__(zelf) | | documentation(zelf, auto_open=False) | | ---------------------------------------------------------------------- | Data and other attributes defined here: | | description = 'Impute missing values on numeric variables with ...eate... | | label = 'Missing Values Imputed (quick median)' | | task_code = 'PNI2' | | task_parameters = scale_small (s): select, (Default=0) | | threshold (t):... | | ---------------------------------------------------------------------- | Methods inherited from datarobot_bp_workshop.friendly_repr.FriendlyRepr: | | __repr__(self) | Return repr(self). | | ---------------------------------------------------------------------- | Data descriptors inherited from datarobot_bp_workshop.friendly_repr.FriendlyRepr: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) ``` ## List task categories ``` w.list_categories(show_tasks=True) ``` ``` Custom - Awesome Model (CUSTOMR_6019ae978cc598a46199cee1) - "My Custom Task" (CUSTOMR_608e42ac186a7242380a6a98) - "My Custom Task" (CUSTOMR_608e42ecd5eb0dc5f28d0dda) - "My Custom Task" (CUSTOMR_608e43fc01f9f466aa8d0d81) - My Custom Ridge Regressor w/ Imputation (CUSTOMR_608e5a4ed5eb0dc5f28d0ea0) - My Custom Ridge Regressor w/ Imputation (CUSTOMR_608e5bc8b66a4934d58d0d4e) - My Custom Ridge Regressor w/ Imputation (CUSTOMR_608ef72b6f13f54305667783) - My Custom Ridge Regressor w/ Imputation (CUSTOMR_608ef74c5dda651931052422) - Second model (CUSTOMC_6019d18adfa83afbad99cdb8) - My Imputation Task (CUSTOMT_6188b0e6fb465717f029fd05) - Image Featurizer (CUSTOMT_61b452e57fd5b0629a2f4fd3) - Maybe Broken? (CUSTOMT_61b7d3f26f8e01a1a8f7bc0c) Preprocessing Numeric Preprocessing Data Quality - Numeric Data Cleansing (NDC) Dimensionality Reducer - Truncated Singular Value Decomposition (SVD2) - Partial Principal Components Analysis (PPCA) - Truncated Singular Value Decomposition (SVD) Scaling - Impose Uniform Transform (UNIF3) - Log Transformer (LOGT) - Smooth Ridit Transform (RDT5) - Standardize (RST) - Search for best transformation including Smooth Ridit (BTRANSF6) - Transparent Search for best transformation (BTRANSF6T) - Transform on the link function scale (LINK) - Ridit Transform (SRDT3) - Standardize (ST) - Sparse Interaction Machine (SPOLY) - Constant Splines (GS) - One-Hot Encoding (PDM3) - Numeric Data Cleansing (NDC) - Missing Values Imputed (quick median) (PNI2) - Missing Values Imputed (arbitrary or quick median) (PNIA4) - Normalizer (NORM) - Search for ratios (RATIO3) - Binning of numerical variables (BINNING) - Search for differences (DIFF3) Categorical Preprocessing - Categorical Embedding (CATEMB) - Category Count (PCCAT) - One-Hot Encoding (PDM3) - Ordinal encoding of categorical variables (ORDCAT2) - Univariate credibility estimates with L2 (CRED1b1) - Buhlmann credibility estimates for high cardinality features (CRED1) Text Preprocessing - TextBlob Sentiment Featurizer (TEXTBLOB_SENTIMENT) - NLTK Sentiment Featurizer (NLTK_SENTIMENT) - One-Hot Encoding (PDM3) - Pretrained TinyBERT Featurizer (TINYBERTFEA) - SpaCy Named Entity Recognition Detector (SPACY_NAMED_ENTITY_RECOGNITION) - Fasttext Word Vectorization and Mean text embedding (TXTEM1) - Keras encoding of text variables (KERAS_TOKENIZER) - Matrix of word-grams occurrences (PTM3) Image Preprocessing - OpenCV Detect Largest Rectangle (OPENCV_DETECT_LARGEST_RECTANGLE) - OpenCV Image Featurizer (OPENCV_FEATURIZER) - Grayscale Downscaled Image Featurizer (IMG_GRAYSCALE_DOWNSCALED_IMAGE_FEATURIZER) - No Post Processing (IMAGE_POST_PROCESSOR) - Pretrained Multi-Level Global Average Pooling Image Featurizer (IMGFEA) Summarized Categorical Preprocessing - Summarized Categorical to Sparse Matrix (CDICT2SP) - Single Column Converter for Summarized Categorical (SCBAGOFCAT2) Geospatial Preprocessing - Spatial Neighborhood Featurizer (GEO_NEIGHBOR_V1) - Geospatial Location Converter (GEO_IN) Models Regression - eXtreme Gradient Boosted Trees Quantile Regressor with Early Stopping (ESQUANTXGBR) - ExtraTrees Regressor (RFR) - Elastic-Net Regressor (L1 / Least-Squares Loss) (ENETCDWC) - Light Gradient Boosted Trees Regressor with Early Stopping (ESLGBMTR) - eXtreme Gradient Boosted Trees Regressor (PXGBR2) - Ridge Regression (RIDGE) - Nystroem Kernel SVM Regressor (ASVMER) - eXtreme Gradient Boosted Trees Regressor with Early Stopping and Unsupervised Learning Features (UESXGBR2) - eXtreme Gradient Boosted Trees Regressor (XGBR2) - eXtreme Gradient Boosted Trees Regressor (XL_PXGBR2) - Nystroem Kernel SVM Regressor (ASVMSKR) - Partial Least-Squares Regression (PLS) - Gaussian Process Regressor with Rational Quadratic Kernel (GPRRQ) - Eureqa Regressor (EQR) - Auto-Tuned Char N-Gram Text Modeler using token counts (CNGER2) - Frequency-Severity Generalized Additive Model (FSGG2) - Hot Spots (XPRIMR) - Linear Regression (GLMCD) - Frequency-Severity ElasticNet (FSEE) - Gradient Boosted Trees Regressor with Early Stopping (Least-Squares Loss) (ESGBR2) - Light Gradient Boosting on ElasticNet Predictions (RES_ESLGBMTR) - Support Vector Regressor (Radial Kernel) (SVMR2) - Regularized Quantile Regressor with Keras (KERAS_REGULARIZED_QUANTILE_REG) - Auto-tuned K-Nearest Neighbors Regressor (Euclidean Distance) (KNNR) - Lasso Regression (LASSO2) - Gaussian Process Regressor with Radial Basis Function Kernel (GPRRBF) - XRuleFit Regressor (XRULEFITR) - Frequency-Severity Light Gradient Boosted Trees (FSLL) - Ridge Regression (RIDGEWC) - Stochastic Gradient Descent Regression (SGDR) - Eureqa Generalized Additive Model (EQ_ESXGBR) - Elastic-Net Regressor (L1 / Least-Squares Loss) with K-Means Distance Features (KMDENETCD) - eXtreme Gradient Boosting on ElasticNet Predictions (RES_XGBR2) - Auto-Tuned Word N-Gram Text Modeler using token counts (WNGER2) - Auto-Tuned Summarized Categorical Modeler (SCENETR) - Keras Neural Network Regressor (KERASR) - Elastic-Net Regressor (L1 / Least-Squares Loss) (ENETCD) - eXtreme Gradient Boosted Trees Regressor (XL_XGBR2) - Gaussian Process Regressor with Dot Product Kernel (GPRDP) - Dropout Additive Regression Trees Regressor (PLGBMDR) - Elastic-Net Regressor (L1 / Least-Squares Loss) with Binned numeric features (BENETCD2) - eXtreme Gradient Boosted Trees Regressor with Early Stopping (XL_ESXGBR2) - Auto-tuned Stochastic Gradient Descent Regression (SGDRA) - RuleFit Regressor (RULEFITR) - Gaussian Process Regressor with Exponential Sine Squared Kernel (GPRESS) - Adaboost Regressor (ABR) - Elastic-Net Regressor (L1 / Least-Squares Loss) with Unsupervised Learning Features (UENETCD) - Gaussian Process Regressor with Matern Kernel (GPRM) - Light Gradient Boosting on ElasticNet Predictions (RES_PLGBMTR) - Gradient Boosted Trees Quantile Regressor with Early Stopping (QESGBR2) - ExtraTrees Regressor (Shallow) (SHAPRFR) - Statsmodels Quantile Regressor (QUANTILER) - eXtreme Gradient Boosted Trees Regressor with Early Stopping (ESXGBR2) - LightGBM Random Forest Regressor (PLGBMRFR) - Frequency-Cost ElasticNet (FCEE) - Frequency-Severity eXtreme Gradient Boosted Trees (FSXX2) - Gradient Boosted Trees Quantile Regressor (QGBR2) - eXtreme Gradient Boosting on ElasticNet Predictions (RES_ESXGBR2) Binary Classification - Stochastic Gradient Descent Classifier (SGDC) - LightGBM Random Forest Classifier (PLGBMRFC) - Bernoulli Naive Bayes classifier (scikit-learn) (BNBC) - Dropout Additive Regression Trees Classifier (PLGBMDC) - Auto-Tuned Char N-Gram Text Modeler using token counts (CNGEC2) - Gaussian Process Classifier with Matern Kernel (GPCM) - Gradient Boosted Trees Classifier with Early Stopping (ESGBC) - XRuleFit Classifier (XRULEFITC) - Support Vector Classifier (Radial Kernel) (SVMC2) - Multinomial Naive Bayes classifier (scikit-learn) (MNBC) - Adaboost Classifier (ABC) - eXtreme Gradient Boosting on ElasticNet Predictions (RES_XGBC2) - Elastic-Net Classifier (L1 / Binomial Deviance) (LENETCDWC) - Gaussian Process Classifier with Radial Basis Function Kernel (GPCRBF) - Light Gradient Boosted Trees Classifier with Early Stopping (ESLGBMTC) - Eureqa Classifier (EQC) - ExtraTrees Classifier (Gini) (SHAPRFC) - Logistic Regression (LR) - Keras Neural Network Classifier (KERASC) - Nystroem Kernel SVM Classifier (ASVMEC) - eXtreme Gradient Boosted Trees Classifier with Early Stopping and Unsupervised Learning Features (UESXGBC2) - RuleFit Classifier (RULEFITC) - Regularized Logistic Regression (L2) (LR1) - Nystroem Kernel SVM Classifier (ASVMSKC) - Naive Bayes combiner classifier (CNBC) - Light Gradient Boosted Trees Classifier with Early Stopping and Unsupervised Learning Features (UESLGBMTC) - Light Gradient Boosting on ElasticNet Predictions (RES_PLGBMTC) - Elastic-Net Classifier (L1 / Binomial Deviance) with K-Means Distance Features (KMDLENETCD) - ExtraTrees Classifier (Gini) (RFC) - Hot Spots (XPRIMC) - Partial Least-Squares Classification (PLSC) - Auto-tuned K-Nearest Neighbors Classifier (Euclidean Distance) (KNNC) - Eureqa Generalized Additive Model Classifier (EQ_ESXGBC) - eXtreme Gradient Boosted Trees Classifier (XL_XGBC2) - Auto-Tuned Summarized Categorical Modeler (SCLENETC) - Light Gradient Boosting on ElasticNet Predictions (RES_ESLGBMTC) - Elastic-Net Classifier with Naive Bayes Feature Weighting (NB_LENETCD) - eXtreme Gradient Boosted Trees Classifier (XGBC2) - Elastic-Net Classifier (L1 / Binomial Deviance) (LENETCD) - Gaussian Naive Bayes classifier (scikit-learn) (GNBC) - Logistic Regression (LRCD) - eXtreme Gradient Boosted Trees Classifier with Early Stopping (ESXGBC2) - eXtreme Gradient Boosted Trees Classifier (PXGBC2) - Auto-Tuned Word N-Gram Text Modeler using token counts (WNGEC2) Multi-class Classification - Stochastic Gradient Descent Classifier (SGDC) - LightGBM Random Forest Classifier (PLGBMRFC) - Dropout Additive Regression Trees Classifier (PLGBMDC) - Gradient Boosted Trees Classifier with Early Stopping (ESGBC) - Light Gradient Boosted Trees Classifier with Early Stopping (ESLGBMTC) - ExtraTrees Classifier (Gini) (SHAPRFC) - Logistic Regression (LR) - Regularized Logistic Regression (L2) (LR1) - Light Gradient Boosted Trees Classifier with Early Stopping and Unsupervised Learning Features (UESLGBMTC) - Light Gradient Boosting on ElasticNet Predictions (RES_PLGBMTC) - Keras Neural Network Classifier (KERASMULTIC) - ExtraTrees Classifier (Gini) (RFC) - Light Gradient Boosting on ElasticNet Predictions (RES_ESLGBMTC) - eXtreme Gradient Boosted Trees Classifier (XGBC2) - Elastic-Net Classifier (L1 / Binomial Deviance) (LENETCD) - Logistic Regression (LRCD) - eXtreme Gradient Boosted Trees Classifier with Early Stopping (ESXGBC2) - eXtreme Gradient Boosted Trees Classifier (PXGBC2) Boosting - eXtreme Gradient Boosted Trees Regressor (XL_PXGBR2) - eXtreme Gradient Boosting on ElasticNet Predictions (RES_XGBC2) - Light Gradient Boosting on ElasticNet Predictions (RES_ESLGBMTR) - eXtreme Gradient Boosting on ElasticNet Predictions (RES_XGBR2) - eXtreme Gradient Boosted Trees Regressor (XL_XGBR2) - Light Gradient Boosting on ElasticNet Predictions (RES_PLGBMTC) - eXtreme Gradient Boosted Trees Regressor with Early Stopping (XL_ESXGBR2) - eXtreme Gradient Boosted Trees Classifier (XL_XGBC2) - Light Gradient Boosting on ElasticNet Predictions (RES_ESLGBMTC) - Light Gradient Boosting on ElasticNet Predictions (RES_PLGBMTR) - eXtreme Gradient Boosting on ElasticNet Predictions (RES_ESXGBR2) Unsupervised Anomaly Detection - Local Outlier Factor Anomaly Detection (ADLOF) - Mahalanobis Distance Ranked Anomaly Detection with PCA and Calibration (ADMAHAL_PCA_CAL) - Keras Autoencoder (KERAS_AUTOENCODER) - Isolation Forest Anomaly Detection (ADISOFOR) - Mahalanobis Distance Ranked Anomaly Detection with PCA (ADMahalPCA) - Keras Autoencoder with Calibration (KERAS_AUTOENCODER_CAL) - Isolation Forest Anomaly Detection with Calibration (ADISOFOR_CAL) - Double Median Absolute Deviation Anomaly Detection (ADDMAD) - Keras Variational Autoencoder (KERAS_VARIATIONAL_AUTOENCODER) - Keras Variational Autoencoder with Calibration (KERAS_VARIATIONAL_AUTOENCODER_CAL) - One-Class SVM Anomaly Detection with Calibration (ADOSVM_CAL) - Local Outlier Factor Anomaly Detection with Calibration (ADLOF_CAL) - Anomaly Detection with Supervised Learning (XGB) (ADXGB) - One-Class SVM Anomaly Detection (ADOSVM) - Anomaly Detection with Supervised Learning (XGB) and Calibration (ADXGB2_CAL) - Double Median Absolute Deviation Anomaly Detection with Calibration (ADDMAD_CAL) Clustering - K-Means Clustering (KMEANS) Calibration - Calibrate predictions with RF (CALIB_V2_RFC) - Text fit on Residuals (L1 / Least-Squares Loss) (XL_ENETCD) - Calibrate predictions: Weighted Calibration (SWCAL) - Calibrate predictions (CALIB) - Text fit on Residuals (L1 / Binomial Deviance) (XL_LENETCD) - Fit High Cardinality and Text (XLF_LENETCD) - Text fit on Residuals (L1 / Least-Squares Loss) (RES_FDENETCD) - Calibrate predictions (CALIB2) - Calibrate predictions: Platt (PLACAL2) - Fit High Cardinality and Text (XLF_ENETCD) Other Column Selection - Converter for Text Mining (SCTXT2) - Single Column Converter for Summarized Categorical (SCBAGOFCAT) - Single Column Converter (SCPICK2) - Single Column Converter (SCPICK) - Converter for Text Mining (SCTXT4) - Multiple Column Selector (MCPICK) Automatic Feature Selection - Feature Selection for Ratios/Differences (FS_RFR2) - Feature Selection for dimensionality reduction (FS_RFCDR2) - Feature Selection for dimensionality reduction (FS_RFCDR_LASSO) - Feature Selection for dimensionality reduction (FS_RFRDR_LASSO) - Feature Selection using L1 Regularization (FS_XL_LASSO2) - Rare Feature Masking (RFMASK) - Feature Selection for Ratios/Differences (FS_RFC2) - Feature Selection for dimensionality reduction (FS_RFRDR2) - Bind branches (BIND) ``` ``` ``` ## Search for tasks by name ``` w.search_tasks('keras') ``` ``` Keras Autoencoder with Calibration: [KERAS_AUTOENCODER_CAL] - Keras Autoencoder for Anomaly Detection with Calibration Keras Autoencoder: [KERAS_AUTOENCODER] - Keras Autoencoder for Anomaly Detection Keras Neural Network Classifier: [KERASC] - Keras Neural Network Classifier Keras Neural Network Classifier: [KERASMULTIC] - Keras Neural Network Multi-Class Classifier Keras Neural Network Regressor: [KERASR] - Keras Neural Network Regressor Keras Variational Autoencoder with Calibration: [KERAS_VARIATIONAL_AUTOENCODER_CAL] - Keras Variational Autoencoder for Anomaly Detection with Calibration Keras Variational Autoencoder: [KERAS_VARIATIONAL_AUTOENCODER] - Keras Variational Autoencoder for Anomaly Detection Keras encoding of text variables: [KERAS_TOKENIZER] - Text encoding based on Keras Tokenizer class Regularized Quantile Regressor with Keras: [KERAS_REGULARIZED_QUANTILE_REG] - Regularized Quantile Regression implemented in Keras ``` ## Search custom tasks ``` w.search_tasks('Awesome') ``` ``` Awesome Model: [CUSTOMR_6019ae978cc598a46199cee1] - This is the best model ever. ``` ## Flexible search ``` w.search_tasks('bins') ``` ``` Binning of numerical variables: [BINNING] - Bin numerical values into non-uniform bins using decision trees Elastic-Net Regressor (L1 / Least-Squares Loss) with Binned numeric features: [BENETCD2] - Bin numerical values into non-uniform bins using decision trees, followed by Elasticnet model using block coordinate descent-- a common form of derivated-free optimization. Based on lightning CDRegressor. ``` ``` w.search_tasks('Pre-proc') ``` ``` ``` ``` [a.task_code for a in w.search_tasks('decision')] ``` ``` ['BINNING', 'BENETCD2', 'RFC', 'RFR'] ``` ``` w.Tasks.RFC ``` ``` ExtraTrees Classifier (Gini): [RFC] - Random Forests based on scikit-learn. Random forests are an ensemble method where hundreds (or thousands) of individual decision trees are fit to bootstrap re-samples of the original dataset. ExtraTrees are a variant of RandomForests with even more randomness. ``` ## Quick description ``` w.Tasks.PDM3.description ``` ``` 'One-Hot (or dummy-variable) transformation of categorical features' ``` ## View documentation for a task ``` binning.documentation() ``` ``` 'https://app.datarobot.com/model-docs/tasks/BINNING-Binning-of-numerical-variables.html' ``` ## View task parameter values As an example, let's look at the Binning Task. ``` binning.get_task_parameter_by_name('max_bins') ``` ``` 20 ``` ## Modify a task parameter ``` binning.set_task_parameters_by_name(max_bins=22) ``` ``` Binning of numerical variables (BINNING) Input Summary: Missing Values Imputed (quick median) (PNI2) Output Method: TaskOutputMethod.TRANSFORM Task Parameters: max_bins (b) = 22 ``` ### Set task parameters with a key Alternatively, use the short name directly. ``` binning.task_parameters.b = 22 ``` ## Validate parameters ``` binning.task_parameters.b = -22 ``` ``` binning.validate_task_parameters() ``` ``` Binning of numerical variables (BINNING) Invalid value(s) supplied max_bins (b) = -22 - Must be a 'intgrid' parameter defined by: [2, 500] ``` ``` ``` ``` binning.set_task_parameters(b=22) ``` ``` Binning of numerical variables (BINNING) Input Summary: Missing Values Imputed (quick median) (PNI2) Output Method: TaskOutputMethod.TRANSFORM Task Parameters: max_bins (b) = 22 ``` ## Validate task parameters ``` binning.validate_task_parameters() ``` ``` Binning of numerical variables (BINNING) All parameters valid! ``` ``` ``` Update an existing blueprint in a personal repository by passing the `user_blueprint_id`. ``` blueprint_graph = keras_blueprint.save('A blueprint I made with the Python API (updated)', user_blueprint_id=user_blueprint_id) ``` ``` assert user_blueprint_id == blueprint_graph.user_blueprint_id ``` ## Retrieve a blueprint You can retrieve a blueprint from your saved blueprints. ``` w.get(user_blueprint_id).show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABa8AAADECAYAAACY9t2uAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdZ3gUVR+G8Xt2N4WQ3ukgvUOE0JuCCogUAQUUyysWVOwF7NiwFxR7o4iKghTpqFTpndBReghpkABpu/N+ADGEBJJN2Q08v+tCZHd25rQ5Z+a/Z88YASFhJiIiIiIiIiIiIiIi7sJgksXVaRARERERERERERERyUnBaxERERERERERERFxOwpei4iIiIiIiIiIiIjbUfBaRERERERERERERNyOgtciIiIiIiIiIiIi4nYUvBYRERERERERERERt6PgtYiIiIiIiIiIiIi4HQWvRURERERERERERMTtKHgtIiIiIiIiIiIiIm5HwWsRERERERERERERcTsKXouIiIiIiFwWrFS48SN++3IoLUOtrk7MZcTAr95NvPvrOIbV93J1YkREREoVBa9FRERcwAjoyLM/zmbFul95uE7+AwhG+HW8MmkOq9ZN5O7Kl+Yw7n3lw8zY+DeHYn7jmda+rk7OpcuzFU9OmsvK9fN5obmHq1NzUYVtF0ZAB4ZPnMnyNVMKdM4VlquO6+5ULq5hrXILo9/oS+sed3Fb80AMVycoN6Wsb8rVeXnwpnavexjY5hpGfPwIUd6uTqCIiEjpcWne9YqIiBQj/25j2BkXR/KRVbze0tO5nZSpQvN2UdSO9MOjANEDS9kraNG2KTUjfLG6ZdShCBgWLBYDw7BiNQqfSWu1vrw7aQ6r131MLx8nduDZkQ+2xpIcf4Cfbwlxz2CPM2yRNGzdhFrl/PEsDZnKo13ku37LVKVlh2bUKe9foHMuL646rlNstRk8ehLzF60gZvtODh06RGLcYeL27WDbynlM/fI1Hu7dlPAiiBOWqnIpYgUaGwpTJ872SZYI+o4cTsdAk0M/DWfE7ATMAuaxRBRh31To/t9Z5+XhFKvfeZTRWzLxqj+UN++pg60EkyMiIlKaKXgtIiJSEEY4PW69jlALYK3Ejbd0ws/VabrEpK1+l24NqlKu7nWMXJpS6P1ZwhrSqX1TaoT7oPmdpVde7cJV9Vuq2pWlHFd2bkezetUoHxKAj6cNi8WKp08gkVc0pkOvu3jx81msnv8+N9co3JIGpapcilJBx4YSrJN/eUffz5PXBsOxBbz22jyS3DJyXbTcqj2mr+eDlyay3+FN1NDH6Rl6iXxrIyIiUswUvBYRESkA6xV9ua1D2TMz3SyEdb+VHuG6ARWR0iCDhU9HERkZSVBYJGGVa9OwQ1+GvjOb3WkG/vUHMHriSDoFqE8rKOfHhhKqEyOEG+69mWq2LP6e8CG/HHYUbn/ilNSFH/PpqjSM4Ou4b2AN1wfURURESgEFr0VERPLNk6hbbiHK0yR5/lh+3mfH8O3I7f2q6QZUREqFzLRTpGc5ME0HmSeT2L9lEd+/fhvX3vk9/9gNPKoO5LF+FXSTUCCFGxtKok4sFXpya+cAjMwtTJywhrRC7EsKwbGXSeP+JBVPmgy8mSaldElvERGRkqTrUhERkfzy7cDt/a/AZo/l189G8t7ErWTiSdQtg2ji5NLXGD7U7PYIH/64gE079xF3cDc7V81i3MjbaBlWsBUxPaJfYt2ROJIPfsON563taRB483iOxMcRv+wZmua2a6+KdLj7dX6Yt5o9+w5w5J+trJs7llF3tCAi5/betej+4AuM/nYyC1du5O99B0g4coCDO9bx19RPefmOtlTIrUw8KtB60KO88cVPLFi2jj1793M09gD7t65i8Y8PcqUHGJG3MfVQHMmHF/BE3WyhnzJ16P34q3w6/lcWrdzAnn/2czQulviDO4lZ9DMfP3wtV5S5QAF5deerfXEkx//358ikwRT5xPl/0zlhKotXbmDP3v3Ex8USv38b62Z9zvDu1SnjXZF2g5/ji18XsXXPAeJj9/HP+gX8OOo2okNyuzyzUqvf64yf9jvrYnYRe/gw8Yd2s+2vaXw1ojd1/fLOhOFXm95Pfsz0JRv4Z/8BYv/exIrpnzNy8JWEXSjvBWkP57FQ6e6pxB2NI3HLW3Q6ry2UodsnO0iKP8C0O8udd0Fqi3qOVbFxJB2azJDyp9/Ns12cTW8+69cSQZcnv2TW0vXs2XeAo/ltP3kp7uMWqh7yyyT+908ZvyUTDE+atrqSMlip88g8jsbHkbD2FVrluvZyB97bEkvy0b38OCD43LWXi7tcPMKIvvUFvpmxjO179nNk33Y2/TGR9x+8luq5rW1c2P7jQopjbMi1Tpxlodw13WnuZZAV8xszdtnPedfpscPZceDfPRe4b3K+H8xXe3TiXCt4Hkzi589gyUkTW9XruL6BVr4WERG5GI2WIiIi+WIQfv1geoRbyNrxE98uOc7WXeP4a9go2lfvy23t3mXNghMF362tOv2HP5ntBW/Cql1Jj6FRdOvbjWdvuoNPNp0sslzkxQhuw4jxX/NYdFC2QGII1aKu496mV9O97SP0uOcn/sk6s31gK+4efj8dcgQmygZXoG6bPtRt04vbbvmSuwe9wOzY/wIlRkhnnnrr6fM+5xFWhTrBBikX+CW7EdCC2x8dct5n8QqgfL32DKrXlm6dRtKj/xg2pztRCEUkz3SWCaZa8148+U0nbjvqQUSEzznBvsCKDbn2rjfp0KYaN3Z7iaUp2RektRBQvxNdW1+RbSanH5E1W3Ljo9F0aVOJ6/t8yMYc+bZV7snonz7i5hpe2Y4VQe1Wvajd6sw/z41jnc5DAdvD+RwcXrqEnfZW1A9uypXVrPyxPduBbHVpEeWLgYUGUQ3x/PpwtpmgFsKbNqWyFbK2LmXJkSJe3sAaTnSP7tle8CyZ9uPEcQtfDwXgiONwnAkY2AIC8TXs7F72FwftjalaLppWVa38tfPcxmKrGX36y5asrSxdccy5h/85Uy6BzXns2+8Y3iY024NrvajU8Gpub3gVN908kaEDn2DK3sz/PlNs/UcxjQ2QS53ACacKuSzRbZrgZdj5Z+kSdudyzjvD2XEAnO2bnOsH85UXJ841Z/tXM2kZCzdl0rVlFVq3qohl3T9oERcREZG8aea1iIhIflir0n9wR/xIZ/X479mYCY4Dv/Lt3GQclkh6Du6KU89eMjM48OcYHurbjjpVKhJRtTHtbnuD2fszsYR3ZORXz9CubJHn5lyWcvR753Meiw4kc+9sXrm1I3UqVySyVit6Pz+DfzJtVOrxCqP6nz9DlqxdfD4gimqVKhAcXpGKDTpy8/M/szXVwL/xEL788h7q5DZj0/4P4+/uSMPaVQmLqEil+m3p9vBP+QuqZO3mq8GtqFOjKmERFShftx03vTKPg1kGQa2f5LVbKuV+gZP+G/+rHE5g6H9/IvqNJa64Hlr2bzqrVyEssiJVm/fj5d/jMC0BRIY72PrzSwzqHEWV8uUIrxZF1+Ez2JcF3nXuZMSAnHnIYtuk4Qzq1o76NasSHlGeyNqt6fPibPZnGvhHP8SI3mHnzny11eSeMacf9mYmrubTYdfTtGZlwivWJqrbPbwyZSvHc4uYFKY9ZE/xzqUsibWDrRYtmgWdkzZLhRa0rGQDLPg3i6beOdMpfGjWsiGehp3DS5ewK7+BtvzWr/0AvzzRk5aNahFZrgDtp6SPW0T1kG+WclQqZwFM7MeTSTUhc8MfLIx3gK0u7VvnaF8YhDZrQXUr2A8uZ/n+HBVVbOUSwY3vfMOItqEYqZsZ+1gvompWJqJKI9rf8Ta/H7bjXXsAY75+iMa5zfp1tv/IS3GNDZBrnTjFVpMmDcpgmBnEbNhB5sU/UTAFHQec7Zuc6Qf/daH26My55nQeAMcRNm2Kw44H9ZrUo2gexykiInLpUvBaREQkHzwaDuTWKC84tYyJU/aeniVlJjHnxznEOwz8r76FvpWdGFaztvPdcy/z3Z/biT2RQXrqYTb99g63DXqL1afAo9pAhvWOyP1mvIh4XnkPT3YNwzi1mtcH3cXbs2KIPZlBWuJu/hgzlLs/30mWJYBOA3pQMWcWzVMcPXCYpFOZOBwZpMbGMHvM/XT/31j2ZIFv9DCe6Bp0fvodKezdup39CSfJtGeQcmQHq7bE5jZJ7XzmSWL//ofY5JNk2jM5eXQ7cz54gBHTk3AYZWjR/aqiXwrEGf+m89gpMrMySP57Ie898g5L0kwwM1j/85f8tv4AxzLsZKQc4K8vH+eVOSmYhhdR7aPxO3dnpGz5g9krt3Mw6SQZ9izSEnbx+8cP8sz0ZByGLy3aRZ0TBCnT9m7ub14Ww76Xb+8ZwPDvV/J3UhoZaUnsWTmFtx96n4W5RLEK1R6yy9jAouXHcRieNGvbLNuSBwZBrdvQ6Ewwy1a5Fa2z78izCe2ifTEcx1i2eGPRB9ocSWxbvpJth5JJyyzB9lPA4xZZPeSLhbAu9zCwjg3MDDasWMcpgLSVzFmYjMPwpPk17Qk5p1zKENWyIV6Gg2Mr/mKTsxVV0HJpeg9PdQ/HYj/MLw/dxEPfLWNPUhrpJ2LZOP1NBt7yHhvSoUyje3iyZ+j5fU8R9x/FNjbkVSfO8KpM1XJWcBxl7wGn95K3Ao4DzvZNzvSD+eHMueZ8HgDsHNh3EDsGXpWqEqk7chERkQvSUCkiInJR3rS5pS81bCapi35h5pH/pr+dWPQzMw7bMbyaM6h/7SJ7cGP61nF8tfAkpuFD6y5t8S+i/Z7Pg6Y9ulHNZpK2ZBzjtmfkeD+NdfOXEOcw8KwfReN8RQVMEv94j4+XpWNagujSoy3FPXkcM5lFC9aQYRrYqtehhpsujOY4spK/9tjB4ssVNcLPvRAzk1m1YgeZGNgqVKZ8fhqTmcLmDbuxY+ATHo7/2aCbB407X0WEFTK3fM/ni/K7pENRtoeTLJu/nJOmBf+W7bI9mKwM0e2a42Xfx6o1R7B7NKBty8CzgS1brTa0jbBipi5l7vISeqycq9pPnsctjvMyB4sH3gGR1IjqzO3Pj2XOFzdT2WqSdeAXPpi0/8wyBidYNP13khwGPq2upUNAtqiuRz1aN/PHME+w7PeVRfsAwDzLxUaj67tSzQZZOyYyeubR89p12sav+GRBKqbhT8ceHQnITyDa6fov4rEhX3VScBb/IIJsBjiSSEgqqQUq8hoHnO2bLnSovPrB/HDmXCtsHhwkJSThACyBwQTpjlxEROSC3PTWTkRExI34X8XAHuWwmsdY8Ms84rPfpaYtZ9L0A9x2bxXq9buJKz94kZU5732dYR5j44a/ybquAV7Va1LVBhuKYl3bnAw/atYqjxWDMl0+ZPfRD/Pe1juMiEALnMpH8MMRx4oVf2NvXxef2nWpZpvOpuJI/1kmJ2IPc8yE8MDA/AWsXMGewJF4B2DBP8AfC2QLSDlIjE/AAXiU9cPXwjnrpdrCouh/913069SIGhXLE+5vIfXIfg7ZI7ACps12ev1fEzD8qVkzAisOkjdt5O/8Lr1RpO3BJHnRPJanXUfncm1oX8vGsi1Z4NmUTq39IX4673/qw+jP+hDdoQU+P8ziBBYqtm9HdZvJqeXzWHS8uNZ0OT+trmk/eRy3uM5LPOn8/laS3s89LSd3T+PZO0cwO/G/ck9dNIXZ8X0YFNqWHp0CmTwlCROwVm5Jy4pWzLSVzFmUXPgAZI605F4u/tSpWwkbDhLXr2F7bn2KeYy1q3eS1S0Kr9r1qGGF1Rfte5ys/yIZGwpeJwXm6Y0ngJlBekZJnVPkPg7YneybzihQP5gfzpxraYXLA0BG+ulp2YanF57uOl6JiIi4CX3PKyIickEGYV1vpmuwBUfifCbNT8pxT5zB6l+msjsLrFV7M7CtTxEd1yTleOrpOKSPLz4Furk18r/MiFEWX9/8buyNV75neDo4lnwMB2D4+lG2BG7OzfR0MkwwbB5u/O18BhlnAlgWy/lzMTMyz0TZrNZzZmrarhjIdwum8/EjfenUpBaVQn3x8vQhpFJtGlYNPP+CzvChbFkAk9SU1PzP2Czi9mDG/8nctZmYtpp07lQZK2CrfxWdIi2k/LWQxYv+ZMUpg8B2V9HcCzDC6XRVIzzMDFbNW0hh4nUF5ar2k+txi+28zHZc007GySQO797Aoqnf8Or9N9C8/RC+3pLjAbEnFjFx2kHslkC6DLieCAuAhciOV9HQZpKxdj5/JBR9RV24XExSjh3Po107OH7sdJu3+J75EsjZ411Q0Y8N+a6TgspIIwPA8MTrgpHSAowd+ZLLOOBs34QT/WB+OHOuFSIP//L0Ov1TFDPjdLsTERGRvLnvvZ2IiIg7sFSg94CO+BpghPRlwt99L7BxBDcMuJoX/pjOsULfjFoICAzAApipKfl6UJeZmUWWCRheeJcx4GR+PnSSEycAHMSPH0i9h3+nKCaOg4Gfvy8GYJ48wSmX3pyXpshALmk1QrnxxZfoVt6G/fCfvPfcKCYs3s6h5DQMn3CiH/ueyQ/Uz7GbE6SmAljwDwrACvlbO7qo24PjEHNnr+fl1s1p2LkjkWP24ndVR6paTzBv3hJSEj2ZszKd69p3oHNjDxbubsc1zb0g4y9mzj+Sz6CQq+q3GI9bbOdlBvMfbky/8QkFSH06y8d+z5bBT9Kw7UD6V/ueD/cE06nLlXiSyYrZ8zl0TkUVZ7n8264N/M78cuF8FvwDfE/3nSdSSS2uVTKKbGxwpk4KxnE8iaQsE7yCCAnK8ZMOnBw78iWXccDZvsmZfvB07i68X2fONcPJPJxlISgk6PQvb5ITKbGVXEREREopzbwWERG5AGv1Ptwc7ZXP2WgWgrrcRPfQIpi7ZgmnZcsrsGJycsdW/snHkhuO5ASSTMBSgSr5WjAZMI+za9cR7FgIjGpGraL6WtsIoFHjatgwSd+9M1/pLy5megbpZwIzXl6l8PfZHvVp1dwPw0xj/qv38Nqva/kn4QQZdjvpKUfYezjl/PCMmcL2bQewY+B3ZUsaeOS241wUeXtwsG/mNNZkgmezrlxTvjpdr6uDLW0Vs/9MxjSPMn/OGjIsFbn2uoaEtr+ONj6Qvmoavx3MZ+jaRfVbrMctrvPSSVlbJ/D5wlTwbML/hrTCN7IrfVp7Q8Zqpsw4dy3m4i2Xf9u1Bf8mUdTOrVwMf5o2q4kNk7TtW9jtxJIO+eGyscEZ6fv457AdLGFUqVjmvLedGjvyI7dxwNm+yZl+kHy0R2fONWfzcJaVipUrYMUkff8/xCp4LSIickEKXouIiOTJRv0b+9LI08B+aCx9KoUTGJr7n5A2r7Euw8Qo254BN1Qo5ABrENzpEYa28sRwJLNgxmJSz7xjmqf/YHji7XHujbjjYAwxSQ6wVqfLNdXz+fOqTNbN/Z1YO9hqD2RY17Ai+dm4V93B/K+jD4Z5kmXzlnC8CPbpLEd8HAkOwFKN2tWKMDBTkszT/7FnOfI5OzOTDb/N5p+s0z+1f+LmKi5rD479M5m6OgO8WnDjvbdzfX0b6StnMi/eBBwcmjuLNZlWqnUdyMM3tsePdFZMnZ1jNu8F9u+i+i3e4xbPeek0M5YpX0zhkN1K5Zsf5vH7bqJ1GUhfOZUZOb5kKO5y2fDbbP7OAlutAdx/beh55eLd4H/ce5UvhnmchdMXklws05ldNTY4KWsn6zefwjQ8qde4Fjljrc6NHReX+zjgbN+EE/1gftqjM+daIfIAYImgYcNwrGQSsz6G9IJ8VkRE5DKk4LWIiEhePJvS/8aa2LCz99efWHIq703tO39mwoo0TMOLFv17cUV+YzaGDxGVKxBcxgOLYcMnoi7X3vcxM74azBU2OLF6NG/+9t9aqubxBJIcJtjq0vvOLtQO9vzvRjtjOT9PPYDd8KDxsE9455ZoKgd4YjFseAdGUrWcf6435WlLxvDekmM4rOXp99FkvnigG1GVg/C2GVg8/Yis1YIbbr2OOrndndtqM/jl5xjcoTYRZT3x9I2kQbdH+W784zTzhsw9E/hgyhGXLtzhiFvLmv1ZYKvGoOFDaVOhLDaLJ34VGnFdj2gi3f1qKHMrazaewjTKcPVjr3NXuxqElrFhYGDzCiAkIPfZnxlrPuX13+JwWIK55o0pTBzem2aVA/C0GFi9/ImsVoHAXD5YqPaQG8chpv+ynFN40XrIHUR5pLN8+jyOnIl5Og7MZNrqTKw1BnFvF384uZSfZx7O9zqyrqrf4j5ukddDIZ1Y+AkfrzoFvu146L5meHGSJZNnnfclQ3GXS8aaT3ljZtzpchk9kXdvbUnVQE88fSJo0P1xxo1/hKbekLbxc96cerR4+p6SGBuK1AlWLl1PummlYpu2VM+ZBifHjrMKOA441Tc52Q/mpz06c645278CGEGt6dDQA7L2suyvA06tmS0iInI50ZrXIiIiefBu2Y/ela2QtY2fJ6298DqYjoNM/XERL7a9Ft/GN9K3zieM2pKP36vbqnPXhNXcdf4OSdn0Lffc9Qlbsy2maSYvZvriY3TpEkiju8ey7NpP6d7yeZZnAqSx9J0RjOv0FbfVaMBt78/gtvfPP+R5K3jY/+br++6l2vefcl+T2vR98Vv6vphjm4xVPLNoLtv25rjNNjyp1HEoH3YcmuMDJvajC3lxyOssOXHxYihWmev56uOF3PL21YR2epbfNjx79i3zxCzuWbyKn4pnembRMOOYNGo0g1s8SfOafXl7Sl/ezmWz89qnI5ZfHr2dyoHfMqJ9Rbo89hldHsvlgzmbaWHaQ64cHJ7+PfOea0fPQCvmqcX8Oif2v4CN4xDTf13Bi63a4W11kDBnAtPiClAfrqrf4j5ukddDIdl38+3rExgy+S6qWcGRNJ8JM3L5Yqq4y8URy8+P3knVkO8Y3roxd7w3jTvey76ByakdP3D/ne+zvpimtJbI2FCkHBye+xurRralbb3udK8xmm3bs6fBybHjXwUdB5zpm5ztB/PVHp0415ztXzEIvfp62voYZO2ezYzNLlxTS0REpJRw97lGIiIiLlKWjv26U85qkrHhJybFXOwG0yR+zmQWHDPBVps+fRqf99Psc7Y+toLv3v+an39fy7YDCaSkZ+FwZHIy6RDblk7hoyf70eq6p5l5KMedr+MQ3z8wiKe/+4PNB5JI3bmDPdmSZsbP5dHuPXnok1ms+SeRk1kOTEcmacfj2bdtDQumfMcHn85lf87dxi3gmW4d6fnkp0xdvoPY4xnYHVmkpSawd8sypo2byvqTuWQkaw8/v/kOExfGcOhYOpkZJ4jfu5aZnz9Nj46D+HijqyPXAA72jrubHg+MYdbGAxxLt2PPOMHRf9Yx95el7CsFK4mkrX+PPl3v5vUfFrJpfxKnMu3YM9JISYxl746NLF8wnYmztpz3YE/z2Gre7d+eLkPfYsKCjexNSCHdbmLPOElS7B42Lp3JuI/HMOPvcxuE0+0hD2biXCb8dhQHJicWT2bmOYu8Ojj022QWnzLBfpipExYU8IGnrqrf4j9uUddDYZ3861M+X5WBiYMjMyYyJym3iir+cjGTV/JW3450f+JTpq/aw9HUDDJOJXFwy5+Mf/UOOlzzMJP3FuwRevlXvGNDcXEcnMq4BccwPeozYNCVeOdMpZNjB+DUOOBM3+RcP5i/9ujMueZU/2qpQt/BHfElgw3f/8D64mqmIiIilxAjICTMjacaiYiIiDsyIm/j17Vv0cHYxKtXXcNbW0t6JqGIlDSPmvcxdf6LtDLW8XLXHry7RZG30sS75UiWTr2XailzebDdYCYcLtyMfY0DBefb6S2W/XAbFZOnMaTNEH6J1624iIjIBRlM0sxrERERERHJwSCoSm0qBnhi8w6mZvu7+WzCM7TySeWvUQ/xkQLXpU7ayjG8OTsRAq9m+PDOBLn0KaCXIa/GDHthAJUsaaz95G2mKnAtIiKSLwpei4iIiIjIuYxgur8xn027DxB/YBurJr9Cn2qwb/Lj3PPZ9guv8yzuyRHLz8+/zp/JBhVuGsWr14Zc+EGMUoTK0Oyx9xhW34P0LWN46tNtea8hLiIiIufQAxtFRERERORclhC80v7h6KnqBFtSObJzNbPGfcCb360gTqtDlFr2veMZ9nQ0n3fdwrg1yec/cFOKSRrbp37OxBbX8/eI91iT5ur0iIiIlB5a81pERERERERERERE3IvWvBYRERERERERERERd6TgtYiIiIiIiIiIiIi4HQWvRURERERERERERMTtKHgtIiIiIiIiIiIiIm5HwWsRERERERERERERcTsKXouIiIiIiIiIiIiI21HwWkRERERERERERETcjoLXIiIiIiIiIiIiIuJ2FLwWEREREREREREREbej4LWIiIiIiIiIiIiIuB0Fr0VERERERERERETE7Sh4LSIiIiIiIiIiIiJuR8FrEREREREREREREXE7Cl6LiIiIiIiIiIiIiNtR8FpERERERERERERE3I6C1yIiIiIiIiIiIiLidhS8FhERERERERERERG3o+C1iIiIiIiIiIiIiLgdBa9FRERERERERERExO0oeC0iIiIiIiIiIiIibkfBaxERERERERERERFxOwpei4iIiIiIiIiIiIjbUfBaRERERERERERERNyOgtciIiIiIiIiIiIi4nYUvBYRERERERERERERt6PgtYiIiIiIiIiIiIi4HQWvRURERERERERERMTtKHgtIiIiIiIiIiIiIm5HwWsRERERERERERERcTsKXouIiIiIiIiIiIiI21HwWkRERERERERERETcjoLXIiIiIiIiIiIiIuJ2FLwWEREREREREREREbej4LWIiIiIiIiIiIiIuB2bqxMgIiIiIlIS6tSpTe9evV2dDBEppCm/TmHbtu2uToaIiIiUAAWvRUREROSyEBoWRtt2bV2dDBEppMVLl4CC1yIiIpcFLRsiIiIiIiIiIiIiIm5HM69FRERE5LLz4UdjWLlylauTISL5FB3dnGEPDHV1MkRERKSEaea1iIiIiIiIiIiIiLgdBa9FRERERERERERExO0oeC0iIiIiIiIiIiIibkfBaxERERERERERERFxOwpei9EdS9gAACAASURBVIiIiIiIiIiIiIjbUfBaRERERERERERERNyOgtciIiIiIiIiIiIi4nYUvBYRERERERERERERt6PgtYiIiIiIiIiIiIi4HQWvRURERERERERERMTtKHgtIiIiIiIiIiIiIm5HwWsRERERERERERERcTsKXouIiIiISC4slO8xkl/nTuelth6uTsw5jKCOPDdxBotHdcbL1YlxE9aI1gx9axx//LWGnTHr2LR4CqO6hmK4OmEiIiIihaDgtYiIiIiI5MKgbMW61K8UhHexR0C9iBr2I2tXz+SNa0IuGnA1vMtTr2E1wnxsZ7Yt2OcvOZ71eeizj3j8hiiqBntjs3riGxaJV3oqpqvTJiIiIlIINlcnQERERESk9PKmSqdbuH9wd9o1qEyoj0H6saP8vW09S2Z9z+c/byDJbaOHVurfMZp3bvFl+tA7+Hi7vUj37t3yIcY9dz3VwoPxK+uJ1czgZPJR9u3cyJI5P/PdLys5nPHf9oZhYBgWLE5Gns//fEHy50efTxbyzlUXm8edyapXu3Pz2IM4nEtmsfBq0Z8BtT3J2PkTjz/yIfP2pOIZVh7f1HRXJ01ERESkUBS8FhERERFxipUq/d7l55faE2r9L+JqC6lIgzYVqGlbz9hfNuC+U18tBFapR81ySXgWw1Rla1gNGtYon21ZD2/8QitRP7QS9Vt1ZWCfLxnyvw9ZcdwE0lnzQX+afuDs0XL7fPHmz31YiKhRnQAjncVff8BvO5MxgfTYvaS4OmkiIiIihaTgtYiIiIiIM2xNuG1oW0KI4/d3nmfU5HXsTc7EM7gS9Zu3p9GpPzniTtNzXSKLmI9u5sYx20g3rZQJiKRG1DUMefx+uje8k5F3zKPbBzEU7ZxvZ6Qw+b4oJp/9t42oJ6fz0x1+/HJ3J55anOnCtP3H4uVHSKA3WSlJJJ3MOvOqgZe3J4Z5ivj4E+77XYmIiIiIExS8FhERERFxhk9FqoRYceybwYdfLWHnmQhsRtxuVvy2mxVnNzQIaTOEZ27rQL3qlSgf6o+Ph53jh7by5/cf8dmGCvQa1JNrWtShYqCVEwc3M/e7txn1/SaSs0ciy1TlmjvvY8gNralXviyOpL2s+f1nxnz8AyuP5gj/FmRbay2GTd3IsDP/dMT9wK1Xvcyyf+O1hg8tH/ySOa/WpnKIN/bkfayfP5F33v+BdflYE8WRlUGm3cQki5NJB9i44GseOxZGo7GDqdY8ighLDIccVmre9wMzh5Vjco5gsSW4MQOG3sugLk25IsSDk4e38dfyY0Sc8/SevD9/0fwVmEFAk/48dNs1NK9fg6qRgZQxThG/dw4vDX6R2UYHhr/7EF1rVyTC3wvzVDy7V8/hy3c/Ysr2f4PLubUJSEvKvWwtwc0Y8vwI7ulciyAPA9PMJGXfPEbe/hS/HDq9PyxB9P9iPf3//VDmKl7ocidjDzvy2R4ulK+XWFn3rsK3YREREZECUvBaRERERMQZpw5zIMmOpdLV3Nr1F3bM2MupXDe0ENyoCz061Mt28e1BUKWm9H7qK3rn2NqzSjNuevYTgk/cyL2/Hjm9trJ3Pe75/EuejA7474nrEbXoMGA4rds35rFbhjP90JkgZEG2zQ/Di8qNm/3379DqtLn5GZrUKkOfW79mR1beH82LI8t+Ol8WywWfIG/4t+bZsaO5vab32YcwelVuQrfKp//fNSs6Wwhv1Zdbu2WvTz/Cwj1ITzWhTCDVo2pR0fPMW74R1O04mLcahJPZ83Gmx5vk3iagbG5la4mk36jRPNnBHyPrBAlHUjF8QwgM9yD9mAOwXji5+W4PF8qX4XQbvvvXIwUpXBEREZFzXOhaUURERERE8pK5hq8/WkK8UYUb357C7xNGcu+1dQjOa3qIeZxZw6+hcaNG1GjYlu7PzOSA3cSRvIqPHuhLqyubUCuqC7eMWcNxAunQ5yrCLQBWatz6HI809ydt68882f8q6jdoStNrhvD674cxynflxSe7EGQUdNsz7Dv4sGcjqtWuT7Xa9aneLsesZDOVP1/vT9voK6nZoAVtB41iXqyDso0HMSjKI9/FZVg9KRtckfrtBvDK8/2obLWzb81aYvNcWsVGg/89zeAanhxfP46H+57OS+OrBvHwl6uIz++SLBfLn7PMFOa9cD3NmjahRqNWtOv3ASsywUxZyluDe9Gq+ZXUqNuIui2v584vN5IW0okbOwZxzvLb2dpE9fq5l63h14JrWvjh2PwZvVu1oln7q7jyymha9n6bJSez7cuRxE93NTmbz2oNbmfsYaPg7SGPfOVMb37bsIiIiEhhKHgtIiIiIuIUO3snPUyfez9kWswJQq68kac+/Jkl877l1VubE5EziG3aSTkax/F0O/aMJGKmjGHCFjuGLZ6YpVuJTc0k88Qhln7+FfOOmVgrVaWyBbDW5Iae9fHM3MTox0YyacMRTmZmkLx3GZ8//gI/HTYJ6ngDVwcZBds2v8xM4nbv4OCxNLIyUzm4+nteG7eZLGsIdeqEXeSGwkaDh6exa/sW9sSsY/Nfc5jx5bPcVL8sp7aO54WvtpDnxG1rba7tXBVL+href+xNpm46nZfjB9czffxcdrl6oWwzi6SDB0g4mYk9/TiH9h7hhAkYBoENb+a1r39h6YpVbPxjLC9eWx4rNiLKhZ5bXtnahCMrj7I1TUzACKtDdJ1QvA3ATOfo3wcuviSHM+0hr3zlSG9+27CIiIhIYSh4LSIiIiLitAwOLPqch/p0pv2gZ/lk3i4yIpox8JmvmPZxf6pfaJE++2H2HsoCrzAiArNdlmfEciDOxCjjQxkD8KxCjYoWHPtXsPSfHBHbE2tZvC7t9DaVLAXb1ml2Du3+hxOmga9vWfIdBjdNTNMEx3FWfzmMHgPfYumFoq8e5alawYLjwFpWHy4lT740/Gn/7LeMHXEznRpWJcLfC48ywVSuFIqXARbLxVZtPL9szZRlTFkQjxHRgRFj57N+2Qx+/uQlHuxak7IXK/zibg/5aMMiIiIihaHgtYiIiIhIoaUTu2YKbz7Qhw59X2TaPgdh7R/iwat8L/AZB5kZWWB44OGRPQqZSWaWCYZx5mK9ALOkC7St88yMDDJMA8NyseNlsfn9G6hRuz7V6kRx7WvLScaX6nXCIf0i04YN6+n8G5YSylXhGcGdub13ZaxJK/jo/htpdWUTatRrRssHpxCbz5ni55WtGc9vI27lzle+4scFa9lnlieqU18effcr3uwWepGyKe6Su3gbFhERESkMBa9FRERERIqMg2MxU3j/p63YLb7UqFHuYo/Tu7iMvew+4MBSqQVtquTYW9ko2jX1hox97DngKNi2mGRlZWHig49PSQQZM9g5fgTPzDyKf5vHePuuOnhdcPN9p/NSuTUdq+d/be3/lHT+wBIaSaQnnFwyjtHztxGbmondfoqEo8cL93DJtP0sHPcuT99/G9e060C35+dy2Aym47XRXHBuc4Hag4iIiIj7UfBaRERERMQZnk25+7XHGXxVAyoHeWM1DKxlgqnWvDdDe9XCamZxNC6RQocF7TuYNi2GDI+GPPjuc/RtFIGPzZOAKq25+62X6F/OIHnhdOYnmgXbFpO42KOY1nJ06deFar42rN7BVG9Wn/KFjrjnwRHHrJEvMOmgF02HvsSQup4XyPd2ps/YSqatHvd/9AZD2lUn2NuKxepNQGgQF49Hl3z+HAlxxGVCmRZ9GHRleXxtBlg88PX15mILhuTJWo2rbryKRhX88bQYWD1sZKWkkM7pic0XLIYCtQcRERER9+P0NZSIiIiIyOXMVu9qBva6gyo33pHLuyando7lizmJmIWeL2Jn57iRvN/+S55o3o+3JvXjrWzHyTw4ixffmMPp+GPBtt238HdihjWkUZ+3+b3Pmc0y1/Nat1v5Yl8hk50H89gS3nh5Ku3G9Oa+5wcx55Zv2Jnrkhp2dnz3Im+3/oKno69lxJfXMiLHFheezXyx/BX9bGMz4Xd+WvAA7bpfxfPfX8XzOdKzw4l9GiEt+N9Lz9E65+RzRyKz5q7ixAU/XZD2ICIiIuJ+NPNaRERERMQJjj3TeOPdCcxauYNDx9KwOxxknUrm4Pbl/DrmafoOfJtlKUUUFTwVw6dDBjJ09G+s3pvIqcwMTsTtYNHE17nlpqeZdsju1Lb2nd8x7Ilv+GNnPCftdrJOJrBn3S6OFutaxSbJi0bzzh/JeDe5i0e7heQ9e/jUVr4YchN3vP0zS7Yf4Xi6HXtWGinx+4lZOZ9fFu4h8wJHKvH8mYnMenYIj331B5sPHSfdbicr/QRJcQfYsWEFy3cdo6AtwjAOs/bPjexNPEWWw4H9VBL7Ns7n86f+xxMzjl58fwVpOyIiIiJuxggICdP37CIiIiJyyWvbri3Dn34agA8/GsPKlatcnCIRya/o6OYMe2AoAK+PGsWSxUtcnCIREREpdgaTNPNaREREREodi0WXsSIiIiIilzqteS0iIiIipc7gwbfSrFkzNqzfwKbNm4iJieH48RRXJ0tERERERIqQgtciIiIiUuocO3acqlWrUrlyZXr26gnAocOHWbtmLZu3bGbLps0kJSe7OJUiIiIiIlIYCl6LiIiISKmTmJgIgNVqPftahfLliQgPp3v3blgsFhLiE9i4cSObt2xh3bq1rkqqiIiIiIg46bzg9b8PsZELi9m2lam/TnV1MgpM9et6U36dwrZt212djAKpU6c2vXv1dnUyREQkn14fNcrVSSh2iYmJGIZx3us223+XtyGhIbRr356OnTpiGAYpKcfPvufv51ci6RQRKUm63xOR0qo0Xr/27NWTenXqujoZl5Tc4q3nBa/btmtbYgkq7aZS+oLXql/XW7x0CZSy4HVoWJjajohIaVL6rv0LxMPDI9/b2mz/zcz28/PP9rp+gChSWtWpU5vE+ASOJiSQlJhIVlaWq5PkNnTNLiKlVim8fq1Xp6763WKQM96qq3YRERERcQuenp74+voSHBxMcHAIwcFBBAcHExISfOa1038CAwOxWCz53q/DbifLbmfjpo00u7IZAIlJScWVDREpZr169jr7q0DTNElOTiYxMZGEhHji4xNJTEzk6NGjJCUlcjQ+noSERE6kpro41QXnYfMgMyvT1ckQERFxqTyD1ytXruLDj8aUZFpKhfFjv3F1EoqE6rdkRUc3Z9gDQ12djCLx4UdjWLlylauTISIiOQx7YCjR0c1dnYxcBQUGEhAYQEhIKIFBgYQEBxMYFERwUBAhwSGnXwsJwcvL6+xnHA4HycnJJCUlkZCYyLGkZHbv3s2xY8c4Gh/P8eRjvPb6a3nOwjZNExM4kZrKtGnTmTZtGk2aNjkbvBaR0mvUG2+wft16IiMjz/uiq3z5cjRoUJ+QkBDKli179jMZmZmkpqSQmJhI7OFYEpISSUxIJPHM37GxscTHx7vVLO7uPbrTqmUrfvrpR9asKdi6/brfE5HSwJ2vXwvq6Z1VXJ2EUm1Uzb15vqeZ1yIiIiJSYB4eHvj5+eHr50twUDDBIcFn/w7599/BwYSGhp6zREdmViYpx08HkBITE9m3fx/r1q8n9UTq2dcSE07PmrTb7RdMw/FjxwgJDT3nNdPhwLBYiI+PZ/KUKcyeOYuMTM1cFLnUpKamsmvXLmBXntt4enqe/sXGmf4oMiLybB9Vs0YNgqODCQ8PP+eXHKmp2fqixCQSExM4fDj27P8nJiaSlJSEaZrFnsfQkGDq16/HyJEj2fvPXr6f+D3Llv2Fw+Eo9mOLiIi4CwWvRUREROQsZ5fuyMjMJDEh4WzweeeuXaSmpJ4TACrqoE9CYuLZ4LXdnoXVamPv3n38MmUyf/7xpwI8Ipe5jIwMYmNjiY2NzXObf7+Iy63Pi4yMpEaN6oSGhuLj4/PffnP0d8U1izs0NAxMEwyDypUrMXz4cOKPxvPLlMn6Yk5ERC4bCl6LiIiIXAZyzkAMDg4mJGewJjiYsr6+53wu5yzEffv2kZCYePr1M8Ga+KPxnDx5ssTzdPRoPDVr1gRg3br1/PjTT8RsiSnxdIhI6ZWZmXm2j8vPLO7IyMjzfmmSn1ncsbGxJCQknveF3unj5i4iPALjzP7+/TskNIQhQ4YwcMAApk2bztSpUzlx4kTRFIaIiIgbUvBaRERE5BLz4IMPEBwcRGBgECEhIQQEBJyzdEdGRgZJiUkkJCaQnJzMvv372bRpMwkJ8SQnJZOQmEhycjLJycluPXs5Li6W+Qt+Z/Ivv7Bv3z5XJ0dELmH5ncUdHBxESEgoISEhBIcEExYaSnBwMJUqVaJxk8aEhITimW2t/vS0NI4cPUpSYiIJCacfOpmYkEhc/FHCwsPOO4ZhGBiAn58fN998E71792LKlF+ZPn06KSkpxZF1ERERl1LwWkREROQSU65cORISEzl48CDxCYkcSzpGfEI8x5JPB6YvlVl6X3/9rVsH10Xk8pKZmcmRI3EcORJ3we0uNIu7fv36BAefnsV9sSWWrFYrPj4+3HzzTfTr14/Zc2YVZXZERETcgoLXIiIiIpeYESOecXUSSoQC1yJSGuVnFndIUDBjx4/N1/6sVitWq5Ubetxw9jVvL69Cp1NERMQdWC6+iYiIiIiIiIiUFL9A/3xtZ7c7zn6Rl5pt2RCLzVos6RIRESlpmnktIiIiIiIi4kbCQkJzfT0rKxOr1YZhGCTEJ7Bx00Y2b95CzNYY9u/bz4wZ0wE4eaLkH6IrIiJSHBS8FhEREZHLTtdrr6FldHNXJ0NE8ikoKMjVSShRwSEhANiz7FhtVhwOB3v37mXjho1s3rKFrTExJCUnuziVIiIixU/BaxERERG57NSsWcPVSRARyZOPT1k2btjIps2biYmJYdu2baSlpbk6WSIiIiVOwWsRERERERERNzJlymSmTJns6mSIiIi4nILXIiIiInJZWLJ4Cd0XX+/qZIiIiIiISD5ZXJ0AEREpDlaq9HmdaXMnMyJa31O6IyOoI89NnMHiUZ3xuujGAbR/5mu+HNqcEKMkUlcQFsr3GMmvc6fzUlsPVyemQApUB0XCm5q9XuL7DwdQTaeliIiIOEXX+RdXUten7lkX1ojWDH1rHH/8tYadMevY9OdHDKyk8J+cy/Dy4sFrQ5jU2gtPVyfmItR6RQDwImrYj6xdPZM3rgnB7WJDkk1x1NWlWf9+lepRr1Ig3salkqNLi+FdnnoNqxHmY7tImzPwa/MQLw9qxhUhFjKKLUXOngcGZSvWpX6lILxLWVM7vw6Kuy/IJMuvEg26PMTLN1XCWuT7FxERkcuBrvMvpuSuT11XF3lct3rW56HPPuLxG6KoGuyNzeqJb4iFjOMm1pqDmbB0JUtH96GyooGXPcNqpWaIjWCbcab9GDRoHMyMm0J5urLFreIixdZcvVs+xKTf5rF61Rq2x2xi16ZVrFv8G1O/fINnbu9MnQBnb9ms1L9jDLMXjOX+2rrtKwlle4xm27bVzHgimsBcW68H7V9ZzO6Y33i6UemtE8MwMAwLFnc6Qy9V1ho8MHkDezb+xAO1L/RNeFlaPTuLXdvW8lUvv7OvFkdduab+y9LlzcXs3raasTdF5N0hezRhxNyN7Nk0ljsqlv6rjLI9RrNt+wam31fdzYN3bjLe2Gpy26O9qZAwkzdGryTFLL5DqR8s7jKw8/fEUXwe40XLoUO52v8yLmgRESkV8r5us1HlhrdYtHkTmyY9QsvcbxTdxuVyT5td8ebZTa6Ti5PFnzrX3cOoz39i0V8r2R6zni1/zWHGt28yYlArKpbMz/YuKLfrVq8W/RlQ25OMnT/x4PVtqVOvMQ2vfo7Zx00wDCyGgcXq3ufrpc4r0pePe4QyrX84vw+KYOHAcGb3DeWbLgE8WMeLii6exG9Q8GBxzbqBfNcriFuDiiNFxbjmtTWsBg1rlP/vZ7hWHwLDqxIYXpVG7bpzx70bGfvsE7w2/yBZBdqzhcAq9ahZLglPnW8lxyhD/Tvf5cPDt3HX+N3FOPPPVdJZ80F/mn7g6nRcJiyhRIRZMLzqctcj1/Pz0CnEOs7fzFpzAE/2q4TVsBMUGoyVFOzFUleuqv8TLJ25kMQevYi+vgvlJ43nQC7l4BXVnW4VLaSvmsXsQ7lsIMXEPcabsm0Hc2tdg5gPv2R+cjFGrtUPUiJlkLWT8Z/P5473r+Wu3mOY/91+dFaLiEjpYqXctS/x7avXEbJzHHff/T7Li/UapYhc8ve0uSi2PLvHdXJxMfwbcdfb7/Fk+0hs2fLnGVyR+q0qUq9JWbbPXM6BdNelMffrVgsRNaoTYKSz+OsP+G1nMiaQHpdw+u0d3zGg9XcuSKtkZy1jo06A9b+lOgyDst5WanhbqRHhzQ21TvHy/OMsOlnSKTPZvCGR7hsK+jmDAD8PqpZ1FNvyI8U8hS+LmI/7Ua9eA66oF0XDNt3ofd8rfLnkIFmBjbn9vc94ppWfW01FLy1CQ0MZOXIkV3e+Gh8fnxI4oond9KftU+8zvHWA6qwUi27enCeeeJzo6GhsNhd9pecVSkSAQUbycazt7mZIlPf52xiBXHPvbTRMTybZYRAcHJSvdmfx8iMsIowgn9zzdrH3S9rJFTOZG+fAs2l3ulfObdaCNy2u70w5SxorZszPNcjvLtytbC8JRgBX9elMaMYaJv26B7ur0yNFwCR54S/MPGKjaZ/rqXmJTlYSEZFLlYWwDsP59s0biNw3iQfvfpulSa4LXBfs+vNyvKe9HPNcSJZy9Bn1MU93iMBIWMe4l++je6eW1GkQRaN2N3DTw2/x7XfTWXqs6Np90d3DGnh5e2KYp4iPP0Ep+ErJrURHR/P4E48R3bx5icRKdm5MoPOEI7SfcIQuk+K5848UZiSaePqX4eFGnuQSJblsFfvvzx2Z6WTYTUx7Oqnxe1n/+0Revas/t3+9jTSPqtzy9K3UOXPjZoR0ZMR3U1i8fBU7Yjayfc3vzPzsKfrULnt+J2utxbCpG/l7+xb+3r6F3Yufo7WHE/sppQyLhSuvjOLRRx7hhx8m8syzz9C6dWs8PYrrYQRZbBg3mlkJVbj1zZfoVf5id9setHnxT3bHTOGROtm3NQjoPYbt29fxbd/gs+vqhLS5m3c/n8DsBYvYuGE9u2LWsHb+eN69syW1r7yRp94dy4KlK9m+ZQ1r537HqEENz/vpk1G2Btc/8i5TFixj66Y1rJ3/PR/e34GKHtmO3eQmnn/vK6bPXcimjRvYtWk5y2e8RNdgKzXvm8TOrUt4o12OMixTmc73vcYPsxayedM6tqz8nbnjXqRXldIZcfD09qJjx4688MLzTJw4kQcfeIAGDRpisZTcchSW4FBCLA7iZn7CuF3l6H9fD8rnOLyt5s0MvcaLv8Z8yfJ0C0GhgWc6rNzryhLcjHven8zqNX+xctGfrFm7mg1z3+LGMzu+8Pu57TO3drmezUunMf6lATQNyqU38a5IpyEvM37GH2zcuJGdG1eyesEUfvrkZYZEB+be/5xazdS5h3F41KNn91yW0Sjbil5Xh2KkLuPX+fGYFKaPK8h5eeadi55XFy/7iytsH2AQ3Op/vPnJWGbOX3zm3F7JqjkT+OixnjQMzJ6OgpfBBcebfJTP6TJqzKBnP2HmwuVs27yGtfMmMHpoWyIuVkQ+0XRp5UvWpj/4/UiOby48y9P2juf5Zsp81q3fwK7Nq1m/aAZTv3mXl/rUOtOWCpLfouwHLYS2f5Y5Gzaxbtz/aJjrd6wl1fefSVG+6iD3MsjfOVeAPiNtPfOXJmOp0YmrS+lYIiIilyOD4DZP8M37N1H1yDQevetVfj967vVJ4e7JjHxf5zp3/VnQe9r85qlg975Fkf/8K648n5HrdbKVho9MZ9e2VXxyvV+2jS1EDvya7VsX80b77GtuWGn4yDR2bV3Mmx3OhOvKVOWa+99g0pzFbNm0lk2LpvDti4OIDstRvnmWZW65ys/1Kfi0vpfHOwZD/B+MGHA7z49fRMyhFNIz00mJ283KWd8y8r3ZF5xUVFTtuOD3sKfLBUsQ/b9Yf7Ze/t78LYPLWTBC+vLdpi1s+6Qn/tk/Ucjz9lLh7e1Np46deOHFF5g48XseeOB+GjRoUGyxEtMBmSaYJqSl29l58CRvLT3BTgcEh3lS2QC/0DIMaxfEVz3DmD0ggj8HhDPlen/+PVUMDxtXNwngs15hzB8QzoxewbzY0IvIHEm2eHvQq3kg3/QJZ8HAcGb0DObFRp6E5qi+qg2C+WNQGE+Xz/GGzUrbBgGMviGMOQPCmdc/jHFd/Lkm+ylu2Li9ewSLbzn9Z+GN/kQVUdG5ZnqceYzlo99g0rVfMrhmV7rX+YytW+yQFUj1qFpU/HeeuW8EdTsO5q0G4WT2fJzp8fn83qio9lNKWK1WWjSPplXLlqSnp7P8r+UsXLSYtWvXkJVVsEVZLsR+cBbDH/Oj2td3MPKdO9l+xxfEpBXFni0EN+pCjw71sjVID4IqNaX3U1/RO8fWnlWacdOznxB84kbu/fXI6Z9b+zTkga++4OGmfme/kfGu1JgeD35Ik/LD6PnsQpJMC+Gt+nJrt+zH8SMs3IP01DyS5lWHIZ99xdMtAv/7psczghpNKuF7yo2nwOaTj08ZOne5muu6Xsex48f5888/WbJkCTFbYor1uEZgMEEWk+S4FYz9ehEDX7uDO5tO45U1Z353Zfhz9d0DqRM/ndunbKfrXSbewSGUNSAjt9PXEkm/UaN5ssP/27vv8Ciq9YHj35ndTUIK6YD0ErpAAAlFehVBioKCFCt6VeRaEQuI2LBeLyoqAioo+gMFpEsP7dJbIECooZPey+7Ozu+PQAgxZTfZFOL7eR4fgZk9c86cmbPnvHP2TGUUayqx11JQPP3xqWIiM9FW+PY8V17O67oEj4AG3D3iTYIbVeL+MXOJuHGLuTVh3KzZTArxzbHmmAf+NRvhX7M+LvvnMnd396dYEAAAIABJREFUQh4zZ83sW7qC06P+RaMB99L8uwgOZ9+2Ct5dB9LDF+JX/MmGG7NaSquNs+e+Ugo7t/Yobhug4h98D0N75vy8kYC6wQx4qhW9+7XjhTFTWJM7+FtcdrU7oFTuxFvzvuTRhm7ZnVXX2sHcWzvrzwX92tDYuDWtPGxcPHjo1g6yWxPGffc9r7X34+aSdUa8q9ajZdV6NE5ey4eLI5wzU7vQdjB3j0TBu90EZv/nIaqf+J4nnptLWJ4/eSuttr94dQDYec850mZkcnj/USwPhNA22AvOJBSWAyGEEKKMqXi3m8DcGaNpHP8Xrz75Nquv5OppFHtMpkMlO75zC+3b58/hMa2dfQ1HzmOxy++g0i+zxomdu4l+ajitWjfBtGIPFgAq0fquZphUd4Jb18ew5VhWX1UNIDi4Fmr6FrYfyAS3Zjw9azYTQ7xv9jKrNqLbyNfp1LUVL49+neWXtULOZe482ds/daPjfb2popo5MOcT/jhfxHiKPX3HEhnDFoEz7tsKyN3dnT59etO/f/9SjZWgcMsDDv9qlRhax5TjvCv4uYPZAhhNPNLTl8cCley6c/U00auVD808Ehi3M5NEQHFxYXxvH4b5KNlpu3iZ6HE98FzockIGIyN6+PJMVfXmPWlQqBNgwN15IccCld2bv9IPsXlXIja1Bk0begCgJ2/nk7FD6NiuLUFNW9K0w0Aen32YDP8ePNA915IBWgQzBrekXuPm1GvcnAZd3mVHVovoWDoVhMFoQFEU3Nzc6NK1M1OmTGbBrwt4/vnnada8GYpT3nyrk7LvK174zz5swc/ynxfb4eXMk6knsfr1vrRq2ZKgFp0Z8OYqLmo6toQ9fDV+GB3bBtOoTR9Gz9xHEj50u78nVVQAA43GTmZ8sDtRoTN4/N67adK8Le2HvcXvZzRqDhnPqKAcDbuezLq3B3JX62CCWnaky/D/ssuSV4YM1Bs1mZdCvMmIWMpbY/rTpmUwTdv15J6xH/FXBXkIYjRmPU71rlyZgfcO4JOPP+aneT/x2GOPUrNmzRI5purjh4+ik5yYRNSaH/n9Ug2GPdY3+6mfofZQnuzjSdgvP7MzJYnEZB3V1w//fFosxas9fdt7YTvyHUM7duSurj1p2zaEDkM/ZVta4dsLlOO6bNC8PZ1HTWfdVRserUYxqs2NR9EGGoyewsshPpjPrODtMfcQ3KIlQS3a03lKKGmFXCra8eX8EWZGrdufwcE5VolSfOk1qDPe+jVWLd5O8o0slUobZ999Vaxzm1uR24Cbn181sSfNm7egwZ0d6PzQa3y/Nx5TncG8N7EXeU2Wt0ue3zf2tjtG7nxiEmODXEg6OJ8XhvWk+Z2tadVzFC/M3kNMgeMrBc+69ahm0Dh35nyOdZENNBw7lZfb+2I9v5YPnhpMu+CWNGjWhtYPfM0hp3YgHG0HVbzbPsecmY8TFDmPfz39JbuTCrkBSrztL04dXM+iI/ecXW2GTvLZc1yzmahbv5addSGEEEKUFQXXhqP46qsnaWHeyXtPv8nSvwX1nDMms+c7t3j9T0fGtA6WyRHFKH8RDlZyZc4nLmM+/D92p6gEtr2L+jd2NzWnQxt3FFTq3dXm5q/fPFrTobkJy5Gd7EpRCRozmRfbVSbj2O9MfDCr39a67zg+3HgFpXp/pk7sc2u/vtDxvQP9U0MtmjXyRNXOsmXbpSJPBHHGdVys69wWz8Ing7Prpd6djzLvSl6d3pKKpVQMOWMlA/rfW2KxEuX6mtdNa7jzaicPglRIiLFwPvsy1dm2K5ZBv0XR/ddoHlydyiEN6jfxYmygQuylFCYuj6bXgiiGrE5idaJOtfoeDPbJ+nTjZl7c76OQEpPGtNXR9F0QRf+lcUw7aibOjrBWrUaVebKqSmZCOp+ti2Hgr1H0XhjNI+uT2ZLzQZhu5ceV1+jyc9Z/3f5IYr+T5o+VXfAaK3FxieiKirune1ZGFAWfFiP4YO4fbN+1h8Ob5jG1X3UMGKl6R4D9mXVWOrcpg8GIooCHuzu9e/fkk48/5uf583jq6aeckLqZiPlvMG1jMkFj3uNNZz4M0DWSo6NIytTQzPGEL5nJL0c1FGMM4duPcTXFgiX1MttnzWFdoo6hVl1qq4ChMfcNbIIxaT3vvzqLTacTyLRmEBW2hHdmbCLF0JBOITnqXbcSf+kisWkWtMwkLkdeIzWvG9ZQn4H33Ymr5TAzJkzhl93nic+0kJF0jYgDEUTf/hOv/8ZgzPpiCvD3Z+jQoXz33bfMmvUdnTt3dupxXL29cVdtpCSnYss8yLyfD+DSfSwjGxoAN0LGPkxw2gZm/34OjTSSU3QUH7983pIN6HrWchqBTQhpEoCbAuiZRJ+9SIJux/aC5LgubdYULu1dwAfzj2A1+NOkSWDWdWWoz70DmuOiHefbF99i3u4LJJo1NHMK0bEpha81pp1n2eK9pKvVGXh/B278gk29ox8PdPLAFrmK3/fk+FYojTbO3vuqOOc2t6K2ATk+nxIXR5rVhs2SzKWDK/jw2an8GQ1+vQbT3duJT9vsPT+GxvTrXRc1cx9fvPwxf4ZdI81iJunSQZb/vJZTBfaIVfwD/FBtacTGpt28jgyNGDSoGS7WcL4eP5HvQ08Rk65h0zJJik0g3ZnP1RxqBxX8OvybH797iqYXf+GZcZ/ZtwZmSbf9xaqDG0Vz4J6zp80A9LgY4vSsOhZCCCHKNwMNBwynow/EHdrAjrzeUuesMZk937nF7n/aOaZ1tEyOKE75i6SUy5y2l0170jA0aE/761FqQ8P2dAhI4ujRS6h3tifkegTdtVVH2nloHN2yg2ilIYMGN8fFEsaXL09j0aGsfltC5A5mvfI2C6/o+HYfRK+c0esCx/cO9k8VD7w8FLDFE5dQjAG/M65jZ46z8lNSsZQKyGjKmvucO1ZSu3btIqfZKNif0NFV2TKqCmuGBTCrhxcD/RS05Ay+PJxJdgRA10lM1Yi36miajWvJGmmKiV51TRjMGXy9PZX/Jdow23RiY9P57+FM0lQjbasaUBUTXWsZUTUzc7clsy7WRrpNJyXFwoYTmUQWVn+KkV71TLhoFn7cksTSaxqJmk6m2cbZaKtdwW9nKMO3ahnx8/NG0W2kpaahK5Xp+taPzB5ZB1N2O+RK7VoAGqpqZ1adlU4hOnfpzMouK5ySVkm68aTIx9eXwYMGZf+7t7d30RPVLrP47Xfo1Ow/DHvndULDJpNa3IzmeZwrRF62QtNAqvqokHb9y8N8lYtROkoVdyopgKkW9WuqqJX68eXufnz594SoXvMOVGIcO76xDg3rGrBd2M2O8857TdrrkybBJKclV2IMhqxAdo0aNahRowYAOuDn61vstL28vVB1M6mpZsDGhaU/89fTn/PwmI78MMOfxwdV48Ki11ifoIOSSkqqDbVOZSrnE3vUk3ewZEMMPQZ0441563k5PpIjB/cRuuxn5q45SWph2x1qcDUunz5Hqt4cT8/ra5ZlXys72HyqKI+ebVxb8zsbXuzAgD5D6fXxVpYnqNS/byjtXK0cXbKEIzcmt5RSG4eLffeV4tRzm/sQdrYBBdATd7Bhv5khvWvToIYKzlqdwc7zo5oCqVtDxXZxP3vznO1QyGHcXFAwY875Wy6X2jSoqWZdb6dLeKqDI+2g6kPvJx8BWwIbf/2F/8UWscPv7La/mHVQ/HsujzYD0M1mLDq4uJbUO7mFEEIIZ7FyfMHHrKnxKM92f5NF82rz4vjP2HQtR9/A3r5RQWMyO79zC+3729P/tGdMa3eZ4uw4oB1Kup9fmmXWE9m+8QAZPdvSrYMP8xcnUqtTJ+ql72LizHgmzriHrm0rsXSjmRZdOuBni2De5otoLr0Jqqliu7CL7edy9T1T97P1QAYj76lDUC0Vu067o/1TPY3UdED1xsdbhagixAGcdR2X5DjrBmfct060cmX5j7HBrbGSnKq4WIgyO/4eOptNJ91s40qihUOXMlh6MpNzhQ3zDAZqeYJqdGPqg25MzWOXKp4qqqpSwwNsKRYOFyVwpxqoWxlsKWb2Jxe+e0kpu+B1pVZ0b++Naovk+MlU8BvMo0NrY4jfxVeTP+aXnaeJTjcS0OtNln4xqPD0rlP8ejslncIcP36cJUuXOi09R1X2rsxzzzxr176apmEwGIiKiqJKlSoAJCYmFuv4esxG3p3yB22/e4Cpb27jk/Q89sEGuOLmVtTZjjYsZisoJkymnGlYsFh1UJRbnljmT8G1kqvjM8QVNWvtYt25j5KWLF3K8ePHnZqmI5o0acLQIUPs2lez2VAVhWvXrlGtWjUUIC4+vth58KrshaJnkJaRdW71pFB+WBzJwFGP8oruRzeXg3z46+GstZf0dNIydBQ3L7xMQF6NuB7DyjfGkHJgOP07tKJN6xa06VGPtt170ES9n/ErC9vuWJl0sxmzrqDcWNxaNWFUAau16D8tSwxlwcor3DuqCyPuvYOVv1fhwfubYEzbwYKl57LTLW4bZ/d9ae99Zce5L/odZGcbUHBB0G161v+z/6W4bRP2nx/FcP2XRWqRfqVizjCj44JLzvimnlUCNJtd57ZY5XWkHdRT2L9uHwHdu9Jjyhw+TXucl1cU5eeWTm77i1kHzuhX/K3NIGvtOZMC5sxCV5kTQgghypw1aidfvbuS7f/6nG/Hj+WbeT68+sQUll+8PsPCCWMyu79zndT/LHRM60CZnNG/LI1YhjPLXMiRiN22if2ZdxPSowPef+6jS9fGWPb+xpYd8XSMf5CuXVvhGppEjy53wOllbDirgYuTF3l1tH+qXeLk2XT0xvXo0C6QmSev4ujUB2dex84cw+appGIpRfTh9OmldKS8NW3alCGDB9u1r6ZpqKpKamoqnp6eAA4HriMOxjLuiNXhawwAnULbOVeDgpJjzFy0X27cXCe7LCfZl03wWvGmw/iJDK+hYo34i5XHNNSgalRzgbR18/ly/fHrC4ZbiI1OyvUiJR2r1YqOO+7uf7+F1AB70ymemOgYtm3d5sQUHRNYpQo8k/92q9WC0WgiKSmJzaGb2bp1G8fCj7FixXIn5UAnYdvnvPlrCD+OfJUXrrmjkJRju43kxBR0tQaNg7xRDsaW3IVuucS5SzZs3n/yZN/JbMp3/ScH1yO7nq5aO4SOtQyE5X7yW0THjx8v02unMBarFZPRyJXLV9i4aRObN2+mfoP6WTPGnaRyZU8UPZO07PUNLBz57Vf2jHmDRx7SiV/9Kksv3mjCzaRn6OiKB16eCuRXvxkXCJ3/OaHzAYMXTR6YxtypfejeLwT3latILXD7X8UrkOUql2NsqLXvIqS6ypELRfn6yWDP/y3h+IjnCBk5jA7xNbi/tkLssoWsirp59zjexhkwZLf0DtyXdt9XFH7uHTwTTlWpBSF3uoA5qzyAA21TAd839p4fQzNOX7Sh1u1E9wZfExbhyExpG7ExcdjURvj7u6OQmJVXy0XOXrKh1mpD22oqRy4VdL0Vsy12pB3ULZxa9BJPL3yBeV+OZtB7X3Dl2uN8vCe5ZNr/UqmDkutXKH4B+ClZdSyEEELcFmwJ7J35DA9dfY850wbx+TwXjI9OYsl5q1PGZA595zql/1nImNaBMjlj7Fu8fr69nFVmY4FxGQDbtY2s2PsKHTv0oVsDL/q01Nnz/nbi09JYvz2JB7r1pN3SJHrX1Tnx1VoiNMAcmdVvq9Oeu+sYCDuTo+/p0YYurd3AfJ4zF/N6aXhexXW0f5rG/zbsJKlfLzqMm0DfdW+xxq71Qm/WhVOv45Icw0LJxVKKqKzjJKqiQgGx67xiJY+MHUvnLs5dZtUuNo1LqWBzSWfSn0n8L7/3HikmzqeAWtmFDt4Kxx1dc+b6cVRPF9p4wYmkvHbSseo6oFCphKLMJb78s2o0YVAAgwseAXVo1WMEb85eyA9PNMXNco4F0+dxTANbbBRRFqjU/n5Gta2Op1EB1YSnp1uuCLtO1NVodMMd9Bneh3qeRgxufjS4qznVDY6kU/Fo1qyrNSMjg21bt/POO+8yevQYvvt2FuFHw9GdPIMYPZkdX0xjwQVvqld3y/U0TuNM2DGSdFc6/+t1Hm5dFXeDisHNi0DfSs59cqed4K91Z7AF3MfUj5+gV7NqVHYxoBrc8K3ZnO7t6hSt7rUTrFl7Gs3Uin9/OY3R7evi62bAYPKkWuNgGuf39sDb0I1rJyEhgVWrVvHqxIk8OW4cCxYs4PLly04/nrunOwrpZGTevCZtl1cyf2MCNu0yfy7YlOMN1jYy0jLQVS8qe+Zzzg316PlAT1rWqIyLqmAwGbEmJ5MJKAoohW0vboGsx1i38Qo219b8+7NXua95VTxdXPGt0477+zTF3kUBtFOL+WVnOoagEXzxVl/89PMs+W0bOX+d40gbZ7ZY0RUf2nTvQE13Aw7dl/beVyV9bh2heBAy7GF6NPSnktFE5VrteOSDdxhZUyV11zq2JuqOnYOCvm+w8/xoJ1i+4hgWYzOe++ojxnVpgJ9b1n7eAb7k09fPPn7KuXNc0wzUrV/75he2dpJ1G85hc23LC5+9wsBmgbgbTXhVb8Wg0X1peEvfsphtsaPtoK4Rs/VjHnlpMZGmpoz7ZDL3BJZQW2nvNVqsOiipfoWCV926VFUtnDtzscipCCGEEKUvk9OLX2fs66u5Vu0eps96k57+ilPGZHZ/5zqz/1nQmNbuMjln7Fu8fn5pl7nguEzWLtGsX7mbdM/OPD51OO3Yx5rNseiksWPNVhKr9ubFSQNooB9nxZozWbOhtQiWLQvHbGrB859PZljLqrgbXfCu04mnPnmHB+9QSAhdznpHFtp1qH+qE7/mG2YfzUStPogvfpvJq0Pb0SDAHaNqwMWrCo3aD+RfLw6l6fVy5q4Lp13HpTHOKqlYSgWSHSuJjy+VWInddAuhF6zYKrnxwt0e3O1nwNMAqqLg7WmiQxVDVt3pFtafs2BVTYzpVpkR1Y34XN/Pq5KKmx3H2XzeimYw8VjXygypasDbAKqqEOhrov71BGLTbNgUA52D3KhlAoNBpU4VE1WdFBAo4evQSLPxf3BifO5/19ESwvjprZd5f0dS1hOv2I0s3DCeLgN6MmVBT6bcsr9GRI4/nw/dSPiEFrS8/1M23n/9ny0H+eDeMXx/wd50KoYbAWmr1crOnbvYtGkT+/fvx2IpnVe+6sm7+fz9JfT8dhi537WaunU+84/15vnm/Xnvt/68d8tWZ/5M2sqROe8zt8c3jOvzErP7vHTLVsuBj+nz8E9EOjwZ1srROe/xXdeZPHvnEN6dN4R3b2zSU1k+oSsT1mYUlEC5dmM5mZTUVEI3bWZzaCjHjh1z/kOOPLh7VLo+8zrHP+qJrH6pMw1eyr23TnpGJrriQWXPvNNT/NvzxDuT6ZT7Vzq2OFav3UOaf68Ctxd/ZnAGe2Z9yrJenzKk1VhmLB6ba3t+j0Fz5+cay+f/xb87DaVqgE7G3t/4+dCt94ruQFt58dhxEvQmNBn7DX9VeYV2/17jwH1p3311vpBzX6qzrhUX6t4zkbn3TLw1Kwk7mf7pCm5MYLf/HBT8fTPbrnZHI+KnqXza6XsmhfTjjdn9eCNXtguavWuN2M/B1DH0a9WSqmoYl20AFsLmfMQvvWcwpvUjfLnkkb99LmeaxWuL7WkHc3/f2Ije+AHPflGXhS/35713dxL27GIuOv0lt/a2/cWrA/vvOUe40qJNM0zaafYfynP6ghBCCFGOWTm/fDJPVa3Cry8P4/PPzzD8yfnFHpPZ+51bWN/f0f5n/mNa+8eZzhj7FrefX9APJZ1f5kLiMudtgE7s+iVsmNiFQW2bkBY6hQ0xWR3y1J1r2BA/kOGtFdJ3/sSy7F/3aZycP40vus7m1XbD+WTRcD65mWssl1Yz9aO/ivCSOAf6p5bjfDPhNarMfJ9RTbvw7PQu/G3BVutR1D+XcexMHnXxgnOu45Ifw0LJxVJub5pmw2BQSUlJYfOmzYSGhnLs+PFSiZU4IuJoMotq+DCilifTa90aLLFEJzNmbRqXdDh7PInv7/DlX1XdeK6nG8/lSqewFupkeDK/VvdhtH8lXu5TiZezt+hs2BLN1PM6ly5lcrKliaYNvFnQ4Po79mwWvl4ex29OWCu7xKaOatGnCDt9hdjkDCyajs2STlLMBY7sWM2Pn7zEoH6jeGfdpZshHT2O1W+N4+U5mzhyOYlMTcOamUp81EUiDu1i56nE7J91aCd/YsKrP7DpZAxpmoY1LZYzB04RrSgOpXO70zSN/fv28+mnnzFixEimT5/Orl27Si1wnUUncet/+WBV1N/X6ck8woynnub9P/ZwNi4DzaZhzUgm5tJJ9m9ZTeipdKfVhZ68h+mjH+aFmSvYeTKKpAwNzZJKTORhQvdeKHKoXE/Zx2ePjGbCzFXsPRdLqlnDkhbHhfB9nE4yle6sUidKT89g8+ZQpkyZwsMjH2bmN98QHl4Cs/Pz4eFuQNHTScu053g66enpoHhS2SvvJktRrrB/82Ei49Kx2mxo6fGcP7yeWa89wasroqGQ7c4otS16HRNHPcPHi/dwJjYDqzWD2DO7+XN9OGk62HT7vvFTti1g0Wkrui2B9fOX8bcVSBxo49JCv+Clr9dz5Goyly5eyboPHLgv7bmvCjv3pdre6qkcWrWYrSejSbNayUi8yKG/ZvH8yPH8cDJHu+jAOSjo+8budif9GN+Pe4jHPv2dbSeukZSpoVkzSI65QPju9fwReibPpdwBSN3Nhp0pGFv0oEeVm9e/nridaWOeYOqvO4iISsFszSTxUhhr/gjlbO6VPYrZFhetHczg2Ny3+HhHKj7dXmLyfVVLpMNRKnVQEv0Kt1b0vtsX/XQoG520JJUQQghRujIInzuRt9fF4tX+BT5/pjmuxR2T2fmd6/z+Z/5jWrv7Gs4Y+xa3n1/KZS4wLnMjraSt/LbyCpqewtblm4jJLsAu/lwXhWZLZvPC1dcnaFyXHs634x7m2S9XsjcyjnSLmdSoCLb8+iGjH5rEsstF7TvZ3z/VLq9nyoj7eeS9+azZf5aopAw0TSM96RqnD23l9+9/ZXt8Vqb/VhdOuo5LYwwLJRdLuV1lxUo2M2XKFEaOfJhvvv2W8FKa5Oco3WLmm7VxTAvL4ECCjRQNNJtOXLKFXVHazfGN1cqvG+N4dX86e+I1UrSsl0SmpmucvJbJ6kvWAqfb6RYz36+P452wDA4n2UjTwGK1cSXOTKQ561cAtoQ0pm1P5X8JNjJ0sFptnI+2Out1tije/oG31MCNt3vu3r2HGV/NdNJhKo6f5/0AZK3FU5aLybuYTLhVciMpybFHGFK/ZSMkpB0Txmc9r/1w+vQyXcvJy8uLzIwMzA485OjcpXP2mtczvprJ7t17Sip7FYxC4IOz2DrtLnZN7sWji+IqzMOz8sVAw2d+Y9WEO1j8VA9e21qaD/BKnmfP99n49QCufPEA9393usAXIKp3PMwv696k9caXCZ6whtv3tyEVmYJPv49Y/0Vvzn48hBE/nC/yi15zmzD+WUJC2gEwYMBAJ6UqhBDidiLjPSHE7aQ89V+LEit5fdKk7DWvJ52sU1JZ+0eY3jASyCPeqrCo4iza+w9jtlgcDlwLAZCcnOxQYyzso1Zpy4A+bWlU3Q9PFwNG9wAadXmM959pj4vtHAfCKs6vPkTpStnyEz8fh+ajn6SXz+36ew+RzdiQ0eN64xO3ltmLLzgtcC2EEEIIIYQoOomVlF//9LXXhRDCKVyDRzB9xr145o4t6hqXV3zDgggJUYkisp7kx8+WMGzWA0wav5T/vb+LZHkScpsyUG/Ea4xrbmHX+zNZnygVKYQQQgghhBAFkeC1EEIUm4JLwgk27a5PcMPaVPN2hcxErpwJY+uyn/jql11E/cNeciGcSSdp+3+Z/EsdxsTruCpI8Pq2ZcKUepHw9euZ/JvzlgsRQgghhBBCiIpKgtdCCFFsOom7ZzNh7Oyyzsg/lMbJb4bT8JuyzkcJ0hMIff9xQgvZzXZlASPvXFAqWRJFkUHEkrcZuaSs8yGEEEIIIYQQtwcJXgshhBBCiDI3eMhgmjVpWtbZEEIIAMKPH+PPpX+WdTaEEEKIfzwJXgshhBBCiDLXrEnT7Le1CyFEefAnZRe8vvvuTri7uxMefoxLly6VWT6EEEKIsibBayGEEEIIIYQQohypVas2Y8aMBiA5OYUjR8MICztC+NFwzpw5g6bJmxOEEEL8M0jwWgghhBBClCujxz5W1lkQQvxD/Tzvh7LOAgCxsTHYdB1VUfDy8qRD+w60bxeCajBgsVg4fuI4YYfDOHL0KMePnyAzI6OssyyEEEKUCAleCyGEEEIIp3J1dcWm2bBYLWWdFSGEuC3FxsaiKkr23xVFQTEYADCZTLRo3oKmTZrysNGIzWbjXOQ5Dh88XFbZFUIIIUqMWtYZEEIIIYQQFUtQUBA//fQDw4cPx8PTs6yzI4QQt52Y2NiCd1DAaMyai6aqKvXr1WfI0CHZm318vEsye0IIIUSpkZnXQgghhBAVzIB77yUmNpbExARiYmJJTEzEYim9WdB+fn5U9vZmzJjRjBw5gpUrV7F06VJiCwvGCCHEP4y7uzsBAQH4+/vh5+9PYEAgfn6+VKta1e40bLoNBYWoqCiqXv9cQkJiSWVZCCGEKFUSvBZCCCGEqGDGPjIWz1wznhOTEkmISyAuPp74uDji4uOJi48jIS6e2Lg4EhISiImJIcMJ66b6+vmiaRpGoxGDwcCgQYMYMmQwW0K38H+LFnI+8nyxjyGEEOWZwWDAx8eHwMAAfP38CPQPwM/fD3//AAIC/PH186VKQCCubm7ZnzFbLMTGxhAXG0d0dDSa1YrBmP+Q3abZUFSFSxcvsXDRIkI3h7Js2Z+lUTwhhBCi1EjwWgghhBCignnooRGYjCa8Knvh5+eHn58/nl4e+PlMaa1MAAAU4ElEQVT6ZQVPfP1o1qwpfn5+BAYGYri+jipkBU9SkpOJi4vL/i829saf44mLiyXuerDbZrPleXx/Pz909Oy/G41Z6Xfu0plu3buxf99+Fi9ZwsGDB0v2RAghRAlwcXHJalv9/a63sX74+/lRrWq17H/L3bampKRkt6kxMbFEREQQe/3vV69czbNdbd68OYGBgX87vlXTMBoMnDx1kt9+W8iePbvRdf1v+wkhhBAVgQSvhRBCCCEqIIvVkh0ogVP57qeqKj4+Pvj4+ODv54ePrw/+/gFZf/f3o3bt2rQObo2vny8uLi7Zn7NarSQmJhIbG0tCQjxxcfHExsaRkBBP3Tp1MaiGvx3rxvqswa1b0fautpw9e47FSxazedNmZxdfCCGKxd/fj/vuG4ivrx+BAQH4+fvj5+9HlcBA3HLMlrZYLcTGxGbNlo6NIeJEBFEx0cTHxRMTG0NsbCxxMXFFeoFtdHTMLcFrTdMwGAyEHz3K/HnzCT92zCllFUIIIcozCV4LIYQQQvyD2Wy27CD3mTNnCtw3r9mGnh6eWWu1+vkRFBSEn58fNl1HVfN/L7jBkNUFrVunNi+/9BJjRo8hJTnZqeUSQojiaNqsGQ0aNiQuNparV7NmRp86dbLQ2dLOFBV1jabNmmKzWlEMBrZt38bC/1vIuXORJXI8IYQQojyS4LUQQgghhLCL2Wzm6tWrXL16tcD9Zn33nV3pKdcD3FWqBFKlys3ZhQaDAU3Tip5RIYQoph3bd/D+Bx+UaR5iYmLRrFbWrlvLH78vLrTtFUIIISqifIPXQUFBTBj/bGnmRZQiqd/S5evrW9ZZcJr+/frSIaRdWWdDCCFELkFBQWWdhWzePj6F7mO1WjEajei6zvnI8xhNRmrUqAEggWshRJkrqdnUjggNDWXp0qXEx8c7/FkZ7wkhbgflqf9aXKOqRZd1FiqsfIPXfn6+hEiAqsKS+hVF1bBhxflyEUII4XxGoxEPD/e//bvVar2+XIhOZGQkhw4fJjw8nIMHDpKSksLrkyZlB6+FEEJQ6FJOBZHxnhBClK4WXmllnYUKS5YNEUIIIYQQTuPj44OiKNhsNlRVRdOsnDx5ioMHD3LkyBHCjx0nMyOjrLMphBBCCCGEuA38LXg9YMDAssiHKCVSv6Iotm3dxoCtcu0IIYQonKenB4cOHebIkTAOHz5CxInjmC2Wss6WEEL8I8h4TwghSs+H06fD9LLORcWX/2vghRBCCCGEcNC5c5G88cYbLFjwK0eOhEng+jZnqNqJZz+Zz6b/7eNk+AHCti5hev8AlLLOWIVhoM79H7Js7WLeCLmdfxSrUn3Qeyxbt4J3OpsK2beilFkIIYQQpUGC10IIIYQQ4h/OlTYT/o/9e1fxUV//Eg7MluaxismlOf/+7iteGdSGun5uGA0ueAZWwzUzBb2s81ZqSr6+vGo1o1ktH9yUsr4ailNWBY8ajWla0wc3Oz5YfsoshBBCiPJOHnULIYQQQogKx6398/z45gDqVPXD16sSJqxkpCRw7XwE+7eu5Kd5KwmL17L3VxQFRVFRSyGWVprHKg7X9g8ysrEL5pMLeeXFGaw7k4JLYHU8UzLLOmul6napL2f4J5VVCCGEELcHCV4LIYQQQogKx1ClEcGNa+Ga/S8uuHtXoV6LKtRrcTeDh3ThpYdfY/kVG5DJvv8+SOv/lkbOSvNYxaFSNagB3komW+f+l5UnE9CBzKuRJJd11krV7VJfzvBPKqsQQgghbheybIgQQgghhKigrBz6YghNm95J/WZtaXX3PQz614f8fjwNQ/V+jH+wMYayzmK5peDq5oKipxMTk/oPWiZE3C5UVy8Cqwbi6y7zsYQQQoiKTILXQgghhBCigtLRMjMw23R0LYOkmAuEbfqZt7/bQYYOXpU9r6/ra6DhM4s4eWwbH3W58bI5Bf+7n+LzWb+wZsMWDh86yKnwgxzZvoyf3xlJa9+c6yo4sm9xj3WdW016jHuXn1ds4vDhw5w8vJu9G5aw8Jt3GRfiU/B6xZXq0ve5j1j011aOhu0nbMsSfpw6ipDA3KF8BVRfHvz+IGdPHM3678iPjL0j9xDCkfybuHvqZk6HL+HFJoZb0vAeOpMTJw7w4zC/6/nPK9197F//M58/3oHGbR/gtc/nsWH7bk4c3cf+tT8xfVQLfHIVXvEIYuCLn7Nkww6Ohe1j//oFzHiuGzVNOY4d/BBT/jOH5WtDCTt8iFNhO9m54h36++VVXzfOY216P/MBv60O5UjYAY7u3sja+VMZUierXIp/d974aQlbd+4hIvwwJ/ZtZNV3r3F/Yw8H1pM2ctfrf3Hq+E6+7u+Vq2D+PDRnH2fCfmDsHaqdx3O8rA6VQ3Gh0ZC3+OHPjYSFHSR85xp+//w57qlbya7SFl5XoPrdxdNfLGbvvv+xe8tm9u3fy6G1n/BAdRnaCiGEEBWRPKYWQgghhBAVn2rEzcOHms2689iTHXG1xbB163G0/D+AX8s+3Net2S0dZo+ABtw94k2CG1Xi/jFzibA6um9xjwW4NWHcrNlMCvHNsTaxB/41G+Ffsz4u++cyd3dC3mVza8bTs2YzMcT75iyWqo3oNvJ1OnVtxcujX2f55fzPilPyX6x0TfjWas3Q1+YwNNfeLnXu4qG3vsEv9QH+tfQaNgD3Foyf8z0vtPbKLq9brVbc9/wMgqtPYPBbocTrKlU6DmPMvTmP40VgFROZKflkzbUJ476bw6T2PjfPo0tVgoJr4Zluy/q71YcGbRpR0+X6ds+qNO0+lk/urIJl8Cssj7FnPruVsC3/I2bsA7Tr1ALX1TvIXnHcoy2dW7qiHdvG1igbeNpzvCKU1ZFyKB4EDxx28+8utWg74FladwrmndHPMu+UJf+i2lNXSjWGT/+Sid0qo1hTib2WguLpj08VE5mJNjvOpxBCCCFuN/J4WgghhBBCVFAm2ry2htMnjnL22CGO7Q1l3bx3GFk/ij8nP807m5MLXw5DT2L1631p1bIlDZq3p/Oo6ay7asOj1ShGtTEVfd8iH8tAg9FTeDnEB/OZFbw95h6CW7QkqEV7Ok8JJa3AAhkIGjOZF9tVJuPY70x8sCfN72xN677j+HDjFZTq/Zk6sQ+3TJS2xbPwyWDqNW6e9d+djzLvSj5BwuKW347zEtSiMwPeXMVFTceWsIevxg+jY9tgGrXpw+iZ+0jCh27396SKmlXeRmMnMz7YnajQGTx+7900ad6W9sPe4vczGjWHjGdUUI7Z33oy694eyF2tgwlq2ZEuw//LrjxjrQbqjZrMSyHeZEQs5a0x/WnTMpim7Xpyz9iP+Ot6MFdP3s4nY4fQsV1bgpq2pGmHgTw++zAZ/j14oLuv3bOvMw9sYUci+HXsQoscTwcqtelCB08bp7dt47zm4PHsLquj6Zo5u+ojHr+vG83ubEPrvk8wbXUkVp+OvPrKQKrkW2j76krxak/f9l7YjnzH0I4duatrT9q2DaHD0E/ZlmbnCRVCCCHEbUWC10IIIYQQ4h9FqVSPfk8/x4iWXoUHEHWN5OgokjI1bNYULu1dwAfzj2A1+NOkSeCtnWlH9i3qsQz1uXdAc1y043z74lvM232BRLOGZk4hOjal4GC8oSGDBjfHxRLGly9PY9Gha6RZzCRE7mDWK2+z8IqOb/dB9MprmRJ7FLf8dqSrmeMJXzKTX45qKMYYwrcf42qKBUvqZbbPmsO6RB1DrbrUVgFDY+4b2ARj0nref3UWm04nkGnNICpsCe/M2ESKoSGdQgJu5ku3En/pIrFpFrTMJC5HXiM1rxNqqM/A++7E1XKYGROm8Mvu88RnWshIukbEgQiib8T2FQWfFiP4YO4fbN+1h8Ob5jG1X3UMGKl6R4D95yNtF6u3JKBU70KvZjeC7a606Xk3vvpp1qw9lTXL3pHj2VtWh9NNZc/iX9kUEUO6JZOEyJ388PpUfrtkw6NDH7rkXtMl+5zaWVe6jg4ogU0IaRKAmwLomUSfvUiCLMwuhBBCVEiybIgQQgghhKigLOz/6D6Gz72ADQXVxR3/6o25+4HxvPFEb974bxwR905jW7ojaWpcPn2OVL05np6FrV3syL52ft5Yh4Z1Ddgu7GBzQUsw5MWlDkE1VWwXdrH9XK6lQVL3s/VABiPvqUNQLRXiHM6sffl3SrJXiLxshaaBVPVRIe16tNh8lYtROkoVdyopgKkW9WuqqJX68eXufnyZR/6q17wDlRjHjp9dB7vZcT6fJVaUynR960dmj6yDKbvgrtSulXVcVXVkGJbK9hWbiL1vCL17N+HTw0fRXFrSp1sA+vH/Y+VJzcnHc3I50g+zK8zMmL41qFtdhfg89nGxr66U5B0s2RBDjwHdeGPeel6Oj+TIwX2ELvuZuWtO5h+AF0IIIcRtS2ZeCyGEEEKIfwAdmzmV6HP7Wfr5JGbstmCo2p7OjXK/pNCOlMxmzLqCohYejnVkX7s+r5owqoDVWsB63flxWvjYbnmVX8cGuOLmVtT82LCYraCYMJlypmHBYtVBUbIGOddn6eZPwbWSq+NnRVGz1hrX809d8evNo0NrY4jfxVfPPUDHtsEENbuLDs8v4arjFUfa7lWsuQr1+t1DSyO4tr2HvlU1Dq1YzRnN+cdzbjlu1L+a/7m2t670GFa+MYbH35vD/23Yz3m9Om16DOOlz+fw8b0B9hdMCCGEELcNmXkthBBCCCH+WRQTLi5ZwTRVUaDwla/LD8tVLsfYUGvfRUh1lSMXHHhJnTmS0xdtqHXac3cdA2FnckQfPdrQpbUbmM9z5qKNkpvjYiM5MQVdrUHjIG+Ug7Eld/Ytlzh3yYbN+0+e7DuZTfmuiezgA4zr6aq1Q+hYy0BY7lnsgBpQjWoukLZuPl+uP44564PERifdfOFirjwYChqZZexl0bJzPDyuH4PbzMVnYC+qZOzki+UX0ACDw8ezj+Pl+DvFuxO92riAOZIzl3JeWznKbHddARkXCJ3/OaHzAYMXTR6YxtypfejeLwRWripSOYUQQghRfsnMayGEEEIIUUEpGFwr4aICqolKXv7UadGDJz74khdbm7AlHGDPKWtZZ9Ix1mOs23gFm2tr/v3Zq9zXvCqeLq741mnH/X2a4lLQZ7UIli0Lx2xqwfOfT2ZYy6q4G13wrtOJpz55hwfvUEgIXc76uJIM5mucCTtGku5K53+9zsOtq+JuUDG4eRHoW8m5c8O1E/y17gy2gPuY+vET9GpWjcouBlSDG741m9O9XZ2izeTRTrBm7Wk0Uyv+/eU0Rrevi6+bAYPJk2qNg2nsr2KLjSLKApXa38+ottXxNCqgmvD0dPvbMc0WK7riQ5vuHajpnl8g3Ur44sUc1O5g4GOTeKSvH4kb/2DN9ZdDOnI8RzicrmLAK8AfD5OKavTkjpYDeP3rqQwOhLiNy9mUqOddZnvrylCPng/0pGWNyrioCgaTEWtyMpmAUvo/LBBCCCFEKZCZ10IIIYQQooIy0uqFJRx74e9bdPNFVnz4NRtTSj9XxZPBnlmfsqzXpwxpNZYZi8fm2l5QMF7j5PxpfNF1Nq+2G84ni4bzSfY2Hcul1Uz96C9KNHYNpG6dz/xjvXm+eX/e+60/792y1ezEI1k5Mud95vb4hnF9XmJ2n5du2Wo58DF9Hv6JSAcmr99I9+ic9/iu60yevXMI784bwrs3NumpLJ/QlQnrNrJww3i6DOjJlAU9mXLL5zUicvz54rHjJOhNaDL2G/6q8grt/r2GvCYea+eX8XPo03zWZyDdtEhmL9hK0vW60mPtPZ5jHE5XqUz/6RvoP/2WVMiM/JPJH60jXs+/zPbU1Xn/9jzxzmQ6mXId1xbH6rV7ilhKIYQQQpRnMvNaCCGEEEJUOFr0KcJOXyE2OQOLpqPbNDJT4rgYsZc1C/7Dc8OG88LyS0VYN7rs2aLXMXHUM3y8eA9nYjOwWjOIPbObP9eHk6aDTS8gGpsezrfjHubZL1eyNzKOdIuZ1KgItvz6IaMfmsSyy6VwRjKPMOOpp3n/jz2cjctAs2lYM5KJuXSS/VtWE3oq3WlLiejJe5g++mFemLmCnSejSMrQ0CypxEQeJnTvhSKHyvWUfXz2yGgmzFzF3nOxpJo1LGlxXAjfx+kkE4oex+q3xvHynE0cuZxEpqZhzUwlPuoiEYd2sfNUYnYZ00K/4KWv13PkajKXLl7JP096HH/9vJLLmk7m4YX8cijzlm32Hs+xgtqbro3YQ2tZvuUQJy/Hk2bW0KzpxJ0/zJo5U3jwocmsvnbzusyrzPbUlaJcYf/mw0TGpWO12dDS4zl/eD2zXnuCV1dEF6WEQgghhCjnFG//wNtokT8hhBBCCFERvT5pEp27dAZg9NjHyjg3tyOFwAdnsXXaXeya3ItHF8XdTit5C1Fu/DzvBwC2bd3Gh9OnF7K3EEIIIUqUwiJZNkQIIYQQQojbiFqlLf1bwcmjZ7kck0iG0Zf6bQfxyjPtcbGd5kBYEWfZCiGEEEIIUc5I8FoIIYQQQojbiGvwCKbPuBfP3C+o0zUur/iGBRG342IoQgghhBBC/J0Er4UQQgghhLhtKLgknGDT7voEN6xNNW9XyEzkypkwti77ia9+2UWUwy8gFEIIIYQQonyS4LUQQgghhBC3DZ3E3bOZMHZ2WWdECCGEEEKIEqeWdQaEEEIIIYQQQgghhBBCiNwkeC2EEEIIIYQQQgghhBCi3JHgtRBCCCGEEEIIIYQQQohyR4LXQgghhBBCCCGEEEIIIcodCV4LIYQQQgghhBBCCCGEKHckeC2EEEIIIYQQQgghhBCi3JHgtRBCCCGEEEIIIYQQQohyR4LXQgghhBBCCCGEEEIIIcodCV4LIYQQQgghhBBCCCGEKHckeC2EEEIIIYQQQgghhBCi3JHgtRBCCCGEEEIIIYQQQohyR4LXQgghhBBCCCGEEEIIIcodY1lnQAghhBBCiJwmjH+2rLMghBBCCCGEKAckeC2EEEIIIcqVkJB2ZZ0FIYQQQgghRDkgy4YIIYQQQgghhBBCCCGEKHcUb/9AvawzIYQQQgghhBBCCCGEEEJkU1gkM6+FEEIIIYQQQgghhBBClDsSvBZCCCGEEEIIIYQQQghR7kjwWgghhBBCCCGEEEIIIUS58/859fmlysXg/gAAAABJRU5ErkJggg== ) ## Retrieve blueprints from a personal blueprint repository ``` for bp in w.list(limit=3): bp.show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABa8AAADECAYAAACY9t2uAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdZ3gUVR+G8Xt2N4WQ3ukgvUOE0JuCCogUAQUUyysWVOwF7NiwFxR7o4iKghTpqFTpndBReghpkABpu/N+ADGEBJJN2Q08v+tCZHd25rQ5Z+a/Z88YASFhJiIiIiIiIiIiIiIi7sJgksXVaRARERERERERERERyUnBaxERERERERERERFxOwpei4iIiIiIiIiIiIjbUfBaRERERERERERERNyOgtciIiIiIiIiIiIi4nYUvBYRERERERERERERt6PgtYiIiIiIiIiIiIi4HQWvRURERERERERERMTtKHgtIiIiIiIiIiIiIm5HwWsRERERERERERERcTsKXouIiIiIiFwWrFS48SN++3IoLUOtrk7MZcTAr95NvPvrOIbV93J1YkREREoVBa9FRERcwAjoyLM/zmbFul95uE7+AwhG+HW8MmkOq9ZN5O7Kl+Yw7n3lw8zY+DeHYn7jmda+rk7OpcuzFU9OmsvK9fN5obmHq1NzUYVtF0ZAB4ZPnMnyNVMKdM4VlquO6+5ULq5hrXILo9/oS+sed3Fb80AMVycoN6Wsb8rVeXnwpnavexjY5hpGfPwIUd6uTqCIiEjpcWne9YqIiBQj/25j2BkXR/KRVbze0tO5nZSpQvN2UdSO9MOjANEDS9kraNG2KTUjfLG6ZdShCBgWLBYDw7BiNQqfSWu1vrw7aQ6r131MLx8nduDZkQ+2xpIcf4Cfbwlxz2CPM2yRNGzdhFrl/PEsDZnKo13ku37LVKVlh2bUKe9foHMuL646rlNstRk8ehLzF60gZvtODh06RGLcYeL27WDbynlM/fI1Hu7dlPAiiBOWqnIpYgUaGwpTJ872SZYI+o4cTsdAk0M/DWfE7ATMAuaxRBRh31To/t9Z5+XhFKvfeZTRWzLxqj+UN++pg60EkyMiIlKaKXgtIiJSEEY4PW69jlALYK3Ejbd0ws/VabrEpK1+l24NqlKu7nWMXJpS6P1ZwhrSqX1TaoT7oPmdpVde7cJV9Vuq2pWlHFd2bkezetUoHxKAj6cNi8WKp08gkVc0pkOvu3jx81msnv8+N9co3JIGpapcilJBx4YSrJN/eUffz5PXBsOxBbz22jyS3DJyXbTcqj2mr+eDlyay3+FN1NDH6Rl6iXxrIyIiUswUvBYRESkA6xV9ua1D2TMz3SyEdb+VHuG6ARWR0iCDhU9HERkZSVBYJGGVa9OwQ1+GvjOb3WkG/vUHMHriSDoFqE8rKOfHhhKqEyOEG+69mWq2LP6e8CG/HHYUbn/ilNSFH/PpqjSM4Ou4b2AN1wfURURESgEFr0VERPLNk6hbbiHK0yR5/lh+3mfH8O3I7f2q6QZUREqFzLRTpGc5ME0HmSeT2L9lEd+/fhvX3vk9/9gNPKoO5LF+FXSTUCCFGxtKok4sFXpya+cAjMwtTJywhrRC7EsKwbGXSeP+JBVPmgy8mSaldElvERGRkqTrUhERkfzy7cDt/a/AZo/l189G8t7ErWTiSdQtg2ji5NLXGD7U7PYIH/64gE079xF3cDc7V81i3MjbaBlWsBUxPaJfYt2ROJIPfsON563taRB483iOxMcRv+wZmua2a6+KdLj7dX6Yt5o9+w5w5J+trJs7llF3tCAi5/betej+4AuM/nYyC1du5O99B0g4coCDO9bx19RPefmOtlTIrUw8KtB60KO88cVPLFi2jj1793M09gD7t65i8Y8PcqUHGJG3MfVQHMmHF/BE3WyhnzJ16P34q3w6/lcWrdzAnn/2czQulviDO4lZ9DMfP3wtV5S5QAF5deerfXEkx//358ikwRT5xPl/0zlhKotXbmDP3v3Ex8USv38b62Z9zvDu1SnjXZF2g5/ji18XsXXPAeJj9/HP+gX8OOo2okNyuzyzUqvf64yf9jvrYnYRe/gw8Yd2s+2vaXw1ojd1/fLOhOFXm95Pfsz0JRv4Z/8BYv/exIrpnzNy8JWEXSjvBWkP57FQ6e6pxB2NI3HLW3Q6ry2UodsnO0iKP8C0O8udd0Fqi3qOVbFxJB2azJDyp9/Ns12cTW8+69cSQZcnv2TW0vXs2XeAo/ltP3kp7uMWqh7yyyT+908ZvyUTDE+atrqSMlip88g8jsbHkbD2FVrluvZyB97bEkvy0b38OCD43LWXi7tcPMKIvvUFvpmxjO179nNk33Y2/TGR9x+8luq5rW1c2P7jQopjbMi1Tpxlodw13WnuZZAV8xszdtnPedfpscPZceDfPRe4b3K+H8xXe3TiXCt4Hkzi589gyUkTW9XruL6BVr4WERG5GI2WIiIi+WIQfv1geoRbyNrxE98uOc7WXeP4a9go2lfvy23t3mXNghMF362tOv2HP5ntBW/Cql1Jj6FRdOvbjWdvuoNPNp0sslzkxQhuw4jxX/NYdFC2QGII1aKu496mV9O97SP0uOcn/sk6s31gK+4efj8dcgQmygZXoG6bPtRt04vbbvmSuwe9wOzY/wIlRkhnnnrr6fM+5xFWhTrBBikX+CW7EdCC2x8dct5n8QqgfL32DKrXlm6dRtKj/xg2pztRCEUkz3SWCaZa8148+U0nbjvqQUSEzznBvsCKDbn2rjfp0KYaN3Z7iaUp2RektRBQvxNdW1+RbSanH5E1W3Ljo9F0aVOJ6/t8yMYc+bZV7snonz7i5hpe2Y4VQe1Wvajd6sw/z41jnc5DAdvD+RwcXrqEnfZW1A9uypXVrPyxPduBbHVpEeWLgYUGUQ3x/PpwtpmgFsKbNqWyFbK2LmXJkSJe3sAaTnSP7tle8CyZ9uPEcQtfDwXgiONwnAkY2AIC8TXs7F72FwftjalaLppWVa38tfPcxmKrGX36y5asrSxdccy5h/85Uy6BzXns2+8Y3iY024NrvajU8Gpub3gVN908kaEDn2DK3sz/PlNs/UcxjQ2QS53ACacKuSzRbZrgZdj5Z+kSdudyzjvD2XEAnO2bnOsH85UXJ841Z/tXM2kZCzdl0rVlFVq3qohl3T9oERcREZG8aea1iIhIflir0n9wR/xIZ/X479mYCY4Dv/Lt3GQclkh6Du6KU89eMjM48OcYHurbjjpVKhJRtTHtbnuD2fszsYR3ZORXz9CubJHn5lyWcvR753Meiw4kc+9sXrm1I3UqVySyVit6Pz+DfzJtVOrxCqP6nz9DlqxdfD4gimqVKhAcXpGKDTpy8/M/szXVwL/xEL788h7q5DZj0/4P4+/uSMPaVQmLqEil+m3p9vBP+QuqZO3mq8GtqFOjKmERFShftx03vTKPg1kGQa2f5LVbKuV+gZP+G/+rHE5g6H9/IvqNJa64Hlr2bzqrVyEssiJVm/fj5d/jMC0BRIY72PrzSwzqHEWV8uUIrxZF1+Ez2JcF3nXuZMSAnHnIYtuk4Qzq1o76NasSHlGeyNqt6fPibPZnGvhHP8SI3mHnzny11eSeMacf9mYmrubTYdfTtGZlwivWJqrbPbwyZSvHc4uYFKY9ZE/xzqUsibWDrRYtmgWdkzZLhRa0rGQDLPg3i6beOdMpfGjWsiGehp3DS5ewK7+BtvzWr/0AvzzRk5aNahFZrgDtp6SPW0T1kG+WclQqZwFM7MeTSTUhc8MfLIx3gK0u7VvnaF8YhDZrQXUr2A8uZ/n+HBVVbOUSwY3vfMOItqEYqZsZ+1gvompWJqJKI9rf8Ta/H7bjXXsAY75+iMa5zfp1tv/IS3GNDZBrnTjFVpMmDcpgmBnEbNhB5sU/UTAFHQec7Zuc6Qf/daH26My55nQeAMcRNm2Kw44H9ZrUo2gexykiInLpUvBaREQkHzwaDuTWKC84tYyJU/aeniVlJjHnxznEOwz8r76FvpWdGFaztvPdcy/z3Z/biT2RQXrqYTb99g63DXqL1afAo9pAhvWOyP1mvIh4XnkPT3YNwzi1mtcH3cXbs2KIPZlBWuJu/hgzlLs/30mWJYBOA3pQMWcWzVMcPXCYpFOZOBwZpMbGMHvM/XT/31j2ZIFv9DCe6Bp0fvodKezdup39CSfJtGeQcmQHq7bE5jZJ7XzmSWL//ofY5JNk2jM5eXQ7cz54gBHTk3AYZWjR/aqiXwrEGf+m89gpMrMySP57Ie898g5L0kwwM1j/85f8tv4AxzLsZKQc4K8vH+eVOSmYhhdR7aPxO3dnpGz5g9krt3Mw6SQZ9izSEnbx+8cP8sz0ZByGLy3aRZ0TBCnT9m7ub14Ww76Xb+8ZwPDvV/J3UhoZaUnsWTmFtx96n4W5RLEK1R6yy9jAouXHcRieNGvbLNuSBwZBrdvQ6Ewwy1a5Fa2z78izCe2ifTEcx1i2eGPRB9ocSWxbvpJth5JJyyzB9lPA4xZZPeSLhbAu9zCwjg3MDDasWMcpgLSVzFmYjMPwpPk17Qk5p1zKENWyIV6Gg2Mr/mKTsxVV0HJpeg9PdQ/HYj/MLw/dxEPfLWNPUhrpJ2LZOP1NBt7yHhvSoUyje3iyZ+j5fU8R9x/FNjbkVSfO8KpM1XJWcBxl7wGn95K3Ao4DzvZNzvSD+eHMueZ8HgDsHNh3EDsGXpWqEqk7chERkQvSUCkiInJR3rS5pS81bCapi35h5pH/pr+dWPQzMw7bMbyaM6h/7SJ7cGP61nF8tfAkpuFD6y5t8S+i/Z7Pg6Y9ulHNZpK2ZBzjtmfkeD+NdfOXEOcw8KwfReN8RQVMEv94j4+XpWNagujSoy3FPXkcM5lFC9aQYRrYqtehhpsujOY4spK/9tjB4ssVNcLPvRAzk1m1YgeZGNgqVKZ8fhqTmcLmDbuxY+ATHo7/2aCbB407X0WEFTK3fM/ni/K7pENRtoeTLJu/nJOmBf+W7bI9mKwM0e2a42Xfx6o1R7B7NKBty8CzgS1brTa0jbBipi5l7vISeqycq9pPnsctjvMyB4sH3gGR1IjqzO3Pj2XOFzdT2WqSdeAXPpi0/8wyBidYNP13khwGPq2upUNAtqiuRz1aN/PHME+w7PeVRfsAwDzLxUaj67tSzQZZOyYyeubR89p12sav+GRBKqbhT8ceHQnITyDa6fov4rEhX3VScBb/IIJsBjiSSEgqqQUq8hoHnO2bLnSovPrB/HDmXCtsHhwkJSThACyBwQTpjlxEROSC3PTWTkRExI34X8XAHuWwmsdY8Ms84rPfpaYtZ9L0A9x2bxXq9buJKz94kZU5732dYR5j44a/ybquAV7Va1LVBhuKYl3bnAw/atYqjxWDMl0+ZPfRD/Pe1juMiEALnMpH8MMRx4oVf2NvXxef2nWpZpvOpuJI/1kmJ2IPc8yE8MDA/AWsXMGewJF4B2DBP8AfC2QLSDlIjE/AAXiU9cPXwjnrpdrCouh/913069SIGhXLE+5vIfXIfg7ZI7ACps12ev1fEzD8qVkzAisOkjdt5O/8Lr1RpO3BJHnRPJanXUfncm1oX8vGsi1Z4NmUTq39IX4673/qw+jP+hDdoQU+P8ziBBYqtm9HdZvJqeXzWHS8uNZ0OT+trmk/eRy3uM5LPOn8/laS3s89LSd3T+PZO0cwO/G/ck9dNIXZ8X0YFNqWHp0CmTwlCROwVm5Jy4pWzLSVzFmUXPgAZI605F4u/tSpWwkbDhLXr2F7bn2KeYy1q3eS1S0Kr9r1qGGF1Rfte5ys/yIZGwpeJwXm6Y0ngJlBekZJnVPkPg7YneybzihQP5gfzpxraYXLA0BG+ulp2YanF57uOl6JiIi4CX3PKyIickEGYV1vpmuwBUfifCbNT8pxT5zB6l+msjsLrFV7M7CtTxEd1yTleOrpOKSPLz4Furk18r/MiFEWX9/8buyNV75neDo4lnwMB2D4+lG2BG7OzfR0MkwwbB5u/O18BhlnAlgWy/lzMTMyz0TZrNZzZmrarhjIdwum8/EjfenUpBaVQn3x8vQhpFJtGlYNPP+CzvChbFkAk9SU1PzP2Czi9mDG/8nctZmYtpp07lQZK2CrfxWdIi2k/LWQxYv+ZMUpg8B2V9HcCzDC6XRVIzzMDFbNW0hh4nUF5ar2k+txi+28zHZc007GySQO797Aoqnf8Or9N9C8/RC+3pLjAbEnFjFx2kHslkC6DLieCAuAhciOV9HQZpKxdj5/JBR9RV24XExSjh3Po107OH7sdJu3+J75EsjZ411Q0Y8N+a6TgspIIwPA8MTrgpHSAowd+ZLLOOBs34QT/WB+OHOuFSIP//L0Ov1TFDPjdLsTERGRvLnvvZ2IiIg7sFSg94CO+BpghPRlwt99L7BxBDcMuJoX/pjOsULfjFoICAzAApipKfl6UJeZmUWWCRheeJcx4GR+PnSSEycAHMSPH0i9h3+nKCaOg4Gfvy8GYJ48wSmX3pyXpshALmk1QrnxxZfoVt6G/fCfvPfcKCYs3s6h5DQMn3CiH/ueyQ/Uz7GbE6SmAljwDwrACvlbO7qo24PjEHNnr+fl1s1p2LkjkWP24ndVR6paTzBv3hJSEj2ZszKd69p3oHNjDxbubsc1zb0g4y9mzj+Sz6CQq+q3GI9bbOdlBvMfbky/8QkFSH06y8d+z5bBT9Kw7UD6V/ueD/cE06nLlXiSyYrZ8zl0TkUVZ7n8264N/M78cuF8FvwDfE/3nSdSSS2uVTKKbGxwpk4KxnE8iaQsE7yCCAnK8ZMOnBw78iWXccDZvsmZfvB07i68X2fONcPJPJxlISgk6PQvb5ITKbGVXEREREopzbwWERG5AGv1Ptwc7ZXP2WgWgrrcRPfQIpi7ZgmnZcsrsGJycsdW/snHkhuO5ASSTMBSgSr5WjAZMI+za9cR7FgIjGpGraL6WtsIoFHjatgwSd+9M1/pLy5megbpZwIzXl6l8PfZHvVp1dwPw0xj/qv38Nqva/kn4QQZdjvpKUfYezjl/PCMmcL2bQewY+B3ZUsaeOS241wUeXtwsG/mNNZkgmezrlxTvjpdr6uDLW0Vs/9MxjSPMn/OGjIsFbn2uoaEtr+ONj6Qvmoavx3MZ+jaRfVbrMctrvPSSVlbJ/D5wlTwbML/hrTCN7IrfVp7Q8Zqpsw4dy3m4i2Xf9u1Bf8mUdTOrVwMf5o2q4kNk7TtW9jtxJIO+eGyscEZ6fv457AdLGFUqVjmvLedGjvyI7dxwNm+yZl+kHy0R2fONWfzcJaVipUrYMUkff8/xCp4LSIickEKXouIiOTJRv0b+9LI08B+aCx9KoUTGJr7n5A2r7Euw8Qo254BN1Qo5ABrENzpEYa28sRwJLNgxmJSz7xjmqf/YHji7XHujbjjYAwxSQ6wVqfLNdXz+fOqTNbN/Z1YO9hqD2RY17Ai+dm4V93B/K+jD4Z5kmXzlnC8CPbpLEd8HAkOwFKN2tWKMDBTkszT/7FnOfI5OzOTDb/N5p+s0z+1f+LmKi5rD479M5m6OgO8WnDjvbdzfX0b6StnMi/eBBwcmjuLNZlWqnUdyMM3tsePdFZMnZ1jNu8F9u+i+i3e4xbPeek0M5YpX0zhkN1K5Zsf5vH7bqJ1GUhfOZUZOb5kKO5y2fDbbP7OAlutAdx/beh55eLd4H/ce5UvhnmchdMXklws05ldNTY4KWsn6zefwjQ8qde4Fjljrc6NHReX+zjgbN+EE/1gftqjM+daIfIAYImgYcNwrGQSsz6G9IJ8VkRE5DKk4LWIiEhePJvS/8aa2LCz99efWHIq703tO39mwoo0TMOLFv17cUV+YzaGDxGVKxBcxgOLYcMnoi7X3vcxM74azBU2OLF6NG/+9t9aqubxBJIcJtjq0vvOLtQO9vzvRjtjOT9PPYDd8KDxsE9455ZoKgd4YjFseAdGUrWcf6435WlLxvDekmM4rOXp99FkvnigG1GVg/C2GVg8/Yis1YIbbr2OOrndndtqM/jl5xjcoTYRZT3x9I2kQbdH+W784zTzhsw9E/hgyhGXLtzhiFvLmv1ZYKvGoOFDaVOhLDaLJ34VGnFdj2gi3f1qKHMrazaewjTKcPVjr3NXuxqElrFhYGDzCiAkIPfZnxlrPuX13+JwWIK55o0pTBzem2aVA/C0GFi9/ImsVoHAXD5YqPaQG8chpv+ynFN40XrIHUR5pLN8+jyOnIl5Og7MZNrqTKw1BnFvF384uZSfZx7O9zqyrqrf4j5ukddDIZ1Y+AkfrzoFvu146L5meHGSJZNnnfclQ3GXS8aaT3ljZtzpchk9kXdvbUnVQE88fSJo0P1xxo1/hKbekLbxc96cerR4+p6SGBuK1AlWLl1PummlYpu2VM+ZBifHjrMKOA441Tc52Q/mpz06c645278CGEGt6dDQA7L2suyvA06tmS0iInI50ZrXIiIiefBu2Y/ela2QtY2fJ6298DqYjoNM/XERL7a9Ft/GN9K3zieM2pKP36vbqnPXhNXcdf4OSdn0Lffc9Qlbsy2maSYvZvriY3TpEkiju8ey7NpP6d7yeZZnAqSx9J0RjOv0FbfVaMBt78/gtvfPP+R5K3jY/+br++6l2vefcl+T2vR98Vv6vphjm4xVPLNoLtv25rjNNjyp1HEoH3YcmuMDJvajC3lxyOssOXHxYihWmev56uOF3PL21YR2epbfNjx79i3zxCzuWbyKn4pnembRMOOYNGo0g1s8SfOafXl7Sl/ezmWz89qnI5ZfHr2dyoHfMqJ9Rbo89hldHsvlgzmbaWHaQ64cHJ7+PfOea0fPQCvmqcX8Oif2v4CN4xDTf13Bi63a4W11kDBnAtPiClAfrqrf4j5ukddDIdl38+3rExgy+S6qWcGRNJ8JM3L5Yqq4y8URy8+P3knVkO8Y3roxd7w3jTvey76ByakdP3D/ne+zvpimtJbI2FCkHBye+xurRralbb3udK8xmm3bs6fBybHjXwUdB5zpm5ztB/PVHp0415ztXzEIvfp62voYZO2ezYzNLlxTS0REpJRw97lGIiIiLlKWjv26U85qkrHhJybFXOwG0yR+zmQWHDPBVps+fRqf99Psc7Y+toLv3v+an39fy7YDCaSkZ+FwZHIy6RDblk7hoyf70eq6p5l5KMedr+MQ3z8wiKe/+4PNB5JI3bmDPdmSZsbP5dHuPXnok1ms+SeRk1kOTEcmacfj2bdtDQumfMcHn85lf87dxi3gmW4d6fnkp0xdvoPY4xnYHVmkpSawd8sypo2byvqTuWQkaw8/v/kOExfGcOhYOpkZJ4jfu5aZnz9Nj46D+HijqyPXAA72jrubHg+MYdbGAxxLt2PPOMHRf9Yx95el7CsFK4mkrX+PPl3v5vUfFrJpfxKnMu3YM9JISYxl746NLF8wnYmztpz3YE/z2Gre7d+eLkPfYsKCjexNSCHdbmLPOElS7B42Lp3JuI/HMOPvcxuE0+0hD2biXCb8dhQHJicWT2bmOYu8Ojj022QWnzLBfpipExYU8IGnrqrf4j9uUddDYZ3861M+X5WBiYMjMyYyJym3iir+cjGTV/JW3450f+JTpq/aw9HUDDJOJXFwy5+Mf/UOOlzzMJP3FuwRevlXvGNDcXEcnMq4BccwPeozYNCVeOdMpZNjB+DUOOBM3+RcP5i/9ujMueZU/2qpQt/BHfElgw3f/8D64mqmIiIilxAjICTMjacaiYiIiDsyIm/j17Vv0cHYxKtXXcNbW0t6JqGIlDSPmvcxdf6LtDLW8XLXHry7RZG30sS75UiWTr2XailzebDdYCYcLtyMfY0DBefb6S2W/XAbFZOnMaTNEH6J1624iIjIBRlM0sxrERERERHJwSCoSm0qBnhi8w6mZvu7+WzCM7TySeWvUQ/xkQLXpU7ayjG8OTsRAq9m+PDOBLn0KaCXIa/GDHthAJUsaaz95G2mKnAtIiKSLwpei4iIiIjIuYxgur8xn027DxB/YBurJr9Cn2qwb/Lj3PPZ9guv8yzuyRHLz8+/zp/JBhVuGsWr14Zc+EGMUoTK0Oyx9xhW34P0LWN46tNtea8hLiIiIufQAxtFRERERORclhC80v7h6KnqBFtSObJzNbPGfcCb360gTqtDlFr2veMZ9nQ0n3fdwrg1yec/cFOKSRrbp37OxBbX8/eI91iT5ur0iIiIlB5a81pERERERERERERE3IvWvBYRERERERERERERd6TgtYiIiIiIiIiIiIi4HQWvRURERERERERERMTtKHgtIiIiIiIiIiIiIm5HwWsRERERERERERERcTsKXouIiIiIiIiIiIiI21HwWkRERERERERERETcjoLXIiIiIiIiIiIiIuJ2FLwWEREREREREREREbej4LWIiIiIiIiIiIiIuB0Fr0VERERERERERETE7Sh4LSIiIiIiIiIiIiJuR8FrEREREREREREREXE7Cl6LiIiIiIiIiIiIiNtR8FpERERERERERERE3I6C1yIiIiIiIiIiIiLidhS8FhERERERERERERG3o+C1iIiIiIiIiIiIiLgdBa9FRERERERERERExO0oeC0iIiIiIiIiIiIibkfBaxERERERERERERFxOwpei4iIiIiIiIiIiIjbUfBaRERERERERERERNyOgtciIiIiIiIiIiIi4nYUvBYRERERERERERERt6PgtYiIiIiIiIiIiIi4HQWvRURERERERERERMTtKHgtIiIiIiIiIiIiIm5HwWsRERERERERERERcTsKXouIiIiIiIiIiIiI21HwWkRERERERERERETcjoLXIiIiIiIiIiIiIuJ2FLwWEREREREREREREbej4LWIiIiIiIiIiIiIuB2bqxMgIiIiIlIS6tSpTe9evV2dDBEppCm/TmHbtu2uToaIiIiUAAWvRUREROSyEBoWRtt2bV2dDBEppMVLl4CC1yIiIpcFLRsiIiIiIiIiIiIiIm5HM69FRERE5LLz4UdjWLlylauTISL5FB3dnGEPDHV1MkRERKSEaea1iIiIiIiIiIiIiLgdBa9FRERERERERERExO0oeC0iIiIiIiIiIiIibkfBaxERERERERERERFxOwpei9EdS9gAACAASURBVIiIiIiIiIiIiIjbUfBaRERERERERERERNyOgtciIiIiIiIiIiIi4nYUvBYRERERERERERERt6PgtYiIiIiIiIiIiIi4HQWvRURERERERERERMTtKHgtIiIiIiIiIiIiIm5HwWsRERERERERERERcTsKXouIiIiISC4slO8xkl/nTuelth6uTsw5jKCOPDdxBotHdcbL1YlxE9aI1gx9axx//LWGnTHr2LR4CqO6hmK4OmEiIiIihaDgtYiIiIiI5MKgbMW61K8UhHexR0C9iBr2I2tXz+SNa0IuGnA1vMtTr2E1wnxsZ7Yt2OcvOZ71eeizj3j8hiiqBntjs3riGxaJV3oqpqvTJiIiIlIINlcnQERERESk9PKmSqdbuH9wd9o1qEyoj0H6saP8vW09S2Z9z+c/byDJbaOHVurfMZp3bvFl+tA7+Hi7vUj37t3yIcY9dz3VwoPxK+uJ1czgZPJR9u3cyJI5P/PdLys5nPHf9oZhYBgWLE5Gns//fEHy50efTxbyzlUXm8edyapXu3Pz2IM4nEtmsfBq0Z8BtT3J2PkTjz/yIfP2pOIZVh7f1HRXJ01ERESkUBS8FhERERFxipUq/d7l55faE2r9L+JqC6lIgzYVqGlbz9hfNuC+U18tBFapR81ySXgWw1Rla1gNGtYon21ZD2/8QitRP7QS9Vt1ZWCfLxnyvw9ZcdwE0lnzQX+afuDs0XL7fPHmz31YiKhRnQAjncVff8BvO5MxgfTYvaS4OmkiIiIihaTgtYiIiIiIM2xNuG1oW0KI4/d3nmfU5HXsTc7EM7gS9Zu3p9GpPzniTtNzXSKLmI9u5sYx20g3rZQJiKRG1DUMefx+uje8k5F3zKPbBzEU7ZxvZ6Qw+b4oJp/9t42oJ6fz0x1+/HJ3J55anOnCtP3H4uVHSKA3WSlJJJ3MOvOqgZe3J4Z5ivj4E+77XYmIiIiIExS8FhERERFxhk9FqoRYceybwYdfLWHnmQhsRtxuVvy2mxVnNzQIaTOEZ27rQL3qlSgf6o+Ph53jh7by5/cf8dmGCvQa1JNrWtShYqCVEwc3M/e7txn1/SaSs0ciy1TlmjvvY8gNralXviyOpL2s+f1nxnz8AyuP5gj/FmRbay2GTd3IsDP/dMT9wK1Xvcyyf+O1hg8tH/ySOa/WpnKIN/bkfayfP5F33v+BdflYE8WRlUGm3cQki5NJB9i44GseOxZGo7GDqdY8ighLDIccVmre9wMzh5Vjco5gsSW4MQOG3sugLk25IsSDk4e38dfyY0Sc8/SevD9/0fwVmEFAk/48dNs1NK9fg6qRgZQxThG/dw4vDX6R2UYHhr/7EF1rVyTC3wvzVDy7V8/hy3c/Ysr2f4PLubUJSEvKvWwtwc0Y8vwI7ulciyAPA9PMJGXfPEbe/hS/HDq9PyxB9P9iPf3//VDmKl7ocidjDzvy2R4ulK+XWFn3rsK3YREREZECUvBaRERERMQZpw5zIMmOpdLV3Nr1F3bM2MupXDe0ENyoCz061Mt28e1BUKWm9H7qK3rn2NqzSjNuevYTgk/cyL2/Hjm9trJ3Pe75/EuejA7474nrEbXoMGA4rds35rFbhjP90JkgZEG2zQ/Di8qNm/3379DqtLn5GZrUKkOfW79mR1beH82LI8t+Ol8WywWfIG/4t+bZsaO5vab32YcwelVuQrfKp//fNSs6Wwhv1Zdbu2WvTz/Cwj1ITzWhTCDVo2pR0fPMW74R1O04mLcahJPZ83Gmx5vk3iagbG5la4mk36jRPNnBHyPrBAlHUjF8QwgM9yD9mAOwXji5+W4PF8qX4XQbvvvXIwUpXBEREZFzXOhaUURERERE8pK5hq8/WkK8UYUb357C7xNGcu+1dQjOa3qIeZxZw6+hcaNG1GjYlu7PzOSA3cSRvIqPHuhLqyubUCuqC7eMWcNxAunQ5yrCLQBWatz6HI809ydt68882f8q6jdoStNrhvD674cxynflxSe7EGQUdNsz7Dv4sGcjqtWuT7Xa9aneLsesZDOVP1/vT9voK6nZoAVtB41iXqyDso0HMSjKI9/FZVg9KRtckfrtBvDK8/2obLWzb81aYvNcWsVGg/89zeAanhxfP46H+57OS+OrBvHwl6uIz++SLBfLn7PMFOa9cD3NmjahRqNWtOv3ASsywUxZyluDe9Gq+ZXUqNuIui2v584vN5IW0okbOwZxzvLb2dpE9fq5l63h14JrWvjh2PwZvVu1oln7q7jyymha9n6bJSez7cuRxE93NTmbz2oNbmfsYaPg7SGPfOVMb37bsIiIiEhhKHgtIiIiIuIUO3snPUyfez9kWswJQq68kac+/Jkl877l1VubE5EziG3aSTkax/F0O/aMJGKmjGHCFjuGLZ6YpVuJTc0k88Qhln7+FfOOmVgrVaWyBbDW5Iae9fHM3MTox0YyacMRTmZmkLx3GZ8//gI/HTYJ6ngDVwcZBds2v8xM4nbv4OCxNLIyUzm4+nteG7eZLGsIdeqEXeSGwkaDh6exa/sW9sSsY/Nfc5jx5bPcVL8sp7aO54WvtpDnxG1rba7tXBVL+href+xNpm46nZfjB9czffxcdrl6oWwzi6SDB0g4mYk9/TiH9h7hhAkYBoENb+a1r39h6YpVbPxjLC9eWx4rNiLKhZ5bXtnahCMrj7I1TUzACKtDdJ1QvA3ATOfo3wcuviSHM+0hr3zlSG9+27CIiIhIYSh4LSIiIiLitAwOLPqch/p0pv2gZ/lk3i4yIpox8JmvmPZxf6pfaJE++2H2HsoCrzAiArNdlmfEciDOxCjjQxkD8KxCjYoWHPtXsPSfHBHbE2tZvC7t9DaVLAXb1ml2Du3+hxOmga9vWfIdBjdNTNMEx3FWfzmMHgPfYumFoq8e5alawYLjwFpWHy4lT740/Gn/7LeMHXEznRpWJcLfC48ywVSuFIqXARbLxVZtPL9szZRlTFkQjxHRgRFj57N+2Qx+/uQlHuxak7IXK/zibg/5aMMiIiIihaHgtYiIiIhIoaUTu2YKbz7Qhw59X2TaPgdh7R/iwat8L/AZB5kZWWB44OGRPQqZSWaWCYZx5mK9ALOkC7St88yMDDJMA8NyseNlsfn9G6hRuz7V6kRx7WvLScaX6nXCIf0i04YN6+n8G5YSylXhGcGdub13ZaxJK/jo/htpdWUTatRrRssHpxCbz5ni55WtGc9vI27lzle+4scFa9lnlieqU18effcr3uwWepGyKe6Su3gbFhERESkMBa9FRERERIqMg2MxU3j/p63YLb7UqFHuYo/Tu7iMvew+4MBSqQVtquTYW9ko2jX1hox97DngKNi2mGRlZWHig49PSQQZM9g5fgTPzDyKf5vHePuuOnhdcPN9p/NSuTUdq+d/be3/lHT+wBIaSaQnnFwyjtHztxGbmondfoqEo8cL93DJtP0sHPcuT99/G9e060C35+dy2Aym47XRXHBuc4Hag4iIiIj7UfBaRERERMQZnk25+7XHGXxVAyoHeWM1DKxlgqnWvDdDe9XCamZxNC6RQocF7TuYNi2GDI+GPPjuc/RtFIGPzZOAKq25+62X6F/OIHnhdOYnmgXbFpO42KOY1nJ06deFar42rN7BVG9Wn/KFjrjnwRHHrJEvMOmgF02HvsSQup4XyPd2ps/YSqatHvd/9AZD2lUn2NuKxepNQGgQF49Hl3z+HAlxxGVCmRZ9GHRleXxtBlg88PX15mILhuTJWo2rbryKRhX88bQYWD1sZKWkkM7pic0XLIYCtQcRERER9+P0NZSIiIiIyOXMVu9qBva6gyo33pHLuyando7lizmJmIWeL2Jn57iRvN/+S55o3o+3JvXjrWzHyTw4ixffmMPp+GPBtt238HdihjWkUZ+3+b3Pmc0y1/Nat1v5Yl8hk50H89gS3nh5Ku3G9Oa+5wcx55Zv2Jnrkhp2dnz3Im+3/oKno69lxJfXMiLHFheezXyx/BX9bGMz4Xd+WvAA7bpfxfPfX8XzOdKzw4l9GiEt+N9Lz9E65+RzRyKz5q7ixAU/XZD2ICIiIuJ+NPNaRERERMQJjj3TeOPdCcxauYNDx9KwOxxknUrm4Pbl/DrmafoOfJtlKUUUFTwVw6dDBjJ09G+s3pvIqcwMTsTtYNHE17nlpqeZdsju1Lb2nd8x7Ilv+GNnPCftdrJOJrBn3S6OFutaxSbJi0bzzh/JeDe5i0e7heQ9e/jUVr4YchN3vP0zS7Yf4Xi6HXtWGinx+4lZOZ9fFu4h8wJHKvH8mYnMenYIj331B5sPHSfdbicr/QRJcQfYsWEFy3cdo6AtwjAOs/bPjexNPEWWw4H9VBL7Ns7n86f+xxMzjl58fwVpOyIiIiJuxggICdP37CIiIiJyyWvbri3Dn34agA8/GsPKlatcnCIRya/o6OYMe2AoAK+PGsWSxUtcnCIREREpdgaTNPNaREREREodi0WXsSIiIiIilzqteS0iIiIipc7gwbfSrFkzNqzfwKbNm4iJieH48RRXJ0tERERERIqQgtciIiIiUuocO3acqlWrUrlyZXr26gnAocOHWbtmLZu3bGbLps0kJSe7OJUiIiIiIlIYCl6LiIiISKmTmJgIgNVqPftahfLliQgPp3v3blgsFhLiE9i4cSObt2xh3bq1rkqqiIiIiIg46bzg9b8PsZELi9m2lam/TnV1MgpM9et6U36dwrZt212djAKpU6c2vXv1dnUyREQkn14fNcrVSSh2iYmJGIZx3us223+XtyGhIbRr356OnTpiGAYpKcfPvufv51ci6RQRKUm63xOR0qo0Xr/27NWTenXqujoZl5Tc4q3nBa/btmtbYgkq7aZS+oLXql/XW7x0CZSy4HVoWJjajohIaVL6rv0LxMPDI9/b2mz/zcz28/PP9rp+gChSWtWpU5vE+ASOJiSQlJhIVlaWq5PkNnTNLiKlVim8fq1Xp6763WKQM96qq3YRERERcQuenp74+voSHBxMcHAIwcFBBAcHExISfOa1038CAwOxWCz53q/DbifLbmfjpo00u7IZAIlJScWVDREpZr169jr7q0DTNElOTiYxMZGEhHji4xNJTEzk6NGjJCUlcjQ+noSERE6kpro41QXnYfMgMyvT1ckQERFxqTyD1ytXruLDj8aUZFpKhfFjv3F1EoqE6rdkRUc3Z9gDQ12djCLx4UdjWLlylauTISIiOQx7YCjR0c1dnYxcBQUGEhAYQEhIKIFBgYQEBxMYFERwUBAhwSGnXwsJwcvL6+xnHA4HycnJJCUlkZCYyLGkZHbv3s2xY8c4Gh/P8eRjvPb6a3nOwjZNExM4kZrKtGnTmTZtGk2aNjkbvBaR0mvUG2+wft16IiMjz/uiq3z5cjRoUJ+QkBDKli179jMZmZmkpqSQmJhI7OFYEpISSUxIJPHM37GxscTHx7vVLO7uPbrTqmUrfvrpR9asKdi6/brfE5HSwJ2vXwvq6Z1VXJ2EUm1Uzb15vqeZ1yIiIiJSYB4eHvj5+eHr50twUDDBIcFn/w7599/BwYSGhp6zREdmViYpx08HkBITE9m3fx/r1q8n9UTq2dcSE07PmrTb7RdMw/FjxwgJDT3nNdPhwLBYiI+PZ/KUKcyeOYuMTM1cFLnUpKamsmvXLmBXntt4enqe/sXGmf4oMiLybB9Vs0YNgqODCQ8PP+eXHKmp2fqixCQSExM4fDj27P8nJiaSlJSEaZrFnsfQkGDq16/HyJEj2fvPXr6f+D3Llv2Fw+Eo9mOLiIi4CwWvRUREROQsZ5fuyMjMJDEh4WzweeeuXaSmpJ4TACrqoE9CYuLZ4LXdnoXVamPv3n38MmUyf/7xpwI8Ipe5jIwMYmNjiY2NzXObf7+Iy63Pi4yMpEaN6oSGhuLj4/PffnP0d8U1izs0NAxMEwyDypUrMXz4cOKPxvPLlMn6Yk5ERC4bCl6LiIiIXAZyzkAMDg4mJGewJjiYsr6+53wu5yzEffv2kZCYePr1M8Ga+KPxnDx5ssTzdPRoPDVr1gRg3br1/PjTT8RsiSnxdIhI6ZWZmXm2j8vPLO7IyMjzfmmSn1ncsbGxJCQknveF3unj5i4iPALjzP7+/TskNIQhQ4YwcMAApk2bztSpUzlx4kTRFIaIiIgbUvBaRERE5BLz4IMPEBwcRGBgECEhIQQEBJyzdEdGRgZJiUkkJCaQnJzMvv372bRpMwkJ8SQnJZOQmEhycjLJycluPXs5Li6W+Qt+Z/Ivv7Bv3z5XJ0dELmH5ncUdHBxESEgoISEhBIcEExYaSnBwMJUqVaJxk8aEhITimW2t/vS0NI4cPUpSYiIJCacfOpmYkEhc/FHCwsPOO4ZhGBiAn58fN998E71792LKlF+ZPn06KSkpxZF1ERERl1LwWkREROQSU65cORISEzl48CDxCYkcSzpGfEI8x5JPB6YvlVl6X3/9rVsH10Xk8pKZmcmRI3EcORJ3we0uNIu7fv36BAefnsV9sSWWrFYrPj4+3HzzTfTr14/Zc2YVZXZERETcgoLXIiIiIpeYESOecXUSSoQC1yJSGuVnFndIUDBjx4/N1/6sVitWq5Ubetxw9jVvL69Cp1NERMQdWC6+iYiIiIiIiIiUFL9A/3xtZ7c7zn6Rl5pt2RCLzVos6RIRESlpmnktIiIiIiIi4kbCQkJzfT0rKxOr1YZhGCTEJ7Bx00Y2b95CzNYY9u/bz4wZ0wE4eaLkH6IrIiJSHBS8FhEREZHLTtdrr6FldHNXJ0NE8ikoKMjVSShRwSEhANiz7FhtVhwOB3v37mXjho1s3rKFrTExJCUnuziVIiIixU/BaxERERG57NSsWcPVSRARyZOPT1k2btjIps2biYmJYdu2baSlpbk6WSIiIiVOwWsRERERERERNzJlymSmTJns6mSIiIi4nILXIiIiInJZWLJ4Cd0XX+/qZIiIiIiISD5ZXJ0AEREpDlaq9HmdaXMnMyJa31O6IyOoI89NnMHiUZ3xuujGAbR/5mu+HNqcEKMkUlcQFsr3GMmvc6fzUlsPVyemQApUB0XCm5q9XuL7DwdQTaeliIiIOEXX+RdXUten7lkX1ojWDH1rHH/8tYadMevY9OdHDKyk8J+cy/Dy4sFrQ5jU2gtPVyfmItR6RQDwImrYj6xdPZM3rgnB7WJDkk1x1NWlWf9+lepRr1Ig3salkqNLi+FdnnoNqxHmY7tImzPwa/MQLw9qxhUhFjKKLUXOngcGZSvWpX6lILxLWVM7vw6Kuy/IJMuvEg26PMTLN1XCWuT7FxERkcuBrvMvpuSuT11XF3lct3rW56HPPuLxG6KoGuyNzeqJb4iFjOMm1pqDmbB0JUtH96GyooGXPcNqpWaIjWCbcab9GDRoHMyMm0J5urLFreIixdZcvVs+xKTf5rF61Rq2x2xi16ZVrFv8G1O/fINnbu9MnQBnb9ms1L9jDLMXjOX+2rrtKwlle4xm27bVzHgimsBcW68H7V9ZzO6Y33i6UemtE8MwMAwLFnc6Qy9V1ho8MHkDezb+xAO1L/RNeFlaPTuLXdvW8lUvv7OvFkdduab+y9LlzcXs3raasTdF5N0hezRhxNyN7Nk0ljsqlv6rjLI9RrNt+wam31fdzYN3bjLe2Gpy26O9qZAwkzdGryTFLL5DqR8s7jKw8/fEUXwe40XLoUO52v8yLmgRESkV8r5us1HlhrdYtHkTmyY9QsvcbxTdxuVyT5td8ebZTa6Ti5PFnzrX3cOoz39i0V8r2R6zni1/zWHGt28yYlArKpbMz/YuKLfrVq8W/RlQ25OMnT/x4PVtqVOvMQ2vfo7Zx00wDCyGgcXq3ufrpc4r0pePe4QyrX84vw+KYOHAcGb3DeWbLgE8WMeLii6exG9Q8GBxzbqBfNcriFuDiiNFxbjmtTWsBg1rlP/vZ7hWHwLDqxIYXpVG7bpzx70bGfvsE7w2/yBZBdqzhcAq9ahZLglPnW8lxyhD/Tvf5cPDt3HX+N3FOPPPVdJZ80F/mn7g6nRcJiyhRIRZMLzqctcj1/Pz0CnEOs7fzFpzAE/2q4TVsBMUGoyVFOzFUleuqv8TLJ25kMQevYi+vgvlJ43nQC7l4BXVnW4VLaSvmsXsQ7lsIMXEPcabsm0Hc2tdg5gPv2R+cjFGrtUPUiJlkLWT8Z/P5473r+Wu3mOY/91+dFaLiEjpYqXctS/x7avXEbJzHHff/T7Li/UapYhc8ve0uSi2PLvHdXJxMfwbcdfb7/Fk+0hs2fLnGVyR+q0qUq9JWbbPXM6BdNelMffrVgsRNaoTYKSz+OsP+G1nMiaQHpdw+u0d3zGg9XcuSKtkZy1jo06A9b+lOgyDst5WanhbqRHhzQ21TvHy/OMsOlnSKTPZvCGR7hsK+jmDAD8PqpZ1FNvyI8U8hS+LmI/7Ua9eA66oF0XDNt3ofd8rfLnkIFmBjbn9vc94ppWfW01FLy1CQ0MZOXIkV3e+Gh8fnxI4oond9KftU+8zvHWA6qwUi27enCeeeJzo6GhsNhd9pecVSkSAQUbycazt7mZIlPf52xiBXHPvbTRMTybZYRAcHJSvdmfx8iMsIowgn9zzdrH3S9rJFTOZG+fAs2l3ulfObdaCNy2u70w5SxorZszPNcjvLtytbC8JRgBX9elMaMYaJv26B7ur0yNFwCR54S/MPGKjaZ/rqXmJTlYSEZFLlYWwDsP59s0biNw3iQfvfpulSa4LXBfs+vNyvKe9HPNcSJZy9Bn1MU93iMBIWMe4l++je6eW1GkQRaN2N3DTw2/x7XfTWXqs6Np90d3DGnh5e2KYp4iPP0Ep+ErJrURHR/P4E48R3bx5icRKdm5MoPOEI7SfcIQuk+K5848UZiSaePqX4eFGnuQSJblsFfvvzx2Z6WTYTUx7Oqnxe1n/+0Revas/t3+9jTSPqtzy9K3UOXPjZoR0ZMR3U1i8fBU7Yjayfc3vzPzsKfrULnt+J2utxbCpG/l7+xb+3r6F3Yufo7WHE/sppQyLhSuvjOLRRx7hhx8m8syzz9C6dWs8PYrrYQRZbBg3mlkJVbj1zZfoVf5id9setHnxT3bHTOGROtm3NQjoPYbt29fxbd/gs+vqhLS5m3c/n8DsBYvYuGE9u2LWsHb+eN69syW1r7yRp94dy4KlK9m+ZQ1r537HqEENz/vpk1G2Btc/8i5TFixj66Y1rJ3/PR/e34GKHtmO3eQmnn/vK6bPXcimjRvYtWk5y2e8RNdgKzXvm8TOrUt4o12OMixTmc73vcYPsxayedM6tqz8nbnjXqRXldIZcfD09qJjx4688MLzTJw4kQcfeIAGDRpisZTcchSW4FBCLA7iZn7CuF3l6H9fD8rnOLyt5s0MvcaLv8Z8yfJ0C0GhgWc6rNzryhLcjHven8zqNX+xctGfrFm7mg1z3+LGMzu+8Pu57TO3drmezUunMf6lATQNyqU38a5IpyEvM37GH2zcuJGdG1eyesEUfvrkZYZEB+be/5xazdS5h3F41KNn91yW0Sjbil5Xh2KkLuPX+fGYFKaPK8h5eeadi55XFy/7iytsH2AQ3Op/vPnJWGbOX3zm3F7JqjkT+OixnjQMzJ6OgpfBBcebfJTP6TJqzKBnP2HmwuVs27yGtfMmMHpoWyIuVkQ+0XRp5UvWpj/4/UiOby48y9P2juf5Zsp81q3fwK7Nq1m/aAZTv3mXl/rUOtOWCpLfouwHLYS2f5Y5Gzaxbtz/aJjrd6wl1fefSVG+6iD3MsjfOVeAPiNtPfOXJmOp0YmrS+lYIiIilyOD4DZP8M37N1H1yDQevetVfj967vVJ4e7JjHxf5zp3/VnQe9r85qlg975Fkf/8K648n5HrdbKVho9MZ9e2VXxyvV+2jS1EDvya7VsX80b77GtuWGn4yDR2bV3Mmx3OhOvKVOWa+99g0pzFbNm0lk2LpvDti4OIDstRvnmWZW65ys/1Kfi0vpfHOwZD/B+MGHA7z49fRMyhFNIz00mJ283KWd8y8r3ZF5xUVFTtuOD3sKfLBUsQ/b9Yf7Ze/t78LYPLWTBC+vLdpi1s+6Qn/tk/Ucjz9lLh7e1Np46deOHFF5g48XseeOB+GjRoUGyxEtMBmSaYJqSl29l58CRvLT3BTgcEh3lS2QC/0DIMaxfEVz3DmD0ggj8HhDPlen/+PVUMDxtXNwngs15hzB8QzoxewbzY0IvIHEm2eHvQq3kg3/QJZ8HAcGb0DObFRp6E5qi+qg2C+WNQGE+Xz/GGzUrbBgGMviGMOQPCmdc/jHFd/Lkm+ylu2Li9ewSLbzn9Z+GN/kQVUdG5ZnqceYzlo99g0rVfMrhmV7rX+YytW+yQFUj1qFpU/HeeuW8EdTsO5q0G4WT2fJzp8fn83qio9lNKWK1WWjSPplXLlqSnp7P8r+UsXLSYtWvXkJVVsEVZLsR+cBbDH/Oj2td3MPKdO9l+xxfEpBXFni0EN+pCjw71sjVID4IqNaX3U1/RO8fWnlWacdOznxB84kbu/fXI6Z9b+zTkga++4OGmfme/kfGu1JgeD35Ik/LD6PnsQpJMC+Gt+nJrt+zH8SMs3IP01DyS5lWHIZ99xdMtAv/7psczghpNKuF7yo2nwOaTj08ZOne5muu6Xsex48f5888/WbJkCTFbYor1uEZgMEEWk+S4FYz9ehEDX7uDO5tO45U1Z353Zfhz9d0DqRM/ndunbKfrXSbewSGUNSAjt9PXEkm/UaN5ssP/27vv8Ciq9YHj35ndTUIK6YD0ErpAAAlFehVBioKCFCt6VeRaEQuI2LBeLyoqAioo+gMFpEsP7dJbIECooZPey+7Ozu+PQAgxZTfZFOL7eR4fgZk9c86cmbPnvHP2TGUUayqx11JQPP3xqWIiM9FW+PY8V17O67oEj4AG3D3iTYIbVeL+MXOJuHGLuTVh3KzZTArxzbHmmAf+NRvhX7M+LvvnMnd396dYEAAAIABJREFUQh4zZ83sW7qC06P+RaMB99L8uwgOZ9+2Ct5dB9LDF+JX/MmGG7NaSquNs+e+Ugo7t/Yobhug4h98D0N75vy8kYC6wQx4qhW9+7XjhTFTWJM7+FtcdrU7oFTuxFvzvuTRhm7ZnVXX2sHcWzvrzwX92tDYuDWtPGxcPHjo1g6yWxPGffc9r7X34+aSdUa8q9ajZdV6NE5ey4eLI5wzU7vQdjB3j0TBu90EZv/nIaqf+J4nnptLWJ4/eSuttr94dQDYec850mZkcnj/USwPhNA22AvOJBSWAyGEEKKMqXi3m8DcGaNpHP8Xrz75Nquv5OppFHtMpkMlO75zC+3b58/hMa2dfQ1HzmOxy++g0i+zxomdu4l+ajitWjfBtGIPFgAq0fquZphUd4Jb18ew5VhWX1UNIDi4Fmr6FrYfyAS3Zjw9azYTQ7xv9jKrNqLbyNfp1LUVL49+neWXtULOZe482ds/daPjfb2popo5MOcT/jhfxHiKPX3HEhnDFoEz7tsKyN3dnT59etO/f/9SjZWgcMsDDv9qlRhax5TjvCv4uYPZAhhNPNLTl8cCley6c/U00auVD808Ehi3M5NEQHFxYXxvH4b5KNlpu3iZ6HE98FzockIGIyN6+PJMVfXmPWlQqBNgwN15IccCld2bv9IPsXlXIja1Bk0begCgJ2/nk7FD6NiuLUFNW9K0w0Aen32YDP8ePNA915IBWgQzBrekXuPm1GvcnAZd3mVHVovoWDoVhMFoQFEU3Nzc6NK1M1OmTGbBrwt4/vnnada8GYpT3nyrk7LvK174zz5swc/ynxfb4eXMk6knsfr1vrRq2ZKgFp0Z8OYqLmo6toQ9fDV+GB3bBtOoTR9Gz9xHEj50u78nVVQAA43GTmZ8sDtRoTN4/N67adK8Le2HvcXvZzRqDhnPqKAcDbuezLq3B3JX62CCWnaky/D/ssuSV4YM1Bs1mZdCvMmIWMpbY/rTpmUwTdv15J6xH/FXBXkIYjRmPU71rlyZgfcO4JOPP+aneT/x2GOPUrNmzRI5purjh4+ik5yYRNSaH/n9Ug2GPdY3+6mfofZQnuzjSdgvP7MzJYnEZB3V1w//fFosxas9fdt7YTvyHUM7duSurj1p2zaEDkM/ZVta4dsLlOO6bNC8PZ1HTWfdVRserUYxqs2NR9EGGoyewsshPpjPrODtMfcQ3KIlQS3a03lKKGmFXCra8eX8EWZGrdufwcE5VolSfOk1qDPe+jVWLd5O8o0slUobZ999Vaxzm1uR24Cbn181sSfNm7egwZ0d6PzQa3y/Nx5TncG8N7EXeU2Wt0ue3zf2tjtG7nxiEmODXEg6OJ8XhvWk+Z2tadVzFC/M3kNMgeMrBc+69ahm0Dh35nyOdZENNBw7lZfb+2I9v5YPnhpMu+CWNGjWhtYPfM0hp3YgHG0HVbzbPsecmY8TFDmPfz39JbuTCrkBSrztL04dXM+iI/ecXW2GTvLZc1yzmahbv5addSGEEEKUFQXXhqP46qsnaWHeyXtPv8nSvwX1nDMms+c7t3j9T0fGtA6WyRHFKH8RDlZyZc4nLmM+/D92p6gEtr2L+jd2NzWnQxt3FFTq3dXm5q/fPFrTobkJy5Gd7EpRCRozmRfbVSbj2O9MfDCr39a67zg+3HgFpXp/pk7sc2u/vtDxvQP9U0MtmjXyRNXOsmXbpSJPBHHGdVys69wWz8Ing7Prpd6djzLvSl6d3pKKpVQMOWMlA/rfW2KxEuX6mtdNa7jzaicPglRIiLFwPvsy1dm2K5ZBv0XR/ddoHlydyiEN6jfxYmygQuylFCYuj6bXgiiGrE5idaJOtfoeDPbJ+nTjZl7c76OQEpPGtNXR9F0QRf+lcUw7aibOjrBWrUaVebKqSmZCOp+ti2Hgr1H0XhjNI+uT2ZLzQZhu5ceV1+jyc9Z/3f5IYr+T5o+VXfAaK3FxieiKirune1ZGFAWfFiP4YO4fbN+1h8Ob5jG1X3UMGKl6R4D9mXVWOrcpg8GIooCHuzu9e/fkk48/5uf583jq6aeckLqZiPlvMG1jMkFj3uNNZz4M0DWSo6NIytTQzPGEL5nJL0c1FGMM4duPcTXFgiX1MttnzWFdoo6hVl1qq4ChMfcNbIIxaT3vvzqLTacTyLRmEBW2hHdmbCLF0JBOITnqXbcSf+kisWkWtMwkLkdeIzWvG9ZQn4H33Ymr5TAzJkzhl93nic+0kJF0jYgDEUTf/hOv/8ZgzPpiCvD3Z+jQoXz33bfMmvUdnTt3dupxXL29cVdtpCSnYss8yLyfD+DSfSwjGxoAN0LGPkxw2gZm/34OjTSSU3QUH7983pIN6HrWchqBTQhpEoCbAuiZRJ+9SIJux/aC5LgubdYULu1dwAfzj2A1+NOkSWDWdWWoz70DmuOiHefbF99i3u4LJJo1NHMK0bEpha81pp1n2eK9pKvVGXh/B278gk29ox8PdPLAFrmK3/fk+FYojTbO3vuqOOc2t6K2ATk+nxIXR5rVhs2SzKWDK/jw2an8GQ1+vQbT3duJT9vsPT+GxvTrXRc1cx9fvPwxf4ZdI81iJunSQZb/vJZTBfaIVfwD/FBtacTGpt28jgyNGDSoGS7WcL4eP5HvQ08Rk65h0zJJik0g3ZnP1RxqBxX8OvybH797iqYXf+GZcZ/ZtwZmSbf9xaqDG0Vz4J6zp80A9LgY4vSsOhZCCCHKNwMNBwynow/EHdrAjrzeUuesMZk937nF7n/aOaZ1tEyOKE75i6SUy5y2l0170jA0aE/761FqQ8P2dAhI4ujRS6h3tifkegTdtVVH2nloHN2yg2ilIYMGN8fFEsaXL09j0aGsfltC5A5mvfI2C6/o+HYfRK+c0esCx/cO9k8VD7w8FLDFE5dQjAG/M65jZ46z8lNSsZQKyGjKmvucO1ZSu3btIqfZKNif0NFV2TKqCmuGBTCrhxcD/RS05Ay+PJxJdgRA10lM1Yi36miajWvJGmmKiV51TRjMGXy9PZX/Jdow23RiY9P57+FM0lQjbasaUBUTXWsZUTUzc7clsy7WRrpNJyXFwoYTmUQWVn+KkV71TLhoFn7cksTSaxqJmk6m2cbZaKtdwW9nKMO3ahnx8/NG0W2kpaahK5Xp+taPzB5ZB1N2O+RK7VoAGqpqZ1adlU4hOnfpzMouK5ySVkm68aTIx9eXwYMGZf+7t7d30RPVLrP47Xfo1Ow/DHvndULDJpNa3IzmeZwrRF62QtNAqvqokHb9y8N8lYtROkoVdyopgKkW9WuqqJX68eXufnz594SoXvMOVGIcO76xDg3rGrBd2M2O8857TdrrkybBJKclV2IMhqxAdo0aNahRowYAOuDn61vstL28vVB1M6mpZsDGhaU/89fTn/PwmI78MMOfxwdV48Ki11ifoIOSSkqqDbVOZSrnE3vUk3ewZEMMPQZ0441563k5PpIjB/cRuuxn5q45SWph2x1qcDUunz5Hqt4cT8/ra5ZlXys72HyqKI+ebVxb8zsbXuzAgD5D6fXxVpYnqNS/byjtXK0cXbKEIzcmt5RSG4eLffeV4tRzm/sQdrYBBdATd7Bhv5khvWvToIYKzlqdwc7zo5oCqVtDxXZxP3vznO1QyGHcXFAwY875Wy6X2jSoqWZdb6dLeKqDI+2g6kPvJx8BWwIbf/2F/8UWscPv7La/mHVQ/HsujzYD0M1mLDq4uJbUO7mFEEIIZ7FyfMHHrKnxKM92f5NF82rz4vjP2HQtR9/A3r5RQWMyO79zC+3729P/tGdMa3eZ4uw4oB1Kup9fmmXWE9m+8QAZPdvSrYMP8xcnUqtTJ+ql72LizHgmzriHrm0rsXSjmRZdOuBni2De5otoLr0Jqqliu7CL7edy9T1T97P1QAYj76lDUC0Vu067o/1TPY3UdED1xsdbhagixAGcdR2X5DjrBmfct060cmX5j7HBrbGSnKq4WIgyO/4eOptNJ91s40qihUOXMlh6MpNzhQ3zDAZqeYJqdGPqg25MzWOXKp4qqqpSwwNsKRYOFyVwpxqoWxlsKWb2Jxe+e0kpu+B1pVZ0b++Naovk+MlU8BvMo0NrY4jfxVeTP+aXnaeJTjcS0OtNln4xqPD0rlP8ejslncIcP36cJUuXOi09R1X2rsxzzzxr176apmEwGIiKiqJKlSoAJCYmFuv4esxG3p3yB22/e4Cpb27jk/Q89sEGuOLmVtTZjjYsZisoJkymnGlYsFh1UJRbnljmT8G1kqvjM8QVNWvtYt25j5KWLF3K8ePHnZqmI5o0acLQIUPs2lez2VAVhWvXrlGtWjUUIC4+vth58KrshaJnkJaRdW71pFB+WBzJwFGP8oruRzeXg3z46+GstZf0dNIydBQ3L7xMQF6NuB7DyjfGkHJgOP07tKJN6xa06VGPtt170ES9n/ErC9vuWJl0sxmzrqDcWNxaNWFUAau16D8tSwxlwcor3DuqCyPuvYOVv1fhwfubYEzbwYKl57LTLW4bZ/d9ae99Zce5L/odZGcbUHBB0G161v+z/6W4bRP2nx/FcP2XRWqRfqVizjCj44JLzvimnlUCNJtd57ZY5XWkHdRT2L9uHwHdu9Jjyhw+TXucl1cU5eeWTm77i1kHzuhX/K3NIGvtOZMC5sxCV5kTQgghypw1aidfvbuS7f/6nG/Hj+WbeT68+sQUll+8PsPCCWMyu79zndT/LHRM60CZnNG/LI1YhjPLXMiRiN22if2ZdxPSowPef+6jS9fGWPb+xpYd8XSMf5CuXVvhGppEjy53wOllbDirgYuTF3l1tH+qXeLk2XT0xvXo0C6QmSev4ujUB2dex84cw+appGIpRfTh9OmldKS8NW3alCGDB9u1r6ZpqKpKamoqnp6eAA4HriMOxjLuiNXhawwAnULbOVeDgpJjzFy0X27cXCe7LCfZl03wWvGmw/iJDK+hYo34i5XHNNSgalRzgbR18/ly/fHrC4ZbiI1OyvUiJR2r1YqOO+7uf7+F1AB70ymemOgYtm3d5sQUHRNYpQo8k/92q9WC0WgiKSmJzaGb2bp1G8fCj7FixXIn5UAnYdvnvPlrCD+OfJUXrrmjkJRju43kxBR0tQaNg7xRDsaW3IVuucS5SzZs3n/yZN/JbMp3/ScH1yO7nq5aO4SOtQyE5X7yW0THjx8v02unMBarFZPRyJXLV9i4aRObN2+mfoP6WTPGnaRyZU8UPZO07PUNLBz57Vf2jHmDRx7SiV/9Kksv3mjCzaRn6OiKB16eCuRXvxkXCJ3/OaHzAYMXTR6YxtypfejeLwT3latILXD7X8UrkOUql2NsqLXvIqS6ypELRfn6yWDP/y3h+IjnCBk5jA7xNbi/tkLssoWsirp59zjexhkwZLf0DtyXdt9XFH7uHTwTTlWpBSF3uoA5qzyAA21TAd839p4fQzNOX7Sh1u1E9wZfExbhyExpG7ExcdjURvj7u6OQmJVXy0XOXrKh1mpD22oqRy4VdL0Vsy12pB3ULZxa9BJPL3yBeV+OZtB7X3Dl2uN8vCe5ZNr/UqmDkutXKH4B+ClZdSyEEELcFmwJ7J35DA9dfY850wbx+TwXjI9OYsl5q1PGZA595zql/1nImNaBMjlj7Fu8fr69nFVmY4FxGQDbtY2s2PsKHTv0oVsDL/q01Nnz/nbi09JYvz2JB7r1pN3SJHrX1Tnx1VoiNMAcmdVvq9Oeu+sYCDuTo+/p0YYurd3AfJ4zF/N6aXhexXW0f5rG/zbsJKlfLzqMm0DfdW+xxq71Qm/WhVOv45Icw0LJxVKKqKzjJKqiQgGx67xiJY+MHUvnLs5dZtUuNo1LqWBzSWfSn0n8L7/3HikmzqeAWtmFDt4Kxx1dc+b6cVRPF9p4wYmkvHbSseo6oFCphKLMJb78s2o0YVAAgwseAXVo1WMEb85eyA9PNMXNco4F0+dxTANbbBRRFqjU/n5Gta2Op1EB1YSnp1uuCLtO1NVodMMd9Bneh3qeRgxufjS4qznVDY6kU/Fo1qyrNSMjg21bt/POO+8yevQYvvt2FuFHw9GdPIMYPZkdX0xjwQVvqld3y/U0TuNM2DGSdFc6/+t1Hm5dFXeDisHNi0DfSs59cqed4K91Z7AF3MfUj5+gV7NqVHYxoBrc8K3ZnO7t6hSt7rUTrFl7Gs3Uin9/OY3R7evi62bAYPKkWuNgGuf39sDb0I1rJyEhgVWrVvHqxIk8OW4cCxYs4PLly04/nrunOwrpZGTevCZtl1cyf2MCNu0yfy7YlOMN1jYy0jLQVS8qe+Zzzg316PlAT1rWqIyLqmAwGbEmJ5MJKAoohW0vboGsx1i38Qo219b8+7NXua95VTxdXPGt0477+zTF3kUBtFOL+WVnOoagEXzxVl/89PMs+W0bOX+d40gbZ7ZY0RUf2nTvQE13Aw7dl/beVyV9bh2heBAy7GF6NPSnktFE5VrteOSDdxhZUyV11zq2JuqOnYOCvm+w8/xoJ1i+4hgWYzOe++ojxnVpgJ9b1n7eAb7k09fPPn7KuXNc0wzUrV/75he2dpJ1G85hc23LC5+9wsBmgbgbTXhVb8Wg0X1peEvfsphtsaPtoK4Rs/VjHnlpMZGmpoz7ZDL3BJZQW2nvNVqsOiipfoWCV926VFUtnDtzscipCCGEEKUvk9OLX2fs66u5Vu0eps96k57+ilPGZHZ/5zqz/1nQmNbuMjln7Fu8fn5pl7nguEzWLtGsX7mbdM/OPD51OO3Yx5rNseiksWPNVhKr9ubFSQNooB9nxZozWbOhtQiWLQvHbGrB859PZljLqrgbXfCu04mnPnmHB+9QSAhdznpHFtp1qH+qE7/mG2YfzUStPogvfpvJq0Pb0SDAHaNqwMWrCo3aD+RfLw6l6fVy5q4Lp13HpTHOKqlYSgWSHSuJjy+VWInddAuhF6zYKrnxwt0e3O1nwNMAqqLg7WmiQxVDVt3pFtafs2BVTYzpVpkR1Y34XN/Pq5KKmx3H2XzeimYw8VjXygypasDbAKqqEOhrov71BGLTbNgUA52D3KhlAoNBpU4VE1WdFBAo4evQSLPxf3BifO5/19ESwvjprZd5f0dS1hOv2I0s3DCeLgN6MmVBT6bcsr9GRI4/nw/dSPiEFrS8/1M23n/9ny0H+eDeMXx/wd50KoYbAWmr1crOnbvYtGkT+/fvx2IpnVe+6sm7+fz9JfT8dhi537WaunU+84/15vnm/Xnvt/68d8tWZ/5M2sqROe8zt8c3jOvzErP7vHTLVsuBj+nz8E9EOjwZ1srROe/xXdeZPHvnEN6dN4R3b2zSU1k+oSsT1mYUlEC5dmM5mZTUVEI3bWZzaCjHjh1z/kOOPLh7VLo+8zrHP+qJrH6pMw1eyr23TnpGJrriQWXPvNNT/NvzxDuT6ZT7Vzq2OFav3UOaf68Ctxd/ZnAGe2Z9yrJenzKk1VhmLB6ba3t+j0Fz5+cay+f/xb87DaVqgE7G3t/4+dCt94ruQFt58dhxEvQmNBn7DX9VeYV2/17jwH1p3311vpBzX6qzrhUX6t4zkbn3TLw1Kwk7mf7pCm5MYLf/HBT8fTPbrnZHI+KnqXza6XsmhfTjjdn9eCNXtguavWuN2M/B1DH0a9WSqmoYl20AFsLmfMQvvWcwpvUjfLnkkb99LmeaxWuL7WkHc3/f2Ije+AHPflGXhS/35713dxL27GIuOv0lt/a2/cWrA/vvOUe40qJNM0zaafYfynP6ghBCCFGOWTm/fDJPVa3Cry8P4/PPzzD8yfnFHpPZ+51bWN/f0f5n/mNa+8eZzhj7FrefX9APJZ1f5kLiMudtgE7s+iVsmNiFQW2bkBY6hQ0xWR3y1J1r2BA/kOGtFdJ3/sSy7F/3aZycP40vus7m1XbD+WTRcD65mWssl1Yz9aO/ivCSOAf6p5bjfDPhNarMfJ9RTbvw7PQu/G3BVutR1D+XcexMHnXxgnOu45Ifw0LJxVJub5pmw2BQSUlJYfOmzYSGhnLs+PFSiZU4IuJoMotq+DCilifTa90aLLFEJzNmbRqXdDh7PInv7/DlX1XdeK6nG8/lSqewFupkeDK/VvdhtH8lXu5TiZezt+hs2BLN1PM6ly5lcrKliaYNvFnQ4Po79mwWvl4ex29OWCu7xKaOatGnCDt9hdjkDCyajs2STlLMBY7sWM2Pn7zEoH6jeGfdpZshHT2O1W+N4+U5mzhyOYlMTcOamUp81EUiDu1i56nE7J91aCd/YsKrP7DpZAxpmoY1LZYzB04RrSgOpXO70zSN/fv28+mnnzFixEimT5/Orl27Si1wnUUncet/+WBV1N/X6ck8woynnub9P/ZwNi4DzaZhzUgm5tJJ9m9ZTeipdKfVhZ68h+mjH+aFmSvYeTKKpAwNzZJKTORhQvdeKHKoXE/Zx2ePjGbCzFXsPRdLqlnDkhbHhfB9nE4yle6sUidKT89g8+ZQpkyZwsMjH2bmN98QHl4Cs/Pz4eFuQNHTScu053g66enpoHhS2SvvJktRrrB/82Ei49Kx2mxo6fGcP7yeWa89wasroqGQ7c4otS16HRNHPcPHi/dwJjYDqzWD2DO7+XN9OGk62HT7vvFTti1g0Wkrui2B9fOX8bcVSBxo49JCv+Clr9dz5Goyly5eyboPHLgv7bmvCjv3pdre6qkcWrWYrSejSbNayUi8yKG/ZvH8yPH8cDJHu+jAOSjo+8budif9GN+Pe4jHPv2dbSeukZSpoVkzSI65QPju9fwReibPpdwBSN3Nhp0pGFv0oEeVm9e/nridaWOeYOqvO4iISsFszSTxUhhr/gjlbO6VPYrZFhetHczg2Ny3+HhHKj7dXmLyfVVLpMNRKnVQEv0Kt1b0vtsX/XQoG520JJUQQghRujIInzuRt9fF4tX+BT5/pjmuxR2T2fmd6/z+Z/5jWrv7Gs4Y+xa3n1/KZS4wLnMjraSt/LbyCpqewtblm4jJLsAu/lwXhWZLZvPC1dcnaFyXHs634x7m2S9XsjcyjnSLmdSoCLb8+iGjH5rEsstF7TvZ3z/VLq9nyoj7eeS9+azZf5aopAw0TSM96RqnD23l9+9/ZXt8Vqb/VhdOuo5LYwwLJRdLuV1lxUo2M2XKFEaOfJhvvv2W8FKa5Oco3WLmm7VxTAvL4ECCjRQNNJtOXLKFXVHazfGN1cqvG+N4dX86e+I1UrSsl0SmpmucvJbJ6kvWAqfb6RYz36+P452wDA4n2UjTwGK1cSXOTKQ561cAtoQ0pm1P5X8JNjJ0sFptnI+2Out1tije/oG31MCNt3vu3r2HGV/NdNJhKo6f5/0AZK3FU5aLybuYTLhVciMpybFHGFK/ZSMkpB0Txmc9r/1w+vQyXcvJy8uLzIwMzA485OjcpXP2mtczvprJ7t17Sip7FYxC4IOz2DrtLnZN7sWji+IqzMOz8sVAw2d+Y9WEO1j8VA9e21qaD/BKnmfP99n49QCufPEA9393usAXIKp3PMwv696k9caXCZ6whtv3tyEVmYJPv49Y/0Vvzn48hBE/nC/yi15zmzD+WUJC2gEwYMBAJ6UqhBDidiLjPSHE7aQ89V+LEit5fdKk7DWvJ52sU1JZ+0eY3jASyCPeqrCo4iza+w9jtlgcDlwLAZCcnOxQYyzso1Zpy4A+bWlU3Q9PFwNG9wAadXmM959pj4vtHAfCKs6vPkTpStnyEz8fh+ajn6SXz+36ew+RzdiQ0eN64xO3ltmLLzgtcC2EEEIIIYQoOomVlF//9LXXhRDCKVyDRzB9xr145o4t6hqXV3zDgggJUYkisp7kx8+WMGzWA0wav5T/vb+LZHkScpsyUG/Ea4xrbmHX+zNZnygVKYQQQgghhBAFkeC1EEIUm4JLwgk27a5PcMPaVPN2hcxErpwJY+uyn/jql11E/cNeciGcSSdp+3+Z/EsdxsTruCpI8Pq2ZcKUepHw9euZ/JvzlgsRQgghhBBCiIpKgtdCCFFsOom7ZzNh7Oyyzsg/lMbJb4bT8JuyzkcJ0hMIff9xQgvZzXZlASPvXFAqWRJFkUHEkrcZuaSs8yGEEEIIIYQQtwcJXgshhBBCiDI3eMhgmjVpWtbZEEIIAMKPH+PPpX+WdTaEEEKIfzwJXgshhBBCiDLXrEnT7Le1CyFEefAnZRe8vvvuTri7uxMefoxLly6VWT6EEEKIsibBayGEEEIIIYQQohypVas2Y8aMBiA5OYUjR8MICztC+NFwzpw5g6bJmxOEEEL8M0jwWgghhBBClCujxz5W1lkQQvxD/Tzvh7LOAgCxsTHYdB1VUfDy8qRD+w60bxeCajBgsVg4fuI4YYfDOHL0KMePnyAzI6OssyyEEEKUCAleCyGEEEIIp3J1dcWm2bBYLWWdFSGEuC3FxsaiKkr23xVFQTEYADCZTLRo3oKmTZrysNGIzWbjXOQ5Dh88XFbZFUIIIUqMWtYZEEIIIYQQFUtQUBA//fQDw4cPx8PTs6yzI4QQt52Y2NiCd1DAaMyai6aqKvXr1WfI0CHZm318vEsye0IIIUSpkZnXQgghhBAVzIB77yUmNpbExARiYmJJTEzEYim9WdB+fn5U9vZmzJjRjBw5gpUrV7F06VJiCwvGCCHEP4y7uzsBAQH4+/vh5+9PYEAgfn6+VKta1e40bLoNBYWoqCiqXv9cQkJiSWVZCCGEKFUSvBZCCCGEqGDGPjIWz1wznhOTEkmISyAuPp74uDji4uOJi48jIS6e2Lg4EhISiImJIcMJ66b6+vmiaRpGoxGDwcCgQYMYMmQwW0K38H+LFnI+8nyxjyGEEOWZwWDAx8eHwMAAfP38CPQPwM/fD3//AAIC/PH186VKQCCubm7ZnzFbLMTGxhAXG0d0dDSa1YrBmP+Q3abZUFSFSxcvsXDRIkI3h7Js2Z+lUTwhhBCi1EjwWgghhBCignnooRGYjCa8Knvh5+eHn58/nl4e+PlMaa1MAAAU4ElEQVT6ZQVPfP1o1qwpfn5+BAYGYri+jipkBU9SkpOJi4vL/i829saf44mLiyXuerDbZrPleXx/Pz909Oy/G41Z6Xfu0plu3buxf99+Fi9ZwsGDB0v2RAghRAlwcXHJalv9/a63sX74+/lRrWq17H/L3bampKRkt6kxMbFEREQQe/3vV69czbNdbd68OYGBgX87vlXTMBoMnDx1kt9+W8iePbvRdf1v+wkhhBAVgQSvhRBCCCEqIIvVkh0ogVP57qeqKj4+Pvj4+ODv54ePrw/+/gFZf/f3o3bt2rQObo2vny8uLi7Zn7NarSQmJhIbG0tCQjxxcfHExsaRkBBP3Tp1MaiGvx3rxvqswa1b0fautpw9e47FSxazedNmZxdfCCGKxd/fj/vuG4ivrx+BAQH4+fvj5+9HlcBA3HLMlrZYLcTGxGbNlo6NIeJEBFEx0cTHxRMTG0NsbCxxMXFFeoFtdHTMLcFrTdMwGAyEHz3K/HnzCT92zCllFUIIIcozCV4LIYQQQvyD2Wy27CD3mTNnCtw3r9mGnh6eWWu1+vkRFBSEn58fNl1HVfN/L7jBkNUFrVunNi+/9BJjRo8hJTnZqeUSQojiaNqsGQ0aNiQuNparV7NmRp86dbLQ2dLOFBV1jabNmmKzWlEMBrZt38bC/1vIuXORJXI8IYQQojyS4LUQQgghhLCL2Wzm6tWrXL16tcD9Zn33nV3pKdcD3FWqBFKlys3ZhQaDAU3Tip5RIYQoph3bd/D+Bx+UaR5iYmLRrFbWrlvLH78vLrTtFUIIISqifIPXQUFBTBj/bGnmRZQiqd/S5evrW9ZZcJr+/frSIaRdWWdDCCFELkFBQWWdhWzePj6F7mO1WjEajei6zvnI8xhNRmrUqAEggWshRJkrqdnUjggNDWXp0qXEx8c7/FkZ7wkhbgflqf9aXKOqRZd1FiqsfIPXfn6+hEiAqsKS+hVF1bBhxflyEUII4XxGoxEPD/e//bvVar2+XIhOZGQkhw4fJjw8nIMHDpKSksLrkyZlB6+FEEJQ6FJOBZHxnhBClK4WXmllnYUKS5YNEUIIIYQQTuPj44OiKNhsNlRVRdOsnDx5ioMHD3LkyBHCjx0nMyOjrLMphBBCCCGEuA38LXg9YMDAssiHKCVSv6Iotm3dxoCtcu0IIYQonKenB4cOHebIkTAOHz5CxInjmC2Wss6WEEL8I8h4TwghSs+H06fD9LLORcWX/2vghRBCCCGEcNC5c5G88cYbLFjwK0eOhEng+jZnqNqJZz+Zz6b/7eNk+AHCti5hev8AlLLOWIVhoM79H7Js7WLeCLmdfxSrUn3Qeyxbt4J3OpsK2beilFkIIYQQpUGC10IIIYQQ4h/OlTYT/o/9e1fxUV//Eg7MluaxismlOf/+7iteGdSGun5uGA0ueAZWwzUzBb2s81ZqSr6+vGo1o1ktH9yUsr4ailNWBY8ajWla0wc3Oz5YfsoshBBCiPJOHnULIYQQQogKx6398/z45gDqVPXD16sSJqxkpCRw7XwE+7eu5Kd5KwmL17L3VxQFRVFRSyGWVprHKg7X9g8ysrEL5pMLeeXFGaw7k4JLYHU8UzLLOmul6napL2f4J5VVCCGEELcHCV4LIYQQQogKx1ClEcGNa+Ga/S8uuHtXoV6LKtRrcTeDh3ThpYdfY/kVG5DJvv8+SOv/lkbOSvNYxaFSNagB3komW+f+l5UnE9CBzKuRJJd11krV7VJfzvBPKqsQQgghbheybIgQQgghhKigrBz6YghNm95J/WZtaXX3PQz614f8fjwNQ/V+jH+wMYayzmK5peDq5oKipxMTk/oPWiZE3C5UVy8Cqwbi6y7zsYQQQoiKTILXQgghhBCigtLRMjMw23R0LYOkmAuEbfqZt7/bQYYOXpU9r6/ra6DhM4s4eWwbH3W58bI5Bf+7n+LzWb+wZsMWDh86yKnwgxzZvoyf3xlJa9+c6yo4sm9xj3WdW016jHuXn1ds4vDhw5w8vJu9G5aw8Jt3GRfiU/B6xZXq0ve5j1j011aOhu0nbMsSfpw6ipDA3KF8BVRfHvz+IGdPHM3678iPjL0j9xDCkfybuHvqZk6HL+HFJoZb0vAeOpMTJw7w4zC/6/nPK9197F//M58/3oHGbR/gtc/nsWH7bk4c3cf+tT8xfVQLfHIVXvEIYuCLn7Nkww6Ohe1j//oFzHiuGzVNOY4d/BBT/jOH5WtDCTt8iFNhO9m54h36++VVXzfOY216P/MBv60O5UjYAY7u3sja+VMZUierXIp/d974aQlbd+4hIvwwJ/ZtZNV3r3F/Yw8H1pM2ctfrf3Hq+E6+7u+Vq2D+PDRnH2fCfmDsHaqdx3O8rA6VQ3Gh0ZC3+OHPjYSFHSR85xp+//w57qlbya7SFl5XoPrdxdNfLGbvvv+xe8tm9u3fy6G1n/BAdRnaCiGEEBWRPKYWQgghhBAVn2rEzcOHms2689iTHXG1xbB163G0/D+AX8s+3Net2S0dZo+ABtw94k2CG1Xi/jFzibA6um9xjwW4NWHcrNlMCvHNsTaxB/41G+Ffsz4u++cyd3dC3mVza8bTs2YzMcT75iyWqo3oNvJ1OnVtxcujX2f55fzPilPyX6x0TfjWas3Q1+YwNNfeLnXu4qG3vsEv9QH+tfQaNgD3Foyf8z0vtPbKLq9brVbc9/wMgqtPYPBbocTrKlU6DmPMvTmP40VgFROZKflkzbUJ476bw6T2PjfPo0tVgoJr4Zluy/q71YcGbRpR0+X6ds+qNO0+lk/urIJl8Cssj7FnPruVsC3/I2bsA7Tr1ALX1TvIXnHcoy2dW7qiHdvG1igbeNpzvCKU1ZFyKB4EDxx28+8utWg74FladwrmndHPMu+UJf+i2lNXSjWGT/+Sid0qo1hTib2WguLpj08VE5mJNjvOpxBCCCFuN/J4WgghhBBCVFAm2ry2htMnjnL22CGO7Q1l3bx3GFk/ij8nP807m5MLXw5DT2L1631p1bIlDZq3p/Oo6ay7asOj1ShGtTEVfd8iH8tAg9FTeDnEB/OZFbw95h6CW7QkqEV7Ok8JJa3AAhkIGjOZF9tVJuPY70x8sCfN72xN677j+HDjFZTq/Zk6sQ+3TJS2xbPwyWDqNW6e9d+djzLvSj5BwuKW347zEtSiMwPeXMVFTceWsIevxg+jY9tgGrXpw+iZ+0jCh27396SKmlXeRmMnMz7YnajQGTx+7900ad6W9sPe4vczGjWHjGdUUI7Z33oy694eyF2tgwlq2ZEuw//LrjxjrQbqjZrMSyHeZEQs5a0x/WnTMpim7Xpyz9iP+Ot6MFdP3s4nY4fQsV1bgpq2pGmHgTw++zAZ/j14oLuv3bOvMw9sYUci+HXsQoscTwcqtelCB08bp7dt47zm4PHsLquj6Zo5u+ojHr+vG83ubEPrvk8wbXUkVp+OvPrKQKrkW2j76krxak/f9l7YjnzH0I4duatrT9q2DaHD0E/ZlmbnCRVCCCHEbUWC10IIIYQQ4h9FqVSPfk8/x4iWXoUHEHWN5OgokjI1bNYULu1dwAfzj2A1+NOkSeCtnWlH9i3qsQz1uXdAc1y043z74lvM232BRLOGZk4hOjal4GC8oSGDBjfHxRLGly9PY9Gha6RZzCRE7mDWK2+z8IqOb/dB9MprmRJ7FLf8dqSrmeMJXzKTX45qKMYYwrcf42qKBUvqZbbPmsO6RB1DrbrUVgFDY+4b2ARj0nref3UWm04nkGnNICpsCe/M2ESKoSGdQgJu5ku3En/pIrFpFrTMJC5HXiM1rxNqqM/A++7E1XKYGROm8Mvu88RnWshIukbEgQiib8T2FQWfFiP4YO4fbN+1h8Ob5jG1X3UMGKl6R4D95yNtF6u3JKBU70KvZjeC7a606Xk3vvpp1qw9lTXL3pHj2VtWh9NNZc/iX9kUEUO6JZOEyJ388PpUfrtkw6NDH7rkXtMl+5zaWVe6jg4ogU0IaRKAmwLomUSfvUiCLMwuhBBCVEiybIgQQgghhKigLOz/6D6Gz72ADQXVxR3/6o25+4HxvPFEb974bxwR905jW7ojaWpcPn2OVL05np6FrV3syL52ft5Yh4Z1Ddgu7GBzQUsw5MWlDkE1VWwXdrH9XK6lQVL3s/VABiPvqUNQLRXiHM6sffl3SrJXiLxshaaBVPVRIe16tNh8lYtROkoVdyopgKkW9WuqqJX68eXufnyZR/6q17wDlRjHjp9dB7vZcT6fJVaUynR960dmj6yDKbvgrtSulXVcVXVkGJbK9hWbiL1vCL17N+HTw0fRXFrSp1sA+vH/Y+VJzcnHc3I50g+zK8zMmL41qFtdhfg89nGxr66U5B0s2RBDjwHdeGPeel6Oj+TIwX2ELvuZuWtO5h+AF0IIIcRtS2ZeCyGEEEKIfwAdmzmV6HP7Wfr5JGbstmCo2p7OjXK/pNCOlMxmzLqCohYejnVkX7s+r5owqoDVWsB63flxWvjYbnmVX8cGuOLmVtT82LCYraCYMJlypmHBYtVBUbIGOddn6eZPwbWSq+NnRVGz1hrX809d8evNo0NrY4jfxVfPPUDHtsEENbuLDs8v4arjFUfa7lWsuQr1+t1DSyO4tr2HvlU1Dq1YzRnN+cdzbjlu1L+a/7m2t670GFa+MYbH35vD/23Yz3m9Om16DOOlz+fw8b0B9hdMCCGEELcNmXkthBBCCCH+WRQTLi5ZwTRVUaDwla/LD8tVLsfYUGvfRUh1lSMXHHhJnTmS0xdtqHXac3cdA2FnckQfPdrQpbUbmM9z5qKNkpvjYiM5MQVdrUHjIG+Ug7Eld/Ytlzh3yYbN+0+e7DuZTfmuiezgA4zr6aq1Q+hYy0BY7lnsgBpQjWoukLZuPl+uP44564PERifdfOFirjwYChqZZexl0bJzPDyuH4PbzMVnYC+qZOzki+UX0ACDw8ezj+Pl+DvFuxO92riAOZIzl3JeWznKbHddARkXCJ3/OaHzAYMXTR6YxtypfejeLwRWripSOYUQQghRfsnMayGEEEIIUUEpGFwr4aICqolKXv7UadGDJz74khdbm7AlHGDPKWtZZ9Ix1mOs23gFm2tr/v3Zq9zXvCqeLq741mnH/X2a4lLQZ7UIli0Lx2xqwfOfT2ZYy6q4G13wrtOJpz55hwfvUEgIXc76uJIM5mucCTtGku5K53+9zsOtq+JuUDG4eRHoW8m5c8O1E/y17gy2gPuY+vET9GpWjcouBlSDG741m9O9XZ2izeTRTrBm7Wk0Uyv+/eU0Rrevi6+bAYPJk2qNg2nsr2KLjSLKApXa38+ottXxNCqgmvD0dPvbMc0WK7riQ5vuHajpnl8g3Ur44sUc1O5g4GOTeKSvH4kb/2DN9ZdDOnI8RzicrmLAK8AfD5OKavTkjpYDeP3rqQwOhLiNy9mUqOddZnvrylCPng/0pGWNyrioCgaTEWtyMpmAUvo/LBBCCCFEKZCZ10IIIYQQooIy0uqFJRx74e9bdPNFVnz4NRtTSj9XxZPBnlmfsqzXpwxpNZYZi8fm2l5QMF7j5PxpfNF1Nq+2G84ni4bzSfY2Hcul1Uz96C9KNHYNpG6dz/xjvXm+eX/e+60/792y1ezEI1k5Mud95vb4hnF9XmJ2n5du2Wo58DF9Hv6JSAcmr99I9+ic9/iu60yevXMI784bwrs3NumpLJ/QlQnrNrJww3i6DOjJlAU9mXLL5zUicvz54rHjJOhNaDL2G/6q8grt/r2GvCYea+eX8XPo03zWZyDdtEhmL9hK0vW60mPtPZ5jHE5XqUz/6RvoP/2WVMiM/JPJH60jXs+/zPbU1Xn/9jzxzmQ6mXId1xbH6rV7ilhKIYQQQpRnMvNaCCGEEEJUOFr0KcJOXyE2OQOLpqPbNDJT4rgYsZc1C/7Dc8OG88LyS0VYN7rs2aLXMXHUM3y8eA9nYjOwWjOIPbObP9eHk6aDTS8gGpsezrfjHubZL1eyNzKOdIuZ1KgItvz6IaMfmsSyy6VwRjKPMOOpp3n/jz2cjctAs2lYM5KJuXSS/VtWE3oq3WlLiejJe5g++mFemLmCnSejSMrQ0CypxEQeJnTvhSKHyvWUfXz2yGgmzFzF3nOxpJo1LGlxXAjfx+kkE4oex+q3xvHynE0cuZxEpqZhzUwlPuoiEYd2sfNUYnYZ00K/4KWv13PkajKXLl7JP096HH/9vJLLmk7m4YX8cijzlm32Hs+xgtqbro3YQ2tZvuUQJy/Hk2bW0KzpxJ0/zJo5U3jwocmsvnbzusyrzPbUlaJcYf/mw0TGpWO12dDS4zl/eD2zXnuCV1dEF6WEQgghhCjnFG//wNtokT8hhBBCCFERvT5pEp27dAZg9NjHyjg3tyOFwAdnsXXaXeya3ItHF8XdTit5C1Fu/DzvBwC2bd3Gh9OnF7K3EEIIIUqUwiJZNkQIIYQQQojbiFqlLf1bwcmjZ7kck0iG0Zf6bQfxyjPtcbGd5kBYEWfZCiGEEEIIUc5I8FoIIYQQQojbiGvwCKbPuBfP3C+o0zUur/iGBRG342IoQgghhBBC/J0Er4UQQgghhLhtKLgknGDT7voEN6xNNW9XyEzkypkwti77ia9+2UWUwy8gFEIIIYQQonyS4LUQQgghhBC3DZ3E3bOZMHZ2WWdECCGEEEKIEqeWdQaEEEIIIYQQQgghhBBCiNwkeC2EEEIIIYQQQgghhBCi3JHgtRBCCCGEEEIIIYQQQohyR4LXQgghhBBCCCGEEEIIIcodCV4LIYQQQgghhBBCCCGEKHckeC2EEEIIIYQQQgghhBCi3JHgtRBCCCGEEEIIIYQQQohyR4LXQgghhBBCCCGEEEIIIcodCV4LIYQQQgghhBBCCCGEKHckeC2EEEIIIYQQQgghhBCi3JHgtRBCCCGEEEIIIYQQQohyR4LXQgghhBBCCCGEEEIIIcodY1lnQAghhBBCiJwmjH+2rLMghBBCCCGEKAckeC2EEEIIIcqVkJB2ZZ0FIYQQQgghRDkgy4YIIYQQQgghhBBCCCGEKHcUb/9AvawzIYQQQgghhBBCCCGEEEJkU1gkM6+FEEIIIYQQQgghhBBClDsSvBZCCCGEEEIIIYQQQghR7kjwWgghhBBCCCGEEEIIIUS58/859fmlysXg/gAAAABJRU5ErkJggg== ) ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABrQAAAFkCAIAAADuU0y9AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3iUxdrH8Xm2pW1624RkQ0looQSQUKUpICiKiB4FQRGxIIaiKPYuRVSkqQiKguUFDkhVAUGkHAkgPZAG6b1n07a+fyzEENJAYCH5fi7Ouczuk3nu2QSS/HLPjOTq6S0AAAAAAAAAND0yWxcAAAAAAAAAwDYIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAOpn323alhPn06K2vtZbXc+lql4vrd0eeWznW92VN6Q0AAAA4OopbF0AAACAjSjajP/0/fGdtf6+Hm7OTvYKyVihy8tMjj995M/f1n+//n8p5VUulmQymSRJcrkk1TespmPvsNaKhB31XXgtXdFcAAAAgIskV09vW9cAAABgC6oBnx3/6THvGhdSWMoTtr454fllJ0uueFjH+1fFfjlCkbBkRJ/XIg3/usqGuU5zAQAAQGPHsmIAANDE6ffM6qrRaNy9NV6Brdv3GfH4Wz8dL7DYNb979urZwzxuZPvfv9eY5gIAAIAbgXAQAAA0dYbysgqj2WIxG8sK0qIP/rxk6j2PLos2CLn//ZPvb3ZrfbfUmOYCAACAG4BvEQEAAKqxFB9etzneJCRl+06trTs0S5rHNqZlFaT/PrOdvOqlknOb+19asnnf8YTklIzzJw9uXvbu+G7eNbboKby6PfLKF+v3nI5JyM5My0yMPv2/X9d/Nfe1iX39q31HZhfQ/6nZP+04fC4pJTPhzNHt382Z0MP3KneKrmEuV3aXhpStbNZ77Iy5X635/cDRc4nJ2RkpyWcO7f2/57spr+Be9s0HT/ts7f5j0enpqRkJUcf3bPzhk8c621/BBUIIofQOH/fWN1sORJ9LzkyKPrn7xwXPD23lWO2a+qoFAABoSjiQBAAA4HImk0kIIclk8jrW4iq09y1as/jhYLuL1/i26TWyTS/rAJdcKbl0m7by29du91FUDufk3izEvVlI1wH97Q+t2pemv3ilR59XV3/9Qrj7xeTNs0XXu57pcsfdfaePeHpNgvHazKWBd2lg2ZLnnS9/NKu/6p9bKr2D2npIxeaG3kvZ5skfN70/0PPiJUqvoFCvANfTSy6+jPVeIISQ3Lq/sPLbV/p4XZynXWDHOx7vOOg/D/84eczMDYkX9n+su1oAAICmhs5BAACA6uzbDhsaIhcWQ/TpmFqPFFGEPL10wcPBdpa8w19E3NMlROsT0Kbr8Kff33CmqFrMJHnfN/+b1/v5yCviNrw1tndoC28fP03Ljr1e+q3IcumVMr8HP172QribIfHX98cNaKsN0LTudf+bWxIMisAR7895yO8qvnWrYS4NvEvDy7YyJax+akDHNs29fQMCQ/sOn7Ym3tTAe6mHTJ/Z31MqObHymbu6Bfn7+wSFdhs6bvq7605eiCnrvUAIme8DH3/zal8vSXfquxdGdg3R+gZ16jdh/q50k32bR5Z+PbWzqgHVAgAAND10DgIAAAghhJAp7BycvQJadx0wavLUcWEqyZy55fN1SbX1kzn0feq57k6SKfGbpx95ZXehRQghys9Fbph/SnQe/uWIKt9kKTtPmnWvRm7O+/Xlh578PsWaQRmKMuMTcw2Xpmyqbk+/NMxbKjs0e+yTC87qhRCiNH730slP+ezcNiVk4CMjAn5aVmtBDZ5LA+/S8LIvMBcnnolOzjUJIQyZMYcyG3wvqVm71mqZMBz9fsGaw2lmIYQ+O/7Ib/FHLo4sr+8CIVRdnn75bh+ZKX3t1P9M3ZRtEUKIjBOb541Jsvy67cWwTk+/dN/KR9fmWOqsFgAAoAmicxAAADRxqjsXnMnPySrISstMjD69f/OqDyb08pHrU3e+Nf6lTTk1xmBCCGXnOwf5yoXh9A/L/iys7SIhhBCKDvcMD1YIU8JPn6xNqbM7TdllxPAWCkv5vlWrovVVHi8/unNflllShXbtbPfv59LAuzS87H89I3N+do7JIpRdHnq8l6e8hmHqvUAoOt0zrIVCGGN+XLQtu+qHo/zEis9/11kklwEjBrhyXDMAAMBl6BwEAACwsljMFiHJJFFx6ptnJ7y7NVZXe+gnuYSE+MqFueDkifN1J2eSc9vQILmwFB3564S+nitDWvvLheQweGF89sIaLrD39nWTibKGtA7WPpcG3qW8wWX/+xmlZ29e/t8Zt48J6jZ106ERe37+v/9bs27zwZTSypffUt8FkkvbdoEKYc47diS62raMlsK/D8cah3e1a9M+WC4OX82mjQAAAI0ZnYMAAKCJ0++c1s7dy8fNy9ddO3TO0VKLpArp3d1b1NkOKDk6OQkhLLpiXT1ZnaR2dZYkYS7Mza+n/05yUqvrvsLerp7OwQbMpYF3aXjZdWjojCx5218eMX7+5qgCs3OrQeNf/XJzZNSfX7840O/i77Hru+DCjSzFhdX3exTCXFSoMwshUzur+c4XAADgMnyLBAAAcFH58U8j5h3UCbs2Ty6Y1aeuXMtSotMJIWQu7q41LnOtcmVFebkQQnJ0cqxnVaultKRECGHOWf2wj5eP2+V//O9ektDgI3Vrm0sD79Lwsq/NjPRJOz4a169Tx6GT3lm1P7lC7tbuntdWr3mzp+PFseq84MKHQ3J2dbnsu1uZi6taZr2E84gBAAAuQzgIAADwD/3Zr6bNjywRypAn5r7S06nW6yzF0WdTTEJy7tazg7LOES0FiUkFZiFz7dS5Rd05oqUoLi7TJGRuXW9rfS22fql5Lg28S8PLrsMVz6gi4++Nn06/v3vfp76PNwi71uMe6+fQkAsufDhkLmFd21S7keTS5bYQhbCUR5/mPGIAAIDLEQ4CAABUZTi7bNZnxyuEKmTiB890UNV62fGtvyYYhaLlmJkPB9UZfOmP/P5nnlkoO4x9ul/dR2IYjm7flWESijZjIoZ5X4vDM2qcSwPv0vCy6yrg6mZUnrB12cZzJiE5+vjWeO/LLjAc3/rreaNQtH7kuaFeVd/DvsPEZwapJUvRns17CupcKQ4AANA0EQ4CAABcSn968Rsr443CvtOz744JrO27Jf2RL2ZvzTLLPIbM3fDjK/ffpnVVySS5nYumRTO3S/Osoh1fLD9dYZFrH/9i9fsPhrfydLBz8mnT9+FXn+nvcumV5fuWfrqv0Cz3f3Dx+q+mDO+qdbdXSDKVs6Z1j3vH3dX2KtoJa5pLA+/S8LLr0LB7qfs89erTd3dr6eWklCS5nVtQ+EPPjGghF+b8xMQCS0MuEPojX8zdlmWW+z+46MdPxvVs7qZSOfp2uPvFVaund7EX5SeWzduYTTYIAABwOU4rBgAAqK70rwUfbH5gxf1e/afPuHP9jO1FNcVK5oz/znhc67by1X4Bg1/4cvALlz5bdQWr/sSnz74etubDof49nvt8y3PVhrFY/hnddP7rZ59p8cMXz4a1Gf32ytFvVx3k0Gt/bj+beMXb5tUwlwbepeFl16Eh91K2G/7U8881nzb3kve0mHP3fPz5/nIh6r9ACGHOWDfjieae377Su/OETzdN+PSfy8pifnruiQXHKhpSLgAAQJND5yAAAMBlLLmbP/7iaIVF7j9q6sPa2r5hshQe/uShfoMnf/T97ycSc4srTBaTvjQ/49yJ/dtWLVm65fw/AWHF2W8fHTzqhS9/PZKQW6I3lBemntr9w5wv/sgzC0tpSUmVmM2c9ftrwwfc99IXG/+KySjSm8zGcl1u4ukDm1ZtPFZ6zebSwLs0vOw61H8vS/a+//tp+9HzmcUVRrPZWFGUEXdk24q3Hhwy7qtYQ4MusA5TEPnR6AF3z/xi86Fz2Tq9viw/9fQfqz+Y0H/ItPWJhpqLAwAAaPIkV09vW9cAAADQBMkCJ60/9GEvsWdm2IPfZdwyS15v0bIBAABQM5YVAwAAXGeSa5/HJ7bO+t/hM0lpmdkFFTIXTauugx9/7eWe9qLot/U7sm7OiO0WLRsAAABXgnAQAADgOlOE3jt15tMB8uqPW/SJP7/68pr0K95H8Ma4RcsGAADAlZDbOzrZugYAAIBGTWHvpLZTyZR2Dvb2SqXMoi/KSji5f8vXs2c8//GeTFP9A9jGLVo2AAAArgR7DgIAAAAAAABNFKcVAwAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRClsXQAA4Ppq27bN/SPvt3UVuOVt+HnD2bPRtq4CAAAAwDVGOAgAjZyXt3ff2/vaugrc8vbu3ycIBwEAAIBGh2XFAAAAAAAAQBNF5yAANBULFy+NjDxk6ypwiwkP7x4xZbKtqwAAAABwvdA5CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CADAzUVyH/DGj1v2zrnTztaVAAAAAGj0CAcBALi5SPb+7Tu28HZUSEIIYdc14v/+Prxt7hBPydaFAQAAAGh8CAcBAPWRu4Xe/cycZWv2HDh49uSRY39u+r9PZz7a09/+Ot/WacSis9HHNz/bSn7p45LPf1afPB2z+lH/Bn0Rk4dOWPrr798910Zex0VOIxadjT59/rI/Zxfedb2nWS9JkiRJJiMaBAAAAHAdKGxdAADgpia53TZlwfypPb3lF8MpO99W4cNbhd81eszat55579ckg03rq5/MLah9iF++6lYN1yqOfPZQl89sXQUAAACARopwEABQO3ngmI8XTuvlYkr/6+sly9fuPp5YaHbyb3f7iMenPXlHu4c+XFaUMWr+sVJbl3mNGE8tGDXy83iTresAAAAAgBuGcBAAUCt1/2en9na1ZP46c8xLG9MuhGb6xKObFh/be/TVNV8+0vrRaQ/+9MS3KWYhJM8+k157rH/7VoH+Xi6OSlGen3Rs548fL/jpaL6lckDJKfjupyZPvKdnWx+7sszofRu+nLdsT8q/7z10aD7kiWcn3du7vb+TOT/xyK51S5f8FJldJeWTt47YeCJCCCGEOeuncYPeO3BlN23Q7ISD9s7Hn3ny3j4dAlyksvzU6D+Xvv7ez4mmesuTeXR+ZPIzYwd3aempLE0/+7+/Cn3/WTEtD3n2p20RfuufGvjyXkNDK7EPGDju6Yn39e2k9XQQ5YXZqediTu345uOvIguu7IUFAAAA0NgRDgIAatV7+ABPqeLQ8k82p1Vrp7PkH1iyYOfwhXeF3XOH/6pvU8xC5tFp8Ij+7Su/rjh5terz8GthrR1Gjfs6xiiEEMKx45QVX03r4mwNvuwDO494fmGYf8R9r++pmmtdMfv2Ty9b/lK464U8zbd1/0de6d2v8wuPvnJZ2VetAbOzazvpyxWzerhdKEPlGxwWqC4z11ue5NL79e8WPR5ib133bKcNG64VQoiKq67Evu2kZctnhbtf3KbQyTOgtWdAS9XfXxMOAgAAAKiGcNAGtm7dYusSmorZc+bs27vP1lUAt7A2wU4yU/Te/Rnmy5+zFB7Ye9I4rE9w25ZykXLhAkvRL6+OnrU1Q2dy8Au7/62PZw7uPHZs11VvRRqEkLce/8aUMMesPQtfnft/BxLLXdsNmzn3jQdGThm7ct/i2NpSPEWHaZviptXwxMXOP3nwuDemd3cpP7Pu7beWbo3KV/nf9tCsd2YOHPb2S7v2Tf/1Quxoilk4avSnZ+vOCqvfy1K67dkeM3/TV75d1+xajH1jRrhreczPH7735bbj6WV2HtpWrvk5suCJdZen6DBx1vhgVdGxVW+9/82Os/kKn/YDx0x7/YnuznVUWmclrR5984VwN/25LbPfXrzxWJpOOGjun7f9nT51zh0AAABAE8VpxQCAWjk7SsJcmFtYQzYohKUkP7/cInNwcvrnF00WU3F2VlGFyWzUpR7+4cNVp4xyz7ZtvWVCCHmbEfe0VRTt/GDmst3xBRXG8qyTG95ZuFsnD+kdrmk/ZX1c5RnBUZteDK3rZOFLyEPuvS9UZTi56IV31x7PLDXoCxIPLHvxrTXpFvcB997hfk1PIalrdi3vGdHBznBiYcSb30cm5VcYyosyY47GZEv1lSdvM/TO5rKKIwtemLfxZGapQV+Uemzz6u1xdceYdVYy/O5QlensF9Nf/y4yuVBvMul12bm6f9OaCQAAAKARo3PQZvLy8uPi4mxdRePk7u4eEhJs6yqAxqC41CJkrp6uMpFzeVglObm720uW0pJSY83vbUqLTyixhKrVTpIQQhXYMkAmcxi6KHLooksv8w/QyEtqK6GGQ0Ikn/+s+v3NcOsbqqDgAJk5+eD+hCqXlPy992j5I3cFBQfKRF5DJ3uFB5JcOjtFUEhzuTk58kDSpe9db3ml/s2bycwpfx9OrzGBvepKDvwRd7MfIw0AAADgZkA4aDNxcXELFy+1dRWNU3h4d8JBNAX9+/cfPPjOyEOHDh86nJaWdj1uER1Xamnbqm8v38/j06pnV5JLr74dFBZjXHStaZpFr9dbJMm69Z3FUkvzmmTnoDg7d1Tw4qur8Zr2Bl6JS2YnyWSSEDVMsb7yJLlMCCHJ/s00LqlEplTIhDAaOXMZAAAAQEMQDgLArcpgMHTp0qVTp05PP/VUdk7OgQMHDh86dOrkKb3hmrWM/e+X3bl339d90owRu17eeMnhHpJ7r+emDXaTyo9s/f2y3LDmclMTUs1m141PDnljd+m1KlAIfWJ8ilkW1KNPkPzkuYsVOnW9vYu90CedSzELIRmNRotwdHS8njGiITUh1SzThvcKlJ+s2iRYb3n6pPgUs6x57wGtlpyMuRYfOENGWo5Zpr0t3F92KvmquxEBAAAANBXsOQgAt6qyslIhhFwuF0J4e3ndPWz4u+++t3bd2tmzP7z3vnt9fLz//S2K//jiswOFkuauj374Ytao7q28HJQKO7eATsOf/XTt0jEhCkPs9wvWNDCBMkX/tuOc2WvE2/Mm3tFe46KSy+T27gGhA7oH/avfU5liNm2K0is7Pv/JG6M7+ToqVK5BvZ/66J2H/KSCPZt35lmEsGRlZFvkfoMfHNxCrZDbe7S6LdS/wVsaNriM6F+3x5uUnacuevfRHs3d7eVypVrTJqyNW2w95ZmiN285Y1C0f27x3Em3t/Kwl8vk9q5e7lefZBrP7NiVbrbrMvXjmSNCfdUqO/eg7qMGt1Ndy9kCAAAAaDzoHASAW1Vp6SUNeAqlQgihUCg6dOgQGhr69FNP5WTn7P/fgWqXXRlT0vcvTvVYMD+iR++nZ/d+uupTlpKza998esHRBo9uPLXig68Hfj5p8Izlg2dUPmo4Om/wmG8Tr77FzRS76t0F/ZbP7P7gR2sf/OhicYbUX96e+1ueRQhhStqzKyqiY6dR83eNst7y2IfDx32VdPktazoZ2Xhq3ogxn5+rtwzj6RXvf9lv6eQOI9/7buR7F6oo2RzRL6K+8mK+fXt+769mhQ99dfnQV6uMWHElr0IV5YeWzd90x/yRnccvXD++aoVXOR4AAACARo3OQQC4VRkqal6FKpPJrO2EXt5edw8f/sjDD1sf1wYGXMVdLPmHFj4x8v6Zn/9335mU/FK9saI4+/zh7avennj//W/8mnQlC2EtxYfmPDpm2tItf8VmFZWbTIaSnMQTew4n66+irKrKor6YNGbyoq2HE/PKDPqSrJg/f5z96H9mbbq4DtoU+23EzG92x+aUmkzG0txzR+OypWu/xNiiO/LxY49GLN12OCG3RG8ylOYlRx2JL1JK9ZUnys58Nek/E+av2xedWVRhMhnLi3OSoyJ3/nfPuatbZmzO3vHS2GfnrT90LrfcaCzPPRe5cWdUqUWYLawyBgAAAFCd5Op5Ddad4Yps3bpFCBEZeYgDSa6T8PDuEVMmCyFmz5mzb+8+W5cD1E91kdpZrVar1U7OamcnlVKlslOpndRqZ7Wz2lmlVKrsVOpKzs4qpbLekY0mk0J+YQ3toiWfHzwYeZ2ngpuQ5P3Qsr3v3nbwjTseW3sFhzdb8S8qAAAA0LixrBgArhlJkpycnJycHB0cHBwdHB0cHRwcHdVOagcHRwcHe0dHB0dHRydHJ0cnJwcH65MODo6OarX68qHKy8tLS0tLS0vLyspKSkpKS0vLSsvyCvLLzieUlpaWlJSUlZUZjYZXXnmlxkrMZrMQQq/X//HHnuTU5EkTnxR1HBeMxkXm021YZxF7+nxaTmG5wr1lt3tffLaHyhx/9GShrUsDAAAAcNMhHASAmqlUKrVabW3lUylVKpWd2tlJrVZbW/nsVCqVyk6tdrK28Vn7/tzc3GSy6ts16A0GXXGxXq/X6/U6nU6n0xUUFCSnJBv0+ooKva5Ep9PpdMUlen2F3qDX6XS6Yl1xcbGhAScOS5JksVikS1fImkwmuVyelJS0efOW3bt3V1RU9L2977V8XXDTswt7eM7C4eqqnxcWU9qWz3+IMdX6Pg0Q1rmzvrwiKzsrP6+gsIicEQAAAGgkCAcBNHKXr9hV2SkvBH+1rNi1Pnv5UNaYzxrwXQz7StIzMnTFOr1efzHaK9GVFOt0On2FXq/XFxQUWJv4rgeLxVJRXm7v4GB9w2Q2W4Q4+L+/ft64MSoq6jrdFDc9SVUQvTuyZViIVuNqJyoK08+d3Lvp28XfH8z6d5+Jd95557Bhw6z/rTcY8nJz83LzsnNz8qv8f15uXl5urr4B0TYAAACAmwThIIBbhjWzU6lUF4I8J2drK59KeSH7c1Y7q9VOF6LAC/vyOStr2pivasxnDfIqYz5rK59er9dXGKwxnzX7KykpuQmX5ZaVl9vZ20uSlJmZtXHzpp07fy/R6WxdFGzLUhi5PGL88ms+7vyPP448GOnh4eHh6WHl6eGh8dW0btO6p0dPb29v+cXdLa3RYUZGRl5efl5ebm7eRbl5WVlZ1y8uBwAAAHAVCAcB2EDlit2LZ244V67YtbNTKVUqa8xXdcWuq6trZfRQ6fIVuzpdiV6fV6HXV8Z81VfsFhUbjI2nramgoODs2bObt2w5cfzETZhdopHR6/UZGRkZGRk1PqtWqz08PTzcPTQajYeHh6enh4eHh1YbaE0SrdcYDIbi4uLKtDA9PSMvPy8vNy8vPy8nO6e0tPQGzgYAAACAEISDAP6lqhvzXb5it9rGfNakz8nJqdo2eaJKK1+VpK/mFbv6Cr3eoNcV6woLC02mf7WHWiPw0ksvk6fgJmGN55MSky5/SqVUenh6Vm059PDw1Gg0YWFhXl5eCsWF70ZoOQQAAABuPMJB3ELkQaPeX/RMm79ef+jDSKOti2lsalyxe7GtT13zil0XZ6WinhW71Tbm05XoLkZ7JVVX7JaWlvJj/9UhGcQtQW8w1NZyKJPJ3NzcvLy83N3dvb293D08vD29PDw9g4NbeXl5OTo6Wi8zGAz5+QU5Odl5uXl5eXnZOdn5eQWZ2Zk52dl5eflGI18UAAAAgKtEOIgbwK5rxHfLxzvveHXcrO25/2bdo3Ng+/aBzscuazpDVbWt2K22MV/lcl21Wu3i4lLZuVNJbzDoKyqqLNetYcVutY35dMXFHEQA4IqYzWZrd2CNz9bYcqgN0ob3CK/acqjT6TIyMqotVc7IyKDfEAAAAKgX4eBNT+bSdsgjj4+6o3fH5r6uKmNh5vno4wd2bPhu3f9SKup+T3nohEUfP6rePHnCkmgbL72UJEmSZDIyvStUdcWuSqlSqewqN+a7fMWuNelzc3OTyWTVxqltxa5Br6+o0FduzFd1xW5RURGdOABsro6WQyGEWq3WaDQeHp4eHu5+fprKpco+Pj7WfwmNRmNRUVFeXl5GekauNTHMzMhIz8jLy8vPz2ebTgAAAEAQDt7kJJdOT87/9KV+GsXFWE3lERDaK6B9mFP0tr9SKur+qUbmFtQ+xC9fZftIruLIZw91+czWVdiO6qLKjfkuX7GrUiovrOe9eMiuqr4zdqut2K22MZ/1BF69Xl9QUEDjDIBGSafTxcXFCRFX7XGlUunp6enl5eXj4+3l5eXl5eXt7d25YydPL09nZ2frNXq9PjMrMzcnNzs7JysrKyc3Jyc7OysrOyszkw5oAAAANCmEgzcxmd+oOUtm9Xe35Bxd9fmyn3Ydjc/Wq9wD2nW7/a62GfsL6XewgRo35qt7xa6zs7Pyspjv8hW7+opLNua7fMWu9WKbzBoAbi2G2vsNq65T1vhq/Pw0Hh4eLVo09/Pzc3Jysl5TbZFyZbNhbWufAQAAgFsa4eDNy7H3My8O8BA5u159ZPqapAsLPCuy4iN/iY/85cI1kueAVz6ZOqxNgK+LnaUsJ/7wb8s/WbwhuuSf4FDeOmLjiQghhBDmrJ/GDXrvgEFITsF3PzV54j092/rYlWVG79vw5bxle1Iq+yTsAwaOeyIetXgAACAASURBVHrifX07aT0dRHlhduq5mFM7vvl4eWTBhWEdmg954tlJ9/Zu7+9kzk88smvd0iU/RWZbVy5LrmEPTX1sSPfQ4OYaNwepLCfxt3fGvxv3n5+2Rfitf2rgy3sv3sZBe+fjzzx5b58OAS5SWX5q9J9LX3/v50RT/TO6RupYsWtnp1KqVJUxX0NW7FZZrlvzil29vuJiW5+uuLjYQFsKANhCHeuU1Wq1h6eHh7uHRqPx89NofDVarTYsLMzX19d6wHrlYcoZGRnsbAgAAIBGg3Dw5tVrxJ0+Mv3RFR/9N6n2rd+Mbq26tg5QCSGEUPu2GzD+ow4+hvte3JxTe5jm2HHKiq+mdXG2plz2gZ1HPL8wzD/ivtf35FuEsG87adnyWeHuF/cHdPIMaO0Z0FL199dfRxaYhBD27Z9etvylcNcLIZlv6/6PvNK7X+cXHn1lc5pJCJlPr9Hjhre/+Inl7O2jrLi83c2u7aQvV8zq4XZhEJVvcFigusx8lTOq3V1Dh/bs2cPRwdHR0dHBwUGtVjs4Ojg4OF6+YtdkMpWWlpaUlJSWlpaVlZWWlZWWlubl5SUlJZeVlZWWlZaVlpaVlel0JWWlZWVlpSXWyzgoFgAaC+uvd5ISk6o9rlQoPb08NRqNNTr089NYdzb09vaWy+VCCIPBkJubm5d3YU/D9HRrfpiRnZ1tMtl4z18AAACgXoSDN6/2rdUyU/yf+1Lr+MHCUrz/o/EjX4tPztYZlK7aXk9+uHjiwAcGuG9Zl3chSzPFLBw1+tOzlWPIW098Y0qYY9aeha/O/b8DieWu7YbNnPvGAyOnjF25b3GsaPXomy+Eu+nPbZn99uKNx9J0wkFz/7zt7/SpfPfgcW9M7+5Sfmbd228t3RqVr/K/7aFZ78wcOOztl3btm/5rvuVCWTvefuSVTSkFJgdfjUOhQfhfUrW8xdg3ZoS7lsf8/OF7X247nl5m56Ft5ZqfY2nQjK6Eu4dHRYW+pKQ0JyenrKysuFhXZo30ykrLyspLSkpKdCVlZaWlZWV6vf7KhwcANH4GY83NhgqFwsXFxcPDQ+OnqVyhHBYWds89Afb29tZrrCuUM9IzqoaGdBoCAADgpkI4ePNydpKEOT+voM6fHyTJrePDL73Ws32Qn4eyJD3HLBcKXz8vmcirOVKUtxlxT1tF0c4PZi7bXWgRQmSd3PDOwr5DF9zRO9xr6TmX4XeHqkxnP5v++nfR1nWvuuxcXZVFyiH33heqMpyc98K7a+NNQojSxAPLXnwraMsXjwy49w7339ZZt2OyGPNTU3JLDUIY0hKLhJBfWkPLe0Z0sDOcmBvx5vfnTUIIUZEZczTzKmdUpx9//HHf3n1X/n4AANTDaDRaNyKMi6t+IoqHh4ePj7ePj6+vr6+vr4+vr6Z3797e3t7WLWitS5uzMjMzMzMzMjIyMzMzMjKzsrKKi4ttMQ8AAAA0dYSDN6+SMiFkrm6uMpFVSywmufR7feXyR4KUF5YA22kDhRAmmaz2D6sqsGWATOYwdFHk0EWXPGHyD/CTKbxCmsvNyQf+iKtlRzxVUHCAzJx8cH9ClZJK/t57tPyRu4KCA2WiIXu1K4JCmsvNyZEHki6b11XMCACAm4w1NDx7Nrra42q1WqPRVHYaNmvWrFu3bj4+PtYNbavtaWg9CCUtLY0tLAAAAHBdkbncvGLPl1natOjZ3XtpbEaN3YOSx52P36+V5x9c/Ma87/+Kzy5TeN3x2s8L7q1rUIullsW5kp2DnSRTKmRCGI219+hJVzCBWseQySQhairkamYEoMGGDR3SM7y7ravALcbd3d3WJTQeOp0uLi6uWqehUqn09vbWaHx9fTW+vj6+vr4tWrbo1buXq4ur9YL8goLMjMouw8z09PT0jIzcnBzWJgMAAOCaIBy8ef3v97+Kht7Rc1LEkB2v/5pdww8AMi+NRiVKd6xatPOsXgghDLnZRRX/PG8xGo0W4ejoWCXRM6QmpJrNrhufHPLG7ssbERRd0nLMMu1t4f6yU8k1/cihT4xPMcuCevQJkp88dzFCdOp6exd7oU86l2IWovphvjUwpCakmmXa8F6B8pMJl+SQ9c3ISi7n0xa4KiEhwbYuAUB1BoMhLS0tLS2t2uP29vYaja+Pj0bj5+vnq/HV+PYID9f4aawbGhqMhvT0jIz0tPS0jPSM9LS09PT09KysLKOx9kPMAAAAgJqQsty88n/9fPljfWd0vHfBTx4rFq9Yv/d0Yl6FzMmzefvwQb2VexZuOJublWUQrXuMGtsteu3xdJ1ZoVbbK4S4mKZZsjKyLfLQwQ8O/iFmR5LRpXkHv7Kj0b/tOPf0MyPenpcgW7r1UFy2zqR09WvV2U+371Ci0Xhmx670x8d3mfrxzOx3Vu6OLVD6dRo6uJ2qsiZTzKZNUZNmdHz+kzdy3vx8W1S+stlt/3n5nYf8pILfNu9s4JEhpuhft8c//WznqYveLX3/q63Hk4tMDt4tg11zTsTUMyOhNxgtklvXAT0Djh5IKeUISABAo1VeXp6QkJiQkFjt8Wprk4OaB/Xo2UOj0VifzcvLS0pKqrowOTU1tays7IaXDwAAgFsG4eBNzHD284iXfZZ+MLbd7ZPn3D656lPG07KNm86c37Xm9ym33z3ozR8GvfnPc6aYi/+RtGdXVETHTqPm7xplHfDYh8PHLV/xwdcDP580eMbywTP+udXReYPHfJtoLj+0bP6mO+aP7Dx+4frxVe9XOXjsqncX9Fs+s/uDH6198KMLD1oMqb+8Pfe3Bh8nbDy94v0v+y2d3GHke9+NfO/CGCWbI/pF7KhnRilnzhZY2rYd//lvPi+GTv21gfcDmrh9e/fdvfceW1cB4NqocW2ySqnU+PtptVprYqjRaMLCwip3M7z80OSkpKS8vIbsEwwAAIDGj3DwpmZK2/nmw2e2Pzhu7PC+XYP9PJ2U+pKctPMxRw/8tj/fLCx5v7w+6YWMqROHdQvxdZIby4sL87PTk/6KK7TGdKbYbyNmurz1/L09WrqrKgqSTsVlS5Kl+NCcR8ecfmLiI4PD2wd6OsnL89Pijx1O1gshhDBn73hp7LMxEU892L+j1lUUJp3Yd049+I7WZsvFVcZlUV9MGnN+4uQn7+0V6q825ycc+X3dkiU/RWZfQR+fRXfk48cePTvxqceG92jn76YyFmacPxVfpJTqm1HpngUzlqhfejDcLiX9Wr7QAADcyvQGQ1JiUlJiUtUHrYmhv5+fn5+/n5/Gz8+/T98+Pt4+crlcCFFcXGzdvjA9LS01JTU1NS0lLbVEp7PRDAAAAGAzkqunt61raHK2bt0ihIiMPLRw8VJb11IvyfuhZXvfve3gG3c8vrbBrYG2Fh7ePWLKZCHE7Dlz9u3dZ+tyAAC4WSgUCm8vb/9mfhqNn5+fxt/f38/f30+jUSqVQojCosLU5NTUVGtamJqampqemm4wGmxdNQAAAK4jOgdxCZlPt2GdRezp82k5heUK95bd7n3x2R4qc/zRk4W3SjIIAABqYzQa0zPS0zOqN+B7eHhotVqNRuPnp9FqtR07daxclVx1H8OkpOSkpMSsrCzOSgYAAGg0CAdxCbuwh+csHK6ucr6xsJjStnz+QwynfwAA0Gjl5eVV24VQqVR6enpqtUFabaB1H8Pw8HAPDw8hhMFgyM3NTUpKSkpKsm5iaGWj2gEAAPCvEA6iKklVEL07smVYiFbjaicqCtPPndy76dvF3x/Moj8AAICmxGAwWCO/yMiDlQ+6uLoE+DdrFtDM379Zs2bNbrutu38zf5VSKYQoLi5OTU1LTUlJTUtNSUlJSkxOz0g3Go213wEAAAA3BcJBVGUpjFweMX65rcsAAAA3o6LCoqjCoqgzZyofkclk3t5ezZo18/f3DwgI9Pf379Chg7ePt0wmMxqNaWlpSUnJyclJSUnJycnJKckp7GAIAABwsyEcBAAAwFUym82ZmVmZmVl//3208kGlUunn76fVarWB2iCttnv37g+MHm1tMLTuYJho/V9y0vlz58vKymxXPgAAAAgHAQAAcE0ZDIakxKSkxKTKRxQKhZeXl3UHw6CgoND27e8aOtTOzk5UOfDEmhgmJiTkFxTYrnYAAIAmh3AQAAAA15fRaLx8B0PrEcnaIG2QVqvVavv37+/g4CCE0Ol01tNOrHEhp50AAABcV4SDAAAAsAHrEcnHjh2zvilJko+Pd0BAoFYbFBjYTBsUdHvfvk5qtRCiqLDofML5xITEhMTE8+fPJyYlVZSX27R2AACAxoNwEAAAALZnsVis2xceOXKk8kEPD49AbWBzbVBQ86A27doMHTrEzt7ebDZnZGYmnD9//nxCUlLi+fMJ6enpZrPZhsUDAADcuggHAQAAcJOydhceP3a88hEPD4/g4BDr3oW33943IOBhmUxmMBjS09Pj4uITExOTkpLj4mLz8vJsWDYAAMAthHAQAAAAt4y8vLzIyIOVexdaT0YODg62blx43333enh4iEs3LoyLi4uPP8dKZAAAgBoRDgIAAOBWdfnJyGq1Whuk1QZqtUHakODggQMG2NnbCyHy8vLi4uIq48KU5BRWIgMAAAjCQQAAADQmOp0u6nRU1Oko65symczPz69Fi+ZBQc1bNG/ep2+fUT6jZDJZRXl5QlJiwvmE8+fPx587d/7c+bKyMpsWDgAAYBuEgwAAAGi0zGZzampqamrqvn37rY/Y29sHaYNatGihba5t0bx5nz591Gq12WzOyMiIj4+Pjz9nlZ+fb9vKAQAAbgzCQQAAADQh5eXl0THR0THRlY9UPeRk0KCBjz02XpIk666FsXEXsAwZAAA0VoSDAAAAaNKqHXLi5OQU1DwoODg4JDgktH374cOGKZXKsrKy8+fPV25ZGBcbp9frbVs2AADANUE4CAAAAPyjpKSk6q6FCoXCv5l/cHBwcHBw5QknJpMpNTU1Li4+MTExKSn57NmooqJi25YNAABwdSRXT29b19DkbN26xdYlNBWz58zZt3efrasAAACNh0wm8/Hx0WqDgoNbhYQEh7Ru7e7mJi6ehhwbGxcXF5+UlJiRkWHrSgEAABqEzkHguuseanjuPxyACADA1Vvyfw6HTittXYUQQliPLsnIyKhchuzt49OyRYtWrVq2atnqzjvvHDt2jBAiPz//3Llz8fHn4uJiY2Njs7KybVo1AABArQgHbaDhvWxKpdLV1dXVzc3d3dXe3sFisViEkEmSyWj8+++jFRUV17XORiAn+6b4RryZr3nkID5YAABcvZ//sDt02tZF1CI7Kys7K+vgwQtZoVqtbtWqVcuWLVu2bNmzR4/Rox+QyWRFhUXWs01i42Lj4uKzs7JsWzMAAEAllhXfdBwcHNq0adOlS1j37t21Wq2QJJPRqFD8E+NaLJa33nr7yJEjNiwSV2TkoIqV7xbZugoAAG5hj7/p8vMuO1tXcTWUSmVQUFD70PYhwSHBwa0CAgJkMlmJTpdY5SjkpMQkW5cJAACaLjoHbyKD7hj04AOjA7WBkiQZqwSCVZNBs9mydt1aksFb1HPzArbtd7F1FQAA3DKG9yla8lKKrav4VwwGgzUBtL5pb2/fslVL61HIXcLCRtxzD1khAACwLcLBm8jZM2f9mvlLkiQuDQQrmUym6OiY1atW3/DSAAAAcA2Ul5dXPQqZrBAAANgc4eBNJC0t7duV3z7xxASZTHb5s2azubS0dPbs2Waz+cbXBgAAgGuuWlbo4ODQomWLalmhTqeLi4s7fToqLi4+Li42Ly/PtjUDAIBGhnDw5rJx48a+ffuEBIfIFfJqT0mSNHfuXL4dBAAAaKzKysqqZoVOTk6tWrUKDm4VEhIycNDAMWMekSQpLy8vNjbWerhJTExMQWGhbWsGAAC3OsLBm4vZbF757Xfvv/dutcctFvPq1T8cPXrMJlUBAADgxispKTlx4sSJEyesbzo6OjZv0dzaV3j77bePGTPGmhXGxcXFxsbFxcVHRZ3W6XS2rRkAANxyCAdvIpIkDR06dOLEJxITE1u2bCFJFxYXm0ymqDNn1qxZY9vyAAAAYEOlpaVV+wrVanXr1iGtW7dp0yZk+PBh7u7uZrM5KSk5Jjo6JjY2Ojo6MTHRZDL9mzu2bdvm7Nnoa1E7AAC4eREO3ix8fLynTp3asWPHLVu3rvpu1QcfvN8qOFghl5vN5qLi4tkfsNUgAAAA/qHT6f7+++jffx+1vunh4REcHBIc3CokJPixx8Y7OzsbjcaEhITTUVHWg02Sk5ItFkvDx9doNPPnz//9910rvl5RVFh0fSYBAABsj3DQ9qwNg08+OTE7O3vmiy9Fx0QLIRZ8umDR4kXWCz784MPCInaTAQAAQK3y8vIiIw9GRh60vqnRaNqHtg8ODg4JDh4+fJhSoSwtLU1ISIiNi4uKijp98lR+QUHdA7Zp00YIMXDggF69eq1YsWL79u1XlC0CAIBbBeGgjfn6+kRETO3YscOGDRtWr/reYDRYH09KTl61atWECRNWrlwZFRVl2yIBAABwa8nIyMjIyNj1+y4hhEKh8G/m375d+9DQ0MpDkOvdrLBN2zYmo0mhVDg5OTz//JQRI+757LOFsbGxtpgNAAC4jggHbeafhsGs7BdfnBkTE1PtgvXrN9jZ2a1fv8Em5QEAAKBxMBqNSYlJSYlJv/76q6hysElou/aVmxWmpKTExcXHxsXGxcXFRscajIYOoaEKpfWHBUmShFar/eTTT7Zs2bLqu1WlpaW2nREAALiGCAdtw1fjOzUiokOH6g2DVZnN5u+//+HG1wYAAIBGrPJgk00bNwkhvH182lw82KR371729vZ6vT4+/lzzFs2rvpdcLhdC3D1s2IB+/b5ascLakwgAABoBwsEbzdowOGnSk5kZmS+88CJLMwDAViT3Aa8vfXFI4oI7Z+2suInHBIDrKjsrKzsra9++/UIImUwWqA1s3bp1ePfucpn88ovlCoXaxXnG9OlDBg9ZsnhJckryDa8XAABcY4SDN5Svxnfa1KmhoaF1NAwCAG4Myd6/fccW3tkK6XqNadc14rvl4513vDpu1vZctvEHGgF/H3N4h0b//VuMMMZIFUqzuYespnxQJsmEEKGhbZd+vujoXz8ej/zJZNLf8CIBALeqyFPKtCyZravAJQgHb5DKhsGMjMwZM16Ii4uzdUUAYEP2QQMffW783bd30Ho5ShWF2efPHtv3yw/L1h3Pt8hDJyz6+FH15skTlkSbbF3nvyVJkiTJZNcwfQRgU+EdDCvfLbJ1FTdCTGlwdp0pqEymEEJ06z2uT99BrRxnu8kP3aDKAAC3uMffdPl5l52tq8AlCAdvBI1GM23a1Hbt2v3888+rV39vMDT6XzgDQB3kQQ9+su6dfl7yC5mZwjOgQ59mIYpj3/33uLDI3ILah/jlqxpDoFZx5LOHunxm6yoA4MoVmjoLUa1t0CITJrOQCSETQkjCopAK7GQZjrLEIkMXB1mSnZRpk1IBAMC/RDh4fVkbBp+a9GR6esaMGS/Ex8fbuiIAsDVF2GOT+3qKrF0fvzln/dHEAoPKIzC0e79OZX9kmm1dGwA0zPKNnkejHWxdxfViZ+86aryPJAkhhMViLi8r0hVn6grTdUVZpbqsEl2OrjirVJdjNlft71YKEWCjegEAt4AubcqevC/X1lWgZoSD15Gfxm/a9Ii2bWkYBIAqHAOCPOXmpC0LV+yLNQkhhD4r/uDW+INVr5G3jth4IkIIIYQ566dxg947YBCS54BXPpk6rE2Ar4udpSwn/vBvyz9ZvCG6xCKEEJJnn0mvPda/fatAfy8XR6Uoz086tvPHjxf8dDT/n73+ZB6dH5n8zNjBXVp6KkvTz/7vr0LfKrud1D2+a9hDUx8b0j00uLnGzUEqy0n87Z3xb/+SZ6lzTHnIsz9ti/Bb/9TAl/caJNeRK/Z9MFB16auh/+v1O578PssiOQXf/dTkiff0bOtjV5YZvW/Dl/OW7Unh6wZwszoa7bBtv4utq7hePDzccvQrcrJzcnJyc/PyzOYaf3XjdKPLAgAA1wfh4HUhk8mGDBny1KQn09LSZ0x/If4cDYMAcFFZekq+SRZ4x7hh/43ZkljW8Hc0urXq2jrAGq6pfdsNGP9RBx/DfS9uzrEIIfPoNHhE//aVX9WcvFr1efi1sNYOo8Z9HWMUQgjJpffr3y16PMTeul7ZThs2XCuEEBUNG9+n1+hxwyvHd/b2UVboLPWP2UCOHaes+GpaF2drrmgf2HnE8wvD/CPue31PPueYALjh8vLy9+7db+sqAADADcIBMdeev5/f7NkfPvvsM5u3bJk2fTrJIABcwnDk68X7cqSgB+Zv2PX9u88Mbetx+S+qTDEL7+vUok1oizahrW5/74BBCCEsxfs/Gj+yV/duwe06tet5zxPLT5R7DnxggPs/mxNain55ZUjnTp1ahfboO3bOjgyzU+exY7sqhRBCKDpMnDU+WFV0bNW00YNCO3TpPGjstOWHcqp0wzRg/OIdb91zW5ew4E69bn/ws4OG+sesylL48xMdQ62TahE6dNrmFIO59PQPX/+WI2s9/o0pYY5ZexY+MbxP29BuPUa/vu6cKWDklLHBNZwTCgAAAADXEOHgtSSXy++9797FSxY7OjpOnz7jm29WGo1GWxcFADcbU+LaaaOeWbgpqsSz2wMvL1y3b8fKD8Z19623l12S3Do+/OHX/91/8NCJ3d+9PdRfLhS+fl7/fCWzmIqzs4oqTGajLvXwDx+uOmWUe7Zt6y0TQsjbDL2zuaziyIIX5m08mVlq0BelHtu8enuc6YrGN+anpuSWGkwVRWmJmSWyBoxZI5n3nW99Me8er/M/vTBh7v4cqc2Ie9oqinZ+MHPZ7viCCmN51skN7yzcrZOH9A73urKXFgAAAACuEMuKrxltkHbatKktWrRcu2btmjVriAUBoHb6lD+XTf3z29ndho+f8NiYQbeNeW3FnX3fH/Pcmvja/u2UXPq9vnL5I0HKC418dtpAIYRJJqvtC5kpLT6hxBKqVjtJQgilf/NmMnPK34fTa+nru+LxGzBmzTdy7j518YIHtdnbXpn4/p/ZZiEcAlsGyGQOQxdFDl106RT8A/yE4PRPAAAAANcRnYPXgFwuHz169MKFn1nMYurzET/88APJIAA0QEXGkQ3zpozqP/rtTUlm735Tnx+kru1SyePOx+/XyvMPLn7ugV7dwoLb39bz+Q0ZdfboWfR6vUWSZJIQQkhymRDiwhvXZvx6x6yJIuiBOUueam889NnTr21NsY5vsdSyr6Bk52B3BWMDAAAAwJWjc/Df0gZpp0+b1rxFi9Wrvl+/fn0tp7kBAGpjLozasGDNA8Nntg8O9pNvP280Gi3C0dHxksxN5qXRqETpjlWLdp7VCyGEITe76ArO/dAnxaeYZc17D2i15GRMDWcAX8349Y15Gcm5+9Qv3ujvlrTumWlfn648isWQmpBqNrtufHLIG7tLGz4lAAAAALgG6By8epUNg2azJWJKxLp160gGAaB+qi5Pffji+EEdtO72ckmSO3i06H7/5JGt5RZjdlaeWViyMrItcr/BDw5uoVbI7T1a3RbqLxfm3Kwsg3DoMWpsN3+1QhIypVptfwW/4DJFb95yxqBo/9ziuZNub+VhL5fJ7V293CsTyKsZv74xq5E8Brwx97E2llOLZ8zZlWupOs5vO86ZvUa8PW/iHe01Liq5TG7vHhA6oHsQv8EDAAAAcL3xc8dVCmoeNH3atKCgIBoGAeCKKNrfMWbkhKAHJlz6sKUs9ruvfsuzCEvSnl1RER07jZq/a5QQQgjDsQ+Hj/sqedea36fcfvegN38Y9OY/72WKaehtTTHfvj2/91ezwoe+unzoq1WesLYHWnKvYvx6xqxG2XXYcH+5JHWcvv7I9H/GSFv52LB3V3zw9cDPJw2esXzwjMpnDEfnDR7zbSJfXgAAAABcT3QOXjFrw+Bnny0wGk0Rz0+lYRAAroj53Ka5n3z/S2RMWmG5yWw2lhWkRv/189JZo8fMP1BsEUKYYr+NmPnN7ticUpPJWJp77mhctiQJS94vr096YcXuU2lFFSaTsaIkPysl5vjBv+IKa9mw7zJlZ76a9J8J89fti84sqjCZjOXFOclRkTv/u+ecQYirHL/uMRvMUnxozqNjpi3d8ldsVlG5yWQoyUk8sedwsv5KBgEAAACAqyC5enrbuoZbSfPmQdOnT9dqtd9//wMNg2igkYMqVr5bJIR4bl7Atv0uti4HAIBbxvA+RUteShFCPP6my8+7bH9ED1/TAQC4Ojfb13RURedgQykUitGjRy9YsEBvMDz/PDsMAgAAADeYPGjU7E3b178azuZIEEIIIbn2e+3r5ZO7e9ay4e+/Jw+N+C3qxP43wlXX6w64GdiHjHznh4WPtOCfFjRVhIMN0rxF808++Xjs2DGrV3//8ksvp6Sk2LoiAAAAoMlxDmzfPtDNXroeUZBd14j/+/vwtrlDrl/QhGtLcu4z9b2xt7X0lF23jTjk7Yfc2Upk7vztGHt9NGoGo3Ngh8FT3/tPoNzWpQA2QThYjwsNg59+WlFRMWXK8zQMAgAAANeJ04hFZ6NPn7/sz9mFd9lfs5vIQycs/fX3755rUz0EkCRJkmSyK48GnUYsOnv28JaZ4W7V31fZ7/298VFbZ3VqSOBQa2FV2fecunbrjsOHjkRHnYw7feTE/l+3rJw365FwvybY2KYIeWzG/c1yt81dFFlssX7yHN/8bKs6X2vJqdWwN9cciD/x2QiHBtxC3m7YkOYic/e2S7PBht3rWmnQJ4YQQgj7oIFPzvtmw/8OHYk9/fepA79t/nruyw92dr+V0+4b9VKbzv84Z1mUXc/Jk+9wuZVfL+Bq0TVblxYtWkyfMS2gWcDq1RxJDAAAADQCMreg9iF++arqCUDFkc8e6vLZZlcJSgAAIABJREFU1Y4qOYQ+8cnC9MeeXB1/tS1mtRV2Cbl3cMdg/4ubddk7ewWGegWG9ho2ZtTySRMXHixq6DFdjYBT3/Hj2klRC5fvLGjArOXOLXoOfeD+Bx68q6OPUhIVDbqFInTI0CCR/v32v23ZN9igTwwh5EEPfrLunX5e8gvXKTwDOvRpFqI49t1/j4sm9HlxtYyxq5ftnLBg6JP3L935bTI/+aOpIRysmUKhGDly5Lhxj8ZEx0yZ8nxaWpqtKwIAAACaAuOpBaNGfh5vukbDyeycPd3sjcX5+aXGazRkjSwmi0vflxe8cu7Rdw/UedL9NWCMWvzwA0vPVljkDq6a4K5DJr343N0dn3h3wo7hn0Vdq9ftKtT2Ul+XD4HkOmjUnV76I4t/PteQKct87539xSs9VMKQcfKEObSTZ0Puoegw9I4gkf7db8cN/7LaK3GVL5ci7LHJfT1F1q6P35yz/mhigUHlERjavV+nsj8yb+6g63r/DW3wp6WlYM9/t2UOfXjUPSGrP4+24V8kwBZYVlyDli1bfvrpJ2PGPLJq1eqXZ80iGQQAAABuQpLngFe/3bD3r0MxUSeij+za9uXLo9o4VfZXyTxue3rB+sNH/hf55x9H/j58fPtHD/hf/PFH3jpi4wnrmuX4vW/0Vgoh5CHPro09s2/u7cp/buCgvfPZD3/6Zc+pk0dPR+7avurtkUG1rW40Hl+16JfcoHHz3hnpX9cKSMkp+J7pn2z4/cCZk0f+3vnDwuf6B1S5YU2F1cBs1BtMFovZWJqfcuL3r194Y02yWdGie1ffi/Or5y72AQMnvbd6y+4TJ07Enog8/PuGNZ+/N+nCqmjJNew/b366YvP2PSdPHI87+ddfW94Z5iHVPWZtL3VdHwKH5kOem7v2t72nT/598s8NK98eG+5d+brVWsM/HMMH91IbT+7e1bDoy5x58I/IU1uXzhxx7/R1SQ1LyxShwwcHirRd264wG6zjhbqmn7GXcgwI8pSbk7YsXLEvNqdEb9TrsuIPbv3mq13p/8xW5d93wpvfbNh59NjxuFOHj/25ZeM3n7wzqrVcCCGUfd7+Iz5qw/S2VT4K9y+Njj66cvSFl77O4q/lp80VqOuz6Mo/LcuP7dxfIAseeEetf82BRovOwUuolMqxj44dNWrUmTNnpjw3JS093dYVAQAAAKiF0a1V19YB1u321L7tBoz/qIOP4b4XN+dYhEzz4JxFL/V3kYwluZk6Se3p5qOsKDQL0eAf++3aTvpyxawebhfiCpVvcFiguqzWXMmU+ssrLzi3+HrCux8/ET3hq6jymi5y7DhlxVfTujhbx7QP7Dzi+YVh/hH3vb4n/190G5qNJrMQQiaTNeQu9m0nLVs+K9z94u6KTp4BrT0DWqr+/vrryAKTkPn0Gj1uePuLPyg6e/soK3SWusaUanmpa/0QCGH//+zdd1gU19oA8PfMbN+FXXoREVEQsDfsvcXeEhNr9BoTY4yaeE3sLdEYTbu2zyReYzR6E1M0sYuCKPbeEEGQIkU6u2zfmfP9ASgoLAuCIL6/5z7PjcPMnHfOOTu7++6Zc4Le+2HrJ8HKwoDd/HuMXdC5e8u5ExbsT+GgrBiKETRp3VLOP7x+I83GYXHc/S1T3wIAYNyCbTtC2Lx/fy94uPNYxXKD1iu/+nqsPvVhDsfU7zNx4J/RBxL0z+4gCZj2/Y+fdnB8/Nix0q1hC7eGTTTHvvgr2qZxclaCr8JuYzvrvais+rTSLcF48+od8+jgtq3sIC63ApEg9PLD5OATAQFNZs+Z4+zktGnT5qNHj1KKEzMghBBCCCH0ggmazfnn/pwn/6a6Q+93mHe0tEnfqObMukkjFsUmZeSbhUrvTu+s3ji11+ieDgf+yAa7Dv072PG3v399yqYbag6I2MXHxawrOpKLXj/q9W+jrKRE2Ibjl3wcrDRE71v92feHbqTqxY7ejZQ5mVa+I9D8KxvnfNvi909nfPvR9dfXXNI8vS/rP2nJzFay9PD1C7/87WyCQRk4cN6XS0aPmDl+e8TGGBsDe4KwIpnS1adpt4kfveHNcg+uXE3jyy+l0YSlc4NVprgDXyzf+Pf1lHyQuo9ce2xFl6dqNmT52AX/PMzlpG7u0jwz6/+vMs+5Kb30qib2ZTUB23jiko/a2xvu/rF82eaDkTkiz3Zj5q+Y12vg8k9CIz46UpgnfTqGEpeu8GnoznIRcTYOAqwEYcsBfbwg5acjtyuSG7Re+Vw19ljzlW0bIwau6D76q71dxh74ecfuPSeisp88R8v6TVo+t4ODJfHYl6s27b34INsksA+cuv23DwJsvjYrwdOiPZ6/29he1dZ7UW4Z9Vl2twQAqnkQ/4jv4uNbHwCTg+jVgo8VAwCIhMIpUyavW7cuMyNzxgczjxw5gplBhBBCCCGEajtCVM3fWr3tzzMXLt0M27F8gCcLAjcPZwYAKKUAxCUgOMBZQgCoMePBQ1sWrijE+g4Z2kxsvrl+1tJdFxNzjGaD+lH0tVinGX/df7yMcuQ//2761KguU/TOhStDNY0nfr6o5zOLxLJNhg4JEKiPr5r3Q1hsrtFiSL+1d8X6sHzWr3Owc0W+mAmazfnn/r07cZHXbp87emDr4jebyvV3f1n23zuWckthfQcNbiriorZ8tHjHxaQ8E8eZ8jOy8p+uGGrJSX6YpTNzRnVKwiMtY/WcZVV1WdtZv2HDm4rMtzbMXfn7jUc6syk34ewP/162J5U69BzW53G1PRVDiRAZJ2dHhtdlZemq62ubsPnAvp7w8MSRWxXKDZbXxFXTY9mAmc/2Qy7h9zmjpq//J1Lr1Hb0p+v/iAjZvmpie7eC4UCs/7BhQSJL5KaZn/wYfj9Tz/GcUZ2Vq69Q9VkJvkCVdBtbq7q8XlTRbllwBdmZ2ZRxcnasSL0gVBfgyEEIDAyYPWe2kyMOGEQIIYQQQqjG2bwgCbHvvnj71rENhIXZJLF3fQDgGEYAAFRzdu+JzF6DeyzccXxuTsLt61fC//ll25EYrY0f9gUN/HxYPuni2cQKLkzApfy1bEXnoG9fX7Eg/NYSbfE/ier7ejGMdMCGiwM2lDzG08uDgeyKFQRFaQ6qubxt0Sebwh4UJMqslyJw9vNh+aSzJ+9XJOdl9ZykrKoua7uoQWMvhk+6cCa+WN1qr56+Zhj7WoPG9W2qCZFERMBkqrZFhIUt+/fzhKTtITcqtEiG9con+q7V12MBAEwPT/0w+9TPX7QdNGnK2+N6txu36L99u34+7oM9sULvRl4Mn3T2ZGxl11ax+nIrXeW6jY3XW14vojcr2C0pAAA1mcwURGJRheoGoTrglU4OikSi8ePHjRo16tq164sXL83MyKjpiBBCCCGEEEI2IY59J4/0ZnMubFyydtf52Ay9wLnPon3fDSv8M808uHBi/rU3BnZs2aZ18za9Grbt2SuAGTXzYJ5tZ2cYAvD0uAEuauOoxhvLOZRmhn629M+2349evihiXfG538ochkDEUvHTwwyteZw/FflN2rxnQYdGAa5gfDz2yWopjFDAAFgsFUt5Wj9nmVVdxvbQilxrGUwGEwWRqLpyOKI2A3p7QtJ/j92u2AK6ViuKqbIea70fGtOu7F175e/vg0at/G7x0O6zP+x9aM4pygMAx1t7Kh54ALFEUnrrlPNyK/2Mlew2tqUHy+tFFe2WBzMpABGJhARMxmrLOiNUW726ycHAwIA5c+Y4ODjggEGEEEIIIYReHiwrAABgnN3dRaAL2bnheJQJAMCclaE2Ft/RkBS+85vwnQCsXcDolduW9+s5IFh28JjFYqEgk8msJhfMyfHJPOMd3Kk+eyu+goMHgeZGfLPof8Hbx86b80hGQF38nLzy73f6LwkrZW41gU2BlWCK+WXhopa/rh8896t3ro/7PspYbimC1imZPOPdLtiTuZ1k83x95UReVlUf0pa6/ciD2Ic806BDlwbsrbiiupW36dZaAqbEuIe8DZNf8VmZ2Tzj7+QkI5BX9V/kRC0H9XWHpO2HK5gbtF5RbJMZ1dhjn8bnRe79bs/oQfOCGjf2YE88fJDMM/XbtHVnbieX2u68Ji+fMvWaNFaS61nPVmn5L7dnVbrb2HJ9poTyelEFu+XBQ1oA4ujsSPiszIoP40XoJfcqzjkoEommTJm8du3aR48e4QyDCCGEEEIIvSxMZgslqjY9O3rJWD4rPd0M0g6jxrf1VAgIMEKFQvJk7APbsPfo3i3q2YsYwgoFFo3GCEAIEKDpaRmU9ej3Rr+GCgErcWzUrqnns+vBcveOHIvlhC1nb1g5oYOPg4RlhQr3Jq2aONn2BYpqzn63cneS0tOz2EAs7t7RkDjeeejytVP7BLnbi1iGlTh4Ne3ZvoEAAGwM7Cl8+uGVy35PFreesWJaoKj8Uix3Q0JTeXHr2V/PG9rUTSESOzRoP6pfYDkj8Kyfs6yqLms7F/3PP5EmYfMPv1nyegs3mUCkbND53XUrxniQ3PD9x7Nt+WpG8+PjH3Gsj693dXyhFbcZ0NcNEkJCKpobtF5R1dtjRa3fXf3vSb2beTtIWEJYqWPD9iNnjPBnqSUjPZvnYkJOxPPitnO+/veQIBeZQGjn2XLYhP5+T87Dxd26q6birtMXjGvtJmMZVmLn4iB93HvLCb7itVH29dpY1eX1oop2SwAAYufj48aY4+Me2hgFQnXGKzdyMCgwcPac2QUDBo8cOVLT4SCEEEIIIYSKe3q1YgAAy+21Q8f9Xxz38G5ULg0ImPR/R13/3X5O6J4TM7sN7r10d++lT3blogEAgDh1mLpiSWdhsZPw2YePXdICpw8PjZzVvMWor0JHAQCA+frqQRN/THwqDMud/37+fffNM5qN+GzHiM8KtlHt/lndZx0z2HIZVHPxm1V7e2953av4Zfx31bZe/zet38db+338eKv52tp+435O4LnE0gMrZ3wfzYv48rO/u20e+f7S8Ucn/BTDWS/FcOmHr/7p89WIlpPW/zWp+PVaLcTaORPLqGqdU5+ymiBm58rvum+d1/6Ndb+/sa7oOszJh5d/edSm3CCAJfrqde3EAS1buDG3Up7U0LOdh3v48+Req69WJMsnbjuglxsk/nA00upRpZfVe0PZlZ9VtT22RMcQBPUZN2JKg9FTSgZJ9TE7fjyaTYHe+u+Xu/qun9j67Q173y6+x+PRf9rTO3fe7fth04Gf/zrw8yd/L3zAlloNvgyV6TalDRsso1m/tdaLyqrPsrslAIibtwkScrFXb6gBoVdMieTggvnzayqOFyD6frS9nf2oUaOuXL26aNHizMzMmo7ouQwfMTwoILCmo6hGX6xZU9MhIIQQQgih2kUX/t3HmxSfvBEsfphqotmHF0+bmzZ76sC2fm5y1mLQ5OVkpCaev59HAQhJvXryZr22fvVUYmLMS465cmTnpvUHMigAF/PzrHn2yz4c1sHXQWTMTbx9P4OUMlyJ5l/5+u0JUVPffXtQh0BPlciSl/bgdqxaSMBgW/6K5p3+z+pD3TcOKrZJc2nNhHF3/jV1bL/goPpOctaQkxJ7/XJSQQLGxsCeLSj31Iavw3p91fudjwf9M2N/lvVS+IyQT8a/Hz3r3Td6NPdWQl7izYg4Rb8+/jy1loW0cs6yqhpcy2wC0EdumTbuwdQZ7wzr1NRTwefEXznxx6ZNv17MsPkJbu3FE+fzB3fr1cv1f7vSbH4+2hbiNoN6u0D8tsORFX2cHMB6E1dnj+Xj/vnyG9HQHu1bNvF2sxNRo/pRYtSlE3t//OlQpIYCAM07s3Li1NgPPxjfp4WPo1D/KPrc+ewmI3p4Pj6F8fb6d99TfzxzfK/m3iohNelys9ISYyPD7+tpecFXojasXK+trPaiynRLScu+XRxo7J7QCs8kgF5KAQFNRo4YWdNR1KTiWReidHJ5/I+DBw/URDwviF6v5zlu20/b68aAwQXz53ft1rWmo6hGgwcPqekQqsyI3sbtK9UA8MFar0Nn7Gs6nMqhbcbFbx3ChWzwmX9OgM/hI4QQejEGdVFv+uQhAExear8vVFzT4dSN93SEnkJcxvxwemW7C0v6TP7dxnF7tYKi96rQTYNTvxs96nsb1ra2maTr8rAfR2l+GDvw2zt1O0XEeIzbFbKodejcVrOO2DQato4jqgFfHv+u74O1I976qaKLlCOb1Lb39K7dutbtEXLlKp51eYXmHNTrdO9Nf79uZAbRq0PVK+nuvshT7+jKnwKA0c/beDduT9IIebVEQoASAkwVLC6HEEIIIYRqDOPadnC/tv6ejgoRK5A5+3ebsur9DiI+/tqtaljZozrln/r5lyhoOuGdPqoq/IQqCR7Qw5XGHz0WhemhV4vAb8K0vqrsY1v/SsKmR6+gUhIOFy9eWr9x84sPpfr8suMnAIiMvJuTk1PTsVS9CZOmlL/Ty2PWzBnBwe1rOopaRJMgTqQanwZGJyJ7ZPXzGpEZA1wplyK5Wy0//JEru31b766OMyOEEEIIoRdH3OqtNesHKYrn0yiXcuD/dke/bCkRS8z2r/e+/sPo+TP3nVt1QVMlqU1pu0G9nGncX0cwN/hqYRu+9em0puYLqzYff8mS5KgK7EpzuaWR1XQUL85494zmdk+vIP4KjRxE6GXEJUui9CDwNhRbSgwY1+zf/oiM+Saz+EbW2+gngPx4ScJL9UmGEXEuThYHSdF7MOG6TI67ujNuYcuX6jIQQgghhF4ORJR7L+xiVFK2zsxxZl124u3wXV9MGz3/WHqVTtz3QlD1mf8s2XU5LpuKq2jsoKz9gN5ONDbkOOYGXzFCofZh5PHvlvyKDxSjV9Qrt1oxQi8Zi+T2QzKssSHIBSJSC7e5t9e0FIDAR9PXwznmYeFG5wYGd5Zci5WYAIhKs2BuxkAfk5ucp0ZBbKT91p2ue+OZggycsknO7GHq9o2MPk6clJDMVOWKxR4XG2YuGpYf5GXyVHEyARg0ouvnHb/e5XCtaKkuvzFxh8aa//rM/9OrBACcWpWzPwCA2NRrcObUXvkt3DkpkLwcUVyCJORvt623WQrAKHXT3k17r6PBQQCUEk2a3colXn9mG/p3MzjYw9BOhtU3qucBaYQQQgihVxfNu7h11qStNR1GFaG54av+FV5159OdWhIcuKTqzler8am7xzbDJ4MKGKL3Lhu7t6ajQKjm4MhBhGo3TngzRsAxxpa+RT/mMua+nXVCLZtLDK91MBaNHaSBjQ0sJ7wRI+ABwMI1CjR42fNCFkQyS2C77HUrUoaoCnd1bZkzsau2mYdFIaKskHdxpEYdOPqrh7bVNXGz2Ikpy1K5ytjltdSdC7P82WdjAgAb9hcbpi15sHVSbhdvi52ICkS8k5uhfXDeYH+OAQDG/MbspE+6GlSEycoS5BhA4QjGfABeEnJGkpMvOXheUk01ihBCCCGEEEIIocdw5CBCtRy5Gy0xDtYENTYKzkgtAIybemgTiNrrHtYhZXo3td8+lygOgDU19eWJQXE1ngAA1SnWLW60KEmUoQehwtxpVPLGEZrR7SwHjhctNEzZkC0+C06KcnnezYnPs4AnAFD28Abf+aeF+Rzv0SR32dy0fv7Z4wMdl90u4zkNq/s3Gpw6txlneqj8YovL3/dE+cC7904+9n5+4VXJtP2bcfx959eXut7QAhDq4mkxGwAoG7HNt822F1CxCCGEEEIIIYQQwpGDCNV62mhZNA/1/PUuDACAbxd1K0Z8+JT9/jNi2kA9zJcCAJHpW9ejlgfSG6bCo1R+2atXxJ75Jerm1gfLO5lZADcXy5MXPIWcdFGWgXAmNiVVqKWFGzU5ArUJeI5JjnRcfVBqYS0BDS1l3ias7M8aB3UziHjJlq88d9wW5ZmBMzMZucyT2X0poQDEwRjc0CwhAJRkJAtzcfJfhBBCCCGEEELoxcLkIEK1HZcuu5QOgob6FkIAxjC8h4GPUe5PIffP2N/hjcN66yQAwkb6ZkLyIFKWzgMQrvu78Tum5vTyM7nJqVDMebtbxASYCs7TnJIk0lJQSHkbjyuxP2v086R8muJkYulHU51870UBcdIs/Dzm+s/3/1iU+mFXo7yKZpJGCCGEEEIIIYSQjTA5iFCtx0nO3hFQib5tQyoOzB1ej1w6aZ/EAZei3BdFPDrndpdT3wC9ExWcvyHmAIhSM7m3iVXLN6727TQ2sPHIgI5rVGkVX3aLmhkTBcLYOpyvxP4EBASAgzKLpYKDG3z+9aPzbxdlidTcpn3Ox3MT1na1YHoQIYQQQgghhBB6kTA5iFDtR67dkOkZc8eWhh591J4GxZ4zQh4AeOHBEwqdvWZMJ0NwMyMxyM7eJwDAqMzuQtBdc9xwQZKmIxzPZOWwxhccskWQkguMuy7Ytex9jKLwA67zV/v0/5f/oM32qdTSs7NO9uJCRAghhBBCCCGEECYHEXoZqG8qLptoQOdHH3S2ZJxRHVcXbs84rzqq5rsNSXvdn+pvKy4aAQD4XGG6GaTNc8cHmRUsAEMVMv5Frz3ESUMuiniRbvbctKGNzAohdfDQjupkED3egTX17qtp4cqJGGAFYNExRgKEUEK4LpPjru6MW9iy4mMdEUIIIYQQQgghVEG4WjFCLwGapzgayXRvrWvBi7eEyHWP/6BT/O+kaMQIfXPKhF6QFyzoQfMUey4KunXTLP1Cs/TJOUj0Cw2ZXPrT9Z8OySP8s9d/k118e+H/KbVT30/tXPwOxAsOn5NrGUP/bgYHexjaybD6hvxFRowQQgghhBBCCL2CcOQgQi8DKgi7IDVSMN9X/R5bfF4+cv24KtIC1CQLuSqgRTsf3thg7l672xmskQOLicnJFkVHy88nsS9yNWA+x/6TBd5rT8jichkLx2Q9lP99QaKjwBfETYRXL0sT8hgLD5yRTYy2++E/DeadElBeEnJGkpMvOXhe8gKDRQghhBBCCCGEXlE4chChl0Pa4QaBh0vZziU5Dxvt/NRGahDv215/3/bSTxWzx9dvT/kbzdc9gkd6lLVDufsDgCVTvmW9fEvRP136J74WbNRoGB6AZim+/kLxdSnRsRHbfNtsKz1yhBBCCCGEEEIIVS0cOYgQqhaMo25wR52/i0UhpAKJxb9N1qoxWhEVX7v/QgcwIoQQQgghhBBCtRwRiz8c4PR7Z7Go/H2rHo4cRAhVC3GT7DWfqhXFn4GmJOWU8+4EUuYxCCGEEKp2tM24+K1DuJANPvPPCfAXO4QQQqg2ICzr5yRw1BMCAECatXRYE8BEnMv+MpF/AW/WVZAclHScvXPJkIaujnZyEcsZNLnpifduX4wI+XNvWFSejeuNsk2nbPh6gmL/jCmb7uESpdUIGwu9MCKNJOy2qZW3yV3Bg5lNfSg9fdJp4yF5Ol/TkSGEEEI1QdUr6dxsTcYBn95bZRbruzL6eevj33dVfDyl/j5t1UdCgBICDP5ahxBCCNlA7K74pr2kvpRRCAlLqd7Ep+aZryYZ9t43PiznHf25ENue9vULVC1uQo6H5+zMqXxZVZAcZF0aN2/sKS78h0zl6qNy9WnRbfCU6Td3LJ63+niyDXXFqBoE+XnkiPAzSjXDxkIvTN5t51mLn54MESGEEHplaRLEiVTj08DoRGSPrI4BIDJjgCvlUiR3DdURCLmy27f17uo4M0IIIVQHsVJBgJItfNqXELmEbSxhG7tJhvnrPzuuPqWrjjLp7RvZg2/YsidR2gl95PxzPoxcVXMOWiI3vREU1Mw3qE3zLoNGvv/51ohki6rl5G+/X9TJDpNIFTJy5Mjp06cHBQYSUk01h41VZep71V+2bGnPnj3FElxaFyGEEHqJ9erZa86cOa1bt2KY6pqSm0uWROlB4G3wY59sZFyzf/sjMuabzOIbWW+jnwDy4yUJL9VDGoyIc3GyOEiKEp+E6zI57urOuIUtX6rLQAgh9JJTKZWff/55v3595XJ5FZ425mZW312Puu961O/3zH+FaQ5kU5G9dE4LUd3IBVTZnIO82WjiKAVjfmbC9dCE62GHTszbtu1fTSbMn7hn1Oa7HBCnngu+mT2wiZebvZjqM2MvH936zca997RPfjdl/Wf9fXNWwdnSf53Y+7OzZhuOqnPs7e2HDh0ydOiQrOzsE8dPhIefjI9PqNoiqqux5I0Hvztj6pCOAa5i/aN7EXu/X/tD+ENz1cZeuzACJjg4ODg42GwynT9/ITTs5LVrV83mOn3NCCGEUF0kk0n79evbr19fjUYTdjIs/GT4vXvRlFbpR06L5PZDMqyxIcgFIlILt7m317QUgMBH09fDOeZh4UbnBgZ3llyLlZgAiEqzYG7GQB+Tm5ynRkFspP3Wna5745mCyJRNcmYPU7dvZPRx4qSEZKYqVyz2uNgwc9Gw/CAvk6eKkwnAoBFdP+/49S6Ha+rC8/uNiTs01vzXZ/6fXiUA4NSqnP0BAMSmXoMzp/bKb+HOSYHk5YjiEiQhf7ttvc1SAEapm/Zu2nsdDQ4CoJRo0uxWLvH6M9vQv5vBwR6GdjKsvlGVX88QQgghKwjDtG7dqnXrVh9++OHly5dDQ0MvXrxkMpme87SUBzMFCmAwcjHJunX5xG+worGLyJuYUp2kUwIlLR0F9WSMBGiOxvDdcXW4AYhQ0LupfIyPqJGUGPSWy7HaLXeMaUUTbTES4bDm8uH1Rd4S0GstVx/xzsWGa/k0c/ypJXs0LHNNStGnEQHbNUDxpq/IX04YjqblGHeeVx/TFFyzYPJgt8kAAMDr9R/tVV+t4HRe1bYgCc07v+HL3wdsneQ3cHDA93fvcGBRNWrj71Uw0lHhFthz0rpmrubh/96fafVTV+WOesmZLWahQOjk6Dhy1MgxY95ITUkNDQs7efJkSkpKtZRXJY0laz7zvz/OaW1X8Gu7pH7LoR+ub+U5a/ji8Jy63FaFhCJR586dunXvZjAYzp87H37q9JUrlzkOfydHCCGEXhoyogOwAAAgAElEQVQWi0UgENjZ2Q0aOHDY0GE52TmnIk6fOH4iNja2agrghDdjBJy/saUvD6kMAABj7ttZJ9SyuTLDax2MPzwUcwAANLCxgeWEN2IEPACxcI0CDV5CAACQWQLbZa9rbDHP9tqfCwDg2jJnYldD0Qd66uJIjTpw9FcPbft4I8hVxi6vpbZqwI9a5BRd2meT8vcXG6YtSZjfjCuappA6uRmc3Iyiu87bbrMcY35jdtInbTnCMVlZDJFxKkcw5gPwkpAzkqF94OD5ujGoAiGE0EuGZdn27dsHBwdbOO7ihQvHj4devXrFYqmiaQIJPE7lOblLRzYQFr2TEkcZmMwAAuHbvR2muJCCJIlYIezTUhUkz5123pgHQESimX1Vr6sKnxgV2Ql72QEAlJnCZAVv9XJ4363oAQeWNHBmy5vDuAKqc7Vi/Y2TF/ImjKoX6CeHO2qqObNu0ohFsUkZ+Wah0rvTO6s3Tu01uqfDgT+yC3NHXPT6Ua9/G1XiM0v5R9VpQoEAADw8Pd56883x48clJycfOxYSGhqanZ1dxSU9b2Ox/lOXzGwlSw9fv/DL384mGJSBA+d9uWT0iJnjt0dsjHklcmSsQAAAEomka7cuPXv1VKvVJ8NPnj4dcTfybk2HhhBCCKEKEAiEAODg6DB40MDhw4YV/kwbFpaSmlrusVaRu9ES42BNUGOj4IzUAsC4qYc2gai97mEdUqZ3U/vtc4niAFhTU1+eGBRX4wkAUJ1i3eJGi5JEGXoQKsydRiVvHKEZ3c5y4HjRQsOUDdnis+CkKJfn3Zz4PAt4AgBlD2/wnX9amM/xHk1yl81N6+efPT7QcdntMqaQsbp/o8Gpc5txpofKL7a4/H1PlA+8e+/kY+/nF16VTNu/Gcffd359qesNLQChLp4WswGAshHbfNtse746QwghhJ5DwWwhQoGgY8cOnTt30et1F85fOH7ixI0bNs3n9yxCiEzMeDuJh7WUN2YgJ9OcSMEdAIBGXMhe+4BTU+IsIxoOfJvZTXIhWcn5667qr2ionYPkvc52r/nKh0cZd+RCkyC7USqSn6n79pI2IoeyMkEnP8XMIJGijHLr+9u/48YYc/WbL2nDMnkDSzyVTN7juYmpZfuhrP/W7IIkZbNkZ+dRYidTyBhQ84Somr/1yaKOQQ08HIXa1EyeBYGbhzMD2dZSR5U7qs5hBSwA1POsN+ntSZMnvx0dfU8oeM7pJp/yfI3FNhk6JECgPr5q3g9heRQA0m/tXbG+64Dv+nQOdt4Y86hKQ63tCr5R2NvbDxo4aNjQYVnZ2Q9jQ/TcX1I2vqZDQwghhFAFFLyne3h6vPnWm+PHj3vw4EFa/GETPSAiWZU7oTZaFs1rmvrrXRhpKg++XdStGPF/TtmHWLLee0s9zNc5KoYQmb51PWqJkd4oGjyg8sv+5B1tkKfZUcCk5hAWwM3FwoCg8FMZhZx0UZaBALApqUUzF1LQ5AjUJgBgkiMdVx/M6/W2IaChhbktLP0xIyv7s8ZB3QwiXvKfrzx3xBfkFpmMXObJ7/SUUADiYAxuaL53R2igJCNZWLn6QQghhKoJywoAQCaTde/erVfvXrk5OTFxFXsywL+VU3irElvMGsOGm8bCBB2leVoux0IB6CMNABH28RGyJsOmM9pzJgCArCz9f26KuncTt3Vjf8ljutcXMJxpW4QmpOC3tnzziXvGoYGipqWWTQR9GgpFnHnLKfW+gkk/OPogo4JPDltVrclBgaOjklBep9VRYt998fatYxsIC3+tFHvXBwCOYawGULmjytC1W9eD3Q5U4sAXLD4+vsy/EWAJAwD+TQIe/+xrZ2ev0ajLPMRWz9dYovq+XgwjHbDh4oANJf7AeXp5AFQmOXjw4EvQWNYJBAIAcHJ0dHJ882r+mwr2nqvHLoCkmo4LIYQQeslc0fw5ba7XtLnVW4qV+UAELAsADX18GjaccVn9noPwrNxuN0CF1xLm0mWX0qFlQ30LIaSaDcN7GPgY1/0p5OEZ+ztj0of11q2PkfON9M2E5EGkLJ0HIFz3d+O3vmYq+lTGebsDAGEquIRcSpJISw0KKW/jcSX2Z41+npRPU5xMLP1oqpPvvSjo1U2z8HPNXLXo9j15eLjjtjPiujxHOEIIocqKN3w4be6E6n5Pt6LgmT+Vg0P7tu0KttSTmG5pZDYezvNUb+JT88w3kg37YozxZa04wLL1FcAIJMvHSJaX/IurgmEYpp4c+HzzTa1tpTKsjz3w+aarGhvDrLDqTA5KW/bsoGT4hKgYLTgOnzzSm825sHHJ2l3nYzP0Auc+i/Z9N8z6CYhj30ocVZaoqKi9+/ZV7tgXqX279vW86pX1V0ppwdzYeeo8lcoBAKoiM/jcjVXmfN1ELBVXLqIv1qyp3IEvkouLyztTp1rZoWD2InVuajP3/S7Co+mpAGD/oqJDCCGE6ghfydfr9zheul2N49HatG7Tp2+fsv5KgfI8ZQhJT4ns3Gi/s/CEVmNXmfd0TnL2jmBqL33bhvQkmzu8Hrn0o30SB3yKcl9UxpLOud1/kSUG6J2o4MgNMQdAlJrJvU2sWr5xk9uuW+IMA3XukLZvXl5Fi6VmxkSBMLam60rsT0BAADgoM3VKBQc3+ORH5Q5soWvTRN+mfU7bdpoA4jvztADTgwghhJ7iIjz8858PqvU9XS6Xz/rwQys7cBaOFbAajdrOzh4Akg02PZcZfT1r2m2LraP1KJT1JihmCSGFExEyNp4NCqcmrL431mpLDhJlx5mfvFGPsUQfPXiXYxq7u4tAF7Jzw/EoEwCAOStDbXyyN7VYLBRkMlmJHyQZZ+tHVUxmRmbE6YjKHv3iNPJt9OxGSinH8wKWjY+PPxYScjr81PTp07t261o1RT5/Y5mT45N5Xvn3O/2XhOmqJqiXorEa+DSA0pKDFotZIBDm5uWFh4dHRET4u10buLIgh+v1giNECCGE6gAH4dkH9+wjTlfyF0dbKO3s+/Tp/ez2gt/5UlNSw8JOhoaGdgxKGFX4nm5XqXLItRsyfR9Nx5aGHm5qT4PiqzNCHgB44cETirkfasZ0MpxqZiQG+7P3CQAwKrO7EHTnHDdckJgAAEhWDlvpD8OVZBGk5ALjrgt2hdtpZexjFIUfcA0/AMByAX1St01X9+ysk522t3E8BEIIoVeHnL3/4F56tb6nOzg4QGm5QY6zMIxAp9edPnX6ROgJJyen+Z9+Wl1B8FyyFniRfv7f6nPPLhtChIn5wNiLOipJVK4NGT+eS9YCoxC1sYN7Tw8PoxZKAYj0+dJ7VZYcZARClgDHiOQOHo2bdxoyYcqELl5i84Oda3bc5QCy0tPN4N9h1Pi2936/kZrPCxQKiQCg6MMNTU/LoGzTfm/02x0dkmix92nmob92J7Wco14JhR9JU1PDwk6GhYalpj3nTNgA1dJY946GxL03fejytfHM5oOX7mfkc0KlR6OWHvkRlxKqbv2c2o7jLCwr0OsN58+dK5jitGBIpb9bTUeGEEIIoYoo+ACWlZ0dFhp6PORE0sOiWUGCnvfM6puKyyZ1586PPnCzZJxWHS/6iJ9xXnX0bc3QIWmunlR/XXHRCADA5wrTzeDfPHd8kOT3e8J8ShUy/kV/GOakIRdFk4fqZs9Ny9jiFJYoEDrrBnQqNsqCNfXuZcy8KYvKZDkBWHSMkQAhlBCuy9sJG/rAH181WH2DtVICQgghVH04i4VhWaPReP7c+fBTp69cuVwwkUiVDbcqFTWHJ1nGNpPM6cIxtww38jgdT+zkgkAZfzmds1Dz8Xjz2FbCiT3sDZe0Rx5Z1DyxkzKSss92MtEytrlwSnd7/SVtWCanocRJKbDTm+MMkKXjeSLs2ljyd64hhWe8nFhDhvlRBQcZVlVyUBA08897M0vEzuXe+nnx3FVn1RQAskL3nJjZbXDvpbt7L32yDxdd9B+J4aGRs5q3GPVV6CgAADBfXz1o4o9J1o+qs1iGLRjmmp6RceL48VPhpxKTqnCiumpprK3/XbWt1/9N6/fx1n4fPz7GfG1tv3E/J1TlLJm1Ec9ThiEmk+nsmbNhJ09eu3bNyqRFCCGEEKqdWJa1cBYBK8hTq0NDQ8NPhsfExFR5KTRPcTSS6d5a14IXbwmRP3niQqf430nRiBH65pQJvSAvGEZA8xR7Lgq6ddMs/UJT7FMZebEfhsmlP13/6ZA8wj97/TfZxbcX/p9SO/X91M7Fv1XwgsPn5FrG0L+bwcEehnYyrL4hf5ERI4QQQgVzsvE8f+HihdDQsCuXrpgtZU0QWC2i72h+r6d6q75iTf0nSxCbMzQTj+mSKTyIUv/o4TDdTfJBb8kHxY4yPXsiAACIidT8z1M1wUk6t5+0aMJGeuJUxvJEmpxsjGkhDGyk3N1ICQDAmzftz/61grMTVkFykMu4fys2sKGrg71MzPKG/LzMxOjbl88c+/2PE5G5RSkSmn148bS5abOnDmzr5yZnLQZNXk5GauL5+3kF2Uwu5udZ8+yXfTisg6+DyJibePt+BiHlHlVXqTWa8LCT4eGn7kXfq9ozV19jUc2lNRPG3fnX1LH9goPqO8lZQ05K7PXLSWX17DrDYjFfvXI1NOzkhQsXTKY6f7kIIYRQnaXT6SJOR4SdPHnnzh2er7bfNqkg7ILU2ErL3lf9Hlt8Rh1y/bgqckh6M04WcrVotj4qOLyxwdzM9KlddX6OHMsxmnxBRqbofBL7Ij8M8zn2nyxgo8dlvNHW4G0HeanSiGSuX7CxoI4IEV69LK0XaKxnxxMzm5wgO3LQdf0pAQUSckYytA8cPF/mSAiEEEKoOnAcd+PGjbDQsHPnz+v1+hqJgZpN/3csOzpIPqy+yM+OkRKap7VEpnOFGUqL5X+h2bFN5G81FAXaszJC9UY+RW2JTLaU+vAlNZt+PJ4dGyQf1UDUWM4IKZ+pNieYgADwubqVZ8isFtLWSkbA8SlZluzSzmAdUTq5PP5HwfqwFy9eWr9xc2Uuvbb6ZcdPABBxOuKlWOPCwcEhLy+v3I+kC+bPLxgEO2HSlBcS1wsya+aM4OD2ADB48JCajqV8YolEIBBo8/Ot7zait3H7SjUAfLDW69CZV3RBEmKvWbwwvX+qS9//2L9SMwPUFNZJ+96kjDdaG7zsqCFb8cWi+rvLmijKNtiCVcuzZ8qWt/TXvvdddq2Ca46+erDvveIGdVFv+uQhAExear8vtDrnHLRX6vQ6s7mcMQX4nl7ApX/i6RnaC5v8JofgqiMIIYRs8sLe04VCoVQmVeeVs3Zr125dF8yfDwC70lxsX624DhjvntHcTgclsy42L42CXpScnJxq/LEaVSmjwVBuZhAVICJzkJ/RRQKYCKkGtM24B1d33/+yk6WweoX62UsS/91T56PkBQxVKKkpH1jv7F0/R52Zn+tdqRs/tmCRZ2q7UuSuhkA3i6Tu1GbVVEupsO+hFyNPnVduZvCVxTjqBnfU+btYFEIqkFj822StGqMVUfG1+y90ACNCCCFkC7PZXG5mED2l2lYrRgi9EPIeSVc+zo/Z5Ttij/iFzHRImw5P+nowv391g03x1fRV/fmLeAFB1i4EKCHAFF2ruHnO2AbUlOjw73WuIcmMUMmBFsCJMgBMrfxJSN4j6cpH2vt760/YIS+5WhftPjP6pz7sj582WhNdW5ryqdquIbWuk9dItUhapO98N6+hI2cn5VmO0WgEifHSi9fs/wy1i7L1h5taV5MI1ULiJtlrPlUrir9EKEk55bw7AV81CCGEUF2AyUGEUMWoPAx+LqyoOr8OPH8RLyDI2oRc2e3beveTf7vVNykJOb3P9WAiSwGM2QIAgASnsW871VSI5SN80xEP12f6vHNQXLsn73y6tmtKLevkNVMtrIOxeX1z4UMpDK9yNKkcTS3a5E15Q7pjo9fqC8JSZ2x5Si2rSYRqI5FGEnbb1Mrb5K7gwcymPpSePum08ZA8HZ91QQghhOoETA4ihFC1YESckx21aNkcw4vOOojFPKFMZi5Te572sqU2OMp1nfJwwUOflTfwOTVkOxL5W8PXf5MYKJXbmxv7awcPypzYSj95XiJZ6bPyJvYlhKpA3m3nWYudazoKhBBCCFUXTA4iVKc4tcpcNCw/yMvkqeJkAjBoRNfPO369y+Fa0ZQLji0y5w/Nb+Ztqu/ISQnJyZBcOKf6fq/qVuFK57TL9JgdA9iNH/t++6AwiaPsnXRxdv65Df5Tjhd9zWaMs767OwsAAPhsh4nTPM4+Mz6HUerHjskY31Hnq6K6DOm5W6xbsQdaiUqzYG7GQB+Tm5ynRkFspP3Wna5744sls0orovyjSkTw1BncTZPv/zqEP7TWb9bZolAYw8f/efCBSvnuO57XmlivGQAAIjUOHp0xtbs2wJHqs8QRoS5r/1Q8LOXaddPeTXuvo8FBAJQSTZrdyiVef2aUW/k2FCE29R2e8U4PbTM3jhjZ5Hi7zRs99qWC35i4Q2PNf33m/+nVotQbw41ZendMwX9b5Mvea7DTkrt9a0qHa/WCVynVNpRlvQWLs94uZdXGs6e5cdAlrXP6xI9Sb87z2lvKDmBL/yzxEhCCOl168rDL99HmEYNy+zc3eNmBNl1y7B+3NYelj59ftlIPyiY5s4ep2zcy+jhxUkIyU5UrFrvfH/Dg6douo13KQah/r9SfJmnbeVlYvTDyhnLrbqcjKYyNVwpQ/iuxtPg9DudVsptZr6tinRDaTY39dajl6Ff+H0QUdRpieXNZzBfNZMunN9iRWWV9rwBvISYOKJD8XNH1i6Lrl+xPvJ2wbYRhwr+y93zscpev7A3Httc7QgghhBBCLztMDiJUpzj6q4e2NTx+YctVxi6vpbZqwI9a5BTNAQA4BahHBj/egTp76gaP1vXtrJuzyPNIVpWFQeTaxZ8nTvamBVkNsYdukAcAwJOVRi1co0CDlxAAAGSWwHbZ6xpbzLO99udaPW/ljioK6vZVefbg3PYt9KKz8oIHVxknXbAnNV6TXTWBc7k1I9HPXJ4wJ4AvSFNI3PVDxya1cvUavtEup3h6kjG/MTvpk7Yc4ZisLIbIOJUjGPMBbKl860WIDNOWJMxvzhXmSYSWxk1Mikqv3mq1rPJbsDgr7VJ2bTyLS1cu+JpruDxr5dzMe8ucIyt1aU+9BBzcdSOnJIwstoPIQ/fmtERHfaPpYQK+vHpwbZkzseuTJnNxpEbdMyMfK90uhG/VvajvCk1tu2W0bqlbsdB7R1KVDTUtLf7n6GY2vgSA3LoizxyS076lThxRVBNSXVd/yj1QnM6p0r5XKsqe/5/b750TJjVQD27ofDeWVObWYevFIoQQQggh9NLD5CBCdQ5lD2/wnX9amM/xHk1yl81N6+efPT7Qcdlt8niHQ981mndaYCC8RyPN25PTpgblfj5ZceEbe1u/9PLi9cUGND0bQbORaZPqU/U9x2U/Ooc8YAWOhl6D0hcP19o93kOnWLe40aIkUYYehApzp1HJG0doRrezHDguoGUXUf5R1oO8Yxeuzh3ZRtNSIL9kAQBQBOiaseTOLVkeBedyaob6D02b2YSmX3Fd+JPD2RSi9FXPm506ulfG+L/tNiY+KYTItP2bcfx959eXut7QAhDq4mkxG560TqWLaDgo7eNmnCFBtfoH50PRQr2Q865vySkru8Gze4oPbQMgqhJ1abWs8lvQxnYB67XxjPxI1zm/GH6fkvHtBOnr2+SaymVhCl8CAi3lm/RI/X6G2lMn37zRbddNcRa1BI9K3vy6rkcftWu4YxpvQ7NSNmSLz4KTolyed3Pi8yzgWbK0irVLiTjJgzOun+1Rnk9mxS66kRNS53fRzns778gqVXrVvBJLjZ/6j6hcN7P1JQAAxijF2fycYS20zVnFZQ4AQBqY31FKYq/JEznqP6rK+l6ZjLKTt9gJfUyB3jzEshW/4VD/kbZeLEIIIYQQQi+7WrluJUKoSMBbcff/jnxQ8L+9sf9uZEPOgIImR6A2Ac8xyZGOqw9KLawloKGFKbZDvprVccBbmOR7yi9Wef6dA44d8noqqihoxjigo4kxyb772v3vGIHOQtTp0v0H7O6XnLZc5Ze9ekXsmV+ibm59sLyTmQVwc7GUe0uq3FGFjPJDlwXEOb9f44JqpM2b6aS8+NQ1UWFoVmqGMQ7tbhBo7VZ94xyWxBo5Jj1GteJ/dvmMsXMzc4kAKKEAxMEY3NAsIQCUZCQLn6zAW+kiWOOQHnqxRbr+S49dt0U5JmLQCqKjJBmVmwzeelm2tWBxZbaL9dooBYk+6LnyEtt4SMqidlwlR9AVvgQIZ2Yjw1x3xRLCspHXJWk6YtYLz/zhFJIPrLvJm9jWrBRy0kVZBsKZ2JRUofap4J+nXShz6YRjWIJAbyG5qfKf1nv8mg7yFppuFUuDlV8bJeInle1mtr8EAMAgP3yVJa6aPoW3LNqmvdYBxEfOibmq7ntlyVazlIBMVjj0r2K3jgpdLEIIIYQQQi85HDmIUB2XkiTSUoNCypc5zC9ffuIuM6KjqZErBU0ZO1WI0OTjSvl02eXSp40DIFz3d+O3vmYSFsbEebsDAGGsp4Iqd1QJzLlw+9Re2QO66NdFycyMoXMzjqaoTj4sfe8SNfPQ5OtGGbF6w67IDSV383SzMCB8nLugOvnei4Je3TQLP9fMVYtu35OHhztuOyN+OqlU0SJYs58n5dPkZ1Or4plTodWyhOZyWrA4q+1SodooxAn/2uzR+auk199PC4/x1FbuAoudLSGDgK/FzQ6gYMSiWfgwmxBHXkoABOU0a/nnZ41V1i5G6YUYZmJnk48LBXX5u1eS9aa30s2sH/h0XTFnTtlldc/r28HwVbSUE+r6tTPTBw4HE0lV9j2rHO05QkGnZ2glbh3lXSwu0IoQQgghhOoSTA4iVKtF/erb+NfnOgM1MyYKxOq6tZQHACgc4QMAlEpEz1Vowciasr56E6Vmcm8Tq5Zv3OS265Y4w0CdO6Ttm5dn/ZyVO+ophjuqfSk50zup2+2UXXTTdvOgD/+xu1v2F/2na6Y0YnHJxCsVHNzgkx+VO7CFrk0TfZv2OW3baQKI78zTpd9vbS2CQEXSoOWzfjnWW7C4ctql7Nqw0iNprt1nmx3aLslZPk2xruQzyJXon2YzAKHCx9VPiJkDIIXXaGuzlqVK24UwBacEqKJXYqkq3c0qVFe628ojWbnjOqtb7JZGBqr7O5IbB+zjOABBlfU9a8S6ns05hoqjEhlQ5lXi1vG8HQMhhBBCCKGXByYHEXrlifXBjXkwi+LTCQDV5LOUMTepz5F7padvLByhlMokZZ/QLIp9RBjP/J71XW4llPI9mlGZ3YWgO+e44YLEBABAsnLY4qsNlFpEuUfZFCQn+f2o9J0p6pEtXJM88psQ0U9nJWWuPlq8Zsyi+HTCK5TvTPcMK3vKvEJGUfgB1/ADACwX0Cd123R1z8462Wn75yqCMcSnE8Zd28md3kp57uxEeWVZb8ES+5bbLmXUhvUhgbnX3BYd1m5/7dGcbFo8gnL7Z8VUqFnLPkOVtAtRaPsE8mAWxVXVK7HsgCvTzSpaV0bZ7yfF40aphwc6q3poXE3y78JFnA0B2N73ykS4jm89esMVLAn2B+MI413xG87zdwyEEEIIIYReHjhzDkKvHsIH983u5W2RstTeXfv2rNSxbqC9ZXc6HwBIXIxETfmub6SNCzDLWGDFnIt98ZEyJD1TQFlTv36ahjLKirlGQXpPtuT5efH+UxIza/hgQfK0NkZHMTAMVTpwsqKz8LnCdDNIm+eODzIrWACGKmS8oLwiyjuqxBVaCTLxtMNJg2VA/5wRwXo22f7g/eIXV3bN8OKj50S8Km/5R5l9fM32QmAY6uCm79nU9HQMrKl3X00LV07EACsAi44xEiCkKMNV6SJ48ZGzIk6gnz0/dUJzk4OYsgLe3UfXRAWVUV5Z1luwxJmst4v12rCCMmd3eex+xHm6lOh+5fXPKq0H285Q+XYhYKeyyAXAsLyHf96ChSnDHSD7ojKsql6JlbhkK5dT4boikSdU13nzkOFpb3ey5F10OJJrUwC2973HGJayBIClcpWpZfucRcvifhppkFhEu7c53uUrdcMhVoMkXJfJcVd3xi1sydlQ4wghhBBCCNV2OHIQoVcPoT5dHm3r8ujxBl4jX/OzsmCBVO01x50PNB82Un/+pfrzYsc8/q/EK3aR4/Qt+jwM7QMAABbp6pkNf0wtUUD0Px5ftUyY30y9cJl6YbE/FIzWoXmKPRcF3bppln6hWVr8KOtFpJVzVHFWgqS5djtPCfsOSP+AQtRvysjizxRbqxlye5/7tvZJ0zqmb+2Y/ngHc5RbvwVOCcVOQpTaqe+ndi5+c+UFh8/Jtc9bBLmzz+P7tokzGud+9nnuZwV/o8z+L5vMOleJ5Fg5ZVlvweKst2Y5tWEV1cq/2arsvSjXq9j1lds/K8jWZrVyhrLbBZqOj/t7jOnsJr+3j5U2+o9wA2fHDJz9ZIMxVbXkp8JFw6vilViJS7bWzSpaV1ya8pcrmV93zOvBibYeVqipTQHY3veK0KC34u69VbJojfTnjV6rbrAUACp1w9lqJUhi6N/N4GAPQzsZVt+QlxkXQgghhBBCLwkcOYjQq4cyNyJUpxMFOo4Y8oU3zjp/OL/+T4lFSQeTdP1K71XHZQ/yGI4Hi5HJTBdfvWIfnkQKvtpziU6zvnUKSxToeLAYBHFRklJWDjBKflzpO2WHKiJeqDYRjiOaXFHkbbs/r4jNAEAFhzc2mLvX7nYGa+TAYmJyskXR0fLzSay1Iso7qjirQTLnDznc5anYIvs9TFxi5I/VmqFa+ZpFDef8pjyfKFAbCWdhMlOl4ZEiU8miCRFevSxNyGMsPHBGNjHa7of/NJh3qig39BxFUJ3s6yUNZ/1mfzlFoDUTs4FNipPFam0YhVeacgalfCoAACAASURBVC7HeguWOJG1dimnNsqTd9V1dYSgRN6pvP5ZxfVgyxnKbheWBaAkX1fKlJ9Z9+z3X5HGZLA6M+E4JjtVemSv55hPPA9n2XqlNr0SK37JVi6nwnVFBUcPKlM4MMY47IomNgZQgb4HwOWIbyUJs7SMmQPewqhzRbdv2G/f7jXs/YYrzgstRWFU4oZjLUheEnJGkpMvOXi+og91I4QQQgghVBsRpZPL438cPHgAAC5evLR+4+aaC6nq/bLjJwCIOB3xxZo1NR1LlVkwf37Xbl0BYMKkKTUdS1WaNXNGcHB7ABg8eEhNx1JlRvQ2bl+pBoAP1nodOlPaxHMvkN+YuENjzX995v/p1Vd4Vn1Wv3BT/Lgkjx6rVVlFaZsXUDNY+a8Y2vej6B+6ib6a1XBzGStiI4TKNaiLetMnDwFg8lL7faHimg6ndr2nI4QQQi+R2vae3rVb1wXz5wPArjSXWxpZTYfz4ox3z2hup4OSWRccOYgQelXYO1jkLIjkpl5jH41xY48dt8uugiUtECoDa2jnz1vi7I6W/6gvQgihQg36JP+zJW5hs1frHZp10s74KD5sR1TM3ru3/ps0zv15T0jsNUvWxJ6era75L9/V46l+Uqu6je2V79kz5Z8tsSta14qwa7k636WfU616CaCXVClzDjZu3HjWzBkvPpTq5uLiWtMhVIs61liNGzeu6RBQHcWYR39yf2kQBQCgkHnJ89vLpTyPjFBVYRzMSr3o0H5VHK5agRCqFrTNuPitQ7iQDT7zz1ViAffnPLyaUDs3Q5Abd70uj7B/puaF+tlLEmc2LJwnRKGkpnxgvbN3fJbuc9d97FpVok1z4JZAROYgP6NLTuUn5a3dnuontavb2F75cldDoJvlbu0IuypU412l6ro03jlRKboq1S1smhS9jvCWljKbdynJQUdHh4LnOuuYJgH+S5Ys2bRpU3Z2dk3HUpXqZGMhVA04MWV1HAdq0YVw5y92K5MwZYOqE59pv+BjfOQQIVQRrHHmuriP60u+meezMf7ZL3l8p2lxOwebw//jPzWMBQAClBBgKvtt8DkPR5X2VM2Lm+eMbUBNiQ7/XucakswIlRxoAZwoA8DUzae8aNPhSV8P5vevbrCplH6ObFfravKluKvgnRM9q0FpybJXzSu0WvHNGzfr1/fa8n+bt/20/ciRIzUdDkI1I2aPr9+emg6iRvCSLQv9t5T99xdQM69u5SOEELIFsbg5UCLSvzMh74/VqrSS48VY7+xP+plYQhwcOBZYDsiV3b6td1e+sOc7HFXa0zXvVt+kJOT0PteDiSwFMGYLAAASnMa+7VRTIVY3lYfBz4UVYXrludWymnwp7ip450SodCWSg3VpCYhSiUSi8ePHffDBjE6dOm7YsDEzM7OmI6q8L9asgbqzvApCCCGE0CtPZHFTgEnDsm0ypwUoP4ss9o2fcP3fyG5uZnOFnKM9ZyUVwIg4Jztq0bI5BmLL9leBLddeg/UjFvOEMpm5pSxtX1Ne5d6CXk1453zWq3DtEacjBp+u40kw271CIwcBwGQy/fTT9gvnL8yeM3vz5k3btv109OhRSmvPGzFCCCGEEHpFMUqLEwPpES77Ax5NHJP735UOKUWDBwXe2TM68ed2uORPeNTZwVLwsKnfmLhDY81/feb/6VUCAIxSN+3dtPc6GhwEQCnRpNmtXOL1Z0aZ25863KlV5qJh+UFeJk8VJxOAQSO6ft7x610O19TFQhSbeg3OnNorv4U7JwWSlyOKS5CE/O229XbxaXxpu6mxvw61HP3K/4OIosdiieXNZTFfNJMtn95gp0WzYG7GQB+Tm5ynRkFspP3Wna5740tNjdEu02N2DGA3fuz77YPCb6fK3kkXZ+ef2+A/5XhhoURqHDw6Y2p3bYAj1WeJI0Jd1v6peGgBK3VSstpL38exReb8ofnNvE31HTkpITkZkgvnVN/vVd3SPDnWStEF1dV3eMY7PbTN3DhiZJPj7TZv9NiX+nTNAwAw3Jild8cU/LdFvuy9Bjstudu3pnS4Vi94lVJtQ1mMUj92TMb4jjpfFdVlSM/dYt3KeCSZqKzVvy01ZmNX6T8iY1rP/CAXnleLr1xUbf7N4WJOsRQDY5z13d1ZAADAZztMnOZx1lKiFOtxlq/MAGjzCXF7XzeHfOP//qnCOnIfmBD+rmHf5/6fXimIkDafELd3NPfX536fXCFVWPmlINS/V+pPk7TtvCysXhh5Q7l1t9ORlILjber/5daksknO7GHq9o2MPk6clJDMVOWKxR6H8yrZe63XRrG+DdbvAzsyq6xWS3RIIajTpScPu3wfbR4xKLd/c4OXHWjTJcf+cVtzWJpLnwoS75yVv3OiuufVSg4WiLx798MPZxUMIezcudP6DRszM7CnI4QQQgihmkTsLA4Myc2W79inGDcr618Bqs8LBg8Srs/onIBc5eRQycBRILHn5ARMT30jZMxvzE76pC1HOCYriyEyTuUIxvyytz/D0V89tK3h8XcDucrY5bXUVg34UYucogtm6RUbpi1JmN+MK5psizq5GZzcjKK7zttus8Um8iW3rsgzh+S0b6kTRygKp3GS6rr6U+6B4nQOgJRrFGjwEgIAgMwS2C57XWOLebbX/txK1ZpEP3N5wpwAvuDLtMRdP3RsUitXr+Eb7XKIDddedv04BahHBj+uEOrsqRs8Wte3s27OIs8jWeUVTQFEhmlLEuY35wq/5QstjZuYFJWe1cpqWUSuXfx54mTvwiVNxB66QR4AAKWXZim7/m3rLbZ0lfeWJnzSrOjanQw9BqZ1bqufu7Deftu/dVmJs1zWAiD3bsoyRue0DNALT8nNAAB866YGIcO3CjCyVyQcADBcqyYmxqg4E0UAqrTyn0X4Vt2LLkloatsto3VL3YqF3juSqmyslmvLnIldn/RkF0dq1D1H77V+YLELK+c+UHW1+lSHdHDXjZySMLLYDiIP3ZvTEh31jaaHCZ5e3QfvnAX/quidE9VFr2JyEIqGEJ4/f372nNmbN23EIYQIIYQQQqhmMXacCiBNw6afdfrjrYTXh6u33FVmUmDdc9/pyN361em8juusJYy9xYmBnJLLahGZtn8zjr/v/PpS1xtaAEJdPC1mAxB56dtLR9nDG3znnxbmc7xHk9xlc9P6+WePD3RcdpsAQKPBqXObcaaHyi+2uPx9T5QPvHvv5GPvl/KV0RilOJufM6yFtjmruMwBAEgD8ztKSew1eSIHVKdYt7jRoiRRhh6ECnOnUckbR2hGt7McOF6ZlUP9h6bNbELTr7gu/MnhbApR+qrnzU4d3Stj/N92m7LLv/ay6u1xhRz6rtG80wID4T0aad6enDY1KPfzyYoL39jnUGtFb0yEhoPSPm7GGRJUq39wPhQt1As57/qWnLK+xvPsnuIDCQGIytbL3JhIm41Mm1Sfqu85LvvROeQBK3A09BqUvni41q7UKiu7/sF6bZQ4i7Wu0nhw6kdNOcMDh+WbnQ/GCUQu2jH/Sp3XPm/5ZLuIr+wLU0i8eH2xYU0VirPcflJOADHyi/qcQUE6X1Z+jwMQ6DsGcgSgYZDOjZGk8ABSXcdG1HxfcUFfxZVf2nWSB2dcP9ujPJ/Mil10Iyekzu+infd23pFVqnQbXw/l1SQAAGVDtvgsOCnK5Xk3Jz7PQv1HVK73ltPti7N6H6D+o6q0Vgs7pEBL+SY9Ur+fofbUyTdvdNt1U5xFLcGjkje/ruvRR+0a7vjUXK5456zcnRPVSXVzBSwb3b0bNWvmrMOHD3/wwYwVK1a4uLrWdEQIIYQQQqhuCngr7v7fkQ8K/rc39t+Nnv5CJ1ZwMgbydQxvku04KBW1yxrrTQH44KHZrQx2W0NEHGE1OiB2FtWzZ6eEAhAHY3BDs4QAUJKRLMylZW8vFQVNjkBtAp5jkiMdVx+UWlhLQEMLAwCscVA3g4iXbPnKc8dtUZ4ZODOTUdY0eQb54asscdX0KbxG2qa91gHER86JC1KaKr/s1Stiz/wSdXPrg+WdzCyAm4ulMl9LGOPQ7gaB1m7VN85hSayRY9JjVCv+Z/f/7N13fBTV9gDwc2e2l2yy6W1DSAKhI9KriMBTQQGR90NAsKCIGJogXUVUVJo0RUV9IogVUBCQbqihQwiQSjZts+nb28z8/tgQQkiymxDYEM73vc97yWRm7tmTmxv25N47Bsras62dcue1134OBwYdbWKAdVA51xUffxiyowSU3coek7lqmrYO6WcWOsSrPwnenCgosRGLkZd8TVRQddpSQ7xMyjq4u42ySVYtD9qRwjM5iE4r/munPLXmtmrMv/u9pZauQlme6W8ROMRrlgf9msw3OUhpnuyrFcG/FIJPl9IBXnV43fXsJy4DsEgPJVJ0mKmbEgCAVhm7e9NX0vhUtLGrBABA2MLQRUyunJMWsA2f/DsySZ0+oDyUyTM7SGme9LvVwVu1IG2v7+NucdHNVqBEKyiyEMZG5+bxjaS+vbf2bFRptJZxoMGzWt4hCWOnkw4FbE4jhKaTLog0JmI384/95rvPAHSQTXVnBRVHzvqNnKgpekhnDlaw2e3ffff9iRMnpk6btn7d2o0bv8UphAghhBBC6P6Ty1iKI0YzAYCsg757R2a/MMT43RbHy485sv7x3a8HIJTBAlQI63XHW1zOJN2WwOvfRz9viX6mTpB4XXrkiPLbY0JjTcfd+NdubpbAyFlkYpYAAG2NCeFYjeyw2p3VjtSxf+VFfcue6GZZlixm+KaBne1chs8uNQHC9H3txjf/sfHLb8OoggCAUPVbQ8m3NQ/kKKFuzeakNbd/JSTQQdx47TXmrdp9vAzSA1epYd1tUQEcZNfWNEXbY0I4ViM9ntcQi0NrfZkU394sgGO1kjPurNitNf91ykZlt3UVvi06kGM10mM5lV67WRp/jRrdyxYdyEHZ3cbpgusA6GMJEktXU7/2jk0HeOEdjZFW6eyf6dlzdH1bs9sTSLtOJiUn/OG0gIEGTb47rOJTKdS4nrZm/hzoXJ9eT7W/qFp6b+0XAv/2YzWPA/c0qww/s4BAc0egHMA5383Ozy4mRMmKceR0uuuREzVJD3tx0OnatetT34obM3bMm29O7t279+erVxdotZ4OCiGEEEIINR3XtjaP3lrbCXIpQzjKZAUA4Iyy7w4IhjxV9DbH9OOJP94jtgEAR0xWIAJGzgO4/ZkDwPF2rWlmuFb6ZHtTp5bmTl1KHu2sjyXNp8TXeNxlwJydsnFAnHNcCPAIAAOMq6ucTImKPUWlL/TUtd8iTmqlG6QkF3d6pTNAvPUTHrfROunadYGbLwsLLJxfN832WTWWizgA4DiRoOYgazguFLKk5pzcuqrmc6pvjr3VaG1NE6jnm/Ya1NbWzbVg7jRIFLXm352MVRte5a7SEFzEedeKLsjP2QxduxgVhyV9OlnsV3z+vcjroSvt28ksPEP1f8QOWX4HcspPbqjku4lQt27osv/XW717b+3ZqKKmcQB49zardjsA4fgVP8SE2BkAUt2qSRw5b+fuyImaIiwOlnNOITx+/Pi0adM2fPnFli0//fHHHyxbv6n/CCGEEEII1QnnJWMrioMAJHGPz+khmvGDoeRo2PZ850HKbCUcYeWSmzNiKrMKjuwMOLITgGZiB+R9O0n3WE+TJN7LWMPxukXn4OWWAhVk6hoAiRo3zrdKfj0sfGGE7tlWft799AE26aojAgaA9rYH8cF0QrnmlMgGAECKSug7HzJA0+Uf6A00R9lbhjPkenXvS+2CG1rCyhSvTgo5VO2WWDXlxI1zqrmb0Nw1mnU26qJpynJDS6ggY48g7nLuXReOXLWVlk+oEMNj4f6XM120RbnMvzsZcxVtWj6hgo29QrjL2TfjERv7xLJgF6RrCQDnYAjHcRLRXcQJAJX6yW2fug4A2CL5zqT8Hu31/cLYgS3I6a9lJRZq/wXquc76LofoJ0Lh+k9eyWz53Roq+e4gMuOAVrcS5aL/A7jMZPXq3Xtd/sRVUcM4cJ+z6ipIHDnvDOOuxwH0AHqo9xy80/XryW+9Fbdly0/jxo1duvTjkOBgT0eEEEIIIYQeChIJSwix2Mo/ZQsUmxJoluHv2H3rMaAWC+Eo1ktyx8W07fEn9O0DGAEFNA8cJspKgBCO1HS8rsEx4n0JAlZgmjpTMzTKLuNzPsHGET0sNU9MIUkHvC+w9iHPasb3cJQl+OwpBQBgS/laO4jblY5pbZfRABQnk7CVZisQm4NwhOnU2RgmAgCSniLScWzv5zUvxNolNNBCxt+r0gQlVrj3hID1LntveuGA5nYvPlAU5xNofqyNjVdLTtzJW3k4bNcnivurHGKa8woyjo/LGx0IxsvyeIOrplnhnuMChmeeOidvbDubj5CjeWxQM1PLanaLdIOrtv76V2SnLW/OzZnYyaoUAkVxCh9GUt332EX+G6S3sKI/D4tsPPNbb2tGtnBIaE4RbHxtet4oPyg9472/DACItpDH0baBA/WREo4WMlGtzSG3l/nq2k9u+9R1AAAcb3+81Cw2vDyppAtI9pzhcUAdPyYr89VNf7ksihPtPF6+y1sDJr96BOTeDikPKJoNblE2d17usz5QnKA4ZADX/d+NTNbwPapv7639wupeXrXjwD3Pqvtw5HR/5CRMrwnp5zalz+vg5jxI9IDBmYNVORyO33777ezZs9NnTFu7bi1OIUQIIYQQQveBRMwSjm+y3nwHx9G7l7WMuv0cs43iCOMl4aosuSMK4ytv5PWs/E97lrf7hNSk0Fd7vO4TQMjp3wP+7JYzrEXx6hXFlY/XdAGjUfx4tnB597J+jOCb3TIdBwDAlcl+SeD16aNf9LF+UaWbJN/8KDtDVMpZYoeq9yrDunzqZTyv3JShfytKt+QT3ZJqGiWJ24O+7ZI1sbv2m+63NgWyXwscONdXXUNOKr/2mvJWfg7hmvXK/7ZX/q0v6qVL/6fQci6azmTJle3BGx5VT44u/WBJ6QfOr3HUX5+0jDtRjwqHi7aS/wxe1iFzTlvdvHd18ypddufEotrz7yIbbkvZGbzq0cxZbUo++6zks4potYr3vvMq5gAA1GflSS+Y2w/IPjgAAAAc4o+mRH6d526ccEc/qfKpywAAoOik4sAE/TOtLaazIQdKAQCMlxQHdGXPx4L5UvCfubcavbvkc23GpO8YZTu+Lmb8P9XN4SLMk1NTnpxa6cI874XflT/T2VX/d53JGtS/99Z6YTUtVTsONGCXvks4ctZh5KQsg/pYfLxgaA/LRxeldc4EavRw5mD1MjIyZkyfuWXLT2PHjvlk6dKQkBBPR4QQQgghhJouwkpFHOGIyVbbWWYrAahm5iAh/HNnxJlllIMFxkqrk+VffR4x618e1HC8HltHsSVes+eqPj0gSS+lHAxVlC3dcUpk4qDGP6FzvL27FLkMWFN8NidXVDx5u9dGzNwmTyygrQw4bFRJsSA5WXoyi3aGZDobMONneWIhnaPl2wDAJl69WPXhfklGGcWw4LBShVrhubNeR7JI+cZ/RunS+ZHTflacVPN0VsI4qMI88ZEkga3mnFR+7S7O4aiLR73j1TwTQywG/sXjfm/NCf/u5oMFamkaADiTZPnCyLifvc7k8ox2YrfQWemSNGPdZx650RZYRV8vbv7SD95Hb/B1NsIwRF8qSEqU/35WaL/jm1JL/t3JmFusoi/fj5z8k+JMHm12EGOx8N/dQWNnh/x58/kSjNo3bqXvITXPxILDwku/Jqr65Ik69pOq3cZVAADAGeVb4/kMR8UfkRc6b2qR7jjBZ1j68D9euWzlM+8q+TQNwBGDqZodGYuue/11VpxSQJvshGGo4jzxnm0ho2aH7C66eYar/u86kzWod+91kY1qWqpuHGjALn13cOSsw8jJivYdE5UYRLtO1nUdO3owEIWvv6djaNSaRTabMX16eHj45s1bcAohqp9hj1u/X6wDgDc/Dfv7WB03qkBuiBiQs+Z568m1kR8l3vt9SRoN2tf4+osFzz9iCZNzlmLZx/PDt7izlUnNiJd+wTztoDz/Jz73avA/zDYGVfpJo+o27ic/5LHcL//PfH5D83fPez7sRq7Jd+m71Kh+BGrxVC/dutnZADBhkdf2g0JPh4O/02/jP0gdP9l4al3MhH1NcKP6mFHpf4+2//FBi3fONeqfEdS4cU9MT/6qj2BZXOT6bE/HghqHpj1y1q6x/U5HleHMQRduZNyYMWPm5s1bxo4d88knS0NDQz0dEUJ3g+v0Qsa5Lamf9HDU69+5d3n5PcLJAy2tAx2ixhRTQ7sj83zz1IXqtx8zNVOwPIqTKTibAWhV8eb/XTs2p1RVr6GdCOytY6z+ooZ84l5jUqWfNK5u437ypQGWVo0m7IZwD0eVhuvSOHIiBABAKU1Pdze18HfI+BxP5GjRqejDUUYBJzyfSj9s728Rchdt6dyCdaTL97pe6ouaJhw50YMC9xx0zbkL4ZmzZ2ZMn7527RqcQogaF9o65bP0GeGiFbOarb1x55s8tsfE9E1P24983uKVQzQAEOAIAaq+7wbv8nJUb1UyL2xXMjqCs6l93v4sYF8OxVcwYATw5SgAqmn+0Ydr82zW8qfZvz6KWFdNP0fua3SZfCBGFRw5EQIAYcvipe/oZJV7Mkdy//Xbch8eJ4rQg4nysSvMgr//8k7HRzg8rHDkRA8KLA6660bGjenTZwwfPnzs2DHdunf7fNXn2dk4NRw1AsQR6MMRgfnVsWW/feStub1qTauKZw+00YT4+DA00AyQs1uaP7Kl/o3d3eWo3qpmPjDcpiAkfnvALjXNAViLeQAAmb6jx/t6KsR7zTvYEuNPC/DfUXetkWXygRhVcORECABAoBcdSrR1VNmCZCzY6bxscfxh37V/S7X4F3OEasAWes2d8bDvP/CQw5ETPSiwOFgHDMP89ttvZ86cnjZt2po1q3EKIWoUBI5AGdj0NN2pcGKs4oOkSu/4CTPo+eJ2drqUzyi9mFpKAZSA8ZVzDiNdYiHuHH8YuPPaPZgfoZAlHFVYWs3m1p7yMPcW9HDCkfNOD/NrfxiUJfrFLfDzdBT3T8ovzWN+8XQQCKEH3MM2cqIHFxYH6+zGjcyZM98ePnz42HFjunfv/vmqz7OyszwdFHp4UQqHLwXao/5/xeaPG1W6cbFPxePVeKriyT3YEz/4G8bm9/RxOBebVtldm1KYJr6meb27xYcHHEf0GvnihWG/F9R4vMrlvh0L5z9jaB1mC/FmJDyw6AUXTiqXb/Y5r6sUotDW/+nCV/ob2gcxYiBlJYL0TNG+HYHfJFbeaIPr/Era1qGOvctavHn05rJY4vjvuykft5W8Nylik0M/d2bBk81sgVKWs/LSkry+2RSw7Ua1pTGu16SUHwbTa2c0X5lR/u5U8XhWwlTDiTUtXtpf3igRW59+ruCVvsZYJWcuEh496P/p77JsB9SSk9vTXv05yvaFc4Ya2qps4UpGTEhJgejUCe8N27wv629dW0vTznQ98WzBq/2MbQMZYqVzbsjXrw3enlfdtugUM2rR1VHOjx3Sd1+P2OQo/f6b3G7nQ7t+qNC50RalMI8eVTCmu6m5N2cqEJ+4TAfWsCSZeNeWf3cy5mZXGTSsYOJjhtb+LKsTnk3wXv+zT0JJpRIDZY1bdTUOAADYYp9xE4OPO25rpfY4XasxAK7d2PRtI+37VrR449/yHAU9mXnkNcv2JS3eOeuMkGs3Nn3bc8wfS2JmnyUNmPxqEK5F/7zvXjR2DnPQZn7SRcU3W3z35Dqvd6v/u8ykomXJ1Gd0XaKszXwZMSGFeYr3FwTvLqtn7609G5X6NtQ+DvxQ2GBZva1D8kGnFR/e7b8h2T7sqdJB7SxhcjBqRf/8Gbh0t7iUqxIkjpz1HzkRQgghhFDjhMXB+nBOITx95vS0qVNXr129+cfNOIUQeQqRO3woUlos/WG77IW4opdjvZc4Jw8SZsBzJbGligkHRU+OAJEXIyVgq/KOkLI/PzVr9qMMYaiiIopIGG8lWA01H7+DsoVu6KOWinFE6m3t9Z+8jhHsiPm+yc6tVYSWiQsz57Rlbm62xfkGWnwDrYKrft8m0pV2XyGXz0oLh5R06WASHpWVP1dUbOrdgmMyZPElAGImqpUljA8AABJHq87Fn0U77FPD/iqtV9ZE5invZU6LZZ1vpkVB5qGjszoGhD27Vl5C3HjtNefHN1Y3vGtFQji/ENPTz5me6GmaNj9kT5GrpjkAgWXiwsw57Zjyd/l8R3RLm6zej1mttS0iNS5Yop6g4pzfGWGw6algAIDqW3PUnH/3eos7XeX1RZmz29587b6Wfk9qej5qnjkv9C/36wu1xOlSbQGQ65ckBc+VdIg18/+V2gEA2EfaWPgU2zHWSp8VMQBAMR1b2iir7Ng1AtCgyb8TYTv2vfmS+LZH+xQ80sH0/jzVD1kNNlcroEPJuN63erK/krOa7qL31n5hpRfmYhxouKxW6ZA+QabhL2UOr3SCINj034lqpTlq0iFe1d/uOHI6P6vryIkQQgghhBorLA7WX+aNzLffnuWcQtijR4/PV61SZ+EUQnS/UXLGG0Cjp7XHfX/7v8yRz+q+vKoo5IAOKn21O3N5q+9JE9PTSCgvhy8FJbfvhUwkxkFtGTbVb+SigItGAML5hzjsFiDS6o9Xj6N3r2k+J55vYNjglqXvztQMbFE8ppXy3UQCAFFP581sy9iyFR9/6b/jusAAbNDjOf+8Uc1bRus12XFDyTPtje1o2RkGAEDcytBdTNLOS9UMcCbZZwui5mcJCszAl9l7jMhZO0z/XGfHzv28uq+r5VoM1UxpyWnPBsz7zud4LlE0182amvdc/4IxO+Tril2/9pryVpGQv1dFzYrnWQgbHKUfP0HzSuvSJRNkp1Z4lXC1Nb1WDZFPaWa0ZSyZ3h995fd3Mt/MR0LivAAAIABJREFUZ1ThjpKa3saz9C+VJxICEG93X+ZaNdd2uObFcE53Xfnu1377Mmie0tL/Ke2CZ43yalNWc/6h9mzcdpfaukr003nT2zCWDJ/31vvtSucJ/I2jXs6b1aXsvQnyo8u8yktIrHB1pWlNdYrTZT9xEUCKNMFc8lRrU3Naep0B4Jm7t2IIQGRrUyAlymUBxKbuUZw9VXbK3MDJr+51koxjAR/8ojiZQwv9TcPH5s3pZZw1vmzPh95aN38eXGUSAICj933ZbO5hQSnLBvqyZQ6uxbD69V4X3b6yWscBrsWIBs1qeYfkGTm2Zb+8DZN1ISbp+rWBmy8JizhH1xE560ea+g3QBRxRVtnLFUfO+o2cCCGEEEKo0WqaT7W8b5xTCOPiplIU+XzN6pEjR1JN9EGhyFNi/y89dUdShvO/29Lejqr6hk4oYyQUGEwUa5P8sEss6Fw0WsUBsF2HFne0yL/ZJ2AIrTcBkTu877w7RzgA4mPtGmkXEQCOFOTwS7maj1eLA30JT2cDlqFykpQf7RI7aEdspIMCANr6VB+LgBV9uSzkh0RBmR0YO1VQ0zZ5FunuczQJ0A8of41cpy5GHxDuOSF0ljS9Y4o/ej/t2I/XLn2T8V4POw0Q6O+oz88bZR3a18Izyj9c4Xcoi7YylDbF+/2f5AbK2rOtnXLntdd+DgcGHW1igHVQOdcVH38YsqMElN3KHpO5apq2DulnFjrEqz8J3pwoKLERi5GXfE1UUL9JybW3RVkHd7dRNsmq5UE7UngmB9FpxX/tlKfW3FaN+Xe/t9TSVSjLM/0tAod4zfKgX5P5JgcpzZN9tSL4l0Lw6VI6oC4bedezn7gMwCI9lEjRYaZuSgAAWmXs7k1fSeNT0cauEgAAYQtDFzG5ck5awDZ88u/IJHX6gPJQJs/sIKV50u9WB2/VgrS9vo+7xUU3W4ESraDIQhgbnZvHN5L69t7as1Gl0VrGgQbPanmHJIydTjoUsDmNEJpOuiDSmIjdzD/2m+8+A9BBNtWdFVQcOes3ciKEEEIIocYKZw42AHWmumIKYc+ePVetWqVWq11fhlBDkMtYiiNGMwGArIO+e0dmvzDE+N0Wx8uPObL+8d2vByCUwQJUCOt1x1tcziTdlsDr30c/b4l+pk6QeF165Ijy22NCY03H3Xinl5slMHIWmZglAEBbY0I4ViM7rHZntSN17F95Ud+yJ7pZliWLGb5pYGc7l+GzS02AMH1fu/HNf2z88tswqiAAIFT91lDybc0DOUqoW7M5ac3tXwkJdBA3XnuNeat2Hy+D9MBValh3W1QAB9m1NU3R9pgQjtVIj+c1xOLQWl8mxbc3C+BYreSMOyt2a81/nbJR2W1dhW+LDuRYjfRYTqXXbpbGX6NG97JFB3JQdrdxuuA6APpYgsTS1dSvvWPTAV54R2OkVTr7Z3r2HF3f1uz2BNKuk0nJCX84LWCgQZPvDqv4VAo1rqetmT8HOten11PtL6qW3lv7hcC//VjN48A9zSrDzywg0NwRKAdwznez87OLCVGyYhw5ne565EQIIYQQQo0WFgcbhnMKYUJCwrTpU9esWb19+/ZNm350OByur0SoVte2No/eWtsJcilDOMpkBQDgjLLvDgiGPFX0Nsf044k/3iO2AQBHTFYgAkbOA6jSJTnerjXNDNdKn2xv6tTS3KlLyaOd9bGk+ZT4Go+7DJizUzYOiHOOCwEeAWCAcXWVkylRsaeo9IWeuvZbxEmtdIOU5OJOr3QGiLd+wuM2Widduy5w82VhgYXz66bZPqvGchEHABwnEtQcZA3HhUKW1JyTW1fVfE71zbG3Gq2taQL1fNNeg9raujlv3J0GiaLW/LuTsWrDq9xVGoKLOO9a0QX5OZuhaxej4rCkTyeL/YrPvxd5PXSlfTuZhWeo/o/YIcvvQE75yQ2VfDcR6tYNXfb/eqt37609G1XUNA4A795m1W4HIBy/4oeYEDsDQKpbYYEj5+3cHTkRQgghhFBjhcXBhqRWq9+eOWvQoEGvvTbx0U6Prly5Ki09zdNBoaaN85KxFcVBAJK4x+f0EM34wVByNGx7vvMgZbYSjrByyc0ZMZVZBUd2BhzZCUAzsQPyvp2ke6ynSRLvZazheN2ic/ByS4EKMnUNgESNG+dbJb8eFr4wQvdsKz/vfvoAm3TVEQEDQHvbg/hgOqFcc0pkAwAgRSX0nQ8ZoOnyD/QGmqPsLcMZcr2696V2wQ0tYWWKVyeFHKp2S6yacuLGOdXcTWjuGs06G3XRNGW5oSVUkLFHEHc5964LR67aSssnVIjhsXD/y5ku2qJc5t+djLmKNi2fUMHGXiHc5eyb8YiNfWJZsAvStQSAczCE4ziJ6C7iBIBK/eS2T10HAGyRfGdSfo/2+n5h7MAW5PTXshILtf8C9VxnfZdD9BOhcP0nr2S2/G4NlXx3EJlxQKtbiXLR/wFcZrJ69e69Ln/iqqhhHLjPWXUVJI6cd4Zx1+MAQgghhBDyENwgr4GxLLtnz543J08xmowrVi5/6aUJfD7f9WUI1ZdEwhJCLLbyT9kCxaYEmmX4O3bfegyoxUI4ivWS3HExbXv8CX37AEZAAc0Dh4myEiCEIzUdr2twjHhfgoAVmKbO1AyNssv4nE+wcUQPS80TU0jSAe8LrH3Is5rxPRxlCT57SgEA2FK+1g7idqVjWttlNADFySRspb9sEJuDcITp1NkYJgIAkp4i0nFs7+c1L8TaJTTQQsbfq9IEJVa494SA9S57b3rhgOZ2Lz5QFOcTaH6sjY1XS07cyVt5OGzXJ4r7qxximvMKMo6PyxsdCMbL8niDq6ZZ4Z7jAoZnnjonb2w7m4+Qo3lsUDNTy2p2i3SDq7b++ldkpy1vzs2Z2MmqFAJFcQofRlLd99hF/hukt7CiPw+LbDzzW29rRrZwSGhOEWx8bXreKD8oPeO9vwwAiLaQx9G2gQP1kRKOFjJRrc0ht5f56tpPbvvUdQAAHG9/vNQsNrw8qaQLSPac4XFAHT8mK/PVTX+5LIoT7TxevstbAya/egTk3g4pDyiaDW5RNnde7rM+UJygOGQA1/3fjUzW8D2qb++t/cLqXl6148A9z6r7cOR0f+QkTK8J6ec2pc/r4OY8SIQQQggh5AE4c/CeyNPkzZ07b9CgQa9NfLXzo51XrFiJUwjRPSIRs4Tjm6w338Fx9O5lLaNuP8dsozjCeEm4KkvuiML4yht5PSsPAyxv9wmpSaGv9njdJ4CQ078H/NktZ1iL4tUriisfr+kCRqP48Wzh8u5l/RjBN7tlOg4AgCuT/ZLA69NHv+hj/aJKN0m++VF2hqiUs8QOVe9VhnX51Mt4XrkpQ/9WlG7JJ7ol1TRKErcHfdsla2J37TfdtRVftl8LHDjXV11DTiq/9pryVn4O4Zr1yv+2V/6tL+qlS/+n0HIums5kyZXtwRseVU+OLv1gSekHzq9x1F+ftIw7UY8Kh4u2kv8MXtYhc05b3bx3dfMqXXbnxKLa8+8iG25L2Rm86tHMWW1KPvus5LOKaLWK977zKuYAANRn5UkvmNsPyD44AAAAHOKPpkR+nedunHBHP6nyqcsAAKDopOLABP0zrS2msyEHSgEAjJcUB3Rlz8eC+VLwn7m3Gr275HNtxqTvGGU7vi5m/D/VzeEizJNTU56cWunCPO+F35U/09lV/3edyRrUv/fWemE1LVU7DjRgl75LOHLWYeSkLIP6WHy8YGgPy0cXpXXOBPIcgUBACFitNtenIoQQQujBhzMH75XyKYRvvmUw6nEKIbpXCCsVcYQjplr/9W62EoBqZg4Swj93RpxZRjlYYKy0Oln+1ecRs/7lQQ3H67F1FFviNXuu6tMDkvRSysFQRdnSHadEJg5qfIIox9u7S5HLgDXFZ3NyRcWTt3ttxMxt8sQC2sqAw0aVFAuSk6Uns2hnSKazATN+licW0jlavg0AbOLVi1Uf7pdklFEMCw4rVagVnjvrdSSLlG/8Z5QunR857WfFSTVPZyWMgyrMEx9JEthqzknl1+7iHI66eNQ7Xs0zMcRi4F887vfWnPDvbj5YoJamAYAzSZYvjIz72etMLs9oJ3YLnZUuSTPWfeaRG22BVfT14uYv/eB99AZfZyMMQ/SlgqRE+e9nhfY7vim15N+djLnFKvry/cjJPynO5NFmBzEWC//dHTR2dsifN58vwah941b6HlLzTCw4LLz0a6KqT56oYz+p2m1cBQAAnFG+NZ7PcFT8EXmh86YW6Y4TfIalD//jlctWPvOukk/TABwxmKrZkbHoutdfZ8UpBbTJThiGKs4T79kWMmp2yO6im2e46v+uM1mDevdeF9mopqXqxoEG7NJ3B0fOOoycrGjfMVGJQbTrZF3XsSMPUyqVG7/e8NWX6z5asjjurTdfGP3fgQMHdHykQ1hYmFAo9HR0CCGEEGpgROHr7+kYmjhCyODBg1+b+GpenmbFypVpaTiF8KEz7HHr94t1APDmp2F/H6vj5lNNjv8gdfxk46l1MRP2NcGN6mNGpf892v7HBy3eOXfvdz1DTRb3xPTkr/oIlsVFrs/2dCyocWjaI2ftnuqlWzc7GwAmLPLaftDzZamK3+lNHTmpO8hw5X9XJMAA4TiufHYoTYxCSiui1CKSK6LyhFSukMoT0Tk0mD0XMEIIoQdGI/mdjirDmYP3HMdxe/bsmfzmFJ1et3LlCpxCiB4qlNL0dHdTC3+HjM/xRI4WnYo+HGUUcMLzqfTD9v4WIXfRls4tWEe6fK/rpb6oacKREzUCnJxOJDcfUs0BXVEZBACGk5qYyGJH31zbqHTLjKumTzW25yqfgBBCCKEHC/4Wv080Gs28efMHDx48ceKrnTt3WbVqVUpKiqeDQuieE7YsXvqOTlZ5Fh1Hcv/123IfHieK0IOJ8rErzIK///JOx0c4PKxw5GzMcvKph2SygyT4iiT4EUJq/pM2RziOMNZ0/Y1PCoxXLwMAPBSZQQghdJdy8nGaWqODxcH7xzmF8PyF81Pj4pYvX7Zt27YfN222Oxp2KySEGheBXnQo0dZRZQuSsWCn87LF8Yd91/4t1da4dRZCDzu20GvujId9/4GHHI6cjdnpK/wJi5r+EhCpVPrU0/SE8TW+UoZhWIb5cfOWP/74g2VZABy1EEIIoQcYFgfvt3xN/vz5C5xTCLt07rISpxCiJq0s0S9ugZ+no7h/Un5pHvOLp4NACD3gHraREzUGfD6/eWTzmBbRLVu0jG4RExYaSlEUx3GEVJ2vyrIsRVHnz51fs25dYYGbDzRCCCGEUKOGxUEPKJ9CeP5cXNzU2qcQNm/ePD09/f5HiBBCCCGEmjalUtm6TevWrVvHREdHx8QI+Hyz2ZyRkXH+/Plff/31ypUrn3z8sX9AQOVLWJYtKipas2bt2bNnPRU2QgghhBocFgc9Jj9fu2DBgsGDB7/66itdu3RduWpVcnJy5RPatm334YeL58yZe/XqNU8FiRBCCCGEmgalUhkdHRMdHRUTEx0b28rLS84wTE5OTmpq2oEDB5OuJmVnZbPsreXriVeu9PX1pWkaABwMQxHyxx9/bP5xs82Ou+IghBBCTQoWBz2p0hTCuGXLPqs8hVAoEs18ezrN48+fP3/ym2/qynSeDhYhhBBCCD1IxGJxZPPI6OjomOiY6OgolUoFABqNJinp6k9bf0pNTU1NTqml0nft2rW+ffuyHEcAriReWbtmTW4ePkYdIYQQaoKwOOh5+fnaBQsWlk8h7Np11crPrydfnzBhvJ/SlwB4yb1mz5q1aNG7lf+QixBCCCGEUBU0TYeGhUZHR7dp3bp169ZhYWEURRUXF6empsbHH01NTbt6NUmv17t5t+vXk2maLisr++KLL+Pj4+9p5AghhBDyICwONgrOKYTnzp2Ni5u6bPln+/b+M+g/g507QNM8ukOHDs8/P/Lnn/ExBwghhBBC6DY1bh144cKvv/125cqVfE1+/e6ckZHxxx/bfvrpJ5PJ1LAxI4QQQqhRweJgI6LVFixcuHDIU0+PHjOaZTmaLn88HEVR48aNu349+cKFC56NECGEEEIIeVZQUFBMTEx0dHR0dHRMTLRUKrU77Gmp6ckpybv/3p2ckpKTk8Nx3N035HA4Nm7cePf3QQghhFAjh8XBxoXjuODQEJlMRtPU7cdhzpx3Jk9+s7i42FOxIYQQQgih+y8wKDCmvBQYEx0dLZPJGIbJyspOSUk5fvx4cnJyRkaGw+HwdJgIIYQQelBhcbBxadUq9plnhjoXFFdGUUQsFs+dO+edd+bg5oMIIYQQQk1Y5ccKt2jZ0luhYFk2Ozs7NTVt85YtqampaalpVqvV02EihBBCqInA4mAjIhAIZr39NsdxdxYHAYDH48W2ajVmzAubNv14/2NDCCGEEEL3SOVqYEyLFj7e3hXVwJ9//jk1NTUtLd1qsXg6TIQQQgg1TVgcbERGDB8eGBTEsizLMhRF33kCRch///vfK1eunDt3/v6HhxBCCCGEGsSd1UAAcD5W+O9df6empiUlXTEYDJ4OEyGEEEIPBaLw9fd0DOiWoKCg1m1at23dplu3rt4+PizDAIHKhUKO5Uxm4xtvvFlUVOTBOFGdDHvc+v1iHQCcuybWFPE9HQ5CCCH0wAjytXeKNQPAhEVe2w8KPR1O/d1WDYyJ8fHxgZvVwJSU1NTUtKtXk/R6vafDRAghhNDDCIuDjVTvPr39/PxUKlVERESzZhEioYhlWUIIIYTjuLy8vE0//viQbz547eq1wsJCT0fhloriIEIIIYTq54ErDvoHBEQ1j2ze/FY1kGXZ3Nzc1NTU1NS0lJSUtLQ0s9ns6TARQgghhHBZcWM1d86cKkcoqvz5xYSQkJCQd2bPvu9BNS4fL116NP6op6NACCGEEAKapkPDQqMio6KiIyMjo6KimsvlcucfdFNSUn7/fVtqakpaWprJZPJ0pAghhBBCVWFxEKF7bvtBofdBnKKLEEIINR18Pj84JDg6OjrauVo4KkooFDIMk5OTk5qadirhlDpTnZ6eptPhSmGEEEIINXZYHGzUUlJSd+/9x9NRNC7R0VFP/Wewp6NACCGE0MNFKpNFRKiio6NjomOio6PCwsIoijKbzRkZGWq1Ov7o0dTU1NTkFJvd7ulIEUIIIYTqBouDjVpJSUlCwmlPR4EQQggh9NCpeIRIhEqlilCFh4cTQgwGg1qtPn/hwq+//Zaampqdlf2Q7wGNEEIIoSYAi4MIIYQQQuhh59w0UKVSqcJVMTHRLWNbKrwUcPOBwvHxR1NT01JSkktKSjwdKUIIIYRQA8PiIEIIIYQQeujIZLJmzSIjIyOcDxCJaBbB5/HtDnvmjcy0tLTNP27JSE9Pz8iwWCyejhQhhBBC6N7C4iBCCCGEEGri+Hx+aFhoZERks8iIZs2aNWvWzM/PDwAMBkNGRkbi5cQdf/6Znp6epc5iGMbTwSKEEEII3VdYHEQIIYQQQk2NUqlUqVSqCFVMdIxKFa6KiBDw+c6nCasz1Xv3/pOamqZWZ+bn53Mc5+lgEUIIIYQ8CYuDCCGEEELowSYWi0NDQ1URqujo6AiVqllkpLeifMdAtVp9JSlpx59/qjPV6sxMfJowQgghhFAVWBxECCGEEEIPEpqm/f39VaqIikcJh4WFURRlNptzcnLU6qxTCQnqTPWNjIzSsjJPB4sQQggh1NhhcRAhhBBCCDVqMpmsYlagSqWKiooSCoUMwxQUFKjV6vj4o+ostVqtzs7KZlnW08EihBBCCD1gsDiIEEIIIYQaET9///Cw0HCVKkKlCg8PDw9XeXnJAaC4uPjGjcxr167v3rMnMyNTrVbbHbhGGCGEEELobmFxECGEEEIIeQZFUQEBAarw8HBVeHi4ShURHh4WLpFIAKBMV6bOVN/IzIyPj1erszIy0nU6vafjRQghhBBqgrA4iBBCCCGE7gfnXoFBQUGqCJVzgXDz5s1FIhEAGAwGtVqdkZ5x+PARdaZarVYXFxd7Ol6EEEIIoYcCFgcRQgghhFDD4/P4oeGh4WHh4eHhzgXCIaEhfD6f4zhtvladnXXt2vV/9u1TZ2ZmZ+cYjUZPx4sQQggh9JDC4iBCCCGEELpbzmeGqMJVwcFBQYFBFU8Qdj42RKPRXLh08c+//lJnqdPT0i0Wi6fjRQghhBBC5bA4iO4bYae4H755Ub5v3rg5/xRxno4GIYQQQvUjFotDQ0NDQ0PDwsJCw0LDwkLDQkKFIhEAGA2G7NycnOzcw4cP5+TkZOfk5GTl4GNDEEIIIYQaMywOPtikQ9ec/axH6sbJY5cllN5Wb+P3XXLwuxG6r//vmaWXGE+FVwUhhBCKIp6OAyGEEEJuUyqVKpWqYqPAoKCgwMBAQkjFlMCU5JT9+w+oM9UajSY/P5/j8C+ACCGEEEIPEiwOPviIuM3LK1bnjX/1xzSbp2OplfXs56Me+dzTUSCEEEKoBkqlMiQkJDgkOCw0NDQkNDQsNDg4mM/nA0BJaWlOdlZ2Ts7Zs2ezs3NzcrLz8/MdDoenQ0YIIYQQQncLi4NNAMdwXr3fWTU3fezi42X4x3qEEEII1Y4Q4uvnFxIcHOL8T/n/BQuFQgCwWq25Obk5OTnHT5zIycrOycnNzs0xGgyejhohhBBCCN0TWBxsAhwXN63TDH5r3KfvXxo1c1tutYuI+b3e2/fDqJK1I0auvOY8gSiGr0tY2uPE/AEv/VbMAfHtNXH++H6to8JD/LwkfEaXe/XwlrUbLoYOG/PsoG6xYd60MSfxn/8tW7rlcsX6ZSKNfvq1ya8M6R4bIDTnXz+6bcOnXx3JtgMAUXQcNXX8oC5topsFeYuJuTBz7/svLk7979a/44L/eK3/O/E39x4Sq56YMOnVZ3q1DfMi5pKc6/+uX/DB9szGsg4aIYQQagJkMtmtRcGBQUHBQWFhYSKRCADsDntRYZFarT5z9kzeTo2TVqtlWdbTUSOEEEIIofsEi4NNAZOze+5MeeS3Ly1e/vL1l75Oqs8DACll+4FD+7W+2SH4PuGPDH9n4/BKZwgiOv93wRdK43OTtuezACBpN2Xj19MekVMAACAK7zD0rdUdQ+KeXXCkhKMCeowc91TF3eT+AXzrnRMOhLETN2yc082bKm8gMLpjuMyM70YQQgiheqrYH9D5yOCg4KDwsDDno0JsdrsmL0+dqb5w4cLu3XucdUDcIhAhhBBCCGFxsGngDGfXTlvZ/td3Jq+cfmHk0tP6+v07n9Ptnjdyzi6NkfNq+cy8DYufDNGfWb/gk80nUos4/66vLl0/qVO/EY8H/PmThqVbvLhwSkeJ9sjqeZ/8fDzTomj15KxPFj43bMqY74+uTXHeTb/vvdFz/8wuZcSBQeIyO4Tc1hgdOWbhjK4KS/L2jz7Y8PfFPLNQqYpSlBTiWxSEysXGthw+bLjr8xB6oGzbvu3ateuejuKBJ+Dzlb6+QUFBt9UBw8Od64INBoNGo9HkabAOiBBCCCGEXMLiYJNhS940b3GXrZ+OWzL/5P/NPVSvjYE4Rl+g1VkZgJKkbes3/3fQ7OaFSceuakwAkHvsq437Rj8yLLyZigINaTl0SCxPt//DWV8dKuMAQHt52/urew9eNaBnV7/1KYUAAJyjJCe7yGQHsOdm6gDo29qimw8Z2lZov/RJ3KLNGQwAgDU/+Xz+XWYBoabEz9+/d5/eno4CoQYWf+woYHHQbRRF+fv7BQUFBwYFOiuAQUGBwYHBXgovAGBZtri42Fn7O3ny1PZtO3LzcvLy8nQ6vacDRwghhBBCDwwsDjYhTO4f777fs/XKke/PPXJ5ofFu75aXmeuAVv6B3hSYWAAAmyZby5EAiZgA8MObh1GUePCahMFrbr8sJCyYgkLX9+dFxDSj2ayE42rcYRAhhBByMRnQuTmgRqPJSM84fuyEJl+jydNkZWdbLfXZTAQhhBBCCKEKWBxsUrjCgx8s+v3RDc+9N//oZ+bbvwQsgFAkIm7fjLXbHED4fH7FJXa7gwNCKACocWESEYqFbrVBKIoA4PomhNyweu36hITTno4CobvStWuXuCmTPR1FoyAQCIKCggIDAgICy/8nMDAwKCjYy0sOzsmARUWafE1eXv6JEye3bduu0eRp8jSlZWWeDhwhhBBCCDVNWBxsYrjSoyvm/9T1+9GzpuVLCOhuHmf1ZQaOCm0ZrSAXihqgIGfPuZHDsoodrw5aeMh055fpOw9VewdK1bVHOH35Rn0mD4pEIoqi8HGKCCGEGidnETAgIMBZAgwMLP/Ix9vbeYJer9dqtfn5+VeuXDlw4IBGk6/RaPI1+XaH3bORI4QQQgihhwoWB5scTn981eItvb8e14wmt2puTPrlqzquee9Jc19I+2zbpQIrX+bvI3Z/GmFVzPW9+9JfnzT0vU9vUOt3nU4tMDB8RXBUh2DD0dOZDvfusOeftNff6DB1zWLTkq93XczSMWL/5tGKwkvXi9yq902fNm36tGk2u91mtdpsNsNNNqvNZrfrDXqD3mAwGgwGg81ms1ntBqPeYDAY9AbnyfV+6QghhFBlfB7f1698ObBSqfT1VTo/DggIoCgKAGx2e3FR1RXBeRqNEX8ZIYQQQgihRgCLg00Qp09Y8eG2x78cGVbpoDF+06arT7zV5sklW59ccuuwrb6NOBI3fvht/y8mDpzxzcAZFUft5z8d+ML/Mt0q7jmubFyyoe/6yW2HffDDsA/KQzf+Fdc37h+3tk/6aevW9LR0sUQsFUvEErFEIpFKpWKxWCKRePsoIiJUUqlUIpGKxSI+n1/lWrvdbjZbTCaj0WQ0m0wmk9nJYDCYTCazyWwym00mk8lkMhqN5psfW3BfJ4QQeojJ5XI/Pz9//4CgwAD/gAB/f7+AgMCAAH8fHx/nCXq9Xlug1Wq0NzJuJJxKyNPkawvytflak6maOfYIIYQQQgg1Eli48Y7CAAAU+UlEQVQcbJK4svjPP/q779qnKh2zJq5+7XXdjClj+rdTefM5m6m0SKNOSzqSaq7fKmNOf3rp2BeuvPzK6IFdW4f7SmlLSW7ahTNZ7pcbOcPZ5ePHXnvltfFPdWsV4i1wlGkyEtN0fAIWd0K6cePG8ePH3WxLIBDIZDKZXCaTyQR8gUAglMmlMplMJpXJ5DKhQCAQCJVKpUqlkslkAoFAIBB4e3s7Z3xUZrPbDXp9+WzE8umKRr1Bb7fZrFabc6KiQW80GPU2q81mtxn0Bp1O53C4NZkSIYSQx9E0rfT19ff3CwoI9A/w9/Pz9w/wD/QPCAgMEIlEznNKy8oKtNqCgoKrSUlHDms1+fn5WiwCIoQQQgihBxVR+Pp7OgZUjV27dgJAQsLp1WvXezqWxqViS/uPly49Gn/0nrZVUVIU8AUCoUAmlTtLigK+QCAQyOQyuUwuk0kr6okymczLy4vHq1pzv3Pts8FgtNmsVputprXPBr3eZsc9px52vfv0njtnDuADSVCTcD9Hb3dUfjRw5bXA/v7+NE0DgMPh0Ol0xcXFmjyNJl+Tl6cpLi4pLi7Kyckxm80u748QQgghhNCDAmcOIlQjm81WXFxcXFxcp6sENzknKjpLijfLi7KKkqJA4FNefJTJ5F5yPq/q2meoNFGx0lxFY8V2ijcnJxqdJUWb1Waz2UpLS/EhLQgh5CQSiQL8/X39fH19/QIC/H19fZW+voH+Ab5+vjKZzHmO0WAoKCjUFuRnZWWfO3eusKCwoKBAk5+PwylCCCGEEHpIYHEQoQbmrOIBQD2qihVrn2VSuUDILz9Sae2zTCYNCgp0lhQFAoFUKiWk6nNlalr77HwYi81uu3Ptc1lZGcPU55nRCCHkcRKJxM/fz9/PT+nrG+Dv7+frp/RVBvgH+Pr5SqVS5zk2m62goKCoqKiwsCgjPaOoqKigoCBfm1+gLcC1wAghhBBC6CGHxUGEGot6T1Ssdu2zTCoTCgV8gcA5UTEoKLBi7bNCoXAumrutdbvdoNdXqidWs/bZoDfabNab5UWDXqe3O3DtM0LofhAIBEqlMigoSOmrVPoog4ODlEql80jFHEDnQ4GLi4uLi4oTTicUFRc7FwUXFxfjNECEEEIIIYRqgsVBhB5sd1NSFAgEAqGgYu3zndsp3rb2WS6/87nPcPva5/LVzXZ7TWufnbMXjUYjx9XvQThNh1AksuLzrxG6XU0VwOCgIGl1FcALFy5UrgCWlJTg2IIQQgghhFBdYXEQoYeRs6RY16vu3E6xytpnuUwu4POrrH2umNRzWwA1b6dYZe1zU91OccXyZVlZ2Tt37kxMTPR0LAjdVyKRSBWhUvooq1QAQ0JCJBKJ8xxnBVCj0RQXl6jV6gOVKoD1GLsQQgghhBBCtcDiIELIXQ2ynaKALxAIhDWtfa549LO3tzdFUVUDqG7ts96gt9tsVqut+rXPer29UT732cdHGRER0adP75zs7B1//nXo0CHc+Aw9JKZPm+b8wGAwFBUWaQsLiooKU1JStNqC4uKiwsIirVZrwXm1CCGEEEII3S9YHEQI3XMNu51ilbXPlbdT9PLy4vGqDms2u91mtVaqJ1Zd+1w+e9Fqr1j7bNDrbfe4pCgSi5wPkwkNDZs06fWJE189eODgzl270tPT72m7CHnclq1b/z10RFugtVqtno4FIYQQQgghhMVBhFBjVe+SYpW1zzK59GZ5sfq1zzK5XOBqO8Uqa5/v3E6xTmufKYq61SIBilAURQ0Y8Pjg/wzOyMjYuXPXwYMHnZM0UZNAR4xYsmZSy5MLRn2U4PB0MJ6XeeNGVnaWp6NACCGEEEIIlcPiIEKoSanf2mcBny8WSyQSiVQmlUgkYrFYIpZIJGKxRCKTSZ1fkojFSqVSpQqXSMq/KBQKq9yHZVmTyWQ0msxms8lkNJvNZpPZYDSYzSaTyWwym80ms9FkZBwO57TBynh8PgA0i4iYMuXNia++snffvpzc3LtLxoNC2Cnuh29elO+bN27OP0VN8nES8vDWrcPlF+74pt9HTT/JCCGEEEIIofrB4iBCCIHNbrfZy8p0ZXW9sKa1z5W3U1QqlTKZqmLts0KhoGm6phsSigIAkVg85OkhFFVeS7qzkuiSdOias5/1SN04eeyyhNLbSkH8vksOfjdC9/X/PbP0ElPX294jhBBCKl6uu0Tdp25aOCQyQCmXCmjGoi/Vqq8nJhzd9/u2Q9fK3HxpdJuX1iwfK/tr8kvrrt9dNiiv2EGjJ4wY0LNds0CFwFGWn3H94vF923747UR241g7W78kI4QQQgghhJo8LA4ihFD91W/tc8uWLVesWF77ORzHElI+RIeFhp2ChDoHR8RtXl6xOm/8qz+mNe4lytazn4965PM6X0b7R7eLDimfvUlLvAOaeQc0a9/n6ZcmXfphwayP9ue4sYKX8o5oHRNcIri7khnxav/qspWz+wbxbt5HoAxr0yOsdUfp9b9PNo7iYD2TjBBCCCGEEGrysDiIEEL3W03TtziO41iWoumiwqLTZ07rDcbnRz4HAPXdoI1jOK/e76yamz528fGyJrqS1JG0bvTIdVctIJD6BEW37/n0mAnjenWYsHIDeW304hP6+/GqqeARS9fN6efDFZ7f9MVXWw+eTyuwCXzCWj3a5z+xmmMeSjwllPt6ixz6khIT7nKIEEIIIYQQqg0WBxFC6H6TSCSVP3UwDh7NczDM1aQrZ86cSzh1Sp2VBQC9+/S+u3YcFzet0wx+a9yn718aNXNbbrXLZvm93tv3w6iStSNGrrzmPIEohq9LWNrjxPwBL/1WzAHx7TVx/vh+raPCQ/y8JHxGl3v18Ja1Gy6GDhvz7KBusWHetDEn8Z//LVu65XLF+mUijX76tcmvDOkeGyA0518/um3Dp18dybYDAFF0HDV1/KAubaKbBXmLibkwc+/7Ly5O/e/Wv+OC/3it/zvxNx8SLVY9MWHSq8/0ahvmRcwlOdf/Xb/gg+2Z1bwE1m61MRwHVkNh5oWDmRcO/X1g1rffvtxy7Jxxv4xYf5UB4vvY3BVTn2wZFugl5MyFaWf2frNi7bbrxltFO7pF3I5Lcc67abeOe/yD43Y3rrpJ0nPS248pofDgvNHTf1GXV+Ks2rSE3WkJu2v83tScotqbrvLtAEuJ+sL+n5av2nq+pDw0Stl54qJ5rz/RwodPOM6uV+9bPOGd33NJzBuVk+z6PgAAorD+415/5dne7VW+YrCUFeSkJyfu+275N1WWqiOEEEIIIYQeZFgcRAih+00sFgMAw7I0RRUVF586ceL06TMXL12yWht4ASqTs3vuTHnkty8tXv7y9Ze+TrLU4x6Usv3Aof1a3/xtwfcJf2T4OxuHVzpDENH5vwu+UBqfm7Q9nwUASbspG7+e9oicAgAAUXiHoW+t7hgS9+yCIyUcFdBj5LinKu4m9w/gWw13tCmMnbhh45xu3lR5A4HRHcNlZtePgQYA4MpOrvnk18HfvBjz5NOxG65eYcDhHdWpRZgAAABkga0ee/GztgH2Z9/+q7DWApe7V4l6DH0igLKd3/jZ72q35+jVlqLam67y7QCpX1Sv/5vfsYV4xLhvkx0AVNDzS9fM7udFHMaifAOR+XoH8K1lLECVbS5d3QcARLETv/pmTlefm/Ncpb5hLXzDmgvOffttQmlj2a4SIYQQQgghdNcoTweAEEIPHQFfcPny5e+/++6NNya/OO7Fdeu/SDh9usErgwAAwBnOrp228izbcfLK6V3k9d5Zj9PtnjuoQ/v20e16Pz3/72yGY0tPr50yssejHVt0Gjh2/VkdePcb8XgABQB0ixcXTuko0R5Z/fJTvWLbPNpt5ILf0pmwYVPGRN+sT3H6fe8O6fxIx+j2Pfo8//kpe5XG6MgxC2d0VViSty8Y92Sn9h1bdXn8Py9+srf2Wl5l5ouHT5WxVGirGCkAcPpjn704rEeXR6NbtW/VfcjL31yy+PZ/7jGfW8lgklc/2z6yZZvIlm2i+nxw3A5uXVUebHjrFjKKyfj3aI7b9TIXKXLd9M1vR1Sbbr3HLN2nYaUdxozpxAcAIu82qJucTdwwvEePzn0ff/TRrt2HLztqqiGQmu8DQEeNXTSzq7ctfee74/7TsV376Hbdei86YsIZgwghhBBCCDU5WBxECKH77eChQ3PmzP3jj21qtfret2ZL3jRv8UF99Lgl8++sbbmJY/QFWp2VYWwlSdvWb77CEF5h0rGrGoPdbsw99tXGfWUcHd5MRQHQLYcOieXp9n8466tDaaVWh0V7edv7qw8Z6JieXf3Kf+VwjpKc7CKTnbHqcjPzq67UpZsPGdpWaL+0Om7R5gR1idVu0eUnn08ucG/iIAAAOIqLyzhCSWQSCgAI8W73fx99+/uxU6cvHfrhvcEhNPACg/1c/P5z8yoilUsJsCXFpW7H5zJFLpu++e1gHYacM1s+2pTooH1jY/0pAOA4DoD4x3aN9RMRAM5akJFd4xrgWu5DN3/q6TYC5tqX0xf8kJBVZmMYm6GgyIC1QYQQQgghhJoeXFbcqPn4+HTt2sXTUTQu0dFRng4BoQcNk/vHu+/3bL1y5Ptzj1xeaLzbu+Vl5jqglX+gNwUmFgDApsnWciRAIiYA/PDmYRQlHrwmYfCa2y8LCQumoND1/XkRMc1oNivhuLreS1d5SqWCcKzJaOKIV98F338zOoJfXhYVqsIBgKGoWn/9uX8VZzKaASiFt4ICrXsBC2pNETH3rlvATG7aDSPXRiaTEgBWf3zbgcL+T/eb98P+mSWZiRfOHvnzx2/3pFSzV2Kt97n5XTh+OLXqxE6EEEIIIYRQE4PFwUYtJiY6Jiba01EghB54XOHBDxb9/uiG596bf/Qz8+1fAhZAKBK5P6eQtdscQPh8fsUldruDA0IqZq5VhwjFQrfaIBRFAGq6jTvEHR7rpqDYzGspRlA+O2G4ii45tXbhp5tPphWYeX4D5m9f9YyLEJRPuHsVk5OSYeZaRnbv4r8+RePW7MFaU0S533TF/Ww2G0eIc2tArnDXvHGG888/2b1Dp0fadeof+ehj/WOpEVN2lbiOq/J9KD6PAnA4cG9BhBBCCCGEmjwsDiKE0MOAKz26Yv5PXb8fPWtavoSA7uZxVl9m4KjQltEKcqGoARaN2nNu5LCsYsergxYeqmarO/rOQ9XegVJ17RFOX75R99oUUXSfMvv5UMqRvHfXVYaKDgoSgGnfpjX7r9kAAOxFBbpKmztyDoeDA4lEclvdkvKr/arKTCcOnNQNHtB9YtygfQv21Lb4maZ5t15gTSmiW052u+kaWLKObFpxZBMALY99bvG37w18bHBXya69dboH2DW5hSyl6tw1hErMqsOKboQQQgghhNADB4uDjdTHS5d6OoTG7trVa54OAaEHCqc/vmrxlt5fj2tGk1s1Nyb98lUd17z3pLkvpH227VKBlS/z9xHX+8klwFzfuy/99UlD3/v0BrV+1+nUAgPDVwRHdQg2HD2d6dbTfJnre/5Je/2NDlPXLDYt+XrXxSwdI/ZvHq0ovHS9qJoqFcXj0wQYSiD1CY5u12PI2JfG9goT2jM2Lf3hKgNQpNXaoUW3EWMevf7rxTwDy5PJRDyAm+U2Tqsp4Og2A58fuCV5n9rh1axtsPn8lTwXV1XGlez54pvxvWe0e2bVVuXGtRv/iL+SWWylpL7NWnd9vCf/yOptVxmw2R0c8e70WPew88ezTbWliKlD09WhIx8fFll48sy1PD3D5zn0eisAIVDnb6jj6r6DeRNefGTq8lkF739/KKWUH9x+8MBWgrreByGEEEIIIdToYXGwkToaf9TTISCEmhpOn7Diw22PfzkyrNJBY/ymTVefeKvNk0u2Prnk1mFbfRtxJG788Nv+X0wcOOObgTMqjtrPfzrwhf9lujUFzXFl45INfddPbjvsgx+GfVAeuvGvuL5x/1juOJnXesrv16dUPsIxpZf/t2Dmh8d1HAAUHfzlwJQ+Tz++aMvji26dwyTf/EB95GBSXLv2I5YdHOEM9MJHT437Oqv2q25nv/ZF3DsB6z8c06rP5KV9Jt/+Uqgdf15NZ7KvXivlYmNf/GJvwNtdpu6pLUUuAnaB+HZ75f2FPfmVDrHFu/85XfeNJi2nv1r254Blwzq8uPqPFyu/pDrfCSGEEEIIIdS44dOKEULo4cGVxX/+0d/a22p01sTVr73+4e+nM4otDMs4LPrCnJRz/+4+kmqu3ypjTn966dgXpq3feTJFq7MwjN1YmHnpyJks98uNnOHs8vFj49b/feZGkdHG2E3FWUln03T8KtPfmILUy2l5RXqLneFYu1lXmJV4fPf3n814ZvCY9/fllBexuOLdCybO3HgoMVdnZRiH1ViizU6+eOpkapnz1TEp/4ub9d2hlEITwzhMRennUwsIcXlVFUzu/kX/N2L8kk17zmVodRaGYcy6/LSL8b99/dOxEhYATEdWzVi3P1Gjz8nOs9Weojo2XQUheecOX8osNjtYljGXqC/t/+qdV2btLKjHt5It2Dd7zBuf/nE6vcjicFiK0hN27E8yccByuMoYIYQQQv/f3v27RB3HcRz/nsjdIhzHcdLUIN3cwenlYJNQ6NAYTtJahFODDS4u3nyIgrg41OIW7Q1OnoODOJiUm4JEdJsOZascQqnI57z34/EXvOYnnx9AX8kVy5XUGwC4wsTTiffz81mWtZZX2u2d1HMILld5uba1OLq9MPlq8+cNamOjMTb39k2WZUvNptPxAADQO1wrBgC6DQzXpx5nh/tHxz86Z4OlkfqLd6+f5P982937rzOMAADAfSEOAgDdCrWZZmt66PJd7ovfx59XP369/hfSAABADxMHAYAuufyvgy/tkVr14YNiITvvnHzf2/q0sfxh+9STgwAA0F/EQQCgy0WnvT43u556BgAAcOf8VgwAAAAAQYmDAAAAABCUOAgAAAAAQYmDAAAAABCUOAgAAAAAQYmDAAAAABCUOAgAAAAAQYmDAAAAABCUOAgAAAAAQYmDAAAAABCUOAgAAAAAQYmDAAAAABDUYOoBAPzD1PNn442x1CvgVkqlUuoJAADAFcRBgF5XrT5KPQEAAID+5FoxAAAAAASVK5YrqTcAAAAAAAk4OQgAAAAAQYmDAAAAABCUOAgAAAAAQf0FWnzkRUCz9CMAAAAASUVORK5CYII= ) ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABlkAAAFECAIAAAAr489KAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzddXQUVxsG8HdmLe5GiEICwRMowZ3gwV1bvJQPKJRSrKUUCqXQUqxAgRanuBeH4AR3TYgSIa5rM/P9ESSySTYCm7DP77TnwGZ25p07ws6Te+8y5ta2BAAAAAAAAAAAoAdYXRcAAAAAAAAAAADwkSALAwAAAAAAAAAAfYEsDAAAAAAAAAAA9AWyMAAAAAAAAAAA0BfIwgAAAAAAAAAAQF8gCwMAAAAAAAAAAH2BLAwAAAAAAAAAAPQFsjAAAAAAAAAAANAXyMIAAAAAAAAAAEBfIAsDAAAAAAAAAAB9gSwMAAAAAKDUiSr2WnFk3biGNiJdVwIfH2Navd9v+zdPqCHTdSUAAKABsjAAAIAPjrHrMG/X8eu3t492ef8vr0G9SYfvvXz16MjMxiYfqQ5po293nQi8c+qH+pIytK08Sxa1ZXTQkrpV3OOo8TwsoQ+xTu3ld+gN3NtPWb7n8t1n0dFRcRHPHp39qa0xkbT2/3bfigh/ePqH5ubMB69N5Dp4+S+9G/uPHFbf4sNvDXIoA/cEg6rdxwxs0m7Gyq/rGuioBAAAyB+yMACAskHa8o/H0UlxsZr/iwmc7/vhwwudklXw7T9l0ZajFx8+DY6JfhUd8vj+xcM7/pgxom0V8/L/jxVrXKlBUx9PexNR9mdihmVZhmFEIuZjPSmLHWo19q5SwUyq9QbNOq16HhubFHN9QUPph9pW3iXztIzIvfdvu47fuL2yu5GmNXz0lpQ0mHs3JjYx8r+JVcQFLCbzmXk1KiYp6vjXHqXaM6joxzGL5vOwQDLHBgOm/rrt2OXHz17GRkdEvbh34+S/a34c3bmGpai46yxNmg69uNKwTcc2zh7QrHpFCwOxSGxgYW8pylQSEcOKWJZhRSL2gxfL2veeO72lhfBq5/QZx+IFTYtIKw/ccu9VYlzUvXkNNNzfxXZNRi3aFXD7ZXh45JNrJzfM7lfDtEhlf9o31bJ2T8gj88aSycsfqmQ1xi0a41XQbQIAAHQBd2YAANA1xtxn5K9rZ3XzNM720GJi7exl7ezl22HAoLqD6n91MkN39X0o8hu/dar5m66rKBBj5z+kgw1LRM69Brf6+erx1I+y2bwtw9rWatXcx139SmOk9PFbUmTnYMMSI/P5cmLbf8YfS9YYdTB2PSYPqyJhiLOxt2HpBfcxKywFjEW9UYtXz/L3NMp2YVo4ePg4ePi06jukwQifEXuTdVceEWk89Ix5x2nftbVmMh9tnfrN74fvvkpjzVwqGkSriOjuHz28//gohRn4fvVteytKPvHzzycTNZ0ejI3fom2LOjuKGdJ0YkjcB6/b93snR0lW2xu61+/6v8/a+tX9osd3pzUnaznX/unfVMvaPUEDxZ0/ftzeb8ewuuO+6bZ11J64wo8bAAB8NOX/t0IAAJ8U5alJ1Sxt7Cxy/WfvOzNQpevaPgzG1HfqjgPzu3sa8UkP9y0c17NhTU+HCk4VPL0bdh46cdGOyzf3/HuhfD+zlV+iSr2Htch6mGZtOw/xt8NIryystYOdhOGir1xJ6jx+sLvmPl+Sml/8r8mLizfkAmtlW+5mjGLMGn63Y//8rp6GfNLDfQu/6tWoVpUKjk5OVT9r1vurH/8+fevQwbMpui5SI0ldvxaWLB+7+8cZWwLDkhRqdWZC8ItXH/Umwlh3HdvfXax+uXXZniheU5EeI1YtH1o5v459khrj//y1k6NY+fzfyZ1ruLu5Nxyw4Fwsb+g1atXivg6FfXrHTbXMSAtYufq6nLHq8OXA0u0aCgAAJYUsDAAAdIgxazF7zeS6ZowqZN//2viNWbjz4pPoZLlKmZn46sm1YxsXTejU8fvzcl2XqaekdQcPrisVkk5t2h3GMSYtP++TT+qjd1h7BzuWhPhLq/++WX3kiPqapgMybTN6kP3p9evuJwqMxMa2fM0YxZi3nr1moo8pKYN3j2/tN2bhvxceRyVlKpVp8WH3z+36feqANhMPaOzupHOMaYUKZixxseGvlLqqga3YbUhbc0b1cPvWm5ruXgZ1JqyY28qK4q/8dzVRQ1Rm5jdxrLchKW4tHjF+0/XI1IzEF6cXjZi4NZJjrdt/PaJWgQPmcVMtS/jQXZvPpZHUe2B/7098ngMAgHIGWRgAQHki8f3xdkxsUuTfvXLMkMJY9N8SExcbd3mmz7ux75KKjQdN/uWvnacv3w4ODX8dHRH++PqFf/9X793HcZlTi9ELdpy8ERwWERPy+PaJTQu/aGCfd+i8yLJO76krdpy88+RFdHTU64gXz2+dP7Z91cKpA32t3z3di6r0WbDl4Jnbj15ER0XFvQp6cuXg+hk9qhU6uY3IY/iMgW5iUj3/a+zXu4O0fHQttHJDrx7fzF+9Zf/5wLvBIeGvY6PjIp8/Or975aT2lQyLtcKCGrO4+07EOAw78Co2Ker01GoiIiJR5YnHIzVPGBd95ce62QrS7tgxplV7fLvy0MW7IeER0S/vXzu0du7QerbaRzImLT7vW0nMRe9fM/f37Y9VJK07eJB3PpOGab8tbZbM3TLvd7zz+rD3zRKza2hWT7Wcy4s8Jhx7HRcbf+fnJrmqZZ3GHo5Iint1ZXqt962l5YWQc0U2ttYs8ckJj/ZuPWfRe0Qnq9y7ylbsOboLd2jriZCEZJ5YG1ur3DGixNZ3yA9/H778NDg8Juzp/bPbl/6vfWVNMx8V4TgWZ180EVUZ8d0AFzEpH60a9fWe4OJkStpeFwZufpP+2HXpztOoqMjokEd3Aw5s+21YHQNtF9BwqkgkUiKS1JpxIerdqZLw7I92UiLGavCOiKS4V1en187RKiW6A+TFVmjXub6MUT86cljTwFjDz6asmuJjqHy6buyYtU80NK5J867trFgh4/z6zU/f9QcWkgPW/ftCTWIPf/9aBRxTfbupanVPyL5dLa67ouxsYSewEHfq8MUMQezWoUtNTE0DAFCG4KYMAPBpYqzbTvv1uxbZsgCJrauXFZPKExExVk1mbNkwxdfy7a9ErN3rdhjr06Zz06/9x+wMUb9diVnd/637Z3YrB8m7ZxCxma2Lma2LV8PWn3EBuwLjs57UWPMarTo2rvT2mcPUwbNhr8m+fk2cu/Rcdk+Rb5Hi6r361ZYyQvr5P9cEpmu3X1pUzpg3+HzyqOz7TjJzx+rNB1Vv2qnVXP++qx4oirjCghqzmPtebFoeO7FLt+U7V/T3kL09dPZVG3Wv2oiISOP0RHm3Y9dlqL8dq36285+LKY9fbL4yYWHzyr2HNfvt5unch0r7bZW4Km1wIecCgmbU9XJo0txTfOmh+t0PGMvGLWpJiAs/d/ZJ1qtaNmYeBpZWRiwJKUnJcce3HZm7blRf1/2rQ7J18JHUGPxF45h9P19KT66WwhNjaW2Z41sTLOpP+Wfj9CY2b+eblznXavN5rdb9+m8fN3DqvtD3A6K1b7Hi7osG4ho9+9aSMkLamT/X3ipmByKtrgtJ1ZHbD85rZf22ZImNaw0bJ/OHKzltFyi5Et8B8jL2beItY7iQSxeD8tZpWG/qki+rSzKuz//y+7MJDTvnfbuken0fU4bUT64E5uh6p35y5Vo8X9Xepd5n9uytSM3bxk21oD3V7rrTfme1OT+FxMsB91UdG7o2buTE3g7RfNgAAOCjQ78wAIBPGheyZXTLWlXdbO2dnGs07TRpZxBHxFbos2TtFF8LVeixeUNaerk4OVRp1OP7wyEqsbP/vIV9K7z5t4G1777knx9aO4hVEaeWjPWrV9XB3sGmYuWqzeecz93XQP1k1/RBnZrV8HSzs3d0qNq455xj4SrGzHfijB4F9ENi7XwbVBaToLp19GS0Vk8IWlb+pqig9UMbeXm42dpXdKzWrN+8k5FqxrLxtz8PdmaLt0KNjVnMfdeEC/qjfcXs88TZfDbpaAxPxCdc3vhvVqajZcFizzGrlvb3kAkJN1ZP6OLj6WLnVLVupzHz9j1O0fJRTOTWd2hLU1Lc2LLtnor4iP3/nEjiWYduQzva5Nor7bdV8qoUR0a4vG8f+z6bYjUN01M/OXvuFUdijzatXbP3CTFt0raREcPHnz99R1WExsyLtbC2YEhQpabJhfTz2w7E1Rs22Cd7FyGjJiMGeTzfvfO2SkhPTROINbeyeF8Ja99ryd8zmtowaQ82Tele19PF3rV28y8Wn4niDKoOWLVhYp13D+Hat1ix90XT7tn7+lYSk6C6dfysFtO0a6bNdWHS7uupLayZ9Hv/jO1Qz9XR0c61Rr32Q76eu/u+WssF8qe6P79ZhXenilWViSc09pAqhTtAHmJP75qGjKB8dPdZnlkepbW+WjCuujg54KdxKx7kEzMauLlXYElQhL2MyLl+dWhwKEckcq3kkt9gZf27qWp3TyAqynWn7c5qd37yMffvx3Ikqe5dXabNEQEAgI8CWRgAQJkibbv0cWLuIXLh23qbFHN9fGro46fh8RkqTpka8+z6w2iOSFpvzLcdbZnMGwsGjVz836PoDKU8IejsqnGj1z5Xs+atBvg7sUREUp8xM/wdRHzCie96Dliw93poopzj1YrUmJCoPN+aJ6Q+PHss8GlkYoaSU8vjX5xZ+b+Zh5J4xqRBs7r5f/oXubg5i4iEuOfPtHvi1rLyt0VlRL8MiU7KUHGqjNdPj/8xfsahRJ4xbNC59bsp4Iu2Qk2NWdx9Lxxj6jtjw4KO9owyaMuXo9c8UhShYMOmo7+qb8xwof+MGTB9W+DLRLlSnhgcuG/xxKUB2n0Hg6TWwCF1ZZR5efu+UJ6IhMTj/x6P4xmzNoN7u+T48KD9tkpelbaUd04FxPMkrt2hXbZndJNmnVqYMXzyxdPX5ERFPfrZsZbWViwJGWnpRKS8vnNPsGufoc3ejbNirNoP6WZ9Z8fuZxwJ6WnpgsCILa3N3511PmOmdbZjuag9E/tN3Hg5OFGuSI++d2jRwMG/31WQYe0x33Z7kzdq32LF3xcNRM6uziIiPu7Fi4RiTwmmxXUhqlitiglLqttbl+68EZ6sVCvTXwfdPL5p760UQbsFSqw07gB5yFzcKoiIfx0akZnrJyL3oT+PryNLu7zg240v8ovzWHMrS5YhPjk+Kdf6haSEJIGItcjZyzDHFnBTzXdPtb7utN1Zbc9PLiIskiNG5uxW6NceAADAR4NbMgCAvpH4+HdyFwvyi5s3P83eU0J++9TFWJ6R1qhbR0ZE4tr+nSqJSR289dftIUVOKoTUB3eDOGKM7OzM8u0cxRgaGzBEfEZ6hlZPbVpWnl9JSedP31QKjLiyl4e4NFaY74a02ffCsPZdflk9oZYhpQYuGDHreJxQlIIlddq2theR6uG2tefz5JZaMWgyuLeHWEg7v+dozJsVpJ/ffTiKY2T1B/Wtmq1TivbbKnlV2pNfOnQilmckPh07vAvDjJv6tzJn+eRTBy+kEpXo6LPmVuZsVsxFRKoHe3Y+su42uP2bScPYij2HtDW4smtfGE9EXFpahkCshaXF2z57tbt0dBeT+tn25UdfZ28H+b31f55OExizlv4tzRkqUtuW6pn85sIUtL0wtZP3uuATX8dxAkl8+n7eyFpTR6dCFyipD3IHYM0sLcUM8YnxuabFZ6z9Z3zdxFh+Y+mMv4PzH+TJSCQShohUytw92QSVSiUQMRKpJL87C26q+dH+usuvhjw7q+35+eZUYC2sLPHgBQBQZmC+MACAMkV5alKdPluKPS5JC4ypZxVHETGGfsuCXi/TsICBrb0FS3IzLy8nEQmJN6890CIJE9vW7Tt6ZJ9WtT2cHO3M2LSY8FecvYhIEItFDJHm/RHkmQqBiDE0MtT4BMK6fnXo0vwGouCV/g1+uKHSsvLM/AYGCenRUckC2VlYvHngKekKS7LvBZN4fL7sj95OYj76wNQvl70bSqX1sfP0tBcRn3T/3sviTatk1nqgfwWRkHx6z8m4d/XLr+46FDFsrGv1Pv3q/TEnMOs5l9F6W9ovWRoyLx/4L2bgFw51u3d0Xrs6lCcyae7f1pLlE0/vO5dCVLKjz5iamzIkZKRn/ZR7sXf3rW9nDu5Z8cC6CF5Upc+QhlzA5CNRPNHbRMnc3NKMJeKIGDOvas5i4hPu3Hyaq2eQkHzrxnN1p7qyqtU9RHSD075tS+dMfldHIRemdgq/LoTXh9btmdxsoGu9iQev+wfs//ffnbsPXYt4H+IUukAJlXK7vSU1kBKRoFQocxQq8/nyuy42fPCa7zVOmP/Om8SLJNLcX1TxJiUTVEpVfk2Am2o+tL/u8h1+m2dntT4/lQoVETFSmbRcfZssAMCnDb+eAAAoj5jif6JmjE0KGXBpIJO9W4xPSUwuNLgQVxq48fShlV/3buVdxdnGRCY1snauWsvNorB/Y7ioyGiOiLVxd9fmd/1aVp4/QaFQCsSIJeJSWiEVf98L3GqdiWvntLRiVMFbJn29J9uUQdoeOyNjYyIS0lLTijVPM2PbsX9HK5ZPOLXrVPa5u5U39hwIUpPIrcfApm/HA2q/rZJWVUTyq/v+i+IYSd2eXSuLiBjzNj3bWrF8/Mn9AWlZ9RT/6DMm5qZihgRFxpsxcHzYgT1XqMHgvp4iktYb0K96xpmdx98k2oI8M1MgxszszTn+ZrtCanLeSdL4lOQ0nog1MTVhi9S2pXAmZ8NFRUQV4cLURLvrQkg4Mc1/6OJDj5J408qth85Ycyjw0fkN37SqINZ2gZIp5XZ7SylXUp7kg7HpNmFIFVHKqaWrruUeOpkTn5yQyAvEmluZ5+prxFhYWRARnxSfmF8ahJtqPrS/7vKXe2e1Pj+lMglpSEcBAECX0C8MAKA8EVRqtUDEyAwMGSpe7wghIz2diPi4LQOrTzqTb/cEJmsxprCnAyLGptecHzs5irmoc7/PXrj1wtNXSXLGyM53yra942sU+E4+8vadGK6+s8SnVVOzLQcKGzenZeXaK/kKi7/v+TOuP23ZRB8jkj9cPe6HMznma9L22KWnpRERa2ZpLiIq8vhWtmKPAS1NGGKse2992VvTEvZdB7T54eyhZIFI0Hpb2i+Zz/uLuLzi6p6DocO+rFS7Zy+vVb9Et+3TxoLlog7vvvDmu/VKcPQZU3MzIhIy5Yo3VfHRR3Zd+GFZv/71Npwf1NMl+eSPJ9+mFYI8Uy4QY2xhlhVsvGkHxtTcLM+VxZqZm7BZi/BFatvSvTT4yNt3ojlfFy0vzLyKcF0ow07+OuTkMoe6HQYM+Xx4n8bO1brM3OJp1qv991cztFugBEr9lkJERHxKYqJaIJmltSX7/ss+TVt2b2vBskyH5fcSlud+i8vYQ6/Hqp//0bnRT7fVJA8JieLJXebiXlFE2afnF7u4u4iJ1KHBYfn9hkKvbqpFOTG1v+6KRpvzk806FfikhER8iyQAQJmBfmEAAOXJmw4BbEVXx+LOniOkvHgRwxFrUfezKgX8QkRIef48hiPWrFZt94I3JanRqL4pI8hPzR/z8/5bIfHpSo5TpMaERqUW+qSiunnocARHrEXHMUOqSgpbWsvKtVfyFZZg3zVjzJrNXDq+uowybv067peracUqWEh9+iSCI8a0XsOahbZqHqLKPfv7ygrsUsJa+vXrnDXNtPbbKllVgkKpEIgYmazg0rJR3vh35xM1ib369fet1HVAG1NGHbJ36+W3fXJKcPRZEzMzlojkGfK3x1mIP7YnIN2t11c/juxqF//fnoDUd0tnZsiJGNbUwjR7i7Fm3nWr5touY+bzmaeYBPnTh0Fckdq2lC8N1a3DR95cmIOrFP1gFf26UETfOvD71z3qNx29NUhFsipDhjU3LNoCxVLqt5QsirCQKI5YW1en9zUyIrGY1fLUVT26fjtVIHG1Rg1yzJEvrtqogQ1L6rBbN2LyzVT056ZatHuC9tddcRR8foqcXCqKSFCEh2j31Z4AAPAxIAsDAChP+MhHjxJ5ElX2a1e5uE8uqtsnzkRzJK46cELHPN9Pn22xu6fOxHAkqTFwdPOCJhQmyvr1vMCp+SIHQMrAP5ddSOYZw/rfrJnd3LaQf5S0rFx7pbFC7fZdEEgQiBipQb6TXhMRmTT+9rcvPCSUEbh48vKHeTtVaH3sjhwLUZO40sCp/V2LeJ6Ia/TqXVvKcK829XS2s7DJ/Z91k59vKwXGuPmArhXZom2rJFURHxcbzxOx7lULiWaz4Z7s2H5TLohc+85aMaKRAake7vz39vtGLf7RZ0xMjBgiQSF/l4WRkHBqz6lku26D/Myi/9t98X2Xpax+Ydl6WKruHjn2Uk3iKgO+ap/je+sMao4Y29qEEVICDgUkCVSkttVyX7Q8D0kZuHr5hRSBMaz/zeqZzawLHjimcZ3FuifIQ46sPRDMEWNkZ6/xplPoAkVU6rcUIiJSP7/zIFNgpNXrvA8ShcQdfR3zXlBO3f+J4YkLW+1va+NY/6fbWXNVpZ8/dDKBZ4yaDx/0fg2MWfMR/TzFpA46fPh+vnNa6dFNtYj3BO2vu+LTfH6y9rVq2YlI9ehO1rcBAwBAmYAsDACgXFFe3X0ggmMkdSb8uWSwr4u5lGXEBhYObhWKMK+P/OKq3y8m8yLHPiv2/jW+U10XSwMxw0pNHao06Dqkg9fbZ+7MC2tXXk8XRK6fr96+YICvm4VMYmDp7tv9m6WTWmTvbqB6fPNepsAYtpmyYGQzDxtDMUOMWGZuba7NL+v5sC3ffHc0iiOjOl9uC9j786h2dVwsZSKGFcvM7CtVdc45nEXLyrVX0hVqve9CSnwiL5C4Wo/hflWt8plAWVpnws+fe0hI/mDltNWPNQ4v0rJg5c3VC47E8qxVu1/2bZ/e4zMXcynLiGRmDu4VLQo+KlKfvr08xcSF7t95UdOsRtzz3VuvyQVG1qBv90qiom2r+FUR8bG3boarSew+aPq4JhWNxazUtGLtDv6+DgV9kOFD9245lyqw1vUbeUlIfnXzzufZu30U++gzRsZGDAnqzMzsAxeTz+05lcgT9+rogavy9y8LCnkmETHGpsbMu3b45WgsL3Lss3z7b0MaullIpUb2NTt/s3nL1z4GJL+3dtGBN99zp32LabkvWp2HWU23Zeq0I1EcGft8tT1g97wRbWs5W8hEDCsxsnKp0azP+AVb14z0EuWzTm2vC5Mmo2eM6Vyvko2xhGFEMgtX375j/d1FxCeGhiYJ2ixQUqV+SyEiovTAS3cUgsipSdPKxeq/KySfXPbXfTkZ1Ju67o/BdSuYGFpUajn5r6VDnER8/Knf198tcIixvtxUi3pP0P6605pW5ydj2bhFLQmpQy9fiUC3MACAsgPzhQEAlCnStksfJy7V8APV3Z+btlv6lJNfWjJjc6v1wzxqDlt6eFjOJQvoK5AD93LDl2Pdt63+0rtq7zn/9J6T7UfK6zPPn3gSmvXteM/XjPu61q7l/Sp/Nnb54bG5V/L2D0LsroXLhzb4tr5n78X7ei/OuVTh88VwYdvH9FIvWPPr4JqOTUf+2nTkr3mWKHLl2ivhCrXedyHpwqELyX5+FrVHb7rcfnXnht9fzfM4K3Jt0baKlCEyqDX13KupuVb25gtGtSyYj94z+XMXi39mNHfym7LGb0quvc53hwwa9unhIiL1k927bmk+dnzkgX/Pz2na3qROr95efy58yBVhW8WtiohIdWf9yoDBi9vYtJp15O6srNeE9P/GXLi+M/9QRIg9suHQzHaDHFjiE45t3JPrUbS4R58xNDZmSJDLc/bySDs6rprVuDw1ZL7tF/Y2CyM+evfk4W7WG6c3rvPF7we/+D3bss92fDV86Z1369W+xbTbF23OwzfUoe8uTKfmo5c0H517AdWt67/+/eQFp2md2l0XkmqdRv/vK7dJv+RsMD4+YMmfl+TaLFBypX5LISLio04cuT63adPqnTt7LH/ytBjj7pT3l42d5r1vSfuqA5ceG/j2Pi/In24YP+XfqMJK0o+bapHvCdpfd1rS6vxkbNp0aWrEqIOOHX6g7T/RAADwEaBfGABAOSPEnZjcudvEP/+7GZKQoeYFXiVPiQt7cvP0vo1/rD4Rrt1jFx97emanlt2+XX3g6rPoFCXHq+Vp8aEPLx/cfOBOtgmp1WH7v/RrP3rJnkvPYlIUKmV6XND1I3/+sP5qzphEfuf3nh1HL9gRcD88MVPFcUp5akJ06LN7V08f2v7fw7RCf9WueLFrcvv67cbOWXvg0uOI1ykKtVqZkZLwKuj+lVN71y354Yedz989Q2hZufZKuEJt951/tW38oO82nn0QkZj2/FlwCZ6JtCxYSL7xW9/mfuN+3Xr6Xmh8qoITOGVGYnTwvUtHN69cdfilxhPFuGWfzhVEgvLuzl2P8itRiDu+93SyQOKqPXvWkRRxW8Wq6s1+h24e7T9+1X/3IpIVHKdMfx1y+8SeS2GF9LtJO/fPzudqIi5i9z8n8n75XvGOvszYiGVIkMsztelGwskzlQIxxqYm7/u1CEmBv/Zu2Xnq6kPXg1+nKZWZiZEPz22Z/0WLdpP2huZIp7RvMa32pUjnYd4Lk1fLU+NCH145umXp1M8nbczauqZ1anVdCK8v/rvjxO2XMakKNc+rFSnRL24eXf9Dn3ZD/nqu0mqB0lDqtxQi4iMPbD6dLEhqDBhUz6B4q1AGbfy8XY9ZG08/fJUsV6bHh944tPLLjp2/Oald3yV9uKkW/Z6g/XWnFW3OT9a199CWJqS8u23HnVI7ZwEAoBQw5ta2uq4BAADKFaMem5+v8ReHrPRvMjMQn+4BAHIzaDj30oGx7qkn/tds6NZCe3LBJ8qk1a+XdwxzSjo4qsmoPXGlMawXAABKCfqFAQAAAACUJnngqkXHEsiizfTpbS1LbVp+KFdkdSb8MMCZld/6c/EBBGEAAGUMsjAAAAAAgFLFR+/+fsG5JKZiv4Xz21sjDdM/hp9N+X1CDYni4appq59gqjAAgLIGc+cDAJQAHK0AACAASURBVAAAAJQyLnTLhO9813Z8uPlmqXzpJZQv8qcH1m5v0OXljN9vls4XPQAAQGnCfGEAAAAAAAAAAKAvMEYSAAAAAAAAAAD0BbIwAAAAAAAAAADQF8jCAAAAAAAAAABAXyALAwAAAAAAAAAAfYEsDAAAAAAAAAAA9AWyMAAAAAAAAAAA0BfIwgAAAAAAAAAAQF8gCwMAAAAAAAAAAH2BLAwAAAAAAAAAAPQFsjAAAAAAAAAAANAXyMIAAAAAAAAAAEBfIAsDAAAAAAAAAAB9gSwMAAAAAAAAAAD0BbIwAAAAAAAAAADQF8jCAAAAAAAAAABAXyALAwAAAAAAAAAAfYEsDAAAAAAAAAAA9AWyMAAAAAAAAAAA0BfIwgAAAAAAAAAAQF8gCwMAAAAAAAAAAH2BLAwAAAAAAAAAAPQFsjAAAAAAAAAAANAXyMIAAAAAAAAAAEBfIAsDAAAAAAAAAAB9gSwMAAAAAAAAAAD0BbIwAAAAAAAAAADQF8jCAAAAAAAAAABAXyALAwAAAAAAAAAAfYEsDAAAAAAAAAAA9AWyMAAAAAAAAAAA0BfIwgAAAAAAAAAAQF8gCwMAAAAAAAAAAH2BLAwAAAAAAAAAAPQFsjAAAAAAAAAAANAXYl0XAAAAAMU3/bvvdF0CAJRv+/bve/Lkqa6rAAAA+HiQhQEAAJRjTZs11XUJAFC+Xbh0kZCFAQCAPsEYSQAAAAAAAAAA0BfoFwYAAFDuBQZeX7Zila6rAIDyxNe3/oTx43RdBQAAgA6gXxgAAAAAAAAAAOgLZGEAAAAAAAAAAKAvkIUBAAAAAAAAAIC+QBYGAAAAAAAAAAD6AlkYAAAAAAAAAADoC2RhAAAAAAAAAACgL5CFAQAAAAAAAACAvkAWBgAAAAAAAAAA+gJZGAAAAAAAAAAA6AtkYQAAAAAAAAAAoC+QhQEAAAAAAAAAgL5AFgYAAAAAAAAAAPoCWRgAAACAyLXngoMn9s7wFeu6knKGsWw5e/vhCwvbyvJdwrz5zA3rxtW3Zj5mXayj/9z9Jw792FTyMbeqUeFNVBwGnt1/3LZsgDtOWAAAgKJDFgYAAADakNWd8O+tG0d/aVe8TKOEb//gTJ2rV3e2MGDKZnVlF2PgWL2Wu62ROJ+GY0ybTPxp0GeVrFllKWxN+7OIMXaqVsPZ0qAMHM+cTVRaF4JKbepc02/iT/2cRaVSJQAAgD5BFgYAAKCvRB7j994NvrdzfFWNfWeMG83678WTW+u7m2b9nWEYhmHZ4j7Bl/Dt+TP2W3Qh6MmNTf3sNXyskXjPOHEv+P6mL5zK3GceY//lT57ePfRlZR1lGaIaX6w6dnrTV1U/5PbFnsMm96gYf/SX5YGpQims74OdRR9PKe0C93L7wrWPZA3HjWtjVp6bAwAAQBfQr7qMOnLksK5LKOsWLFx48cJFXVcBAFCesTb2tiwjqzby6y67x+2L5nP8UOQ54Ns+ziKGs7SxElEqR4qbf/T1+aPYGyvh2wuQfuloQIJ/d98ufo67tkTk3AtZ3c6dnFjF9f+OveLzebveYi1cq3tWSJR+yCDFuOnQIdWYR8vWnUoqjSTsA55FH03p7YL6+Za1p75Y2n5kj1WnNobj/AYAANBemfsdKQAAAHwkMht7c0aZlCJqNnpUXYMcP2Is2o0dVkuRlMQzVlaWBaQlrMzU1t7W0ij3b9fye/1DyLh29EQsL/Xp3NklVxcngwZd2lZg5dcOn4rWRVTwMRuhLGLMW/dsa6O8uWt/MKfrWj5FQlLAnqMxYp+eXTwxThIAAKAo9PXDWTmRkJD44sULXVdRtlhaWnp6eui6CgCATwFrZWPN8rFH/zxU95shX/qvH7XrXd8psWf/ce1kV5asSpv0TWMbC5aISOT55Y6jEyrsHd1q2gUVEbFWn436fsaYtlUsJYwgqFLDTs79fNqeV3w+rzM5385YNxk1c1iL6pWdHW3MjCQkTwy7c2r7kqU7bidm60Bk4NRqyJgR3ZrWdrE2JHny68jgZw9O/r1kXWDOXkaZNw6ciOo3tHq3zpXXrXz2PnYxbtS9jQ2Tdnb/qTiBiLFuOf23iR2rOtmbyYTMuKAbx9f9tmLf03RNHZYkTeac3NQ3cUXP3r8/yVofY95jZeDCRldmtvlid0LWWxhjj86jx43o0tDLTpYZ8/TivjWL1gZEqKiAxsn/aORqEC7l1eNz21asuVux+6Bu7Rp4OVmI0iMfnNi4eOG2+0kCETFWjYZ/N7RFTU93ZzszQ0aR+Or5tRM716w/dD+J134XSFRlwoF7E4iIiI/dMaT1T5dVBe0XEbFWdQaMGzvIz6eStSQj6smVq8maxqYSEZGRr18jE/X9s2disu241LHpoJEjujb3drc1FavSEqJDg57dObR67t5nXOE15z4JiYgMXdp+PnZk1yY1ncyYzMTIp+dXzfppf2iuUlib5jO2Lu9nd2/p0DHr72eUpOULOfSFNVHuXSjwtCzsMpHfOXUpaWD3Vm1c1z5F3ggAAKA1ZGFl2osXL5atWKXrKsoWX9/6yMIAAEoFY2FlyQpJsdc2bTg/8OcvhvscnHdTQUTEmLUZPdAr7tDn+552HCkYWFkbM6TMlRixDn0WLv+2hRmjTo+PSWNMrC3sJIpkPt/XKVfHFdaqtp9/i+rvPogY21Ru0n+mdxXDnkM2PFMTEZGB16i1677ztXw7s5KxtVMVa6dK0lsbNgQm5XzuV97cfzho0NgqnTvVWPPsnvrN/pk379LKkhIPHzidFRyoLSrXreIkJSIiE/tqLYf+WtNO1e2bQ3HFGr5nVGv8+r8m+ZhmpRwGznX8/7fM23FCt1kBiUx+jVCAXA0isXT26TFtfY9sS0hdP+s360+r9F5j98fwxFp7d+jR+t3yYhs3786j67RtX3/SkO+PxZSgF1wB+yUQY9Z41qbln3u+mZJe5uLdyYWISKFpTeKqPnWM+Yg7d993yjPwGrXmr2kNrERvjqnY3N69tr171dQTC/Y+K06WI/MatWb9dw0s3mRNUnsPb2eTTD7n0AfGvP6Edb/3c3z614ivNuQMwqjoLV+aTURU8GlZ6GWiuHfroaqXbz1vUwpOKnLrAQAA6CuMkQQAANBTrIWVBSOkJqfEHvtnd2TF3l+0s2GIiEQuPUb6mdzfuuVqWkpyqsBaWlnn+bzAmDZo18CUf7CmR6NGnzVvXa+eb8Meiy9m5Pu6ZkLKf9Pb1aldu3KNBk0HLTwZzRvXGTSobtZE/qLKg7+f4muhDD78w5AO3rVqe9Rq0PT7gIx8YivuyaE995WsW8du3tK3JVq26drUXIg5uvdSatbWUi/9OrR7o/r1PKrVrtawy/B19+TWrXq1LGgEaP5EVYbOHu9tFBuwbHinJl416jXoPWt3MOfUffwgD1HRGkFTg3jUatp55tEITuCTrq8Y37tRPe8qdf0Gr7qZQhYtera2Y98vf/Tb1jVq1Kpcs2HTftP+upEoce0279s2Rdgl7tmybrXdq9Zwr1qjcrOfLqsK2i8icc0R3w31kKbc2Typd+saNX3qtB40ad31OM3JG2Pi5u4g4kKCw97+XOQ5dM6UBpbqsBM/j+5W37t25ep1fXqtvKvWutrcRO6DZk/2NZc/2z9rSMe6tb2r1W/dYegvx3OEm6x5va/WrxruEbpp7JjlgSn5nEBFaPlSbCIibU7Lgi4TIfVlSAwvcavkXOxGBAAA0EPIwgAAAPSUzNzciOXTUtN5xZ1NW25LWw4d4CkiMvAdOtA74/S63SEcZaSmCYyFlUXebEUQBCLG1svXy8aAIRIUr19GJAn5v66RwKW+jk1RcLw6LfLGtp83P1CLrL28bFkiElXq1LmGlHuy+utZmwLDk5Ucp0x7HZ+Wbw8uLuzg3huZrGOXng2NiIiIrdC+V2NjPvTo7uvyN8swjEWt/j9v2HPp2vV7ZzfNae8oIrF9BZvifBgSVfXv4iVOOTV/6tqzQUkKtTz2/r4fl51NE3k29rVhi9QImhqEUyY+2rdq60OOEcc9uvQ4Ok2lSn91ae36k8mCyNnN5X0WxqUlJGSoeV6VGnnn8IJxcw68Jqs23VqaF3c+/IL3S1S1fVs3VnFz6ZRFB+7HZKiUKZF3Dm058UJzhy7W2saK5TPi4zPejcfs2rW6VP1o5fhv/wp4EZfJ8ZwiJT4ps9iz6osqdfGvKVPdWzbh+62BYYkKlTwl5tntZ6/fB0+MVcOJ/6wZXS1i65ejllxKzH9L2rd8aTZRVo2FnZYFXCZEQkJcgsBa21gVtxEBAAD0EcZIAgAAlEU+Pt4PHz5SKpUfbhOm5qasoExPVxLx4fu3HB/z28Ahjf5eZj28q0P4rmmnkgRi0tPSedbVzCxPtCKkXt53Oq5V5xYzNp2akhj64M7NgINbNhx7np7f64XnHdyroJB0oYaJiTFDRGJXTzcRH3753AtVoe8kIiI+5tju01837OzXo82iC4eS2Er+PerL1A/37XuQ1e2IMWs+6591A1wlb/ZF5uJMRBzLFuuzkNS5khPLGrZfHth+ec69cHSqwBS/EbKvKSr0lZqq2dpbsJTBExEpoyNiBcbOyDCfpEtIvnz6lrJ7W5fKFVkq3oC5AveLldi6VWT5iFs3orQagyk1kDKU7RSWulR2Yvnwy+eCtDymhXlzkgReDssnamIt2o4cRnzSme1br8RrPW604JaXlGYTFf20zHmZEAlKpUogqUyaz/IAAACgAbIwAACAsmjy11+bmpnduH7j0qXLgTeup6ellfomTM1MGUGeIReISEgJ+HtvaJdBn38jWLWQ3lmw/Z6SiITMDLnAGJiaSohyxRdC3JEZQ9Ju9+nYsE5dn1p1W7nXa9nKi+05/kh+rycWWo+gVCoFhsmaHoyViFkitVr7OaSE5IBtR6I6DWrWv1OFI7vt+vb0Emdc3rY/5M0c7FZtP+/hIkq8tmL2oq1Xg15nim3azNy/tGu+ayOeSGZgkF/sJOSTazEyQxmTf+MUJQ3jVUo1MRKJ5F0NKpVaIIbJvyObIPAC0ZvaCtmFfFZQ0H4xIpaIGFbLNSrlSoGk0ncpjSDwRMTxBTRC0WpmWJZ5t7saV5d26+RNm5bNW32/fnHG8CmHI7U7nQps+VJtoqKelpTrMiFipFIJQ0rFBwzNAQAAPj3IwgAAAMoijhckEkmDhg0aNmrI8/z9+w8uXLhw7erVxKRSmyHbzMyEERQZb4aoqR7s2H59yIxh/YTE/6buj8jq1aLMlAsCY2xqwlDe6a7k4QGbfwvYTCQy9eo1d8Mcv5btfY2OHE3X/PrxohWnin4Vx7Mun/k6sg/CtezRI7/+774n/b/yHdC7YWLFni5M/MGdR2PfBBesjYODlDJObl5+6omSiEgV/zolz3TmItGbT0Z8anKawFas6mHO3InXkH2oIkMied78wMh2s89qnAgsv8bRbk+KybCWb00pKSNDInkiKmQXSFCr1QIZGRlly20K3i9R9aAInnVr3LLyyvvPCu3bxcfHJfBsFWtrI4aSBSJSRbyM5FnnuvUc2AeRGo9pYc2eiyoyJJJnXXwbOYvuh2iKuQTVi12Tx+yctGn54K7zlkbFDF90PbXYIzKzb7SUmkjL07IgjJWNFcPHxyUU5U0AAAD6DvOFAQAAlEVqlZKIWJZlGEYkEtWpXeurr8Zt3rJ5xYoVAwcOrFixYsk3YWRixFCmXPEmHOBfHdl8JonnXh3YdvbtxEq8PEMusKZmJnk+MIjcW/dqXbuimZRlRBKxOjVVQcQwxOT3elGLUz8+eSaKl/lMXDLVv4a9iVRm6Vq/p1+1gkeCcS/2br2aKfLov3RWOyshbN+Oi6lvf8THx8aqyLBBz0H1HE3EDLESExOD7L8SVKrUAmNRt2VDJyMRERd8/3GKIGs6dvpAH3sjESsyMLW1zDY2kXt6/GQwb+M/Z9GINtUdzKQiVmRg6VSjZX1XcQGNU9RGKBRj7Nt7YCtPa0OxxMy5/rCffxzgxKZfO3khWSh8F0iIjX4tiCr49fFzNxGLDKwqf1bDkQrcL+7pocOPVeLqX634ZVSzylYGIlZkYG5jaaR5x4S0kJAYTuRW6e38Ztzzk6dDeFm9SUu+6VLd1kgsMXWs03VwO8/3XzFaaM05cU+PnQjiJHUmLp87uIGbpYFIJDFxqOpdNft3PQhc3IVFwybvDZVUG/Xr7A62Jf7oW/ChL1oTFX5aFoYxdXOzZ1UhwREl3S8AAAB9gn5hAAAAZZFKlePb9bJGgxGRu5ubs4vzoEEDo15FnTl7tiSbMDI2ZARFRubbvwvJ/01uWnly9kWETLlCYIzNTHK/l7FuMOLH2Y0l2V7iE/47cT3Duo3G14veH0p+fe3ig20Wd68zdNneodleL/BLB/mYQ5uPT2zcw95GkN/YseXu+4FjQvyZnafHN+vc+vttrb9//wbu2ds/RDx+kiR4eQ3987jdN/UnHku/sHnz47b/q9Fx3o6O894v/26F6gfr529o9ecov8nr/N43mer2Ir+BG8PyaZzS7xTGSN06fLuhw7fvt5N0deHiw1md4QrbBS4s4MyjCbVq91x8pmdW9Xd+7jRkXf77FcpzzzbOWdz4r+98289Y135GtkI0dmVSP7t1J31I+zq17dn7r3giUt1f/8vWtsuG+Axbvm8YaXp7YTXn3sLD9fPWNF81rmb3nzZ1/ynrNSH90ITmE05k75PFvz7z87ilbjundJz309X74/ZGaD11mMaNFnDoi9pEhZ2WhZLVqltdwgXduptS9B0BAADQX+gXBgAA8MFJxBITExNTU1MHBwcHBwc3dzcPD49q1by8vb29vb2bNmvatFnTtm3bdujQoUuXzr179+7du7eBoYHmdTEkFomIyMHRYeDAAVmvOTo6isVF/v2WsZGIETIzFAVN35SZmUmMiZlp7g8MDBN169y90IRMNc9zmYlh906tnTZi6uHXlM/rxRiYxr8++e2gLxftvR4cL1er5fHBgQdOPcoQiBcKSjLSLm7bFaQW+KRTmw/mGFspJPw3a9SU9WcfvEpRcJxakZ4YG/Hs7rWrL5KzassIWDp55akH0amREVFKIlI8WDZ6zPw9118myDmeU8tT4yKf3zr/X8CLN2NKhdTrCwcPnLTq8NXnsSlyjlOlx4XeC7gRrsy/cUo6Oi8vIf3u0b0Xnr/OUKvlyRF3j6/934Dxfz9/GwMVtgvc840Tpv599nlcBsepM+KDb794zTAF7BcRUebjv0b1+2Lx7otPY1IUHKeWp8aFPwo8tScgWMOAwPTA01fTxLVatbJ7c/4IyZfmDhkxZ/vlZ7FpSrUiOfL+sT0BL7OPbiys5twNkHZzybDBE1YdvRESn67kVBkJ4Y9uBqVI8nTDkj/eMGvR5XSLFpNn+9uX8ONvaTZRYadlIQzqtG1iKQQFnNE4RBQAAADywZhb2+q6BtDgyJHDRBQYeH3ZilW6rqVs8fWtP2H8OCJasHDhxQsXdV0OAJRvhoaGIpFIIpHIZDKGGGMTYyIyMDAUi0USsVhmaEBEJsYmRGRgaCAWicVisaGhAREZGxszDCOVyaRiiUgkMjQyJCIjIyOWZaVSqVQqZVmRkZHhu01oU4xSqVQqlTzHZ2RmEJGBgYGFhUUBy6s5TsSyDMMQ0Y0bN5cuW1HC1ijzGNu+ay/M/eza7Daf70oo/VypnBF5frnj6IQKe0e3mnahlL6W8QMwaT3/zMrOUUt79VwTpDGtYSsM3Hpyps+ZKd4Tjsk/dnXlHWPR/pdTS9u+XNS9/9/5fZVmQfCZCgAA9BbGSAIAQFkkzU4mlUqkRJT1B6lUJpVJpNJcL0plMqlEKpVJpUQklcqkEolU9n4dudZJRCYmeQb+5UOpUikViqysit6GVu8plGmpaUqVKjo6WqFUqpRvliUipUqpzHpJqVQqVEqlIseLb1apJKKMjAyez9Hdad78n3y8ffIWw3Ecy4rSUlPOnDt3/PjxVStXElGu934aWLt6HevQ84cvX8Uly8WWlep1/ebLBlI+6PZ97brMQBmQdn7jliedJwwe2ebfGSeScNxKldhz8Ki2Fgkn1u0NR68wAACAIkEWBgAA2sodKmWFUO+CqmwRVbYX30dUWfkUZaVX+URUMplMIpEUWMUbWfkU5UymcvxVoUxLS1cqE4goe0SVFUJRVib1NqLK8aLi3RoUSpXOetwoFTk2LQhCVuB15/btE6dOXbl8heM+8edfmXf/hcs6mWQf7SZwrw7/ue3ZJ77jnxT183+W7Ou9ttd34/dfmX+txF/iCO+I3PtPG1VDdW3+qlMIhwEAAIoIWRgAQLmXq7tT9ogqK596s8z7F6UymZSIskdUWX/SsDapVCqVZo3+06YYjRFVjqDqbUSVlU8RUfaIKiufIqLsEdX7oEqpVCqVcrlcrS5wAvVPgkIhFwRiGFJznFgkioyMPHL06Llz51KS9WSSbEaa9PRsYCVvTxcHcxkpkqOC7184uHHF1muxn2AfuE+YkHLpj9lbXYckCjKGkIWVHokkPeLRqVOzdxRndCQAAICeQxYGH42s7oRN64aanpwx5LsT8fgwDPqgFEf55V7b26Aqa9YqbYopdJTf24hKUYqj/KAk1Co1w1BmZuaZ02dOnDz54sULXVf0kQnJgesmDF2n6zLKLO75n308/9R1FdoQkgLmDw/I54d81LYBNbd91Ho+EfJn+34YsE/XVQAAAJRPyMLKN2P/5Td/bfRi/bjBiwNzzsIhaT7vzN89U/7q33XhvbLy+0KGYRiGZbV6bAf4UMrUKD/KGVF9eqP8oCQePHx44+bNK5cv4wgCAAAAAJQiZGHlH2NYY/hvy6KGjdwSpNR1LQVS3Pyjr88fuq4CyiqM8gPI5fjx47ouAQAAAADgE4Qs7BMgcIJZ02lLpwcPnnsZs6dCKcMoPwAAAAAAAPiUIAv7BKjvbl4Z3f5/Qxb9eK/vlH2vNI6IlDSZc3JT38QVPXv//iRrAca8x8rAhY2uzGzzxe4EgRjrJqNmDmtRvbKzo42ZkYRLefX43LYVa+5W7D6oW7sGXk4WovTIByc2Ll647f67wZiMsUfn0eNGdGnoZSfLjHl6cd+aRWsDIlRExJh79504rF39Gh5uDhaGTGZc6PEfh8590W/H0QkV9o5uNe3C2/E+hi5tPx87smuTmk5mTGZi5NPzq2b9tD+0rAzqLLMwyg8AAAAAAACgeJCFfQq4yP+mTzF13/DF3CXDn37x1yN5MdbBWtX2829R/e0JIbF09ukxbX2PbEtIXT/rN+tPq/ReY/fH8ERkVGv8+r8m+ZhmDTkzcK7j/79l3o4Tus0KSBRYu0a9h3R6tzZTWzuJIi3PNmVeo9as/66BxZtBa1J7D29nk8xy3CWnVEb5Uc6IKldQZWhoKBKJtCnmI4zyUygUKkRUAAAAAAAAUK4gC/s0CGk3V0z6vfauaeN+//pO74XXi/md5ULKfzN6f3ckOl0wq9p1xpq5HR1Tb6ya9cvWKy/iBVvfkQtXja3bomdru4Pbo3lRlaGzx3sbxQYsm/HLv5dD5ebVOk79ZXav7uMH/XNxxfOstaWenDNg+sGIJM7Q3sEwWUWOOTYmch80e7KvufzZ/p9/WnP0blSmzMqlsnliXOmP8ixfo/yIKHtEhVF+AAAAAAAAAKUIWdgnQ/ls84y59XcsGjJv5tX+08/m7YWlBYFLfR2bouCIEh/tW7W1X7tvK8U9uvQ4OoOIXl1au/7kAJ/uzm4uLEUzVf27eIlTTs2fuvZsskBEsff3/bisafulbRr72qx6HkdEJKgTIyPiM1REqlehKUQ5ezOJKnXxrylT3ftlwvdbX3JERIqYZ7djtC+2R/fu7fzaiURsVmrFsIyxkTERGRoZiliRRCKRyWTarIfn+YyMDCLKyMjkeS4rURIEIT09nYjkmXK1Wq3iVGkJaUSUlpZGRHK5QqVScRyXmZlJROnp6QIJSrlCme3FjPR0XhCy1vZuEwAAAAAAAACgW8jCPiHcq70//Ni4+u+9f5wecH92eknXFhX6Sk3VbO0tWMrgiYiU0RGxAmNnZMgQSZwrObGsYfvlge2X53ybo1MFluIKX7/Y1dNNxIcHXg4r/uxgCQkJHKdWqdQKhZyIUlPTiEihkKtUao5TZ8rlRJSeliYIlDXh1PugKiOd5wXMQgUAAAAAAACgb5CFfVKEuDM/fb+n3ppec2Ze/DUz54+IJ5IZGGg1jo+IiHiVUk2MRCJ59xaVSi0Qw7BEJAj5DGVkZIYyrbbBsCxDlN9qtLBv//6LFy4W++0AAAAAAAAAoIeQhX1ihKSLv83c7vvPgKmTYowYSnn7Op+anCawFat6mDN34kthRi5VZEgkz5sfGNlu9lkNg/+0mNxdFRkSybMuvo2cRfdD8MWRAAAAAAAAAPAxsLouAEqbkHp56dxt4eaOjtn7gHHB9x+nCLKmY6cP9LE3ErEiA1NbS0PtO4nlxj09fjKYt/Gfs2hEm+oOZlIRKzKwdKrRsr6rtvEq9/TYiSBOUmfi8rmDG7hZGohEEhOHqt5VrXFOAgAAAAAAAMCHgtzhEySkBv42f19kzr5W6Rc2b36sYJ07zttx5uGj+y/uXjk5zVdS/I2oH6yfv+Gp2tlv8rp9p+/evxf06Oat0zvXTmlZUdtzSv1w/bw1D9MNq3T/adORW3fvvXhw7cqBtV/Vkxa/KAAAAAAAAACAAmGM5CdJSL7wx89Hm6/olO01xYNlo8ekTB4/qFUtFwuJoMxIio8OC3oU8CKzeEMmhdTrCwcPfDh8xAA/3+rO1sYieeKroDs3wpXaryHt5pJhg5+MGD2sU4NqjhZSdXL0ywdBKRKG5KUwihMAQJ94eHhMGD9O11UAQHliaWmp6xIAAAB0gzG3ttV1DaDBkSOHiSgw8PqyFat0XUvZ4utbP+t5b8HChZg7HwAg698LAIBiw2cqAADQvHjYzAAAIABJREFUNxgjCQAAAAAAAAAA+gJjJAEAAMqxzp276LoEAPiAxGKxjY2Ni4uri4uzq6uri4uzq5urRCzhOO7169dhYWFhYWGhb/4PUyq1n6wCAABAfyELAwAAAAAoo9RqdXR0dHR0dGDgtaxXcqVj3t7eXbt1k0o0pGPh4eEKhUK39QMAAJRByMIAAAAAAMoNpGMAAAAlhCwMAAAAAKAcQzoGAABQJMjCAAAAAAA+KXnTMZFIZGtrmzcdI6KEhIT3k46FhwW9CEI6BgAAnzZkYQAAAAAAnziO4wpOx2pUr96xQwepVEp507GgYIVcrtPyAQAAShOyMAAAAAAAvYN0DAAA9BayMAAAAAAAQDoGAAD6AlkYAAAAAABooE061qF9e5lMRnnSseCgYDnSMQAAKJOQhQEAAAAAgFaQjgEAwCcAWRgAAAAAABRT3nSMZVk7OzsHBwcXVxdXFxdPDw+kYwAAUKYgCwMAAAAAgFLD83xWOnbnzp13L1pZWbm4uLxPx9q1kxkYUJ507GXwy8zMTN3VDgAAegFZGAAAAAAAfFgJCQkJCQlIxwAAoCxAFgYAAAAAAB8b0jEAANAVZGEAAAAAAKB7haZj7du1M8iWjkVHR2cFZC+DXyanJJd6PTIDAwWmMwMA+BQhCwMAAAAAgLKo4HTMxcWlZcuWWelYWlpaWFjYu+5jpZKOtW/frnmzZuvWrXvy5GkJVwUAAGUKsrAyzde3/pZNf+u6CgAAAACAMqHY6VjIy5dJyUVLx1xdXLy8vBYvXnzl8pW///77VVRUKe8MAADoCLKwMsfY2NjNzU3XVQAAAAAAlAOFpmMtWrQwNDSkoqdjlSpVYhiGiBo0aNCwUcOTJ05u3rw5MSnpQ+8RAAB8aIy5ta2ua9BrLMs62NtXqlzJzc2tUqXKHpUrW9tYE5HA88QQw7BEJM/MvHPnrlqt1nWxZcu+/fvQXx0AAAAACpUrHXN3d9eYjoWGhGSPunbt3mVkaPjurxzHcRx38ODBHTv+xeT9AADlGrIwnfn882He3t6urq5SqZSIVEqVWCxiWDb7MjzPp6elT5w0MSYmVkdlAgAAAAB8auzt7ZydXVxcnF2cXVzcXFycXbLSsaxZ+cNCw17HxY0YMTzvGzmOS01L27xp84kTJ3ie/+iFAwBAKUAWpjNdu3UdM3p0AQsIgqBWqaZ+O+358+cfrSoAAAAAAD1kZ2fr7Ozi6uri7OTs6uZmZmZaoUIFzYsKAi8I0TExGzduvHjh4sctEwAASgGyMJ0RiUR//rmqgmMFlmE1LiAIwrz5869eufqRCwMAACgvunXvVt2rmq6rgHJvwcKFui4ByMurao/uPXRdxXuOjo7uldyYfD6oE5EgEMNQcnJycHBwelr6x6wNAMqXR08eH9h/QNdVEBFN/+47XZegS9nnWcLc+TrDcdy6det/+OF7jT8VBGHNmjUIwgAAAApQ3ata02ZNdV0FlH+IwsoAG1vb8nU5MwwRkbm5uY+Pj65rAYCy7gCViSysfN1mS92FSxfpbRaW7y864CMIDAx8/vy5mss9Kb4g8Pv27T906LBOqgIAAAAAAAAA+FShX5jOVHCoMHzEF56enoIgZH+d4/hrV6/+/fffuioMAACg3Bk89AtdlwDlz4Tx43x96+u6Csht2YpVgYHXdV0F/bX2T5lMRoLAsqwgCEmJSS9Dw8LDw8PDIyIiI6KiojmO03WNAFDWbdlUFp/r76cabY3Wo8myaplmDHJ4netFZGE6YGBg0LNnzz59+8TGxPz441xf3/p+fn5isZiIOI57GRKyeMkSfCsNAAAAAIBOGBsZBQcFhUdERkREhodHRERGKhQKXRcFAAClBlnYR8WybMtWLYcPHy4WibZu2XrgwAGVSvX0ydNWrVqJxWKO4+Li4mbPmoV/awEAAAAAdCU9I2PBL4t1XQUAAHwoyMI+njp16owaOdLZxfnUqVMbN21KSU7Jej05JXnr1q0jRoyQyzNnzpiVkpKq2zoBAAAAAAAAAD5VyMI+BkdHx2FDhzZt1vTOnTuLJvwaFhqWa4FDBw+3adt2+bJlUdFROqkQAAAAAAAAAEAfIAv7sIxNTPr26d2te7eoqOg5P8y5fuOGxsVUatXkyVMUcvlHLg8AAAAAAAAAQK8gC/tQRCKRn5/fkKFDWIbdsOHvw4cOFzwdPoIwAAAAAAAAAIAPDVnYB+Ht7T1q1Cgnp4pHjh7dumVrenq6risCAAAAAAAAAABkYaXNyclpxIjhvr6+gYGB836ah/m/AAAAAAAAAADKDmRhpcbU1LR3717dunULCw+bNm36gwf3dV0RAAD8n707j4/pah8Afu69s89km+yRxRJEQsQWROx7EJqi9j2oEmqrtqjq21JLX7uW0B9KvaUUVXuIxBYldrKSfV9nvXO33x+TkHUmiUQSnu/Hx2dy595zn3vm3jMzz5xzLgAAAAAAAACUArmwWsDj8QYMGDB1yhSaoX/++ZeLFy8anhoMAAAAAAAAAAAAANQLyIW9LS8vrzmzZ9s72P9z7txvh35Tq9X1HREAAAAAAAAAAAAAqBjkwmrOydFpZuDMLp07R0RErFn7bUZ6Rn1HBAAAAAAAAAAAAAAMwes7gEbJ1NRk9pzZO3ftsDA3X778i2+/XQuJMAAAAAAAozCLPqt+/zts/QBhfUcCAHg7hEvAutMXT3zlXbe9K2q90TBa4AfVTFXnYN/yFX9HJ8yb/dn6zNt46OqtezHPIh9f2zHBCVIfDQ4mFC4YbHnMRyioj73DCVE9PB7Pf6R/cHBwT1/fXbt2f/754qdPn9Z3UAAAAAAAjQMmcnBv18xawsMQQkjYMeh/9//958dBllh9BwYAqC4TJ3d3J3MRpr986+pyLt1ovIsCa9RMNdbWrFrVW/oVr7a33NygcvUv8Fj4y46l/h2bykU8QiCzxHWFHNFyyuEbETe2BzhDFqRhwAiipSVPztOfE1jb9vK/P7Fa4Yy/m4sIxkhWg7d319lzZskt5GfOnDl69H8ajaa+IwIAAADA2yHMPYaMmzyyX/e2LrYmPG1eStS9sDP/O3z8dqq2LncrHbH93qZeMVsCRu2OY0osx2w+OXRltffDdX2m/JZq/E48hMf07Zsnyc7Mm74ziqlsJemI7fc29Sv/mz95YYlX0Pk6PUyjMAzDMPwdfewF77WGfJ5XFW7aeuAnUwMG+LRram8uYlU5yXHP79249L/fz0RmV3qBNxz1eTkTLrN+O/FVByrkqxGBJ7K4Wi279HFV2urWyeETrvOP/bnYNeqnMRN3RFHlnpZ2X3n80CTb0BW9Z/6lqNUd1xRu6jZo/LSA/j7tmtqaCeiCjJdRD29eOnnw+K1ksm73XKb+hV3Hjm8t0MX8sfTzbZfiFXxLM1TIIVsMxzCcgLcc44R2sp+6iJzEuIyPERyn0bFpBdT9JO3JWDKZrsP9YlXrrtWyjfnK1tjl0LxDeTXfF+TCqqRF8xaz58zy8Gh7I/zG179+nZGRWd8RAQAAAOBtYead52/ZtLCb9esPxkLbFt5+LbyHjJ5w7Ju5351PLP/Vo2HBzV3cW9rnCRrrB3vy3taxHbbWdxQANACYqeesjf9d1tuO//pyNrV17WDr6tmG+vdcY8iF1eflLO4yeUp7EYYJe08f5356+9Pa/K5e5rgqa3Xr5vBxK1trHBO2mfX58OPzTqaX/oGEaDl++RgnAmMsrOQEUtT7KYKZes7a9N/lvexe9zETyB09uju6e0mj/rldx7mwsi+TrWsLM4wM27/1bEw+hxCZmYMQQtEHxvscqNM43huEmOdmRhQNXcQwqYhwFRGutiL/VprvLhder5P7BXJPHuYOe1iVNTEzE35TKfuWIyshF2aEqZnpuHHjRgwfHhsbu3z5F8+ePavviAAAAABQGwinCZu3LepuyqTd3r8z+NjVhwkFrNShTc8R0xbN6t9m7A97CtMDNj14X+4PTT8p1wcNgHfDzs5u7qdzr18LvXX7dh2Pq2i05zluH7B+54o+Flx25KFde46GRMblMDJbx6aunr17m96IrOMeNY0dbus/Y0QT9Y3DVxzH+Y+Z1efA4suFtds1rN4IrWzNMF1+AdFzdmDHc9/9W6KDI2Y+aO7UdmR+vsBULrfAUEL9RYkQKj6He1tw2ZGHdu85GhIZl6UTWDi26dRziFv6jYIaviC40MTSXEQr8vLUdFWWF8OEIgHGabKzVe/JmVAF8+d/lpubd/369eTk5NoqM+ZRzqePaR1CQgHhZCUMaC/zk4sXeWojbusaR2dbgyAXVik+nz/Ub+jkSZPUavV/t2y5GnKV4z6cSwkAAAB4z8l6f7rQx4zLOL9swvJTqUXfnXUJkad3PAiL/OqPX8a3mrRozNEZB5JZhDDLHoFfT+3t3sLJwcpUwkfavMQHl3/fvOVoZN6bzwaY1HXY7Hkzh3dzsxFqMqLCT/6yYU9o8tv3LBM3HTTj00B/H3cHKZuXcC/k+K6dRyOySnzZJ1oFnXoUhBBCiM08Ornfdzert9MqHR0SOw+YNneWf4+2jqaYJi8l6vquld/9lcAYDQ+Xtx8/b+7EgR2aW/LVaS9u3S6wfTP+gWj56dF/guxPzO77RRhV1UhEjn0nz5k50tfT2VKMtAVZKfHRTy79unlvRH71Kha8KziOd+ncuUvnzjRN37lzJyQk5N6/9yn6Xfa65PdYc+ng2LwdAaP/+0J/cmJmH+2MWN/91tf9px/P5RBm5jV24dRBXTxcm9qZizFNdsKFb6esOZfLGbmuBQ6+E2fN9O/l1czahEcpc9MT4qIfnPl57YloBiHMss+XPy0c2trR1lTIabLj/r0Q/NOOk1EVfDmX+Mxd2keOsq+tnLDoaEJR6WRSdE5S9L2rpdasPJ63bKYqroHzWO8qHkKZyxkzG7Uv/Pu+Zbpt6G6v7D/rcKaRWjXYaFS04xajJvUQZ548+GNwyzZ+SwZN9LMPOVpyjLnRAqvTTOmXlW91S67Gdf7y7NGpZhc+H/jZueKhi5jlJ8EX13k/WjNo5sE0torvF7jcyhJnM//Zfabj0smfjtgXeOz1cfFajps3SHhr8y7loqU+VuZv4n2rNhmhmr6XFZ/DIV+N//yPxKL8FJkZF3EuLuJcxZsYvkBweefA1V/NGdDKgo9xHKVIvLR22hd/prKVLMfKvUwYwi3G7n0wVr8z6u43A2cc0gX837Vvu4Z/5f3pqUIjB1tpm2CkIuqPo6PT0KFDJ06ckJCQePnypevXw7Kzs9+yTI5FFIc4hLQkE5Oi3qjEWg6TuVoLnDFdmqV4ehtRezmviQQXIS5Pod1yuTBUizA+r5+HdGxTQQsxptXQ/8apfn5Kvu7SiIv4/u2kI50EziKkUdH3M1irEl0sm7aV/9qeuHA1e31qcT3zCF832SfNBa2kGM5w6XnkoduFF/VXFcabNsx2GkIIIVaj+fxk4X3jM0uUArmwinl7d507Z7a5udnxP08cP35cp9PVd0QAAAAAqE0+fn0sMfJu8E9nUst0IuHybu7cctlv2xCv4f0dDh1IZhEu9xw4orf7649NUqsWPcZ97dVKHDB5f7T+M7+k3fx9exd1MNF/pxA5tR+xYJuXQ9DIlaF5b/PJWeQ+Z0/wcm+zoq8qtq16j//Sp1f7JZO+LBd2jVXh6IRugb/sW9G1+OuWwNbVy0mmYY2Gh5n6rDy4fVpLUdHc2s5efs4IIVRJL5cqRCJyC9wTvMLbonhSGKmlYytLx+aC+/shF9bw8Xi8bl27+vj4kCR5+9bt0Oth9+79yzANoRcXbtN99GS/1+eeibUNn1RyRq5rkVvgL3u/6CovHmTNM7Nt5mnbrLXi4roT0QxCiDZv0bGVoz4fJLNt02fKxrY21MilZ7LLNAqi7iMG2OC6h79uOJZgMOVgKJ63bKYqqQFxFQ+hOgzWajUbDYSQsOPYgDbYy59/v61IeH44fM6mXqM/dj22PbrovDJaYPX3aBT9+Pqt7Ckfd/FpJzx3s6gcaSdfTyHzPDwsk636+wVmLrfAufzMOwf3X5/ww/QZHU7/5x6pD7r/7Alu2WemnYwaOosTyS2lGNJxxt8yjB9sDd/Lis7hyH0b/0ys8ghVAxcIbjdm/fblvU0xWpWTocRkluY2fLKArXQ5Iqq605JqcEU0Bs7OTtOmTZ0xY0ZMTPTVa6HXr4XmFxTUTtEYep25srQTf+TCL64fTC5BOgohHn9qP4vp1pi+SoUyfv/25u7S/MDbZAFCmEAwf4D5aPOiuycITPh9TRBCqNJUC8Eb19fiU1u86HwmMBcrQlJ7I6DhDgplubq6btjw4+rVK2NiYuZ8Ou/IkSOQCAMAAADesfnzP/Mf6W9jY113u2jtKsWZ2LAb6RX8jsgV3Ax7TGOEq1vzN5+vucJzXw5q7+nZwqOr78T1l9JZafuJEzvyEUIIEa2mrJrvJckM3TbDr4ebR6euo1cej2ccR82f6GrgAzqv7aLTsVFPX5b4Fx+2usebnhSE6+RVn3cx1T4/vnxsP4+2HToMClwXkoY5DF2zfKDF6w+kTPS2kZ7NWns0a+3RomdlncLK7is+cuPgkl02DB5ds4mrFnubaaP/Wjl5aEdPrzZd+g2Z8uOFbNxYeLy2M1dMcRUUPji0aHQ/j7Yd2vebuCj4brbhX24NRtJi0uol3ua6+L+/mTzEq52na7uuvqtD1Y3j6wlACCGCx8MwTCQS+fbssXr1yt+P/r5gwQJ3D3esdm4tZ+w8N4xTXPpmeOcOXq6e3XuO2XqHMnxdEy2nrFnS1YJOvPjD7JFdvDxbuHfs8PHOh3TJ8m5snDKqe5dOrm0823QbPiP4kday78d9LMoeKuHUppUMZ1+GhiUbzAtWoZ15y2aqbA1U+RDK12XBXzPaeejbpWYegxedSaZY9dMj+y9k4wbDqHajgZn2nDDcgXl44thzGnE5F/64ko23GjO2o6joeaMF1qiZMtbqkpHXbxYgefee7YpTBeKOPbvJ2Ljw8ESmGu8XuLncHOMUBYWZ5//veEqT0dMH6TvREM4fzRooe3z4t9vKwgIFh1vILXFUhbcMowdbs/cyhAgn91YynHl5PTyl6rltA2cXZtJ1UFcT9skvH3Xv3rlXv06dvLt9tClcXenyirF5f8zyKjoJ2047mFbmRa3JFdEoYBhGEDwMw1q1bB04a9Zvh39bt+6Hfv37icXiGhcoFRFtmkiW+UhdcZSfTSUWve1y4Xdy/I9m9vk9a+w51UMGNXczmWKN5aQol5/J6n8kc9S5wnMFnF1z6UhzhBBq7W4SYI4ps9Vrz2UNOpI59K/ctU91BnraObUynWWLk/mazZeyh/+eOeCPrKmXFddfD87k6P87m9Hzt4yev2X0/rPancIQ5MJKksvlCxbM/+9/f+Lx+EuXLl23fn1WJsyRDwAAANQDN7c2c2bP/vXXX3ft3Dlu3DhnF+da34WJBENsQU5BhZ+eOFVenpbDxVLpmy70HKPIyiwkGZZWpvx75IdDT2jC0s3NGkcIEa1HDHfjFV7+ftmeq3H5JK3NfHzy221XlURLH2879/kn3nw5f3Z6qUeVf74mWvqP9BBQj7cvWXvsYYaa0uUn3Nyz9Js/0jiLPv79jX8brQ5DR9d8+Ii2QurRtqDVhyMS80hKW5gRHRmdhRkLj2g9eEBTnLy3ZcmGU48z1JSuMOXBmd8uxhr+qmQwEr9hHgLmxc+frzwYkVSgYxidMiunkfxSD0rj8fgYhkklkgED+2/csOHgwQMDBg6o55g4Oi8lOUdNMWRhakKGCjdwXVvhRCt/f3cB/Wzn/OV7Q2OzNQzLkIU5+ZqSpyOGmbcb98P+P2/cufvo6sE1gx0IxLO1tyr7BQyTmkoxxOTl5Bn8MmeonSkus4bN1OvNS9cAV+VDMAC3HvDNzxuGW708umT6jzeyMcO1Wt1GA7MaGDDQXHvzz38SWYQQUt3483QyauIX0FNWVGlGCqxZM2WU+s656/mYQ8/+7voGX9ixXw8LLu78xVimKi9EMaGZmQRnlQoVSz44+FukoM+U8S0JhETeUyZ4qa8EH3/FILVCyWHmcnOsCm8ZVaiNqsdW+nWQmkgxxObl5lcnIWHg7OI4DiHM2s3bzUqEIcSRWS+T87nKl9dAza6IxgVDOI5jGNa2bdvPFy06+r+j365ZU60CWnlZhk6yvT7R5vxoqz19TYbLMUah3f6ILMpHcVyBismjOYZhMxSMGuP3b8ondNqdN1S3Clgdy+XkaLY+ItU4r5MtgWP8Xk48nNHtD1dcymE1LKdUUleiyITKahXj9W/GFzDU/10v/CuDKWA4Use+zKJrcZQqjJFECCGhUDhixIhx4z5RKZUwNRgAAABQ72i6qFO2S1OXCU6OkydPysnOCb95Izw8/MXzFyxb/Z//ylGoOYSbWZrhqIIbtGFSCwsRxqlVFc/KixCTGvdKxXnIZFIMISRwau6I4+LB2yMGby+9moOjHaGqLIQK5vnGbD45dGW1t/4PgYurI84m3bnxqsQqqvthkdrxQ1xcnXCUW9WDreac4qWPjufSsinBJkXcTCy9tdHw1A5Nm+Bs8v1/y/4aX3UVRnLzWmwt/zp/9uzftVsgqDoewUMIyeVyuVyuX+Lk6BQRcbdGhdXq3PmGrmt7XGDbwhFnk25ei6vkbMRMe638v+DxLsX3hRQ6OyGEGBwv9/2LU6s0COFm5mY4yiwVu2Dw5odbBybuHue39RljOB5UfmKgqjdTFW5enUOoDGbSZeGOLWOcs/75cuZ/rmexCIkNhsG3rl6jgTcZEdBNXHjlj4vFgzZ1D4/9FTN1Qd8x/eWXT+VyfGOtkNEVakh14++rOSNGDRjgtunRU0bgObC3Fffif2djGGMvREbJic5MzExwTqdS6RBik/767cKcnyZM7v7rNssZ/nZJx764nM8hTKVUsbiLqSmGEP+t2+QanCR6lZ/DlTJ4dnGKmyevZPcd1vurg5eX5CU8eXAv9PRv+8/HqCpbXoPv7jU+2Brx7el7tmfdvtGoVJV+2sBxHCGEI9S5S2f9EjshxcMQXbV6Y1lOo2PTCqiHKdq/YshXlb0DE4STDOE80ZqxojWln7GR4TiON5EiVkk9qjTMMkETTU0Rq9TdVxhft2Y+9FwYhmE9fHvMmD7D3Mz0+J8njh87pqMaSd9HAAAA4P2l1b6ZoIAgCISQpZXlMD+/kf7+KpUy4s7d62HhkZH332YXUbFqzq2Fb3fb3XGpZb8WYKbdfdvyODo2qtIv1ZxOp+MwTD9tVaW/oWFCMe/FjwGuO2oWY632/KqOUkeH4TiGUAWHaCw8jMARQhj+NodRKhKcz8MRoulan19q3fr1tV0keMPCwmLunDkGVmAYhiCI/Px8c3NzhFBSclKtx8AhFiGhSFSdk9HQdS3EOI5FCDFspX0a5AOmfeRM5N3ZsWrD4dtxWRqeVf+v/9riX8GqTGrsKw3Xuln3Lta7YioatV2VeCrcoqrNVMWbV+MQKsZz+Xj9ztnu9N0tc74+WzT+03AY1Ww0CNcRH3sJcZ7frrt+pZ/hegYMtj/ze6rRAmujmaqQOuKf8+mjJgwe4rnt6bNOQwbZMg8PnYtnqvVCYCamJhinVWs5hBBXGPrriYThE6ct5eS9BQ/W/f5IhxDiNGoth4lMTPgIcW/dJlf/JCnCpMS81HCtm3UzfA6XLNHw2cVln/1qsjJyzNBu7Tt2aNexb7NOffq64QHzz1a2PK8K+yytxgdbIy9evDj511+1XWopn4z9pHnzZpU9y7IshmEURQkEAoRQBsmvSiIs+kFO4BO6qnliDlVWpJDAMKxoErEq9yotGjlfd32USuXC3u/fxMLDwst80GnVqlVg4Cw3N7dr10L379+fl1f9S6hG3u96blDWrV8fHhZe31EAAACoNlJXwbTFPB4PISSVynr26tm3X1+tVqsorPnPhbfOXc0ZNrJL4OIRIV+cKjUPPWbR/bNFA80x7b2zV8qlySpEpbxKYVmzU7MGrbpa2cQlNaBLiEtmcZeuPVyIx/HFEUo79uwgQrrE+GQWIYymaQ5JJJK6zJpRKa9SWNzZu7sT8bhkdwOj4ekS45JZvKlPnxY7H0fXxm+NVHpqNos7d/Z2wJ8k1WYnDvi0UKccHBxQRakwmqZ4PH5BYeG1a9fCw8MtLS1XfPFF3YTAKgqUHN6ktasZ9iCnqt+sDF/XhPvLFBZ36tjJDn+SUsHZiFvZ2QmQ+tKh7Zdf6BBCiMrJKqxkOnbVzcu3Cwf17xq4aPDlr89V1q3GSDtjbPB19Tev2iEQRMWdKzCTLgt/XtXbPPH43EX7n2qqFgbhXp1Gg99+1IjWlexd2GnUyKZ/7E4w1gpVu5niqtrqav89dvrVhMDBIzvuNx/e30Z7e8uZJAZV6/0CMzWVYRypLhp5Sz05+vvdyV9N/YTLO7fsr2T9WafTaDkOk5rIMJTz1m1ylU6SCl9x9a0rtwsH9+8WGDTo0srzWQba56LNjZ9d2qTQQz+FHkKIMHH7eO3+NQP7DPaWnP1HVfHyCwZrsiJveUFVU3ZWdl2/0QwfNrz8Qo5lOY7jEPcg8uG166E3b9z888/jqI4STCyTokKsQLPiVOGt8p3qMX6iEuGmgm5m2IuqjGtlmRQVwmWCjiYoqrDMcxzNcQhh4rfr2fWBzhdmZWW1ZMmSn37aTNN0UNDCzZs3v7NEGAAAAPCekUgkMpnMytrazs7OpamLq6uru7u7l5dXl86dfXv69u7de8iQIX5+Q0ePHj169Ojp06dNnz7ts88+W7Bg/rJlS79cseKbNd98//1/Nm7YsHXrlp07d+7bF/x/v+5v1rSpgT3qk2Iikci6eHJ9obDq82MXUVz7eevNAsxuyMYjP68I6NLCSsznCc0dPf0+/e+xXRNa8qiYw1v+qGLChYm6cCmetRqxZsPM/u52pgICJ0QWjh7iGSsZAAAgAElEQVR9uri81ec0Jvr06Wc6frsFP60a7Wkr4QnMXHxmb/x2rD2WH3rmci6HEJeZnsUR9gPHDGwm4xEieYvOHg61/AEeISbq/MU4ht9+4fa1k7o2tRARBF9m19qrtXmMkfCYqDN/P6d47p/t+DGwZwu5iMAJkZmVRc0Td/TzSyFprLDDws3LRnjYygRCC5cuAQPbVPu1B/WKYWiEkFarDQsL//bb7yZNnLTnlz3Pnj6ryylKmPjHzws5oe/cLyd0sJUQOCEysbYQGzkTDV/XTMylK69YYadFm5cOd7eW8PgmDu39Jw1qWXwBsjmZmRQSdw2Y2MlBxsMQzpfJRJU0CFze+d3BT0ncYfiW//284uOuLa2lfALnieVOdqZvgnzLdqb6mxs9BB1Fc5h5xz7dHCVl2x1M3mfVj1Nbc092LF4fUjL9aKRWq9NoiDp95OeIa25+1atd0fzoRf+8/H+OZfltRo1oRRgtsNrNVNVbXfrZiRMPGPvh01dMHSQvCPnzvH4cZzVeCEwik2BIoyWLKpBNPXsoJJ9lUk8duVp8V0dWq9ZyuImpDDf+llGF2jAcW+Wv+Otz2H/L0V3LPurSwkrCwwmBiU2rrsPnfv5RG6Ls5kbOLqJZv4/7eTYxFeAYwefRCgWJEIYhrLLllb1cBtTRG3fDwHEcQ9Mcx8XExuzdt2/ypKmrv/km5EqIVqs1vnHN90qFJtGsWLSoh7SHnJARCMcwMxm/mw3BQwhx1OVXFI3zJ/c2HefAMycQjmEmYlxUeWnXEmmG4E/vZTrKljAjEI5j1hb85iKEEMpRsyxG+LqKnPiIIHAXG75t9U+CCl7o3Ny82NjYapfUgHl7d3n9WCgUfvzxx2PGjM7Ozl7/44/1+DPg+1fPDYeFhUXLlq71HQUAADQUQpGIz+MJ9Ph8gVDI4xEikZggcLFEgiFMKpUihGQyKUKYRCIhCFwkEvN4hEAoFPD4AiFfICjaBMdxiX4TmRQhJJVKjd79jWVZtVqNEFIqlQghlVrFsZxWq6FphiRJSkcpFUqdjqQomiS1NMNo1BqOY7t07iK3kFc2koNmGALHY2KicYxwbemKECLJ6t/0mUk8vHShfMumoK4+c9b5lOq2wqleHFs9Z0tklft40U/2fb+/7+7AgYuDBy5+vZSK3DBwwoGEmndgYmIOrd3SK3hZlzEbj43ZWBwclXJuzY8XcjmEEJMYGvIsqJ1nwKaQAP0uH/zgN3lvYvld8touOh27qGzUG0ZM2B1vNAz66b7//NJr17y2o747OOq7oihUZ4J6BRkLL/rAmk0+e1d4D/4qePBXJUqspHeMUdq7ezad7r9pVPsp205MKRlhDcsD75A+1UVR9K3bN69dDb1//z5N1/oLV/l5Hnbo0PMBCzyG/ufo0P+8ec5wu2H4uqYe7/vx8IBtkztM3X5yasnN9Kc3lxPyx5X5PYf1W32k3+o3TzLRFe6KerE76AvbXd9PaOMz54fSzRF63U3sLduZam9u7BCY5Ocv8jk3tym7L9gs7bLwUslt+R2H+jkQGNbu8xP3Pn+zaer/TR261lAY1Wg0JN1HDLHFCi78eS6zTPTUsxMnH8xY6jVsuNeujfeMFFjdZqqyVrfCVU//Fjpn88DhvZmE4CNhhUXZq6q/ELhEKsY4Uv26Vx1XcG6xb4vFJdfhNFqSw6SmMlSVtwxjB2s4tjKv+PlSb5HUi91BX9js+n5im57z1vecV/Ip+il+6vTz+NKbLzJ0dmGWXWd+u8qHX6IQNvfcxbtqy/4VLq/iDFSl1dEbdz3TDzmPj4+/fOVKeFh4bm41JhZ9e9FPFceamI9zkq13kr1eSGUpJl9Up3Do5YvCvfYWc21Fn/UTfVZiq8oa4phnit8dzCdZipcMFC8pWsZduZ61JpFLSSFjPPltWpgdaWGGEEIstfNM7tFqDhWoIBcWGxu7bceu6hXTsP128Ff9A9+evrNmzJBIJIcPHzn11ymKrs+pwd6/em44vL27QC4MANC4CEoSCgT8ov8RQgKBUCDkCwRvFgoEAqGw6C+BQCjg8/WP9MqUJpFI9HOmGqajKB1J6ipE6pRKtU5HkjodpStaCyGko3Q6/SKdTkdSOh35ZklxSaSWrNm7rZOTs0dbd6z0IAWGoXGcV1BYcPnSpfPnLqSlp325YoXrWzT4XN7dbTNGXfWbOHVUn64eLjYmBJmXGhUZ/vfRQ/+7kVKtH085xd31kyY8nTFz/EBvdydLKaHNS4178G9S9VN0pWme/Rw44eXMebP8u3s4yNi8V/euHN+582hEVtHXYybmQNAy028W+HdtbiEg8xOfxGYZy07WAKe8t3nqpBczZ0/169rGwVxAF6S/fBJXyMeMhYc0z/cGfhI1eXagf0/PplZSglLnZyXFRz0Ija/ZhzA269LyiZ9GB80e07udsxkqSHwUHi8b2L8VyzXaLy4fBpqmIyMfXA0JuX3nDknWNBf6Nsgn22bPKVw8f2Lfds7mfE6nzs9JT4x7FhqrMdAbzfB1zRXcWDt5ZtyCzyb292wq52syom/dzm09qrdD0ca551YGLklfOHNop5a2UoLWKgrystISb8cWVLhHJvXyqnHPL4ydMnFoj46u9nIpQakLs9MS46KfXb9VNAHTW7Yz1d7c2CGoQ7cs3ilbPsZbmJxW9bbOSBhVbDQwk94j+shR1h9/hpYfbsUknT91b0FH78EjO22/d9tYgdVspqrR6nK5F347u6TfOOtHfxx++Oa0r+oLgQmlEgLjNGrS0Emq0WgQJjM1wRFijL5lGD1Yw7EZfsWZ1Murxz2/OGbyRD/fjq72llK+TpWd+jI68uaFG3ls2c0Nnl0Ylnb/2qMmnVo2MRdiZEFKzL3zh3Zu+zsL2VS8nKvRkMa6euOuJyzLpqamXrkSEnotNC09rV5i4Cjd7ou50e5SfydBSxNcjHEFKvpZJlN0KdH07yG5ca2l45oJ2pgSEozTkGxqIf0speIfRjhKt/dybpy7NMBF4CrF+RybXUgl6BCGEJuvXnsDC/IUdzDDeQybmkPXIOeHmVlav/5DP49VRMTd9yxHo8+FFRYqpFLJpUuXDh08lF9QUI/xvK/13HB4e3cJmj8PwXxhAIDaVpSqEgr5fL5ILOIRPLFYTBCEVCrBcFwmkSEMyWQyHMekUimOExKJmMAJsVjM4/NFIiGfzxcKhSULEYqEfB7f6H41Gg3DMFqtlqbpomQTRelIkqZprUbLcqU7XqlULMvpNyG1JMVQ+swURVEkSdIMrdVoWZZRqzUcQiqlss5rrUbmfTZv8KDBPB6BEEIcYjmW49jI+w8uXbl86+Ythin6WP/lihW+PX0RQpOmTK/HaEH9wazH7glb2/nOqv5Tj1X7k3DQ/Hn60QPDKppmBdQWfaOnNNba+Pb0/XLFCoTQth27anofyXqG2084fOnrDiFLvILO1+VIJABAo6HPRZSfu7zWyeXyqvQC0+ciHiskh9Otja783mhnop5ol4VK5wfeg8GwVcWyTFBQ0KtXCfUdCAAAgDqnzzqVzz2JREIeny8WiQiCJ5GIcZyQSqU4jslkMoQhmUSG4bhUKiEIQiwW8wieSCzi8Xgikeh1lysD1Go1y7IqlZrjWKVSyXGcPi2lVqlIUpufn0fTjFar0ekokiRLdqqiKUqrJRmG0Wg0HMup1CpUnNjSl/kuqqyBoXUUjmM0w/AIIik56dw/50KuXlUo6uzG2qCRwG06DW2PYp6+TM0u0PIsmnfyX/ppVwEbF/m4Pn/mBIbpe4nWdxQAAPA+e8fDId8DH1Au7MnjJ5AIAwCABqiy4YFVHBsok8kEpenTXoZ3amBIYG5uXpnxgFUZDPjBJq3qiI7SaUny6pWQCxcvxsXF1Xc4oKEQeo1bv81PVnJAEsek/r37SHQlt94DAAAAACjnA8qFAQAAeEv6madkMpl+6nQCJ8QSMUEQYomYR/BEIpF+XnZ9Tyx9xkooEvIJvr53lVgiJnBCIpXiGCbV/y+TGd4jRVOkliw585R+nKBWo6VpWq1WK1XKpKRkjmOVSlXx/5xKrWIYRqPW6Pth0TSl1b6ev4rUUfU5WSSoonPnzx85fAReLFAaJsiPuhrR3Kuls52ZEJEFafGPw04f2HH4Ttm5swEAAAAAKge5MAAAeJ+V6XIlk8nK97eSSWWVTcRessuV0SnYDfS00mq0OorSpadX2NlKqVBV2NNKn/Z6Z3UFGpSM9Iz6DgE0QFxBRHDQlOD6DgOAirFpR8a3PVLfUQAAADAOcmEAANCA6Cdil8lk+vmq9EkofacqmURGELhYIhEI+EKhUCgU8fk8iURCEDypVFI0p5VQKODzJWIJQeCGu1zpp1EnSZKiKLVazTCMSqXS97fS6XRKlVKjUTMMq1AoGYbVaNT64YGve2OxLKtSqljEqvT9sFQ1upk1AAAAAAAAALxzkAsDAIC3JZFIeHyeRCzRz2kllUr14wGFQiGfx5fJpDweXywW6bNXMqlMf0tBkUjM5/OkUimPIERisT6NZWAvpTJWWq2+C5VGrWEYJj09XT/tur6/lVaroWlGpVIxDKNWq4vSXlqSoim1WsUwLKSuAAAAAAAAAB8syIUBAD5Q+i5XUqmEx+OLxWL9bOv6vJVIJBKJRTyCZ2Ii099JUCQS8Xh8mUyqvy+hWCzm8fhSqcTwvQX1YwbVag1NU2q1Wp+9UqnU+lsK6pNTSqWKYWi1WkPRFEmSpEZL0bRarWZoRqUuSnvp13yXlQMAAAAAAAAA7yvIhQEAGhn9xOxSqVTfCatoQiuhQCaR8QU8oVAkFhfnrXh8oVAoloh5PJ5UKtVPiCURS3h8nkQiqax8lmXVarV+viqVWkVTjEajJkmS0lHpGek0xWi1Gq2WpChKqVTSNK0ltVqNlqIplVKln+hdo9bQDA19rwAAAAAAAACgAYJcGADg3SkzHbtUJhXyBQKRUCqRCgR8oVAkkUgEAr5YLBaLxQK+QCwRi0RigYAvkUiEIpGAz5dKpZUVrlarKZrSqDWv+1vRNKXVaPPy8miaViqVFEVptaRWq6EpRqlWUjqKJEmNRk1TjEqt0ie/9DNhvcs6AQAAAAAAAADwLkEuDABQVW/SWCVuRygzkVZ4L0KZTFom82ViYsKvZD6sMvcfVCqV+ge5uXk6XTqp0ykVSv1tB4ueIimlSlF8C0KlTqfTaDQMw7zjCgEAAAAAAAAA0OhALgw0IoRLwH+2z219e+XYHyLo+g6mkcEwTCqVikQi/VxXEolEJBQKREKZVCYUCUVCoUQiEYvFIqFIKBTKTGQCoVCoH4dYnM+qrGS1Wq2jKK1Go1FrdBSl0ai1Gi1J6XJzc7UarY6iVCqVjiR1OkqpVurIolSXfiyhWq0idRSp1b7LqgAAAAAAAAAA8CGDXBh4B4Qdgw4GTzG59NXkFRdzuLcoyMTJ3d3J5AGG1VpojQeO4xKJRCyRiARCoUgolUmFApFIVJTDEgqFYrFIIpGKhEKBUCiTSYVCkVgkEukXioQVzu/OcZxKpdJqtSRJajQatVqtJUmdlkxPTye1JKnTVZjG0lE6HalTqVQ6nY4kyXdfFQAAAAAAAAAAQI3VQi5M1G3hoVXDm9nITaQCgtEq8jMTo55EhF/68+TVFwVVHLJEeEzfvnmS7My86TujYJRTabip26Dx0wL6+7RramsmoAsyXkY9vHnp5MHjt5KNZCEaUK1iGIZhOP4hprBKqWyMYfFjgcxE9nqA4euhhfoHUqkUqygJqKMopUJRfmihQqmgdDqS1ClVyuKxhCqdjtQPM9SPK4S5sQAAAAAAAAAAfGhqIRdGWLu2c3UQFv0hMbdpam7T1LPnsOlzHx1cueyHyylVGMyGm7u4t7TPE3zwuZIyMFPPWZv+u7yXHa+4ZgRyR4/uju5e0qh/bieThrtYNZxaJe9tHdtha31HUXtkMplYIhGLRRKxWCQSy0xk+rneJWKJWCyWyaRiiUQsEkkkYrFYIpVKJWKJWCrm8yqYKothGI1Go1ZrtFoNSZIqlUqrJUlSm5ubm5ycou+xpVQqdSSpJUm1Wq2/oaFWSyqVSv0M8e/+8AEAAAAAAAAAgMartsZI0s92jh+987kWCaQWdq6ePsMmTpvco/20//6CzR6/9pbibYbFNRYCPl8oEikUilorEbcPWL9zRW8LLjvy0O49R0Mi47J0AgvHNp16DnFLv1HwIVRqnSMIQiqVSCRSmUwmlUmlEqlUKpFIpVJJ8V9SiUQikcmKsl165ctRq9WaYiqVSq1WKwoVGRmZxX+qtBqtWqvRarRqtVpLakmSVClVJElSFCSzAAAAAAAanN69emq02pTk5LT0DOhHDwAA75lamy+MpUgdw3GIVGYnPAhJeHD1nyvL9u+f0XrSisl/BOx6ziDMss+XPy0c2trR1lTIabLj/r0Q/NOOk1GqNxkdolXQqUdB+tIyj07u991NqgpbNRgWcos9e/Y8fPAw5OrVW7dvv/104BKfuUv7yFF2yFfjP/8jsah3HZkZF3EuLuJc0To1rFWp67DZ82YO7+ZmI9RkRIWf/GXDntDk1zkZkWPfyXNmjvT1dLYUI21BVkp89JNLv24OjsgvKlbcdNCMTwP9fdwdpGxewr2Q47t2Ho3I0g/DxMy8xi6cOqiLh2tTO3MxpslOuPDtlLWxnxz9J8j+xOy+X4QV70bsPGDa3Fn+Pdo6mmKavJSo67tWfvdXAlO7r7i7u7uZialEKpXJpFKpVCKVyl7nuqRSmVQqFInKbKLValUqlUqpUqpVapVKpVanp6crlUqNWqPRaNQatUarVSqUGo1ao9HqE2AqpbJG0QEAAAAAgAbK07NtV29vhBDD0OkZGa9eJSYlJScnJyelpORk59R3dAAAAN5Knc2dzxXc3v7jscHBU1oOHeb2y/OnDKLNW3Rs5aifv1tm26bPlI1tbaiRS89kG8xy1GyresLj8Tp26tixU0eaom7fvhNy9Vpk5P0ad/zpPmKADa6L3Lfxz8TKh5nWoH4k7ebv27uogwmOEEJI5NR+xIJtXg5BI1eG5nEIidwC9wSv8LYonttLaunYytKxueD+/v0R+QxCSOQ+Z0/wcm8z/ebItlXv8V/69Gq/ZNKXZ1IZhHCb7qMn+7kXn1gm1jZ8snymSOgW+Mu+FV3NiwoR2Lp6Ock0bA2PqHIj/f3102kpixUUFCanpLyeSEupVCoVKqVKoZ9CS6FQQEctAABojILmz6vvEEDj4+rqWt8hgAoMHTyom3eX+o4CNWniwLEchmMEwWvi0KSJgwPHIf3krQzLarUalVKt1qj1P5fSNNziHADQaDiLyYl2WfUdxbtjyqtg/vS6vI+k5uG1OwWTApq0aSlFTws5xY2NU0Z9HZeUpaT4Zs7dZ/2wY2bfj/tY/H08tyjJwURvCxj93xelojS+VQOjf4PkCwQ+Pt179uqp1Wpv37odej3s3r1/GaZ6E9i7t5LhTNz18BQDm1W/VolWM1fN95Jkhm776sf/3UzQmrUZuuzHVR+Pmj/x/8J3xKAWk1Yv8TbXxf+9bs2OUw9SlUhs99GGi9/2eL256+RVn3cx1T4/vuabXWef5QkcOo9d8e2yvkPXLA8J//x8HlcU1qU14788nZzPiG3txAUUcigVNdFs4qrF3mba6L9++O6Xfx6maYRy5xZmedlclY6oOtatXx8eFl797QAAADQy3g3gmzMAoFa0bNkwc5RvbmJE4LhUIpVKpPUaDwAA1JAZj2lnoq7vKOpZXebCEJ2bW8BhJhKZBEeFLIaZtxu3/Otu7i72cr4qLZslEM/W3gpHuYZSRDXbqgEgeDyEkEgk8u3Zo0/fPoWFhddCr4WFhT9/9ryKJZhIMcTm5eYbnJ6guvVDtB4x3I1XePn7ZXuuFnAIoczHJ7/d5jt4S38fb6td8aZ+wzwEzIutn688GKXvHqXMylGWGHHZ0n+kh4B6vGHJ2mNxDEJInXBzz9JvXP7+eXwf//4WF47nIoQQ4ui8lOQcNYUQlZpQiBBROobmw0e0FVKPfgxaffglgxBCZEZ0ZEYNjwgAAAAAAAAAAACgyuo0F8aTy80wjlWr1Bxm2mvl/wWPd+EX/ZwidHZCCDE4bjCAmm1VCStrK9+evjXYsIrMzcwqXM7j8RFCpqamw/yG+Y/wz8jIqHC18lQahHAzczMcZVaSBapB/QicmjviuHjw9ojB20s9wTg42uM8q5ZNCTbp5rXYSsYJClxcHXE26c6NVyVCUt0Pi9SOH+Li6oSj3CocGM+lZVOCTYq4mVjuuGr1FQcAAPDeW7d+PVpf30EAAGpDeFj4sLDh9R0FsrGxdnJydnFxdnFy7jegP47jFa7GMiyGY0+ePNm7NzguLu4dBwkAADUzbFj9N7MNRF2mGMTt+3Q1w9mEFzEqJB857SNnIu/OjlUbDt+Oy9LwrPp//dcWf8MFYPIBNdiqMm5ubl+uWFGzbWsFQRAIIVtbW/2fZmamhtePeanhWjfr1sV6V0x6hX3DalI/HFfJSENMKBZiOJ+HI0TTlffAwip9puowHMcQqiiQ2n3FAQAAAAAAMMDW1sbJydnZ2cnZydm5qbOzk7P+juG5ubmJiYlKhcK03K/d+k/T6ZkZBw4cgLk4AACgkaqzXBhm1m3+8jFNcDr6wtnnDO5qZydA6kuHtl9+oUMIISonq5B8szZH0zSHJBJJqVQLbmV4q+oJDwtft74Ofzu2tbXZv39/Zc/SNM3j8fJy866Hh43090cIFRQUGi7w1pXbhYP7dwsMGnRp5fmsCrJhxuqnolqlUl6lsKzZqVmDVl0tP0CY1yE1m8WdO3s74E+SKsq/6RLiklncpWsPF+JxfHHGTNqxZwcR0iXGJ7MIVfzTWSlUyqsUFnf27u5EPH5VKu1WtVecIKCXGAAAAAAAqCa5XO7s7Ozs4uzi7Ozs7NysWTN95kupVCYmJr6Mf3ntWmhiQmLCq1d5+fkIoa+++qq7Tzcce/P5lmEYhVJ56OChixcvsqzBmUwAAAA0YLWWVMB5fAJDDC6QWti7tus+fNL0ST0chdTLQ+sPPmcQysnMpFCrrgETO0Ude5imZHkymYiHUHGag8tMz+IIj4FjBh6JvpRImzZta6+JfJpmZKtGgGFoHOepNeqw62FXQq48f/ac4zh9LsyovPO7g6f6Lm7nv+WofN+OfSfCnibkkrjUsqm7dz8ffui2ky9qUqtRFy7Fz5k7Ys2GV/ius3djs5QM38y+RXt7ZfjdBJp+fikkbdqUDgs3L8v69v+uxuTz7T0HD2wjeHM80adPPwtc3G7BT6uyV+/+51kev0nnT774dqw9ln/hzOUqzm7PRJ2/GDfn0/YLt69V/2fv2YdJhYzYurmrWfajaGOvuI6iOcy8Y59ujpE3k9UwgRgAAAAAAKiY4cxXYmJiWHh4YkLiq5cv8wsKKiwhISHBu2sXnIcjhBiGYRjm9OnTv/9+VKvVvtMjAQAAUNtqKxfGc5//Z9T8kks4Jv/xgZVLvr9ZyCGEckL+uDK/57B+q4/0W/1mHSa6+EFiaMizoHaeAZtCAhBCCFEPfvCbvDfJ8FYNF0MzOIGTJKm/ieT9+/dqcqNl6sXuoC9sdn0/sU3Peet7lrpXPP0UP3X6+cua1Grwvu/3990dOHBx8MDFb3YVuWHghAMJrPbunk2n+28a1X7KthNTSu7vdeExh9Zu6RW8rMuYjcfGbCxayFEp59b8eKHKN3qkn+77zy+9ds1rO+q7g6O+KypDdSaoV9AlI0eU/PxFPufmNmX3BZulHgvPV3F/AAAAAADg/VYm89W8eXORSISqk/kqLzExkUfwaIbBEPrn3LkjR44UGhvYAQAAoFGohVwYkxX7OK5NMxsLU4mQYLXKguzE6Cf/3rh47PiVZ/nFPXe43HMrA5ekL5w5tFNLWylBaxUFeVlpibdjC/T5EybmQNAy028W+HdtbiEg8xOfxGZhmNGtGiaapiMi7oZcDbl3918dVckk9FXDpF5ePe75xTGTJ/r5dnS1t5Tydars1JfRkTcv3Mhja1arnOLu+kkTns6YOX6gt7uTpZTQ5qXGPfg3SYcQQojNurR84qfRQbPH9G7nbIYKEh+Fx8sG9m/FcsWdwDXPfg6c8HLmvFn+3T0cZGzeq3tXju/ceTQiqxq9tDjlvc1TJ72YOXuqX9c2DuYCuiD95ZO4Qj5m7IjUoVsW75QtH+MtTE57m4oFNTNy1Eh3tzb1HQVo9Op0uDoAAIAPQRUzXy/jXxYUVjXzVV5iUiJCKOL2nV9//TU1DT58AgDA+wMzs7R+/cfZs38jhCIi7m7bsav+Qqp9vx38FdX9fGFW1tZBC+ZfCw29feu2Wl1+Lq43GlU9Y9Zj94St7XxnVf9px6rc8au+eXt3CZo/DyG0bv16mNO0Fn25YkWd3owVfCDg/jUAAACqpbLMl36G+/T09ITExLfPfJXH5/ObN28WFdXwB6UAAACoHpiEvNZkZ2WtXv1NfUfxtnCbTkPbo5inL1OzC7Q8i+ad/Jd+2lXAxkU+btB98SozY/r0/v36ZmXn5GbnZGZn5WTn5OTkZGVlkWQjmnQOAAAAAOADUibz1aJ5c2GJzFdMbOyVKyGJSYkv419qNJo6jYSiKEiEAQDAewlyYaAUode49dv8ZCXv58kxqX/vPhLdKCeqf/nypVqjaeri0rFjBysrKz6Pr1+uUChyc3IzsjJzc3NysnOysrJycnKys3OysrLq+kPV+2HSlOn1HQJofILmz/P27lLfUQAAAGhYGk7mCwAAwIcDcmGgJEyQH3U1orlXS2c7MyEiC9LiH4edPrDj8J3MxnnP6KvXrpUcIymTyeSWcrmF3M7OTi6XW1rK5XK5awtXBwcHiUSiX0dHUbk5Obm5ubk5uekZ6Tm5ubm5uelp6bm5uXl5eRzXGLvHAQAAAAA0CJD5AgAA0BBALgyUxLiyo2oAACAASURBVBVEBAdNCa7vMOqKUqlUKpWJCYnlnxIIBHK53M7OTp8ss7e3k8vlXl5e9nZ2UplMv06FaTL9n5mZmSzbOPOFAAAAAAB1o2zmq0ULoVCIymW+4uPitVptfQcLAADgAwK5MAAQQkin06Wnp6enp5d/Sp8mk1vK5XK5na2d3FJuaSH38vKSy+UWFhYYhiGEKJpSFCr0Pchy8oqSZZAmAwAAAMAHgiAIa2trZ2cXZ2cnFxcXZ2cnJycnyHwBAABomCAXBoARhtJkfL7c0rJMmszD3V0ul5ubm+M4jhCiKEqhKJUmy9Uny9LTs7KyGKZRTsQGAAAAgA+Z0czX02fPzp07D5kvAAAADRPkwgCoOR1FVZYm4/P4JqYmcrnczt5O36vMztaupaur3FtuY2OjT5MhhJRKZXp6un6oZVpa+us0WXZ2Nk3T7/JYxo4dExMTExn54F3uFAAAAAANX/nMl7Ozs0AgQOUyX3Fx8SRkvgAAADR4kAsDoE5QNKXPcMXGxpZ5is/nm5iUTZM5Ozt7eXlZW1sTBKFfrWSaTD9NWXp6em5ebkZGZl18yuzdu/fUqVPj4+IP/34k4k4EjOsEAAAAPkyQ+QIAAPDeg1wYAO8aRVWaJkMIyWQyOzs7udxSLrewt7eTyy3t7OzKp8n0JaSnp5dMk2VlZtX4pkuWVlYIoWbNmq78+uu01LTff/899Pp1GMIJAAAAvN8g8wUAAOADBLkwABoWpVIZGxuLUFXTZO7u7rY2Nvr7kaMSaTL9tGRpaenFD9JUKlVlO+Xz+DKpFCGE4ThCyN7efvGSxdOmTzv+55/nz53X6XR1c6wAAAAAeKcqyHy5uAj4fASZLwAAAB8SyIUB0GgYTpPJLeVyC7mdnZ1cLre0lNvZ2bm6trC2thaLxfp1dBSVm5OTnp5eJk2Wm5srEAj0N8TUw3AMISSXywNnzZo4YcKpU6dPnTplIJUGAAAAgAaIx+NZWVmVz3wxDJOVlZWYmPjgwYNTp08nJiQmJSWRJFnf8QIAAADvCOTCAHgfKJVKpVKZmJBY/ilzMzO5paWVlbW1laXcytLGytrSyrJVq5bW1tb6+z0hhHS6Cj7+YhiGYZhMJhs37pOPPhp18uRfPB60GAAAAEADBZkvAAAAoIrgmy0A77n8goL8goL4+PjyT5mYmFhaym2sbbp27zZ40KCSXcNKIghCIpGMGz+u4qcBAAAA8M5B5gsAAACoMciFAfDhUigUCoXi1asEJ2cnhmZ4/EobBJZjEcfiRNEK9vb2aWlp7ypMAAAA4ENXPvPl0tSFz6sg85WYmAgTfQIAAACGfUC5sLbt2ro0dUl4lVDfgQDQ4FhaWXHlOn2xLIMQhuO4liSjo148fvy0vadn23ZtEUKQCGssMIs+K3ctHZSwZcCKy9AlANQFN7fWH436qL6jAKBxe/bi+am/TpVcwuPxHJo4ODs729naQeYLAAAAqHUfUC6MIIjt27ZdunTpwMGDhQWF9R0OAA2ItZU1jyAQQgzN4ASOYVhGZsbjh4+fPH3y/PmL5ORk/Wouzs71GiaoNkzk4N6umXUWD0MIIWHHoIPBU0wufTV5xcUcrr5jA+8HK2tr356+9R0FAI0bhmGRkZHOzs7OTs4uzs7OLs6Ojo44jkPmCwAAAKgjH1Au7NHDR7cj7syYMaNHjx5Hfv/97zN/syxb30EB0CBYW1uxLBMXF//48ZOnT58+f/G8rvLFhLnHkHGTR/br3tbF1oSnzUuJuhd25n+Hj99OrdPbtktHbL+3qVfMloBRu+OYEssxm08OXVnt/XBdnym/pRpvDwiP6ds3T5KdmTd9ZxRT2UrSEdvvbeonLLecvLDEK+h8/d6dHsMwDMNxmPgNAAAakh6+PXr49qAoKjUlNTEpMSwsLDExMTExKTU1labp+o4OAAAAeA9VkAvz9u7y28Ff330odY3juJArITdv3AwICJgxY/rQoUOD9+69d+9+fcXzvtYzaIy2bt2Wkpyso6g63Qtm3nn+lk0Lu1kTxbkYoW0Lb78W3kNGTzj2zdzvzifW7f7fHm7u4t7SPk/QWHNJ5L2tYztsre8owHtq245dERF36zsKABoZ/UfBF89fbN26NTUtDTJfAAAAwLvxAfUL09NqtUeOHLl27Vpg4Ky1a9dGRET88sue9PT0+o4LgPr08uXLOt8H4TRh87ZF3U2ZtNv7dwYfu/owoYCVOrTpOWLaoln924z9YU9hesCmB+o6j+PdoJ+U64MGAAAAVCg7OzsxKam+owAAAAA+IKVyYeFh4fUVxzvw7MXz149TU1O//Xatl5fXnNmzf/559z/nzh06eEij0bybSN7vem5QsrOy6jsEUETW+9OFPmZcxvllE5afSi3KEekSIk/veBAW+dUfv4xvNWnRmKMzDiSzCGGWPQK/ntrbvYWTg5WphI+0eYkPLv++ecvRyLw3k1xhUtdhs+fNHN7NzUaoyYgKP/nLhj2hyW/fs0zcdNCMTwP9fdwdpGxewr2Q47t2Ho3IKpHUIloFnXoUhBBCiM08Ornfdzert9MqHR0SOw+YNneWf4+2jqaYJi8l6vquld/9lcAYDQ+Xtx8/b+7EgR2aW/LVaS9u3S6wxd+E3vLTo/8E2Z+Y3feLMKqqkYgc+06eM3Okr6ezpRhpC7JS4qOfXPp1896I/OpVLAAAAAAAAAA0DKVyYevWr6+vOOrFgwcPFgQF+Q3zmzRhQk9f38OHj1y8ePEdTCL2odUzAAghH78+lhh5N/inM6llOktxeTd3brnst22I1/D+DocOJLMIl3sOHNHb/XXzJLVq0WPc116txAGT90frh49I2s3ft3dRBxN9nkfk1H7Egm1eDkEjV4bmvc2c8CL3OXuCl3ubFaWPbFv1Hv+lT6/2SyZ9WS7sGqvC0QndAn/Zt6KreVEYAltXLyeZhjUaHmbqs/Lg9mktRfpBnEJnLz9nhBCq5A6SVYhE5Ba4J3iFt0XxFGNSS8dWlo7NBff3Qy4MAAAAAAAA0Ejhxld5r9E0ffrU6ZmzAsPCwz/7bN5PP212b9OmvoMC4D3U2lWKM7FhN9IrSDZzBTfDHtMY4erWnHizsPDcl4Pae3q28OjqO3H9pXRW2n7ixI58hBBCRKspq+Z7STJDt83w6+Hm0anr6JXH4xnHUfMnuhLliy/Ga7vodGzU05cl/sWHre4heL0C4Tp51eddTLXPjy8f28+jbYcOgwLXhaRhDkPXLB9o8XqOMCZ620jPZq09mrX2aNGzsk5hZfcVH7lxsKDE8waPrtnEVYu9zbTRf62cPLSjp1ebLv2GTPnxQjZuLDxe25krprgKCh8cWjS6n0fbDu37TVwUfDfbcHrfYCQtJq1e4m2ui//7m8lDvNp5urbr6rs6VA13oAQAAAAAAAA0Zh96LkxPoVDs+WXPooWfkyS5YeOGL1essLK2ru+gAHivmEgwxBbkFFSYmOFUeXlaDhdLpW+6qnKMIiuzkGRYWpny75EfDj2hCUs3N2scIUS0HjHcjVd4+ftle67G5ZO0NvPxyW+3XVUSLX287dznn3iThHp2eqmHgexYaURL/5EeAurx9iVrjz3MUFO6/ISbe5Z+80caZ9HHv79FrU6Yb+jomg8f0VZIPdoWtPpwRGIeSWkLM6Ijo7MwY+ERrQcPaIqT97Ys2XDqcYaa0hWmPDjz28VYwx3aDEbiN8xDwLz4+fOVByOSCnQMo1Nm5SghFQYAAAAAAABo1D64ufMNiIuP++KLFd7eXefOmb3n593H/zxx/Nixur6zHgAfCIWaQ7iZpRmOssvnZjCphYUI49QqdSU30GJS416pOA+ZTIohhAROzR1xXDx4e8Tg7aVXc3C0I1SVhVDBfPaYzSeHrqz21v8hcHF1xNmkOzdelVhFdT8sUjt+iIurE45yq3qw1Zw7v/TR8VxaNiXYpIibiaW3Nhqe2qFpE5xNvv9vWo0HelcYyc1rsdAMAgAAAAAAAN4fkAsrKyLizoMHkf7+/uPGfTJ40MADhw6FXAmp76AAaPSiYtWcWwvf7ra741LLpmow0+6+bXkcHRtVafKI0+l0HIbpp63iuEq6JmFCMe/FjwGuO2oWY632/KqOUkeH4TiGUAWHaCw8jMARQhj+NodRKhKcz8MRomm4GyYAAAAAAADgfQJjJCug0+mOHz8+Z87cR4+fLP7883XrfmjarGl9BwVA43br3NUcTtQlcPEIhzKDFjGL7p8tGmiOaR+evVIuTVYhKuVVCstmnZjRwUM/b1fxv3bd19ypeRcmXUJcMos7de3hUiJCaceeHURIlxifzCLE0TTNIYlEUpdZMyrlVQqLO3t3dypdUUbD0yXGJbO4s0+fFvxaiiQ9NZvFnTt7O8A7BQAAAAAAAOD9Ad9wKpWTk7N58+bFi5fw+YJtW7cuWDDfzNSsvoMCoLFSXPt5680CzG7IxiM/rwjo0sJKzOcJzR09/T7977FdE1ryqJjDW/5IqtrgPibqwqV41mrEmg0z+7vbmQoInBBZOHr06eLyVj1dmejTp5/p+O0W/LRqtKethCcwc/GZvfHbsfZYfuiZy7kcQlxmehZH2A8cM7CZjEeI5C06e5TN7L09Jur8xTiG337h9rWTuja1EBEEX2bX2qu1eYyR8JioM38/p3jun+34MbBnC7mIwAmRmZVFzRN39PNLIWmssMPCzctGeNjKBEILly4BA9sIjG9phFAkeusyAAAAAAAAAKCGYIykEdHR0cuWLevbr++M6dN9fX0PHzly9u+zDANjhgCoJibx8NKF8i2bgrr6zFnnM6fkU5zqxbHVc7ZEqqtaFv1k3/f7++4OHLg4eODi10upyA0DJxxIqPFkWYiJObR2S6/gZV3GbDw2ZmNxcFTKuTU/XsjlEEJMYmjIs6B2ngGbQgL0u3zwg9/kvYnld8lru+h07KKyUW8YMWF3vNEw6Kf7/vNLr13z2o767uCo74qiUJ0J6hVkLLzoA2s2+exd4T34q+DBX5UokaxOLZSgvbtn0+n+m0a1n7LtxJSSEdawvGIn/jyuf6CjKKVCodPpdDqdUqnUP9CROh1FKZQKSqcjSZ1Op9NRxc+SlFKl0JE6HaXTkTqdTqdQKCiY1bFuYBjG5/Fg0kwAAAAAAPD+gVyYcRzHhVwJuXHj5scBATNmTB/m57d3b/C///5b33EB0MhweXe3zRh11W/i1FF9unq42JgQZF5qVGT430cP/e9GirZaRSnurp804emMmeMHers7WUoJbV5q3IN/k3RvGaLm2c+BE17+f3v3GRfF1bYB/Mxso+zSu4AFUBQLoGIJ2NFYoyQaY4sxYg+vPWjsmohG8ygajYgm9iQWYu+iAQ1FBBFUQFRQinR2F9g+74dFRTqILsr1//kBdmZn7jmDiBf3OfPtrKkjejhZ8VX5T6OuHvv11z8jskvjb2XSPp9Feiu/G9GtlSFXWpAa9yibavj5kow4avPXEx5+O+3rId3aWhlwFYWZT+KShRyqpvJIyYPd3l8mTJzmPcKjYwsTXZa8uCD72eOEmBuP65dnqLIvLx4/M9Fn2ujeHWz1SWFqbOhjvmf/1iqm/okjIWT16rVcHkdXR5fNZmtra2tp8TgcDp/P57A5PB5PW0dbj6Nna2vD5XK5XK6Oji6Xy9GqupVMrpBLJdLi4hKFQl5cXCyVSuVyuVhcJFfIpBJpSUmJQqEQi4vU8VlJcbFMIS8pLpFJpTK5XCwWKxQKiUQikUgUirfN+D4yOjo6e/fuuXrl6oWLF1NTUzVdDgAAAABAg6H0jU01XcOHxMrS8uuvv3b3cI+IiAjYtTsjM0PTFQG8P0t8fd093AkhEyZ9o+laQCMo0zEBIWu6hC/v//XROjxWU81nziw3t66EkKFDh9Xv9Fwul8/nc7lcLo/L5XC5PC5fV8DlcUpf53C5XC6Px+VwuQK+gMvhcHkv939JoCfgsKtcT00ml6tb0V43qZX2rBWpe9TUrWriIrG6N00sKpLJpKVNanKZWCSWyWTFxcUq1VtlhY2EkZHRgQP7VQxDU1RiYsLpM2dDQ0JlsvKBs7uH+xJfX0KI//YdERGRmqgU4AN2cP/vhJDQkND1fn6argUAAKAJQV9Y3aRnZKz38+t0odM072m/7dp59ty5gwcOFhfXemoXAMCHgzbrPLgTSYp/kp5TKGEbtuo8YuHMblxVcvS9Qo3UI5PJ8vLqnMFV9Coa4wv45TK1l5/y1Zkaj8vlcnl8vi6Xa/g6ieNyuVyurq4uVXVXYG2mf77K1Brn9E8tHo8QQlMUIcTB3mHe3LmzZ8++Hnz93NlzyY+TNVjYe0dbDV+14zuXu2u8VoY2ohmjlGGfZTsWDkzZMsD3Sn3nQVeF1dxr3bYZbcKWjfkp4gPol3yXQwEAAAAfJ2Rh9XE35q6Pj4+np+fXkyZ5uLsfOnT40qVLH0cjAADAKzznsX7+Q/hlAx9GmX5m5+HED3vNRHUwRQh5y2StbMdZaQ8ah8vl8vgC3dIgjVOauKkzNS6Hw+frWliYlyZxfD6Xy9XW1maxqnwEQ9lM7XWyJpXJ5KXLqamb0couqaZuVSvd7eVb6nF1ZR9xQNE0RYgWjzegf/9PPx305MmTM2fOXr9+XSKp0+Tmup3f1Wd/4CTB5aUTfS/lMpo8MqVr3dbJxjDhXT4/th4oLat2HVqaZrMpQhp8uAQ27drZCGJK0953dy8axptDAQAAAFAzZGH1pFQqL1y4cPPmza/GfTVr1szBgz/dFRBwP/6+pusCAGgoFLcgITiilbODrYU+j0gLMx7fCzm1b/uh8Cwk/4SQMpnaW9LS0uJwOLq6ui+XSNPhcNjaOjo8Ho/D5vD5uhwOV0uLp6WlzeGwdXV0uTwOX8DX1tHmsrnaOto8XumCa1UdX6VSFRcXq6stKi6SyxQSSYlEIpXLZUVFReqZoSXFJTK5rLi4RCaTyuTyIrHYyqpZxUOxOWxCSPPmzWfPnjV9+rTga8Hpb7FWgO7wbVGb+vHKv6zMOPhtr7WxFEVR6hCOEEJYTt9s2zyBf3rWN78mVBnF6g7fFrWpV9IWr5E7k8vuRJl9eeDqCre76/tMOpiuIm8e+d2g9RwHfjXZq3/PDi3M9bmKwhdPEu7euhy0/9h/z99N59I7vah63ItGQOC188bmil9fpeSRPw4duz+tvt/MPpRBAAAAgMohC3srIpEoYFfA+bPnvadN3bhhw83Qm3v27snKytZ0XQAAb48pjAj0mRSo6TI+furF+0Ui0dsfqq7TP42MjMo2qZWd/llN+w9N0+pzDfD0ZLHo0lNzqlyIrV6kUVvHuGx9fU6D5u0cLPO5DZD1lDtyw6P0Ok7d9L/FvSxe9Slxjaydeli3c9ZNOBf2XPouOqve6UW9u3vx4cIgAAAAfNiQhTWAZ8+frVix0s2t2/Tp3r/99tuxY8ePHT2K59ADAMB71iDTPymK0tXV7dbNbf78+dXsxqhUFEUYhlGvm6ajq1OvsyniKrRxfdhoSy+/X317GzI50Qd2Bvx5LTo5W8Y1tG7b2eNTx8ybhY1wiuHHSnRipuuJ0o/ZrotP//2N4Pi0vt+H4MczAAAAILSmC/h4RESEz5gx8499+0aNGrk7cHe//v2qWVYZAACgcWIYRiwWKxSVx1NKpYJhGIVCcS8ubndg4PZff1W/XlDQsE9UYDnMPJr0IHSDR5l2M1Zrn5OxTxLinyTEJ4cs71nPRrTKjqxl3dd77cEzwbGxsUmxEbevBv29c623m8Hrf8Upne7fBV78N+xBfEzczVMHV3/lYlj5P/E6PWcs7GNEcoKXfjV5xcF/76eLpHKpKCs54vwfa/53IVNFCCGUcZ+l+4JCwiIT78cmRF07t+t7rza6Lw9H6Tt/ueJ/e05funEv9u6je2FhZ1YPNqIIIbRRp/HLdp67EfYwLurO5UPbZrmbv/4hrtxFUcafTPsl4NCFq//G3o15dL+SmqutoRYjVv5esLssvfToYZj/IN0y+7SdfyY2Oeyn/tpvHK76y6+x8pqGom4oXfth834Junrrwb2oO1cO+8/ubc0hhBAdl3nnY+/FH/m2zcuL5rWb8U9M7J3fPrd+tb5fw3xBAgAAgAagL6whyeXyUydPhYaEjh8/bt7cuQM9PXftCnjy5Imm6wIAAKgbLS0tpUrFokszBoVCwWazi4rEERGRYeHhtyNvqxfOd/dw12iZDUHL0Tsg0NfN8OV6WLrG1q2NrVtx7+zdG1FQmghSPNtOXUr3N7H7ZOwPzq21vSbuTSz/lEWtHsMHmNGy6D0/H0+t+gmMCgM719bWXEIIIXzztn0m/dzeTP7ZwtM5DCG0WY8vJg5p9/LnM4GpGUcqZii9nsv2b5vsoFW6mr2t8xBbQgipYvEx2qij5/Derw5CdCvWXF0N9aCIC/kvb6JX1x4duRf/U6+iR5u7urWgpaHhd8o9YqHay6+x8joORbV0OszZs3uui0D9Va5l02n4d/7OVj6fLbuRH71zcUCPv2bP3OB968sdD6Ra7Wet825fFLxwVdDzj6eJEQAAoOlCX1jDy8vL27Zt+/z5C1hstr//1gULFhjo62u6KAAAgDrgcbk0RSmVSkJI2vO0Y8ePz5+34Msvv9q0aXNoSGjDPUGS3X7uqUcJ8U9e/nl09fsuVf2eTpno/1nHlm2cWrZxsvNYe6vyuW7lD/gkIf5xyIpPuFUVwLKbsGKBm4Hs8ZmVEz917tDRvkM39xU3isslQoz4+vox7m6dHdp3cx/vdzlTpdtp/HjXCo1ALJt2rfm08sm/oWnVBCaM6ObPk0b26NrZvm3Htt2HTQmMlRj3/bxPmd4nRnR55bAuLs72HXt4jN4aLme3/9Z3kj1XGHNg7hf9nNq7dOo3fm5gZE71C78zwvNLBnbq2NHOqZKaa66hehXuheT2tRv5xNSjV6eXo8J36dqerYgPjyo3MbQWl19N5fUaisqxWk9aPsdZJ+uG/5Qhnzg6de72xbJjj5XWI+eMt2cRIrkXsGzbXabD9DWzOxq5zloz3VF4bt26U5llzlSrL0gAAABojNAX9q4kJSUtXrT4E/dPpk6ZEhCw6++jx06ePCnHImIAAPAhYLE5sbGxYf+FhUeEv3iRpely3hlWqyFDnbjKh1vnLdufoP43WpydKy7fHMXIs5IT0wrlhJC024d/OjC476J2jo6mdET6GyEMpSvQpYgqP6+g2myGogw6jF38Q/d2zS2NOEUZOSoWYZtbmtAkrzRBYxT5ac9zi+WEyNNThITlNGhAC1oatWXBxpPPVYQQkhZz+uClsV93danmLIxSlJ0llCoJEVdSc4011FVJxLnreV4j+3q23xwZrSCE28HNWVv56N/QjPJjUfPlV105q019hqJSrDbDhzmyhVd+XBQQXMgQQrLuBa32dx+0pX9PN5MdSS9UssTdP2zr9deiGf5HRptY5p6et/pCNh6iCwAA8HFAFvYOMQwTGhIaGXn7cy+v8RPGDxzoGbg7MCIyUtN1AQAA1CAo6ERQ0Ima93tbDb52fiUHpMy+PHB1hVulu7ObO7RgqZ7duv6o9r+sUqYnPy1inPj8CgtsMcVFJYTQ+gb6NMmq4poovV7L/gj8qjmn9M08WxtCiJKmq/6RjGPVohmten7ndoVYqZ4116OGmpX8d+pyxsgxgz7t+HP0HTnLoaebEZMSdP3xm+NQ51O/WXkDDMVLXJtW1jStPWhbxKBtb57RytqSJi9UhCiSD/lu6X16WXez7NOz/K7l4skHAAAAHwvMkXznpBLJ4cOHvb2nJSQkrly18scf19na2mq6KAAAACCE5rBpQqp6UkAVGJlMxlAUXWFCoTIt6UkJw2rZvatpVT9gUUYDJo+yZeWHb5/9eY/OzvbtunT/Liiz+tNTLJoQUsn56llzfWqoBcntoH+ekGYDh3ThEVaLnh625Pn1Gw/KRWF1P/Ubo90QQ/HyuEwV0RbF0+aVHp8yaNPeRocQytilr3OtJ5ACAABAo4cs7D3Jyc7evHnzkiVLDfQNtm3znzZ9mq6ubs1vAwAAAMIoFAqG6OjoNHQcIc9Mz1HRtl3crBrkJ6Li/66GCRled2+fgVWkYbSJhQWXFIce2HblYaZYrlSW5GYLa1j3XZaa/FxF2/bsY9cwjyqsXQ0sVuWtWlXfC8WDo3/HKMwHjOohsPzEvQ317MLFuHJPEKjP5ZfVgEMhT3uaplJln5ji4qRe8+vlnw49VoXLCSGE1Xz02vUjjBOO7rmYZTl63Sovq1ePkHxnX5AAAADwXiALe69iY2O/8/HZ6u/fp0/vwMDAEZ+NoGncAgAAgOoxWZnZDMvSc7RnSz6bpWVk18XpdS7xNhQPLl/LUPFc/m/zouFO5nwuz7B5Vy/PtlUutV9TnfkXdgbGS2mrEVv+3LFoVFc7Ex02zeIKzFp3GzZj3qi2LKLKzcqSE+1uXuM7W/HZFKE5fL5WDVMTlQmnzzyQs9vN3r7B28POSItFs7T0TQzrncPUWINMrmAoA9c+3a11Ko5yNfdClXrm6PUi40Fjxozs15719PLZ8lFYvS6/rAYcCmXCxcuPVSbDV238tn87Cz0ui2ZpGVo79enaXF0Pp/XkTb4enKitc9f8b/H3h5L1+670G2fHrnEQAAAA4AOA9cLeN5VKde3qtYjwiNGjv5jyzTcD+vfftSsgPj6+qv0N9PULCgvfZ4UAAACNjDL1xrX7Ph06em265kUIIUQe89OQibtT334pc0lkwKZT/TeN7DTJ/8SkMq+XD3FqS/5wp8/3Zjt+HN/WY5afx6yymxTx9MlTD55c+/vqHI+h/VYc7rfi9TZlYnUHVSbuW7Wp525ft0FLAwctLbOhDh1VZTC51degfP7gYQHj6Dhp50WzhV3/73K5Yqq5F0zulQNn/2/AmNmzGdbDX8/erzD5saZT16gBh0IRt+fHvX13envOD/Sc/+pVefRGy0dwmQAAIABJREFUz3H7Ulitp66d6aoMW/nDoSQ5Q8K3Lgp0+2u6j9/k/74KfKR4h1+QAAAA8D6gKUkzxGLx77//MWfOd3n5+Rs3bli5coW5uVnF3dw93Nev/wmzKQEAoIlTJu3zWfR7cFJOsVKpKM59HP0om2qY6Wmq7MuLx8/ceCLyca5EoZDkPo44eeV+MUNUTD1zDWX6lRVjvb5ed+DCnSdZQolSqSwRvki+G3Js95Gb+SrC5J1f5r1gT3BculCqVCqkRflZzxPvhoc9KqxuZfaSB7u9v/xm07HQhBdCqVKpkIhynt2PuHL8xuP6PKC6phqKb2yZ/+uVuExR2vMMWcULrO5elIQdOvZAxeUpoo+erOyRCPW7/HczFIwo0m/CuLk7zoQlZQklSqW8KCcl9sbtZzJC23yxeFZHZfi2dUdS1Rchubtr7e/JHNfpi0c3o2saBAAAAGjsKH1jU03X0NQ5OztPnzHdwtz81KlTR478KZFI1K9zudzAPYHGRkZ378auWLFCoajv76gBGsgSX193D3dCyIRJ32i6Fvjw+MyZ5ebWlRAydOgwTdcCDcbdw32Jry8hxH/7joiIj+NByZTpmICQNV3Cl/effDQPTw6sM7bz0nP7xyUv7z3rJB68WKOD+38nhISGhK7389N0LQAAAE0I5khqXkxMzHdzvhsydMjECRP69u37x759wdeCGYb54ovPDQ0MCCEdOrSfM2f2li1bNV0pQCmfObNq3gngTfb29pouAaAStFnnwZ1IUvyT9JxCCduwVecRC2d246qSo+/VulMJCKVnaqbMz5HrWH/y7YIx1vmXNlxDjggAAACNFrKwRkGhUJw6eer69etjx46dN3fusGFD//rzrzGjR6tX1qdpesCAAc+fPz927LimKwUghBB1dw8AwEeA5zzWz38Iv+z8NkaZfmbn4cRKZvhB5WjLz7ecW9GFQwghjDIneOX/rosQhQEAAECjhfXCGhFhoTBgV8CCBQtVSmbJkiVlHzFJUdTkyZN79+6twfIAAAA+OhS3ICE44uGzvGK5UikvzkuNu3FovffnvpeysAx67VH6PKagWKEozk4M/n3JuPlBzxAkAgAAQCOGvrBGJzExcc/ePT9v3EhVWIR1/vx5WVkvHjx4qJHCANb7+RGsZwIAHxWmMCLQZ1Kgpsv4wCkf/Dahz2+argIAAACglpCFNTo0Tc+cMUOlUrFYrLKvUxRF0/Tq1avn/t/c9IyMz0Z+1s6xraaKfA+wiCwAAAAAAAAANDhkYY1O/wH9W9nZVfpcbpqmeVq81WvXzJs7r51jW/UT/T5aiMIAAAAAAAAAoKFhvbDGRVtbe8rkb6rZgc1im5maLVu+rOIMSgAAAAAAAAAAqB76whoXQ0PDS5cvO7R2sLez09XVJYTIZXIWm1V2HX02m+XUrl12To760wmTqsvOPjg+c2bhGYUAAAAAAAAA8I4gC2tc0tPTf//9d/XHJiYmrVq1srOzs7Nr5eDgYGJiQghRyBUURVhstrmZmUYrBQAAAAAAAAD48CALa7xycnJycnIiIiLUn+ry+S1btLSzb2XXslXrNq2tra0xTRIAAN4RmqZVKpWmqwAAAAAAaHjIwj4YRWJxXNy9uLh76k9/WLq05yc9CSFsNluhUGi0NAAA+Njs378vNzcvIeHho+Tk5KTklNQU/FsDAAAAAB8HZGEfqle/rsd/TgAAoMEVFhba29u1bNWCJhRF00qlMi0t7f79B48ePUpOTn765IlMLtd0jQAAAAAA9YEsDAAAAMpLTUm1bd6cRbPUn7JYLFtbW2vrZp4DPVk0rVKp0tPTHzxMYBjMowQAAACADwyyMAAAACgvLT1NqVDQHE7ZF+mX0RhN09bW1lbNrGiq9DHH+nr677tEAAAAAIB6oTVdAAAAADQuHDZHKpOy2axq9mFUDEWo7Jxs9aeFwsL3UhoAAAAAwNtCXxgAAEDTxWazTUxMLCwsLCwsLC0tbG1tbW1tzczMaLrq35YxhCFMZlbm4cNH5HK57/ffv8d6AQAAAADeFrIwAACAJoHD4Zibm1tZWVk1s7KysrKytLKysjI1NVHHXnl5eenpaelpGfHx9zMy0guEwo1+fhWOwagYUpiff/DQ4cuXLyuVSncPd/WGwYMGdnfr+n4vCAAAAACgPpCFAQAAfGzKdXtZmFvYNrdt1qwZi8UihIjF4szMzMyMzJCQf1NSU1NTUtPS0kpKSsodpKREoq2t9epTpVJZXFJy7Oixk/+clCvKP0TSwcH+XV8UAAAAAECDQBYGAADwAWOxWKampmVjLwtLC9vmzbkcDikTe0VERKhjr/T09OLi4tocOSMzo1XLloQQlVIpkUgOH/nr7NkzMpns3V4PAAAAAMA7hiwMAADgg2FkZGRra/tG7GVry+VySZnYKyYm5vz5C6nPUlOephQVFdX7XM9SUlu1bCmRSI4fPx4U9E/FxjFCSGhI6NCQYfW/HgAAAACA9w5ZGAAAQGPE5/Ntm9va2ti+ir1sbGx4PB6pEHtlZmampKTk5+c3bAGPnzzOzsk+duy4SCRq2CMDAAAAAGgQsjAAAAANq1PslZqampeX9x6qOnbs+Hs4CwAAAADAe4YsrGnx2nlncz9eFRvlkT8OHbs/TVXPY7Ocvtm2eQL/9Kxvfk1Q1rdAAICPHJ/Pt7CwsG1u29zWtjT2srbmaWkRQorE4ow3Yy81TZcMAAAAAPBRQRYGDYU2aN7OwTKfS2m6EACAxqFi7GVtba2lpUUIkcnlmRkZqSmpZWOvFy9eMAyj6aoBAAAAAD5yyMKalhMzXU+Ufsh2XXz6728Ex6f1/T5ErtGiPn6Ojm1GjRyl6SoAoCHdf/jg5D8nX32qjr0sLC1sbWyb29paWFo0a9ZMW1ubECJXyHNzclNTEXsBAAAAADQKyMKgPErXfui0Wd8O6+5oxit5kRAatGtjwI3ncqLjMu/4vim28Vu8Ju1JkBNCCK/djL8Oz7INWz1i9vHn6mmRrNY+J2N9CCGEqLL+nNhv7S3kbISYmJq6e7hrugoAaEjmFuYG+gZWVlZWVpbNrKzUkxylEkl6ekZaenp0dPSZs2cz0jPS0tIafEl7AAAAAAB4G8jC4E06Hebs2T3XRUATQgjRsuk0/Dt/Zyufz5bdyI/euTigx1+zZ27wvvXljgdSrfaz1nm3LwpeuCroOdYHA4Amxs7OTiAQZGZm3ouLO3v2nLrbKysrS6Wq76KLAAAAAADwXiALg7JYrSctn+Osk3XDf+mGv26lSPTbDl60YfnnI+eM/yN0e5LkXsCybe6HFk5fMzt0+vUBa6Y7Cs8tWHcqs8x//JSJ/l5f/O8hsrHK+W/fERERqekqAOCtHNz/OyHk1s1b6/38NF0LAAAAAADUGa3pAqAxYbUZPsyRLbzy46KA4OQCqUKSdS9otX+wmOXQ082EJoTIEnf/sC1C7jjD/8ivU+xzT69bfSEbLRAAAAAAAAAA8KFAXxiUwbVpZU3T2oO2RQza9sYGpZW1JU1eqAhRJB/y3dL79LLuZtmnZ/ldy8XSzwAAAAAAAADw4UBfGJRR5VPNKJ42jyr90KBNexsdQihjl77OhtT7Kw4AAAAAAAAA4G2hLwzKkKc9TVOp9E9OHbg8uLjSPVjNR69dP8I44eieVI/Jo9etihw973i6enUwRqFQMERHR6dh8jFXV5fCAqFQLBIKhVKJpEGO+S54T53K4rBOnTydnp6u6VoAAAAAAAAAoAbIwqAMZcLFy4+nzxi+auNTesfZyEfZYiVH39Kuk6U4NDJFQQin9eRNvh6cqJ/nrjmQ6cqx2zNhpd+4mCkHkhWEECYrM5thOXmO9jyceDlVodeivWVJdHx6fZfRX7t27auP5Qq5SCgSCkUikVAoFAmFhUKhSCQSCUVC0csXRWKRSCiqsrPtnbGwsOjWvduwocPuRN05ERR09+7d918DAAAAAAAAANQSsjAoSxG358e9fXd6e84P9Jz/6lV59EbPcftSWK2nrp3pqgxb+cOhJDlDwrcuCnT7a7qP3+T/vgp8pCDK1BvX7vt06Oi16ZqX+m0xPw2ZuDu1nmvrjxrlxefz+QI+n883MjIyMjLi6/L5Ar6ALzA2Nm7evDmfX7qp7LtkcrlYJBKLxXl5eXl5+SKxSCwSi4vEYrFYLCoSF4nycvNyc3LlCnn9B+lNhoaGFEURQpxdnF07u2a9ePHPqVMXL1yUSqUNdQoAAAAAAAAAaCjIwuANjCjSb8K4+CnffuXp1s7GWJclyU9Pjrn9TEZomy8Wz+qoDF+/7kiqutdLcnfX2t8H/DFt+uLR52ccSVMpk/b5LNJb+d2Ibq0MudKC1LhH2VT950vKZLK8vLy8vLzqd+Nyua8iM76ugC/Q5fP5fF2+sbGRkZGRhYW5OjIzMDCg6der472KzF4qysvLzc3LexWZicVisUhcUFCgUtWQ5enp66k/YLFYhBBzcwvvqVO/njTp4qVLJ04E5WRn13sEAAAAAAAAAKDBIQtrshR3Ng6231jJBkaUeGrr96e2VthwZGqHI2++UhKzYZjzhtefy1Iubp5ycXPDFlq9WkZmNE3rCQR6enp8PYGAL9DTEwgEAn19fT2Bnp6enpmZqb2dnUBPIBAI2OzXfykUCsXrqZhCkVAkVBOJ1BM2RSKhSCAQvHEmitAUraWlNXTIkBHDh0fdjvrr77/fxYUDAAAAAAAAQD0gC4MmQaVSFRQWFhQW1rhnuUYzIyNDIyMj9dxMQ0NDGxsbdaPZq6mRVTWOqTM1F1eXLl27ZGRmNuzlAAAAAAAAAED9IAsDeENt52ZyOHyBwMzMdPPm6vrg1BMnLS0s1J+2drCPiIhsqFIBAAAAAAAAoK7omncBgApkcnleXl5+QUFVOzAMo1AoCCGSkpKkxCT1i4lJj95TfQAAAAAAAABQGfSFAdSfvp5+uVfkcjmbzWEYVUpKSlRUVHR0TFxcXPce3Zf4+mqkQgAAAAAAAAAoC1kYQP0JBHyiXjKMomiKykjPCA8Pj4mJiY2Lk0okmq6usWOZ95y+cOZo93bW+rQk9+nZn7yXnM9hNFcPZdhn2Y6FA1O2DPC9ItVcGQAAAAAAAPBOIQsDqD+BQC+/oCD6zp3oO9HRMTH5+fmarqhh8Vx99gdOElxeOtH3Um4Dp1Rcp//btX1OWx5FCCGEb2rBk4o1GIQRQigtq3YdWppmsymNlgEAAAAAAADvFNYLA6i/W7duTRg/YfPmX64FB7/DIIxlP+fE3cexf89pw6lss26PZecfPbyzZ6SgNsdy+mbHhav7Z7dh1ebMFEVRFE2/g3CI123MV224sqS/vxvm7tjOuWOfsRsj0IwFAAAAAAAA7xyyMID6k8lk7+M0tIm5KU3x2k6dN8yiwl9ZlsNXi0fbsCiWoYlRLfIt2qB5OwdLAbdW8ZY0ausYl86fLrrY0E1hhDa3t9OnpDf3bj2blC9VykWZKRkabgsDAAAAAACAJgFZGECjxzMx16dkBUKWxzRvV603NlEGA2d83UFaUKCijIwMP5zJfRRPi0sxJTk5RQjA6oHmCUzNTQ11MMm9PjB6AAAAAABNHLIwgMaONjIxplVZ53YeeGQ5ZuZwqzJ/a9kOY2cN5P23IzBMShuaGKi3UMZ9lu4LCgmLTLwfmxB17dyu773a6L4Rk7Fa+5yMfZIQ/yQhPjlkeU8OIYTSd/5yxf/2nL50417s3Uf3wsLOrB5sxHKYeTTpQegGDw4hRMdl3vnYe/FHvn01U5PXbsY/MbF3fvvcutKGNO0WA2dvOHoxJP7enXv/Bv2xarybadn9KEIbjtkdoy7jSdwfkyxfXRirw7zTjx5G7hz2atYnbTFub8KDkA29eGX2OfXoQcjG3lo1navSS6MIIbRRp/HLdp67EfYwLurO5UPbZrmb1/gdUcu6r/fag2eCY2Njk2Ijbl8N+nvnWm83A6rqE1V7OyjjT6b9EnDowtV/Y+/GPLofdefKwV+mdG/T+fPvf9l/9WZEQnzUnUv7/MZ3MHj5Btqoy/QtJ25H/Rfx7/WoO7fvXvr5c6vKiq5h8AnRth0w86c/z9+IuxcdH3Ht0oFVI5uzqt3E+WTV9eT7QfMcywzsqB0JCdF/fGFU9eXTVQ0+pWs/bN4vQVdvPbgXdefKYf/Zva05lY5JTNzNUwdXf+VSLumtuv6qj1zr0QMAAAAAgI8afjEO0NhRBkaGNFOQFb5/77/jfvpmisupdVFSQgih9PpPG+eYc3pyUMLgqYyWkbEuRWQMIQoDO9fW1lxCCCF887Z9Jv3c3kz+2cLT1T2kkTbr8cXEIe1efkcQmJpxpOI39iiO3rk4oMdfs2du8L715Y4HUq32s9Z5ty8KXrgq6LmywvG02k0PCFzspl+aNJi37v3Vkp69Oi2YsOR0esW9y1EmhEVkTxvdycWRcyZSTggh2i5d2nFoHWeXVqx/HygJIbSJs7MNXfLvzWhpTeeq9NIYSq/nsv3bJjtoqSMWnq3zEFtCCKlu0TItR++AQF83w5cLqOkaW7c2tm7FvbN3b0SBsooTEe1qbgdt1NFzeO9Xb+EY2riM+n7PqDLn5Dbv8uWynUZFn8/454WKthjtt21xbz1KUZT7QkzxjQ3MONJCVZ0Hn+fovWuPb7fS8JRwze2dbfglqmo31ZgZVXr5VOVjotNhzp7dc10E6oNq2XQa/p2/s5XPZ8tu5DPlxoTomth9MvYH59baXhP3JipIDfVXc2SqdqMHAAAAAAAfO/xKHKCxow2MDChGVCjMuvDHsbRmX3wz0IQihBCW7aipnvx7hw6GiYWFIoY2NDKmCSGEEd38edLIHl0727ft2Lb7sCmBsRLjvp/3KdNXo0z0/6xjyzZOLds42XmsvSV/+TojurxyWBcXZ/uOPTxGbw2XlytEci9g2ba7TIfpa2Z3NHKdtWa6o/DcunWnMiumCSz7icvnddWTPDi2eEw/p/YuLgO911/LoKwGr1rs+boOVf7fU53VZbRsP3l/xuvjyGL/ixDTpp27tFL3+nCcurvqUIRu2cW1tHVL16W7E0ceFxYupmt1rvKXxm7/re8ke64w5sDcL/o5tXfp1G/83MDInOqCEZbdhBUL3Axkj8+snPipc4eO9h26ua+4UVwuYKwwhjXfDkZ4fsnATh072ndwH/rDuedKRlUQuX3OFz06O7d29ZywI0pIDHp79TOjCSXoNrCbQBW3a1SPHl169evc2a37qE2hxXUdfFbL8cvnu+lLEv9ZNnGwa0fntl37fTppw8UcptpNtVPpl1D5F1mtJy2f46yTdcN/ypBPHJ06d/ti2bHHSuuRc8bbv2w6ezkmdk7d3Mf7Xc5U6XYaP95V3d9VXf3VHLl2owcAAAAAAB8/ZGEAjR1PX1+HVolFRSppzP6D0dw+k75yYBGi5TZpnHPx1cBjT5WkWCRmKAOj0pl0FGXQYexPe4/fDI+MDd6/apAVi7DNLU1q/tvOKPLTnucWy5VSYXrKi0qW8pIl7v5hW4TccYb/kV+n2OeeXrf6QnYl8RHLYcRnTlz5vW0L1hy9+6JYLitIuRWwcOXfGYxhnxH9a7OsWfHt4Mhill23buY0IYTl0K27iTA+Po1u381NQBFCeJ16dNVVxv97K5uq3bnKXRrdZtCAFrQ0asuCjSfvvSiWy4RpMacPXnpU2rLGcpxz4pF68mZC/JP7pxY6sQir1ZChTlzlw9/mLdsf8axQplTKxNm5FRb8rziGNd4ORinKzhJKlUpZ/v2gHYfilRQ75/7NB5liubwo/WbAnsuFDMumhS1NCMMwhFCmjm6OJloUIYw0+8nzgnIV1Dj4rFbDhrfnyWP9fVYcikjNl8olwheJ0YnZKlLdplqq9EuowuAPH+bIFl75cVFAcHKBVCHJuhe02j9YzHLo6fZyWF6OiUohTrt9+KcDcQqWsaOjKU2qLZJV7ZFrM3oAAAAAANAEIAsDaOwE+gKakRUVyQhRPfvn4MXCNuMm9uCbDpoywuJZ0KErBQxRFYmLVLRAT48ihNLrteyP/UvH9u3QwlyPx9E2srUx4VGEphtmQrQi+ZDvlgiJhbVZ/rk1ftcqf8Akt7m9Na16Fn7zaZnpkEV3QqIlhNvc3qYW33aYwpvXoiWc9r27G1CEZdOzZ8uS8MAdIVk8116dtQlhd/DobqRKvHb9ubJ+5+JYtWhGq57fuZ1R65iH3dyhBUv17Nb1R+X75apT19uhzEhJVxCeqfnL+X9Elvk8i6G0dbQpwohuBV3Nocx7L91/JebWmWM7V3832EG3XLRY44CUXkjErdQKk1Wr2dSAuDatrGlaf9C2iLgnLwPHe1s/FVC0lbVlZTdMmZ78tIih+Hxdqvoiqz0yVZvRAwAAAACAJgDrhQE0dgI9AcVIiiUMIYQR3vj9RMqw8ZMXMka9uTHrj8TKCCFMSbGEobQEAg6h9AZMHmXLyg/fvnzjobDk7BK2Sf8f/tkyosGqoQzatLfRIYQydunrbHgpOK+yNOztAwYmNzT4jvQTt77d9U9GefRqI7/957+38nvkj+nVqxPvhrCvhyVJPnX1iZJw63UuikUTQii6ijcrH273st/+5mvaHDZNiEJRp5SIMqrr7VDJZQpCcTicV6XJ5QqGUBRNCGFyzi6dKI4ePbh7J1eXDq59W3bu09eR9ppztuwkxpoGhKJpihCm0vtW9SZCGKIihKel9dY3l6niBITiafMqPTojk8kYqvR2VVNk9UeuYvRmn82p12UAAAAAAMCHCn1hAI2dnh6fYqTFJer/5svj/jwSSXf/+ksHcfCRf56r25pkJRKGoXQFfIo2sbDgkuLQA9uuPMwUy5XKktxsYZn14BmFQsEQHR2d+iUarOaj164fYZxwdM/FLMvR61Z5WVX2CElZSvJzFW3T7ZPmZbbqunq4aBFZ6uPntWrFUr24dua2RKe7Z2+7Pp4dmchrN/OLb1+5KTTt3a9rh34DWjAJFy4lKut7Lllq8nMVbduzjx2n8h0qkmem56ho2y5udXnyYE23o+4kz24c+MV39tcDPXoPWXEpgzHqM8hNp+wONQ6IPO1pmoq2dethU+HeVbOJqESFYoY2b2Ov/7ZhmDztaZpKlX1iiotT6WpxpX869FhVYZG6Kt5eTf3VHbmy0XvLqwEAAAAAgA8O+sI+eD5zZmm6hIZkb2+v6RIaHR2+DkVyJdLSlhdV+tkD12Z1+7To5OHg/JevSYolDC3Q49Oq3KwsOWndzWt854SjdzPEKjafr8V+/XhEJiszm2E5eY72PJx4OVWh16K9ZUl0fM2PdiSEEMJpPXmTrwcn6ue5aw5kunLs9kxY6TcuZsqBZMWb+ykTT5267z2/w3e/LM9ZsfPc/XxOsy5ffr96jCVVcPH0lUo7ySpisq+cjVjW033KKpvWJGr19VyGMLcuhBQOGzDPV2rHPNx84bGy3udSJpw+82DaXKfZ2zdI1v56PPJpgZyjb2JYXUKoeHD5WsbkSS7/t3lR9uo/gpMKOJYdB3m25VZ7ETXdjjpitew3smVO2O2HGSIlh60QiaSEUNSbnWA1DgiTcOFS8vSZnf5v25ridbvP3n0mVGqbtrLXz4lNyK1mk/LxvQdCppX7jCXjkn8Ois2Wcvimhtr1ycWUCRcvP54+Y/iqjU/pHWcjH2WLlRx9S7tOluLQyBRFLd5eTf3VHbmK0QMAAAAAgKYGWdgHz82tq6ZLgHdLR1ebYqTFJS8/ZwrPz3e3m192F6ZEImUoXT0+YZ5c+/vqHI+h/VYc7rfi9Q7KxJcfpN64dt+nQ0evTde8CCGEyGN+GjJxd2ot6uC0nrp2pqsybOUPh5LkDAnfuijQ7a/pPn6T//sq8NGbEYYy6cCaLb0CF3Ud/fPR0T+/LFKedn7Vhou1jMIIYXKvBF1d7DGis2PxjRVXcxhCSFHYhav5w0a7UCVh+06VrodVv3MpE/et2tRzt6/boKWBg5aW2VB1SiWJDNh0qv+mkZ0m+Z+YVOb16tIbJrf621E3lHG3b1cv71m2lU2Vd/5SZNEbe9U4IIr4Pet29doxq/3ItftHri3dXnTap5fPJUk1m4pCDhx4MOA7p8Hr/hy87vXpZHW/DkXcnh/39t3p7Tk/0PP117E8eqPnuH0pNXcNVld/NUdOrWL06l4/AAAAAAB82DBHEqCx09VhUUxJsbSaDIkpKSkhFF9PQBMm7/wy7wV7guPShVKlUiEtys96nng3POxRofr9yqR9Pot+D07KKVYqFcW5j6MfZdeqN4a2+WLxrI7K8G3rjpSuWS65u2vt78kc1+mLRzer8J2k5P5v3uNmbTt7OyWvRC4rykr898j6CV/6nqplB5r6qoQhf57NUDLikNPBpQtiFYefvJylVImu/30+/VVoUr9zlTzY7f3lN5uOhSa8EEqVSoVElPPsfsSV4zceVzVNT5V9efH4mRtPRD7OlSgUktzHESev3C9miIqpOr+p6XbUCUVl3Lkem5JXolCplCX5qbFXAr7/dtGZ7PKHqmlAGHHU5q8n+Ow4d/tpbpFMKS/Oe3Y/KlnIoardRKRx/tOm/3g88kmeRKlSKiSinLSkO/+ev/GopK7Xwogi/SaMm7vjTFhSllCiVMqLclJib9x+Vstcrbr6qz5yVaNXx9oBAAAAAOCDR+kbm2q6BoCPnLuH+xJfX0KI//YdERHoQ/loUKZjAkLWdAlf3n/y0Vq3u8GH7+D+3wkhoSGh6/38NF0LAAAAAADUGeZIAgDUCm3WeXAnkhT/JD2nUMI2bNV5xMKZ3biq5Oh79WnyAgAAAAAAAI1AFgYAUCs857F+/kP4ZWeUMsr0MzsPJ9Zh4icAAAAAAABoFrIwAIDaoLgFCcERrZwdbC30eURamPH4XsipfdsPhWfVvNw7AAAAAAAANBbIwgAAaoMpjAj0mRSo6TIAAAAAAADgreA5kgAAAAAAAAAA0FS8jJxtAAAA/UlEQVQgCwMAAAAAAAAAgKYCWRgAAAAAAAAAADQVyMIAAAAAAAAAAKCpQBYGAAAAAAAAAABNBbIwAAAAAAAAAABoKpCFAQAAAAAAAABAU4EsDAAAAAAAAAAAmgpkYQAAAAAAAAAA0FQgCwMAAAAAAAAAgKYCWRgAAAAAAAAAADQVyMIAAAAAAAAAAKCpYGu6AIAmZPCggd3dumq6CgAAAAAAAICmC1kYwPvj4GCv6RIAAAAAAAAAmjTMkQQAAAAAAAAAgKaC0jc21XQNAAAAAAAAAAAA7wP6wgAAAAAAAAAAoKlAFgYAAAAAAAAAAE0FsjAAAAAAAAAAAGgq/h/m4fJzSaVQ6gAAAABJRU5ErkJggg== ) ## Delete a blueprint from your personal repository ``` w.delete(user_blueprint_id) ``` ``` Blueprints deleted. ``` ## Retrieve Leaderboard blueprints via the Workshop ``` project_id = '5eb9656901f6bb026828f14e' project = dr.Project.get(project_id) menu = project.get_blueprints() ``` ``` for bp in menu[6:9]: Visualize.show_dr_blueprint(bp) ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABJQAAADECAIAAABGCez2AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd1gURxsA8Jndu+Pg6L2DUqQoYgEUe+8NewE1SjRq0Bg19qhJjD1WjEbNF7HFjr0gimIXwU5Veoejc213vz9A6jUQBcz7e/I8wbvd2XdnZ+/mvdmdxVp6BggAAAAAAAAAQNNGNHYAAAAAAAAAAAAUg+QNAAAAAAAAAJoBSN4AAAAAAAAAoBmA5A0AAAAAAAAAmgFI3gAAAAAAAACgGYDkDQAAAAAAAACaAUjeAAAAAAAAAKAZgOQNAAAAAAAAAJoBSN4AAAAAAAAAoBmA5A0AAAAAAAAAmgFI3gAAAIAvhjQbvfvygTmd9MnGjqQ2rOE0ftv5AD9nlcaOBAAAgHSQvAEAAGgY3A4LLr38kPr28gpP9caOpYkirabs2jjGc9jMqW7auLGDqYXbauSsSV36L9/zQ3tuY8cCAABAGkjeAACgyVExcZ/w46YjV0LfRL3PSE9Nj3/3KvTSiR3LZ/S112rKH9uYIAiMMUnixklMlKo3Ts8d79LzspNPT9GTEyXbY92LjEx+ytX59ix5W2y34lFaRl7a9R9slRhJI4zGrFvWU5tJPbls+bUcRnEwpJ3ftazszOwHK9pViQKr2w2au+nErbAPySlZybFRj68d3zirlyVHzpa5Fp6Tl2w+euX+26j36anxH14/Cj6+fYVPNyvVqkuVPtu6cNcbsYrznE2zHOTtNgAAgEYCH84AANCUYK12MzfvXznCjlelM6+uZ+GgZ+HgPnDi5PaT3ebeLGm8+OQRPNs2uPW2xtl2Q9cbaWisTyCs0u67+X3/N+9aPiN1o4ajFk61Z2NE6RvpEyiWkl8m133ukgG6KP/G+vU3+VILVAxrdfhu34HlAyw4H/eTbWTTfpBNuwGjBq4ZPWnnq9Jaa+h2mrf9z58GWHMra4Zr3FLHuGX7fhO//+HW5rnf//Eghy57QxixY+3x8Semtp+zaMRR3zPZ9YwSAADAZ9KUf8IFAID/GKzhvvhE4G8j7dTovDfnNszx6tTaztjE3MTOtdMQn/mbTjwIO/PvvSaauTWmhq83Qs/YkI2p9IcP84bMm9JC+qgau/X077vEhj4TMISugcJ72LDe8NkTWrAkH47uPJNG1yWY6hs1dGxtijMe/W/ppN6O1ubG9h37fncgLJ8hdD2X/e7TskYUWKf72tNnVw+0VpFkPTuyavqgdq1aGpm2sO04yOfno89yKBWLviv/PfNrD52KxK4oZM+fTwVYd+B3k5QZTAQAAPBFwcgbAAA0EVizx6p9C9trYnH8ufmj552OE318h58a+Tg18vG1fzY1ZnxN1eeoN8LI2JBATM79Pw+L9s6e4XZwxSNBzWU0+nw72ejW2o15q7t21NM30MYoXc5AFWE2wruvFha/PH40rFZRymOyr62ePvXfpKDQdEnZK4nPTq2cru34ZH0Xbvv+vQz/el+ZGWK9Qb/8Oau1GhJEHv7W66drqeWrIGF82IU9YVfO39x2ap+3vdOs3b/e7znvcg6DEEJ0wqmAO8s8BrpOmuC695cwcf1jBQAA0OBg5A0AAJoG0vab5ZOsWUgc89fsH6pkILJXsB/7+5ELweFvY9PT0rJT4yIfXji4fJSjRrWbp9jua8MzMvNS/h6tVvVlrD3hSEatm6m41v0W7Dh1PyIqLS0lPf7ti5DAY9umtuUquwA2nhqYmpmXdmuxY8WYjVJBIlWHUYt++/PI+btPXryPT8rKTM9OiXl79/SeBQNaVrspq0HqTRmEvoEegej83Ldnj97RHjNjsG7NW9IIM69vh1IXj96Iz82nEaFvoCt3nIow6T/ETQVL3l6+pOjqSgUY/tNrFZlbGTr1WVgihTBhaGJQ5Wud5eT70ygTkhG+2vXt8srMrYIk5cqSOXteixjSeOSyGU4fGwKTHXQptIRhWQ8c2hp+4QUAgKYFkjcAAGgSWE6jx7twMFNyd+++J8XKrEFoOfca5Nm6haEml02yOBrGdp1GL9x7/cT3LvWa6Z3daubx6wFrJvdwNtdRZbO56vpWzp0H9G6lTim7QL2DxFoe0xb6Thjo6dLSRFddhU0QLBUtU6fuk1f+c+vEnNZyd6fu9aYMro6uGoGYgrz87OvHLov7+Y6zqv59yXaeMt0z49zR+8X5eQU0wjp6OnInaeG5d3FVwVTy/dC4T8vdpGLpG+gSCNE5WbmVF2SyXLxGt2JjpuDm3kOvhNJXFLw46H+7iMFsBy+vthXZG/9ByCsxYll5djaHXgIAADQp8LEMAABNAWHo7mHDQoz4+ZWb6crdESWJPLVs8uBuznbWhkamxq08vdZcSxJjTff5y0cZ1H22R/X+PyzuoYeLX/5v9sAOVqamhlbOHQZ4/7Du9CuJkgt8cpCSuIM+nR1srQ2MzEwdu43/9WaKBOt4Llk/xUL2d1U96k0JhLaeNkaMuLBIwBTfPRaY3WHqlHbsKguodZkx2Tbm9MlwMVNcWMQgQktXW97IG8vOtbUqZkRvX0Q3/HWIhNFI74G6BFP8KPBqakUdECYdO1qxECN6di04V+b1nEx28PUwEYNYlm5uJh+rmc549SqTQmwnVyd44hsAADQpkLwBAEBTQFpaW5AIMdkx0TlKTvHHFL65fe1JVAq/RERJBDmxwXu+X3Exj8bqHt3a17nPTZo52qsTSBx+dPvJZ0n5IomoOCsu7Prhs88LGOUW+PQgmZL0D/HpeSViSlySFXV9x7zlF/k0VvUY0ttQZjJaj3pTAqGjp0sgpqSoGCEkenryzHursT7dKi48xboDvEfoRZw4HU0hpriomGEwS0dPS07GrGJpbUIiOishudZskAghxOm7/R0/OzOv5n9pT1e3Z0tboQp2y8lb1g3UwcI3e389nliZv5JmlmYkQnRGXJy8I8TkxsZl0wiRZtZmFeknlZyYQiGsYmFtDN0EAABoSuBTGQAAmgKsyuNihOiS4pJ65yBM4esXcRTCaoaGmnUdeqP5WdkUg9jtxk3rrCdtDEnhAg0eJJN391aYiMEsGwdbmfdeNUS91UZo6WoRZXkZQkj8+szJt3ojpgwov/GNMPPy7st9eOpcIo0QooqKShhEaOtoyxkf1NTRYWFE83P4DTY6iBBCiGU+bPO/G/obIv69X+Zsfl51JhSsqsbFCDGlCmqGKS0uZhDCXFXViuNRHiihrasD3QQAAGhK4FMZAACaAkZQKmQQwqpqqlJTGsJq7uXkvOy052s7VgzFsAzaT1rhfy4o9FXk+4zU+LjwkH+m25IIYRaLrGvyxmRdPHAmUYx4HeZfeBp6dtuC8Z3M1XBdFpDhE4JkitPT8hlEaGvLHtSqT70phjW0NDBiSopLaYQQomLPnn7O7TfFy4xACJH2Y707USGnL5dN68iUFJcwiNDS0ZTzjcrhchBCjEgokppIiYIWOOroG2rX/M/Ebd1z2ZdZssxHbDv/50Q7TsnLP32n74usfl8bU1oiYBDCqjz5hwmr8ngYIUZQWloZm0goRghhjgqncR63DgAAQDpI3gAAoCmg0lLSKYQI/RYtlBs1Y7Wc9M+ti3t+GNPL1d5CX12Fo6Zn0aqNtazhH6yoUCb3xk/DfLZcfJtHa9j09lm+7+KTt3cPLeplwlJ2gU8PslZMQqGIQZjFlr2JOtebMrC6lgYLI0ZYUn6RI50YeOYh8pgyzo5EnA4TxzuVBJ+8Xn6ZZlnWgzU15W1fJBChhk2GSLOhW875j7fllL7+a8aYNXdrXTRKpSSmUAgRRjY28iLDOjY2+gRCVEp8SuVUKhwVNpKTbAIAAGgkkLwBAEBTQKeER2RQCLPb9eqqRBaC9UevWTvYlEWl3dk8c6BrqxaGRiZGLdoO2/2mxkANI5ZIGISwClf6yFRVosSbm727u7QZ4Ls24H6SkNR2HLriyMnVndSUXqCeQX6COtabcrCGliZCiCkVCMtzFzr98ql7otbjJ3TQ7jLZyzL/5pmb/PK3GEGpgEGYp60p+2JSuoDPlzCI0NFrmMsQsXaXVSf2TrTjCN797TtmZXCmlIsx6bRnzxIkCHM6DuxV60EHlSXp9xnYgYORJPHp08oHxJUHSuflNvBlngAAAD4NJG8AANAkiMMuXkqmEKE9aJZ3K4VX+LGdO7tpYEYQ9Nus9eefx+cUiyhKWJiRkFZYY6SEzsvhMwgRZlamSt6oJkx/HvjHD6Pcun57NE6MVOy9p3ZXrdsCdQ7yU9St3pRDqGtqEgghQYngY6RMzrUzIcXWo+eunTncMOfqmZDCiqVLSwQIYUJDW0N28ihMjE+jEGFgZa7wuXWKYYPBG/1nteZKEs8uGLdc5iSbkpdnz0SJGazZb843bWTMYMNtO2NOL3XMiKPOnn1ROWsoaW5pRiJGmBTfcDN4AgAAaACQvAEAQNMgerJ35718Gqu6Ldq3qruBwo9nBiHEUBJafiJEp7x9y6cRadOvv02dnrgsiL+8P/A9hbCaoZHUW84ULqB8kJ+krvWmBKyuroYRYoSCiuQNMblBZ4LyDUdM7qeZfvV0aEnFwmUjbwira6jL3rYkJuJ1KYM5Tm3tPznBVOu2aO0YM0LwYse0BeeS5DynQfL2r43n0yms0ub7/b8NMKmVvLNMB230n9uag6mMwN8Pvq0siTBq08aQROK3EW9lPB8OAABA44DkDQAAmgg68ciipVfSKKTW9rtjIWfX+/Zva6mjQmKCpaJp1LKVRdUJMcTvwl6WMli1z4+/z+xmq6/KwgizVLT0tFRqplGiR6cDkynMbuu3d+sUd0stDoFZXG1ja5MaFxmqd/l2+awhHVrq89gYkyraVu7jZg9rQSKan5CQxyizQC3KB/nF6k05WI2nhhEjKS2ten1n/p0zQXwaUalXAh9VmdaREQpKEUKYp8GTs1vFT+5HCBnSvEtXm0+YqhMhhHUHfTvWgqRSjq7b9VzqcweqhJZzdeWcA+9KEddh6qE7FzbPHtTOWo/H4ajpWrUbPHvz5VsHfOxVkCDyr+9XXq5yzxzW8ezRho0kCQ8eJsPAGwAANCl1+h0WAADA50QlHp81WvL7vs1TWpt2nbm568zNtZYo/z+TeWrDLh+PJW52Y7acG7Ol+kKiav8S3N+6PKDXwam2raduvzR1e7X3Kgdb2I6Dv/1+rvWCjdXeZ+ickK177wuUWaC2OgT5aZSvt0qcvtvf8bfXell0b2H7MUdVeTyMGIGg+rhT0ZU5jrpzaq7BlH4ceZOXvNFpNy4/Xde1q9OQIba7IqNqx6MstksnN3WMCPOZZxNm1g7/4Yr2I/6qTLkY/u1Vo8dk79i7qK+lx9QNHlM3VI9dmHJ769x5W0KrPsUb6/cZ2lUNS+KuXXot7/nrAAAAvjxI3gAAoCkRxp5aOODO4WGTxw3u162DvZmBjhoWlRTlZaUkfIh58+J5yIWYsg61IOIPr0Gxc+Z5D+3iYmusyWHEJUV5udmZaUkJUbfeFFXpjDPZNxYOGfF8wfypgzwczbVVCUpYlJ+ZmhDz7nXEvRtJZXkEkxX67wm7/p5tbU31eGwkLspOinkeEnjQ/+9bCWKlFpBG+SC/WL0pQ4WnRmDECASlykRICUpFDGLzNNTlDSjSKYEBt5Z2GeI8cXKHPaufSE93lUBo6ch7GriUDWff/2Oyx9lu4yd5DezZqW0LY111ojQvI/7V4zs3zh0+cft9cfWdJKzG+PRUR6Lnx05ENNzEMgAAABoE1tIzaOwYAAAAgK8ft9O6+4GzWxTe+L6bz9G0JnpBonqvzQ9OTDXPu+DbxfdMNjwoAAAAmha45w0AAAD4EgRP/Dddy0XafZYt66vTNB9+rdLW7+eJFoTg+d4tgZC5AQBA00Ny1XiNHQMAAADwH8AURYYXuo/v59Lew/DV6SuxCuYb+eJUO/50eMdwY+rNrukLzqfA/W4AAND0QPIGAAAAfCFM/uv7qdbtmOu7Au4nlTS1oS2qMFdoYM+6+OOy82n1n1IFAADA5wP3vAEAAAAAAABAMwD3vAEAAAAAAABAMwDJGwAAAAAAAAA0A5C8AQAAAAAAAEAzAMkbAAAAAAAAADQDkLwBAAAAAAAAQDMAyRsAAAAAAAAANAOQvAEAAAAAAABAMwDJGwAAAAAAAAA0A5C8AQAAAAAAAEAzAMkbAAAAAAAAADQDkLwBAAAAAAAAQDMAyRsAAAAAAAAANAOQvAEAAAAAAABAMwDJGwAAAAAAAAA0A5C8AQAAAAAAAEAzAMkbAAAAAAAAADQDkLwBAAAAAAAAQDMAyRsAAAAAAAAANAOQvAEAAAAAAABAMwDJGwAAAAAAAAA0A5C8AQAAAAAAAEAzAMkbAAAAAAAAADQDkLwBAAAAAAAAQDMAyRsAAAAAAAAANAOQvAEAAAAAAABAMwDJGwAAAAAAAAA0A5C8AQAAAAAAAEAzAMkbAAAAAAAAADQDkLwBAAAAAAAAQDMAyRsAAAAAAAAANAOQvAEAAAAAAABAMwDJGwAAAAAAAAA0A5C8AQAAAAAAAEAzAMkbAAAAAAAAADQDrMYO4L9o2dKljR0C+Mr9vmFDY4eAEEIjRo5wcnBs7CgAUKCJnC8AAACAQpC8NYKu3bo2dgjga9c0+qJODo7Q2kEz0DTOFwAAAEAhuGwSAAAAAAAAAJoBGHlrNE+ePN2527+xowBfFb95c9zd3Ro7Cimm+Exv7BAAqKnJni8AAACALDDyBgAAAAAAAADNACRvAAAAAAAAANAMQPIGAAAAAAAAAM0AJG8AAAAAAAAA0AxA8gYAAAAAAAAAzQAkbwAAAAAAAADQDEDyBgAAAAAAAADNACRvAAAAAAAAANAMQPIGAAAAAAAAAM0AJG8AAAAAAAAA0AxA8gYAAAAAAAAAzQAkbwAAAAAAAADQDEDyBpoR0srr9ws3zi53ZzV2JFIRpsN/vXDz0tqubBkLNPH4QRNHmA5bd/7GRdkN7CuEdXquOn7p3oa+Ko0dCQAAANAUQPIGvgCV9n7/Pn92ZWN/PfxpBWlYODlZaHPxJxajvDpFjnlmrRzNtbmyF/3i8YOvCeaZOzpb6MhpYF8fzDV1atPCQI31X9ppAAAAQCYYAWjyCE2H/hOnefXxbGNtpMWR5Gd8iHrx4Oa5w6cfJgvlr0k6T9+1dYr6xTnT90RRXyZYWTDGGBPE5+p/8fptuvbncNX7Pw+Z9m8GXeNNtuvyy4dnmkT8Mmja38k131ToM0cOqiA0W/UbP9Wrr2cbaxNtLl2ckxz3Luz+zX+PXwzPbuQGXBOp7TxwgveI3p1bWxlpsAT8lKiwexf/PXr6UaqgsUOTS6nPBN6wXWFbetce6RJe/9HV71rT3kEAAADgKwfJW5OGNV1mbvljSXfjip+dObrmzp3NnVx5UVceJQsZuWsT2lZOdiZ8TuMnHsKwHePa7fh85RffvxKSO2yk+9B+pqeO1EjQVNoPGWxOCJ9evZZa58zt80cOymFNl5mb/1jcw5hd0Vw1jWzbGdm6OIqfXW1SyRvW7jhv+5b5nQzIj6GqGNm4D7ZxHzhm0qmfZ/9yLVHcqPHJ03Q+EwAAAABQH5C8NWGEideGPUt76DDZ4QF7958IDo/LEnF0zB07dBvokH4/X37m9t9S8vjKjczhE9sNGWJ5fF981Y4+12NoXxNCcO9SUHo9crcmgFDR0NPmSgr5/BJJY8dSaeLEieo8Xsjdu9HR0Q1QXFlT76nDZIcH+O8/ERwel0OpG5lb27r06KF5P1zBEPMXRVpM2rpzQWdNKu3RoT0HTt1+kZBP80wduw2btmBmH8dx6/cXpHttiShp7DA/meT1dq+Re+OaUNIMAAAAAEjemjI1z9mLeuqi7ODlE384mVjecRdmxj25GvfkavkyWK/nsm3zB7UyN9JUYUqz455dP7Bt97mo4srEjrT3C3zphxBCiM484d37lwdihHm2Q76dM2NoJwdDldKMqNBz+zbtD0muGC7gmvfynjVjRFcXSz1VJMjPSnkf/frm31sPPMkrL1bVuv833/kO93Qy5dH8hLDg0/57TjzJKuvmYS3XcfOn9ndztrU21lbFpdkJ19f6rIsdf+KKn8nZb3v9dO/jZlQt+06bPXN4l9bmmriUnxJ113/lL+cTKMV7JFXps8AbaeN9nEYMsTmwJ7qyx8nrPLKPPi66fT4om1FQXUpFrjg8zLEfufLvRd06ttQli9PfPrh4YOeha/GlsgKXcywI3Y6+q5fP6muvw8YMIy5MvLlu2k9n6jN+2PB0dHWGDB48ctTIzMzMoKBbd0PuJiUn1bu0j039zspJC04klLcQYVJ0TlJ02O2KpaQeoDVXcxm5DZLdZc3Nw+P4u73G/BH5sYmO2vNkQ+eHK/pMP53LIKzb+ZulPj1a27WwMNRUxUJ+aszjGyf3Hbz4Kk9KVav3+G6+pxaTcW3xpCWBqeUNTZQQfmF3xL3w5Sf3TbSfsmDsiW/+SaYRwnpdfFdM7eFkY2Gqr6nGRgJ+YkTQ8a3bT4Tzq7QX+SdjbVit0/cHrv/WylKPS+VJKVBubSCEpH8m1IVS+yXrBFcYHqHbduKc2ZP7tWupxy5Ji3z4KN+o+q3ZsmtMdgsBAAAAvhaQvDVdnYf1NSRE4Qc3n0mUPeQi0bZpb2/OQQghpG7k2NNnc2tD8YhFF7Nl91fU2sw7+NeCdhplPSKuRdth3+90NfUbsTKEzyDEdfDdf2Cpu87Hu7x4eub2euYtOc8PHXqSRyGEuE6z9h9Y4q5V3qEysu8xcZln97Y/Tll2MZVCiDDsPMZ7sNPHhqVhYMgWFtWKQcXBd9/BpR7a5YVwjGxdLdRL6XruEUIIicLOX4qbPNt+yGDnfdEvyysMa3Uf2ksH8S8F3irrVsorXLnIFYaHea5Dx5T/zbHoMGROO0/XtVPmHI6V1kGWcyyw8dgNu5b00MSS4pyMIqyup23IFuY3icytjERCsVikoaHhuHFjJ02amJqaevv2neDg4PT09LoWVdbUX/y96VSCnDRC6gFiFDVIhQg914GjelcUy9K3dh3ybdu+A9wWeK++VvMOSlXPwT31sPDpgW21Cmf4D/ZsDxq8c6Dr0D6mAf8k04jQdek3rEdFyYinb9NlwgpXe1Uv70PRZU1U/skoFVaxbNux/O/aBX5qbShDif2SdYIrCg9req48vGuaXfmcLCqWroMtEUKocuxVXo3JaCEAAADAVwRmm2y6nOzVCerD3dAUOX0upvD+Zp+Rnd062Dq6OHYa+s2BlwK9XqN76lTe0kJF7xzh0qKVc4tWzjbdfnkgJu19Vs1zVcsM2fnN4C4Ozh08xqw8/Z4yHzlvsi2JEGkzZfWP7tqi95d+9h7o2sbFto1H19UhJVUG8my9V/3gpil4d3rJuN7Ordu16+/7e3AaNh20Zkm/yq0yhTd/HtqxnautS+duY3c8rtkhJ1tMXrXQXUsQfX6l96D2Lq6Obr0H+my8ns0otUcyUJEXz7wSEdaDRrhyyl/COn2Gd9ViMq6cvV+oZHUpiFyZEkQfrmz8ZlgPp9bt2/Wfse5qgkS78+JFQw2l7IC8Y4E1PPp7aNCv943q3Llj994dOrh3GrUltElejcdisRBCJiYmEyaMO3Dgrz/+2Dp8xHBtLS3lS3C0VyfoDyH3khWnFzUPkHINUnGxBVeW9HZ2bmPTulPX8T/99YzPthrx65I+NUsgzVvZ8ggq9t59aRfhMvkP7r2SYNLWoSVZpeSry/q3dXGxcfboOnnDzXSa13by5PZlc/3LPxllhVp05/dxXd072LWWUqBStVHzM0HqZlitF1yIjXrz4eN/78M3D+BUeV/ufsk4wQlF4bFaz1jqY8spiAhYMKa3c+t2bXtPXnDgaXZlXStRY4pOYQAAAKBZg+St6dLgYUTzc6Vdu1UJY+02E9YfOnP/8dOXtw+vGWBKIpaRib7M40q2GjbUgVUQ9Nvi/bfj8oQSQearc2t33i4i7Tzd9Qmy5eAhzhwq8s8fVh5+kpQvoihRUVZOld+uSbvhI5w54le7flx36kVGiViUl/Bg/6KfT6YxOj2HV3Z1GQk/JTmnREwJC1ITMmpe8ki2HDqstYr45U6/1UefJPKFYkFBRnR4dBZdrz2qQCVeOPuslDAd6tVJDSGEEGEyYLQnj064cvrpxxnyFBYuP3KlSih+evb47ejsUrEwL+HR38vWnEiheZ36ddOulUnIPxYMwyCEDRzcHfS5GCFGmPUhOa8JjyJgjEmShTG2t2vlO3PmkaNHfv99vaGRoTLravIwovg5fCXGFWscIEK5Bqm4WKooN7dEQtPiwpSIS7/PWROYhXT7jOipVb0EzNNQw4jOz5E+BMoU8/kChlDl8SovaWCowqzMAiFFS4pSnh1bH/BaQuo5OBgQSH4DMHaad7Yyd3p7YZFzRXIizoyLTskXSMS1C2yg2lCuxmTvl4wTHCsKj2w1oK81IQzb/uOmwFcZJWJRQUrExSM3YityevmnTHlgik5hAAAAoDmDyyabruJShAgtbS0CZcoYkMCa3Vf+78BEq48T9KlYWiCEKIKQfVg5Fi3NCUJ1wK4nA3ZVe4MyNTchWPp21iSd9OCO1Gv8EEIcK1tzgk56fL/qpCDFz++FCyYOtLK1IFCuEjvGsrKzJumkJw8Sa+1XPfaoEp1x7fStHzoN6Teqz6Z7F/OIlsNGualI3pw791ry6YXXt4TSl49fibz7m1mbEohf/S25xwIXPjh3K7vXkB7LDwf9yE94HREWcuHIoWsxSnZGL1++pOxO1QtFU4ysVAsjAhMIoTYuLhXpghpPraRY5rihrKbOGbD1xY5+iXsnDN7xVvo50CANshYm/8Gt56KRfS1tzAiUV6VkpriwhEGElp4WgaRMgIl5OjpczJQUy5pbhkqNiy9mnNXVeRjJbwDGZLEykdYosAFro04TllQPQ9YJrjC8ElNrM4JOfv4sTUbbkv/xhbKV3z0AAACgmYLkremK+VDKtGrRyc3AP0b6RIlYt7VMt/UAACAASURBVO+0UZYk//HuVZuOPorLKmXp91lxfvtweYUyjIzOP1ZRVcEEm0UgJJHI7rE1xI/3ZY9NkxZIffaoCiY/5NjltMGTu00YbHL5tOE4LwdWyYNj58u7ip9YeH1LwJjACEl7UJz8Y8FkX17uXRQ+dlCntu3btWnfq0WHnr0cCK+5l5Xqof6+YYMyi9Vb//7927q0lfUuzTCIYRiGzs8v0NXVRQjJydwQQrHxpUyrFp1lN3XZFDdIBtEIqXDr9mRrhqEZKW2USo6KLWEcbLp2NtobV2vqGKzZuWtrFiOJjZKZ8zAikYgpaxLyGwArcqOX7e4ar0u5lrJagQ1yetZLtTBknuCKwsMkgRDCsp+qKP+UUTpaAAAAoPmC5K3penjrUcGAPp18/frfXHktS0qfltA3NuagkpsBu4IiRQghJM7JKqgyqzojkUgYpKamVqVXI06JT6FprcCZ/Vfdrt2dZrVLzaYJy47upsTrJGm9aFFCXDJNWHl0sSJfvf/YQeW179aOi0SJ75NppS7EFafEp9CEpXtnC/JVtWn9Fe5RGZKU2WwFT/89FzlhrvvEMZ34Zl6WOOfCySuZTF0Kl6ceJWAtzz7tOUiU8D6lonI+xi//WCCEBEkhAdtCAhAiNRxGrzu0pl/PAe7o8hVlQg29F1qHHas7FxeX2skbwzA0RREkGRsTfftOSMjtkDlzvuvaravC0h4EPSro38fDd8GAoBVXZY0zS6W4QaLC/CKGMGtlq4UjcpS9hk61jXtrDhKlxKfUOAtKH169nTNkhJvvwmHBPwVWmwIE63Seu6CfNhaEXb6l3JSgChtAXSmuDSzlM6HByTrBFYYnSoxLpglrz542e15FSxv8V1Bjsm8UBAAAAL4WcM9b08W/tvfAGyFhOnz7Cf/Fo9xs9NVYBMnRMLT3GDr7h1GOJKJzMjPFSNXDa3IHU3UWRgRbXZ1bJa9hMtOzGNKk39h+LdRZJFfXpqOzKYq6fvM9rT9szaYZfZyMNTkkQXJ1zJ17ulmxEEKSdzeD02iVdvO3Lh7mbKTOUdGxcvPq51g5TwEVfeHCWxG7zffbVo1xMVJjcbSsPL/dvHacCc4LuRik5JTcVNS1G3EUu+38XeumeFjrcEmSrW7cyrWVHqFoj5BILGGwdvuenczVpHfUqNizRx+VkrYTtq/sr8sknjsRWvjxLYWFK6RUCZjU0NfjsQmCpW7iMmTZnjUjDFBu8MXb+UzN+Cm5x4Js0Xt0bxczTQ6BSTZLUlgoRAg31cEFiqIQQqmpqYcDjvj4TP3hhx8vBF7IL8hXcvWPTX3o9n//XDraw86AxyYJlqquhbGmgj1W3CCp96/eFTAqXWcvm9TOSI0kSK6GgY5qzWIxz33MpF52eqostqaF29T1ayeaE8WPb96r9TTFwjt/7niQj40Hbj7251IvNxt9VTZLRdvcZfB3f5zyn2THEscc3X5S6g8fUoKX2wDqQXFtSPtMaPCUR9YJrh2jIDwq6uKld2KW09zdG3272ehySYLkaunrVGaaDV5jAAAAQHMDX3lNmDhyr99Phv6/TXbsNmdDtzlV35K8IQIvvPsQfPLWvG5Deq8+1nt15XvUx6cmU4khwW/92rh4bQn2KiswYv1g7wMHfzvUa69vv4UH+i2s3FT4pn6T/kmgBU/3b7nQZ8vItj47z/pU3V5F4TEB67Z3P7DYbezmU2M3l7/IiFOurtl4XenHKUneHPx1X3f/Oa1H/nJ45C/lZRRf9Ovud1PBHiW/i8xjHBx89l43XOQ2/5qUH9/pjIsB1+d7jjLSZwTPThx5Iap4h8mRX7hiSpWANQdtuDWo8qJFRpgQuGrjTT4jJf7Xso9Fop7HjLWrPNlVdy336o2nSgf72REkIZFIWCxWWlr6rVu3Qu7cSU1Lq2dZ4si9fj8Z+f82ydFz1nrPWdXekz8Qp7hBFt8LCHjX93vnQb+eGPRr5YqiasVgjvXAJYcGLql4gc57tGHLpczaTZpKPLpovu72LX4enrN+rx4qUxx5avWs7eFKj6JJ5DSAhPo8FUJhbUj/TPgrsfbGWK0XXIhdUDPeTcMm7X2veL9knuCKwov+Z80Wz7+Wug9YfmDA8iolfhzfbvAaAwAAAJoZGHlr0qjUoNUTvKb+GnDt+YfMAgFFUaUFGXEv7p3+6/h9Po2Y3KsrfX88ePt1aoGQoiTCYn5mcvSLx49iy8cLqJh//Bb/fTsmu4SiJCU578NjszBmCp9umDJpgf+lRzGZBQKKEhdnJ7wMeZZU1pmls24umfzdprNP3+cIJBJBzvsngUFvSxhEV0xPUfr2T99Jc3ZdfpaQWyoWFWdG3z3++5TxSy/U5SlSTFHY1qlT/PyvPIvPKRZR4pLcpLdhcQVsrGiPSkK2L9wT9Dq9MCU5TSSj8KLQY6fiJAydFxRwodoQiKLClYhbfgl0zosbF+++iEnll4goSlKam/jy2sHV48avuvrxcWE14pdzLDBOe37nZUJuqYSmqVJ+4sug/T/NWHwpS/lK/tzy+PzAwAvff+83c+bM48eP1z9zQwghRKUGrZrgNfW3I2VNXUKJSwtzkqLD71w6ceKh3BvhFDZI4eud38767czTD7kCiqYkgsLslJjnd6+GxJZWHnem+MWVs/diskokEkF+8ovr+7+fOO/vGOnT9jD8pzu/GTlq8d4zoe+S+SUiibAw68OzGwFrZowatepaYl3mppd/MtaHotqQ+plQ763JIvMEV3iwSt/95Tt++pbToVEZBUKKkggKs5PePgk6E/K+rF4bvsYAAACAZgVr6Rk0dgz/OWUzAT558nTnbv/GjkUhbDBu/711HR+v6jPtlNJDa6CR+M2b4+7uhhAaMmToZ92Qrq4un8+XOX/ER8uWLi27522Kz/TPGs8nIO2+O3HFz+Tst71+ugcPBftv+WLnCwAAANBQ4LJJUA1h2GFQWxTz5kNqdr6ApdOyw/BF33lw6LjwV0oPT4H/gNzces3BDwAAAAAAPgEkb6AaFdcJG3YOVq96IRVDpV7aeyy6LnMAAgAAAAAAABoaJG+gKszJi7r9pKWrnaWxlgoS5qe9f3Xvwj+7jz7OhMkAAAAAAAAAaFSQvIGqmPwnB/x8DjR2GAB8MVTM3rF2exs7CgAAAAAAJVRL3pYtXdpYcXwBbyPfBZ4PbOwoAAAAAClGjBzh5ODY2FGAemo6fYyvuy8HvrBz589FRkY1dhTIwaHVqJGjGjuKxvT7hsqHUFVL3srmhfuKBaIm8cEKAAAA1ODk4PjVfwt/3ZpIHwNaEWhA9+6HoiaQvOkbGPzXG3Zl7gbPeQMAAAAAAACA5kDKPW/N5PljdXDk8N+NHQIAAACglCb8XEQgRdPsY3x9fTnwJbm7u/nNm9PYUUhxNN3gVaFaY0fx5Uw2zmqjUVLjRRh5AwAAAAAAAIBmAJI3AAAAAAAAAGgGIHlrMDY2Np6entYtrFVUVBo7FgAAAMry8hrl7uZmYmxCEJ/rO1FbS+szlQwAAOA/BZ7z1mAMjQxXrFhe9ndeXn5qWmpSYmJqalpaWmpKampaappQKGzcCAEAANQ2atQoXV1dhJBYIk5NTk1KTk5OSU5OTE5OSU5OTi4tLf30TXhP9W7RouWBAwfevnn76aUBAAD4z4LkrcHExsZV/K2traWtrdXKzp5GDIskMcYIoYKCgvT0tPj4xLJl1NTUSJKkKKpxwgUAAIAQQsjb24fH45mYmBibGFtaWFpZWnbs0MFr1CgOh4MQKioqSkxMTExMTEtLT0xMSkxMyMzMpGm6Tpuwtmphb2+/edOmsLCwg4cOJcQnfJ5dAQAA8JWD5K3BZGVmlhQXq/F4Fa+QLJKssoCmpqampqa9nX3ZP9V5PIZhvmyMAAAApCguLo6NjY2Nja14hSRJAwMDY2NjSytLK0tLY2NjDw8PHR0dhJBYIk5LTUtMSEzPSE9ITExMSExKSpJ/bYWFhRlGCCHk6uq6Z/fuhw8eHjx0KD09/fPuFQAAgK8OJG8NKTom1rWtC8JY1gIUReGPN1VkZmXV9bdbAAAAXwZFUenp6enp6RERERUvqqurV6ZzRsbu7u5eXl5lH+q5ubmJiYnp6eXpXNm6ZWtpaGjweOplf5MkiRDy6OTh0ckj6GZQQEAAPy/vi+8cAACA5gqStwagwuXatGxpZ2tLkqRYImGz2bWXoRkaIxwdHbV7j/+e3bu/fJAAAAA+UVFRUY0BOjaLbWZhZm5ubm5mbmlpYWdn16tnTxUuFyFUUFCYnJyUnJQsEotrlFOWwvXt27d3n96BgYH//nuypKTmk3wAAACA2iB5qw9VVVUbGxsbWxtbW1tbGxtzc3OCIAoKCtPTUqVmbhRFFRYWHTx08Hbw7YpLJW1tbZvmAxBB82Vra9vYIUgHTR00QQ1yvogl4vgP8fEf4qu+aGBoaG5mam5ubmFhYWZmbmtnQ9N07aksSRZJItJr1KhBgwadPHkKY5j/GQAAgAKQvClFVVW1RcsWtra2drZ2trbl2VpxUVFCYmJ4RMSp06djY2OTEpMMjQwPHTxYdUVKIkEYX7p8OeBwQI0py3R1ddzd3b7sfgDQOKCpg/+UrMzMrMzM8PDy6y2nT58+YsRwWc8hIEhSTY03bZqPSFRzgA4AAACoAZI36dTU1KxbWNfI1srmHKuardWYcSQzI7OkpERNTQ0hVPY7a0TEC/+9e+GudAAA+M+ytLRksaRclFGBYWjEMBXPCNXU1CgoKPwioQEAAGhmIHkrx+PxrKytKrI1CwsLjHFubm5sbOy9e6GJSYmJiYmJCYnyC2EYJi4urk2bNjRNZ2Zm+vv7h4U9r73YkCFDP89OANC0/L5hA9rQ2EEA0Nisra1qzGMlkYgJgiQIgqbpjIyM93Hv4xMS2rm6Ojk7IYQgcwMAACDLfzd546mrW1lZysrWYmPjYmNjcnNz61psVFS0vb39kSNHLl68JK51kzoAAID/FDabra+vz9A0JgiEUElxcdz797GxMe8/xH94/yEpKUkikZQtaWVp2aiRSkWYDlvj/327F+u8fg793N9opJXXr7tmt3q0ctz6J5LPvC0AGgnW6r78Dx/+3p/2Ps35oo+L4tqNXLa2d+SKhcc/wOn1ybCKyrye6t0LiyY/EIq++Nb/Q8kbi8VycnaqyNYsLS0RQlWztZjoqE+fsjn4dnBgYGA9sj4AAABfHy0trdDQ0PfvP3z48OH9+/fN7dsB88wdnS10omQ+AachaVg4OVloRMh+3A742qm09zt8wEfj5nLvpTe+bGrzhWCNLvN/mdyROk588R6/WKJh0brfgF/Gh049mkR96a1/bTBJ2umxdEvLPq1w67Y6GxyI0Ie5GxPpL9BuGyB543aaH7BqaAtDXQ0eh6QEhXmZiVGvn4TePHPudmS+ks2DdJ6+a+sU9Ytzpu+J+lwtqlPnTh6dPNIz0mNjYm/dCo6NjY2LiyssbOCrUxLiExq2QAAAAM1Xdnb2xo2bGrRIrlWvKXN9hnRrbamvhoX5WR8iI0KvHtt/+gWf+RJfpgoQmg79J07z6uPZxtpIiyPJz/gQ9eLBzXOHTz9MlvcYc9BICM1W/cZP9err2cbaRJtLF+ckx70Lu3/z3+MXw7MboQlhjDEmiM+XvyvaX96wXWFbusds9xq5N07K/pO2806dWWgbtW3s5N1RtceieZ1Xng6YYhSytMeM89K6lyy7qQtHmeVcmbvrSSEjc1vYcHzArdXuL37v6XMklUaIY9Jloq/v6F4dW+qyBLmJ7x6e/2v3gbupNdM/Utt54ATvEb07t7Yy0mAJ+ClRYfcu/nv09KNUAUIIUR+Ob9g/8t8Fc+b0ubj8RsHXmBojhBBSMVbf5sa1UCXU2ZhkmFIRnZYvfp4kOBcrTP6cQ44YIWXmC7Zz1F7ZCgeF8AP49d9WAyRvpIFtG1vT8vusSTVtQ2ttQ2uXbkOmz355eOXi9UEpStQVoW3lZGfC53zOn9tev3q97tdfi4uKPuM2AAAAgM+ItBq77fTa7vpk+fclS8+8dRczO1bE4TMvEPMlvkzlwJouM7f8saS7MetjABxdc+fO5k6uvKgrj5KFX21/sZnCmi4zN/+xuIcxu6LBaBrZtjOydXEUP7vaGMmbMGzHuHY7PlfpDbC/hL6RAYFVHGf+MPT0nHPpdLU3SbuJS8ZakJjS0dclUWHt4nhdfbwd8dudB4LylD4XCJMxO05u7KVbntCyDe3cRyxq38Fy7rhld/IrSsHaHedt3zK/k8HHDwakYmTjPtjGfeCYSad+nv3LtUQxQpKYI/uDpm8fMHOUf9A/SbS0rX0FSFWWgxbJKfsHxjwuacslbY24w+1LfwkquPtZHqjJvH6RO+SFMktiLQ22NY/mfNr2GuqyScnbPRPH7HknQByejrGti+eQydO8u7Sd9sc+/O3EdQ8Lm8IHdl5eHmRuAAAAmjGW69Q5XfVQZvDW1RvOhifkiTm6Fs5u3V1K72Q0el+MMPHasGdpDx0mOzxg7/4TweFxWSKOjrljh24DHdLv5zeFjoBSCBUNPW2upJDPL2la9wb9uHBhSmpqyJ2QtPS0Biiu7Hj11GGywwP8958IDo/LodSNzK1tXXr00Lwf3pDjpLKq9ItWdYPsr4q+kRYW5eWT3b71bX/1l2eCyrewdv/ZU9sI8/I4mrq6OhjVug4La/X26qsvCtt9/n0d0mI68/mzD2laV3fvOHotIk2i5zR04fp1Qy1HePfbGnI6q+ysIi0mbd25oLMmlfbo0J4Dp26/SMineaaO3YZNWzCzj+O49fsL0r22RJQgJi/kzJWMARO8htod2dtYY/M1dO3axcPd405ISEREBEU1WEwxL3O+eyURIaTCIS30Vbzaqg/WVV3gInjySCRQvHZT12D3vNFioYhiGCQsyk6ICE6IuH3l1uJDh75pNWWp90kv/3cUwno9l22bP6iVuZGmClOaHffs+oFtu89FFVd+nJP2foEv/cpKyzzh3fuXB2Il1gIAAAD+I9TMrfRIOvHSzoOhMRRCCIky4x5fjntcdZn6fZmqWvadNnvm8C6tzTVxKT8l6q7/yl/O1+x/Evrdlx/dNd7w5XafWQdfVf8NW81z9qKeuig7ePnEH04mlvfFhZlxT67GPblasRXr/t985zvc08mUR/MTwoJP++858SRLao+N3WXNzcPj+Lu9xvwRWbYA1hq158mGzg9X9Jl+OpdBWK+L74qpPZxsLEz1NdXYVEHquzvHdu97YTZy8oj+Hg7m2mRxyusb/2zZcOxVHoNQzeWRgJ8YEXR86/YT4fzyaiB0O/quXj6rr70OGzOMuDDx5rppP51JbfS0uJx1S+vefXp7e0+Ji4sLunUr9F7op9xC+fF43Vk5acGJhPIrAIVJ0TlJ0WG3KxeT23IUHiOZVSrjdWz33YkrfiZnv+310z2xoq0rPqD12F/5CF19PYLOvLL3YvtF3t8NO+h7qqJ1sOwmzOmv8nCrf9GCRZ762lKuoFNz79dZXfLqdnDdfmih3gfM7v9PSXl6mxrx74b/DR3wcydjUyMClZ066j2+m++pxWRcWzxpSWBq+dkkSgi/sDviXvjyk/sm2k9ZMPbEN/8k00gQEXQ/b9LIXn2s9kfVJYX8fFRVVXv36d27T++i4uKQ23fuhIS8e/euxoO46oGhkZhBDEICIRWTUrK5CNsNUbc14FhiUZqe6nRHbltdlpkawUUMv1CwPaggRIAwm9XbmTfOmmOjigWlkmdxxX++EVYMrhJc9vA2vBEWHEsuKi2WPM+g9atc4GDdWvfvtuT129kbUj9GziK7OqiPb8mx52GCYtL5woBHBTfKrqXFrGlDjKYhhBCiS0t/OFfwvI6fMZ9twhIm/9GujacGHPCxGzTEYd+7NxSSaNu0tzcvGylUN3Ls6bO5taF4xKKL2XKPUP3WAgAAAL4+pWnJfIqw6OM96Ez0pYRS5VeU/2Wq4uC77+BSj489To6RrauFeild/SYOrOXmd+CP8aZRf82Ye+hVzauPuJ2H9TUkROEHN59JlDGKwnWatf/AEnet8kKN7HtMXObZve2PU5ZdTK1HP5LQdek3rIfTx34MW8ei3aifDo6qsgTHquP4lXt1i0fPPp9B11we8fRtukxY4Wqv6uV9KFqCEGE8dsOuJT00saQ4J6MIq+tpG7KF+U0lc6uqZcuWM62svvX1jYmJvn0n5M6dOwX5BXUso/x4vfh706kEufOIfko3TFaVyqxqsi5bV3RA67e/cmFtXR2Cyct8fPjQ3Unrp3/T7sKvYUKEEMKafb6d5JB9cdq5qEEzGa6uHg8jUfUaYrVq15ZHJ0e8SK9rmxKVVD3bCB1dbYIRJiZ8zBtVPQf31MPCpwe21TqPGP6DPduDBu8c6Dq0j2nAP8k0Er58/kY82r2DqwZ+r/y1m58TxmUPRlbn8QYOHDBk6JC8vLyQu3dDQ0PfvnnbcFtBFamWnrHqKCv2x2aDddWQSIwQiz21t850A1z26aSizu7TVtuJl+f7SJiPEOZw5vXVHqNdPpkSR4PdSwMhhGTOOkOyJvTS+c6IKP+sI7GVPqnWcKPLn3O2ydIXdx7nT/Eyc7TjoTcFTOH9zT4jV8QlZRWJ2VqWnWeu3z2j1+ieOpdO55a3Hip6Z+WPN+UUrwUAAAD8R4jDDu0OHbS2++gt57pMvPTP4WMnb0Xm1ugT1PnLlGwxedVCdy1B9Pn1v+y78iKtVEXX0kaLX613Tmh1mHvQ/xvbhMOzZu16Unu2A9LCyV6doOLuhqbIyMNIW+9VP7hpCt6dXvOz/+W3fI5px3FL1y7uNWjNkuDQH65JGyxRAlNwdfmYpZfTixnNVsOX71s3yLTwmf/KjUcfxuYwBu4zN/jPbt/Dq7fhhePlPeaPyxdRqiauo37eurhf28mT2wf8/ESMNTz6e2jQr/eNmb7nRQGFsIqBtYH4s9wh86kwxiSLhRCys7WztbGdOeObiPAXd+6GPLj/QCBQ7qIw0sLRXp2g40LuJcvPmz+lGyarSrGmslWteOuyD2j99lc+QltXGzPp+QWZ1/53eu7fY6b3//P5xWwGkZajZvZTf7XryKMiNc9ChtDR1SMQv9qWsLp1C2OSCn2fWD13Y7VecCF2gZRtSU8xOS0nL/V2RAn/+/tmeRWQ5q1seQQVde++tKyQyX9w75VkUBdbh5YkSqYRU/ghPoPuYt3SgkB5TWLorYqyVq2trT148OARw4enpaYF3759586d1NTU+hWIMVZTISz1VIa35dkSiJ8tTmSQMUIIMaGPczd9oAoYrK+GCynUsrWGjwHOSSna/Lw0rJDR0OHO8tQY2JI3IlJ4OA+1ctLw0sZF2SV/PC0O5TOkGquznfo8J466jO1a2GvONCKEeaX+T4tvZ9MCEptqEfkVpyYj+d+VnIONO2GJbJLc3HwGa6ipqxGogMZYu82EJSs6OVmZ6LKL07JpErGMTPQJlCuv9dRvLQAAAOArRCWcWuCVMW3xAu9BHUb/1NHLL/XZuUN7dh5/miH/Z105X6Zky6HDWquIX270W330A4UQQsKM6PCMqivrdpr/v/FT7JOOzvbdel9qmoV5GjyMaH5unoxhBdJu+AhnjvjVph/XnYqjEEIlCQ/2L/rZ6tKfE3sO76Nz/XT9LgBkqMKszAIhhRD/7Tn/o+P7L2mZ/fb+u/QShFDq/f0Hb05sN9LC2pJAH5O3iuWLUp4dWx8wqNdiJwcHA+JJKsMwDELYwMHdQT/qaYaAEWZ9SK5XTF9OxbyM7du3a9+hvZ+f35PHjxWsU74mT5OHEcXP4SsaBvqUbpiMKsXKV7XCrcs+oNV2TPn9lUtFS0uNoIsKi2lhxOEj4ZOW+Ey0u7Irmu3uM8m15Nb80/EUsi4sYrCurnbNSYMIPX1dgi7JySmp/9gDx3rMpj9XebJe7F629cnHZBfzNNQwovNzpI8SM8V8voAhVHk8FkJihJjc7FyGaKGvSyLUZDvSbBYLIWRiajJh/PjJkyelpKREx8TUqQR7V70Q12qviAsFu14KyxMohskvpvgSBiEmoxAhzO5jzSZFgj33ix+KEEIoJ6d0x0tO924qHYzII/lEdwsWQYkOhRbeLJs6o0h8K0o4zJHjLHXbmNWnBZtDif+8W3C+bEScYj5kNeQY/mdN3li6ulqYoUuKSxis2X3l/w5MtPo4w4+KpQVCiCIIuQHUby0AAADgqyVKvrt//t1/fu8w2Gf61Em9O05acbBv118nzT0ZJyt/k/9lyrKysybppCcPEmX05QjtvjOnIjov+PjRhzkyuiBMSXEpQoSWthaBMqWVw7GyNSfopMf346u8W/z8Xrhg4kArWwsCffoD8Ki0hFQJcjQw0iZQCY0QQqL05EwGG6qpSp9+k0qNiy9mnNXVeRghuvDBuVvZvYb0WH446Ed+wuuIsJALRw5di1HyJvuu3bpe7nbpk/dBntJSmdfJlj0Cns1idenSpewVMzNTkiRlzgAh+3hxBmx9saNf4t4Jg3e8pT6tG8bIqlIlq7rOW692QOuzvwp2iNDQ0iAYUXGxCCE66fyR67O2TfLu/PdOvW+GGyed+ikoj0G4uKiYJqw0NWs1OQ6Xg5FIVPNKO8lrWY8KqLEg13bytv1re6m//WeB796XleOUTHFhCYMILT0tAkmZMBPzdHS4mCkpLr9pjhGJxAziqCg73+GypUvRUiWXrSealpnYkCwSIWRmZmZmZlb2igZZl9le6PJHBbxIEZyPEcbLumCWJC3UEcHirhnHXVP9HUN1giAIMx6ii8Qvi5XbKkFaayK6SPS8gR9GVulzZkGqbXt6aBF0QmRMMdIdMW2UJcl/vHvVpqOP4rJKWfp9VpzfPlx+AVi3bz3WAgAAAL52wvSwc5vCAvc5ea3bvnJY9/nf976y4Ib0GfMUfJmWDd/ImSGAKXp+M0y/Z/deqw9uKfnmx0vSLwEbVwAAIABJREFULoykUmI+lDKtWnRyM/CPkXpTT50fX8AgGiEVLlf5FWmxSIIwm105E7xYLGEQxrKev8SIRCIG47LhKyb78nLvovCxgzq1bd+uTfteLTr07OVAeM29nK3MtiMjI8+dP690qPUxdaqPqqqqrHcpiiJJUiAQcLlchFBKSqq8ufuo1Nj4UqZVi84yjxdCSnTDFBwjGVU677Ks16tdSVaPTmC1A1r3/VUEa2hqYEZQImAQQkxByN9nE4ZOnraI0e3Bifj9+EsRQogpLREwmKuhwa554aNIIGIQh1O/SeJVbKbsOLCmO/fVX3O/+eNptcFvKjkqtoRxsOna2WhvXK3ZdbBm566tWYwkNqo8P8QcDhsjkVDZh4SfO38+MjKyXkErxcXFZdDAgXIWoCiKIIi0tHRTUxOEUCFV68ZIaaIjcnxfS5Q90AyS9fGnQmL88QNEmce4IYQQKr817vPd3vXZkjes1WnekrFmhCT6+uV3FGFrbMxBJTcDdgVFihBCSJyTVVDlS4aRSCQMUlNTq3a+Efry1wIAAAD+y+j8t+e2nxw9eLGTra0JeeNDfb5MxSnxKTRh6d7ZgnwVL627z4hjTy2cdXLB4V1Thv+6PS3jm01Paz8BqOThrUcFA/p08vXrf3PltdrXCIkS4pJpwsqjixX5qmKaO177bu24SJT4PrlichSSLO+Y0IX5RQxh1spWC0fkfKG73AVJIQHbQgIQIjUcRq87tKZfzwHu6PIVZVbNzsoOvRf6WaMbP35c7RfLRi0YhokID79z9+79+w/OnjmtRGHFD4IeFfTv4+G7YEDQiqtSB0sVd8OUOEbSqlTt8pVi6a9fr8vW60Sp/VUEa2qqY0ZYUlq2r+LXJ44/9V4+dTzDv7r4fHJZmxeVChgG8zTUMap2gSSdk51LE/Z6emoY1fG5GVi908Ltq7uznu36btbe8Forlz68ejtnyAg334XDgn8KrDZnCdbpPHdBP20sCLt8qzyvw7r6upjOyVb25qPIyMjP2rBVVVWlJm8SiYTFYqWmpt6+fed28G0bO5tlSz/bCCBNpRQjmlO6NLDgYe3rFzA7sQgRmpxOWjhSmUleaCqlGBHqnPYaKKrmREKMhGEQwqqfln4pnUYqLIjFJjFCJIenb9W214QVB07+PcORK44/tuHwOwrROZmZYqTq4TW5g6k6CyOCra7OrRI5k5mexZAm/cb2a6HOIrm6Nh2dTUmFawEAAAD/JZx2365f5NO7taUOl8SYVNVt4TZqzkh7kpFkZebS9fsypaKu3Yij2G3n71o3xcNah0uSbHXjVq6t9Kr0EBgq+96mqQvPJrAdfTevGmhQu/PA8K/tPfBGSJgO337Cf/EoNxt9NRZBcjQM7T2Gzv5hlCOKvnDhrYjd5vttq8a4GKmxOFpWnt9uXjvOBOeFXAzKZRBCIrGEwdrte3YyVyMRot6/elfAqHSdvWxSOyM1kiC5GgY6Mq5/bBBki96je7uYaXIITLJZksJCYcVP6E0Qg2iaZhgmJiZ6zx7/CRMmrv55TfCtYKGSE5ZUHq+h2//9c+loDzsDHpskWKq6FsaVV/wp6oYpOkYyqhTLer16iA3aCVRqfxXBaupqGJUKPj5unk69HBCcR1OpgcdufxwNowUlAobQ0FSvcY4wRfHxGRRp3dKyrj1vViufVZPMovbNmeFfO3NDCKHCO3/ueJCPjQduPvbnUi83G31VNktF29xl8Hd/nPKfZMcSxxzdfrL8odxYw9raiBDHv09uivOoIiShJAghfi7/8pUrft/P9/X99tixYw3zYEM5GHFIkoRW5S7owuuiS6qTiMBYS53dyZBkIYQYcVC8WEKwvXtoTjBlaZOIwFhDleDKLu1OooQi2dO7a440IrVIRBDYQIfdkosQQjklNI3JrrZcCzYiScLKkG1U9w+ZhkqFWE7zzkTNqxY7lffqn5U//vaggEEI5QSfvDWv25Deq4/1Xl25DBX98Y/EkOC3fm1cvLYEeyGEEBJHrB/s/VeS/LUAAACA/xCWU59JI6dbjZ5e/WWmNObwX9dzGcTU68tU8ubgr/u6+89pPfKXwyN/KS+y+KJfd78bVS/8orOC18/Zbn3yx0G//vLo1ZyzNXt/4si9fj8Z+v822bHbnA3d5lR9S/KGCLzgH7Bue/cDi93Gbj41dvPHyMUpV9dsvJ7LIISo5HeReYyDg8/e64aL3OZfK74XEPCu7/fOg349MejXyrKUvdyrrrCex4y1qzzZVV6ic6/eePqZNldvNEVhgoiJibl161bovdC8/Px6FiSO3Ov3k5H/b5McPWet95xV7b2Ps4Eo6Lwh+cdIVpWW6PWR+nqNW4oUbr1ulNjfj2rPAEkl/zOt1/o3ajxVzAhLKu49ZPKvLuxqs7Ba1KUCIYN5mrUmIpREP48o9h7Q1sWIeFWXZweSzsOGtOKokN8df/1d1e0UB87ruiBIhBBCVOLRRfN1t2/x8/Cc9Xv1XWOKI0+tnrU9/OM9cipt2juxqbjnL2rPGNtoMMaUhCJZZGFBQfDtOyF3QqKio75wDNFvCk+ZaU+wUN9gUXnkxFmF3jdKUhj0IbLgLxOd2Ubcub25c6usJevDKOZt4XFT7Sl6qj/2U/2x/DXm1t2sNYlMSoowxoXtaKN1zEYLIYRo8Z6LuSfqeHdcAyRvVFbsqzjHFoY6mmoqJC0oys9OjH797P6NU6dvva2YhpTJvbrS98f0+TMGdbAz4pESQWE+Pyst8VFs+Y8IVMw/fos1f/5+uEdLHY4wL/F1bBbGCtcCAAAA/jvo9xc2buMM6+HWtpWlkQaH+X979x0VxbXHAfzO7C4gS11AKcuistiwoEYswcSuCMaaWFCTmGii8ZlqookaNY1oYowafbEranyJEQtiAxUBI00RFJEqS0d62zo77w/ESNulN7+fczxHhtmZe+/+dpjf3jv3youzJbFh/t77DvnGlLCksX9M2dKIn99cGPvOsjenDu9rbaKjKspKvp9YzKOqT1cue3hw3ZaRf2589ZP104KXn62+2DCT4bdh3sMrry/ymOoyRGxlxucpynIzkuPu3rocXKAm0pj/Ll2Q/M6Kd18b6WhtoC54HOF/6rfnFukuD9j+yW8Gn7/urJuWqSCEyO/vWPZe8ScrPcYOEJnwWEV5YV6WJDEmIEHaEvcAFJV550aUzVAHGxNdSl6UHh9xyeu3HT5PWuBUjcQwTGpqmr+/f0DAjZycZigYk+G3ft7Dy28s9nB9eYjYSsDnKMuLczMliXExN//JUhPtN2+a3yO6ribtWvt2tto6b819E6i9vppRunx9DsVKy+Uazs9KpVJCGRgZ0tVzwrJQ/9ulbqPHju36x/EGPHbHE9paau2sYwvCdiyZcX2qx5szxgx3tOtqyJEXZDy6G+Rz0ut/wen/9sbqDZrwsimb+Oe1WgdItxGpVBocFHz9xo3o6GgNk5e0KFap2HMlP64f/zVbHQdDugvFFpWpYnKYp1dAleqPa/mJvfnzeuj0NeLoU6xUrs4oVsWkq2qdJYpVKvb55Sf248+y0xHzaR6rzi1WpigIRYi6sHxzMLVqYJfBxjSXUWfkqRoxVRNlbGbx7IcLF3wIIaGhYTt27W5M1durY0cPEUKCAoN+8PRs67IAAADUYu2aNS6jXQghCxe/rXVnaD9a7R5DIBDk52u/0+us93KdgMG476795pa5ffas3xPbInmiTCb/6Ld9QvKWGfMO1TW37FPOzsNWrVxBCPnB07NFn3kzMjaSlkmVKi0rp7uMdql45u14lkV0iX7Llae98bB8MsCwnBDi5ub+bGOzPfPW/o0YOWLr1i3L3ls2bvw4kZ2Iar8j2QEAAACqqE/mBu1Z6c0jx2KJ48J3x9dYBq41cB0WLp1gkn9l/+nU9tPvVlxUrDVzg2peoOk/JCmSrMyswYOcprm70zRdWlqakJCQkJBY8S8zK4vVMFEyAABAw4lsbQ0MDZOTkzUs0gUALwRV/OGfvefsnb1m5Zl/vgupOWdrS+L0mPfFUkdlyHe7/fDsUQf3AiVvGRkZP2/bRgjhcrnWNtZisVgsFvfr13f6jNd4XJ5UKk1OTo5PeCotNa2txt0CAECnIbITrV27lmXZvLzc+PiEpKSkpOTk5OTk7Kzsti4aALQytjj41/XH7RYVsLoUad3kjccrS4vx81t/UsuASWj/XqDk7RmVSiVJkUhSJNf8r5GquZyDWOzq6qrD48lksqSkJORyAADQFGlpaYQQiqLMzS3MzMyHDXuJy+URQmQyWUpKSlxcfHJyclJSUkpKSluXFABaHlsY8N2SgDY4sSzO++v53m1wYmh2L2LyVo2WXG7KFB0dHblMlpqWJpGkxifEJyQkxMfFK5VNGqE7ZfKU4FvBJSV1Tg7ap0/vmTNmNuUU0Cl5n/GOjW3tKXSbApEM7VlM7MOzZ8626Cky0jNYlq14ypqiqIrMjRCip6fXu3dvBwcHihCKplmWlcufzjtN0zS+LgQAgFoheauuWi7H4XBshDbPcrmXXx6lq6urUqkyMjISEhKf5nKP4hv0tCVFUUveXbLknSVHDh++eOlSrX+kzS0sKqYdA3heYHAQ6VDJGyIZ2rmzpGWTN4VSmZeXZ25uXutvafrZtGGUrq5Oxf+QuQEAQF2QvGnBMIymXG7USF09vWq5XEJcvEJjv1y3bt34+vosIctXLHef5r5r128PHjxorQoBAEBrMDU1tbW1FQptFArFs863mhgVQ3Pof279o6ujM3TYS61cSAAA6FiQvDVMtVyOpmmhrVAsFtuJRCKRaMH8+YaGhgzDpKen/5vLxScoFFUWYRc7iAkhFWNohELhli0/hoeF79q9+0lOTs0z7ti1OzQ0rFUqB+3XsxVXOi5EMrQrFWtzNRcul2tlaSUUCYU2QqHQxtZWJLSx5hsYEELKy8ulMqlKxfB41f/gqhmG5nBiHsbs27c/MTGxYhUjAAAADZC8NYlara7I5Z5tEQgEYrGDWGzv4CCeP2++kVH1XC4xIdFBLFaqVDwul1SOmRk82Gnf3t///POvU3/9pbnXDgAA2hbfwMDK0tLSylJkK7ITiSytLEV2djo8HiGktLRUIpEkJyfdunVLIkmVSFKys7MnT578wQdVvnxh1SyhqcePU/YfOHDv3r02qgcAAHQ8SN6aWX5+fmhoSGhoCCGEoihra2uxvb3YwcHevudw5wV8AwOGYbJzcrhczvOv4nC5HELmz583cdLEPbv3hIaGtlHxAQCgCoFAIBKJLC0tRXYiO5HI0tKyW7duFEUplcq8vDyJRBIZGXnx4iVJqiQ5qfbF3NLS0p57to1lWTY7J+fQ4cPBQcFYXxQAABoEyVsLYlk2PT09PT094ObNii1Wllb2DvYff/ghRWp5+IGmaQsz8w0b1kdHR4eEhLRuYQEAXnQ8Hs/K2kokEll2s7SzsxOJbG2FQl09PUJIaWlpVlaWRJJ6925kVnaWRCKp/xIyaampFf9Rq9XFJSVeR72uXr3KMFhsCQAAGgzJW6vKzMpk1Cq9Ll3q2oGiKUKIYz/H/v0dW7FcAAAvIhMTE7epU21shbZCW6FQaGFhTlEUwzBZWVmpqWmRkZE+Pj6pqalpaemlpaWNPkthUVFZWTmHpk7+789z587J5fJmrAIAALxQkLy1NrHYQfMOLMuyrJpDP10LqGfPHpjmAQCgJfQf0L9Hzx6ZmZlZmVlXr16VpEqyMrMkEkm1Waaabv+B/f/c+kfD2p4AAAD1geSttdmL7Rm1mkPThBCWZSsmia54HEKpVObkPJFIJOnpaXw+39XVlRCSlJTcxiUGAOikQkJCN2/e3AonunL5SiucBQAAOj0kb62tl4OYQ9NKhSI7O1uSmpaenpaRmZmRnpmZmZGXl/dsN5fRLhXJGwAAtBBlc/ewAQAAtCgkb63t8OGjv/zya35+flsXBAAAAAAAOpJakjexWNzRlwNuzxITE9u6CAAA0H7hTzA0He7loClMTU3bugi1czEuHsgva+tStB5Rl1omuKoleRMITJ2dh7V8eQAAAKA6/AmGpsO9HHRKdrUlMy8aWvsuAAAAAAAA0Naq9Ly5ubm3VTkAAABeZD94ehLPti4EdHy4l4POJygwyC0Qgf0Uet4A6oO2fu3bc1d9Nrnw6tiBYzfrh3NXTn/pjEmAoOkQTo1EmY5Z/4dPoOeEti4IAABAi0DyBq1Jd8iq/90J9/1xkhnV1kVpYGEovk3vvkITvbp3NbTt18/WRI9qBzV7ITQxltpVKNYC4dQ4lJ51vwE9LPSR9AIAQOeE5K3D40/bGRsb7rPa2aT6bR7vlW8DE2MurBnIaZOC1YqiKIqi6QbckfInbglMjA0/OrdbLcHKc/rySlRS9NG3hY2J5IYXBloSR7zy9L2kqD9X9q61e5M/ct3FhNg7B2YYVvzcxLevxd79FozYFsWftjP20b3zy+3b6HrBcXx79yX/ox/0bkfXKwAAgPam3d1AQGNQXRyXbNvhYa/T1gXRRh7x6xuDh05ZfTmPre9LyoJ9A/JZPWf3idY1olV3iNtUIS2/c/FShrpVCgMtiTbvZkFTun3f/djdssZ7zXGY//nrthyKY2ou4BDS5Lev5d79lovYzo02sevnYGWogy9TAAAA6obkrXNgGdbI5Yvta0cZd747n/IQ3ys5ap3Bbm6ial/J6w13n2BFy0J8/LI65p0wrWto0c3CtGMO8bK2tv5649djx4zV09NrniPqmnczphSFxZzRy5YOqXpMymTS+28OkBcWqimBwFRDkNfVpK3Z1O02Yjt0vAEAAABB8tZZqO557byYZ7doy6YZ1nUNOuK9vPFGYoz3x32e7UAZz9z96NHdw3MEFCGEUGYvL9u29/gl/5tR9yITYiLu+B3btmRE76Gzv9h21D849NGDiDtXjnh6DHh+fCbFF7t/vM3b/9bD6Ig7fid2fPCqkFd5cKe5G345cP5KQHTUvYTo27d9NrkKOA7L/4p/GPTj6OfGxXURTVj+/cmLAfej7z4IvXbFa+MMu6pVkIafvZKp5vWb7lZ1QBd/5Izx5lTprTN+uSwhlNmYL494B94Oi4uJehRxzff3L2b15leWtF6F0XiEij10es1Yd+jstejoyJjbl05t+2BK9y4a3pW6G4fQgpfe2346POKf0Js3Iu6E37uydXbNbpr2jcvhOA8b9tnqT0+e/GPNmjXDhw/n8eqazaVeaIG5Ga3O8d3jlWD1xvJpz7cH12Heikm6/+zef1tOm5qb0IQQUv3tq6tJ69he7eXVgj/yfvC5Y5vmD66WJ+oJxy795pjP9aioqPio0HB/7z/3fLO05oDlZojYarR+eAlp5nhr6NWAEox8Z8ueo75+gdFR9xKiQ8MuH9/16fQBJs/OUq8qEE6vVWejkh89SH70IDFw/SielnoRQmjBII91e3wDbsfej7hz9fjOFS61DVcFAADoJPAVbCfBpF9c+6lhj4Nvb/55yaO398XIGnEMWjBw4rRX+1XGBM/UdvDMLw7MfG4PHbuX5q7bIyib/f6ZbDUhRH/AygP7PhpsWHGzpGc7aNp/djhZr5q+LqCApbuOnLNo6rOjGVp05clLa5xTt8/S3w+sGV55i6fTTexkayCt1iuhiDjjk+jxfi+3qY6/x0WpKjZSxq+4jzUlBT5n/QtYQghRmdgP6SWsGDlq0K3vmMVb+3dVTv/sfC5LSP0Ko+kIFefkO7nPqWwL26FuKwaPctq0cMXRBGUtzamhcSjL1z13fv6qEaUqy8supQzMTLry5EUds/eQEB6PN2rkiNGjXWQy2e1/bgfcDIyICGcYpqHHoUwEpjRbmBNy9ODNBd+/vWTwuW8j5IQQQhmNX7agT+75t7wfub7L6gnM+BRRVBvsSNfRpHVtJ9W+46gW/IRvbv/yvK+cenWZtehgXEXI6fVZunf/GmfTysfk+GbCXmbCnjp3Dh4MLaxa26ZHbMM1c7w19GpAmzlNmTnu2f5c8+5ObssGTZg87KNFGy5lNyG2NV1kCGU0at3RnW85PJ1ISFfkNFVECCFYwxUAADorfEXZabClEbs++iVC7bTil4+HGTZ69CRbfHHtpEEDB4oHuLh95ZvGsOrCsF0r54wc6tRryMSFuyOKicmrs8Z1pQkhnF6L16900s8J2LFk6st9HIcOn7PuVBIjnLHSQ1x5Z8yWXP3a/aXBTuKBI0e//mtI9RyH08Nj/SfOxrK4M+sWuQ4Z6NR32Lgpi3+8XOPmlYk9/3e0gu7uOt2p8rE+ynT8ay7GbLbv6eCSp6cK3rp4xshhQ8V9B/Yd4b5kf5TMbOzsMc91nWgpTH2OoEj2/XHJtFf79R8yeNI7my+mqExGrv7MvWstra2pcSjD4ZOGG6rv/z5z5MiXXhk3dKjziJk/BZU37I1qVzhcLiFET0/PZfTLX3+9/tgxr/feX9bPsR/VkMkSaROBCcWWFBXnXDp8Kt1mztuTzClCCOGIZr470SD6+LHbpcVFJSxtKjCrcd2qq0kb1tSVwW/vONzFw/Nqlpo/yMNjSEUvD8d+4YZPnU0UST5fL5riNGCgeMBwlw0B5XXkWc0TsQ3QMvHWgKvB0/19Px/n6DjAvv8Il7lf7Asv4NlN//bz8Q2oEhO3Y/rAHr0de/R2tB/9zS2l5osMt/87axaLdYojvT6aM86x/+BB4zw+2h+W21G/BgEAANAOyVtnoojz+nLztRLxom+/auQtICEsU/Ikp1jOMIqCGO/dxx8wFDc3JvhhVqlSWZYRvPfA1SKWY9tdRBPC6T3NvQ+32O+71XuvJxbKVbKcaO9NO66XchxGOZs/DSxWVZCelleuZOTFGSnZZdXudDk93af111VG7Vi14XiopECulBVnx92Ne1Lz3ouRnDsdLqWt3WeN0CeEEEJbTZ49iq9O8T0VVtnJSFEmA+Z9f/Dv4JCwqOtHN0625hBuNyvzf0Ncc2HqdYSysNN/XI/LlSrlhSm3D63deDJdzR8xcXSNcXNaGodlWUIoiz7Ofcz1KEJY+ZPktMJOMWsKl8sjhBgZGbm5um3dsuXI0SNjx4yp52t1jY31aXVpSZlaHnn02F2dMYvnO3AI0XNevMCp3H//qccMKS8pZSkTQc32JnU1aYOaujL41arS9PAT33vdV3HM+vSxoAkhnJ5T3Rx1mNj/frzuaGhqkYJhFKVP8krrfNOaJWLrr4Xirf5Xg8r9S/Pzy1VqtbIkPdLnhxUbzz4hgvHTxzT6UVzN9eL0njyhOy2P2P7plrPR2eVKRXF65PljVxIa3OkLAADQYWDYZOfCZJz+etOofr/M2bQ2IHp9WVOPlpmSoSJ9LbqZ0KRcTQghiqy0HJbqqt+FIoRn21NI010m7wydvLPqy6yFVjTJ1X58rp1Dd446NfSWROvdljr70in/j0e4TZw5fkvg+UK657SZw3RVD7y971eMSaOMXll3eP98O97Tu0RdkS0hhKHpekd4I44gjQqJViyaZNPdmiYFVX+lo6lxqJJb3v65Y91e/fKo36cFKfcjIwLOHTt4Kb6WfLKGtWvWkDX1rVMb4nA5hBAzgcBsxIiKLUKhTWhomIaXGBob0qyirExBiDr1zLHL721bsGjkoR1mS16zTP3rC79CllBlpWVq2s7IqEYuwNbVpI1vaiYj8XEZ62hgwKfIs1i9daPWIbK1aPmIfV6LxdtzR9J4NagNW3TL/45ixgSRvQ1NChtTLc31onkW3W1oddqd8Ez0tQEAwIsCyVtnw+Ze+2bD30N/n73xq6Ct0qq/ImpCdPU0rDNdnVqpUBGKx+M9e4lSqWIJRdHkaZ9GbSjdLrr1OkfFKlt1HaYqtijgxIXMqR6j5021unCq6xuz+nDLb50487gi7aMEE96aKeIUhOxav+X47cQnUq75+K/ObH+tPkcmjT8CRdEUIbUtFaa5cdjcC18uKr37uuuIQUMGDxgytsfQMWP70LM+uKA94/U+cyY2NrY+NWpp5mZmS5cu1bCDSqXicrkFhYWmJiaEkLS0dM0HNDQypFhZuYwlhLDFAYdOp7h7vPUZK3hVJ/KHP6IUhBBWWi5jKT1DQx4h1XKoOpp05YW6thfUVoSqh1QoFGzFW0wIzePShKhU9e/Uad6I1fLhbVS8rbzQoKfrNF4N6iqWmn32CW/49UdbvSgOTQihsFIjAAC8QJC8dT5sYdC2r/5wPjx/9UfZ+hQprtyuLikqZWmb3mJjKrI51rZSpj9OV6uNz747af31Wp6fqcdKu8r0x+lqWuQ80pYT/VjrXbEs7H/esfM+cJ4/Z0SBzSwRlXfuT9+cp/WgzS0tdUj5Va+dfrEKQghR5j0pbtCkBY04AmU8avwQHaJISUpXV45A5nC4/1at7sYhRJYa4LUtwIsQjmGf2ZsPbpw4ZrIzueCrtZyxsbFBgUENqVlLEdnaktpyN5VKyeXyCouKAgICgoKCBALB2jX16is0MjKgWHm5tOI9Vd4/+UfYoi/fnMsWXFx9Jq2ia0UhlbEsxTc0oEjNVq2tSfUv+JbVvv1yw2qrzMrIVdOil5yt6fup9eznaXrEVoaT1g9vo+JN/4JvUzvnNesywLm/DlGkP05XE0K0XX9YlUrFEn19/edyMc314vRLTFPT3UeNsf8tOq6ePaIAAAAdG55564zYklvbN59INba2fv5bbiYp+mExq+vy/toFg7vpc2iOnqGFaV0jnuqBeXT5apLafNrGLe+M72dppMOhOXqmQscxw+zq+5UA8+jSlUSGN+jDnZsXDu9uqsfh8Awsezv1rjkfRcXuCaeP35ZyxPO2r5skYCXeJ4NKKn+lzsvJUZIuw2d5DLU24FKE5hkY6DXom4l6HYHiGJqb8Xk0zTWwGui29reN0y1I/rXz14tYQohCqWIpkyFjRgj1OVoah9Nj3OxxA22MdGiKw+OqSkrkhDRkao/2iGFUhBCpVBZ4M+j5b+BgAAAFqUlEQVSrr9YtWrho7+97Yx7E1P8I+gb6FJHK5E9v7NUZF7yuFaqZjLMnrhdUbpOVy1ja0Mig5hretTcpVdf2hlZP9fDqtUy17uAPf149zbGbgY6uqd2wWRP76mh8UVMitko4af3wNiremj/iKL7znAVjHcy6cHlGtsPe/H7TfCFdFnI1sIitx/WHzcl6wnKsJr4+sYcBl6MnsH/J0ZporBfz6LzPQyW33we7flw62l6gx6E5esbmpvod/KMEAACgAXreOie2JHTbd97j/jtH+NzGskAvr4cT/uPo+u1J12//3axo7ElU9w98d3DsnqUTP9k/8ZNnW5V3t0xccCSlXp0TqgcHvv39ld0r+s/45uiMb54Wvez8qldWXaltrQN19nmvyx+OmtnNnJWFnzx279+Ss3nX/vRfOdpt3IYT4zb8+wImrt6VqdcRKCNXT39Xz39fJE85u/7HqwUsIYRJexhbyPbps3jP5a6fDfvwkobGkZgNf2fT0zWsKquWf/GKpufB2i21mqUoolSqbt0KvnH9xt3ISJVK1bhD6fO7UKy8/NlYX7bo4icu9p88vwsrlclZim9kUP21VB1NWm42vtbtDe9xkoXt/enc+J9mDFq84/Ti57ZrrGzjI7Z6OGn78Gr6MNYVb83f7UbpdJ/y+cEpn/97nsLbnj/5VHQ3aqsCIwm4FrNqwMBZP12bVVH6yO+nLtqv6SLDxB3Z+NOofWucJ3+5f/KXzxUESwUAAEBnhZ63zootCvz1e9+cKjmU/P6OZe9993dYcr6MUTMqWUluevydmxcDEqSNG0XJloR5Llzw0W6f2/E5xTKGUZblpkQFhKfWPx1kSyN+fnPhqt2+4Y/zyhSMsjw/NSYisZhX11fnpUEn/kpUsepCP69zVQavsfkX1y399MD1+xnFcoZRycsKctLi7oXcTiiqb9W0HEGdd+/K+Zv34jMKyhUMo5LmS6IuHdjwxtz1FyvXsCoP2P7Jb373s0rS0zIVGhuHojLv3IhKyZeq1GpGWiCJ8tv7xTurfZ7Uu9naC5VKGR4etmXL1rlz527d+lNYeHijMzdCCF+fQ7HScrmGd4yVSqWEMjAyrH7hqrNJ69jeiIBXP7n6ucfyLafDkvJkKpUsLyn0rF9MOUvUrKYvKhodsdXCSeuHtxHx1vzzm7Jl93xPB8Y/KVepZEVp9y7v/c/8lYfiKwc0aqsCE39k1epD1+NzyxlGVZ6XdDfhCUVpuchIH+5bOvftn04FPcouljOMSlaSmxoT6vd3QFKzVw4AAKA9oIzNLNq6DFALl9EuFU8K7di1W/McffAicHYetmrlCkLID56e7eSZN11dXR6PV1pac7HzKjpvJFMWb+wN3PxSyPrxb/2V3ykWemgKjsPyk76rrE4vG/tFYMd4/OzY0UOEkKDAoB88PbXuDAAA0E5g2CQANIZcLpfLX6DhaXTXoa6DSPyD5IzcIhnXtOfQ1z5bPlxHnXg3ut69uwAAAABNg+QNAEA7Xad5njumGjw/opdlMnz2nIjDmtAAAADQSpC8AQBoRekUProe2tPJQWRprEvkRZlJ0YHnjuw6HpKDBaIBAACgtSB5AwDQii0K3b9q8f62Lka7xcTved1hT1uXAgAAoLPDbJMAAAAAAAAdAJI3AAAAAACADgDJGwAAAAAAQAeA5A0AAAAAAKADQPIGAAAAAADQASB5AwAAAAAA6ACQvAEAAAAAAHQASN4AAAAAAAA6ACRvAAAAAAAAHQCSNwAAAAAAgA4AyRsAAAAAAEAHgOQNAAAAAACgA+C2dQFAC9fJk0Y4D2vrUkAbMzU1besiNBUiGQAAAKCJkLy1dw4O4rYuAkAzQCQDAAAANBGGTQIAAAAAAHQAlLGZRVuXAQAAAAAAALRAzxsAAAAAAEAHgOQNAAAAAACgA0DyBgAAAAAA0AH8HwYt9a8fcrMOAAAAAElFTkSuQmCC ) ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA4AAAAEMCAIAAAAau0+KAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd1wURxsH8Nndu6P3pohgQaUoVVAQLCAqIKJGjb3EqIkxamI0do1p1sQYNYlRk9hfNTawSxM0UhTsWFBBQOm93t3u+wcWVK7Q7kB+30/+MHu7s8/ezM4+zM7uUToGRgQAAAAAQFFoZQcAAAAAAC0LElAAAAAAUCgkoAAAAACgUEhAAQAAAEChkIACAAAAgEIhAQUAAAAAhUICCgAAAAAKhQQUAAAAABQKCSgAAAAAKBQSUAAAAABQKCSgAAAAAKBQSEABoHGpOs8NvvE4/c7JJe6aMlYVuC04dC4m4cIKF75CQgMAAOXgKTsAAGiGeF0m/vzdRHtzUxN9XS0NVR4lqijOzXiadPvqxbNH9h75L7W82soUTdMURTEMRckqtlU3d4fOvCfnZa3YkGp1LAAA0BAoHQMjZccAAM2NoO8v1w9MMqrxFgpX/uTk8imfb7tZUuti1YftfvBHAO/JloBeS2KE9Y5SPo10LAAAIBluwQNAnVVGLHRq1aqVnlErw7adbXoFTF5x4Ho+p9LO/8c9P/rqK3IYs/7ep2MBAGjqkIACQN0Jy8sqRCzHsaKy/PR70ce2zBk8fts9IWFMh80c1qZ59S/v07EAADRx6FQBoAFxRXGHg5LEhOLb2HWummNOtZp0PD0z/1nIfGum+qqUVpdhC7YERV1/8jT1+eOb0UHbVk10NqpxqJFn6Dxm0e9HIm7ff5KVkZ6RfO/2f2eO/LlmyVQP07f6MBWzPtN/PHA+7lFKasaTu/Hndq2e0sOkjnPdaziW2u1FnrD5bdzHfbnmz4Mhl+MfJT/Nep769G5s5P8+d+bXYl+q7Xzm/nLoUsK9Z8/Snj+5cz3i+L6fJtmr1mIFQgjhG7lOWPFX8OV7j55mpNy7GbZ/4+cDO6q/tY6saAEA5IOHkACgYYnFYkIIRdOMlPvWPPPAXw9uHm2p8nIdky5uQ7u4VRXwxpqUtvPcv/9Z4mnMe1Wchl6bTnptOjn17aMauzsqvfLlmvq9Fu/ZOc9V72V2Z9DeadAnjt7+Hl8EzDj4RNQwxyLnXuQMmzLo//W6hX0Er3fJN7Kw0qeKWHn3xe/y8f4T3/UzeLkK39DC1tBM5/aWl1+jzBUIIZSuy7y//1nUy/Dlcaq07eY9uZvXh6P3zxw7/2jyi/m40qMFAJAfRkABoCGpWvkO7MQQTnjv9n2JjxHxOs3YunG0pQqXG/f77MGOncyNzbo4+c347ujdwrdSGcoocP1fS3sbMxUPj64Y527b3si4dasO3dwWnC3k3lyTbj1yw7Z5rrrC5DPfTehrZW7WqrPbsOXBT4S8tgHfrR7Vug6dXQ3HIude5A+7ivjJnul9u3VpZ2Ri1tbWw2/uwSSxnPvSHPDF/D4GVMmNvz8Z5GxhampsYes8cMIXqw7ffJEKy1yBENrkgw1/LfYwpIpv7Zo31KmTuYmFXe8p60OfiVW7jNm6c469QI5oAQBqAyOgAFBvNE9FTcvQrLNT3+Ez50xwEFBsRvBvh1MkjYupeUz/zEWDEif/NWPMorACjhBCyh/FHF1/i9j7/RFQrVvi209bOKQVw+ae+XrUx3tTq/IcYWFGUnKO8M1MTuA8Y4GvEVUW++O4jzcmVhJCSGlS2NaZ040vnJrVqd+YALMD2yQGJPexyLkX+cN+gS1KvnvvaY6YECLMuB+bIfe+qDbWnTVpIozfu/FgXDpLCKnMSrp6Nunqy5IZWSsQInCc8bW/MS1+dmjOh3NOZHGEEPL8RtDasSncmVNfOdjNWBD49/hD2ZzUaAEAagUjoABQZ4L+G+/mZWfmZ6ZnJN+7fSlo9/dT3IyZyrQLKyYuOJFdY6pFCOHb9/cyYYjw9r5tFwskrUQIIYTXdbCfJY+Inxz46VCq1FE2vmOAX3seVx61e/e9ymrLy+MvRGWylMDWyV6l/sci517kD7veR8TmZWWLOcJ3HDXZzYCpoRiZKxCe3WDf9jwiur//11NZ1auj/MaO30KKOUq7b0BfHbwGAAAaFEZAAaD+OI7lCEVTpOLWX59OWXXyQbHkxJLS7tTJhCFs/s0bj6VnZ5SWla0FQ7jCq1duVMpYs1NnU4ZQaj6bkrI21bCCqpGJLk3K5BkClXwscu6lXO6w639Ez7KCtv/7pedYC+c5J2IDIo79738HDwdFp5a++vo5WStQ2lbWbXmEzU24eu+tabJcwbW4ByI/J5UuNpYMiavLJFoAgJphBBQA6qzywlxrPUNjXUMTPfOBq+NLOUrQyd3FiEgd1qTUNTQIIVxxUbGMfJDS1NGiKMIW5OTJGEekNDRl/MynqoqMEVA5jkXOvcgfthTyHhGXe+7rgInrg+7ks1odvSYu/iMo5s7FnV/1a/1ydEHWCi92xBUVvD3/lhC2sKCYJYTW1NLEtQIAGhQ6FQBoCOXXf569NrqYqHT5eOPCXtJyJ66kuJgQQmvr6dR4S7jamhXl5YQQSl1DXcYdYK60pIQQwmbvGW1saKz77n+m/lueyP2otqRjkXMv8ofdMEdUmXJ+3YTedt0GTvtm96WnFYyu9eAlew4u7/nqFUpSV3hRHZSWjvY71wNaW0eTrloFz7kDQINCAgoADaMy8c+562NKCL/TR2sW9dSQuB5XdC8xVUwoLeeeXaW/P5LLT07JZwmtY2ffXnquyhU+fJghJrSuU/fODTGxqOZjkXMv8octRa2PqOL5teM/fzHMxWP63iQhUek8YVJvNXlWeFEdtLaDU5e3dkRpO3bvxCNc+b3beM4dABoWElAAaCjCxG0Lf7leQQSdpn7/SVeBxNWunzzzRER4HcbOH20hNbmqvBpyMZcl/K7jZvSW/hiMMP5c6HMx4XUZO9u35pfZ11KNxyLnXuQPW1oAdTui8icntx1/JCaUurFJjft+ZwXh9ZNnHosIr/OYzwYaVt9CtevUT7w0Ka4wIigiX+qsCgCA2kICCgANp/L25mV/J4mIqt2nq8a2ldS/VF79/ceTmSytP2DN0f2LhnU31xHQFKOi3ap9G903c6bC879vv13BMeaTf9/z3UjXjgZqKhrGXTxGL/6kj/aba5ZHbf05qoBlTEduPvLnLD8ncz1VHkULtFp17jFkwiCrOgyL1nQscu5F/rClkG9fmr2mL57h79zBUINPUYyKroXrqE8C2jOEzUtOzufkWYFUXv19zalMljEd+ev+nyb0bKcrEKibdPX/aveeLxxVSfmNbWuPZyH/BICGhafgAaAhlV7Z+H3QBzuGGfb54sv+R748V+OL19nn/3452Vz378W9zXzm/eEz781Pq9/trbzx86dLHQ7+MNC0x2e/BX/2VjEc97p08eOdn37Sft/vnzp0GbHy7xErqxcSu+TiucTkWk9jrOFY5NyL/GFLIc+++NZ+0z//rN3cNW9sybE5ERt+u1ROiOwVCCHs88NfftTO4J9F7vZTfj4x5efXq5XdP/DZRxsTKuQJFwCgFjACCgANissJ2vB7fAXHmA6fM9pcUhfDFcT9NKq3z8x1e0NuJOcUVYg5cWVp3vNHNy6d2r1la3C19zNVJP4z3mf4vD/OXH2SU1IpLC9IuxW2b/Xv4bks4UpLSqqlcmxmyBK/voELfj9+5f7zwkoxKyovzkm+ffnE7uMJpQ12LHLuRf6wpZC9Ly4r6n8HzsU/ziiqELGsqKLw+cOrp3asGDlgwp8PhHKtUFVMfsy6EX395/8eFPsoq7iysiwv7Xb4nu+n9Bkw90iyxB+0AgCoM0rHwEjZMQAA1ArddtqR2B/cSMR8h5G7njeb28PNNGwAgIaHW/AA0IRROr0mT+2c+V/c3ZT0jKz8Clq7VUcnn8lLvu6pSgrPHjmf2TTTuGYaNgCAoiABBYAmjGc7ZM78GWbvvM2Iq0w+tvjrg8+a6Ospm2nYAACKwqiqS35dHwCAcvFUNTRVBDRfRU1Vlc+nucrCzCc3LwXv/PHLzzdEZDTZl1M207ABABQFc0ABAAAAQKHwFDwAAAAAKBQSUAAAAABQKCSgAAAAAKBQSEABAAAAQKGQgAIAAACAQiEBBQAAAACFQgIKAAAAAAqFBBQAAAAAFAoJKAAAAAAoFBJQAAAAAFAoJKAAAAAAoFBIQAEAAABAoZCAAgAAAIBCIQEFAAAAAIVCAgoAAAAACoUEFAAAAAAUCgkoAAAAACgUElAAAAAAUCgkoAAAAACgUEhAAQAAAEChkIACAAAAgEIhAQUAAAAAhUICCgAAAAAKhQQUAAAAABQKCSgAAAAAKBQSUAAAAABQKCSgAAAAAKBQSEABAAAAQKGQgAIAAACAQiEBBQAAAACFQgIKAAAAAAqFBBQAAAAAFAoJKAAAAAAoFBJQAAAAAFAonrIDaIkWLVyo7BDgPffj6tXKDoEQQgKHBtpYWSs7Cmii7iTePX7suLKjAADlQAKqBB6eHsoOAd53TSL/JDZW1mjtIMVxggQUoIXCLXgAAAAAUCiMgCpNTEzsps1blR0FvFdmz5rp6uqi7ChqMH7iFGWHAE3Inl1/KTsEAFAyjIACAAAAgEIhAQUAAAAAhUICCgAAAAAKhQQUAAAAABQKCSgAAAAAKBQSUAAAAABQKCSgAAAAAKBQSEABAAAAQKGQgAIAAACAQiEBBQAAAACFQgIKAAAAAAqFBBQAAAAAFAoJKDQjjMXwH0+cO7LYlafsSGpEmw757sT54G88+BJWaOLxQ8tC6fVdtj84cnV/FWVHAgAtEBJQUAAVp9n/uxZ3as0AA6p+BWm1tbFpq6tK1bMY+dUqckqjTRdrM11VyasqPH4AiShVU5tu7Y3UeRQhDXiSAgDIAwlok0drWw2asXrbwYv/xdy7k3D7v7PBf69dPM7NTPaoBWM7ZeuZkF2fdWEUEKZ0FEVRFE031pVNw2dtZFJi3K4PTWpo0HyHxeduPLq5a4pZXVp7I0cO1TC6tv6frN52MOJydOLNqwkXT/zv5/nje5qqNvJuNQJ+Tbx3PejTjm+dJ5Txh3tu3r6/Z7ypXA1HrtNNI+DXxHu3H7/zX+KmQY19mDKhqQOAIuFWYJNGadt9vP7nBb1b8V5eFQT6ZrZuZjYOGvdOXUmt4KRuTeta2HRqnSdQ/hWl4uovoxx/abzySy6disgNGOo62Mf00J5U9o3PVJz8/czoitjTZ9JZCZtL0diRwwuUbvdZG9fP6WnEvGyuKiYdXf06ug4aMfbQik++PZMiVGp8sjWd061u0NQBQKGQgDZhdOvhq7cs7KPHZcfv/m3bgdD4pKxKgZ6ZtbPnIKvnlwqkZ58tS2n0qXOZQ8Y4+vub7//jibjaJ6o9BvdvTZdHBl94Xof8swmgVbQMdFVFRXl5pSJlx/LahAnjBQKV8PDwpKSkBiiOaTt2w6a5btriZ1d2btl+KOx6cgGrYWrtGTB57sfe1qN+2Fb4fPj6hNIG2FNTILq1cfjQ35LEstcEAHhvIQFtutTdP/mqrz7JDl085ouDKS+Sj4rMpJjTSTGnX6xDGfRd9NMc3y5mJtoqXFl2UtzZ7T9tPnqv5HVyynSeffzGbEIIIWzmgQle314WEkrD0n/6zKmDe1oZq5Rl3Is6+sfabRGpr0aYVM36TZgxNdDDztxAjZQXZKU9un/r/F8btsfkvyhWrd2Ajz6dNsTdxlSDzUu+Gnp465YDMVlV11NKx2HUnEkDXGwt27XSVaPKspPPfjNx1cMPD5ya3frI9H5fR77cjZp5/8mffDykV1czbaosL+3exa1Lvz2WLJZ9RDUqizt+7tmHE20C/Ttu33L/9aVdw22otyFVHHbsQjYn4+uSK3LZ4VGCzkOX/vWVZ/cO+kzJ8zuXg7Zv2nnmSZmkwKXUBa3ffdryxTP6d9bjUxwnLEo5v2ry1//WZRy34RkYGPr49B8+fNjz589DQkLCwyPS09PrXJpmn0/nuOtwGWfmj11wPP1F7VUmx5/YnBAZv/jgH2M6j5878sBH/6SyhFAGvaYtmdTHpmNbU0NtdT4pz0tJuLB/w8YD8XnVKkF6C68zaS2fEFLz6VYbch2dpHNHZni0vv2YmZ+M83HsYMAvfZb435WCanNWmE6fVm/q8kUioa/4Mya/dl8sALQ8SECbLreA/sZ0ZfyOdf+mSB76Eul2dOpsJiCEEKJpYt134rquxsLAr4KyJSds6t1m7fhzrqNW1aVHta19wOebHExnBy6NyOMIUbWatm37Qle9l1PBNAzMOhuYdRBc27kzJl9MCFG1mbFt+wJXnRdXLpPOfcYscu9tP2/8oqB0MSG0sduICX42LxuWlpExv6L4nRhUrKb9sWNhD90XhQhMLB3aapaxdTwiQgipvHosOGncJ539/Wz/uH/jxRdG6fQe3E+P5AUfD6m6akorXL7IZYZHaTgMHvHi34K2zv4zHd0dvhk/c9fDmjIRKXVBtRq5+tcFfbQpUUlORjGlaaBrzK8oaBLZZxWWFdM006pVq9GjR48bNy4tLS08PCIkNCTjeUZti3L362tAVcRu/yko/a1hQS7v8paNF/w2DXIY7G26+59UltD6dj4BfV5VE9Ew7Nhr9BKHzmrDJ+y8X1Xv0lt4nclo+Q1CjqOTdO7ICo/Sdl+669fJnV48I6di7uBnTgghFXWORHJfgQQUAGTCQ0hNl01nTVr8+GJUmpSLG1d0ad3EoW4uzpbWdtY9B3+0/Ua5Qb8P+uq9nocmvr8p0K59F9v2XWw7en57Wch0nrhsloN6ZsSmj/x6Wdk69xix9PAjsdnQWeMsGUKYjuOXz3PVrXwUvGLCIIdudpbdengsjyitNqBqOWHZFy7a5XcPLxjlZdvV0XHAtB9Dn1GmvisX+LzeK1d0fsXg7o4OlnZuniN/iX479WLaj1v2patO+f1jSyf4Otk5WLt4DZq45mw2J9cRSSBODPr3ZiXdzjfQQfBiEaXnPcRDh8s4deRSkZxfl4zI5Smh8vGpNR8F9LHp6uQ4YOqq08kiXbf5Xw02ruEApNUFpdVjQA8t9tYfw9zcuvf2cnZ27TlsfVSTvAnNMAwhxNTUdPSHo3Zs3/7zzxuGBA7R0daRv4Qulhq0+GHkpZpmSXAFlyNviijG0qrD66d7uMLTiwbY29l1tO3hMW71+eeshv24cU5Vb7+S3sIl4XWde+Lhmw8GPYpc3kvwagX5Wv7bp5tc+3oUv26goNrnUo9OwrlDywqP13XqwomWgsKE3XNHeNl2dbT3Gjd3e2y29L9opEYiq68AAJAGCWjTpaVBETYvN1/qJYKidLuN/mHnv5eiY2+E7Vo50JQhPJPWhhLrlekSMNiKV3jh+/nbwpLyK0TlmTePfrMprJjp5O5qSDMd/PxtBeLE379YuivmaUGlWFxZnJVTXC3/7DQk0FYgvPnrvFWHrmeUCivzky9v+2rFwWecXt8h3q+uw5woLy01p1QorihMT854+/Y502FwQFcV4Y1Ns5fvjUnJqxCWF2bcj7+fxdbpiF4Rp5w4EldGmw4e3lOdEEII3XrgB+4abPKpw7Hl8n5d0iOXq4SS2CP7w+5nlwkr8pOv/LVo5YE0VqOnj6fuOxmo9LrgOI4QysjK1cpQlSKEq8h6nJrfhK/uFEUxPB5FUZ0sO037+OO9+/b8+OMPxibG8myrpU4RtiCn5vFdriQvr5yj1TQ0Xt+v4cRFWZmFFWJWVJwWt++H3bdEjIGVlRFNpH+rrWxmHXmd+d058ZWt3C+IkLPlNwhpRyfh3KFkhcd0Gdi/HV1xdeO8tcdvZpQKKwvTEoL2nHsofehWaiQy+goAAKlwC77pKikjhNbR1aFJpoSrBKXde+nf28dY8F9c/lTM2xJCxDQtuVoFbTuY0bTawF9jBv76xgdiU7PWNM+wUzuGfXo5vMb7xYQQgYWlGc0+jb5U/UGfkmuR8eVjBllYtqVJrhwHxrPo1I5hn8ZcTnnnuOpwRK+xGWcOh3zR099nmPfayKB8ukPAMBcV0e2jR2+J6l94XUsouxF9s3LCgDbtTGmS9+ZHUuuCKrp8NCS7n3+fxbsuzMtLvpVwNeLEnp1nHsiaDPvCyZPB8h5UneTnS7zH+upNPnZ2dq8WqqmplZVJnAhbVMoRWsdAhybZ7zZ1SkNPT5XiSkskPYQlTk96UsLZampqUET6t9qKKZEUQg0PBlHGH+4OWe5a9T8N0vIl70uyN49O0rkjM7xS03ZtaDb1WtyzOs/iqDESyX0FAIBUSECbrgePy7gu7Xu6GG19UPMD3JR+/8nDzJm86M3L1u69kpRVxjP0XnJs4xBphXKchASGUlFToWg+jyZEJJJ8aWyIkZ6qDKWmQOpyRNVwBRH7Tj7zG+c52q/1ycPGo4Zb8Uov7zv24ppcz8LrWgJF0RQhNb1dUXpdcNknF08ojh/p29PeybGbU7/2zn37WdHDPzuZLU+oP65eLc9qdebj4+Pk6CDpU45lOYriWLYgv0DfQJ8QIiX7JITce1jKWXX0cDP5LemdZ6wobTePrjxO9PCexIyNq6ys5Kq+Z+nfKi9xzXDLzVICkUJpb1d64+gknjuywqMYmhBC1estn29EIruvAACQBglo0/VfyJXCgd49p80ecH7pmawaUlDasFUrASk9v/vXC4mVhBAizMkqrPZIAScSiTiirq5e7aIjTHuSxrI6xz8esCzs3QmFPMf0bJY27+5qSt96WlPSW5mclMrSFj16WTA3H7289Gg4eTqqksqUR6msXJM6hGlP0lja3NWtLXPzjVcmyTyiKgwjsdmWx/7vaOLoz1zHjOiZ12a4OZVz4uCpTK42hUtThxIoHXdvJwGpTH6U9urLeRm/9LoghJQ/jdj9U8RuQhgtqw9W7Vzp03egKzl5Sp5QoyKjanFgtdfduXsNSzki5liaoh48fBAWHhEeHv7ZpzM9PD1klvbf6bAc/0CXaV8GhH59/I0Heig9t8/m+uhS5VdPhsj3/L/Mb7VuZLd8qobTrcFJOndkhleZkpTK0u3c+3bccvN+Q4xZCp/L6CsAAKTCHNCmK+/Mb9tvV9CmQzYe2Dp/mEtHQ3UezQi0jDv3GPzJF8OsGcLmZGYKiVqP4eOcTTV5FKH5mpqq1XIzLvN5Fse09hnp016Tx6jqd+xua0runT3/iDUMWLl2qrdNK20BQzOqema2fV0seIQQ0d3zoc9YFcc5G+YH2JpoClT0LFyG+1i/fkBCfP/EiTuV/G6f/7RshJ2JOk+gY+E+fd03o1pT+RFBF3Lluz0svnfmXJKYbz/n11Xje7TTU2UYvmarLg5dDGhZR0QqhSKO0nXq29NMvebZe+KHR/ZeKWMsR29cOkCfSzl6IKro5UcyC5dJrhIoRsvQQINP0zzN1nb+i7asDDQiuaFBYQXc2/GLpdYF097rAy+7NtoCmmL4PFFRUQUhTfYnPMUiMSEkLT1t1z+7Jk6c9MUX804cP1FYUCjn5kXhv/9yuYBqNWjdvt8XDnfpaKjG56nomtn5ffrzoa1jO/GED/ZuPChnliP9W60z2S2/ptOtwX+DTNK5o/tARnjie0HBd4U8m882r5nm2VFflaEZVR1DvbpnyzL7CgAAqTAC2oQJE3+b/bXx1u/HWXvOXO05s/pHotv08RN3H4ceDJnl6e+1fJ/X8tefie+//EdKROid2d3shq8PHV5VYMIPfhO27/h+Z7/fpvl8ud3ny9e7il/rM/afZLY8dtv6E97rh9pP3HRkYvX9vSr8we5VG3tvn+8yct2hketeLOSEaadXrjkrZ/5JiOj2ju/+6L11Zteh3+4a+u2LMkqCZveefV7GEaXeTcznrKwm/nbW+CuXOWdqGOFiM4J2n53jPszEkCuPO7DneuWrT7gc6YXLJlcJlLbv6hDf1zfAuYrk48vWnM/jaoj/luS6SDHoMfWbZe786oeWe/pcrNzBNjqKokViMY9h0tLSQkJCwiMi6vACphfEKXu/mqO/cf3sHu4zfnSfUf0jriTx0PIZG+PlHs0USflWk+s+VCez5dd8uv2Z8u4ueV3nnng49+2o1waM/e2RzDAknzuywrv/z8r17n8udB24ePvAxdVKrNVNgGpk9hUAANJgBLRJE6dfWD56+KTvdp+59jizsFwsFpcVZiRdjzz85/5LeSzhck8vnTZvR9it9MIKsVhUUZKXmXr/evSVhy9+JUn84J/Z8/8Ke5BdKhaLSnMexT/MoiiuKHb1+LFztwZfeZBZWC4WC0uyk29ExD2tytTYrPMLxn269kjso5xykag851HM8Qt3SjnCci+vo2V3fp82duavJ+OSc8uElSWZ9y/u/3H8hwtP1OZViFzx1Q2Txs/eeiruSU5JpVhYmvv0ztWkQj4l64hKIzZ+ueXCredFaanPKiUUXhy171CSiGPzL+w+8caomazC5YhbeglszvVzQRevP0jPK60Ui0VluSk3zuxYPurDZaczXsTxVvxS6oKinl0Lv5GcWyZiWXFZXsqNC9u+njo/OEv+L7mx5ebmHvn3yGeffTZ9+oz//e9g3bNPQgghXF7spo+GDpv/279Rd1PzSitFFUVZj+PO7V45ddiwZbX7HU7pLbzuZLX8Gk+3eu7zXRLPHZknZtndP6d9OGX94ah7GYUVYrGovCj76Z2YC/9GPKrbLXnZfQUAgGSUjoGRsmNocaqeUI6Jid20eauyY5GJMhq1LXJV9+hl3pMPyT3ECUoye9ZMV1cXQoi//+BG3ZG+vn5eXp7EB35eWrRwYdUc0PETpzRqPNAEvO4rJh2S8VKAPbv+IoRERUY19tNyANBk4RY8vIE2dva1Jw9uP07PLijn6XVwHvLVpz0EbFL8Tfz2PLyWm1uL1w7Be0lKX6Hs0ACgGUACCm9QcRi9epOfZvU7h5w4Pfi3fffxuhUAeA19BQDUBxJQqI4S5N8Li+ng0Mm8lY4KqSh49uhm5Il/Nu+NzsS0LgB4DX0FANQLElCojiuI2T574nZlhwEATRz6CgCoFzwFDwAAAKMneUIAACAASURBVAAKhQQUAAAAABQKCSgAAAAAKBQSUAAAAABQKCSgAAAAAKBQSEABAAAAQKGQgAIAAACAQiEBBQAAAACFQgIKAAAAAAqFBBQAAAAAFAoJKAAAAAAoFBJQAAAAAFAonrIDaLksLS1nz5qp7CjgvWJpaansEGqGpg4AANUhAVUafX09V1cXZUcBoAho6gAAUB1uwQMAAACAQlE6BkbKjgHksmL5Mmfn7gyPkbQCRzjCkX379u/fv5/jOEXGBgAg4PMdHJ08PT3c3HqqqKgkJiZGRkVFRlzMy89XdmgA0OQgAW02err1XLpkCUVRNX4qFotZsXjNunX/Xf5PwYEBAFT3KhN1d3cTCARVmejF8Ij8ggJlhwYATQUS0GaDYZi9+/ZqaWq++5FILCouKl6xYuXDhw8VHxgAQI0EAoGDg6Onp0cvdzc+MlEAqAYJaDPA5/MdHZ28vfp17tLFQF+P4b3x6JiYZR8nPVr5zTd5eXnKihAAQIoaM9Hw8PDCgkJlhwYAyoEEtEkztzAf6DPAy8tLU0vz2rVr165dmz59evUVWI67fOnShg0/VVZWKitIAAA5qaio2Ns7eHv16+HWg6GZGzduhISGRl+JLikpUXZoAKBQSECbIoFA4NrD1XfQIHt7+5ycnPDw8FOnTmVkZBJCtm7dam7elqKoqseM8MgRADRHGpqaPXq4enp4ODo6EorEX4uPjIq68t+V0tJSZYcGAIqABLRpsbS09PUd1KdPHx6fF/1fdEhoWFxcLMuyr1YYEjjk448/JhxhxaK169ZfvnxZidECANRT9UyUIyQhPj4yKuq/y/+VlZUpOzQAaERIQJsETU1NDw+Pwf7+7Tu0T3n6NORCyPlz5wsKa5inr6Ots2vPrqLCwhXLVyY9SlJ8qAAAjUFTU9O1h6unh4eTkxPLcchEAd5vSECViaZpOzs7b29vD49eIpHoypXokJCQhIQE6VuNHj367NmzeOQIAN5LWlpaLq4uLzJRlk1ISIiMirp86XJ5ebmyQwOABoMEVDkMDAz69evn5+tr0srk4cOHp0+fCQ8PR/cKAPDKq0zU2dlZLBZXZaKXLl2uQFcJ0PwhAVUomqZdXV38/PwcHR0LCgtDQ0LPnzv/NPWpsuMCAGi6tLW1uru8yERFItH169cjo6IuRV2qqKhQdmgAUEdIQBXEwMBg0KCBAwYM0NfXT0i4fvr06ZiYGJFIpOy4AACaDW0d7e7du3t6eHTv3l0oFMbGxIaEhsVfuyYUCZUdGgDUDhLQxlU1y9N30KCebj0ryssjo6KOHz+RkpKi7LgAAJoxAwODXh69PD08rK2tS0tKomNiIyOj4uOvCYXyZqIqqqq4lQ+gREhAG4umpqaXt1fgkCGtWrWqmuUZGhqK18UDADQgQ0ND917utc1EtbW1Nv7yy/q16+7cvauwUAGgOiSgDYyiqK5du/r6+bq7uVVWVFwIDT1z+gyGPAEAGpWhkZG7u1tVJlpSUhITExsZGXXt2tUaZzoNHDhw9uzPxWLx9j+3nwgKUny0AIAEtMGoqan16dMnICCgXTuLqiHPsPBw3OIBAFAkI2NjN7eeVZlocXFxbGzcu5no6tU/2HbtSlM0R0jUxciff/kFfTWAgiEBbQDmbdv6+ft7e3sxDBMRHhEcfBKviAcAUC5jY6Oebm6eHh42NjaFhUVxcXGRkVFXr8ZpaGrs3bOHpumq1URicVZW5qpvvsWtKgBFQgJad1UPGAUGDnFxccnIyDh9+sy5c2cLC4uUHRcAALzWpk0bTw8PT0/Pdu3b5eXlPXj4sLuz86sElBAiFotFIvFPP22IirqkxDgBWhQkoHWhq6PT38fHz8/PyMjwxo0bp8+cuXzpcvVfbAcAgKamrVlbz94e3t7exibGNEVX/4jjOEJIUHDwju078II8AAVAAlo7lpaWvr6DvLy8hEJhZGTk8WPHU57iNfIAAM2DjrbOnr27qw9/VseybOK9ez98/wN+6xigsSEBlQufz/f08AgYEtC5c+ekR0nBwScjwiPwIxwAAM3L4MH+M2bMkJSAEkJEYlFxccn3336HNzQBNKo3EtBFCxcqMZTGdifx7vFjx2u7la6Ojq+fn5+fr46OzqVLl4KCgu/cudMY4TU4K6suw4YOU3YU8Iajx44mJt5TdhTNW+DQQBsra2VH0eLUrf9sgtatW2tlbfXW/fe3cBwhhHv06NGz9GeKigsaVxPpe3Fd/nH16lf/5lX/wMPTQ+HBKNRxUosOtEOHDv7+flV32y+EhBw9eiwrM7PxYmtwhkZG732FNjuRl6JIE+gEmzUbK2s0bKWoVf/ZNOnp6VlbW3Mcx3KslEFQiiKEUB07duzYsaMCo4NG1ET6XlyXyev8880EFAghNE137+4SGBhgb2+fnp7+199/nz1zFnfbAQCaO4GKYPPmLYQQjuNKSkteLX+/7/4BNE01JKAxMbGbNm9VfCiNZ8+uv+RZTV1dvb9P/6GBQ6uebV+16rvY2JiqRyObtU2bt8bExCo7ihbN1dVl9qyZyo7ifTN+4hRlh9AiyNl/NgsZzzPOnDlTwwcLCXkfr33QZPvevc+NbhapKzsKxRnXKqubVulbCzECSgghpq1bDx4SMHDAAI7jIiIi8Gw7AAAAQONp6Qmoja1NYMAQ917umZmZ+/cfOHPmTHFxsbKDAgAAAHiftdAEVMDne/T2HPnBCHML8zt37qxZuxZvkgcAAABQjBaXgOrr6w8aNCggIEBNXTXyYtSatWuePElWdlAAAAAALUgLSkA1NTXnzZvXp0/vwsLCoKCg4JPBhQWFyg4KAAAAoMVpQQmog6PD3buJ69dvuHz5Mn7qFwAAAEBZWlACej3h+uIlS5QdBQAAAEBLJ+3nyN4zRUVFyg4BAAAAAFpSAgoAAAAATQESUAAAAABQKCSgAAAAAKBQSEABAAAAQKGQgAIANBZKr++y/cGRq/urNO0yQWFQfdB0UCoqnw80OOSuIlDG3pGAQt2oOM3+37W4U2sGGFDKDgWgyaJUTW26tTdS5zXgafJmmTgTmxTZ1dEYTQKgbiiG6WTA0+dRFCGEUF3t9YM/NFxoTiumcTbAe0BVe87ZvWxwe2N9LQ0BIy4vys9MuXcrJur8v0fDEgvE8pXB2E75dcN4zaCZU7bck3MTaDBv1CBXWVaQ/fThrf8uHNt1KDK5TOJWFEVRFK2gdlpLaJNAiKpFv/GfTfT37GpuqE5VFGQ9TkyIOr1v2+Hredx7VblN+Ux8z7zVVZbmZ6U8uBF19vA//8Y8q3yxjhKrQ57woLlQaaX5k4tqWzVak08xHFdWyT4rEF57Wn70YUVqY/6QDiXfwGQna92lXagLEXm78+q+rwZIQBkjy26Wpi/uJjDqusbtdI3b2Xn6T/nkxq6l83+4kCbHd0XrWth0ap0nQB+qDG/UIFHVNDCzNjCz7jFwzJh/v5j8zdkMtqaNKq7+MsrxFwVGWRtoky0eYzHyp8Pf9DZkXtQfz8Csa682nXgJu/69Trj3qXKb9Jn4nnmrq9QybGtr2NbWzXfs8O3Tpm6KLuSUWx1yhAfNBqPGs9JhXtwZpygNVcZSlbE0UR3SuezbC4UXSxtjn9yt67n+1+VZk9LR4rfTYOt5476hbsGL7mwZaWPTtYONU7defsM+/W57VJpI137yz38scdN6Hzr5psS0desVK1f069tPVVW14UoV3dk8wtqmawdre9ueAwZP//5/t4tVOgxfOctNveH2oVhok83MuHHjPv74I0tLywYoi+cwaaaHAckM3TBjQC/XLraO3TyHjP5yw4a/w2v+ewreU/O+/HL06NGtW7VuuCKrd5UDA2duCH5Uqdnto1VTrJmG20c9NNHwaBUtIxMjPfW3x7wkLW92bGxsFi9e7O7uLuDzG7DYBzdy+u/N6L03w+dQ9kdhRcG5nEBbba6doAGv/UrUYLXOCisqxRxHKoqzkxNCkxPCToXM37nzoy7jF044OHzrXTGhDPou+mmObxczE20Vriw7Ke7s9p82H71X8vqPMqbz7OM3ZleVlnlggte3l4VybNXyMAzP1cXF1cVFKBReuRIdFhZ27do1oVBYz2JZUaVQzHFEVJqXdjti35JUTZsTc2ycnDswl592GzVn0gAXW8t2rXTVqLLs5LPfTFz18MMDp2a3PjK939eRQkIog17TlkzqY9OxramhtjpfXJh+N3zf5j+utxk6LnBADyszXaYk7da5f9av3ncznyOESK9ZSsfh3T1+l/XJyQMTtU/N8Z59tuTld2H95fH/fWYYPL3f4pB3Zgs0VpvUsPSfPnPq4J5WxiplGfeijv6xdltEan2/fiAGBvoDBw4cNmx4xvPnIaGh4eERaWlpdSxL3czCgGFTgjftiHogJoSQysyk6JNJ0dXXqXWH81YjJ+V5KQkX9m/YeCA+73WTofXtx8z8ZJyPYwcDfumzxP+uFJhU+zO/9s1+5elcTmqZTKdPX5+JlM7QHVHf93trXKLyylLvj/dmci2w6bbr0M7L22vChPFJSUkXQkKiIqNyc3PrWWa1rjL1RsjOeQVGdrsmtndxMqHvpLNvVEfV+tKbBCGEqJr1mzBjaqCHnbmBGikvyEp7dP/W+b82bI+p6ixr1+dIDU+O0qQFI7GJSimT1u8+bfniGf076/EpjhMWpZxfNfnrf9NZScsJIUSt3YCPPp02xN3GVIPNS74aenjrlgMxWVWzZSTGUM9qrSc+n9+rl3uvXu7l5eWXLl8KDwtPSLjOsvX9e5djiZAjHCHlFeIHaaXriqlO/pqWRgJzqvKZgdoUa1V7fV4bdVqVcHlF5RsvFEaUE4rP87LVGNVO0FGNKi8TxSWV/H674vnLQGhV/pBuGoFtBeaqpKxEdC2DNaw2HtOuq/5f9szZsOzV6S+/Tx7jYaX5YQdBZw2KFnPP8yp2Xyk8V/WzkhRvsr/JZEIIIWxZ2RdHC6/V8nAb7c8OruDKr2sODdw+sZOvv9Ufd2+LiUi3o1Nns6qeUdPEuu/EdV2NhYFfBWVLbTd126pl4PP57m49PTx6VVRUXPnvSsTFyKtX48TiBprT9noWE23sNmKCn83LtqJlZMyvKH57bX07n4A+r9bh67V1HPb1jmHV1hBYdP9w6W/6JR98ciyDJdJrtsY9Cm9F/pc7YbiLm53g7H9VM5poEyfXdnRFVPS1cjmOqEHapHq3WTv+nOuoVXUFUW1rH/D5JgfT2YFLI/LQJOtNLBYzDGPSqtWHH344duzY9PT0sLDwkNCQjOcZtSuo7Flqnphu6z3B99/7wVJmMr9DarN8s5ETDcOOvUYvceisNnzCzvsiQgihtN2X7vp1cifVqnNHxdzBz5wQQirkK7+mZs/JLlNOLbvpdujQ4WMLi+nTpj14cD8sPCI8PLywoLBBSmZFYpYQQtM13k+UXX2qVtO2bV/oqveyw9UwMOtsYNZBcG3nzph8Malvxb0dnvTSZARTcxOVVibVauTqXxf00aZEJTkZxZSmga4xv6KAJbSE5YQQVZsZ27YvcNV5EbBJ5z5jFrn3tp83flFQuljSaSL7i1AUVVXVvn36ePXzLisrvXgxMiQ05O6duxzXQBFS5FW6aNBKbZgF/+X3QOmrk0ohITz+JC+9KUZU1benosn3tte10cifdqWigBBKIJjVX3eEbtUjR0Sgxe+nRQghEmcIM7zR/fQ+NXnZeBjKwpBRb7gZqI35FHzZ9fDoApZuY91JgxDCFV1aN3Gom4uzpbWddc/BH22/UW7Q74O+eq+Tb/H9TYF27bvYtu9i29Hz28tCItdWLRvD41EUpaqq6uHZa/nypXv27J7xyXQbWxuKquM3RDEqmobmdv3Gfb9msg2Pzbme8KgqoeWKzq8Y3N3RwdLOzXPkL9E1/vHNFZ5eNMDezs6ym4f/klOpYo7Nj908a4Sbs0NnJ5/xW68WEt0+w72M6aryZNXsO3ssjwuNyCNGnr3tX97i0HR06coT3Y6+WiDn2V3fNsl0nrhsloN6ZsSmj/x6Wdk69xix9PAjsdnQWeMsm8b9t/cFwzCEkNatW48ePWrH9u0//7xhSOAQXR0debcXXt25OSqbsvhg/dHQvas+GWil/+7f2nXrcF428o62PTzGrT7/nNWwHzfOqapF8rpOXTjRUlCYsHvuCC/bro72XuPmbo/NrjYqUPtmL7vM6riCYx91s606qPa2A+cGpQrZ0tv7dp7Nplt406Uoqqq37GTZadrUqXt271r1zTde3l51nshEMQINfTNbzzHfLR9pzohTrl57XkOlyKw+puP45fNcdSsfBa+YMMihm51ltx4eyyNKq9+FqVPFSQhPemkygyGEvNtEpZVJafUY0EOLvfXHMDe37r29nJ1dew5bH1VKJC0nhLGcsOwLF+3yu4cXjPKy7eroOGDaj6HPKFPflQt8Xp8n8lyPlIdheBRF1NXV+/f3Wrd27Z7du6bPmF6fyUUURWmoMtZt1Oe7a1jSJD9bmPKiUrio6JwhBzL77s8adbrkuph0sNKaaETlpBUvCMry3pc59HTh6QKuVQeNQF1CCOliozVclyrOLl11OmvAvkzfY7mrbldKGTtu21n7YxO6Ir9sw/nswfsz+x/MmnSh6OKr4R5O9PfJDM89GZ57Mvr8W+vhT9KII6CEECLKzS3gKC11TXWaFLIUpdtt9IIlPW0sWuvzS55lswzhmbQ2pEmutCG7um3V8vB4fEKItra2v6//kIAhObm5D+7fr1UBXeeeeDi3+hKuMiX4281RpVV/cXGivLTUnFIhIcL05EJCaur7OHFRVmZhhZiQvDtHt+79cMCCDtl3Lt19XkoISb+0bcf5MY5D27Yzp8lzVo6afXuPhJTFnArPHT60n0/XDbHxIkIE3Vwd1MQPL0Y9k7vl169NMl0CBlvxCi98P39bWAFHCMm8efSbTR4DN3q7uxpuflDLUTqQhaIohuERQjp36mJp2Wn6tGk3b96UL2MQJx+aOzxj8vy5E3ydP/i6+/DZ6XFHd27ZtD82Q/qf77Kb5atGXpwWt++H3b795ttYWRnRMeks02Vg/3Z0xdWN89YeT2UJISQtIWjPudGTXBxrUf6bzZ6xlV1mjWij/it+XzvY8PH+L6asuZRN2UxC0yWEEPLqCXUnJ0cnZ6fZs2fHREfL2OYNNXSVJXd3rdhxu4aWJbNJMB38/G0F4sRfvli6615VGlWclVNtQE9qn7P1wbtTmqWGJ720R9oygnlR3ltN1EZKmb8FcRwhlJGVq5XhvdiMcq4i63EqIYTial5OmE5DAm0Fwptr5606lCQmhJQmX9721QqL4N/H9B3irXf2cG5NMTRVVddlXT09fz/fwCFDnqU/e/goqVYldHYwiHB4Y4mwqPzXGxUvkkCOKygR54k4QriMIkIovnc7PlNZvuVSSdVdwpycsl9uCHp7qjibMHsK6N5tebS4cmdU0fmqe5jFwpB7FQHWAtsa903xvNvzBWLh7xcLj1V9x2LucVZDTqJv1ASUp6+vQ3FsaUkpR2n3Xvr39jEW/Bd/waiYtyWEiGlaagB120oCD0+Pk57BddiweWF4DCHEQF/foGfPqiVmZm1iYmLl2pjjxOKK0vzs1Ee3Y8JP7j0Y+qCIqznXlEn8LDldRKyNTHRpUsoSQkjl89RMjjJWV6PqXLNl/504/2zoqIGD7NbFXxMyndxd9bnko+GP5P9jpH5tUtC2gxlNqw38NWbgr28eralZa0JkXMU5QhYtXEgWyh1sC8NyLMdK+GOcIjRFE0Ls7OxeLVNXVy8tlfIsaGXqxW1zLv7zo7PfxCmTxnp1H7tkR3+P78Z+djBJUg5a6yYhTk96UsLZampqUIQQvmm7NjSbei1O0h9EdWhyMsuseUdaLnM2bxxpnnVq0dTvLmaxhKjVq+mSZtt/FhQWSPqIomlCCJ/H69WrV9WSNm1MGYaRdxZTVQrFFcXtXLJgS9jjt8cJCSFyVB/PolM7hn16OfyhhEE8qX0OTSQ/U1djeNJL4xnKCKb2EVJFl4+GZPfz77N414V5ecm3Eq5GnNiz88yDEknLBRaWZjT7NPrSk2q1UHItMr58zCALy7Y0qesMXuX2vVWZaGvT1q1NXzwPp8+vxZ1sln3xGqbraeXHHlQ8kVQ/DNNWk9A81ZWjVFe++YmxJk3TdBsNwhYLb5TUuPE7aKadNmGLK68VyR9p7TRmAqpm37eHDs0mJz4oIfqBk4eZM3nRm5et3XslKauMZ+i95NjGIdILoPT712ErSRITE48eO1a3bZsUA32D6dOnSVlBJBLxeLz8/HxdXV1CSGqqPE9yiG5tHD70t6SGG1dmhZUiQvH5/Fd3TYRCEUcoiq5HzZbHHT32eNQnA/y6/3Qtxszd05yk/hNxV/6g69kmJU7koVTUZP+sCUXI0WPHEhMT5Q63ZfH1HdStazdJn7IcSwjFseKCgkJ9fX1CiNTs85WK51ePrr16/A+b4as2Lg3oPedzr1Nzz9U8f7IOzZKrrKzkKKpqTI1iaEIIJfEVkHVp9rLKrAnP4oPVW6bbiGI3zlhyMvXFFJp6NV3SbPvPCRPG62hLnLlRNee4vLy8alg9LS1djuzzVVcp6DRx68FFPTpaGZMKSd+urOqj+TyaEJFI4l6lVlxNxUoNT3ppMoOpQ4Rc9snFE4rjR/r2tHdy7ObUr71z335W9PBZJyUsD22s6XWN3fe2s7AYM2aMlBWqGltWdraRoSEhJFcoV/Z1PyFn2i2RvH99ckTS7XQVhqKoFxND5Z55+WImX+NNsG20BJTS6Tlrwcg2tOj+2ZN3xbRlq1YCUnp+968XEisJIUSYk1VY7SLAiUQijqirq7/R+GhD6VvVTnZWdlRkVF23bkLamrWtcblIJOTx+AWFheHh4VFRUfr6+osWNtHRtrrXrOjuoYMJH3/df5jbL0/NPbpQT/86e0vePyTr3yaFaU/SWFbn+McDloXV6TVsiYmJ70cjbAxOjo41JKAcEXMsTVEPHzyoenbks09nenh61LJstuDO0Y0HP/Cbb2Np2Zo597hROpzKlKRUlm7n3rfjlpv3axijqEv5ssp8B6XlMuf3ZX10Uw5/Mnfn7VePX9W76TbT/vPDD0e9u7Dq2WSO4xLi48MvXrx06fKRfw/XvuzKB3sWL7E/sMl/3vqPE8b+kVhDVcqsPuHz9GyWNu/uakrfelpTmlH3iqspPOml8RxlBFMjmRGWP43Y/VPEbkIYLasPVu1c6dN3oKv6yVMlNS4/8zgplaUtevSyYG6+urWl4eTpqEoqUx6lsnV+cKWx+96iwqIa80+xSMzwmJzc3LDQ0AvnQyzaWzTidZkVp5UQVlC28Hjhf+9eFyl+SjGhtQU9dajEfHmeXxOnlRBaU+CkRe69Pc2BE3EcIZRa/VLIBnsIiebxGYoQRqBhaGHfb/SS7Qf/mmqtKnyyb/Wuu2LC5mRmColaj+HjnE01eRSh+ZqaqtUi5zKfZ3FMa5+RPu01eYyqfsfutqaMzK2AEELEYhEhpKysPPJi1DfffDt+3Phtf2y7c/uOsuOSph41y6YEHwovMRg4atRQr67Mk/MnJeefDd8myb2z5x+xhgEr1071tmmlLWBoRlXPzLaviwWaZYNjxWJCSFp62q5/dk2cOOmLL+adOH5C3ieXBY7Tf/hqoldXcz1VhqIYNf32LsNmDu3McKKszFy2kToc8b2g4LtCns1nm9dM8+yor8rQjKqOod6rLLcu5csq8y2Uft9layZ14W5t/nJ1aA5XvRw0XcIRlmU5jnvw4P6WLVtHjx6zfMXK0JDQinJ5XqJREzbz9KoVh9JUHGd+M826pndyy6w+0d3zoc9YFcc5G+YH2JpoClT0LFyG+1Qrqz4V92540kuTGUyNpJfJtPf6wMuujbaAphg+T1RUVEEIRRFK0nLx/RMn7lTyu33+07IRdibqPIGOhfv0dd+Mak3lRwRdUPa7luQnEgkJIYWFhSdPn5y/YMGkiZP++uvvp6lPG3evnDDiqYhVU53bS6OXPqPJEJqidDT5PY0ZHiGEE154IhTR/Al9tEeb8nQZQlOUlhotcU49JwxPEYkZ/pTe2kNNGB2G0DRlpMfvoEoIITmlLEsxHpaqbfmEYWgLY75J7QevG6rz4dnM+vferDdiF+ff/GfpvO8vF3KEkJzQgyGzPP29lu/zWv56HfHLx2TEKRGhd2Z3sxu+PnQ4IYQQYcIPfhP+fCp9qxat6r6HUCi6fPlSeFh4fEKCSNSYv8/VoDgZ7UH6thd2n5zTf9Rnn3FM4paTdyTeLmqUNrl9x/c7+/02zefL7T5fvtpGGL/WZ+w/yXjDeb3RNF01gSQ9PT3kQkh4RMTz58/rUA7Pxnvs0CkWH0x5czFX9mDXn2dzOcI1Tocjvv/PyvXufy50Hbh4+8DF1T6oGhurU7OXUeZb+E6+fqYMRXX74sjVL16Xkf73JN9VLbjpsmIxRdMPHjwICQmJiozKL5A4MbS2uIKoNd8e99w67NPl486O/+vB292RzOorj922/oT3+qH2EzcdmVjt81eduehWPSrunfCklyYzmBpJKzPFoMfUb5a5V381O5t7+lxsqYF3jctLiPjB7lUbe2+f7zJy3aGR614ehzDt9Mo1Z5t+/ilmxTRFV1RUXIy8GBYafuvWrfq/CrRW7t8uOtRGd3RbzdVtNV8tFGYVTThXmsaRx4mFf7bW+8RE9TMv1c+qbSXpNUwP7hTtN9Udb6A2z0dt3otlXMjFrJUpXFpaxQM7vnVHnX0ddQghhBVuCco9UMvZog2QgIqzHt5Msm5vrKetrsKw5cUF2Sn3b8VdOnfocMid/JenI5d7eum0ec/nTPV17mSiwYjKiwrysp6lXHn44v054gf/zJ6vveLzIT066Akq8lNuPcyiKJlbtVgikfDatfiwsPDo6OiKijrPSlCeetVs2ZW9vqPFygAAHvhJREFUh++OmGUrjjt0vOZJq43XJrmi2NXjx97+aOoYH1ebtgYaTHleelJC3FP80nKDyM3LCwsNDQsPf/L4SX3KYR+dWPOTIKCPi30XcxMtAVdRmJGSGBty9M+/Tt0p4kjjdThld/+c9uG9CdOnDfG0a2eowQhL87OePrqXEPFISOra7KWXKbeW2XTFYvHTp6khISEREeGZmVmNsAcu/+KvG8L6rff6+Eu/EzOD8t/+XFb1sVnnF4z79P7s6SP7dDPXIQUpN6Ieafp4d2a5F4lL/SrurfBypJcmM5ia9yG5TIp6di38RhvnTm10VaiKgrQHV8/s3rIpOIsY17ycI4SU3fl92tjHU2d+PMTN1lSTzXtyNeTwltcvom+6Kisro6NjGuqnYeqGE1b+di73vo3GkLaCTlq0GsUVlIjuZIpfRCMS7Q/NTeqiMbq9wFqbUae4sgo2vVB0J63m4StOWPnnhdwkG43hFgJLDZrPsdmFwuRKQhHC5peuukTNtlNz1KF5YjY9R1SHx8MoHQOjV/9z8mQwISQmJnbT5q11OfSmas+uvwghUZFRP65erexYGoCKigqfzy8ufvtd8G/x8PSommuyafNWeZ+Cby54DotP7RqbtKzPzOM5zeFvEVdXl9mzZhJCfly9ujlOpFMMfX39vLw8mW9sXrRwYdUc0PETp0hfExpEs+4/9fX15fnpo6Z07aOMRm2LXNU9epn35ENKH/JrUsHUhcL6Xg0NDbFYXC5rOser6/Le50Y3i5rtD13X3rhWWd20Sgkh/v6DXy1sQfN/3hsVFRXNctSzvihtI2NxXrZQ3azX1HmjzPLOrQltlj0iSFD/30gEeEvTb1S0sbOvPXlw+3F6dkE5T6+D85CvPu0hYJPibyrhRl+TCqZ5KSmR8+VG8BoSUGgm6NYfbDy1vDufEEI4cXbYip/Di9AnAkCzpuIwevUmP83qD3Bw4vTg3/bdV8Id5yYVDLz3kIBCM0HpqHD5pSI9kvcoOmj7j5tOPkWXCADNGyXIvxcW08Ghk3krHRVSUfDs0c3IE/9s3hudqYQnw5pUMPD+QwIKzYT47u/j+/6u7CgAABoOVxCzffbE7coOo0qTCgbefw32HlAAAAAAAHkgAQUAAAAAhUICCgAAAAAKhQQUAAAAABSqBSWgFFX7XyoFAAAAgIbWghJQ5+7OI0aM0NLSUnYgAAAAAC1aC0pAs7OzR40c8ffff33++Sxzc3NlhwMAAADQQrWg94A+efxk7i+/9OnTZ+iwwIEDB16/fv348aDY2BiZvz0NAAAAAA2oBSWghJCysrIzZ86cO3fOzs4uMHDI8uVLnz17FhQcfPbsuYrycmVHBwAAANAitKwEtArLsgkJCQkJCWZmZn7+fpMnTZowfvz5CxeOHT2amZml7OgAAAAA3nMtMQF9JTU1ddsf2/bu3eft7TV86NCAwYPj4uKOHz+RkJCg7NAAAAAA3lstOgGtUlJcfOL4ieCg4O7dXQIDA77//rukR0mnTp4ODQ2trKxUdnQAAAAA75saElBLS8vZs2YqPhTlYlk2JiY6Jiba0tIyMDDw008/mTBxwoXz54OCgrOzs5UdXb34DhzQ09VF2VG0aHp6esoO4T3UArspqCc+n6+iokII4QgpKS5+69OWee17vzXZvtdDp9BOo0TZUSiOuVrFuwtrSED19fVcW3C+8vDhww0bNuzcudPX13dwgH9gYGD0lehjx4/dvZuo7NDqqFMnS2WHANDwWnI3BXVjbm6+adMvkj5t4dc+UCSLmhKylqYFvQe0VvLy8vbt2zdpwuRNv/7apk2b9evX//LLRi9vLx4PkxYAAJqlpKSkzCw8aQrQJFA6BkbKjqEZsLG1CQwY4ubuVlBQEBoa+h7clwcAaIEmTpw44oPhjOShhKo3QwcFB+/YvkMkEikwNICWhVFV11B2DM1AVlZWVFRUWFi4QCAYMGDA8A+Gm7U1y8vLQxoKANCMFBUV+Q/2l/SpWCwWCkXr1q07euQoy7KKDAygpUECWgvFxcXx8fHHj51IeZrS3dll3LixPd16cBx5mpoqxh/KAABNnqaGplsvN3U19Xc/EovFmZmZixYtvnnzluIDA2hpcAu+7iwtLX19B3l5eQmFwsjIyOPHjqc8farsoAAA4G1t2rTx9PT09PBo177d82fPDI2M3prQzxESdTHy519+wa/iASgGEtD60tXR6e/j4+/vb2hocOPGjdNnzly+dBn3bgAAlM7Y2Kinm5unh4eNjU1hYVFcXFxkZFROTnb1Z+Gruut//tl1+PBh5UUK0OIgAW0YDMP06NFj8GB/Ozu75xnPT586feF8SEFhgbLjAgBocYyMjd3cenp6eFhbWxcXF8fGxkVGRl27dvXVQ0V//rnN1NSUECISi4qLSr7/7rs7d+8qNWSAFgcJaAMzNzf38/Pz8uonUBFcirp06tTp27dvKzsoAID3n6GRkbu7W1XeWVJSEhMT+1be+cq4ceNGfziKUNTdu3d/+P6H/AIMFgAoGhLQRiHg81179vAdNMjBwSE1NfX8+Qvnzp0tLCxSdlwAAO8bQ0ND917ub+Wd8fHXhEKhpE3MLcy3btny779Hdu3aJRaLFRktAFRBAtq4qh5U6tu3L80wMVeiT585k5CQoOygAACaPQMDg14evaryztKSkmg58s7qunXrdvPmzcYOEgAkQQKqCOrq6r179x7s79++Q/uHDx+ePn0mPDy8HM9aAgDUkraOdvfu3T09PLp37y4UCmNjYkNCw+TPOwGgiUACqlA2tjZ+fr4eHh6VFZUXL148d+78/fv3lR0UAEBTp62t1d3FxdPDw9nZWSQSXb9+PTIq6lLUpYoK/KY2QLOEBFQJtHW0vb36DxjQ39zc/MnjJ+fOnQsNCysqwgxRAIA3aGlpubi+yDvFYnFCQkJkVNSlS5fxtk6A5g4JqDJVzRDt06cPj8+L/i86JDQsLi5W5jtEu3TpfP/+g6ofLAYAeP+8yjudnJxYlq3KOy9fuoyZSwDvDSSgyicQCFx7uPoOGmRvb5+TkxMeHn7q1KmMjMwaV+bxeLt3706Ij//5558rMecJAN4jmpqarj1cX+SdHJcQHx8ZFfXf5f/KysqUHRoANDAkoE1IW7O2/X28+/v4aGtpVf2oUvSV6Ldm1vfs2WPp0qUcyz5+/GTFypV5eXnKihYAoEH8v707j6sp//8A/jnn3NuqvSwpMULKUtKqUClLWcq+f5lh/AwxY99p7AzGki8jM6MZfA2yJHukRClLlkkJpUKlvW7de889vz8qwq1uyb3J6/lf95zz+bzP+bzv47z7nOWqN2liZ2fr7ORkZWXFEYK6E+BrgAK0weHz+VZW3dxcXRwcHQQCQURExOng4OfPnpctXbF8mbV1d4bHiMVsYWHBihUrnzx5otB4AQDqonLdSShy5/ad8IiImzduFhcXKzo0APjsUIA2XHp6ei4uLv3792vevHnZy5tiY2MDAvYxDFO2AstKJKx4w6ZNNyJvKDZUAAAZKSsrd+1q6ebqYudgx9BMXFzc5dDQqJtRRUVFig4NAOQHBWhDR9O0lZWlh7uHnYNdfl6+jo4OTdNvl5Y9inTw4KFDhw7hsSQAaLCUlJQsLa2cnZ16ODrwlZTi4+PDIyKuXr2an5ev6NAAQAFQgH4xNLU0t27Z0qxZM4qiPlgk4bjI69d/+WWLUChUSGwAAFJJrTvDroTl5ePn1wG+aihAvxitW5vs2rWrqqUSln369NnKVavwWBIAKNzbutPR0UGpou68djUsNw91JwAQggL0CzJ1yhQvL0+Gx6tqBTErLiwoxGNJAKAoSny+pVU3Z2cnBwd7ZWXlsrozPOxaTm6uokMDgIYFBWgDZWbWwXuI99s/KYqys7fjVV19luE4juO4x/Hxb95kf+YAQTGCTgTFxz9WdBSf5IPchsZBV0/PwMBAV0+Xoem83LysrMysrDdv3yLXCPIWAOpXDQUNKIq+gYGTs1Ntt6IoiqKojubmnyMkaAjCr0eQL/xEXrfchi+IlraWlrZWW1PTt580grwFgPpF17wKAAAAAED9wQxoQ7d9p3909C1FRwEKZmtr4ztjuqKjqGfI7UavUeYtANQLzIACAAAAgFyhAAUAAAAAuUIBCgAAAAByhQIUAAAAAOQKBSgAAAAAyBUKUAAAAACQKxSgAAAAACBXKEABAAAAQK5QgAIAAACAXKEABQAAAAC5QgEKAAAAAHKFAhQAAAAA5AoFKIAsaMNBq09dDF7lxK9iBcbEZ92pC8cX2/LkGhfAe5CHAPBlQAEK8qTczfd/t2NCNnjoUYoOpZbBUOotO3Q00lapelUNY3NzY20VqgHsGXzF3s/DBvWNAwB4BwXoF0994I74+JjgebbaH55h+D1Xhyc9OrOwC6OQwKSiKIqiaLoWJ0N1943hSfExB0Y2k5KsfMvFF+Ke3j8wyagumVz7YEARaE2zft+v33vk2o3ox4/uPrxxPviPjYvHOhgpf74uGYtJ/ucuH/ihQ/1+d2RsVsXE5buNvwfduBWb+PD2g8jzp/dvWDC8q06dEhVJDgANEy7TNAqUqsXkLdtfTvzuryShomOpVmnsryOsfq3VJkXXQ8KyBw6x9XI3/OevVMl7y5S7eQ4woktvnT2XLqli8/oNBuSN0uzy3eat83s251WUUEq6RhYORuaW6o9DbqaWcp+nW1rbxLxdixyleq7bZGmWMRm+5eiqnvpM+Uo8PaNOPVq24909cOweqfXuIskBoIFCAdo4cCyn6bRg26Kn4/wi8z7TOVlRiqNCLmQMGm3l6dnq0J7nbKUlKnZefVrQJeHBl17Vof5sAGhlDT1tFXFBTk6xWNGx1AMtTa158+eFhV2NjLxRVFRUDy3SLXzW71rYS4fLuhO4e+/h0DtJmUIlHaOO1s79zF5dr79Mb0ADwbOcON1Jj2SE/rJ8/fE7ybkiJV1jC5ueXQRXXzeYJG9AhwsAvlgoQBsH8b3AXa/6zhy/cVXciDlB6ay0dfg9Vl48MCJnp8+wrfFlK1Ba3rui1zvcWOI26Wg2Ryi9HlOWTOxl3tbYUF9Tjc/mp/979eDOPfdaDhk72MPOzEibKUp7cOHPzesP3s+tOPVT6qaeU6d/62Vv1lRZ8PpxRNCejXvDUkWEEErLcsSsiR42Fqatm2urUoKs5POrJvg9GXk4xLfF8akuC8JF5U2oturzn2nfDerRyUiTEuSkPb7mv/TnE8mVdkEQc/LCy5ETzAd7tt23K+HdAnWHIW76VOGVE5eyOEIovd6Ltszq38GomaYyJ8hKijm/b8vOoMdFHJE1mGpbKNtbpfZDlv4+17n7N7pM0atHkaf3bd9/7rmgqlGp+uAQWrf7lOWLv+/TXodPcZyoIOWi338WHKvLPG4DQtOUlZWllZXlzJkzY2JiQkNDo6NvCYV1n5RXc5w2t7cuyQpdPPrHIynltU5pRlL02aTosxUrqbb2mPx/UwY5mhuqS3KSY0OP+u86HJ1ZnuHvpzQpyUm5e+nQL9sO38kpH9UqBoIQQgjT3vdknC8hhBBJxuHxrj9HimpIsxq7q6rZSvtsZKLHSFKCtwdEJLKEECLMSIo6kxRVvpjSdZi8cEKvTu3aGDfVVKVKc9IToy4c2RNw+n6u1ORh2v1f5SSXKcKvLW8BQCFQgDYSbNrZRXM02uyf5PfL5MeTfntUUoc2aN0u7gN7mVfkBF/H2Mp7QYB3pTWUTLqPXLpbt2jotBOvJYQQtc4zAn6bbaVRdgOminHXgTO3Wxr6Dl4alsPRTR2GjR/wtjUNg6b80sKP+lQ2m7InYKGddvktnErNTC2Nmwg+OJ8JY08EJ42d1t5zgMWehLjyOoTS6unlokNygk9eLjt3irXbdmtvpEQIIaRJs469J2zq1FQ0eO7pLI4Q2YKproWyPtUtvYZVHAtja8/pVo6Wq8ZNP/BE9FFb1R4cqvnw9Tvm99KkxEVvXhdSTfS0m/JL8xrPWZxhGBsbG1tbWzHLRkdFXboUevt2rFhc69kyh4F9mtLCOwGbjqVUsa2K+fd798231SrPn2bte41e5Niz65xxi06nsx+lNFHXb9tj1BLL9qo+4/cniAmhqxqIqu/RrDbNauhOFoKXqTksbew2vv+xhODkj/65ofUs+3m7vu2Cp9/a0nNq1z59bWaPX36u5jlSGSL8ivMWAOQJDyE1Glxh7M7ZW2MlltO3/mijUed717j8s4s8unbpYtrZyXNJSCrLSXJv7ZwxzMHasn0393H+sflEu5ePa1OaEMK0n7BshqVaRtj2yQN6mFlY2w1bevQpazRkxljTivM3V3BxhVd3K0vTLg7Ow3+N+rBOY9qMXfaTrVZJwoml4/t362LZ0ca134QN57M+vLTKxp8+dl9It+4/2FKp/CNKx22Qkxb3OuT49YLyrq5vmjDEwcbatGOXjvZek/fFlei5DO1d6cmNGoKRpQXhs5ANkwf2Mu/UzcrjW7+zyWJth3lzvZpKOdrVHRxKw87DTkPyYI+3g0P3nq7W1rb23psjims3UA0cTdMURfF5PHt7uxUrlh06fGjunDmWlpZUbd4SYN6+Cc0+uxaRJnVKnxDGdPyyH200S/49On+Eq0UnKyuPKetCX1KG/VfOd383bBUp3dbCzmns+ouvJOpdx47txieE1DAQbML2wV3adLBo08GirXP5PKUMSVJld9U0+44odv/OiCzKZOjmoNC//ab1NdP9eJaAyw+Z72ph0bltJ3unkQt+i8nhmwxePd9N1qeUqovwa89bAJAbFKCNiTAhcLFfaIHp+NVLetftkVlCOLYgMyO/lGWFOY+C/P9+yFK8rEfX/31VKBIVpV/fG3Axj2OMW7eiCWE6DPQy4+VfWjNv75Wk3FJxScb9oFXbrxQy7Rxt9csTixPnpKW+KRaxpfnpya+LPigsmW+8BnZSFsVt913+d3RKTqmoJP91wp2EzI+nVNiUU8djBLShl4+9GiGEELpF36GO6pLkkKO3KiZ7KUq786i1+49dj7oVd+XAyr6GDOE1a6H/LsWrD0amFopuHT90JSFLICrNTb75+6KVh9Mk6vbuzh+9gKCGg8NxHCGUgZmtmb4KRQhXmvksNbeR3bpbgWF4hBA1VdWePZ3XrFn911+Bfdz7yLithjpFJDnZ0i8uE8K0GzTYQkl0f8ccv3/uvS4WCXOTI/fOXXHkJafTe9C7cqwipSXiwrSYg2sDH4gZPTMzA5oQUoeBqDlJqu5OJmzyP7N9pm0/9ahIz3rogu1HIy7+sWa8TbPKZSjHFmZnF4slElFB2t3gddNXnswkum6De2vJ9qWvJkLkLQDICy7BNy5s+vEVqxzNtw5btSjs/rJPfQyEfZmcLiYdDZpp06RYQgghwlepGRzVVE2VIoRv/I0RTav23RHdd8f7mxkataBJVs3t80zatWYkL6IjU6qY4XpH8vrc0cs/2nu6e7ttDD+dS38z0NtGWfwwKOhB2XVDSrPn0j/2jTbhl5+ClVsZE0JYmpY5w+vQgiAu6r5wvEfL1oY0yXl/kVJ1B4cqiAy6nOXi2WvxgUtzcpIf3I0NO/XX/nOJUmrijyxauJAslHWfGhSGxyOEaGtr21h3L/ukdZvW0dG3qtmkSEAIraWtRZMMaRmiZGJqREteRF2v/Gha0e3wOyWj+5mYGtMk++Nt2PSk50WcRZMm6hQhktoORK2T5L3uZCZMvbZ31rU/11kPmDBp4hjX7mOWBPRxWj3mhyNJ0q7jc3mRl28Lh/Rp1bYlTXJr0Y2UCD9b3gIAfAAFaGPDZYX+vPyY9Z6hK5dEbHr/DjKOSAhRVqnmXeofkoiEYkLx+fy3m4hEYo5Q1NvZI2koZVVlmfooez1hVc28j8sLO3jm5YCxzqMGtDhztOkIHzNeceTBE+WlB6Xb5z/erZicqJ3LNv59MylTwNN3W3Ji2yBZWiZ1b4GiaIoQae9YrP7gcFlnFo8vvDO8v33Xbladu7m0se7tYkb7/HCm5qo96MSJ+Ph4WfZI/tTV1X1nzqxmBTEr5jG8goJ8DQ1NQsjzZ8+rbzDxmYDr0MbexsA/Uep7Duoyy88JhUKubOAIqWIgZpzJk7ptHZLkve5qp/RVbNDG2JN7zH38ti0d2HPWTNeQ2RekPvHGcRJO1i9S9RF+trwFAPgACtDGh8uN2LLkkO0fo+fNfq1GkfyKzyUFeYUc3bKDqRZ19009zFmI0p6nSSRaJ7/zWHZFyn1gMrzBW5T2PE1Ct7J1MGbuP69xErTk1v+C4kf9YDt6mH1OS59W1JtTR0IyKp5l1m/eXIkUXwzccSleSAghojeZ+aW12Zs6tEBpObp1UyLC5Kdpkoq7WcouONd4cAgpeREWuCUskBBGw2yo3/6V7r372pIzITXGGR8fHxEeUZs9kx8dbW0irQBlWTFN84oFxeHXwi+HXtbT01u4YIEsDd64fDO/r5v9FF+Pi0vPfXxnhjA5KVVCm9j1MGHuP63IH/VuzlYqRJjyNFUi0y1G0gZC7cwFsVjMETU1tfcKx09OM05qs9WS5D0K2nZk6IB55qamLZgLT6WsotrZtpMSEaY9/zgPa+uz5S0AwAdwD2hjxBVEbvM7+ELL0LDybCf79P6/+Zyy07RFY6yaqTE0o6JhoKNa9zdts4/PX3wq0R+4cuO3bubNNZUYmlHRMbLobWMi67mPfXzuQhLL7zprh984u9Y6KgzDb9K8g2UHPelpyT45/vdNAWM6attSD10uJehwREHFIsmbjAwRUbXzGWtt2IRHEZrfpIlKrU7BMrVAMRr6eup8muY1adHFc9GulYMNSHbo6St5HCFEKBJzlHa33vZGakwNB4dp4zrUtUtLTSWaYvg8cUFBKSGN7yc8WbGY47iSkpLwaxF+fj+PHjV6x44djx4+qnKW7SM553bve1hKGw7adth/nrdNW301Hs0oaTRtb+c17UfvjiTh1KlHQn7nmVuWDevSTI2npGXiOHXTqhEtqNyw05eyZeilioGgCJfxKpNjWrgPd2/ThMeo6LbtbmHIfHqaSW/2PUpWU9fOneDaqZWOCkNRjKpuGxvv6UPaM5w4MyO7vAan1G2HjXFpp6fK42sa20xcu2q0EV0UdTH84zysLeQtAMgLZkAbJ64gesuaINf/DjOq9GFReGDgv31mWvRffbj/6ncf1/k1jeIHAWv2u+ye4v7TPvef3n4qurPRfcyfyTK9m0X8MGD1np7+0zsN+fnAkJ/LQy867dvT94K090hJXp8OPD/L0buZPlcSc/ive+8i596EHrk8w9nTdflB1+XvNmATZN4ZmVqgNPuvv9x//buNSpNPLttwMYcjhLCp/8bncmZmE3afbzrXZta5ag5Oip7dt6uWOVZ6MJpIss9eqO5uyC8Ix3Ecx0kkkqjoqNDQK7G3YkViaa+pkoUofrfvgqb+a8Z2dJ6+3nl65UXih/TJU/6Bftt67ptnM3zTP8M3VfQvSju7csN5WepPqoqBKCKsICz0kW/nLj6bQ33KIrm7dsD43158YpqxKVKbTXn3beGZu40ZMslk6KT3N+QEiQd+O5/NlU0ZUEqt+83f32/+u6hzb67fHJwhJQ8vyhpaueq+1I07bwFAzjAD2lhxeeG/rg3JeK8OLH2wfer3a47depZdwkpYcUlBVlri7Wtnw54I6nj3WMGt9ePGzPYPvpmYkV/CsqKirOS4sJgXspe0XGHsLxPH+fqHxDx/UyRkRcXZLx7FJuXzq5pVKYw4+E+SmJPkXgo89aLyvnHZZ5dOmRNw5UF6finLikuLcjJSE+5F3Xwi86/l1NCC5M29C6ev3UtMzykWsqxYkJ0Sdy5g+YiRy85WvHyxOGzbT7suPXhVkJb6UljtwaGol7evxiVnC8QSCSvISYm7tHfBt/OCM2U+bA2XRCK5e/fO1q1bR40avXbNups3bta9+iSEEMKmX1o+ymfi6sBzt59l5JewLCvIf510L/zob4eu50iI4NF/p4yZvuNMTHK2QCQsyki4dmjduJELT0n/LYYPVTUQHCFs4p++836/kphVzLLi4jdP7zzJpKhPTzPpzVYieXpqw5a/z0YnpOeVsBKJWJCb9vjmCf+Fw8Zsjiyo6IQruhdyPDwxs1gsLslLvXd+78zRM35PLD/OH+RhbX2deQsA8kdp6RkoOgaQwsnZadHChYSQ7Tv9q39SGL4GtrY2vjOmE0LWrV/fYO8B5fP5qmqq+Xn51a+G3P4EH/yyUUP3ReQtACgELsEDQP0QiUSivC+gKgIAAIXDJXgAAAAAkCsUoAAAAAAgV7gEDwDwpWATdw9vt1vRUQAAfDLMgAIAAACAXKEABQAAAAC5QgEKAAAAAHKFAhQAAAAA5AoFKAAAAADIFQpQAAAAAJArFKAAAAAAIFcoQAEAAABArlCAAgAAAIBcoQAFAAAAALlCAQoAAAAAcoUCFAAAAADkiqfoAKAG/ft62NvaKDoKUDAdHR1Fh1D/kNuNXqPMWwCoFyhAG7p27UwVHQLAZ4HcBgD4auESPAAAAADIFaWlZ6DoGAAAAADgK4IZUAAAAACQKxSgAAAAACBXKEABAAAAQK7+H66O6T5fWhm1AAAAAElFTkSuQmCC ) ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAACIgAAAFUCAIAAADKkAKgAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzddVwVWRsH8DNzi+4SAVEBEUQBlbI7sBtr7V5b1651Q1dd115rbX0tVAwQDBQUQRRbTFA6pOHmzPsHqCCXG6jgrr/vxz90Zu6cZ86cM8J57plD6RubEgAAAAAAAAAAAAAAAPj66OoOAAAAAAAAAAAAAAAA4HuBxAwAAAAAAAAAAAAAAEAVQWIGAAAAAAAAAAAAAACgiiAxAwAAAAAAAAAAAAAAUEWQmAEAAAAAAAAAAAAAAKgiSMwAAAAAAAAAAAAAAABUESRmAAAAAAAAAAAAAAAAqggSMwAAAAAAAAAAAAAAAFUEiRkAAAAAAAAAAAAAAIAqgsQMAAAAAAAAAAAAAABAFUFiBgAAAAAA4BvHqdl307mdk7xMONUdCfw3ULpOA9ed2j/VWVDdkQAAAAB8j5CYAQAAAACoapRZ55XHgqLuHh5ngx/IQTlOraEbV/Xz6T7mh6YGVHUH82XwveceuxgZE7K0Ka94g0bj6Wfvv056fG6hj071hvaZ/iUXolGv1/jBzTou2DzDXaO6YwEAAAD4/uD3QAAAAICq8NeTlKyUO7v7WXPl7uY1XhGZnJ2RcOoHsy846sqp3W/dsaDbdzf30vpyJ/1G8Fv/9SQlOyPh+FDjT2tM233exddZGanpt//yNftGf9yltet4NnezN9fhKL3fXOPGA3/aevTinccvkpOTUt88fxp18dSOX2f0dbfgE0I02qy99y4j7d2zrd31KjwHp/6sa8lp2Wmxu3voF2/hea64l5qWlXhhmoP8JllM4LYwIjk1Ozlohp3CiRrcesM3Hgu5dutx7POkpKR3aclpb58/vX353N4/fhrsbYVh389Em/dbMb+1AZt0dP6CwLy2fz55l5GWrfhPSuh8Z0V3tvpxLVx8XB1q6PE/dAGKpmmKojgc6rOfgp80yNSExBePHt64cHTryll+zW21v3Ju6wteyFdUdHvtzI2PJALnSavHO37bbQUAAADgP+gb/U0VAAAA4L+H4lr1/nPvYm+9Khuro01d2rR0szPT+o5efsSvM3TLP3PctNn0ywuGzj2XxlR3QJ9H22368esXN83ya+tax0xPk8cVaOlb1HZt3XvMko2/DbblECK8eeZiiozQhu36tK1oLgXXsXt3Jx5hsq/4X80p3sQxszChCSVwmzitvX5FLZIy6z3zBwceRWgTcxOFvzjQNRq3b9HEqbalsb4Wn0vTHL6mvoVtg2a+P8zfcOp22O5xLtqfUQvfOw2PyXM7GZGcS7/+GpzFVnc0X43w9rquDWxr1O+8Ijzvc8/1SYPk8LUNTK0cGnfsP27xxhO37wRuGOby9R7EX/JCvipRzF/LD79lNNwnze5p8i3nkAAAAAD+g5CYAQAAAKg6lKbzlB3rB9p8R4mSKsWz6b/hf+u61qDeXV/uN3bHU1F1B/SZuM6T1i5oZkIz6eGbJ3fxcKxhUcPMpl6Dlr1HLvr7zMGdR17ICCHCiDPnU2SE0m/Xq52h3MFVbv1e3R24hMm6fPpqbvEm2tjCjEfJUm7ezPadMrS2/AbJazDyx2Yvwm4LWdrIVKWlTcSh89wtLCwMTS1MrB2cmnUfsfTIvWxWYOv724Hfuhhh5LdSKOMeEwbV5kpfH9xwIpkhRHxpRn0jEzODkj/mdtMviwhh0vf3sfyw0czAotVvj6TVHXq1E4f+9L5BWjk4Nes+dN6WC8/zaSO34evOnPu5jel33yTzQzdvixJSRp0nDlY8Jw4AAAAAvjAkZgAAAACqDCsSSWgL3z+2Tm6A5Za/OJ5137+Obepbi5sVvtJvxIaY/OoO6LPRli1bOfAptiB4+fBlx26+elcklYkLsxIeh/tvW/zDnGNJxdOBRLdOnE2UEUq3dc+O8pIf3AY9ethzCZMZdPLa+y/w0+YWZjRhM8O3/RPtNGZ0U3kvG9NtN26I+aVdOx9ksRTPxFSlpU0kwiKRlGFZRlqUnRR769Tmad2Gbo+VEI5l70m9a+J3j0qga/Yc1l6fkjw6fDBaWN3B/OtIRO8bpDA7KfbW2Z3L/Fq3G3PouZBou4zbutHP6ntvk0z8sf1X8wnfdfAgV151BwMAAADwPfnefxAFAAAAqELiS6sWnk1hdD3m7lzUrOIX6XAcZwSnZ6Rl3lnp/clIGb/Vn49SstPj/+dXMgCvYdth+l/HwmNik5MTU+Ie3ws9fWjdD40+GWUX+O5683HxidRjw0vWseHV9Bkyc9WOo5du3H0V/zY9JeHtk6jr//ux8YdCeaYew5b+c/ZG7Ku3qW9iH1w5vP7HTnXlLlcjsGo17rcjwbdfvUlIjXty9+K+30d6mpdetUDTsffsX7YdPH098t6r+LcZaSkZb5/evbB9vm9dTQ2rFsMX7zh17cmrhIyUN3Exl/73+w8exmr+mMqr1X/jia39a/OybvziN/zPO3lyXvikNEgFFVIc/4FT1yLvvYp7m56WkpH4/PG145und6qjWakKUQXF5fEoQggjFIoVvcBKfNs/4LWUUNot+3Ypv0YRz61Xt7pcwqQFnbhe8H4jbWJqTBMm593jkwevGvQb3bVcRoeu2WdcN1nAwYtx73IYQpuYGlXqC/Vs3u3jAS9lhOI5NSy7mo2KVcQ1aew3f9vJ0EfP4tJTk1LjYx/dDDy5Y9XC0c0tP7QRpS1ZhbKUdiWV+poqXUZptGXQNTr6NhVQ0sfnzr6QKansUp+yHnc6LT3t3aM/2vBLb9fsuvVZVkbCmVE1SncwrvviqJS0rKSTYy1LbVbxBql2GKVbr/fczQFh9+LeJqS8fnArYPuK4Y0/mbBCWfxwOiktO/nSnPpl29oX6U0fiOJOzhn1y60CljbqMHday0/ujqKyOHZTA9Mz0jJjfm3GL/sp2mrC2YTsjKSb81248i+E49D/twNnLt99/CIlOTkj6eXTm2d2LehdX7dsFaj7nOGaNBm84O9T1x8/j09NeB57J/Tczp+Huep8PKnyqmMzQs6GFbJc287dGmChGQAAAICqg5+9AAAAAKqO9O2xqZMcHI6MqTdu45rILhMCUuUtgSJ7eeNmoqyRbQ0Pb1vOzecfR2O59h4exjSRPgm/lcMSwqs35vCZlW0+5DB4JrWcTaz0H21WbfyWMm7/0x/zWpUaYeSZ1nI0ovIYQgihDJrO2rN3fjOT96vTC6xd2o1waTtw0OFJg+f4x0s+nseo2YIDu2d5GL6Pw7i2e+cJbu18m8/oPv5onJQQQih9zxEzx5Yui2ga1W7aa+4/bX5I55mba30YSTSwcuk0ZnWrZrX7dl0eLi+9IofAbvCmI+t72XCzbvziN2xdtJyPqRRkxRUiJ36BvqVTyyFOzbu2WdF9wJaHIvXKUons7Y3wOIm7vW7X+Wv8ns068jS/gvqQxJw48Wz8T06aLfp0sTy0J7F0q+I37tPdlktkb8+eCC/6sFXD0EiLJmxudk5G0KFzK3aOHVDr1La4Up/jOQ8d6ZPq/2t4QU79XIZQhsbyX5OmymXIZIQQiqY5H1d5V62KKL3G0/fsXdjCjPvhk9qGNe0Na9q7t26lEbU/LElMiNKWrEJZSruSKn1NxS6jONpytD2auQooWVx42EvV8zKESQ4Pey7zdjZya1ybcyX2/Se59T3ddShCN3B34e9Ofj//hjZzc7PhEOmT8LD3DyRVb5Bqh3Ftem48ummQneD9bTSv592rnjchhBBlF/XFelNp4thda09MPDLc0rL7oJbLrgYWqFaWLO5q6MsF7o4WzVrac8NLvSmOMvRp5cIjsrdXrzyVEiKvp9D6zm26+NR5n6jRtbD36jvTo0Mz6259Ntx///RQ7zmj6z51z/7FLU3f9w5Ncxt985r6T/b+vF+dqmOzboQ+kHTxquXjbUXuxlWmPgEAAABAfZgxAwAAAFCV2OxrP49fd7eItuq3dv1wW/nfkpHcuxKawRBu/ZY+pb9TTpk08azLIbLEiIi3MkJ0Os6Y08qYKri/Z0LnxrUsLc1qOTfuNGzGiuMPPhmsFJ0bbfNx8Qnz/vvSSo/vy+IOjGvtUs/W1NzK2rl51+lHX8oIoc37rv1nQXMTKv/hvlm93O1tzGs1bDlyzeVkmUY9vy27pzX6MHRI1+i/dvssDwNJfODKYa0dbawsHLx7LzkbJ+Fad1/5+4AyX8wn0pe7hns71q1lamFl27T/z5fTWFrfwox5cnz5kPbutSxrmNV27zL/7Bsp0XActcDPWpUfVTVsOi45dmZjL2uSGDi/71C5WRn1gpRbIaXjt7M1Na9pWb/FwJXBiVLK0Gfur0NLhapWWUpIotdPXxOZxfLqDtwQdG3PlOYVTROQPjl27L6YpQSevbuXrTYNz17drThE+vrE0ciPL8KiDYwNKMJK8vKFbMG1Q6czGv8w1K30jA2tZqOH2D0/fvSuhC3Iy2cJrW9kULklKDQcu3Sy5xBWEvvoWUlqQsUqokx7rvlnUUszjuiF/9IhPs61Tc1qWNRx8Z4blCs3QSW/JatSltKupEJfU73LKIi2PK69awNNihU/vvdMIm9/RaTPw8NSZITr4NnkY0aNrunpZc0lhNZr4uH0sSlpNfFy4VOy5PCwkjk5Kt4gFQ/j2o/fsn6QnYB9d3vb1G5u9jZmVvXcu45f6f8kV34uqpQv2ZvKKIwIvpbDEFq/qZcjT+WypE+vXE2SEa5du7a1SncH3WbtvbUoJvPapZiK7pL06bH5Q7q2cLa3NTO3tKjn02dZ4FsJpecxbUHvcivdqPKcoUx8V+1e0tKUUxh7bMmQZi51zSysbFxadp8w55/bYvWqjkl98CBNRnhOrk6VrU4AAAAAUBsSMwAAAABVTBjz1+SloVnEqO3PG8c68uUeEhkUms1Q/KYdWxp/HLTTdPdyEVBMzq2bDySEcGrWd9ChieTuwfVHb7/NEUvFBekvo4P2nbwjf9i6Ikxe/JPYt5mFEpk4L/VZ1KMUGSF8t/E/+ZrRsuQT0wZO23vjVZZQVJByP2D14KF/3hMRzYbj5/Y0KY6L33j83C6mVNHt34aMWXPhcUqhWPju5ZUtk8Ztfy6l9dv4dS+zhgNbmPI6LiWnSCIVZ78O/XPG2jAhS1hxzPGd52IScsQycV7CzZ2zVwblsZTAvaWHrvLo+c1n/jXDx4Quiv5t8Ji/H8ifUqJekPIqpEz82YUSmaQwPTborykLArIYStPTt+2HN4ipV5YybG7Uqt7t+iw9FJXOqe27+HRY4JYxnuZy3nkle+X/vwgRS/Eb9+tZp9SQsVbzPl0tOUQSe/LovVJDxrShsRFN2ML8AkKIOOroiVe1+g9v8eGlTpRRp2E9jWOOHH8mI2xBfgHLUlxDY33Vp8zQXIG2Yc16nt3Hr/I/OtOVTzFpZ7cef8OoU0W8RmPn9bDgMO+CfhowZnPw49QCCSMT5qa+jM+UyL3NcluyKmUp7Uoq9DXVu4yCaOUQ2NjW4BAmPT6hSO7+ConvXYvIZSh+k+ZN3r8EizL0adaQRwghXBtvnw8Nke/awkOHYnJuXL8vUecGqXiYZvNxk5tqU7L4PeP95h+KfJ0lFAuzXkX6r5m2PlRZrunL9qay9fMqNl5GCG1pXZOrelnimJDQTIZwG3buWCpJotOiays9iskJu3SrwlWA2LxHVwIjYxOzCsUyqTDzxeXNPy4MyGYoHc8W7p+uOKbCc4brMmZhnxocJvn4j33HbQl+lJwnlopzk59e9w9+VKRu1ckS3iTKCCWwtq1sbQIAAACA2pCYAQAAAKhykpe7pi8JzGB1vX7aONn501E5QggpuBZwOYuhtLw7tfowHM5z8mmiR7EFNy5HCgkhTFZ6howlPLcBI7yNKzeZoSLcht261OYS6bPDG8+nlx4DF97ftfVSPkvpte7eWp8ihPDcunetzWWFYfv3x4pLH3g3JCyNofjO7o3kXV4xJjXy5isZoXXq2Jl9/KmUzY669UxCKG5NG0uVL4vSajxv/+ZhDvLWsP+8IBVhs69dihazFLeuox33q5UlTri6ZXrHpi0HrTzzgucy+LfTt6/tXdir3idrFDGJAYdDC1iK79qvT/0PkyH02gzsakaz4rtHT8aWHvin9Y306eKcCyFE8vDE0cfGPYd2Kllohq7ZZ1h7jZvH/N8whBBZfn4hS2gDQwPlvzrw269/kpWRlp2WlBof+yg8YP8vI73NOOLEkKXD557JYNWpIm6Dbl3tuEQWd2TdsQQ1XuJVhmplKe1Kyvua6l1GPbSeoSGXIkxWZpbS2SWfKLwRElHI0npeLd4v6q7p0aKpQPYmKjpVxmvQ3MugOByuQ7Pm5hw2P/xiRHFWQcUbpOphjdq3NecQyaND26/lqJUy/oo9lxDCFhYUsoRQtKamQI1HmTA84GIaQ/HcunT+kJnRbt69jT7N5IScuZ6nRgB5D++9lBFKy8ys4vXGio8s/5zhNujezZ5LSZ8d2nA+TV6tqlV1Jc2LNjBSPXwAAAAA+ExIzAAAAABUAybh2Mz5p5MZrSYz101vKGdwMf+af2AGQ+k0796mZPyUY+PlZcVhhZFB17JZQgibHrDzxBsJ0W487UxU2Ml10wd6WWlVdhmQMig9x/rWXMLkxkTHfvJWNDbnzu3nUkIJ6jnZcQihdO0dLDmE0uyw4WV6WnbGxz/pp0ZZ0oTSMDVXMJwvy0zNYAih9fT1Sh3EvMvIZAihtXV1lP+sKg77c8b68DQpSwlq9Vjvv3ts/XK5mc8MUhG2ICU5hyW0gUHJmPtXK4steBW0fmyrtqM338rUtOsyZ0fI1e1+9qWnW7GZ549cfMcQbr0+g9yLd1DGnQZ0MqJZYcSRU3FlchuUrr4uRdjCgiKGEEJkL04ev6PRYWifmjQhhOPQf5iXLPT4uWSGkPcj2LS+oZ4agbMsw7AsIazo4e7RzZsN2Rj9fm6JilVE6To61+IQNj864r5YcVkVU7EspV1J6QGqdxl18TX4hBBWLBKrmdQgbPa14Aghy6nRrKUDlxBC+G5tfPRIxtX1267nsAKPVp5ahBBCW7VsUZfLFkUEXyu+RSrfINUO07O3N+cQJvfB/dfqpte+Ys8lhNLU1qIIYZmiIhGrRllFN05fSGUonnuvLiWZGZ2W3dsb0kzWJf+ruQoK5Jq6D164xT8k7MHTV6lJcS/vhu4dacchhOJyOUqe23KeM04NanEIkxtz55ncVXbUrDqxSEIIofiVTnMBAAAAgNqQmAEAAACoFkzKqQWzTyQymq4z1k5pWP6FZgXXDp9JlNEGHfy6mdOEENqidVsXLiu+E3Ils3iIln138afuw9cEPM5mdOu2Hb7g74DIx9d2z25To4J1SFRGaevoEELYvJzya0AwuTn5DCG0jq4O/eFIBTQEisb6xOLi1RDoMiPWYomUEEI4HFXGsYXxgcv7dR6+834eS3HM2/12eNMg27Kv+vrcIBVhRSIxSyguj/v1yyKECF+dXdirbd814RmsoE6vP3ZNdynVcNic4IP+iTLCqdV7gJcmIYSu0WNgK12KyQk+eCqpzJ2kdPR1uRRhRYUl78di3pw+cZN4Dh1gzyH8xn4DnQovHw16386ERUUsofT0lHyvnxBCxCHT6xuamBmYmBvadPr9biFL8e19mpqSUkkFFauI0tHXpSjC5GRmVXa6jOq3Q2lXUnaA6l1GXWKhmBBC8QV8tZOubMbVi3ckLNe+fRsbDiFc57ZtLOi8m6HXr129VUQZtGjbVEAIZdambUMeK44KDn3Hlr4WBYpvkIqHaWlrE0LY/Lx8daf8fN3exKvjYMMhhEl6myhVqyxhhP+FZBnFc+/Toy6HEEq/XZ/2RjSTGXwqNL/CD3PrDN57KWDzjH5tXB2sTXQEfC1j63outqqmlco9Z3T0dClKfnsrPkC9quMLeIQQVixSLRwAAAAA+AKQmAEAAACoJmzm+cULjiUyGq4//jG+/Fozooh9hx6JiVbzwQNqcwhl3KZDYz6R3A0MKTXGLn4T/Mewlg1dOo1dvj/8rYhjUL/bwgNHl3h9WCxE3W/ZF3+oID+fEELp6pefI0Hr6evQxYcwhLCFBQWEECbjwCAzEzOD8n8sfTfHqT0eq3bYkoTzC/r3X33zHUNxrbr/dWS1r0WpwL9WkHID//plSVNDV4+ZdzaLofjOAwe4l05CCW/uP/pcSjg1egzrbEhx7AcM99GkmNTTB4Pela1RSldfjxDCFglFJTuYlHPHrosbDBzU2KDZkD42OcEngrNKdrHCIiFLKG0DPbUmfAjv/Tl19a18Iqg3Zv28Zh9HiVWsIlYkFBJCKC3tz5gGpsbtUNqVFB6gepdRE5OblSVlCW1obKj+L25M0sXAGDHLdWnf2oLm2LVtbcspCA8Oy3sXGhQpokxbtW/Eo4xadGwqIOLo8yGpjFqVpuphxTVD6xnqqz1f6Gv2Jk3vDq0MaMLkRN16KlGvLFHEiTPxMorfsE9fRw5l1L5/OwNalnL2+PWCigqjTPouW97VkitLvvrHmM6u9Wqbmdcwr92o+6ZHyhbZqQBbWFhIivMz8puFelVX0ryY7HeVCwcAAAAAKgGJGQAAAIBqw2YGLpp/IpHR8pg+v5fJp+PP0icHt4fmE77r6LHeOhZd+vhoEPFt/7Nvy41EilLunP5zRu+mzccdfCkhAodhP7QsXu6bFYlFLCGUQCBQZ3CbzYt9miAjtJ6re71PZt9Qem5N7LmEFcY+eikjhM198SJVRmgD9yYOnztP5/OwWRF/DO615EqajAjsBm/bPd3tY3KqCoOsmrLYrOjbL6SEcCysrcrMDpI+OHL4toilDTuO6F3Ha9jghnxK+urE/uuFn5yA1tHTowkhwkLh+4wNmxl4IrTAtu/k5WN6mGVeOBH6cbWMokIhIRSta6CrXopE/HTH9DWRBYRnP2rVfC/t9wWpVkVsdvybbIbQ+g0b1a70Ckpq344Ku5KSA1TvMuoSvYlLlhHatJaVpvKDP8W8OX8mWkL4Tbp0tKzbpbMjVxgVeDWbZdNDgqLFtFWnzi4mLTs30yKiqDPnEt8/V1S9QSoeVlwzlG5jrwa8ig/7nCIqgWc/cma/GhwiSw44eq1A3bLEt/939KmUcB0HDvKo08OvnS4ljTt58EZRxcU5ezfVpVhhyC/jfz11Jy6zQCyTifJS45PzKpU5J4TNff48TUZo/UZu9nKjVa/qOFY2NTmEFb2Nq1w4AAAAAFAJSMwAAAAAVCM248LyBadTWC0zi/JftWdT/Hf4J8k4NoOmz5440EeTiCJPn02s8Bviwrhz20+/khFKy8y8eCkCJiMtkyGErl1PvbFtyb1zga+lhOvgN7lTmXyRRoPRE9rqUGxuaEBoNksIkdy9eDlFRrj1Bk/tYvpFFrj5DAX3/x49aNWtHIbSbTr7n187vg+9KoNUtSyWJSxLCMXX4CmKiKLk7tWp52TDJYTJz8ouu8IEE3d8z5U8ltLwGbfx5wE2HFZ879Dhu+XWaKF0dLQoQliR8ENihrDvQk6E5Jj1HNJBL+XC8bCPuZziGTOEqsSruCRPt8/7656I8O1H/zKhQcmkMBWrSBx96do7hvAaDBnfUr+yd62St758V1J2gOpdRk3S5zEPi1iK79TIQd28BiGEeXv+9G0xEXj2nTCimzNXFHk+OIMlhEm6eCFawqndZfD0vi11iejW6cBS8/BUrDRVD7t3LjBOSrh1Bs8ZVEvN9MrX6bl86x6rdi/x0qaYd8Gr/7paoH5ZsqdHDkcLWU6tAYs2jfbWIJJHR/9XvpeVwRJCWJmUqWQm5lOSmOBLqTLCdRwytYupvG6pzuXQ5i4uZhwieRzz+MtEBwAAAAAqQGIGAAAAoFqx6QHLfgnMlJ9uKQjdujmqiOi0mDaxiYAUhp28UGr8VKfZuAXjfRvXMdHmURRHYFDLY8CE7rU5hMmKjy8eAmbS7kS/lRJu7SHzJzWrqc2l+bo1G3bu7mGh7GdAcfS2VefTGI5l/42H1w3zsjXg87XMG/jO3n9ghpsGEd7fvvp0evEIozBsy59hOQzHsv+mkzumdHW3MdTgUjRf18LBs8ewzo5VPI2Gzb/z5+jxB1+JCdd28J/rB1jRVR6kimWxuZlZDEu49XuP6lDPqIL1Qyjzof9cO7th9pB2rrVNdQQ0RfN1LOo1G7x8z/p+ZjRhUgPP3/pkOJhND9h9OllG8eyauBvRbP7VXUdelJ+nUfyCMFZaVFT6VUo5V0+EZDFElnT+dISw1ClFwiJCCKWtq63+2Lj40abFe15KiUbDiSsGW6t1O3KDt+18JGI5NiO2HVjZ36OusaZA26xe80ELJrRSYbWbEqqVpbQrKe9rqncZNRVEhseIWI5Vs+Z1KzFziEkKOBFRRAQ+Y0e680QRAcHFLyxjEs6fuS3h2A2Z0EGPFIYfP59c+gGk4g1S8TBx9LbfzqUxtFHHVf6H5/duYqPPpymOQM+idk0DZffx83sul6/BpQghFEegb2Hf1HfUkoOXr+wZVk+DFDzcMenHQwkfLlydspj4kweu5rG0cVNvRx4RRuw/+lzRbCjJk+j7RSyl2W7Wb2Na2JlocilCcQX6xvpqzWQso+j65rXXsxmOZf/N/rumdnG1NtDgcvg6pvYenTs461DqXA5l6NPKhUek8TduJlQ2HAAAAABQW/W+cgIAAAAACJN0fNGagS1+bS7nTVGyl3t+Ozj25JjaHMJkhRw8m/pxbJdXv+u4HyfbTl9V5gMskxm6dmt4ybi6JGbX5tCha9qZtFl07t6ikiMKLoy/HnVU8bf3mZTjM0fZGu+d79No5J9nRv758fxFz45MHrU+5sMq0bLXuydOqH1o20TXev2W7em3rNRJxFELr118Gv+Flm9REZseuGDCWif/eY1Nu674ffCN4QfeMlUapGplsdnXA67ndOhg0HDcvhudtvl6LYkot9wEZdy2e1uH5lpzmw+eW+4ymew7WyatCMn7dAcpuL7nQOygOU48iipflj0AACAASURBVDApp3afTpNzoylNbW2KsEJh2cW+889Pqm80qVxRRe9nzFQiMUNIYcT6XwL67upt0mrGzPYnZ17MZVW9HeL7f05c5Hr0106WnpO3np1c9rQMy6qU51ClLKVdSZW+pnqXUQ+TfPFc1IrmzZ18fe02Po1V921oTHLAoeDFLXoacNii66eCUkoaOpMUcOrWMu8WGhwmM+jgmU8aiYo3SMXDmJQTM0fYGOxZ0NKqw6y/O8wqG6DiC/rcnstvvfpu+upPNrLSzLsHV85ZeOB+bunrVqcsNu3c7oCFHYdY0IR5F7j3RILCMNi0Y79vHO45t6l9vzX+/daU3al4pk2FZHH/TJpQ59DfE10dei/Z23vJxx0vN3XzehQtUfVyKJN23ZprUdKXgWcflp1/BwAAAABfE2bMAAAAAFQ72au9SzbdF8kdaC68uW17lJglTOrZw0FZpQ5h08P+d+Ti3depeSIpw0hFuSkvos/vWtq/47Adzz+M8TPx+8d1n7Llwv2EHJFMJi5Ij7t78UT4GxW+es9mR/7Rr7XvnG0BUa/S88XioqzER1cP/DKyVcfpJ+PL5BCYtEsLu7buOXfb6YhnKbliGSMV5mfGP7pxZv/pmE8XN6kSRTHrpv4Rkc/Sxu0XLfYtfo9PVQapUllM0qEpQ+btvfIwISv/+bNX8kZE2YyjY9v5zViz/+yNR/EZ+SIZIxMXZic/jwo+tGpqN0/fFVcy5TUZyaP9eyOELCHSZwf+kb8guUBbi6YIKxQWqZLckAmLxCyhtHV1KvX9fjYzYO22uyKWY9ln2iCb4l8/VLwdoqd7h3boM+vvwOi4zAKxRJiT+PDKod+3XX1XvPa8ajNQlJeltCup1NfU6DJqYRJP77+Uw/Kc/YY01lD/4+y7iwfPpTOELbh+8nzKh/wBk3Tu5PUilsiSTx+8lFOuJlW8QSoexubcXjegZYdJfxy8dD8+M08kY2XiwqyUV/fDz+/fvOXsa0XJmUr2XCblzqXwO8/epmUXCCUylhEX5mQkPr8TfHzHyqn9mjbu/OP+slkZtcvKv7rn6HMpIbKE43suZilrisKYP/t0GffbkdAHb7OKJDKZWJj3LiX+2f2ISwGHLzzKr9RcKibt8sKubXrP33nu9qu0fLFMKspLj394MzD4QS6j+uXQtfoNb61DxPcOHYmpfCMFAAAAALVR+sam1R0DAAAAAFSIZz/xdMgyb+ruz126r3uEkTP4ztHWY09G/epNQue49t+X8oWW7PjGaXitCD89oXbexR9bDD+YXLVT0OA/TafNHzeO/GCVfWZss7EnMr6P7gQAAADwbcCMGQAAAIBvDWVYq56VPp+rYWTfctzfBxd6a+Xf/H3aJmRl4LtC6TcbOXOkr7dLnZrG2nwOV8PQyrndyFV7f/LSILmhJ4PlvaTtv0kYuWV14Dti0G7+/PaGlV6WBOATgkZTl/pZ08I7W9ecRlYGAAAAoGphxgwAAADAN4YyHno4ZmP79+tCs+L4kz92m+T/Vt3VJQD+1Xg+q6JOjLcq9949Vhx/akbPicfivqcVMTi1fjh2aVUb3aTDP3SYHCj3HXYAatFssuDc2RnO5NGfvp1/jxZWdzgAAAAA3xmOhpZ2dccAAAAAAKVwrJv6tnKzNhAwuclPw46umznh15BkZGXge8PV0NYR8GmeQFNDg8ejWXFuWtyD8LO7f5v549rQ1O+sR7A5D8OTbN3YoI37w98WIi8Dn0+W905k6sANmDX/FP6DAQAAAKhymDEDAAAAAAAAAAAAAABQRbDGDAAAAAAAAAAAAAAAQBVBYgYAAAAAAAAAAAAAAKCKIDEDAAAAAAAAAAAAAABQRZCYAQAAAAAAAAAAAAAAqCJIzAAAAAAAAAAAAAAAAFQRJGYAAAAAAAAAAAAAAACqCBIzAAAAAAAAAAAAAAAAVQSJGQAAAAAAAAAAAAAAgCqCxAwAAAAAAAAAAAAAAEAVQWIGAAAAAAAAAAAAAACgiiAxAwAAAAAAAAAAAAAAUEWQmAEAAAAAAAAAAAAAAKgiSMwAAAAAAAAAAAAAAABUESRmAAAAAAAAAAAAAAAAqggSMwAAAAAAAAAAAAAAAFUEiRkAAAAAAAAAAAAAAIAqgsQMAAAAAAAAAAAAAABAFUFiBgAAAAAAAAAAAAAAoIogMQMAAAAAAAAAAAAAAFBFkJgBAAAAAAAAAAAAAACoIkjMAAAAAAAAAAAAAAAAVBEkZgAAAAAAAAAAAAAAAKoIEjMAAAAAAAAAAAAAAABVBIkZAAAAAAAAAAAAAACAKoLEDAAAAAAAAAAAAAAAQBVBYgYAAAAAAAAAAAAAAKCKIDEDAAAAAAAAAAAAAABQRZCYAQAAAAAAAAAAAAAAqCJIzAAAAAAAAAAAAAAAAFQRJGYAAAAAAAAAAAAAAACqCBIzAAAAAAAAAAAAAAAAVQSJGQAAAAAAAAAAAAAAgCqCxAwAAAAAAAAAAAAAAEAVQWIGAAAAAAAAAAAAAACginCrOwD4Ynr26unkWL+6owA1PH765PSp09UdxVeENgnwr/CffxYBAAAAAAAAAHxTkJj573ByrN+8RfPqjgLUc5r8lwdD0SYB/i3+288iAAAAAAAAAIBvCl5lBgAAAAAAAAAAAAAAUEUwY+Y/aOjwkdUdAihxYN8/1R1ClUKbBPg2fW/PIgAAAAAAAACAbwFmzAAAAAAAAAAAAAAAAFQRzJgBAACAbwJWpfrPe/rkaUZGRnVHAQAAAAAAAFDNkJgBAACAb8L8efOqOwT4un77/few62HVHQUAAAAAAABANcOrzAAAAAAAAAAAAAAAAKoIZswAAADAN+T58xcXgi5WdxTwJdnZ1e3auVN1RwEAAAAAAADwrUBiBgAAAL4hWVlZkZFR1R0FAAAAAAAAAMDXgleZAQAAAAAAAAAAAAAAVBEkZgAAAAAAAAAAAAAAAKoIEjMAAAAAAAAAAAAAAABVBGvMAAAAAMB3xMTExLG+Y3VHAQBfXtj1sOoOAQAAAABAJUjMAAAAAMB3xLG+4/x586o7CgD48nyvd6vuEAAAAAAAVIJXmQEAAAAAAAAAAAAAAFQRzJiB7w1t2WPFth9d7y7vvTRMIu8ATq0+KzdOqBexaMCvkdKqjg7+QzjmPuNnT+zf3MlKnxZmxp37dez8Cxls9cVDGbZetGV2x/j17eeFiKovDACAb8f5wKAXL15WdxQA8Lm6dOpob29X3VEAAAAAAKgBiRn4DxC4T923c7hu8IJh8y5mKhv4prRr1qtvZfCEqvAIXWsnJ2vdGKriI+A/Qq2Woya+87S/N02pLyhuRjqmFgJRfjVmZQghlIalk0tt03QuWjYAQLEXL15GRkZVdxQA8Lm8PJpWdwj/Dlhhq7pg9SMAAAAoD4mZ70ufrXfWthVUsFMS9YvvoH2JTCXPzXEeuXHtUJ2ASSM3x8rkHaDdYXXgth6a4Ut9R/wv9dNSeK4Lzu0bUyPm5y4j/klQOwSKoiiKpjHe/C/FsZty7MRMu9h1/Ydsii0/jUnbe9Hx/UPNQ+e1Gn0qT+m5lLXDMr5eyxF4DvCrxxc/Pzp7xobgV/l8U0udfExTAQAAAIBqgxW2qgtWPwIAAIDysMYMfCm0QS0n+xq6/ArHuAvCz4e+YzU8unWwLNfuBO6+Xa1o0Z0LgUmVSAyJov8a4Na485ygLz3pAaoGbWJuSlOC+mNmdLMo1zY49n5z+1tzKI6hiRFHhXMpa4elfb2WQ5vb1dWnROG7/zr3PEskk+SlxCdX84QZAAAAAAAAAAAA+CZgxsz35eRE95Mlf+W6zw04OlL3xLg2P12Xu9TKl1d46/zFtB5+br6+Nof/jis9m0HDs1v7GrTw+tmQlMpO2KletEDX2EBDmpeVVfhdLEszfsL4wsLC0Kuhb968+QKnE5iY61Pi7BxOi3Fj3S/8fFv4cRdl0HHCDy6i7Gy+npGRIUXiv0BxVYESaPAptigjowDZmEr43jrUl4XaAwAAAMWwwlbVwOpHAAAAoAASM/ApStvOd9yk0d28HM0ERamxYf5/r94emiAhWm4zTuwdZfNofZ/hu4pfNyVwmvC/Q5NsIpb3mHwioTjPwnGYevr+VEIIIUzakWFtf75ROulTdPv0xeSBw516+tbdufnZx8yMtnevdiZU/pVTIRksIZRx6/nrpnWpZ2WuJ2CLMl7eDtq5bpN/bPEAN6XvOmDaDx2bOtvZWhhoUkUZ8UHLh694MfDI+ak1Tr5PMik8Q/FF8h16LfpndosmdYw4BSmPbwTs3LA7MK5I3TohhNBGTcYuWTC+vYMhj2JZSd6b4BUjfjpRmXk//yaWFjWaNG0yaODAt2/fhoRcunYtNC0tvdJno41MjGkm7fzWAPfZwyZ23zX22If649oPmtRRcHPtlvzps31MDIqn0yi/v3LaofKWo1ILL03TtuOoiWN7+DhZajNZ8dGXj2/ZfCQy/cNxFKENB+yIGVD8L0nU0g6j9iUXXxjHZcYp//EWwbPbTjxb/HI22mLwztDF9qfGt//pmuj9Mf7+4wxPTugwN1SosCy5l7bswjuWNmrkN2nCkA5udYx5hclPb0bkmCudJKlh1WbY+NE9mze0MdYkwpz0xFfPHgb/s3ZnZDZbQUGBVCsFHda42diFP7RyqmttaaKnxZPlJj25emjT3/dq9hrSs6Ono5UBpyDx4cW9a34/9CCbLW4MqnUoJZVPiKZN+xETxvRo1sBKjyrKSoy9tmXRz6fiZRXvopstC943IGtTn35/Pn1fsb03R/7ufXNhu5HH31Vw+ctv2PSXW/kVPzQ+qRMizHoTE3J47fojd7NYVeLH4+hLoQxbL9oyu2P8+vbzQvCeQQAA+K5gha2qgdWPAAAAQAEkZqAsLZcpu3ZMd9MtHr/VsG7U/ccNrpZTey4Kzbq7de527/9Nnrhq7I2BW56INBpMWjm2QcGV2cv85YxZyyeOPnX25ZAJDr5dnf9+dr/ky9yUfstubQxJ1tnTl4rHJaUGdd0drPiEEEJ0zOu3Hv5HAzNJz9kBGSwhtJl3v2Fdnd43XF1TM54ov1w5is5QXKa2a7d+JX/nWzf2neTm47p86KR9L+RNHlJQJ5RF/983zm2lR0kLMlPzKR1jAzOeKOc7Gga1trYePnzYyJEjXr9+fTE4+Pq161lZWeqehDIwMqTZ7LRb+3ZfG/zryFFuZ1ZGiwghhNJrN26wY0bACP/YLmNYDSNjbYqIWRXurxzKW06hWi1cw2n89p1zPfRLMh3mDq385vu0bDRr6PyAJKX9QRYbEZk+rn8jN0fe2SgJIYRoujVx4tFarm51ONeeyAghtImrqzVddC38rkhZWXIvjaX0fBbt2zjCXqP4pW4CG9euNoQQomgAWsNx7Pad8zwM3y+6o21s5WBsVYd/Z/fuyGxZBQURTUUd1qhhh+6tPnyEZ2jt1vunXb1Llcmv1WTgoq1GBX0nnEplaNU6lNLKFziO/XvXPM+STB7hm9u5WusUMQp3Kc1Zyb18Sn6dKHhosJ/UCdE2qdts0EJXB80+w3Y/K34qKogfjyNVCdyn7ts5XDd4wbB5F+W/q5DSsHRyqW2azsXyZAAAAAAAAABQxbDGDJTGcRi+eIqrVlrohlFdmzk6N/bst+j4K5lVrylD7DiECB9sX7TxHusyfsXkhkbuk1aMd8w9v3LlmdJvH5M929CzYe16zrXrOddtUXa6TPH+pwEnHohp2y49XfklmyjDdj2a67Op50+GF88dYPPC/xjey7tpY7v6Det7dRu1877QuE3f1oYfx87YvOCl3Zq4udo19G7R/69b5UpR4Qzi1+dXjereyqmBu1vH0SsuxEsNvOfM7mYmZ3xOUZ1Qup4dPXWZh3/39vZu0rJt48YeXr3XhBVWtvr/nTgcDiGktm3tsWPG7N+/b926NZ07d9bS0lL9DLSBkQHF5uXkpgXuOZ5Ys9/IjiYUIYRwbHqP6aDz4OCBiPzcnDyWNjQypglR5f5W1A6VtBwVWvj7i7YbtnhGUz3hk+NzB7R1buDm1nHsb5eTKcsuy+Z2+BgHk3V0jGtxGLUbjHg/XYYQQsT3b0bm06aNm9QpXjaH5+zlrkURunYT95JJLdpuXs48ycOIW/m0SmV9emncBqPnDbfj58bsn96vrXMDt0Zth0zfGZWhaJSeU3foklkeBuJXZ5cO6+zq0tDOxbP5ktDCT4a0y9WhCt0t98L8jo0aNrRzae678HyCjGWyozZN6efd2NXBvcPQLdG5xKBVn7ZmNFGtQymtfE7tIYtneugLn51aNKyLe0PX+k3bdh6+KiiDVbhLNXKb0KcbFT9Iy9RJXWfP5kN+D05htBsNGeLOK75ABfH/ex9Htra1KOqLZUA0vKYdOxd8Oyo69vGDF4+i74cHnt2zep6fRw3+x2MoiqIomq6OrIsq4QEAAAAAAADA9wyJGSiFU697N0dubsgvc7ZfeZktkgrTHvgv33Aln2Pv42FCE0LEz3Ys3BgpcZyw4fDmUXaZASuXB6ar931s2ZszJ28X0Zbd+ngVj9zTNTr19dFm4s8fj3q/sghFGbgM+nX3ifBbUfev7FvWyZJDuOY1TD42VlaalZiQWSiRiXKT4lPlLOKh/AwFUScPX3mWUSQRZcdH/DN/2ZFERturQwuDcmN4iuuEZVlCKFNHD0cTDYoQVpT+OiH7+1xUhCI0TVMUZW/vMHnypCNHDq9YvszM3EyVjwr09bVoJj+vgBHF7Dtwl996uJ89hxANj+GDXQsv7TweJyOFefksZWBUcn+U3t+KKG05KrZwjn2Pns58yYONs1Ycu5daKBFnx9/YPnvp0WTWsHWPdoYqjAQX3r4SVcip6+lpThNCOPaeXia5jx4l0g08PXQpQoigkXdTbdmjazfSKdXK+uTS6Hqd2tvSouj1s1affpBaKBHnJsYEHLj4omQyD8dxyskXsY9eF/95fGa2M4dw6nT1debLnm6bsWhf5NscsUwmzk/PzP+0ksrXofLuJstLT8sVyWTirMf+Ww4+klHcjMfhT1LyJZKCpPDtu4JzWI61rQ1NiCodSmnlc+p0695AILm/YeqSg5FvskQSYW7qs7vP0hmiaJeK5DahcpWv5EFaqk4YaX7i7UO/7n8o5Rg7OprSRGGQ/+bH0bKly3bv3uXn52dmZvr5Z+OY2rnYWRrrafA5NIeroWti7eztO37ZrqCD0zz1ijuFKPqvAW6NO88Jkj9d5qtSITwAAAAAAAAA+K7hVWZQCt+6jhVNa3baGNlpY5kdMkurGjRJZQiRvjw4b32rgEVeZukBk36/rP6IF5MaePzSDC/fDr3brb4ekE3X6d67qUD6yN//YfE7fCi9lov27PSrxSsZvBLYWBNCZDStclutxBmK7t96IB7WsaatJU0+eRGXwjqh8m74X8po49tqwb6QWVnxD2OiQ88c2B34XMUF35u3aH6uxVlVr+tbkpSQWNEumi4ZeXZv3PjDF+T5fL5YLK7oI7r6ujQrLigQE8K8PXUgaPy6wcO8/9lgPKqHxdtjP4Vks4QqyC9g6Fp6etSXaCEKqdTC+bXsrGjm7a3wuFJvLSu4c/2u0K9zLTtrmrxTVgybE375rrBt41ZeBvtP5lj7+NQuujV3S9bcDZ1bNtY8dVns0sLLiHm272qCjN++MmXxLG1r0kzCndvJKuccuLXsbTnM2xtX5b7QryLq3g5ZcnySlNQ3NTegSSFDCCHilIQ0ljLT0qQIq0qHUlr5BcUXEnnjTbl3ynEr3vUFKXmQZpT7gCzpZVwB66yjo00pDvJrPo6+Np6Ab6Cv7+c3aMiQwY+fPAkKCgoPCxcKhco/WSHp402D+m55KmI5mvoWdu4dx86e7OsyasXI4K5/Pf6aN/jfHR4t0DU20JDmZWUVSlXZDgAAAAAAAABfAxIzUArLVjCCRwk0BSXjrpRBvQbWWoRQxm5tXA0vXnmn7qAfmxN66Fxy1yEtBnWtce642YA+jtzCG4dOlQyyUkbtR/S24WTd2rR49cGIl+lFXJN2C0+t76H6+St1BoqiKULkvfNGcZ2wGecWDMu/27+LVyN3Nxf3NrUbt27jSPeZfK782KscT58+9T91SpUjvzU9unWztKpZ0V6GYSiKkkokOTm5JqYmhBAFWRlCiK6eLsUKC4UsIYTNDf3nZHy3ISNms0at+DG/Hb4vJoSwRYVCltLQ1eURSu9zW4gSKrXwz//OO5sZduWOqJlHGy/909EtWtaT3D5y7UaWd9aAli0bCUJz27SoQV6eufRaRviVKovi0IQQqqLXOMmebupjt6nsNk0elyZEKlVryFj97sZIxFJC8Xi8D6FJJFKWUBRNCKmgQ005V/pdY8oqpPjtVXI7roJdhLCEIUSgofHZN1eVB+knnxCLxSxVcrsUBPk1H0dfG5fDJe/ffOjk6Fjf0XHq1Km3IiJCQi7fvh3FMJVZC4eRiiUyliXSwqyE+5d2z8oxbbhveO2m7ub04ySGYz/xyPmpNU6Oa/PT9ZJcI23UyG/ShCEd3OoY8wqTn96MyDH/ZJ6dhlWbYeNH92ze0MZYkwhz0hNfPXsY/M/anZElU48obTvfcZNGd/NyNBMUpcaG+f+9entoQgWpTIXhqXA2RcFQ+q4Dpv3Qsamzna2FgSZVlBEftHz4sgvvWAXnpI2ajF2yYHx7B0MexbKSvDfBK0b8dCKJqWg7IYRo2nYcNXFsDx8nS20mKz768vEtm49Eppf8d11RDJW4lQAAAAAAAADfGyRmoBRJYlwiw+ifHtNx8RX5KxNwavX/+bcexrHHdr1pMaL/ymVR/WecKFnqnJVKpSzR0tJSOqwpjPqf/9NBkz38+nll1exjQ2WeOXo+rWQohzaxsOCTwuD9G0OeigkhRJKZnqtoufJyKnEGSt+nnTufiONfJX5YA5zDKe4cSutE+DZ0/7rQ/YRwdB37rti9rEPrTh7k3HlVQs1Izwi7HqbGtX0zOrRrL2crSxiWoSjq4cOHwSEhN2/cnD5tWnPT5krPpqenQ7GiwqLiNiB5eORw1LAFPwxksy7MOZVQPFwrLhKyLKWtq0PRhorvr+rtUC4FLbwUcfzLBIau5dmsFufBq/d7td1buGkQ8ZtXCaqsJE+Y1Mtnb8/29urQqq5uh4Zs1C/hWYWFIeG5fVu1bXoqt70tG7vp4jNZZcsSv3mZwNC2Pq3rbn7wTLUZMJKUpAyGtmniYUk/fKvqKPnnd9hPyetQWufOF3w4QGmFSBLjEhnaxsPbmvMgruy9U7CLMHk5+Sxds56dPhXzeS+/UvLQ4JTfpGqQX+1xxOVyORyOTPYVJ3IUp2SKFeeeaEK8vDybNWuWnZMTEhx88WJwYmKFU/FUwUhlDCk1a68sSs9n0b6NI+xLMm8CG9euNoQQ8rG5ajiO3b5znofh+3ymtrGVg7FVHf6d3bsjs2WEEC2XKbt2THfTLT6/hnWj7j9ucLWc2nNRaJYKLebT8BSfTUkwtJl3v2Fdnd7/DKdrasYT5bOKzklZ9P9949xWepS0IDM1n9IxNjDjiXIYQlewnRCi4TR++865HvolAZs7tPKb79Oy0ayh8wOSZKSiGAAAAAAAAABABUjMQCmy2KDgV+MndF+2Oo7eci7qRXq+jKdfo26jGvlhUfFSQngOI9bMa8GL/mP6iv0p7ry6u4Yu/X1wzKj9L6WEEDYtJZ3lOHfo3+HQs+A3Uj3bBjWK7j4qP6ZNCJG9OHkwYtQvPoPWL9I0Yt/sPBKW934Xk5mWJiEOnn2GNI49di85n+Hq6GhwS4+dKaPSGSiOromxNi+tiNUyd2o1cu68nqbk3bmAKzksIUQskbKUgXtrL6u7NxIKFdYJp3bbXrUzIm4/Tc6T8bjSvDwRIV9ufet/B5ZlGUZG05znz59duRp67Wpodk6OWmfQ0tGiSKZQVDKixySd2395kmfngtOHrrwf7mSEhUKW1tXToZXdXzXaYXkKW3gpsmdnzjweO9Plx3WLM5ZsPf84i1ezycCflg+oQWUHBYSo+IVxNj3kXOQin+ajllk7kOjlVzNZwt4IvJ7Trf2MeaK67NO1ga9klS5LFhtw9sm46c6TN60S/rz5RFRctoSnb2KoKF0lfRJ8OXnEcLdpa+ekL99z5Xk2r0bDTh3qK16t/PM7bBkVdKgyUSutEDY28OLL8RMbTdu4onDljnP33ubKNE3r2Oln3I/NVLBL9urBk1y2TvMJ8we//MP/frqIp2NqqFmZ3qzsQar04wri/0qPIy9vrzNnThf/XSqVfnjDWGFhEcPIym0sLJ7gIpVKhUWlNrIMIUQikYpEHzfKZAwhRCqTcrlyMlLFCXADff3evXv369fv2bNnKoVbFsXha+mb2Tq3GDajvw1H9jr6ToqcxCK3weh5w+34uTH7l678J/hpFtfMqc3g6YtGNdV9H0vdoUtmeRiIX539bdmm0zFJ+UTTovfqi8ubfQjWYfjiKa5aaaEbFqz63414oX79LnNWLe7ba8qQPWGbnlf4iKkgPMVnI8qCIYQQwuYFL/ObfyYhW6ZpbqGZI+E4jKrwnJvTPDt66jIP/+43cvO9XBmhBKa2ppJCQunJ304Ix27Y4hlN9YRPji9buuXc4yy+ZZMB85bPadNl2dzLYTMCSx7On8ZQiRv4hdnY2NA0FRcXX92BAAAAAAAAACiCxAyUJn2465fdbbaO7TBzZ4eZH7ZK7q7uMHhvPMdhzM8T3WURSxcefC5hya2/5uz0+N/4qb+PuOm384WUyN6EXn481aVhnzWX+xR/LObXrsN2vJH31XsmNWB/0DSf3uYmrPD2kQP3Pr7nis28fPTSlBa+bZccarvk4wdkqg/XqXQGSq/L75e6/P7xQ6L404tXBWexhBBZwpOn2ayj4/CtQWazm04LVFAnb4w9Ry9f7MMrfWnvLlyMUjnYfzeZTMbhcN68eRMcfOn69WsZElE1JAAAIABJREFUGZV8Y5KWtibFigqL3v+bzbkws3ndmaUPYYuEIpbS1tMh7GvF97eidqhCHDzFLbzMpT/fv2J9y51zmvb/41j/P94HKUm8sGxVkMov8mEzQ/wvzW3Ro7FjYeiSSxksIaQgIvBSVrf+blRRxN4zJbMlKleW7NneZWt8dszz6LRgZ6cFpXZUnDIRRm1fc6bdml6Nhm84ObzUdkWphM/vsKVRFXSogjJHKa0Q6aNdK/9uuWVSg14/7+v1c8n+goCpLadeFCrYVXB9//4n7X907rLySJeVH4tT9Ba+Cih8kCqfjKQo/q/0OHr44GHA2YDiNA6fx+PzBcXbdXS0i98dJxDw+Xz++406xX/h8z5uNDM3+7hRUO5IPr/0jJnyivc6ODgU/7P4FYjKcBtMP/NieuktbMGTfUt3PZLTZDn1OrW3pUXR62etPl08Dy8xJuDAxUE/NHUrOaBOV19nvuzpXzMW7YstTi/kp2eWmgDCqde9myM3N+SXOduLU/hpD/yXb2jeaX07Hw+TLc9Ty91YheEpPtsrPSXBlJxPmpWYkFkoIUSSFJ9LOE4Kzrk1gGUJoUwdPRxNYqNShawo/XUCIYRi5W8nHPsePZ35kgerZ6049lJGCCmMv7F99tJaZ7f5te7RzjDo+Dt5MXwD6tevP3Xqj69fv75wITA0NDQ/P7+6I/qSOOY+42dP7N/cyUqfFmbGnft17PwLGf/FaUq0ZfdlW350u7eiz9Kwz0z3cWr1WblxQr2IRQN+jazE+knqRfLd3KAv6DNvEAAAAADAvxgSM1AGmxf1+9DBj0aN9uvg4WRtrM0RZiW9jLn9Vkxo635zJzWU3fpt5eGSJamF9/7++Z/2e8aNn9v/woTDiYzs+d6pc/SW/tjDs44hX5T95uGL9Iq/rZ0fdujYy+6T6+aF7D9T5rVJ7LsLi8bOSpk2uktje3NtjlSYl5OVnvwm4kWOysPdis/AZN67GHBN0sDOpqaJnoAW5yQ9jww+tnXHqftZJXEUhq6fuVlnbn8PQUKyWFGdEIpKvnP1fs3G9jUNBJQoJ/F5dOD+zRvOpleq7v9NWJZNSUm5fPny1auhn/n2IUKIthaHYosKRQruMFtUVEQoHT1dWmkLUasdlqK8hZc5vOjxtrGDX4+eNKaHt7OlDpMVF33p+OaPqy+ohM29fuRcsu8QvesBV0pGbgpvnQ5O6zNQ6+rRC0kfCqxcWUVPdowdGDts3NgeLRrammhzJIXZ6W9fxcaEvqpobIlJD547ZOKzqeP6t3Kx0Sc5b+6HvdLp0M6heCZEBdfw2R22lIo61KenUlYhbH702h+GPh097oeunvUtDfjSnJTXD1/m8igiVLRL9HDDuPG5M6cMaeNiY8BjxYXZmSlvXj4OfVGk9kpaFT80VPq4giC/zuMoOzs7LCxczatUz9mzARXtYhmGUIRh2Pv377u5uRFCMtLVyfIWpxbYvNu7F87dfOV1obzbxbO0rUkzCXduJ1fQmLm17G05zNsbV19U0D/41nWsaFqz08bIThvL7JBZWtWgSfnEjMLwFJ+Na6IkGPUjpPJu+F/KaOPbasG+kFlZ8Q9jokPPHNgd+Lygou38WnZWNPP2Vnjp9+kV3Ll+V+jXuZadNU3eqRFaVeLxuAzD2NraTpgwftz4cTfDbwRdDLp3737l1jH6tvCdp/29aUr9kqWqdEwtBP/Zl8dR2lb1na0NY7/EFGRdaycna92Ykp8EBO5T9+0crhu8YNi8i6q8tVKdSL6VG6TuNVazsjcIAAAAAOA7gsTMd0t6Z3UXu9VydrB5z8789dOZv8rtODzG5XDZLUUxq7q5rvr4b3F80NpRQWtVK//x2m6N5B7KFj4/tXrKKXmxESJ7vrW//VYlGxWegX13Y8fMGzsUxcakX9807XqppdErrJPU0LVTQlW74P+UDRs3vnv3xYblTox3O6HkENm9P3ztPkyOUHR/SUXtUGnLeau8hX+yNy5o09ygTXL3yW2o5Qlv/tzR7ucyW8KXtbNb9oXKEiVe27n02k6lYXwkTb6xbf6NbSX/okwHbO/cjsnLyWMqLkjNDisOmeNZZ07pQ15t7tNoc/HfVe9QiiqEEELY/Odn/5pztnyfVbhLlnF714IRu+SfUqWHz8dSKnpoyPuIJHyZR/1lqsb/L3wc0TRNlRt0YxmGJYRl/8/efQc0cf5/AH/uMtlTRERwi8EBDtxbtGq1DrB1i7vWotW69+pXrVqruOpo1db2Z23ddeBCceEgDsJGQZaAbAgZd/f7A9SACQREg/h+/VO5XJ773Erh3nmehw0Jkd26fTvgSkBWdtaZM6f1blX9ZMvQwTujGSJsNHbHkYXtGrjYEV35LsWjCSEUrfPJHy3g04So1TrTTo7T1bTISKSt2VLLK721MoupQIVc2plFY3KDvfu1b9nKvXmrHvVad+/hQg+dcUbH8ssf60NSgUDEsKyAz6coiiakY6cOnbt0ycrOuujvf+7s+aTkpHK1NnTng009RTpeVN1dO+CrgwkfLPARtRs+oolQGXnk+++2+sfkCms4mOa+05ReVQDP1WfbptGmp6b7bA9/j3NcaaIoiiqc56qyffATpPPovb99BAAAAACASoRgBgDKrRJTGag6aLvW/VqSyJCniWlZBXyr+q0Hff91OyEbHfy4It1fAF7j0W/GMWNZlqIohmEe3H9w7dr1O0F38vPz3615ZeTvixa3/GvrgDkbJ0lH7g7T8ihUGRcdz9J1O3ZvsP1xhLZuKKrkxDSWdmrj4UA/ea7tSbsq4VkCy1qcmNRn6ZXy1autvNJb47uXUYxWZVZY8Dzg0OaAQ4TwzFyGrdq/wrN7Xw/jM//laV1+7ml0PEs7t+vkzHsc8+qZr0mrLu5iooyLiWcJoct1FF5r3769jZW1QqnkOC4vP48QIs+XMwzzeh6jvNw8jnBKpVJZuE5eXllNFiMU8gnRGILu1TxGg4cM8fLyevr06enTZ65cvVqx4g2KrtmwgQWluL7/5zORmRwhiuTYnLLfVcXRls6SRrUyhB8uQlDc/3m4u7bM+519+BOk6+i9v30EAAAAAIDKhGAGAAAIIUTk9tW6rf1NNR/xcEzi6Z2HIz7QF5mhuuLxi4IZlUp19+69wOvX7wQFFT6FrxxsytlVy/922/7l9JWTr43xC31r0Dgm/NTp0CmzXL/xW1+wevs/d59lqgQWtlbGr692daj/5aTxY91nbpqbuvK3K5GZglot+no2FWq0cN4/Zuq0gSs2PKN3nLkblZrLCCxqNWhZKzfwbmwZEyO8XV7prZVZjFalt8mr13NwvbTb98KSchgBX52ToyCEogilazkTcfKkbPLs5t9uXpq2bOd/sgxB7TZfzl85vBaVef7URb2n0nqbu5ubm5ubgM/n8/lisVjPd8nlcoZhVGq1ojC8ycvjOE6pUCpVSpbl8vPyCCHyggKGUdvb1yLaquPzeIQQ57rO33wzffLkSWFh4fps99+vW/37qoFW804d8TH7Z0qP+dffcdYTvdAiMxtLsTonIyO/8PqiRGIhxcnT0vKQlFdJn9YJeuv6BHhfKKvuS3Z83yd2S+8FFyurD9r7aBMAAACgAhDMAAAAIYQSZoZfCarv1sjJ3kJEFFlJMY+vnzzg98edlI9/agYwLJqmr127duPGjbt37ykU7+UZCJcVuH71iS47hny9bNT50b9GlgwTmYgDKzZ23LPAo++ivX0XabzwqpqCu79sPNlr4+CWY7f+O1bj9dfPHNVP9q3d32PnZM/Zez1nv35ZFbzBc+SB2LLukbfKK721MovRqrQ242zaTVy5tKNAY3U2/eyFu/k2vbQuzyNM5KFVW7rundvW+8e/vV8PJKlKOLti/fl3yGXIzl27Aq8Hai4xMTGhKEooFAqFQopQJqYmhBCxWMwvDG+MxIQQE2MTiqJEQqFAKKQoytTUhBAiEokFAj6PxzcSiwkhNjY2NE2Zm5sS3d0vaIomFBGLxW5uLQuXGImNKr4zhLJwGz5zXJ+2rg3r2lsaUfK02PMrx644R3VbuHlmvyaONc1FnDwt+t75vZv9joUXPq+nbDpNXjyum6RBHQdbc2MBKciIk178c9OWv4Izig4rbd1m8rJFU3s3thJQHKfKifNfNX7+P4mEEIrQVsP3SIcXrqe6u9xzwsEklhjV7TPh68mDOkocTNiM2PuXj+54M+eW1gpXBjWdpFEDk50YevWw3+6HtQeP+qJPOxdHS15ewpMLBzauO/w4U+u5LnWLZe5gSbzGvice+RJCCGFT/hrTc/XNwuSLMm7/7d7za5s42YiZTC2NUCYNB0yZPvHz9i52IvmL8MBjuzf8EhBfRmrGa/T1X//51vq3KGArb7W0bddFf2z70u7RlrFT9z0u2TXt7RM0NXrq+YPDM/yGev0U9uqMDNketK7DrcW9fI6mc3oWYOTUe/y0SYM6NXM0p+QZCeHXdixZfTxW19ErsY+VfMp0XJ+s7joZyqa77jtCiwqdWai2KLGDpHm9Gqn8SuxZV7zNj2xaJgAAAKhOEMwAAAAhhMsK2us7tjwz0gDoJz8/f/16nbNCVRIu89q2TVd6bOw5aXb/k9NPZZZ8XR66Z/KX4WOmTB7UpUVdWxOeKj8z9XlMuDQgpvBxH5vqP2/U1xG+U7y7NXeyIFlxjwJjTD17NWa5otSFy7m7bvTIkAkTR3h6SOrYmPAKMhKjpfeev9U9R5/yXpbeWpnFaN+G7jYpKunB1Ue1WzeqbSmiFFkJkffPHdq+9XQqsdO+nCOEyGW7Jo98OnH6pEEdXB1M2Yxn9y8d3f7mYW6lKe9gZaWbPGmSk5Oz1pc4jmNZlsfjxT+Pfxb7rHPnzoQQeYH8HbZG23XwGtNf8uqXabMadgJFLkeMLBu0auxY2MXJtGbT7mN/bGan+uL7U2kcIbR1C8+B3V6/hZjYNuj01WK3xkZDx+yPUBNC23uv2zavmzmlznv5IpcytbG0EyiyWEJ42gogRCyZ+sveeR4WRUPL1WzcbcTCjl1bzhm98FQio6NCqngNAqs67kPm7xui0arQuc2XS3Za5w2bdvxFyWuurC2WsYP6o0ROLdsU/fvtRoybz9i3Z5a7WWEZ4jotB3671c3B94slAboCIG3KVS1l0dZ3709fOoTvmfjN/rdSmYrRowCRy+Td+xa0syw64MKaDd3qmMrLM5xgJZ4ynddnKXUSoi7ljnhL5ZxZqBRi5x6jvxk7oEszJ1tjSpGV+jRMGnj28C9HH2ZwBpgg6v3BtEwAAABgKAhmAAAA4GOSd+pbl1NvLWVTjn3T+dirnyJ3ejfaWXwFRcK1vcuv6Q4f1Uk3dy28uavoJ6rG8F8+68XmZOW8fjDN5USc/Hn+ybImb9CnvDJbK7UYRsveld7mi4BNMwI2vf0GXcsLyZ+d95t33k/razprMCyBSEiV6DLDEZZjaZpOTEy8ejXg2rVr8fHxnbt0LgxmKgGX479ixMKT8ZmMUU17oywV4dQ3fhw7eHH089RclcDCqcOkH/wm9hjW3er00Vd9jbjss4u8FpxJzmWMarkNWb5prmfLUaNaHVoepKLM2vVpZ8Y+2e3ls/1hNkMoUY26NVSvMwA240ixsdR4DScv/a6teUHo0RXLd5yRZQgd2gxfsHJuj34r5l0O/O5c0YPstyp00KghjzNvMmjR7lX9HHLu7Viy/o9bUS+5Gh6T1u2Y1qrb0J52J/9MLpbM8BqO0WeLOndQywFkIra+6VCieWBzr66bsORY9It8fs2WJRrhNR67dIabcUrA1kXr/+9mbIFF035z1y8dNnjGqN8C/d7qMVfWGdSnWtqi9Tf7dkxoGHtw6tRtQdk6IoKSJ0jQ6V0L4NUbtXS2h0VBxPEfVu/+72GSXGTt1MAi43WkoevovVGZp0z39VlanVxOWXeERrWVeWY/MSampq4S1wcP7qvVlTK+HM/Ze/PRlV1teUWfqHwbx2adajfiSw/+85BwH36CqPcH0zIBAACAwVRw7lYAAACAaoO2az3As3VjB2tTIY9vbNu4i8/ar9sJ2WfBj7M+/Le0q1QxHxchX0DRFCGE5ViOY1mWffzk8a6du8aMGTtlytTDhw/Hx8dX8iY5dUZC/Mt8FaPITox9kccRQlGWzb/6Yf8/N+7cfXTl4Iq+DjzCr1nL9s3v3ByTk5qSrWBYdW7CvcM/HHqi5tm4uNSgCSEcxxFC1XDxcLEVU4RwitSn8drHEyOE8BoN+sJVqHq8bc6qvx++yFcpM2Nv/vL98iNJnFX3Qb2sXj0xfbtCjRoYZYbs2I4/QhiKnya7EZqcq1LlJd74ZZ9/FserU9eJrtgWde9gOQ6sKiU6IiGrQK16qxFek4Gfu/CzL66d+8uV6EyFuiDl8bGVW6/k8hp19LAt9982ZVdLWbef+dvuKU3j//h68qYbld5xo5QCePU/H9hMpHq01XfZH0FxGQpVQfaLiOCIVP2HGK3cU6br+iy9zjLviDfVlnZmK3RwPyGmpibLly89fPj3aV9PbdK4ybs2x3cbN72zDUm5vGlqn04eTVzdm3cZ9NXsTZt+u/pWNzoAAAAAqCD0mAEAAIBPncjtq3Vb+5tqfvmXYxJP7zwcYYDvaFepYj4ufCGfpmm1Wv3gwYPrgYF3g+7m5OR80Aoo865Lfts7wllQdPpETnUIIQxN6/qVm0mMfpbHuZqamlCEsDk3j11K6zGg26KDF+dkxD6R3g84+fv+c5Hap+MQOjd0pNnnd24807gw8h5cDy4Y8Zlzwzo0SdevZiYpNlFNmtaoaUmTfJYQQpTJ8SkcZWdsVOLr8BXZYrEdrKjijQjr1HekaaO+24L6biu+moNjLZq8y3NjbdXSlr0njSNs5uU//7j18n0/ky5eAN+5UV0e+zzoZlxF7/1KPWWcruuzlDrLdUeUemYJeVHOnf+08CiaEGJiYtqvX/+Bnw9MTk72v+B/+crllJTUijRn7Ohsw2PjTm/dF1jYVUmZEn3nTPSdYpvUMkFUqVMK6TWhEW3dcsT0aaM83evbCPKTwm7dzqqpGZOW2r7Web/OpnOltllsWibKYvC+wLU9hMWPhvL2kl6T/kjhMAESAAAAVC4EMwAAAPCJo4SZ4VeC6rs1crK3EBFFVlLM4+snD/j9cSfFAF8NrlLFfGSePAq5fev23Xv3FQUFBimAsu49fogTL+OO39INf9yOTpXzbXstPr5lUClv4ZRKJUcVdvQhXNqZRWNyg737tW/Zyr15qx71Wnfv4UIPnXEmQ+vWKqlqVqVUE0ogELxuUKVSc4Si3urTUJEtFtvBiip+lDgdnVYokZHoHQ+Klmq53Af+9227d+2xbN/G/AlzTifoH5JwhCVEJBaXo6hiBRTOe6Frd/VSqadM1/V5SWed5bsjSj2zFdiRTwqPV/R3PZ/HI4TY29uPGDVyzNgxT58+veDvf/XKlezs8qTU8qT4DIau02tMv38iTseWYzKu0qYUKntCI8q845KD28Y3KrpnRE5u/Z0IIUShX/ta5/0qu009YQIkAAAAqGwIZgAAAOATx2UF7fUdq3v+mQ+qShXzkTl3/pxhC6Bt7e2FJN//0LaLYUpCCFG9TM0u3+O/gucBhzYHHCKEZ+YybNX+FZ7d+3oYnzmvZU1lbHQ8Szu36+TMexzzKiwwadXFXUyUcTHx5ZkfXk+VvEVOrVZzxNjYuJzhgSrhWQLLWpyY1GfplfyyV9d4ZF0hnCrq79lTj8w6uG30oDVbkl5M2HA3R7/HsGxOVi5H127S0IKSvqzIk1tVwrMElnby6FCH9/hZiTxIv6NX6ReJ1uvzXKyuOvW7I16doHKfWS1mzZrFMG8mWVGp1ApFsZg2JydX80eFokClerM+w6jlxWPdvNxczbxIqVAoVSqN9Rm5vFhskZ+fx7KcrvXfH4pX8kooTGjqOjtPnjhx0sQJ0mDphYsXebR+Z1x1f79fYL+VXYdtPNZpxOkDBw8fuRSWXmLyGm1THJU9pVBpExrxm01cMLahMFt6aPmaX/3DMvh2kh4jZy2Z0NasHO2XmFWL3+ybMtosVn/W8QnNj786go5frNv34wDriMP7z6fRjadgAiQAAACoZAhmAAAAAAAqAfsyJUVFGrcbOqp1+N8Pk3JZvqmpmK//V7N59XoOrpd2+15YUg4j4KtzchSEUJSOXg9MxMmTssmzm3+7eWnasp3/yTIEtdt8OX/l8FpU5vlTF9+eWP3dVfIWuZTkVI7n6unteTjCP05tXrdZLXlwSGKZTziZ8PP+MVOnDVyx4Rm948zdqNRcRmBRq0HLWrmBd2NLPDpWqtQcZdmqe3vH4JvxFX3WTzgm7fqGcbONj2wbOvnHpY+8F5zVa5oXJuZxaDZXv/O0hSOjfzz2KFUhMK1hVXJ8uFIbCD93IXrq1y1nbluVv2bPmYfPsxmjGvUbWqQ9Cn+p39Gr3FOm6/rUXWdEWXdE8RNUjjOri62tLaVxiIUikVAgeLMHNM/I2EhzfWNjE1qjb1CJ9Ssdy7L5+cUuxPx8Ocu+OW1KpVKpVJayvjxfzjAa66tVSoWCEGJmpjVoKOp2RQhxd3dv1bq1UqnnpxET+/esoS/Gz501pl/rYfPbDPVNvHds//atf959UfqZoCjL5l/NW9xe4lzLWpCXlMa+mlIovajoVxMaEVI4oVG/HnMlLi416KBEltekb++6tOL+ljkbTsSzhBCSID31+4WvxrV1L0f7RbNqEaJKjM0mPNey29SKrtF7+a4Nn9s+/fM7n/U30ijJuNcTIGVxhJCUx8dWbu3cd0uvjh62fpEYZw8AAAAqAsEMAAAAAEAl4F5ePnJpRpcBPZcd7rnszWImQr+3UzbtJq5c2lHzsTCbfvbC3TztqzORh1Zt6bp3blvvH//2/vFVCaqEsyvWn38fuUxlb5GJC7gs823eYujGy0MJIYSopD/0H7Mnrsw3qp/sW7u/x87JnrP3es5+vVQVvMFz5IHYYokJEx8alsm5uIzded7u+7Yz/ctbogY29fIP07fUPTKn35rVtx9P/zdej2gm7/qhQ6G9v3Xtt+avfmveLFbqfkcJ6pB9a3Z33TG92eDVBwevLlzG5Z3y7ep7oUC/o1eZp0z39am7Tv/S74gSJ+ic3mdWpyVLlpRvr0pF07SxsbHmEmNjI5rmvf5RKBQKhcJS1jcyNuLxNNbnC4SiN8OyURRlYmJSbH0jMZ//5i90Po8vNhJrrmBqYqr5o5mZmcDKihBibFRsu1r2hcfjOE4kKmrNwaFW6esTooy/9svMawf+17r/WJ9xI3u2Gbl4X+/Oa0Z+cyRaVzbzbpNsEYFD3do0G//gXpKOk13u9vVoU/uGzNrO9Nvi7ZT638KJa66lsoQYYQIkAAAAqHwIZgAAAAAAKgOXfnbJ5DnJMyf2a92opglPXZCTlZGaFHc7KkufZ+AUlfTg6qParRvVthRRiqyEyPvnDm3fejqVIzztb5DLdk0e+XTi9EmDOrg6mLIZz+5fOrp9+19Bqe9tXJ1K3SITecB3rvnybwe1q28lVGTGPYlKpfTqT8Ll3F03emTIhIkjPD0kdWxMeAUZidHSe8/fTjzyA7bM3m46z9tDFJ+kfx6iQ0Ho/iUbOhxZ0W320oE3vj7xouwHvYonW6dMzZ49Y1SP5k6WAk6Zn/kyOS5aFhAl1zMT4XLvbxo3OmzilHH92zV1sBSqs5KfPonOFlCkQN+jV3mnTPf1SYiuOsu6I0qcIP3P7IfBsmxubrGhz0r8WHU0btz4p582a3+NIyzHUhQVEREe9zzes3dvQkhiYpJ+DSuS7x/bcP/EbsnQVVuWDOw689ue/826oL3bzbtOskXxaEKI7gmpKtB+mW1qw3cetm77FIn67papi8/EF94omAAJAAAA3gMEMwAAAAAApVA/2NCv4YYSC5nInd6NdpZclcuPPL5hxvGSK+t8i+rGCo+mK4p+eBGwaUbAJv3eWET+7LzfvPN++m5O20Llxbnt6s/VXCVm+9CW27U2Wf4tFtvBkpSx5zdNOF9yj/VphMuJOPnz/JM/66ryFTb1ut/M66+qLd5ymdW+tQITd2hC+0NaNqPzBDFp9/YtGr9Pe3F6HS4uN/L0z3NPa9lTfY9eZZ0yVuf1WWqdpd0RJU8Q0f/MQnH029HDqzwmMjLiytWA69euZ2RkdO7SuTCYKSc2S3Zsy5Fh/edKGjasxbvwVOsUR+86yZYyLjqepet27N5g++MILRPzVKT9stp8C2XWduaupd0s445Om7U/5PX8QZUxARIAAABACZU9KSgAAAAAAAAAfChvBljjOJZlOY4LDQ3dtXPX6FGjv/tuzskTJzMyMsrRnNB9yg/fj+3ZzMlKzKMonpF1vbZDpg9uzOPUqSnpbNEEUbU8vT3rmfJ5YusGbVwdeEWTbBm1GzqqtYMpnyK0oHBKIX0x4adOh6r4km/81k/u0sBazKN5Ygtbq9fpT0XaL6vNEijr7kvXj2vCPfGbve7yS06znfP+MaztwBUbJvaS2JsLeTRPbOXo2r2tM77oCgAAABWGXySqCaFQKHo1YDFFUTo7WwMAAAAAAEA1QtM0IYTjuLCwsKtXr964cbN8SUxxfEmvkYN9nIf5FF/MySMP7jmfzhFO+xRHz99pki1CmIgDKzZ23LPAo++ivX0XabxQ2C2mQpN4ldFmCYJW/fo78Ciq+Xf/3v/uTRuJv43rt+qdJ0ACAAAAKAHBzEfA3NzM3NzC3Nzc3NzM3MLCytLS3Mzc3MLCwtLC2srK0tLS1MREoDHtJFIZAAAAAACAT0R2VtbuXbsDb9xIT09/99bYmJPrNwsHdmvbsolTTTMhp8h+ERd299KxPb/+J8vhiK4Jot5tki1CCJGH7pn8ZfiYKZMHdWlR19aEp8rPTH0eEy4NiFGRik7iVXqbeqtqEyABAABANYBgpqqb6evbp2+f1z8yLMs0AoVWAAAgAElEQVSyDMVRPB5N0W9GouMIYRmGx+MRQnxnTDdAoQC64ZoEAAAAAHhP4p4/j3v+vLJaY7Mjzu794ezeUlbRPsXRO02yVUiRcG3v8ms6Nl3e9vVos9i7lBfnNXOZp33bmAAJAAAAKhuCmaru+MkTnn08KapoHFweTfPokjMDMQyTn5cfFRnp3roVIcTDo+2HrhKgVLgmAQAAAAAAAAAAAAqVfMQPVU3ss9hHjx8zDKNrBYZhsrKyZs+Zk5ef/yELAwAAAAAAAAAAAACA8kKPmY/Av//827JFC60vqRkmNTVlwYJFaamp/1u3jqz7wKUBlAbXJAAAAAAAAAAAAEAJ6DHzEYh7/vzly5ccW3JSQzXDxD+Pn/Pd92mpqQYpDAAAAAAAAAAAAAAAygU9Zqo0O7saw4cP7927d2paGqE4QqjXLzGMOioyatmy5Xl5eQasEAAAoHJZWVlhYqpqpmHDBoYuAQAAAAAAAKAKQTBTRdWwsxsyZHC/fv0yMzL3//rrJf9Lv/6238TEpPBVhmVlsrAVK1YUFBQYtk4AAIDK1ahRw0aNGhq6CgAAAAAAAACA9wXBTJWjGcn8+uuvZ8+cValVhJBTp097e3nxeDyWZe/cur3hxx9VKpWhiwUAAAAAAAAAAAAAgHJAMFOF2NaoMXLEV7169UpLS9uxY8eVy1fUavXrV8+cPuPt5UUIuXTp8tatW1mWNVylAAAAle9/69YZugR4v8JCwwxdAgAAAAAAAIDhIZipEoyNjb28vIYMGZyRnuHn53flylXNSKZQenp6QEBATnbOnr17OY4zSJ0AAADvT+D1QEOXAAAAAFDJ+vTt8+Txk8TEREMXAgAAAFUIghkDo2m6e4/uE3x8BALBH38cPnnihFL3AGW7dv+Sl5v7IcsDAAAAAAAAgAqbNnWqSCRKT09/9OjR48dPnjx5Eh8fb+iiAAAAwMAQzBiSm5vb5MmTHR1rX7x48eCBQ1nZWaWvj1QGAAAAAAAA4CPi7T28Xr16bm5urq6SCRN8TExMMjIzIyMiQkJkUqk0OjoaQ2IAAAB8ghDMGIaTs9OECRPatmkTFBS0ds2axKQkQ1cEAAAAAAAAAJWMYZioqKioqKijRwlN0/Xr15e4SlybSoZ7e/n4jM/MyooIDy8MaWJiYjCbLAAAwCcCwcyHZmpq6uMzvk+fPhERkXO/nysLDTV0RQAAAACfooYNGxi6BACoBFZWVoYuAUBfLMsWhjQnT5zUDGm8vb18fMbn5+dHREQEB0sR0gAAAFR7CGY+qO7du0+aPIkQsmnT5oCAAHRYBgAAADCU/p/1NXQJAADw6SoR0jjWcZQ0lbi7uXl5DfPxGS+Xy8PDw4ODpbJQWUR4hFqtNnS9AAAAUJkQzHwg9vb206dPb9XK/cqVq3v2/JKdnWPoigAAAAAAAADA8FiWjYuNi4uNO3funGZIM2zYMB/z8QUFBWFhYYUhTWREpEqlMnS9AAAA8K4QzLx3fD6//4D+48eOTUp+MW/uPIxdBgAAAGBAYaFh/1u3ztBVAAAAaKcZ0hBC7O3t3dzc3N3chg4b4mM+XlFQEBoWFhIik8lkISEhCGkAAAA+Ughm3i+Jq+Tbb2bY17L/+++jf//9N35nAgAAADCstLS0wOuBhq4CAMBgMMPWh1FZsx8lJyefO3dOM6RxdXXt28dz1KiRxUKaJyEqNR44AAAAfDQoC5sahq6hehKLxZMmTfzss8/u37+/Y+fOF8kvDF0RAAAAAAAAfKI6d+m8cMECQ1fxKRow4PP30ay9vb3EVeIqkbRu1aqGnZ1CoYiOjpbJZMHBUllIiBLfCgUAAKja0GPmvZC4SubMnm1sbPLjhh8Drl0zdDkAAAAAAAAAUH0kJycnJydfvnSZaIQ0Xbp09fLyUiqVUVFRCGkAAACqMvSYqWR8Pn/48OEjRnwVHCz9+eefX758aeiKAAAAAAAA4FNna2vr0tTF0FV8ij7w+JnW1tYSV4m7m5u7m3tN+5oMwzx9+lQqlQYHS2UymVKp/JDFAAAAgC4IZiqTc13nOXNmO9Z2/O3AgVMnT3EcZ+iKAAAAAAAAAOBT9DqkkUgkTk5Or0OakJDQkJAneXl5hi4QAADg04VgpnLQNP35wM8nTPCJjorZtGlTYmKioSsCAAAAAAAAACBER0gTIpMVjniWl5tr6AIBAAA+LQhmKkEt+1qzv5/dqFHD3w/9/u+/x1iWNXRFAAAAAAAAAABaWFlZuTZzlUgkrhJJgwYNOI6LiYkpDGmkwdJchDQAAADvH4KZd9WxY8dZM31fvkzfuHFTdEy0oct5w8WlyZDBQwxdBRRz7PixsLBwQ1fxqfhi8BcSl6aGrgJAL7Kw0BPHTxi6CgAAAAD45FhZWro2b6YZ0sTHx8tksmCp9KH0YU5OjqELBAAAqJ74hi7gIyYUCMZP8Pli0KDLl69s27atqs2hZ1ujRucunQ1dBRRz/UYgQTDzoUhcmuIWgI/ICYJgBgAAAAA+tIzMzMDrgYHXAwkhlhYWjZu4uLo2dXNz69OnDyHkdUjz6OHD7GyENAAAAJUGwUwF1a5de+GCBTXta65fv+HatWuGLgcAAAAAAAAAoOIys7KCgu4EBd0hhFiYWzRxeRPS0DSdnJwslUqDpdJHjx5lZ2UbulgAAICPG4KZiujUqePMmTOTkpJ8v52ZlJxk6HLKsNVvR1DQXUNX8Unz8GjrO2O6oav4dI0e62PoEgB0+v3gr4YuAQAAAACgpKzsNyGNkZFRkyZN3N3dSoQ0ITLZo4eP0tLSDF0sAADAxwfBTPkUDl82aODAk6dO/br/V5VKZeiKAAAAAAAAAADeF7lcLpVKpVIp0QhpJBJJ7969+Xz+m5Dm0eO01FRDFwsAAPBxQDBTDk516ixYsMDaxnrNmjW3b98xdDkAAAAAAAAAAB+OZkgjFotdXFwkEomrq6RXr14CgSA5OVkmCw0JCXnw4H5KCkIaAAAAnRDM6Ktjx46zZ38XGxvn6+uLXy8AAAAAAAAA4FNWUFDwOqQRicVNX4U0076eKuC/CWmCgx+8eJFi6GIBAACqFgQzZaMoatiwYePGjb1w4cKuXbsxfBkAAAAAAAAAwGsKzZBGJGrQsIGkqcTd3a0wpElPT5eFyIKlUqlUmpycbOhiAQAADA/BTBmMjY3nzJnTuk0rP7/t58+fN3Q5AAAAAAAAAABVl0KhkIXIZCGyo0ePaoY0U6dNFQrehDSyUFlcbJyhiwUAADAMBDOlcXR0XLxksamJyYL5C8LCwg1dDgAAAAAAAADAR0MzpOHxePXq1XNzc3N3d5s6dYpQKNQMaZ7HPec4ztD1AgAAfCAIZnTy8Gj3/fezY2NjFy9anJ6ebuhyAAAAAAAAAAA+VgzDREVFRUVFaYY0rq6SiRMnGBsbZ2RkhDwJCQmVyUJk0dHRCGkAAKB6QzCjBSaVAQAA+FicOXPa0CV8iv63bl3g9UBDVwEAAAAfK42QhmiGNKNHjTIxMcnIzIyMiAgJkUmlUoQ0AABQLSGYKcnIyGjevLnurdwxqQwAAAAAAAAAwHulGdLQNF2/fn2Jq8S1qWS4t5ePz/jMrKyI8PDCkCYmJoZlWUPXCwAAUAkQzBRTw85u+dKlVjbWCxcsDA0NM3Q5AAAAoJf09IyoqChDV1H9WVlZNWrU0NBVAAAAQLXFsmxhSHPyxEnNkMbb28vHZ3x+fn5ERERwsBQhDQAAfOwQzLxRv3795cuX5eXmzf7uuxcvUgxdDgAAAOgrKipqq98OQ1dR/Xl4tEUwAwAAAB9GiZDGsY6jpKnE3c3Ny2uYj894uVweHh4eHCyVhcoiwiPUarWh6wUAACgHBDNFOnXqOGfOnJAQ2bp16/Ly8gxdDgAAAAAAAAAAEEIIy7JxsXFxsXHnzp3TDGmGDRvmYz6+oKAgLCysMKSJjIjEVMEAAFD1IZghhJBBXwyaPGnShQsXdu7chS9ZAAAYCM956Jpt05rcXjL8hyB8FAMAAAAAgBaaIQ0hxN7e3s3Nzd3NbeiwIT7m4xUFBaFhYSEhMplMFhISgpAGAACqpk89mBEIBL7fftu9R/e9e/aeOHnS0OV8REStfA/uHWvmv2jMggsvOUNXA/BpqP73nVkdiaSOmZSiDFdC9T/IAAAAAADVSXJy8rlz5zRDGldX1759PEeNGlkspHkSolIjpAEAgKrikw5mzMzMli1d6lzXecWKFffvPzB0OQYjbj/z0NLP69lZm5kIeZxSnpX2POrJrYvHD/59PVau810URVEUTRvw8aluxfaIKcjJTIkLfxIU6P/PsSthWYx+bfBcfbZtGm16arrP9nA93wIfJZOB2+7/2CFq3/TRG4Myiz2GF3Rdc/nXodl7vhq07lFVuQYqdt9VrTuCNnfpM2L80F4dm9etaSFUZ714Gv7wpv+xg0dvxSveqeHKUpU/3AAAAAAAoBQlQhqJq8RVIunj2XvUqJEKhSI6OlomkwUHS2UhIUr0pAEAAIP6dIOZmjXtVq5aKRKKv587Ny42ztDlGBKvRsPmDR1ERT+JTW0cm9o4Nm3Xd8SIf74bv/L8C1bbmxT3fx7u/vMHrLI8iu0Rz9jSrq6lXd0WXQb4THt0cMncHy4m6DFGEm3pLGlUK0OIh7OfAsrIdcLmrUnjJv0erTR0LaWq4H1Xde4IyrzFpI0/zetqz3/VjtDa0bWDo8TNJPy/21UjmKnSH24AAAAAAKCn5OTk5OTky5cuE42QpkuXrl5eXkqlMioqCiENAAAY0CcazNSt67xy5cq83Ly5i+empaUZupyqQC3z+2rYjjAFxzOyqFmvRbcxM329XYeumHHh+tIb+YYurkLUsu0jvLaHFhChiZV9wxYdB4waP6ZTy/E/7aamjFh1KwcjFH28Jk+apFSpAgKuPnsWW0lNcgxn3nn+loUxo1fdzKqm10YVuCPoWkPXbV/QzYpLCz6085e/LgdHpyqFVo5NW3f5zCX5hoEOPC0ys7EUq3MyMvIxqw0AAAAAQPWkGdJYW1tLXCXubm5dOnfx8vJiGObp06dSqTQ4WCqTyZTKqv1tPQAAqC4+xWCmZcuWixcvioqKWrv2h7y8PEOXU1WwaqWK4Tiizs9ICAk4vDjeVHJypqRV6/q8m8+bD585rk9b14Z17S2NKHla7PmVY1dFffnXf761/p3SY/51FSGUTafJi8d1kzSo42BrbixgshNDrx722/2w9uBRX/Rp5+JoyctLeHLhwMZ1hx8XjhZF2XRfuHlmvyaONc1FnDwt+t75vZv9joXnFb5o4fb2FtekTjvz11jz/2b28j3/6qzxms4+8X/f2J6e0mPRpbdGXWNVCiXDcUSRmxYrvRwrvfLfpbn7909oMnrBmCNDd4QypddQ2H5j3xOPfAtbS/lrTM/VN1WEMmk4YMr0iZ+3d7ETyV+EBx7bveGXgHh8veYDsqtp17Fjx+HDvRMSEi5dunQ1IOBF8ot3a1L98ND25L7fjtmw8tHwOccStQ7VJei0wv/g8Ay/oV4/hRWuQFkM2R60rsOtxb18jqZz5b8LCCnlctJ6F5S47wghhBg59R4/bdKgTs0czSl5RkL4tR1LVh+P1bIL7+uOKPNdrxh3nPZ9d2uSdnnRiO+OxBWlIIqU6KCz0UFndZ6bUu640j9Gip8OUpARJ73456YtfwVnFJVGW7eZvGzR1N6NrQQUx6ly4vxXjZ//TyLV6OtSPty0tEMIIWLHHmOmTvyicwsnGyNSkJWaEBPxxP/XTXtLDI8HAAAAAACGlp6eHng9MPB6INEIaTw8PDRDmpCQ0JCQJ3hkBAAA788nF8z07NHDd6bvnTtBmzZuRGfV0ryZYYG26+A1pr/k1bViVsNOoMgtubZ1C8+B3V6vI7Cq4z5k/r4hGmsIndt8uWSndd6wacdfsIQQtWWDVo0dhYQQQkxrNu0+9sdmdqovvj+VxunYourJ9VvpY4a27dBCeP5W4TdY6JqtPOrSisA7Dwr02CMu6/a29X/33Tu2Ub8BLrtDQ5hSa9DBuPmMfXtmuZvRhBBCxHVaDvx2q5uD7xdLAjLw/PWDq1279shRo8aOHZuQkHDhgv+lS5cyMjIq1hSTcHbhHLN6+31WbZoQ7rNHps8VVVL574LSLid97jtCRC6Td+9b0M6SLtpAzYZudUzlWocffEul3BGk9HtZk7jDwN52tDJ434//xOndN6X0O67Uj5Hip4OY2Dbo9NVit8ZGQ8fsj1ATQtt7r9s2r5s5pc57+SKXMrWxtBMoslhCeMUrKKsdQojYZfIvexd4WL360DSxcWxs41hf+GD//qDMqjI9EQAAAAAAvKXMkCZEJisc8Swv9+2/xwAAACqONnQBH9SgLwZ9N/u7/86eXb9+PVIZrSieyNTWqUWPUWvXj5fw2ZcPpTGFjxW5HP/ln7dxd2vYokMX75/vaD14XPbZhX1atmjRsHnnAYv/i2c4NvOu3wyvDq3dGrfyHL3jfjax7Da0px1d2N6NH8cO7tC2dcOmLZq2/3zC3kcFNj2Gdbd6M4HFW1ssuHc5IIPU6NK1paBoFVP3ts346pA79/UdA0n+8OqdLJau3bSRiV41MBFbv2hRr4lrvSauDbqsvqniNR67dIabcUrA1gn9O7m4tm7nteRoDOM4eMaohjzdW4X3iM/jEUJqO9QeO27soUMHN2/eOOiLQeYW5uVvicu97zfrp/us2/SfvmtrVuGZVMpxF+hxOZVx3/HqjVo628OiIOL4kjH9WrVwa9q252dj158vPUfR9K53BNHrXUXF1pE0NqWZp9cCE/TOKso4RHp8jBSdjgau7TqPWuefzJq0HDWqlYAQQpm169POjH2ye0iHDm269mzd2qP9kI2BusZt1N0OIbwGo5fN8bBUxpxePuYzt+YtGjZv13lZQD6SWgAAAACAj0phSLNtm9/XX08fPXrMhh9/DJHJXCWSBfPn//Xn4Z9/3jJl6pTOXTqbmpoaulIAAKgOPpUeMzRNT54y+fMBA/bu2Xvi5ElDl1MF8ZvNOhk1S3MJp4w7vdovMJ9QhBDCqTMS4l/mqwhRJcZmv/Wl8sJ3MDmpKdkKhpAM2bEdf3zZZ179NNmN0OR8QkjijV/2+Y9wH1ynrhNNkllCKMqy+VfzFreXONeyFuQlpbE8wq9Zy5Ym6UUPbUtukRB50H9X04cO7uHZbNPdYDUhwuYebkZM1LXAJP06CBBC1OnpWRxlZmxqTJNstswaSuA1Gfi5Cz/74tq5v1zJ4gghKY+Prdzaue+WXh09bP0i33E0LXgHFOFRNCGkUaPGjRo1njRxgjT4oZm5WTlbUUYcWrSq7V8bxqxZfPurhVcq9H0o/e8CqrTLaUdkGiFl3Xe8+p8PbCZSPVrvu+yPpwwhhCheRASX6zp8tzuikJ7vokzMTCjCZqRn6n2/lnrH7Yh8UXbBb05HbsK9wz8c6tdjrsTFpQYdlMhxHEcIVcPFw8U2/O6LAk6R+jReZyW622F59fsPcBUyYT9/t+RgeGF0lpv6Mhe5DAAAAADAxysjI+N1TxorS0vX5s0kEomrRDJo4ECO4+Lj42UyWbBU+lD6MCcnx9DFAgDAR+mTCGb4fP7sObM7dOiwfv36wMAbhi6nCuM4hlHkZ6bFx4QEXT3zx5HLkTmc9gymTExSbKKaNK1R05Im+SwhhCiT41M4ys7YiCKEMu+65Le9I5wFRd9sFznVIYQwNF36BSm/ddI/afDwvp+1+DH4gYrXqKOHNRd77GqM/mMF8a2tLSiOzc/L5ypQg7BOfUeaNuq7LajvtuJ76+BYi5CyH4gvXLCALNC7WNCBZXU+9Kbpwl6AdOs2rV8vFImECoV+8zcyif8uX9lR8pPXyoUBj5e+63DCpd8FgtIuJ5qkld0+37lRXR77POhmXIWHy3q3O4KU517m8vPkhNAWlhY0SdGv4FLvOJqSdy5fwUxi9LM8ztXU1IQihM25eexSWo8B3RYdvDgnI/aJ9H7Ayd/3n4vUMjdOqe28Ogs3r0ahFyYAAAAAQDWUkZn5OqSxtLBo3MTF1bWpm5tbnz59CCGvQ5pHDx9mZyOkAQAAfVX/YEYkEi1atLB5s2arV6168CDY0OVUWeonW4YO3hldedMhsCqlmlACgeD1qEIqlZojFEUTQln3Hj/EiZdxx2/phj9uR6fK+ba9Fh/fMqjMRgvuHTv+dPi0Pv3bbH4Q5NixixOJPxAQqn/RRi27t7Og2diwyDxi/UW5a+A4Hc9sKZGRSJ/tHzt+PCwsTO9yQbshQ4a4NGmi61WWYwlHWJbNzMy0tbUlhOibyhBCCOHSLq9e9k/r3cNWLA78UV78JcISIhKL9R/mrLS7oPTLSa9tUDRNEaKrGX284x1RrnuZSYh8Kuea1GvftsaOyGS9es2Ueojo8n+McEqlkqOowqlguLQzi8bkBnv3a9+ylXvzVj3qte7ew4UeOuNM2dMUFWuHFvBpQtRqzCUDeqKsui/Z8X2f2C29F1xUVOE2AQAAAOBtmVlZQUF3goLuEEIszC2auLwJaWiaTk5OlkqlwVLpo0ePsrOyDV0sAABUadU8mDExNV2xbJmTs9PiJUtCQ/FAvKqgbe3thSTf/9C2i2FKQghRvUzN1utZkjr07yPSSfN7D+nw83Onzk2o57+ef6LvNOKURfsZ87xr0+qI82dCGbph6TVwarWaI8bGxhpPyFUJzxJY1uLEpD5Lr+iai6JUYWFhhd+ygXfRrWtXLUs5wnIsRVGR4REX/C9eu3Ztpq9v5y6dy988lxm4efGfHr+NmDvrhTFFXv8yzeZk5XJ07SYNLSjpy0oYqKqMy0mPnmqqhGcJLO3k0aEO7/Gz8ucC735HlO9ezr916XZ2317tJ/v28V9yLrWUaIbH47/ZQV2HiNdkegU/Rl4reB5waHPAIUJ4Zi7DVu1f4dm9r4fxmfPlaoOokhPTWNqpjYcD/eS53qO0gX5ommbZ6nZUKbGDpHm9Gqn8Ck9lVVabola+B/eONfNfNGbBhcr4sAIAAAAAbbKy34Q0RkZGTZo0cXd3KxHShMhkjx4+SkvTY0QEAAD4xNCGLuA9srK0XPe/H+xr2c+fPx+pTJXCvkxJURGjdkNHtXYw5VOEFpiaivULCdm4039fzbPpO3z44J7NeM/8z+jOZWi+gEcRwhOa2Dq37PHV4r1Hfp3YVKx6dnjdwVCmzBq4lORUjlfL09uznimfJ7Zu0MbVgYSf949hbQeu2DCxl8TeXMijeWIrR9fubZ2recJZhXEcp2YYQsjTZ0/37N07ZvSY2XO+P3fuXH5+haKzokZzbm5Zdfi5hYODZu8YJuZxaDYn6jxt4Uj3msY8mic2q2FlVPFHq8w7X05M+LkL0Yyg5cxtq0a3q2sl5vEEpvZN3JrYaP9gr/w7gleue5nLOLdzb4iCdhi05a8dc4e0bWBrzKd5QjO7xu0+n/bdkKY8QghRqtQcZdmqe3tHY17ph+gdPkYIIYTw6vUc1rNFbXMhTfEEfHVOjoIQiiLlPqHqUP/LSazIfeamuQNda5oKRVbObYd6NhWWtx3Qxtvba9WqlR06duDzK+VTVuzcY9KGX4/duns/MuTBk5vnT+1fP9+7pRVFCOG5+uw4d+ngN00qNHpnFUNRFFXYow4AAAAAPgi5XC6VSn/99beZM2d5ew9fvHhJYGCgk5PTTF/fAwd+27dv77ffzujZq6dtjRqGrhQAAKqKavs82c6uxtq1a2manjd3flJykqHLgWK4l5ePXJrRZUDPZYd7LnuzmInQ670XD52Z2Xv4N99wvLDtZ2Q6+wnwJTP+CZ9R7K1M5uMDS+asvZnNEULKqIGJC7gs823eYujGy0MJIYSopD/0H7N339r9PXZO9py913P26/eogjd4jjwQW92+1V3VMQzD4/GeP39+8eLFgGvX01JTK7FxLido89pjPXd5OWoszLt+6FBo729d+635q9+aN4vLMU5aceon73o5qUP2rdnddcf0ZoNXHxy8uqj0vFO+XX0vFLy18nu5I/Y8L8+9rArb6TvfbsfaUU27TF/XZXrxXaFPnAyNYeJDwzI5F5exO8/bfd925rnSDtE7fIwQQiibdhNXLu0o0FjEpp+9cLf8EwsV3P1l48leGwe3HLv137Gau1SuVpo2bbpwwYLcvFy1mikokCuVSoVCKZfLGYbJzctVq9UF8gKlUqlUvlqYm8swjFwuL7vpj5lYbNSqVatWrVrl5eaePXfuwvkLiUkV/h86z9l789GVXW15RXkF38axWafajfjSg/88JBxt6SxpVCtDWB3CDMX9n4e7/2zoKgAAAAA+VQUFBVKpVCqVEkLEYrGLi4tEInF1lfTq1UsgECQnJ8tkoSEhIQ8e3E9Jqcw/YwEA4ONSPYMZJ2enNatXZ2ZlLluyLDMry9DlwFu49LNLJs9JnjmxX+tGNU146oKcrIzUpLjbUVl6DLoiv/3H0VCvGa7Mvb9PaJ8Uh0mNehzdtJ6dlbmxiMcW5GalxUU8uXfjwt9HL8kyX72jrBqYyAO+c82XfzuoXX0roSIz7klUKkVxOXfXjR4ZMmHiCE8PSR0bE15BRmK09N7zCj+bhwrgCElJTb108eK1gGtxz5+/p41kXf/5h/+6+vXXWKZ4snXK1OzZM0b1aO5kKeCU+Zkvk+OiZQFR8ooNFvTulxOXe3/TuNFhE6eM69+uqYOlUJ2V/PRJdLaAIgWaJb2/O6K89zKTeHHZV6EXvMeM6t+5VcNaNiYCZV5a4tOI4Jvnb2SwhJD8gC2zt5vO8/YQxScpSz9E7/QxQigq6cHVR7VbN6ptKaIUWQmR988d2r71dCqnzyByxbGp/vNGfR3hO8W7W3MnC5IV9ygwxtSzV2OWK0day3IsRdP29vYCgUAkEonEIgFfYGxswuPRJiYmpbyxKK3Jz2dYNjc3l2FYuTxfqVQqC3MdlsnNzVUzjNHkUv0AACAASURBVDxfrlRpLMwpynWKFuYXrVnefX/fxCKRmlEL+AJTM7MhQ4Z6e3s/ffr09OkzV65cUSjKOZcK323c9M42JOXypmXr/g2OzVQJreu4tu3aQn71BWJ1AAAAAHg/NEMakVjc9FVIM+3rqQL+m5AmOPjBixcphi4WAAA+KMrCprr1o2zQoMGa1aufxz9fuXJVXl75v/1cXXTu0nnhggWEkK1+O4KC7hq6nErFd1v038GR0Uu7TT/xUQyf7+HR1nfGdELI/9atwxwz787a2jo9Pb3M1RYuWFA4x8zosT7vvyiA16gaw3+5vqrNnaW9xv+dXuZH1O8HfyWEBF4P/N+6daWsJnzF1MxUKBAKRUKhQCgUioQigVAoLFxiamIqEhW9KBSKhAKBUCQ0NTUVFmdqalrKhpQqlVKhUL6Sm5tb9C+FUqlSKZUKhVKZm5OrVCoLQx2lqvBVVW5eTtGPr96dl5fHce/0Gf2t7wzP3r1fTTpECCEcyxKKUhQUXA24dvr06adPnxJCzpw5TQgJCrq71W+HzrbMv9gX+EO3xH3DBmx+qCXSF3Ra4X9wRI3X4wCyKX+N6bn6popQNt0Xbp7Zr4ljTXMRJ0+Lvnd+72a/Y+F5HCGEUDadJi8e103SoI6DrbmxgBRkxEkv/rlpy1/BGW92nLZuOWL6tFGe7vVtBPlJYbduZzUd1s3h4hw333MFpIz2LdyGzxzXp61rw7r2lkaUPC32/MqxK86mc6W2yWv09V//+db6d0qP+ddVlMXgfYFre5QYXE95e0mvSX+kcJRJwwFTpk/8vL2LnUj+Ijzw2O4NvwTEq0o7Kfg/GgAAAMC7E4lEDRo2kDSVuLu7uTZzFfAF6enpshBZsFQqlUqTk5MNXSAAALx31a3HTKNGjVavXvXs2bMVK1YWFLw9mA98vCjzGnZMRprK2LHTxDnDHTMurL9c9iNPqI70SWUAPhjarnW/liQy5GliWlYB36p+60Hff91OyEYHP9ar746eCqMOUhnXP03TxsbGAj5fJBa/6ppjzOfzjE1MBDyBSCzSWMg3MTHm8/likVgoFpmamRoZG/H5fBNjEz6fL3719lK2lZebq2YZeb68sP6CArlazeTm5jEsI8/PL8xx5PICtVqdl5fHsEx+fr5SoVIqFYVrmpmaUnSxPkwUTRNCxEZGvT17ffZZ38IONHrttjwpPoOh6/Qa0++fiNOx5RgDTm3ZoFVjx8Jgw7Rm0+5jf2xmp/ri+1NpHCG0dQvPgd0kr3+XMrFt0OmrxW6NjYaO2R+hJoQQyrzjkoPbxjcqmrBK5OTW34kQQhT6tW/XwWtM/9ftm9WwEyhyubLb1JNx8xn79sxyNyuMo8R1Wg78dqubg+8XSwIy8L9XAAAAgPdJoVDIQmSyENnRo0c1Q5qp06YKBW9CGlmoLC42ztDFAgDAe1GtghlXV9cVK5aHyGQ/rFmrVJX6hU/46NC1hm35b1kbASGEcEzaleU/Xc3BgyMAMDiR21frtvY31ZybhGMST+88HKFzCizDYlm20kctK+yIU9hRR7NDj6mZicaPwsIOPWamZkKBwNraukQ/HqFQKBaL+fw3v5mkpKTQlPZZX/g8PkeIs3PdGTOKpisqfcw3orq/3y+w38quwzYe6zTi9IGDh49cCksvMRMQE7F1qNdPYcVOHJdz48exgxdHP0/NVQksnDpM+sFvYo9h3a1OH3313QAu++wirwVnknMZo1puQ5ZvmuvZctSoVoeWB6kI4TebuGBsQ2G29NDyNb/6h2Xw7SQ9Rs5aMqGtWTnaz/FfMWLhyfhMxqimvVGWit/smzLaLFZ/1vEJzY+/OmqOX6zb9+MA64jD+8+n0Y2nLJ3hZpwSsHXR+v+7GVtg0bTf3PVLhw2eMeq3QL/IKnr1AgAAAFQ/miENj8erV6+em5ubu7vb1KlThEJhYUgTEiqThciio6PfsUs6AABUHdUnmGnevPmKFcsfPXr4w9p1KjVSmWqHshBxmflqK5IRc+fU3v9tPfMcT40AwPAoYWb4laD6bo2c7C1ERJGVFPP4+skDfn/cSfmUZi5RKpWV1ZWtMKcxNjbi0fxZs2fa2dnpWpMihGVZilc0/JixkZFIJFQodE3SxMT+PWvoi/FzZ43p13rY/DZDfRPvHdu/feufd1+odbyjaDOUZfOv5i1uL3GuZS3IS0pjeYRfs5YtTdKL/i/EMTmpKdkKhpDchHuHfzjUr8dciYtLDTookeU16du7Lq24v2XOhhPxLCGEJEhP/X7hq3Ft3cvRvjojIf5lvooQVWJsNuG5lt2mVnSN3st3bfjc9umf3/msv5FGScZ97sLPvrh27i9XsjhCSMrjYyu3du67pVdHD1u/yBelNwYAAAAA7wPDMFFRUVFRUZohjaurZMzo0cbGxhkZGSFPQioc0nwx+Iu7d4ISk5LeU/EAAFAu1SSYadumzaIli2/durVp4yaGqeYP7Bs3bvwi+UVWdpahC/mwmNBdo7vvMnQVAADFcVlBe33H7jV0GdXH67ltCCH84uOYvaZWq/l8fn5eXlDQ3Vt3bhdOqJaalqY7lSlqO/7aLzOvHfhf6/5jfcaN7Nlm5OJ9vTuvGfnNkWhd2Qxl3nXJb3tHOAuK+u2InOoQQhia1vXrE5MY/SyPczU1NaEIIQKHurVpNv7BvSQdMV2529ejTe0bMms702+Lt1PqfwsnrrmWyhJiVKe+I00b9d0W1Hdb8V1wcKxFCIIZAAAAAAPTCGmIZkgzetQoExOTjMzMyIiIkBCZVCrVM6T5avhwn/HjDxw4eOLECZb9lL5HBgBQJRX7y7/w0cZHx8bGuomLS2pKKp/mzZs7V9dqsrDQE8dPfMjC3pNZs2Y6OzsXFBS8SH4RGxeXmJiQmJSUlJCYmJiYmfWJpTUAAFB9icTiNz9whGEYHp+Xlpp249bNoDtBjx8/LvoqRvl+eVEk3z+24f6J3ZKhq7YsGdh15rc9/5t1Qfv8LJR17/FDnHgZd/yWbvjjdnSqnG/ba/HxLYNKaZ1TKpUcRdEUIYQUducp+qFy2i+zTW34zsPWbZ8iUd/dMnXxmfjCr6/o/NudEhmJytE2AAAAALx/miENTdP169eXuEpcm0qGe3v5+IzPys4KDwsvDGliYmK0hi61a9c2t7AghEyY4NO9e/dNmzdh9hoAAMMqFsx07tLZUHW8u5r2NWva1yx9nROkOgQzkVGRderUEYvFznWd6zg7sQzD49EURRNCFApFSkrKs2fPEhITzUy1jjYPAADwcRCLxBwhHMdRhERGRAbeCLx9+05CQkJltM1myY5tOTKs/1xJw4a1eBeeqtVqjhgbGxfLO2hbe3shyfc/tO1imJIQQlQvU7O1ZzhaKeOi41m6bsfuDbY/jtAyyGpF2i+rzbdQZm1n7lrazTLu6LRZ+0PkrxarEp4lsKzFiUl9ll7J13+X3mjfvr2FmTnDsooChYpRKRUqpVKhVjMFBXKO5fLy8wgheXl5HMfJ5fJq35sZAAAA4INhWbYwpDl54qRmSOPt7eXjMz4/Pz8iIiI4WFoipGnevDnLsjRN0zRdv3697X5+//777++//6HCDM0AAAZSTYYy+6RERUb16N6j8N80RdEa8ySLRKI6deo41q7NsOzr+ZNr13YwQJUAAADvRqVS3g0KunnrVtDtoHcdwFPoPmVFr4KL564GRyVkKojYyqlZz+mDG/M4dWpKOku4lORUjufq6e15OMI/Tm1et1kteXBI0suUFBVp3G7oqNbhfz9MymX5pqZiPiH6ZjNM+KnToVNmuX7jt75g9fZ/7j7LVAksbK1epz9sBdovq80SKOvuS9ePa8I92TJ73eWXnGY75/1jpk4buGLDM3rHmbtRqbmMwKJWg5a1cgPvxpY+7U4hdze3Nq1bUxRlZGTE42kfdE6TQqFQqVQqlUqhUDAMI5fLOY7LyyvMb/I5lpUXFDCMWqlQKlVKlUpVUKDgODYvL58UBjyEk+fLGYZRKhRKlUqlVikKFAzLyvPzOULycnP1KBkAAACguikR0jjWcZQ0lbi7uXl5DfPxGS+Xy8PDw4ODpbJQWcuWLV93maZpmhAydMiQdu3bb9q4KTIy0qA7AQDwidISzAQF3d3qt+PDl/L+/H7wV0OXUJmio6JLfwJC0TThuMIh+AkhCQmJH6o0AACASuM7c1ZlfYOPL+k1crCP8zCf4os5eeTBPefTOcLFBVyW+TZvMXTj5aGEEEJU0h/6j9nz/PKRSzO6DOi57HDPZW/exUTou1km4sCKjR33LPDou2hv30UaLxRGL9zLCrRfRpslCFr16+/Ao6jm3/17/7s3bST+Nq7fqn1r9/fYOdlz9l7P2a9fUQVv8Bx5IFaPIcd37toVeD1Qc4lQk0goFAgJIYX/EApFQpFAKBQKBcJXS4QiUdFPhBChUGQkFgtFQq0IIaampmXXRIhSpVIqFEqtFMqiFZQKhVJZ9B9l4b+USpVSqVQSQgq7/rxZ+KqxwmBJnxoAAAAADIJl2bjYuLjYuHPnztE0Xa9uvWYtmrVo1nzYsGE+5uPVKlWJR0k0j+dQq9bmzZv+/fffP37/Q4lfdQAAPiz0mPmY0DTt4ODgULs2x3EUpf3LsSzL0DRP+iD4zr2gb76e/oErBAAAqCyV+BycjTm5frNwYLe2LZs41TQTcorsF3Fhdy8d2/Prf7IcjhDCRB7wnWu+/NtB7epbCRWZcU+iUimKcOlnl0yekzxzYr/WjWqa8NQFOVkZqUlxt6Oyyp5ctZA8dM/kL8PHTJk8qEuLurYmPFV+ZurzmHBpQIyKkAq2X3qbeuNy7q4bPTJkwsQRnh6SOjYmvIKMxGjpvefK8jSiqTDAqOi79fJ29qMZ/BStoC37EQpFQoFAKBKampmWbKewMZFIIBCUvY8qlVKheL2zb2c/msEPIeTt7Ecz+CGEaGY/GPMNAADg/9m777gmzjcA4O/dZUHCCgiCLCVRCIq4cC/c4sRRF+5d66ij7tXaWlut26pULc6fC/dEFESruHCwAwqC7BUg++5+f4AISEhAlvh8/+jHJnfvPXe5C5f3ufd5QVWhKComNiYmNubihYs4jrdp22b9unWfL1aQqhnu6dm1W7dtW/968+Z1jUcKAADfLkjM1Gk4jltbWwscHAQCgYPAwcHBQU9PT61Wy6Qyfa5+qYVpGtE0lZCQuGv37rDQsK96xiAAAACgClGSqOvev173LmcRZdzNrVNvbi31Ki2NvrBl3oUtZa5CRu8bJdxX4iXVg/VuTutLvKRIDPReF6hh0xVtX4c2S6yl9FvW3HFZ2dtGiM6NurTjp0s7NL1f51R37qdogI6m3E/xxE/BYqVyPzwel8UyKd0Oi8VisfT19QvKhmjZx88G/aDieaCSuZ+CBVFRguez3E/xxA/6OOtP9R1AAAAAoBLMzMwcnRxrO4p6rqCOmaYHfAkcb9Cgwe+//xYS8tLvjh+MEgagXooIj0hPT6/tKOA7HxWvPAGJmbqloCSoQCAQCAR2trYCgYDH45EkmZiYKBbHPHj4UCwWi6PFP8yb171H9+KjUEmSVCqVPkePXrl8pWhuNwAAAAAAoKNaGfSDtBV8Qwhpyv2UKvjG5XI1dbiU2E0NBd8Kj0BZuR9dC77JFSo1dOUAAACoGEcnxxXLl9d2FN86HMMQQq6uLV1dW9Z2LACAavHb5s2lilHXCvjO97g/qOjfkJipZQRBNLJuVJCJEQoEDg4ObDZbrVZ/+PBBLI45fuKEWCwWR0WXqvUZHSPu3r1bwb9JisQxPDAw8OAB7y+dGxkAAAAAAFSbul/wDSGkKffD5rCZjCor+IYQ0pT7gYJvAAAAAAAAgHoPEjM1rfxMzP2gILFYHB0ZXf7zhmKxmGAwaEQjGkVFRu/Zs+ft27c1tgsAAAAAAKBuqiMF39CnkUAVKPimp6dXal7isvex2KAf9HkGqFIF34rakUqlMPocAABqxbUbN8XimNqOoh4yNDSc5DVeoVTKZPK83Nz8/Px8qTRfmi+TyfJypTKZNF8qVSgUtR0mAKBaCAQOA/v3q+0oynA/2zBezq7tKGpOFyOJnV7pb1pIzFQ7BoNh1ciqKBMjEAhYLJZcLo+NjY2Pjy/MxERFV6iIZ2xMLEVRORLJwQMHAwMDoVw4AAAAAACoATU86AcVJXjqS8E3pUKhhNr9AABQFrE4Jjj4SW1HUQ/hOO7vfxceOwAA1Cnxcvbr3NITqNdjLtz8z1+ExEzVK52JEQpZTKZMJnv79m20WHz9xg2xWJzwPuFL/ijK5fIDBw7cuu2nkMurMHIAAAAAAABqVy0WfEMIacr9VG3BN6Q591Ohgm+o+Eigj43J5XK1Wl2dxw8AAMDXBFIyAABQN0FipgowmUxLK8uiTIywqZDJYEql0nfv3lVVJuZzly9fqcLWAAAAgDplxfLlFg3NExM/pKampaampqampqWmJqekVHd3LQDgW1CLuZ8KFXwr3Q6LxWKxOBwOg6H9R1yFCr4hhMqf7AcKvgEAAAAAAFC1IDFTGRwOp4lDE4FAIBQIbW1t7OzsmExmfn5+XFxc9WViKm1Av74d3NrVdhTfNBMTk9oO4Zs2f97c2g4BAFBhqWlpnbt0FggEpJrCcQz/OO1EXn5+WmpqUlJySkpyWmpackpK7cYJAABlqu7cT4UKvhUsX85kP5/aYbFYLJa+vj6O49r38bNBP6h4BkhzwTdUMNCnrNwPFHwDAAAAAADfCEjM6ERPT69xk8YFmRiBwMHa2hrH8fy8vLj4+NCwsIuXLonF4vfx7+vmXC9CoaC2QwCgNrlBYhKAr1BERDiGDUcIYzBLdA7yuFxe48aN7e1VpBpDWNFj42amZrUQJQAA1JK6XPCtcAHNk/2w2IUV4bTvZrkF3woX0DzZTzkF3xQKRYXm+AQAAAAAAKBqQWKmbPr6+vaN7UtlYvLy8uLj41+EhJw5e7YuZ2IAAACAr11kZFR5b2MYk8Gki41MTc9Ir/aYAADgW1LrBd8KF9A82Q+LzeIZ8Eq3U5j6YTN1yP0UJH7Q56XeKlXwDRVlgKDgGwAAAAAA0AYSM4W4XK6dvV1RJsbGxgbDsMzMTLFYHBwcXJCJiY+Lr+0wKyDofpDH/UG1HQUAtea3zZvR5toOAgBQWZmZmTkSiZGhoaYFKIpKz8jYvWvXxo0bazIwAAAAVaVmCr4V/aOeFXxDCOXn58OTggAAAAAAX6lvNzHD5fHs7GzLzMTcvx8kFseIxdGZmZm1HSYAAADwreByuUKhUCQSCYUCkUhEEARF0TiOlVpMrVbjOH7+/Pnjx47DJAQAAAA0qZVBP6hoyp8vK/iGEOLxeDrt5pcVfEMIacr9QME3AAAAAIDq8w0lZhgMhshZVJSJsbW1RQgVz8RER0VmZWfXdpgAAADAt4LBYDRp0sSxWbOmjk0dHR0tG1rSNJ2YmBgZEenzr4+lleXgIUNwRBQtX/BccGRk1K6du94nvK+9wAEAAACEvoaCbwghTbmfqi34hhAqf7KfMgu+yWQykiSr9QACAAAAANRNVZCY4XRYcHTNoMbmfAMuiyDludmp8ZFvgoNun/O9G5Gj4z0W4Txl19YJvMtzp+yJrK7bsg4dO3To2CEpOUkcLb5zx18sFsfExOTm5lbT5gAAAADwOT6fLxAIBQIHZ2eRSCRisVgymezt27cPgh6EhoZHRIZLciQFS4qcRcOHDy9akSRJab70gPdB/zv+tRQ7AAAAUNO+roJvqGQmSU9PjyAILRFUquAbKkrwQMG3um3v3j3+/v7+/nehHgkAAABQShUkZogGghYCK3bh/+gbm9sbm9u7dPWYMvuVz+qlv/olqrW3gRvbiYSWWazS1Uqq0pvXbzb+8kt+Xl41bgMAAAAAJTEYDHt7e5GzSCgQikRODRs2pCgqISFBLI45fORIWGhYbGxsmdMji6PFJEUROE6RJIbj16/f8PHxyc/Pr/ldAAAAAOqrulnwDSFU/mQ/RXkgLpeLYdr7ETQVfCs8Apon+9Fe8E2uUKmh4FvZcBy3s7ObMmXKpEmTnj17fuPG9adPn6nVOnQRAQAAAN+Aqiplpg7bM3bknnA5YnFNGgpcOnmMn+zVueXkv/ZjM8du/C+3Ljygkp2dDVkZAAAAoAbw+XyRs0gkEgkFAmFTIZPBzM/Pj46OvnPHXyyOCQsLzdPhL7JSqYyPi2vcuHFcfPyOHTujo6NrIHIAAAAAVK1aLPiGCuby0TzZj/aCbxw2k6G94BvSNtnPlxR8k8vlX2kyg8liFfwDx/HWrVu1a9dWKpMFBgRcvnLl3dt3tRoaAAAAUPuqbI4ZSqVQkjSNFHnpcSH+cSF3r91ZeujQ1GYTlnud9twbTiLMtMeKbQsGNLO2MGTTsvSYpze9t+32jcz/lLMhms6/+Gp+QWupp7zcf36o0mEtAAAAANQ2DofTxKGJQCBwdhI1d2lhbGREkmRiYmJYWNj1GzfEYvH7+PeVKCQSHPzk5s2bV69eK3NIDQAAAAAAqgOT/RQkftCnkUDVW/ANfT7lT6UKvhW1I5VKq+Nei81iIoRohDCECvZRX0+vd+/e/fv3f/v27ZUrVwMCAmQyWZVvF9Qsws7zl12zmz1aPfrX4K8yg/itwIy6rfxrYta+n/Y9yaiTnaqYSY/Ve5f0jdvee7mforaDqds4wmErNrhHrPrx5Fu45r4YxmbP68Hrlps3/qGieu8kylJliZnS6JxHu34/0897onCAh+P+8FASqY0dWje1Lnhggmfh1GPiH83NVUOXXE4v9/ugcmsBAAAAoJo1bNhQ5CwSCATOIlGTJk1wHM/MzBSLxVevXA0LCwsLC/vyLhIfH58qCRUAAAAA4EtUa+6HyWCyOWwCJ/T09TCEcXlc9LFKG4fDZjCZLCaTxWIzGASHo4fjGJf7aQE9PT0CJ1gcNs+Ax2Qy2ezS7ejr6+M4rjWGgnE5BbupVqvlcjlFUVKpFCGUl5ePaFomk5EUWVC6TalUKRQKkqRkMimiUZ40DyEkzc+nKFoul6nVZEE7BgYGCKFSleYYDAZCqLF94++/nztn7pzH/z2KT3hf1UcU1CgDG5HIxiBEh6KCoPZgBp0X/Dy+LXkSr/muZx1hHCtRi8YN0hgYQgixW8/38Z5ocHul1/JbdSqRVBcCU6kNbJr36ffzd0GTjr+vrqnavxkYQQhNGXxZwVcY1rylyWZHPOi/zN/jqRr4fKstMYMQkr289zhngmcjJyEXhUro3Ad/TBy2KuZ9Wp6KaWTbcfqvu6f1HNHD5MrZzML9JKN2eo78K6LEGaV9LQAAAADUCH19/aZNm4pEIqFQ4OjoZGhooFar3717FxoWdvHSpdDQ0JTklNqKzc2t3TGfw7W1dQAAAACASlOpVao8FUIoR5JTfVupdMG3ggX09PQKRgCVgV1YEa4CMIRjOI5Qh44dOhOdC14rSDiVgzt417M/Oor/mTvhz+DsEl1CzG6/+B/2lBwcM2Tzq8r1UhLOU3ZtncC7PHfKnsg63c/JHbzr2Z/dord7DtsX8zFQht2Q347+2t8k/NCMGdsfZdep3jKOXc8J30/06Nrc1kwfU+SkvY0ICbp+4sBZcdvfb/w9RO/BOo/J/0spPVyL6bryqs90y5CfB0w+nNRk3plzPwoit40avzvy8/mcuB1Xnz06wSJgefdpF3JLb7vDgqNrBjU25xtwWQStlGanxUe/Crp59t9zwUl1NjtRTRjCST8Ob5Rx7ftd4X32Pt/qztawnOrJJo8xPol1oVgBhmEYhuMVz/eV+NxJeW52anzkm+Cg2+d870bkVMHVXenAqg759uTmA8P+t3Du3F6XV96S1KlLviqxG/K2tePY6OE8JkbQtExJJeWonr+X+4oVCdU5VAhDSPvjDAgJnYxXN8P8ArKOZlV+W9WZmEHqzMwcGjPQ5+njSEJhmHGLMctWdRDZWfKZ+UnpFIEYFpZmOMos77Ko3FoAAAAA+GI4jlvbWBeMiRGJRDY2NhiGZWZmhoWGnTx1UiwWR0dGw4S3AAAAAAB1X3UXfOPxeAghfX09HCc4HDaDwbSxsVmyZHE5q9A0jYoNs+BwONo3g+k5T922M2nS9GMxVbozuLGdSGiZxfr6Rn0Qlv02HNnU3zT66MyZdS0rQ9iN2nZ2QzczovCwMkytm3duJGSE+Jx7+eBaQObgYW6D+lidOZZQMg/Abu0x0BpXPLl+4wOFCDOLBjjGdpq+aNDZub7JJZckhGOXjbIhMNLEjE+g3FL9hEQDQQuB1ccUBMfAzMbZzMa544Bxnt4zpu18XH+7sz/H7TLRywkL2+ntl00Pre1gdKN4tmN0qx2VWbPE507oG5vbG5vbu3T1mDL7lc/qpb/6JX5Zl37lA6tK6uhjB/ymbO83ffhev3/f14VEWnUg9BiORkThZGUYxuUQAg4hsOAMaSr72U8SKK2ObdJvXmZ6vNRlSczIgGnPpVhftr1qTcww+HwjjKak+VIaM+y2+oj3WDtm4bcx29YGIUTieLkBVG4tAAAAAFQWn88XCIQCgYOzs8jJyYnNZsvl8tjY2BchIcdPnHj98nW1PstZCUH3g2o7hG9RelpabYcAAAAAgDokLy+v6L8FNM0vWDCfDYZh797FPXz4MF+aP3PGDIRQRkaGDtuhSdqwy0/bV8RO2Pgwp9I96zjbwNSYo87NypJ+1VM04A26rziyZUjD+DPfz/zzQVYdyzQwXCfN7WKKUv23rt18/kVctorFt3Fu181Fdi+FQtTja7dSh4xt5eFhe3L/u+IpFU77Qb0tcfn9K37JFEL6ZhZGmDI7h+g6c0br6z8/lX9aEDPuO3tSC0V2NsuQzzfBUFxZQajDdo8ZsTdCQRN6Rg0FrfvOWPK9R4upG6fcHrgjrBYf+NZ0BlbLmYkZuXv2NlM+230hlkTU+Tmtzxe+wWi97PLpKQbnZvb84776WQAAIABJREFU6X49e9hOHbZn7Mg94XLE4po0FLh08hg/2atzy8l/7cdmjt34X24du1Qqgc4OOHctpd8Yz0HCY/vqyDA/gUDA55u+ePFcparK0yn6Vcac12olQmwWYWPG9mzJG8jXW+giD36klGtfu66rzgyHXsse7Y1wKi4iOh/xh04ebktkPd69ZsvxRzFpMoZZr1UXtg8pvwGM37sSawEAAABAdwwGw97eXuQsEgqEAoGDra0tQig5OTksLPzIv/+GhYbFxsZWx3ywVeW3zZtrOwQAAAAAAFBaQRm0IhRJYThG03R0VNS9wMCg+0GZmZkIoS5du1SkVfXLo3uS+/3gtWXDq9GLfT983h/JaLvy2qmJhtcW9Jp/M7/wNcLpx4v/+97sysyeK+/qtZ2xduWs3k1NmBhNq3Ljb2+c/NO5DwWLNZ1/8dX8gmhTT3m5//xQhRm5jl4wqW87Z4F9Q2M9TJYed3PDxPXXM2mkZ9936pwZQzqJrLhUVtwz/7N795wKTvsUD8YVeMycO21QB0dztiwlMsh3/5YDAQkqhBBm2nnGqkndRQ42VmaG+kxS8iH83ond+182GjZ+aN/2jtbGRH7im1v//rn5xOtyB8Bg/M5LD2//zj7l0sLpm/zTPt2ul7PpsnZnQ7DT9GLxIHlWfIjfya3bT70olunR3KZm+tZ2pgQVf2XnP0HRJEIIKVNjHl+NeVzwruzpxVtJ300UDfVw8N4T9enAcTsO62WG5d294JdOI4TzzUxxKvXavsutl3jNGfzPjDMfPu4oQzhmbl/2f1v35i1c0snMWFPpIUqtVJE0jdTSrIRXdw4tzmng4jOxcbvWFnhYQVNado1j3dNr1rShXVxsTfWQPCctMTbqze3DW72Ds+myj+f665l0OW3i/DLPQErT6wihck82zafopw/CrU9Hnvr1Xf/PisaVRLRYdMF3VsPbS9znXCmoC4c3HOcdsEZ4YVbvnwIVH5fx9Z1pcn52n2UBcq1Hr/x3cX7LsXNnj+/TqokpU5oU8d+jHItPnyIhnHPq2nzL88WSRhU6CSmVQknSNFLkpceF+MeF3L12Z+mhQ1ObTVjuddpzbzhZToOMtiuunppkdHNRn++vf6yPh5l+533rN7dX6/vO+G/YyeKBYaY9VmxbMKCZtYUhm5alxzy96b1tt29kPl2wWomLveyLC+nZ9p48e/qQzs2tDTFZVmJk4N7VP1+II7XssjzE70H2uGE9e9kdiIytE5kZWzvbxT/+KJXK7t8PvHv3XmhoaJV0I9AUUtGIRkiuIKMTpX/kYUIPnqAByxZTJpnqTXHitOQzGunjHERn5cq3+0kC5AhjMtyduaPtWQ56mFymfhqT/3eoomi8Hc5hDmnBHWrDsuUgWb76eQplVmyspH1z/uGWxM276Zs/fPyMGEQXR953TVhNuRhO0slZiqOPJLcKTg2MMdnDYjJCCCFKJlvkK3lewT2utsQMZtRh3rJRjXB11M2r4SQuaNiQhaS3j+7yi1AihJAqI02i+LQ0rVaraaSvr19i1ChuVv5aAAAAAKiMgmExzs5OIpFIIBSymMz8/Pzo6Oj794PE4pjw8LDc3NI1mgEAAAAAANAdg8lECJEUReC4Uql88uRJUNCDp0+fSqVfVICGTLy+YrFB40NTNm6dGjnlYFjpR6bVb+7/l+nl2a6jC+vmfwXlznCL1m72uCLo8XNlw1E7di3rboip8zNS8jCeqbE5U5FDIURo2Bpu3nGk10DRx74zgwbmTEUejTiiWQe8l7kZFXYjWzTtPnZFp24tF09YcbkgV6TfYt4/Bxe2MihYgGPTcvAPO12t5g9dHZBF43yXPoO7F7XJNLFpNfynf4YX2yrLru13q/fx80fMvqCpKx03ajf/0M4JzbJuLp2+7npSsW7Zcjdd1u5gJeNBXDOHzmNWuTbV8/Q6FKXW2qaGI4cQkiUlZJG4TS+vAeeirsTJSr+tfHbhSsz42U09Bjrvj3pVOD4EM+o2qKcJyrpy8U4WjRDCjPkmOJ2d+tjnUOC4X6dMbXXpl2cKhBDCDHvNHOeYfnmyb+SA6TSHb8rFkFKHcRCUmqQQQjiOaz9cCHEcZxzwXu5m8nFCEa6pdVNT6yas54cOBWeTmk6PctrEGo7aXNYZiGt4HSFtJ5uGGIphNGvVkkslhLxM1tJlTEY+Ck6bOaplK0fmlScqhBDSa9VWxMT1XVs1IQLDSYQQbubqaoPLAh+8UGg/euW+ixl2Wu2za7KQU3Bo2bauA20RQkhjr2/lTsIidM6jXb+f6ec9UTjAw3F/eChZToOvA/9LnziiXacW7OsPC+PhtuniwibDg+6n0qV70tXGDq2bWhekoXkWTj0m/tHcXDV0yeWCxKLWi4vtOGP/P8vbf8wssiwErjY8GaXDLitePQ9VjXBr42qAYrN1OAQ1gaIofX29Xr179evXL0ci8ff3D7gXEB0dXZXbwFBR8sC0od5wO+bHw4vx9ZFShRCDOcndZEoDrOC4sXnMXi2NRdzsGY8UOQhhLNa83sYjjQvLaLIMmD0NEEJIY21MgjGmp8kci4/fGARmZ0boV914Nl0ms9GtIQaTwBAiWFwzu5Y9x6zyPn14mhNH9e7EZp9wElEZqakqpNfec3wbKx4DQziTx+MUO5Xp1OQ0mrDsM6pPYx6D4PAd2jpbEVrXAgAAAIBO2ByOyFk0ZOiQFcuXHzt+7OhRn9WrV7q5uSUnp+z/e/+cuXO/+27MqlWrT5w4ERz8GLIyAAAAAADgCzGZTIlEcsfPb8P6DaNHfffrr78FBgZ+YVYGIYQQnfds98K/nlGuc/9a1M7gs1lh5E/9A7JQg67dWjILX+G1atecoQ59/EzCa9+3vQH1Zv/wjh3bdnNv08atw/A/g4oiIqN2DnVp3My5cTNnh64/Pyx6Ep/Ovb1uUNtWrgKXjl1H7XisIgReaxa1M5SHn1022t25eatWfWf85p+EWQ1Yv6yPCYYQIppOXDPPVT81YOfUgZ0dndu0H7n6bCxpPWzeeMHHDBAtub6ib0sXF0GLLh6rriWQNJX9ZPe8kR3buDZt3WfC3mcSZNzd09287E47jC0cv3v39BbKR7/MWnUhvngfoS6bLrU7JeJxcG7fZfzm28kUt+X48a2ZurZZJtWzQ7uD0jG7EX/6+h/fOLufI79kjx4ZcfncayVuP2Co68fBVZhJryFdjOiUa+cfFA7ZMOYbY3RujiT1xpGziY1GTulb8Gw7YTt8eh/e6+PHHuVJcnJp3IRvWm4HJ0awuHxr565jf1k7ypYg4589T6a07hrhMGHtYjdjZeyVdV79XVu4CFq077I2QFoqDfDZ6VFOm5hB2WegptcR0nqyaf5MP+46z75xQ4J8Fxuv9VF+5av/gvPwBm3aNin4YJnOHVrrYwhv3LZ14VgWbqsOzkzVm0eP82itR6/cdxnNpy2fKGBJQo4uHOnu3LxVS/fxC72fpGsMsbInYXGyl/ce51B4Iycht/wGFS8CH+YgfseuLT6esXqtu3bgUTFBQfGfDU2hcx/8MXFYx3ZtBE4uTh0GTfV+JTftOaLHpw+n/Iur8fg1P7oZyaMurPYa0NrF1amde/+Jv99M13p4EUJ07tt3KRTTvomNrkegBmAYQohBMBBCRoaGQwYN3r79L5+jPlOmTLax/qI4MQzjcginRvpLO3EFOMpOV8UXXol00OOMIadSe5xMG309/yWJmjgaTGyAZSTmLbuc1utE6rDrkus5dMMm3KHGCCHUTGTgaYzlpUs3Xk/reyJ1wIXMjaHKTM25PZumhtMtcEW2bOvt9EEnU3ufTpvklxtY9EAArT5yNaXrsZSux1K6n6vwcBlUdYkZhmjeuciI0LdhL948uHbh7zXTuzRi5rz+98dZPz+U0AjRGf6n76RjFu5rT9x+HfrmbXjIC+/vGn26dsj4AP8wBW7n+af/s5fil/f9jqz0aIRrWwsAAAAAGjVs2NC9l/vMWTP/+GPL/06d/GPLllEjR7LYrGtXr61atXrUqNFz5szdunXrjRs34uPiNRUBBwAAAAAAoBLevHkzfvyEHTt2Bj95olJX7QwWyqijKzf65wq8fllVvAO0gCz42r1MzLJnn+YFvaqsFm6ueqQ4MCiJomkaIayBo5ujGQdDiFakvU0ot1wYQgghWp2VmJAhVZEKyYe4lHxcOGSoM0v1etfijWdepkhVyuy4hweWrDudRJv0GNLLBENEs8GDHBkSv01LD9yNyVao5amvfTfsvJtHCDu5mRV2w9FkblqqREGSyqww373HQ0mMkR72IDw5T6XK//DgwD+3c2jCxt627E47QugxqqMxynx552FCyQEGOm265O7QJeKh1HmJT0/8evSNmjB1dGyA69hm2ci4Mws9Z++8FJZv2mbETzvPBt0+ssmrnUVReoaMv3T+qQy3GuTZQR8hhBBu2W9EJy4Vd+3sk8KOT7aRkT5O5eXmU4oQn2MvWD0mjhUSCHHcJo5zld7xPvuORNLcPBoz5ht/lqJDCCHEaL7wkjgyNDbsxZv/bl7xXv2dM1cWfmzdP6FqrbtGNBno4cwiI/5etNon+H2OkiSVeWkZeaXPl9KnR7ltajoDNb1OaDvZyvlMC+GmZnyckmZklM4olUH69O4TKeHQvr0FjhAihO07mElCQxPx5u3dDDCEELtlx3ZcMjTwYRql9ehpebdfb3tc8Wz74i0XX6dIVUpJYsjlY7fEmipyVf4kLE6dmZlDY7g+T19LeNLH1wOzMauuvUQFfdDs1u6dTeiYG2UGiGHGLcb8eujcg8dPXt31Wd/PikAMC8tiUZV3cTUZNLg5W/Vq5/y1x4PjsxQquSQl6kWU9sNb0HBmeiaNm5rxdT0ANY5gEAghUz5/2LBhf+/fd/DggXHjxhkbG1eokaaupgETLALHm98YaXagp8EgPkbmyne9UhR+R9B0Tj6ZpaZJkkrJJaUYs5c9k1DK9zzI/y+HUlJ0RoZsxyuFFGe0sSBwjNnNhoGTykNBubczKBlF5+Wp7kQq4jRdGxijV2Mmi1QdCZRcSCFzSFqhpN6mqctJ5FRUFYw/IdPEr2OcGpubGOqzCUqel5MeH/Xm6YNbZ87eCcv+eMbSmddXz1icvGDagDZCCy6hlufmZKUlxT8SF87VRkb/O3+p4bofhrRvYsJSZMe/EadhmNa1AAAAAFBEX1/fvrG9yEnk7CxybOZoaGSoVqvfvXsXGhZ2/cYNsVgcHxdf2zECAAAAAIBvgkJRnaXoyQ/n123oJPpr5IYVAa/X5Jd4T/bfpdtJw0b36+/yx4vnKkLYyY1Px/neiyVp6qHvnfSeHt1X+vgtzop7E/Is4NKxQzei8yvUx8SyE1jj1PvHD4pPWZ///P4L+dj+dgIbHMlsmljjuF6/XcH9dpUM2sraEkfpn+1LUtwHNXJqYGGMIymFEELK5IRUGjPX1ys72aCOOLHlRqPJc3usOuNju2je1rspHyNhVXDTZSM/xLzLp515PC72pW0qEwIPLAj897c2AydOmTTOve24Vf/07vLLuO9Px6gRQlTKjbN3FnXw6DO815b7l7PxJoOHt2OrQ3193xSOAsINjAxwWpmfr0SIen/h2M1Z28Z5dTy803TqkIbvz/zkl00jLD8vn8LtDA3LPlYfFWQ+6Nynh1Yt23P3bUGSovxdY5gJ7Qnq/cN74oqkFcttE8vVcAZqel3ryZapQ0QcFoaUSo2lmoqhcx74v5C7t+newfjo+RybTp0ayx4v25u1bGf/bm30LvgrW3TtwKeifO4lkNr2FGdZlPcus4F9I5xKeP40SbchBlpOQi2T53zE4PONMJqS5ktpVstyGwx5cOVuxuBhvXs7/vkqlGS59OluRkf872o0WbrsIWbYbfUR77F2zMLTj21rgxAicVxTf3vJi4thJ7QnqPfBDz8fiaPDLtNKpYpGLDar9LoaDB82rGvnCs3pVTHmFuaarkIGg4EQsrSyHDN2DI59PFh4BQaYUBQtU1JJOaqXifIL0Yp3mi5KgrDhIZzBWT+as75UeDwcx/FGXETlqV7ll7nyZ3DC3hBRecrn1VZSpAoSM/JHO0YN3KF1MVoafWHLvAtbNL2vjLu5derNrRVcCwAAAPh24ThubWMtEAicRSKRSGRtbY3jeGZmplgsPnf+fFh4mDgqWqmq2ucTAQAAAAAAqH10uv/Pa8+12T9i/aqgP0pOYCJ/6nvh7ejZfQe23fY82LpTV1uU8G9AOIkQSr+60ivvxagBHVq2btWidc/GbXr0dMQ9513NqciWy88AFOYAylyRrccua2VKpVQjjMlkFr2pUqlphGGahgKoUx/t/vnqg9nb/p43cZ+P8dJpay8nqCu1aQ17oFQqaQwrmFmlCtpUJD/z3fLs4n6R58btqwd3W/CD+7WFt/IQQnROwImrSQPHdx0z0PLqWfPRno4M6cMTF4qyEJiBoQFGy6VyGiFESwIOn48bNH7yEprfnRXy28lXSoQQLZPKaYxjYMBEqIzfPeo32z2H7YshEUs4ce/pFe0dHM2R4uMOlb9rOJOBI6RWV2xe9fLbpDWdgRpe99f9Q9NIKVfSiMXSqfeezgi6+1zR2a1nB6OLz7p2a6Z6eirwYVbHrNHdurVkB0h6drVEMZfuvCV12NNy38UIHCGE4bruXpWc2Hote7Q3wqm4iOh8WluD0uBrN5KHjevX32VnaFib/n0tyJdHr8d+dipg/N6Th9sSWY93r9ly/FFMmoxh1mvVhe1DytuV4hcXhuMYQmXGosMuYywWE0NKhS45t69YVEjGjDdqXXM4NNKUZ2cTGPbxS1XnUVaFSaTqGx/yDc3Y0qFjhz/+2BItLvQ+/j2UbQEAAPDVMTExEQqbCgQOQqHAWSTi8nhyuTw2NvZFSMjxEydCX7/Jyq4rU/8BAAAAAABQbejsoG2rTrodGbt0YYo+hiSf3lGHnzkdMv2n3sM77nhv26UZ9v7wzY9jMOTvA45uCziKEGHgOGLjofV9evRz0796S61W00hfX1+HPl5lXEwChdu172xHvC7qqeW27tqKg5TxsQkUUiW+S6Qoo4vT+665W8aUOlVUoJ/Kfrp3znfJv/yzccg2HxZj8nLfeHW1bFqnNglCe/8ilRPmu/30iIFLRQKBJXErmkQIIfmT//lGjPnebezIDlmNPG2xjEunr6UWddZhhoY8jFZIZQWvqN6cOvnEa+Wk7+is60svJBR01SplcprGuAY8DJVXrEsZfWzlqpandnos/nN6yLj9EQqtu8Zo9SGdwm3bulnhb97r/Gi/lsOl6Qy8ll/m6zfeajnZtPcwUxnpmRTe1NRUH0Pa6w9RKf5Xni7p2KFPdweDPi70k00PsqRSvweSEd3d212Q9LanI3ffiiJ12FPCudx3RTEJFG7fqYfDntdROjxHqPWoaoUZdZi3bFQjXB1182o4iZDWj+npmUvvxs3oN7T1IeNBvczlj7Zffv95ig43a9iQhaS3j+7yi1AihJAqI01SgaGCqsR3iRRu69bRhnj9jvz8rfJ3GeOb8TEqI12HYVMIIYR8L1wIuh+ke3QV5d7LfdGiRWV+h6rVagaDkfQh6e7de5JcyZzZsxFCCqqqJlgphiIT8xHFki2/KPlP/dm7GDM+D+GGrA5GWIT2KpaFreE8VmsDFCkp9R6tpmmEML0vS61UwyGoq+Lj4pOTklu1dF20cOG+vXtPnTq5adMvU6ZM6dq1q5WlJYZVQRYaAAAAqHIEQdja2fbv33/x4sX79u09etRn3bo1vXq55+XlHztxYumyZd99N2bp0mUH9h8Iuh8EWRkAAAAAAPCtoHMfbt944r2RlRWnZJ8OFX/lzL18036jRw9zb068u321IC9DNHYf4e7SyJCFYwSToc7NVSCEYQhDdGpyGk1Y9hnVpzGPQXD4Dm2drTRlMcioS5fClMwWP2xbM9LFQp/BMrLrNPOPDaMtseyAy36ZNCIjb96OpcwGr98yrZeooSGLwAmOibVzj3Z2Vf1wtCLm/IqJK66nNOy/+cAqd1OsWjatrU2lSk1jxq17dLDWL3nIWK1m/rpkontzWxMOgWGEHr9xu+FzhzUlaHVaamZRooMUnz/+SEYIxmxf3ZdPx/ueCipWNAjT5+ljSCb/OMaF+nD1qH82RX64eOJu1sfX5FI5jRsY8rT1cFKp1zeuO5PIbjV3wwwnlvZdU4ff9k+i2K0WbF062NmCx2Kb2LXz7OOkZeRJ+W1qOgM1va71ZNOOznv3LoUk7JtomLGo9OJpfleDZbwuU9ePaoee3biXQSPpwxv3cyx6L1ru4UBHXLnxMUFU/p5qe/fylXAVQ/T97t9ndHXgcwic4BiZmWhMjFb8xMYZTAJDiGBxzexa9hyzyvv04WlOHNW7E5t9wkldGlSHnT8fQloOmrJ8Ul9+jv+5G+llHG0qIzVVhfTae45vY8VjYAhn8nicClxrZOSNWzEks+WCXRsntLc34RAEk9ewmWszU1yHCDEDe3sLXPUuNkH3DdYwUk0ihDIyMy9cuDB71pwZM2aeOHEiu1q7LGhVwHs1pcdZ2JnbmU/wCIRjmBGP2cGcYCCEaJXfO5UaZ3p1NxxjxTAmEI5hBno4R3Nr9+LVJMGc0s1wmAVhRCAcxxqYMJtwEEIoQ0pRGNFFwLFhIoLA7cyZFhXPLXxDI2Y+fPiwdds2hBCDwbBqZCUQCAQCgUjkNHTYECaDKZPJ3r59WzSeJuF9AkVVoNQdAAAAUIX4fL5AIHR2dhKJRAKBgMViSaXSd+/eBQcHHz78b0REmERSbVVOAQAAAAAA+ErQucHbNvm6/z3SutTrGX5Hry7oPfr772kiYs/VMBIhhDDT9tM2rOnELLYclXn91pN8RMoC/MPmt3Dx/NPfEyGEkCrk14FeB8uen5GMPrpxezfvpe1G/XFm1B8fN6hKvL7+95uZNEJI/eafTYd67pvR50fvPj8WraZ6saXPuH/jqrirSR1/ec1MC/OTi0du2xY7avrRath0+btDJoRHZNOOjhP33TRf0m7BjaKH+xmiXuOGTbEbMaVka7Qs2ufgzWIpBSrl8tGbCzoNtzCj5U9PHXtZvC4Trs/Vw2iFtKhUHZ1z/ccuDj+WbFCuoDGuIU/7ntA5Qb//fLHr3uFz1o6/OeFwNFn+rsmfHPjzUq8/h7WcuPP8xOIHpNyNlNdmvIYzUGraS9OZqe1k004d9Twk36tfSxcL/PUH7ecAneHne2dZ1yFtHKUBa++k0wih/Ec37mQNGtUKkz3699KngR3lHz0tp03Uv+v/7HRwuVu/ld79VhbbvIbhJhW9phiieeci55XYLzL79b+rF296KKF1a5CMv3QsYNbWPoO6k3HeJ+5Lyqw3luF/+s68rh7ua0+4r/30MhlV9l6UsV+h//yyv9veuc2H/ewz7OfCRvMvz+82/5ZcW4TsFq1FTDLm+cvSQzlqE00jDFOTagbByJFI/P39A+4FREdH12QIUaG5ZxoZj7Hhbbb59KWgSsv1uiVNpNHbCMlBS5PZFpzv3TnfF1tLUz246LDck1bGE0z1FvfRW1z4Gn0nMG19PJ2YqIh2YTo5GJ1wMEIIIUq153LmqQr203xDiZkiarU6Pi4+Pi7e/44/KpmnEQoEAwYMYDGZBWVhIE8DAACgZrA5HAeHJgKBQCgQOjs7W1iYUxSVkJAgFsfcueMfFh4Gf4kAAAAAAAD4DJ1zf8ev17rtHljqddmj42fDR85zJp+euRhT0JeMYUnP771q1EbYyJiNKXISo5/dOLpn55U0GiEy+t/5Sw3X/TCkfRMTliI7/o04rZzCKrKwv2eMeztt7vQhHZ2teFTWu2d3zu7Zcyo4rbDPms59snnCuNCp08b2cRPZmHIJedaHmJCn76tnLgh52KFl61z+t7Xvwm1zno/aWfWbLn93pAHbf9zDWzbKjZ2QVHwrVOyl37exBndv17KZrYUBi1ZIUuIjntzxPXj4WlhuiU7uvKATZ2IGf++Q63f0UomaYRibq09gtEyqKCcFQctkMoTxDA1whLROB0NnB+7aerfnn+7Tfxx4ae7ljPJ3jUq7vWz8nKj5M0d1b2FrhHLiXwXF8vr0akrR5f0uK6dNTWcgMtd4Zmo92bTLD77zKM+ja8+e5iePJ+uQmZHcP3U1yWO84f3LdwtHiUgfX7yd6vmd/r3T14undso/elquAln4wRnfRXrNnDGkq4u9GZdQSbPT3sdGhgTEllnaTPdrikwTv45xamxuYqjPJih5Xk56fNSbpw9unTl7JyybrECDdObNY1cXu49p8Or08ZcaEkZ05vXVMxYnL5g2oI3Qgkuo5bk5WWlJ8Y/E2qvGFTaQ92zrpAkR02ZOGtjeycqYpc5JfvsmRsLEkFxLhJyWvTub0DGn/d9VbBakaoXjuFQqu38/8O7de6GhobXSg0GrlPtuZUaJuENsWEIDXA+jc/LVYalk4XmlVp/0z4xpxh3TmOVkSOhjtExBfZCowxLVZWZcaZXyoF9mjIjraccScHEmTaVLVHFKhCFEZUs3PsDmu+i1MsIZJPUhQ61rUbliMCPTBkX/c/XqFYRQcPCTnbv3VmbX66pjPocRQkH3g37bvFnrwqXyNAXPKSvk8vcJCfHx76PF0WKxODoqWvVlcyn379f/wcMHubka82iOjs2GDxv+JZsA9ZLvBd+IiMjajuLrMHTYUJGjU21HAUB59PT0DAwNDA0MGCymibEJQRCZmZlRkVHhEREREZHR4miFXF7bMQIAAACgYuAuFHwt6sivyy5du6xYvhwhtHP33uDgJ1XZNMN15TWfcTFrus+9mAFTDIMqgDUYfeD+xraP1/SafEbH8Sp1As99k/8ej6TtIzz3x9ShXnxQeZhxv9/9tvd+u2XYmMPx5X+mbm7t5s+bixD6bfPmap1jRiAQ8PmmL148L7/PvOg7/3hyg9e5+tUXT11Iq42eAAAgAElEQVQzvmFaCwMpQsjDY1DRi9/iiJnylRpPQxBEI+tGRXmazp07sdlstVr94cMHsTimME8TGa1SVyBPg2HY1OlTp06b+u+RI9dv3Cgzf2jWoEGXrl2qbK9AfXH/QRCqA7fOXwWRoxNcROBrkZycvH3HDrFYHB9XdrkEAAAAAHwt4C4UfC3q6a9LzLCBOZmVrtK37jxt8WjrrFu/+39NPeigLsHN2wxoiaJD335Iz5EzTJq0GbJkTnsWFfPita7jIeqIvMB/j0V4zJ8wvdf/Vt7SZc5zUMcxhBNm9DbOvOV9/n3dybSJxWKExLUdxVcGEjNakCRZXp6mU0c2h1MqTyOOilaWmxu0sLDg6uvTCM2ZO2fQ4EG7d+8JDQ2tqR0CAABQ54ijxQV/ZQAAAAAAAACVh1uO2H5tbVsmQgjRZPrddX/dy4V+aFA5bNcxm3cO5BWvaUeTH67sOxFVdzrDdaOOPrLVd+SBEcvnXfhv02O4JL5yROMxP81wVj3etNfvK0sRgtIgMVMxpfI0OI5b21gLBAI7W1tbW9txY8caGBiQJJmYmPgpTxMtVipL1DwUCAUIIQwhhGHW1tZbtvz+9MnT3Xv3pqWmfr7Fqh/MC75CRQMPQSVMmDhF+0IA1JKCYpsAAAAAqH/gLhTUTfX81yVmxKazpWoTlBX7+LL3bzuv1qHnycFXBmNlR94NbuIqtG1oxEaKnKTY1/cv/bv7+OPUr2/qT1ryYMea43ZeWTQbQ5CY+coxmfkJYX5+a05pKWIG6j5IzHwRiqIK8jRFr/D5fIFAKBA4CIWCsWPGGhqWztPEiGOEAoFKrWYyGAghHMcRQq1auR48sP/06TNnz5wpf7QNAAAAAAAAAAAAACgDGf73hB5/13YUoF6gc4K950/0ru0wqgidHbBpakBtRwGqgjzKd91Y39qOAlQFSMxUsczMzODgx8HBjxFCGIZZWVkJHBwEQqGDQ5P2buO4PB5JkimpqQwGUXwtgsEgEBo7dkyfvn327d0XHBxcS+EDAAAAAAAAAAAAAAAAAKAaQWKmGtE0nZiYmJiYGBAYWPCKZUNLB6HDogULMIR9vjyO4w1MzdauXfP69evHjx/XbLAAAAAAAAAAAAAAAAAAAKh2kJipUUnJSSSl5ujpaVoAwzGEkLPIuXlz5xqMCwAAAAAAAAAAAAAAAAAANQGv7QC+OQKBsPwFaJqmaQrHC2udNWnSuPqDAgAAAAAAAAAAAKgyQqHQ0NCgtqMAAAAA6igYMVPTHAQOJEUROI4QommaVJM4geM4jhBSqVSpqWnx8fGJiQlcLnfAgAEIodjYt7UcMQAAAAAAAAAAAEBFDBs2tHv37vHx8U+ePH35MiQ0NEyhUNR2UAAAAEBdAYmZmtZUKCBwXKVUpqSkxL9PSExM+JCU9CExKSnpQ0ZGRtFiXbp2KUjMAAAAAAAAAAAAAHxdciQSiqbt7OwaNWo0cuQIklRHRkY/e/YsJORFdLSYJMnaDhAAAACoTZCYqWlHjvj89deOzMzM2g4EAAAAAAAAAAAAoFrkSnIpkiRwnMFgIIQIgiFychI2FXp5TVApleEREc+ePQ8JCcEwrLYjBQAAAGpBGYkZgUAwf97cmg/lGxETE1PbIQAAAAAAAAAAAABUI0lOTumkC4aYDAZCiMliubRo0dzZecqUyTKZrOBNLle/5oMEAAAAaksZiRk+38TNrV3NhwIAAAAAAAAAAAAAvkZ6enpGRkZGRoZGhkY8Q4Nmjs0IgtC4NIbhBEEjWk9Pr+AFtRqKmwEAAPiGQCkzAAAAAAAAAAAAAFA2JoNpaGRoYGhgZGRkbGhkaGRoYGBoaGhoaGRoXJiIMTQwNGAymEWrKJVKmUxWTpkyklQTBCP+Xfyr0NeDPQYhhBQKRU3sDAAAAFA3lEjMeHgMqq04vgSGYbNmzRo4cMD2v7b7371b2+EAAAAAAAAAAAAAfAVYLBaPx+Pz+Xy+Kc+Ay+PxeFyeqSmfz+fzPjIxMSmeYlGqVJkZGZmZmXl5eRkZme/i4vJy8zIzMzMzs/Lyc/Py8vJy87KysmztbPfu2fP5FkmSxHH8xfMQ3wsXQkJCunTtMvjr7IwCAAAAvkR9GDFD0/T+/fspklz04yKcIPz8/Go7IlD/4FZDNv79g+uLDcPXBanKWoCw8/xl1+xmj1aP/jVYXdPRAVBPwHUEAAAAAFCn4FaD1+/9odXLjZ4afgdVPcykx+q9S/rGbe+93K8eDKAgLDrNWjJnVBeRtREuz3h39dcZK66n07UdVYEFC+Yv/+mn4hkXqVSanZMjyZFIJDm5ktzExITsbEmOJEeSI8nNzZVIJBJJjkSSq2P7khxJyRdokqRIkrxx88YF3wspKalVtysAAADA16c+JGYQQjRNHzh4MF8qXbhwgZ4e5/LlK7UdESgTu/V8H++JBrdXei2/lVHbN6MVCgbjNmrmZG0crnEcNjKwEYlsDEI0j9QG4IvVqSuoWtSB66j+H2QAAAAAAJ1hXGsnZxuTyCq7O9N+r4VxrEQtGjdIY2C6LV+nsZwX7N89z4ldcPx4DRqyFXl1Zy+C7gc9f/4iJyenIN2Sm5urUlVl+i03N5emaQzDaJpCCMvIyDh3/vztW7dlMlmZywsEDlW4dVAJbdu0bmRpKZXJZAq5Qq6QyeUyqUwuV8gV8gIkCfMAAfCVqbNfrbacevD0RQUYMsr4/qwniZkCx48fVygUs2bNwgni4oWLtR1ODeEO3vXsj47if+ZO+DM4u8QtHrPbL/6HPSUHxwzZ/Kqu/O3EMAzDcLwCt/XcPltu/D1E78E6j8n/S6FKvcl0XXnVZ7plyM8DJh9OKP1mNQQD6qH6fgUhhBCnw4KjawY1NucbcFkEKc/NTo2PfBMcdPuc792IHB13jXCesmvrBN7luVP2RH7Z0cANHfuOnezZq1MLewsjljon5W3ky4e3fX3O/pdQN/4owzcDAAAAAGoEx67nhO8nenRtbmumjyly0t5GhARdP3Hg7MusWui5r5qbvRK3nbRSmp0WH/0q6ObZf88FJykLl6novVbJ5bXGaeC5L2CrO1tDY6onmzzG+CRW+KdjZbHbjx7bjKWMPr1k0c7bsXmsBla8vLpxy4sQQujZ8+dB94Oqr321Wi2Xy/X09CIjo86eO/f40WOKKu/YD+zfr/qCAbqjEYIfQwCA6tbVWKJ9ofquXiVmEEJnz56laWrG9OlsFuv06TO1HU5NwfScp27bmTRp+rEYpfala5Hi2Y7RrXZUaJX8B9cCMgcPcxvUx+rMsVLJF3Zrj4HWuOLJ9RsfKnFrXYlgQD1Vn68ghBAiGghaCKwKf54S+sbm9sbm9i5dPabMfuWzeumvfok6VA3Dje1EQsss1pfdoWOGLtP//GtZt4aMj+2w+NbOHa1FrtzIa4/qRmIGvhkAAAAAUAMIu1Hbzm7oZkYU3hUxTK2bd24kZIT4nHuJaiExUzU3eyVuOxHHwMzG2czGueOAcZ7eM6btfCyhK36vVWr5qomzpuAWAgcjTHH/0I6r0dk0QorkOF2rgNUXV65cCQp6IBaLazsQUAFfx+UFAABfv/qWmEEInTt3Xi5XzJ49i8czOHz4ME3XnYHC1YcmacMuP21fETth48OcerbD0sfXbqUOGdvKw8P25P53xZ+K4rQf1NsSl9+/4pdcY488VSmcbWBqzFHnZmVJYTqNCli1amVkZGRA4P201KqqSlyfr6CP1GF7xo7cEy5HLK5JQ4FLJ4/xk706t5z8135s5tiN/+XWxF7jlp6b9yzvbkKnvzi678Ap/xcxaUqWibVTm679HZMf1NKBh8sQAAAAADoaN24sV597LyAgOjr6S9tiuE6a28UUpfpvXbv5/Iu4bBWLb+PcrpuL7N5nVQK+Ouqw3WNG7I1Q0ISeUUNB674zlnzv0WLqxim3B+4Iq/5x6Lnn57Q+X/hvRutll09PMTg3s+dP92tigpzPbiwxNoeF0bL09Px6+hNDuyNH/tW6TER4xG+bN9dAMEAXc+bMNjYyLmcBkqJUKuW1q9cjoyJrLCoAwJeLCI+o7RAQgu/8kuphYgYhdPXq1by8vB9/XGRibLx9x45voAim+uXRPcn9fvDasuHV6MW+H8rcX2bn9bd9Rmft9hz5V0TBApjR8D3Bmzv+t6rXlLOZNMJMO89YNam7yMHGysxQn0lKPoTfO7F7/8tGw8YP7dve0dqYyE98c+vfPzefeF1U8QnjCjxmzp02qIOjOVuWEhnku3/LgYAEFUIIM3IdvWBS33bOAvuGxnqYLD3u5oaJG8Xfnbo23/J88TtjPdvek2dPH9K5ubUhJstKjAzcu/rnC3HFdkH29OKtpO8mioZ6OHjvifr0BrfjsF5mWN7dC37pNEKYaY8V2xYMaGZtYcimZekxT296b9vtG1lwB6xTMOW2ULC3rKbDVh9e0rVtEz6Rnxz28LL3zkM33pVdHrfcg4NwftsZa1fO6t3UhInRtCo3/vbGyT+dq8y4n2+RUNi0U6dOkydPjoqMunPnzv0HQZ/NKllR9foK+ohSKZQkTSNFXnpciH9cyN1rd5YeOjS12YTlXqc994aTOlwCRNP5F1/NL2gt9ZSX+88PVTqs9ZF+p9lLevBRuv/KsYtOxxdmQRSpMcHXY4Kva/xsyrmOyr/qS34cSJ4VH+J3cuv2Uy8+FgfRcBliwjnFD7L2dhBCiGPd02vWtKFdXGxN9ZA8Jy0xNurN7cNbvUuVxwMAAADA18zEhD9w4IBhw4elpqb6+d0JDAh8n/C+km3pW9uZElT8lZ3/BEWTCCGkTI15fDXmceHblbmxRHr2fafOmTGkk8iKS2XFPfM/u3fPqeC0YreFWhco62YPIYQw/Q4/eN/c1MzWlENml3UvVBKlVqpImkZqaVbCqzuHFuc0cPGZ2Lhdaws87ANFlLzXQgghnN9y7NzZ4/u0amLKlCZF/PcoxwL/FNPny2uMUydl3mavv4F1r4YbS4QQhnCT0QdDRhcsp3qyrs9UnySq3M+izAg3BDtNr/Ap8ZVIT0+v1opqoEJaurTs27cPg1FmbyFN0ygkJGT7X9szMzNrOjIAQL0A3/nF1c/EDEIoICBAJpUuX7Fcn8v9ffNmZZVOYVcHkYnXVyw2aHxoysatUyOnHAyTV6INnO/SZ3B30cdzgmli02r4T/8ML7YEy67td6v38fNHzL6QQiGE9FvM++fgwlYGBbfNHJuWg3/Y6Wo1f+jqgCwaN+840mtgUWsGDcyZirzPtsl2nLH/n+XtjQtvvFkWAlcbnqxUfkL57MKVmPGzm3oMdN4f9aqwOxcz6jaopwnKunLxTsHdsNrYoXVTaxZCCCGehVOPiX80N1cNXXI5nUZIt2DKa6Fgm1zXQSM/HgubNh5zW3Vy3TBhro+4rLOrnIODNRy1edey7oaYOj8jJQ/jmRqbMxU5kJWpGAzDmjVrJmwqnDN3TlRk5K3bfoGBgVKptHKt1esrSAM659Gu38/0854oHODhuD88lNR+CZRJ17U4HQf3NseVL/7541y8zmNTyjtEWq76kh8H4po5dB6zyrWpnqfXoSg1Qrimy5AoGYG2dhBCHMcZB7yXu5l8LH3ONbVuamrdhPX80KHg7Hr/XAAAAADwTVGrSQaDMDc3Hz161LhxYz98+HD37j1/f//k5OSKNSRLSsgicZteXgPORV2J++xRr4rfWHJEsw54L3MzKrwttGjafeyKTt1aLp6w4nLBU0daFygHxrZt2bbw35/fC2lDqUkKIYTjeFnvYoadVvvsmizkFNxJsW1dB9oihFC11bgt8zabRno1cGP5kZbPoswIMd1PiZkXUqrsaIFvA0EQjRs3dnV1bdXKtUWL5jhexqmrVqvVavXBg943btyo+QgBAKBeKvPWqJ4IfvJkxYoVzs6iDRs36Onp1XY41Y3Oe7Z74V/PKNe5fy1qZ1DpmqC05PqKvi1dXAQtunisupZA0lT2k93zRnZs49q0dZ8Je59JkHF3T3dzHCFENJ24Zp6rfmrAzqkDOzs6t2k/cvXZWNJ62Lzxgo9/xenc2+sGtW3lKnDp2HXUjsel8xdE4/FrfnQzkkddWO01oLWLq1M79/4Tf7/5WS8wGXH53Gslbj9gqCur8CXMpNeQLkZ0yrXzD3ILN/Xgj4nDOrZrI3ByceowaKr3K7lpzxE9TD4dCS3B6NKC8u2136cO7i5q3rpV32kbr8epjTsuXTLIvIyjXd7BwQza921vQL3ZP7xjx7bd3Nu0cesw/M+gSiYUvm0YwnEcwzBh06bffz/35KmTG9avd+/lzmZrmu2zHPX5CtJI9vLe4xwKb+Qk5CJdLgEyaudQl8bNnBs3c3boWvhkova1CoO1ETXl4eTbwKBEnXMVWg6RDtds4cfh4Ny+y/jNt5Mpbsvx41szEUIVuww1t4MQ4TBh7WI3Y2XslXVe/V1buAhatO+yNkD6tT2oCAAAAIAKKXic3NLScsyY0d7eB//6a+uQoUOMjYx0XV/17NDuoHTMbsSfvv7HN87u58j//JnJitxYCrzWLGpnKA8/u2y0u3PzVq36zvjNPwmzGrB+WR8TTJcFEEJl3+whhBCdd++30V3c2gibf34vpBFGsLh8a+euY39ZO8qWIOOfPS+rADWj+bTlEwUsScjRhSPdnZu3auk+fqH3k/TyHzTSFKfuPrvNrsYbSyrr9HTXgmgbN5/sk4Tp9FmU+UNAt1Oi4ocDfItwHBcIBCNGeG7cuOHM6f/t2LHdw8MjLS197559Zc4IEBoaOmvWbMjKAABAFarPiRmEUGRk1E/Ll1tbW//226+GRoa1HU51U0YdXbnRP1fg9cuqz3tFdUSTuWmpEgVJKrPCfPceDyUxRnrYg/DkPJUq/8ODA//czqEJG3tbHCGi2eBBjgyJ36alB+7GZCvU8tTXvht23s0jhJ3czApPLFqdlZiQIVWRCsmHuJTStY2IJoMGN2erXu2cv/Z4cHyWQiWXpES9iEr7/C6cjL90/qkMtxrk2UEfIYQQbtlvRCcuFXft7JOPQxswzLjFmF8PnXvw+Mmruz7r+1kRiGFhafbpFC8/GJ1ayH9y/uTdqHSZSpEd9+jwivWnEiluhz5djT872OUfHJqmEcIaOLo5mnEwhGhF2tuEr268eZ2C4ziO4wyCaNXK9cdFi06cOL50yRI+n1/BZurvFaSROjMzh8ZwfZ4+jnS4BMqk41oY14CLISorM1vn+LQeIu3XbOHHQanzEp+e+PXoGzVh6ujYAEcFo/B1vgzLaYdoMtDDmUVG/L1otU/w+xwlSSrz0jLy4IIGAAAAvgUYhhEEA8OwpsJmM6ZPP3b82G+//WpuYa7DqmTcmYWes3deCss3bTPip51ng24f2eTVzqJ4eqYCN5bCIUOdWarXuxZvPPMyRapSZsc9PLBk3ekk2qTHkF4mmPYFykerUmOiEnPkatVn90JlYDRfeEkcGRob9uLNfzeveK/+zpkrCz+27p/QMgbYEM369bbHFc+2L95y8XWKVKWUJIZcPnZLXN2Djj+/za6xG0sdP4syfwjodkpU54EDX72GDRv2799/xfLlx48f37Fj+4iRI2RS2REfnwULFk6ZMmX79u03bt4Ux4hpVHgGq0m1SqXaf+DAqlWr09PTazd4AACoZ+ptKbMice/ifvpp+aZNm37/bfOatWvr+R8S8sP5dRs6if4auWFFwOs1+V/aWlLcBzVyamBhjCMphRBCyuSEVBoz19fDEGLaNLHGcb1+u4L77Sq5mpW1JY50OM4MO6E9Qb0Pfhiv9b6bSrlx9s6iDh59hvfacv9yNt5k8PB2bHWor++bwlkVDbutPuI91o5ZeB/LtrVBCJE4rvMZXokWZK8ev1Z69W1kb4WjrJJvsco7OFjuQ9876T09uq/08VucFffm/+3dd3wT5R8H8Ocuu1ltuktpgaZQUqZKQQQEERGQIYqyHYADsSBD2TJE2SJTAfWHKKKgyJSNDJE9S3cLTfdKmjR73e+PlFIgnVBS6OfN6yXlcnnuewknd/e553muXDy+6+cf9idVZUbIaVOnkqlV3acnldFY7rw+LDabEMLn87t261q6kKZph6NqYcATewSV24RMJqUYh0FvYGp2EFX9XYxBbySElnpKaZJXtYIrPI5oytipegXbs1Ju6ZlIkUhIEeKo+WF4Vzu3v4XT/7gc0rBqGEI6de60t/OeGrcAAAAAj4bNVs5pDEVoiiaEtGzVqjTl8BB6GPQV9Iu3ZJxYP/7Epq+e7j3ynbeGvvDM0Bnfv9jpi6Ef/Z5yf4JRyYllqDyYdqSf/fdWmfL0l05eNg15OVTekCbGylaoxmwRd58LVcAZVzDFF36Y8emaYzdddijmBDVqQDsyLl3IduvAztU+E36AE0vuQ/ouyv8rUdW9hnpDJpMpIhVt27R5+qmnfP38zCZTXHz8H3/8ceXKldTU1Psvli+cv9ikcRM2m+1wOJISk5YuXVbt0RoBAKAKnvxghhCSlZU1ZcqU+fPmLVu6ZPacOWm30txdUS1iCo7On/3H09+9NmfGqSV3375miIMQHp9f9Z4ADqvFRigOh1P6FqvVxhCKKn0syBWKJ+BVaRsUTVOElNfM3RjN8S17s3sP6zy4d+De7X5vDIxgG05v+avkZJaSvfj2qyEs9dnVsxb/ciYl38j26T7jrxX9qtIyqXkLFEVThNAudrbiD4cp2Dt9hO7yoF4dWj/VtuVT3Ro/3bVbBD3wo72V34vf8ddf8fHxVdmjJ9gH739QweCEdrudxWIZDEatRhMQGEAIqWoqQwh5co8g1wStu7aX0o60+CQ9kfWvwUFUjQPHnpl008g0a9yhne/aJFfjWNyvwo+Irv4xy1gsFsZ52BJSzmE4bq+6ghZctENz2HQFN2mqhCIkPj5+x19/PUAbAAAAUOte7tmzZctW5b3qYBhCCOOwazRaZ9ftClOZUuacizsWX9z5nWLgvBUz+3YZ//EL+yYcvP85pApPLEmlJ481HqjXhbvOhVywxawYOGBdip1ww0eu/X1a+7AIP2Iu77SORRNCym/r0ajBxeADnFg+rH0t96/EQ2ofHm9SibRl65YKhSJSoZDL5RaLJTk5+fiJE5cvX7lx44a1wmmYr1y5MnToEJvN9r//bdq5c2e1LqgBAKDq6kUwQwgpKCiY/Omns2fNWLJo0bz5C2Jirru7otrDFJ1aPuPXqP8NmTIh14Mi2tvLHcUaHUM3aCaXUlcKH8IwO9bMW5kOh3Tn6JdmHXNxxVHOPIf3tUCHRD3bkHX9VqV3NU3nf9sRP/ijqCGvd1A3GBhCFe76fV9eyX7QPgEBXGI4tHnV4XgLIYRYC/O11ZousgYtUNKO3Z/iEktaaqbj9qiALBb7zq6V/+EQYko/vnn58c2EsMQRr837YU6Prj2jyN59ldYZHx9/6uSp6uzZE2j0qNH3L2QcDkJRNrv96uUrh44cPnvm7ORJk5zBTDU9qUfQfShph3GfDmpA2xIP7I2z0/KKDwHGZrMxxMPD464rveocOIb/jpzR9uzeYUz0S4dm7q9owLUqHUesZmMf8Kh3eRh67D1QrTaINSerwEGHPBMVRMek1/yKpSC/AIc2AABAHde6Vev7gxmGYRx2O81iJSclHvvn+PFjx8eO/bBT507VbNuhid2x4vfXek9RyOWBrIOp1Xu3JS0lw0GHtn8ulHU99fZpofCpzm35xKJMzXBUvgKhXJ7sPRhL0s/TZ7TeurLPpKWjrwz9Lt7FqZpFmZLhoBt17Bq25npiVfofuz4pfUAPfjlZjRPLyr+LJ3zAeag9Eok4skWLli1btm7VOjQ0xOFwJCYmXbhw4fvvf4iPi7NUGMaUlZCQcPXq1W/XfatMT6/VggEA6rl69E++XqebOWP2hUuXvvhi3vPPP+/ucmoTU3x6xbwt6dKgoLLP9ttTr8dpGV6nD6YNbevvwaJZfLGvl6Dm57P2hAOHUh0+fecsHtVdESDhsmgW3ys4smu70KrGffaE/QdT7JzW41fNG96+kRefxeKIApq1aebt+q+lPfnPX84YWfLBK2a+JGOUO7aeKr79kqMwL89KBO0HDns6SMSmCM0RifjVSh2r1ALFEvt4Czk0zRYFtuozbc2c/r5EdXT3MQ1DCLFYbQzl+VTXDsEerEo+HFbjF157oVUDCZemWBy2rbjYTPBgUw0xDofD4WAY5npMzPKvvx785uDP58w5dfJUxU8AVdbok3kE0WwOiyKExRX6hLbuNnjGxt9/HNWcb721ZeFPcfZKDwEmLyefYQX2GNSjsYjN4svCnokMYlXr0GPU+9dtvGGmg/qt2Lp2yqvtwnw82DSLK/Zr2v6VDz55tTmLkOocRw961JdzGFb7C7XFHTqa7eC1Hb9sSt9IfxGX5xXabmCP5tzqtgMAAACPG7vdTgjJysr6afPPI0e+9cknk3bt3KXRaqr0Zm7b976cPPKFFiFefBZFsQSyxu1eHTugKYux5eepqv2shz1x165YC6flx8tnvd7K34PNlYZ2fG/J3DcCqaLjuw+rmMpXKOdk70E58v6e9/m2TF7bsXPHuDw/sifs3hNnZSs+Wr1oTOcwGZ9Fs/hSH6/yY5daqfORnlhW/l0AVINYLO7wbIf333tv9apVv/zyy/Rp01pERl6+fGne3Hlvvjl48uTJmzf/fO3ataqnMoQQm802ffoMpDIAALWtvvSYcbLarEsWL3n77benTJns6+u7fft2d1dUW5jic8sX7Hjh29eDyyzUn9y8Oe7FjyN7fbG11xd3FltquhFbzPcLfui2bkyPiRt7TCxdar28uMfQTWlVupiw3fj+i++6rB3bYsD8nwbMLyldvzu6S/RBk4vVHbm7Nx8Y3/FVfx/GdGHrz1fvVM4UHv39yLjOfV6YveWF2XfeYE+s8s5UqQVK0mvhkV4L77zJnLZz1qJDaoYQYn9jYRMAACAASURBVM+Iiy9iIiJGrjvgN7nd+P0VfDhK7/aj5s7qyCm7a6q/D56vcrFAGMIwdgdF03Fx8UeOHP339L/FxcWVv63q7T+BRxBbMe6PhHFllzD2ouubZk5acFrLEEIqOQTsyuNHY6Nbthq49OhAZ6FXvuw9YkN6dQ49a/y66M/81i4Y1rzz2IWdx969K/TOXXGp1TiO0h7sqKfKOQyrP7GQ6fz6pbu6Lx3QeuTKP0eW3aVqtwQAAAB1Hs2ibTYbm83Ozs45cuTI8X/+ycrOrkE7bEX3oQPeCX3tnbsXM8aknzYcUDHVfoDSnrR53oouG6e0G7Rk26Alt1uzZv49Z9EBFVOlFVyf7ClrsHN375Lm1KL5OzuvffXD2cMODP8x6d5u3vbETXOWdtwwNarn9I09p5d5oZwOK+XV+UBDLT3g5WQ1Tywr/S4AKsHn8yMiItq2bdOmTZsmTZoQQjIyMmJjY7f+/tvVK1cf7qUxAADUnvoVzBBCGIb58ccfC1WFY0aPlnnLNm7Y+IQOl8loTn7z5b4uq3uXWWaOWfne+9qJ44Z1axniyWEshqLCHGVK7PFkY81O/5ji8wuHD73x7qghPaIUDb2FLJM6K+XKhfSq36hmdBeXvTU8ftR7b/Vu3zzIk2vT5NyMSdFyKGJyWZLu1JZtKX0/Cis+vHnXXYMGMaq/Z46ZlDN+VK+nw/2FLJupWKPOz1aeSdZUddcqacFRePXg7hPWFvKQBj4SHm3RZCWdO7Rt3Ya/rqlL6jAcXzFxjejTQVG8jGxLhR8ORWVf+udag6fDG3jyKLMmM+ni/s1rVu7Jr/LHVt/Z7fabqTePHD164vgJlaoaE5VWxxN1BNnzk6+nNG/s5yXx4LEcJp2mQJkYc+Hfg9u2H4ktun1xXNlBZE/aFD1F8vnH/do38eKai5QxyfkUVd1Dz551ePbguIODRgzr3ekpeaC3kGPRF2TdTLx8+sC/agepznH0gEd9eYchU5VB5O7myD/06bAPE6PfG/R8yxAp0SivnUoV9eje1ME8kf+4AAAA1GtFavXRo8eOHz+emlrN0cbu5kjdtWg5t+/z7Vo3C/EXcxmzNlcZf/7Ijg0/7ostrtGppTH22zFDb44aO7rfs5FBIof61sUj29es2Xou317FFVyf7D0ETNGJVcuOdVv6wuiJvXeN3V10X+VxG8a8mTDivTH9Ordq5CNkWQ1F+empCVeOp7p8wr9W6nzEJ5aVflkA9ykNYxQKRdOmTdlsdk5OzpUrV7Zt337t6lWtFmEMAMDjh5J6+7q7Bvfo2rXrhAnjT//339fLvrbaHmDUo9rRqXOnaVOnEkJWrl577hz6UtR3UVHtoseNJYR8tXAhJqKQyWRVyWOmTZ3qHN17+Mh3Kl0Z4OGhfN9Yf3LeM2dndX97W+VPPf7804+EkFMnT321cGFl6wIAAIA7yWQytVrNMJX8846zUKjjcHX5uCgvjLl85QrCGACAJ0C96zFT6p9//lGr1TNmTP/yywXzF3yh1Wgrfw8A1AG11ksGoCZov6d7tSZJN25mFWhMbK8mT/eb/GF7riPl8vUq99gDAACAxwHOQgGgtolEIoUislWryBYtWoaFhVEUpUxTXr1+bceOv2JuxODOFQDAk6T+BjOEkKtXr06eMmXO7M+/Xr587tx5SuUDD6ALAAD1DK/N4IUre4vKDqHB2LP2rNuSiMEoAAAAAACgEjKZrEWLFgqFomXLliEhDSmKSlemX712bdu2bTHXb2i0GncXCAAAtaJeBzOEEGWacsInE2bMmLF8+bJFCxedv3DB3RUBAMBjhOIWJRw716RNeEiAlEfMmuzU6yd3bVr9y9k8TDEDAAAAAACuyGQyRaSibZs2CoWiYcOGDMNkZGTExsb+uvXXa9euoWcMAEB9UN+DGUKIVls8c8as6AnRs2bP2rBhw+7de9xdEQAAPC4YzbmN0SM3ursMAAAAAACo0wICAtq0aRMZGRkZGenv72e322/evHnu3Lkff9x0I/aGXqdzd4EAAPBIIZghhBCrzbp82fK0W2nvvfdeo0aN1q5dZ7djCBoAAAAAAAAAAKgJmqaDGwYrmivatmnTqlUriVRiNpni4uMPHz4cGxsbe+OGxWp1d40AAOA2CGZKMAyzffv2nNycSRMn+vr6Lly4yGAwuLsoAAAAAAAAAAB4PHA5nKbNmrVs2VKhUCgUzfl8vkariY2J3fr7bzdibqSmpjocGPIYAAAIQTBzj1MnTxUWFMycNWvJ4sXzv/giJyfH3RUBAAAAAAAAAEAd5eXpGd60mVweFhmpUERGcjkclUoVeyN2008/xd6ITUlJYRjG3TUCAECdg2DmXnFx8RM/+WTmzJkrVny9ePHiS5cuu7siAAAAAAAAAACoE2iabhjSUNFcoWjePELRPCgw0OFwKNOUN2JvHDl8JOZGTF5evrtrBACAug7BjAu5uXmTJk76aNxHc+fO/fXXrb/++iuebgAAAAAAAAAAqJ94PF6YPEwul0c2V7Rq3VoiEZtNppTU1NP//nvjRlxcXGxxcbG7awQAgMcJghnXLFbr11+viIuL//DDD+TysKVLl2HKGQAAAAAAAACAekImk8nl4ZGRzRUKRXjTcA6bo1KpkpOT//jjj9i42KTEJKvV6u4aAQDgcYVgpiL79+/PyMiYOm3q8q+XLZj/ZXpGursrAgAAAAAAAACAh4+m6eCGwYrmisjISLk8LCQkxOFwZGRkxMbG/r1/f+yNWExFDAAADwuCmUrExMRMGD9hxozpy79etnTpsrNnz7q7IgAAAAAAAAAAeAi8PD3DmzZrFtE0ollEs2ZNBQKBwWCIi4s7efLkjdjYhPgEk8nk7hoBAOAJhGCmcgUFBZ99NjV6/MczZ8745edfft+2zeFwuLsoAAAAAAAAAACoHi6HEyYPa9qsWbNmzSKaNvMP8GcYJjMzMzEh8ccffrwRF6tMU+K2DwAA1DYEM1VisViWLlmWmJj07jvvRLZosWzp0iKNxt1FAQAAAAAAAABAJcrOFiMPD+dyOAaD4datWydPnbxxIy4+IU6r0bq7RgAAqF8QzFTDrp27Ym/ETp322Zp1a5cuWXL58hV3VwQAAAAAAAAAAHcRCASNmzRWNFdERiqaNmvmKZXa7fbMzMzk5JQjR47GxsWmK9MZhnF3mQAAUH8hmKme5OTk6I/HR0d/PG/evF9/3bp161b0bwUAAAAAAAAAcCOapoMbBsvl8kiFQqFQBAcH0zStUqmSk5N3/LkjNi42OSnZYrG4u0wAAIASCGaqzWAwLFy46OWXX/7gw/dbtmyxZMlSlUpVe5vr1fOlDlHtaq99eCx4eXm5u4THWPS4se4uAQAAAADqHZyFQt30xFxd0jQd3KCBXC4PDw+Xy+Vh8jAej2cwGBITk07/919CfGJSYoK6qMjdZQIAALiGYKaG9u/fn5SUNHXa1NVrVi1buuzixUu1tKHwcHkttQxQT0Qh2gQAAACARw5noQAPF03TQUFBcrk8XC6Xh8vDwsIEAoHVar1161ZycvLBw4cSEhIy0jMwrgkAADwWEMzUXEpKyoTxE6Kjo+fMmbNt2/YtW7bYbDZ3FwUAAAAAAAAA8CSQyWRyebhcHhYeLo+IaC6RiEunivn39Onk5OTkxCSL1eruMgEAAKqNknr7uruGx16f3r1HjR6VkZmxdOkyZZrS3eUAAAAAAAAAADx+7kpimkVIpJLSJCYpOSk5ORlTxQAAwJMBwczD4R/gP2nixPCmTX/5+Zc///wTPWcBAAAAAAAAACoWEBAgD5eHy+VyeXi4PEwoEtnt9nRlujOESUpKupl6E31iAADgyYNg5qGhaXrgwIHDhw9LSk7+etnyrOxsd1cEAAAAAAAAAFBXsNnsoAZBcrk8NCQkJCSkWUQzqUTqcDgyMjJK+8SkJKeYzWZ3VwoAAFC7EMw8ZI0ahU6aNCkwMHDjxu/379/v7nIAAAAAAAAAANxDJpM1bty4cePGTZo0bty4cYMGDVgsltlsvpV262bqzdTUmzdv3kxJTTWbTO6uFAAA4JFCMPPwcTmckW+N6N9/wIUL51d+s0pdVOTuigAAAAAAAAAAaheLxWoQ3CAkJCSkYUh4uFwul8tkMkKISqVSKpVpSmVycnJycnJGegZGgAcAgHoOwUxtUTRv/smkiVKx+OctW/bs3oNzDgAAAAAAAAB4kgiFwtBGoSENQ0JCQ8LlcrlczuVybTZbVlaWMs2ZxKQkJSbgiVUAAIB7IJipRQKBYMTw4X379Y2Li1u9eo1SqXR3RQAAAAAAAAAANeHh4dEwuGFIo5CGwQ1DQ0MahYb6+PoSQjRaTcm4ZKmpqbduZqRn2Gw2dxcLAABQpyGYqXVNmjQZ9/FHYU3C9u7bt2nTTxg4FQAAAAAAAADqOJFIFNKwYUhIaMOGwSGhoSENg50xjNlszsjIUKYrb91Mu3nz5s2bN1UqlbuLBQAAeMwgmHkUaJp+6aWXRo16V6PVrFu77uLFS+6uCAAAAAAAAACghEgkCggICAkNCQ0p4e/vT1GU1WrNzs52jkumTFcqlUrMEAMAAPDgEMw8OjKZ7J133nnhhW6nTp5at25dkUbj7ooAAAAAAAAAoN4RiUQhoSHOuWFCQ0ICAgICAgIIIQaDISsrKyc7BzEMAABArUIw86h16ND+ww8+4AsEW379de+evRh3FQAAAAAAAABqiVAkahAUFBgYGBzcIDAoMCgoqEFQA5FIRAjR6XTKkgAmPV2pVKZnFOTnu7teAACAegHBjBvw+fzBg9/sP2BAfl7eDz/8cObM2fLW9PXzKywowMMpAAAAAAAAAFAxHp/fwBm8BAUFBQUFNWjQoEGQVCIlhNhsttzc3Kzs7Mz0jKzs7IzMjHRlOuaGAQAAcBcEM27j4+Pz1ltvdevW9fr16xs3fJ+SmnL/OgsWLCjIz/9m5UpkMwAAAAAAAADgxOFwvL29S2eFcY5F5ufnR9M0IUSlUimVypycnOzsnJzcnJzsHGVamsVqdXfVAAAAUALBjJs1bx4xevSYpk3Djx49umXLltzcvNKXwpqErVz1DcMwx/85vmz5cmQzAAAAAAAAAPWNWCz29/cPCAwI8A8IDAwICAgICmrg4+NN0zTDMIUFBZlZWVlZWZmZ2VlZmVmZWTm5OVZkMAAAAHUbghn3oyiqS+fOw0eO8PXxOXDg4G+//ebsTTx9+rQOHTqwWCyHw/Hff/8tXrwEE9IAAAAAQAV8fHwimke4uwqou06dPOXuEgCgXBwOx9fHJyDQ2fsl0N/fPzAwIMDfXygSEUIcDkdBQWFOTnZOdo4zicnKzsrKzLJYLO4uHAAAAKoNwUxdwWazuzzfZfjQYZ5envsPHDh+7J9ly5dRFOV81e5wXLp0ccH8L602PPYCAAAAAK516txp2tSp7q4C6q4+fV5xdwkAQAghIpEoICCgbCeYgIAAX19fFotFCLFYrarCwpycnLJjkaWnp5vNZncXDgAAAA8Hgpm6hcvh9H6lzxtvDPLge9A0xWKzS1+yOxxXLl/5Yv58DAsLAAAAAC4hmIGKIZgBeMS8PD19/fx8fX18/fz8ff38/P0D/P0DAgP4fD4hxGaz5Rfk52Tn5uRk5+Tk5OTmOn/X6XTuLhwAAABqF4KZuiggwH/9hg0smr5nud1uv3b92rw585DNAAAAAMD9SoOZffsPJCenuLscqCt69XwpPFxOEMwA1A42m+3tLfP19ffz9/X38/f19fH19fP18/H3D+ByOIQQh8NRVFSUk5ubn5eXk5Nb8isnOz+/ALPJAgAA1E/syleBR65Xr94Mw9y/nMVitWzZas7cOXPmzMUwsgAAAABQnuTklHPnzru7CqgrOkS1c3cJAE8CLocj8/Z2Djsmk8m8vWX3DEFmtVoLCwtVKpWqUHX2TEp2dk5OTo5KrcrNzTObTO4uHwAAAOoQBDN1joeHxyt9erNZLJevslmsFpEtFixYMGvWLBNO7AAAAAAAAAAeHj6f7+fr6+3jLfP29vP18/aWeTt/8PEWi8XOdfQ6XX5+QV5+rjI9/eLFi/n5Bfn5eXl5+SqVyr3FAwAAwOMCwUyd06dPH75AUMEKLDarWUTTL76YN3PmbGQzAAAAAAAAANUiFIl8fLx9fXy9vb29vb39fH1l3jJfH19vH2+hUOhcx2K1FuTnq1SFeXkFaWnnCwoK8vLyc/Ny8/PyDQaDe+sHAACAxx2CGTeoeEbWyBYKm9XG5tz5ahwMQxiGoiiKopxLWDSreXPF//73443rMTa7vXbLhcfQVwsXursEAAAAAAAAt6Fp2tPT09vb28tL5uvjLfOW+fn6yby9vX1kfr5+PB7PuZrJZMrLz1cVFhYWFCYmJhYWqgoLC/Py81QqlVajde8uAAAAwBMMwYwbdOrcqVrr0xRFbkcyZYnF4g4dn31IRcGTBbkMAAAAAAA86UQikcxbJvOSlf7X2/mzTFY67wshxGK1qgoLc3JyVCpVcnJSoUqlUqlysnNUKhUGHwMAAAC3QDADAAAAAAAAAHUOh8MRi8Uischl9OLn50fTtHNNZ/SiUqlUhaqk5GRVoUqlVjn/W1hQqNfr3bsjAAAAAPdAMOM2586dX7l6rburgCdK9LixUVHt3F0FAAAAAABAlZTt8iISibxlMpnMWyQSymQymUzm6elZleilIL8Ak74AAADA4wXBDAAAAAAAAAA8ZBw2R+opdeYrUqnEW+Yt8ZR6enp6y7wkUk8vT0+xWFy6stlsVqvVKrVaq9EWFBQmJSVrtZpClUqjLiooLFCriqw2qxv3BQAAAODhQjADAAAAAAAAANXA4XAkEolEIvHy8pRKPMVSsVQi8fT09JR6SiQSsVQs8/QSikSl65tNpkK1ukit1mq0aWlKjeZ6kbqoUK3SarRFanWhWm02mdy4OwAAAACPGIIZAAAAAAAAACjB5XIlUqmnVOrp6SmRiCVSqVQi9fL0lEglYonEudzDw6N0fZvNptVqtdpijaaoqKgoOTWlWKstKipSqVRajValVqvVarPZ7MY9AgAAAKhrEMwAAAAAAAAAPPkEAoFEIpZKPcVisUQskUjEQpFIIpFIJGKxWCL1lEhEYolEwuPzS99itVk1RRqtVqtWF2m0mpzsHK1Wqy4q0miKtNpirVZbVFSk0+ncuFMAAAAAjyMEMwAAAAAAAACPKy6XK7pD7ExZJBKxVCqRiEt+icQisVjM4XBK32Wz2YqLi4u1Jb8K8vNTUpK1Wm1xcbFGU1xcrClSa4o0RUaj0Y27BgAAAPCkQjADAAAAAADwKFBeXWeunfxS2ooXpx7GuE5QgZKsRSwSiUQioVgkFopEIpFQJBKLxCKxSCQsE8SIuWXiFkKIxWrVFRfrbsvLy09JTVUVqlRqla5Yr9MX63Q6XbGuqKjI4XC4awcBAAAA6jkEMwAAAAAAAI8CxQ9StGzsm8+mCCGE91T0TxtHig9NHzH1YCHj7tqgVt3p1CIUlvxHLHT+KCrJXIRCkVAkFIuEHkKR6J636/V6vd6g0xXr9XpdsU6j0WZlZun0er0ze3H+rtfpinW64mKL1eqWfQQAAACAqkMwAwAAAABQX7E8I18ePKL/C8+2CPUXs03qzISLJ3f/9sv2M1mm2tyssO+qi0u7JK0YOGBdir3Mcsrvzc1HZkdd/arryJ+zKn+UnxX5zqplw0W7x76zJsFe3krCvqsuLn2Bd99y84FJbaL31+puVoqiKIqiacqtRUD10TTt4eHh4SEQeHgI+HyBQCAUiYQeQqHQQyQSi0RCYZn0xdnNhcO+q1OL1Wq9Havo9XqdTq9XZWTodXqdrlivM+j0xTpdaeai1+v16NoCAAAA8IRBMAMAAAAAUB9Rns+MW7F0fAdf1u1ggOcfFtU7LOrl14du+/yD+fuVdf2xe9ozVBEeqOY+rsGG+eI3b7T9xt1V1G/OjixcLpfL45YOGsblcLk8bum4YVwOp+RV55pcrqenJ03T9zR1zwBiWm1xZlaWrjRhKdZbLGaL1eIcRkytVjMMekkBAAAA1F8IZgAAAAAA6h9Ww6HLVk54VmLPPvPDmo3bjl1N0ziEQc079317wujuzd/4cr02Z+DSKwZ3l/mQ2GLu650DTwwWiyUQCPh8vodAwBcIPDw8hEIPvkDgIRAIBAKhUCjw8BAIBB58AV8gEImEAoFAIBB4eHjw+fz7WzOZTAaDwWg0Go1GnU5vMhoNJqOmWKtUKvV6vdFoNBiNJqNRrzcYDAaT0WgwGo0mk16ne/Q7DgAAAACPLwQzAAAAAAB1XUREsy6du5w4eSIhIfGhPGgvev7D8R2lTO7+KUM/3ZlVElhY0i7vWn3l5OXpv383pOnwCYO2vrspw0EI5f3cmBlvPa8IaxjkI/HgEJNaeeXwr8tWbL2svlMJJZT3eW/sqFc6RPjxjLkJp3Z8t3j98YwH73MjaPTSux+O6ddRESR0qNMuHt2+ds3Wc/llEhZW0+id16IJIYQ48raOeGH+6epttEp7RwQhL779weh+z7UIllBGdWbCibUz5/+VZq+0PFrWesjYD4b1aNvEm2PIjv/vjMb/TkcLVviHW/dFB/75XrfPTlqrWgk/uNuI90f179QqxFtATJr8zNTEmEM/Lttwrqh6H2zdwC2LxxWJRFwOl8vlle22wuM5f+RyuTyRSOhct7Tzikgs5nI497dssVotZrPFYnH2X7FYLBaLRaVSpSnTdMU6i8Vyu/OKXqcvtpgtFqtFV6zTarU2m+3Rfw4AAAAAUN8gmAEAAAAAqOuEQmH/Af37D+hfWFh47NixE8dPpqSmPEiDHXt39abM5zcu3511TzcSRn16zYrDvVe+3OaV7kGbN2U4CC1r1aPv84rSKwehT9hzg2e0aSoYOOKHROdNbI+W477fMKGt2Bk68Bu27vvxyjZB0f1nHlc/SIrEV7y/fuOnUdKSLMO/6fNDpnXs0nrS8Gn3lV1jVdg7XsSY776f2t6zpAyuv7xNQ5HRUWl5lKTjzJ9WvR3Od461xgtp0zuEEELMNa6EHzFm/capUV63p6URegc39Q5uwr30wyMLZjw8PGia9vAQ0DRL6CGkaEooFFIUJRIJCUWJPESEIiKRiMvl8Hg8gYcHl80RCAR8AZ/D4QiFJbGK8wce7/6pf0rodTqLzWY2mQwGg9VqMxoNJqPJYrWoVCpniFJcrLParGaT2WgwmK0Wk9FkMhmNJpPRYDAYjAaDAZOyAAAAAEBdhmAGHiOs0IFfrPqg2ZmZb3x5rg4+yEYH9Zv37cdtLs999fNTLh/UrOP1AwAAQN1VepfZ29t7wIABr7/+em5u7tGjR0+cPKlMU9agwWZyIW1POPlvjou714zm9Mnrtl7PySOasEhGyQqM9u/pr0/dm6OzCwLbvPr5sik9Wg8b9tTmz89ZCWE1HTlrXBuPvOMrpy/67XSaSdq815RFs14bMG7Y/06tTiovQWG3mLAreYKLF26fSLHkI2Z90k5iits+5/O1e2PV3KBn3pg6d0q3XnM+PXrqk/0lkY89ceXA17+OrzinuXdbjGHfh+2nHLCU/rmivWs8bNbEKKkp8a8v53+372q2kScLCZOqC2j5qIrLY7cYNXWknKu9svnzL348FK9m+ym6DZ0w89124goqrbCSsOGzJ0V5WlL3fDVn9c4rWToiCHh18cG5z1W473eJimrP5XG4XG5JJxQOl8cr+Y3L4fD4PA6LwxPwOWw2ny9gs1l8Pp/NZgs8BCyaJRAIWCxWxe07ExG93mC1Wkwmk8lkNJutRoNBU6SxWC16vd5stlgsFoNBb7FYTWaT0WC0WCxGo9FsMltsFr1ObzGbLda6PrsRAAAAAMADQjADjwDvqeifNo4UH5o+YurBwgd5aFLcUKFoKL5CPbIJXqtVOSVs0Kx5sGdc+dU98voBAADgCVF29DI2m00I8ff3HzRo0JAhQ7Kzso8eO3bixImMjIyqNyj2oIhDU6hx2auA0avVJoYWCIXs0piEsRfn52nNdkJ0mRe2fLm5V7cpiogIX/pcloPVrO8rEWzt4QVT1h/TMISQvOs75q7s1HNF945RAUd7rtr1cbOS2/n2lDWDXl16o2qdXVjh/fpHcq3XF0+aty3FTggxpJ1eP/nz0D3fDunar7vXge2qqu9uZSrauyav9G3Bs15bFD37l5t2Qggx5yZeziWsiBEVl6dp1vPFRrT54opJi3c6463MK7t/Pjj4rXZta1pJ7z6RXHv8N5/M/CnB+bXo8gt11Tq7/vzzWc4fSkf6up/ZaCq2Wi2WHLPFYrXcXstqcXZVsVgsFrPVOY99yZLbzaCfCgAAAABAFSGYqfNoScRLQ94e2L1jy0b+Uq5Nk3sz4erpQzt+2v5fRjmDINzGinxn1bLhot1j31mT4OaJTimKoiiarq08Qthj8f5v+wn+/bzP27/l3nstyGkzfe9PowOvzO/19o8Z1b5QrOXKAQAAAKrE5bwyzoQmMChwyOA3hw0bmpWVlZKaWsUGiw0MoaXeUpoU3H+iSAm9vPgUY9Abyunka89KuaVnIkUiIUUI4TZsEkzTgp6rzvVcdfdqQcEBLH15JdhiVgwcsC6l7OYpvzc3H5kd5fwDN1QeTDvSz/57q8wq+ksnL5uGvBwqb0iTagQzLrZVvrv3jh0a3ojlSD93Wnn3uystzxDUqAHtyLh0IbvGWYXLSk7/k1zzDiUDB75mNldyFQEAAAAAALUNwUydRklajV769addAti3gwGuLDjy2WBFG2HCvjMZ5oofj6M9QxXhgWqu+0MF88Vv3mj7Te21r/9333FV3wFRr/QI2vbzPeEL76k+vYNp8/m/92fV4JK4tisHAAB4YjlnknD5ksDDg0XT9y+nCCUUCV2+hc1m8fkCly/xeDwO18XU38Q5GQblakMUJRS63hCHw+HzXc974RzrqXobommhAFR2UAAAFJFJREFU0MPlW9hsNp/Pd/kSj8/jsO/dUHkfphPNYhFCggIDg4KCnEv8/f0rWJ8QkpBsYCLCOj3rvy7lvpMkSvJspxZsxpacUG6SwVgsFoainE+vuEyNCCGE4gnY8YsGyldXXEt53HYWe9feOZ/RcbGLlZVHsWhCCPVAT/jcVQnNYdOE2GwP8sgVUhkAAAAAgLoAwUwdRgcOXLhm6vNeTMHlzevWbz16OSXfwvUKbv5055cjcv7VPMiQYE8aw9l9B/P6DWnbp0/Ir9+VfWiR8Nu/8mIgbTq557CrAdQfAzRP7O3JtxWr1eU9sQoAAHfj8fkctuszHKFISLm6l0rTtIeH67vnFaQLfAGfXc6GnNNf34/FYgkE5aUL5W+IL2CzXc/rIBKKXC5nsVgCD9cbqigvKX8CCZHI9YYqSBfqAudgTS5fMhmNNrvr+9s6nc7lcrvdYTQaXG/IYraYXXRiYByO/Lx8u6N6G7LZbEaj6Z6F/v7+vXv3crl+6bvYbLZapfaSeRFCcnNzK1iZEPLf38cK+/RvN2Zi36Of7cy6q9eK17MfTejhSZku7j1StQdbrJm3Mh0O6c7RL8065voTqhFLWkqGgw5t/1wo63rq7QqFT3VuyycWZWqGgxDKZrMxxMPDozYjHGvmrUwHHRL1bEPW9bLnmZWWZ1GmZDjoRh27hq25nvgwJk2x5mQVOOiQZ6KC6Jj0x/PUFgAAAAAACCEIZuoyj44fTO4qIwVHpw/55HdlyU15c17Kub9Tzv1dsg7l3XXa8vG9mgX7S3iMsSDlwoGNy1fvSNDfCW1YTaN3XosmhBDiyNs64oX5p62EEsr7vDd21CsdIvx4xtyEUzu+W7z+eEbp1SI/uNuI90f179QqxFtATJr8zNTEmEM/Ltt4rqikWUGjl979cEy/joogoUOddvHo9rVrtp7Ld16OUtI2b4x/66V2kfJGAZ4CyliQdmDuyHnJb27dFx3453vdPjt5ezOCkBff/mB0v+daBEsoozoz4cTamfP/SrNXvkcuGS/sPJj95khF/z5hG9ck3rliFj47oLsPpTv21+ECppKPq0qVV14exW06YOaPkzs/00TG0ufEnt69ceUP+28Zyyu8gu+Clj0zZvb0919s6sWhGMZarDw07+3P/qhJvx8AcIMK7n1X9BKPy+W4eKm85YQQLpfH5bl+hN85t3P1NsTl8niu3+KcJbrcGlx1I+CWnzTU8PMp56UK+jHUBRVkA85ZGar1UiVvMZe8ZLfbdcV3bvpbrFaLxXUNzhkiXCw3l78h650NuXipvHeZXddQSWvlvVT+56DX68vtv/E4UygULoMZu91G0+ziYu2xf/45deqUTCabNnVqVRos/ufbb053ndvp5SVbpM1Xrt92IkZZ5BAGNOvY992J778YxrYmblrxexXv/tsTDhxKff+DvnMW36LX7j2fnK+zc6SBYa0DdafOp9X80RJ74q5dsWMmtvx4+ayC2ev2xao5DZ5587O5bwRSRQd2H1YxhDB5OfkMK7LHoB5bEg8pbZJGLQKNl29kPdxBfO0J+w+mvP9h6/Gr5hm+2LD3arrWLvBtIpcWxFRSHpOwe0/cexMiP1q9yDR/zR/nbxVZOVIfr5qnSLa4Q0ez3x7ZdvyyKflz/3csqYgT2Kpnj+YV9aUCAAAAAIA6CcFM3fVs3xf9aMvl75f8oSz/etbmGfZU02Dn1ZjIv3nXkUta+Fn7T95dUP79CI+W477fMKGt2DncBr9h674fr2wTFN1/5nE1Qwg/Ysz6jVOjvG6PuSD0Dm7qHdyEe+mHH84V2QkhfMX76zd+GiUtGa3Dv+nzQ6Z17NJ60vBpu7PshNB+z74+orfi9l8ssa8fx3z/w6C8iDHffT+1vWdJI1x/eZuGIqOjhntECCGWi3/tSRn2QdM+vSO/S7xW8oFR0i6vdPMi6j07j6iZyhqvWuWVlkcJ27zyesnP3IZP9xnbtmObucPH/uRyKPAKvgsqYNDCVZ8+L6Fs+sJcHSXy9vTjmF1Pzwtwlxrd1K727XtSozygwtYeyzzA5XBDdYdb8oC7lpstOp2+/uQB5fV+AHgoGOau0wDG4SAUZbPbz505e/jI0YsXL9jtdkJIp86dqtqiXfnL5PGyFUuj23d8/6uO79/Vuj5+2+z3V1yucu8XW8z3C37otm5Mj4kbe0wsXWq9vLjH0E1pNT9/sSdtnreiy8Yp7QYt2TZoye3irJl/z1l0QMUQQuzK40djo1u2Grj06EDnJq982XvEBuX9m2S3mLArecK9VS/uO3Rd5ZPy2G58/8V3XdaObTFg/k8D5pdUod8d3SW6svISN81Z2nHD1Kie0zf2nF6mxZqOJmY6v37pru5LB7QeufLPkWUrrGF7AAAAAADgJghm6i5FUxFtTzlxKrOCZ/6Y4n+XjBwwIyU9X2flSEOeHf3l6lHdXuvqtWe7qiQosCeuHPj61/GlbbCajpo1ro1H3vGV0xf9djrNJG3ea8qiWa8NGDfsf6dWJ5Gw4bMnRXlaUvd8NWf1zitZOiIIeHXxwbnPlb5dPmLWJ+0kprjtcz5fuzdWzQ165o2pc6d06zXn06OnPtnvjD8IU3xozpBpuzKK7AL/AIHGSoLuqprVeNisiVFSU+JfX87/bt/VbCNPFhImVRcwVdqjctjjd/9x/d2pLXv1b7P22gULIYRQXt37dZIyub/8+W9xFT+uSiqvSguWm39/PX/dvjM3i3lBbV/9ZPbUl5+dMvmV/R/uyLt3B1hNR5b7XazJa/9Se7Ej5rvX31lzVWsnFM+3ka/1IY4NUgvqTx5QXhhAHnYeUMFbeDwe58nqH1CDnIDUKA8oLwwgDzsPKC8MqLy1auYBFrPZYn0YI+QAQN3mcDgIIYRhHAxDCLl86fKRI0f+O3OmvP9xVQWjPr/y3QHHeg97a0DX9pGhfmKWWZ2VcPnUnq2bf/s3897B1Cpuqvj8wuFDb7w7akiPKEVDbyHLpM5KuXIhvebFORljvx0z9OaosaP7PRsZJHKob108sn3Nnb7axJ60KXqK5POP+7Vv4sU1FyljkvOphz+sGaO7uOyt4fGj3nurd/vmQZ5cmybnZkyKlkNVVh4xxm0Y82bCiPfG9OvcqpGPkGU1FOWnpyZcOZ5as/9xO/IPfTrsw8To9wY93zJESjTKa6dSRT26N3UweHwHAAAAAOBxgmCm7hILKeJQq4oqvMqiKM+Wgz+d0UERGijj6LMLHCzC9g/0oYnKdZzDatb3lQi29vCCKeuPaRhCSN71HXNXduq5onvHKJ+1qZLefSK59vhvPpn5U4LzalGXX6grMzBaeL/+kVzr9cWT5m1LsRNCDGmn10/+PHTPt0O69uvudWC7ihBCCGNTZ2YUGqyEWLPStITcPVo9q8krfVvwrNcWRc/+5aadEELMuYmXbw+DXt09KmVX7vrzwvjWHV4Z2GHZhRMGQujAnq91FDrSftt+/vaNhUobr7jyKrWgP//nr8cSrYQQY9qZH6fNadRy44gOPTp7/vWHuhrfxbrdDEMI5RsRFeGTcD7XxJjzb2ZU/AGU9c03K1wuZ7PYfIHrmQAqyAA8PDxoV1M01xHlzgTgsBsNrgeRs5gtFqvrO0VGo9HuasoBhiH68mYCsNtMptt/x/RV3ZDBYLDbXR7djE6nd7Wc2O02o8n1XTKLqdxb80aD0eXcBoyD0Rtcb8hmK7NHdzObzFYbMgAAADdwDs+WlJx8+PDhEydPajXah9OuXX199+rJu1dXvFLSukHh6+5aZP13TlTzOXdVWJy465vPdn1Tpc3qd38csdvFcibvt+Etf7trkfHWgdWfHii3QEvagWXvHlhWg22VqtLe6ZL2fDNlz/17V0l5hJgzT2z8/MRGl6/d88FW6XO2ZZ/+dtrpb0v+RPm+sf7l7o5iTXH5FQAAAAAAQJ2DYKbu0hsJoaWeUprklRNJUJIuM/+3cUgop+S5QF5IQ0KInabL/1q5DZsE07Sg56pzPVfd9YI9KDiQZvuEN2I50k//43LcLUIIN1QeTDvSz/5bduJT/aWTl01DXg6VN6SJqgo7xg4Nb8RypJ87rbxvv2qwR3c4cvdvP/JJhz49Xu2++OTuIrpJ31fb8Ww3duyIsT144zVtwXjt7HXLiJcaNAqiyT3BTIXfBVV8eseRgm59np/+0+FJ6rSYKxeP7/r5h/1JlU22UyLmeozN7npQC53O9cj7DofdUF6MUf4AQSajyWZzvaHyhvi32+1GYzkbKr9jRHl5CQAAQD2RnZ09etSY7JxsdxcC7kT7Pd2rNUm6cTOrQGNiezV5ut/kD9tzHSmXr2vcXRoAAAAAAFQDgpm6K+mmkWnWuEM737VJOS6fq6dkL779aghLfXb1rMW/nEnJN7J9us/4a0W/ihotdzJciifgUTSHTRNis5V///thDA1B0TR1+7HPe16pwR6VwWiOb9mb3XtY58G9A/du93tjYATbcHrLXyUh0gM2XtMWKIqmCKFdfHAVfxdMwd7pI3SXB/Xq0Pqpti2f6tb46a7dIuiBH+0tqEqpGza6fiwTAAAAHlPFxcXFxegVUd/x2gxeuLK3qOyZJWPP2rNuSyKeXwEAAAAAeJwgmKm7/jtyRtuze4cx0S8dmrk/30U0Q/sEBHCJ4dDmVYfjLYQQYi3M15aZ0ICx2WwM8fDwKHPpZs28lelwSHeOfmnWsfsnLGG3zSpw0CHPRAXRMemuwiBLWkqGgw5t/1wo63rq7cs/4VOd2/KJRZma4SCkCgNeWTNvZTrokKhnG7Ku37rrGrKyPXJiscr9a2s6/9uO+MEfRQ15vYO6wcAQqnDX7/tuT+1StcYrUoMWKGnH7k9xiSUtNbP0w7ldf8XfBSHElH588/LjmwlhiSNem/fDnB5de0aRvfuqUzIAAAAAPDEoblHCsXNN2oSHBEh5xKzJTr1+ctem1b+czcMUMwAAAAAAj5W6O28EqPev23jDTAf1W7F17ZRX24X5eLBpFlfs17T9Kx988mpzFnEU5uVZiaD9wGFPB4nYFKE5IhG/TGbB5OXkM6zAHoN6NBaxWXxZ2DORQSThwKFUh0/fOYtHdVcESLgsmsX3Co7s2i6UTQixxR06mu3gtR2/bErfSH8Rl+cV2m5gj+Z3Jh6xJ+7aFWvhtPx4+azXW/l7sLnS0I7vLZn7RiBVdHz3YVXVhtmyJ+w/mGLntB6/at7w9o28+CwWRxTQrE0zb7qyPSIWq42hPJ/q2iHY477ZX5xtJ//5yxkjSz54xcyXZIxyx9ZTpU+WVtp4parUAsUS+3gLOTTNFgW26jNtzZz+vkR1dLdzFpm76rdX+F2wGr/w2gutGki4NMXisG3FxWZCamEuWwAAAAB4XDCacxujR77W5dl2TRWtmrbt/PxrY2duOpvjelhZAAAAAACou9Bjpg6zxq+L/sxv7YJhzTuPXdh5bNmXbDfonbvibh79/ci4zn1emL3lhdl3XrMn3v5BefxobHTLVgOXHh3obPDKl71HbPx+wQ/d1o3pMXFjj4l3NnV5cY+hm9IcpvPrl+7qvnRA65Er/xxZdnuljSdtnreiy8Yp7QYt2TZoSclCxpr595xFB6qYyxBiu/H9F991WTu2xYD5Pw2YX9KGfnd0l+hDlexRRlx8ERMRMXLdAb/J7cbvd9HPxJG7e/OB8R1f9fdhTBe2/nz1zoQlTGHFjVeuSi1Qkl4Lj/RaeOdN5rSdsxYdUjMu6o8p/7tQercfNXdWR07ZXVP9ffB8lYsFAAAAAAAAAAAAgLoIPWbqNHvW4dmDB771xeb9l27maU12u92ozU25enL7hl//VTsIo/p75phJ3x+LydKa7XabWa/Oy0i8evZMssYZkdiTNkVP+fFYUoHBbrcZClMvJ+dTFFN8fuHwoRPW7jmTlKc12e1WfUHateMX0p0JhiP/0KfDPlz85/nUQpPNZipMPbfzcKyBIQ7m9vgIxthvxwwdu2rvhTSV0WrR5yWe+PWr4W9O3ZVVjYGtGd3FZW8Nj16778KtQr3FbjWo0mMvpmg5VGV7ZDi+YuKawzE5xZkZ2a7niCdEd2rLthQb4yg6vHnXXeOxVdZ4FequuAVH4dWDu09cTcpSGyx2u82oUl7b//3sN96c9XduSR331F/Bd0FR2Zf+uZamMtocDrtRrbx2eP1no6bsya/6hwwAAAAAAAAAAAAAdRAl9fZ1dw31zt69ewgh586dX7l6rbtrqRTl+8b6k/OeOTur+9vbqtwlBtwketzYqKh2hJA+fV5xdy0AAADgBp06d5o2dSohZOXqtefOoa8tlMBZIgAAAABAnYKhzOAutN/TvVqTpBs3swo0JrZXk6f7Tf6wPdeRcvl6lbuVAAAAAAAAAAAAAABAORDMwF14bQYvXNlbVHaSecaetWfdlsRqjFQGAAAAAAAAAAAAAAAuIZiBsihuUcKxc03ahIcESHnErMlOvX5y16bVv5zNc1T+ZgAAAAAAAAAAAAAAqBiCGSiL0ZzbGD1yo7vLAAAAAAAAAAAAAAB4MtHuLgAAAAAAAAAAAAAAAKC+QDADAAAAAAAAAAAAAADwiCCYAQAAAAAAAAAAAAAAeEQQzAAAAAAAAAAAAAAAADwiCGYAAAAAAAAAAAAAAAAeEba7CwAAAAAAgIdMLg9zdwlQh3h5ebm7BAAAAAAAuAPBDAAAAADAk6b3yz3dXQIAAAAAAAC4hqHMAAAAAAAAAAAAAAAAHhH0mAEAAAAAeELEx8V/tXChu6sAAAAAAACAiiCYAQAAAAB4QhQUFJw6ecrdVQAAAAAAAEBFMJQZAAAAAAAAAAAAAADAI4JgBgAAAAAAAAAAAAAA4BHBUGZuI5fLo8eNdXcV8ESRy+XuLgEAAAAAAAAAAAAAKoJgxm1kMq+oqHburgIAAAAAAAAAAAAAAB4dDGUGAAAAAAAAAAAAAADwiFBSb1931wAAAAAAAAAAAAAAAFAvoMcMAAAAAAAAAAAAAADAI4JgBgAAAAAAAAAAAAAA4BFBMAMAAAAAAAAAAAAAAPCI/B94dxX1ki7TngAAAABJRU5ErkJggg== ) ## Clone a blueprint from the Leaderboard ``` ridge = menu[7] blueprint_graph = w.clone(blueprint_id=ridge.id, project_id=project_id) blueprint_graph.show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABrQAAAFkCAIAAADuU0y9AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3iUxdrH8Xm2pW1624RkQ0looQSQUKUpICiKiB4FQRGxIIaiKPYuRVSkqQiKguUFDkhVAUGkHAkgPZAG6b1n07a+fyzEENJAYCH5fi7Ouczuk3nu2QSS/HLPjOTq6S0AAAAAAAAAND0yWxcAAAAAAAAAwDYIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAAAAAIAminAQAAAAAAAAaKIIBwEAAOpn323alhPn06K2vtZbXc+lql4vrd0eeWznW92VN6Q0AAAA4OopbF0AAACAjSjajP/0/fGdtf6+Hm7OTvYKyVihy8tMjj995M/f1n+//n8p5VUulmQymSRJcrkk1TespmPvsNaKhB31XXgtXdFcAAAAgIskV09vW9cAAABgC6oBnx3/6THvGhdSWMoTtr454fllJ0uueFjH+1fFfjlCkbBkRJ/XIg3/usqGuU5zAQAAQGPHsmIAANDE6ffM6qrRaNy9NV6Brdv3GfH4Wz8dL7DYNb979urZwzxuZPvfv9eY5gIAAIAbgXAQAAA0dYbysgqj2WIxG8sK0qIP/rxk6j2PLos2CLn//ZPvb3ZrfbfUmOYCAACAG4BvEQEAAKqxFB9etzneJCRl+06trTs0S5rHNqZlFaT/PrOdvOqlknOb+19asnnf8YTklIzzJw9uXvbu+G7eNbboKby6PfLKF+v3nI5JyM5My0yMPv2/X9d/Nfe1iX39q31HZhfQ/6nZP+04fC4pJTPhzNHt382Z0MP3KneKrmEuV3aXhpStbNZ77Iy5X635/cDRc4nJ2RkpyWcO7f2/57spr+Be9s0HT/ts7f5j0enpqRkJUcf3bPzhk8c621/BBUIIofQOH/fWN1sORJ9LzkyKPrn7xwXPD23lWO2a+qoFAABoSjiQBAAA4HImk0kIIclk8jrW4iq09y1as/jhYLuL1/i26TWyTS/rAJdcKbl0m7by29du91FUDufk3izEvVlI1wH97Q+t2pemv3ilR59XV3/9Qrj7xeTNs0XXu57pcsfdfaePeHpNgvHazKWBd2lg2ZLnnS9/NKu/6p9bKr2D2npIxeaG3kvZ5skfN70/0PPiJUqvoFCvANfTSy6+jPVeIISQ3Lq/sPLbV/p4XZynXWDHOx7vOOg/D/84eczMDYkX9n+su1oAAICmhs5BAACA6uzbDhsaIhcWQ/TpmFqPFFGEPL10wcPBdpa8w19E3NMlROsT0Kbr8Kff33CmqFrMJHnfN/+b1/v5yCviNrw1tndoC28fP03Ljr1e+q3IcumVMr8HP172QribIfHX98cNaKsN0LTudf+bWxIMisAR7895yO8qvnWrYS4NvEvDy7YyJax+akDHNs29fQMCQ/sOn7Ym3tTAe6mHTJ/Z31MqObHymbu6Bfn7+wSFdhs6bvq7605eiCnrvUAIme8DH3/zal8vSXfquxdGdg3R+gZ16jdh/q50k32bR5Z+PbWzqgHVAgAAND10DgIAAAghhJAp7BycvQJadx0wavLUcWEqyZy55fN1SbX1kzn0feq57k6SKfGbpx95ZXehRQghys9Fbph/SnQe/uWIKt9kKTtPmnWvRm7O+/Xlh578PsWaQRmKMuMTcw2Xpmyqbk+/NMxbKjs0e+yTC87qhRCiNH730slP+ezcNiVk4CMjAn5aVmtBDZ5LA+/S8LIvMBcnnolOzjUJIQyZMYcyG3wvqVm71mqZMBz9fsGaw2lmIYQ+O/7Ib/FHLo4sr+8CIVRdnn75bh+ZKX3t1P9M3ZRtEUKIjBOb541Jsvy67cWwTk+/dN/KR9fmWOqsFgAAoAmicxAAADRxqjsXnMnPySrISstMjD69f/OqDyb08pHrU3e+Nf6lTTk1xmBCCGXnOwf5yoXh9A/L/iys7SIhhBCKDvcMD1YIU8JPn6xNqbM7TdllxPAWCkv5vlWrovVVHi8/unNflllShXbtbPfv59LAuzS87H89I3N+do7JIpRdHnq8l6e8hmHqvUAoOt0zrIVCGGN+XLQtu+qHo/zEis9/11kklwEjBrhyXDMAAMBl6BwEAACwsljMFiHJJFFx6ptnJ7y7NVZXe+gnuYSE+MqFueDkifN1J2eSc9vQILmwFB3564S+nitDWvvLheQweGF89sIaLrD39nWTibKGtA7WPpcG3qW8wWX/+xmlZ29e/t8Zt48J6jZ106ERe37+v/9bs27zwZTSypffUt8FkkvbdoEKYc47diS62raMlsK/D8cah3e1a9M+WC4OX82mjQAAAI0ZnYMAAKCJ0++c1s7dy8fNy9ddO3TO0VKLpArp3d1b1NkOKDk6OQkhLLpiXT1ZnaR2dZYkYS7Mza+n/05yUqvrvsLerp7OwQbMpYF3aXjZdWjojCx5218eMX7+5qgCs3OrQeNf/XJzZNSfX7840O/i77Hru+DCjSzFhdX3exTCXFSoMwshUzur+c4XAADgMnyLBAAAcFH58U8j5h3UCbs2Ty6Y1aeuXMtSotMJIWQu7q41LnOtcmVFebkQQnJ0cqxnVaultKRECGHOWf2wj5eP2+V//O9ektDgI3Vrm0sD79Lwsq/NjPRJOz4a169Tx6GT3lm1P7lC7tbuntdWr3mzp+PFseq84MKHQ3J2dbnsu1uZi6taZr2E84gBAAAuQzgIAADwD/3Zr6bNjywRypAn5r7S06nW6yzF0WdTTEJy7tazg7LOES0FiUkFZiFz7dS5Rd05oqUoLi7TJGRuXW9rfS22fql5Lg28S8PLrsMVz6gi4++Nn06/v3vfp76PNwi71uMe6+fQkAsufDhkLmFd21S7keTS5bYQhbCUR5/mPGIAAIDLEQ4CAABUZTi7bNZnxyuEKmTiB890UNV62fGtvyYYhaLlmJkPB9UZfOmP/P5nnlkoO4x9ul/dR2IYjm7flWESijZjIoZ5X4vDM2qcSwPv0vCy6yrg6mZUnrB12cZzJiE5+vjWeO/LLjAc3/rreaNQtH7kuaFeVd/DvsPEZwapJUvRns17CupcKQ4AANA0EQ4CAABcSn968Rsr443CvtOz744JrO27Jf2RL2ZvzTLLPIbM3fDjK/ffpnVVySS5nYumRTO3S/Osoh1fLD9dYZFrH/9i9fsPhrfydLBz8mnT9+FXn+nvcumV5fuWfrqv0Cz3f3Dx+q+mDO+qdbdXSDKVs6Z1j3vH3dX2KtoJa5pLA+/S8LLr0LB7qfs89erTd3dr6eWklCS5nVtQ+EPPjGghF+b8xMQCS0MuEPojX8zdlmWW+z+46MdPxvVs7qZSOfp2uPvFVaund7EX5SeWzduYTTYIAABwOU4rBgAAqK70rwUfbH5gxf1e/afPuHP9jO1FNcVK5oz/znhc67by1X4Bg1/4cvALlz5bdQWr/sSnz74etubDof49nvt8y3PVhrFY/hnddP7rZ59p8cMXz4a1Gf32ytFvVx3k0Gt/bj+beMXb5tUwlwbepeFl16Eh91K2G/7U8881nzb3kve0mHP3fPz5/nIh6r9ACGHOWDfjieae377Su/OETzdN+PSfy8pifnruiQXHKhpSLgAAQJND5yAAAMBlLLmbP/7iaIVF7j9q6sPa2r5hshQe/uShfoMnf/T97ycSc4srTBaTvjQ/49yJ/dtWLVm65fw/AWHF2W8fHTzqhS9/PZKQW6I3lBemntr9w5wv/sgzC0tpSUmVmM2c9ftrwwfc99IXG/+KySjSm8zGcl1u4ukDm1ZtPFZ6zebSwLs0vOw61H8vS/a+//tp+9HzmcUVRrPZWFGUEXdk24q3Hhwy7qtYQ4MusA5TEPnR6AF3z/xi86Fz2Tq9viw/9fQfqz+Y0H/ItPWJhpqLAwAAaPIkV09vW9cAAADQBMkCJ60/9GEvsWdm2IPfZdwyS15v0bIBAABQM5YVAwAAXGeSa5/HJ7bO+t/hM0lpmdkFFTIXTauugx9/7eWe9qLot/U7sm7OiO0WLRsAAABXgnAQAADgOlOE3jt15tMB8uqPW/SJP7/68pr0K95H8Ma4RcsGAADAlZDbOzrZugYAAIBGTWHvpLZTyZR2Dvb2SqXMoi/KSji5f8vXs2c8//GeTFP9A9jGLVo2AAAArgR7DgIAAAAAAABNFKcVAwAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRBEOAgAAAAAAAE0U4SAAAAAAAADQRClsXQAA4Ppq27bN/SPvt3UVuOVt+HnD2bPRtq4CAAAAwDVGOAgAjZyXt3ff2/vaugrc8vbu3ycIBwEAAIBGh2XFAAAAAAAAQBNF5yAANBULFy+NjDxk6ypwiwkP7x4xZbKtqwAAAABwvdA5CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CAAAAAAAADRRhIMAAAAAAABAE0U4CADAzUVyH/DGj1v2zrnTztaVAAAAAGj0CAcBALi5SPb+7Tu28HZUSEIIYdc14v/+Prxt7hBPydaFAQAAAGh8CAcBAPWRu4Xe/cycZWv2HDh49uSRY39u+r9PZz7a09/+Ot/WacSis9HHNz/bSn7p45LPf1afPB2z+lH/Bn0Rk4dOWPrr798910Zex0VOIxadjT59/rI/Zxfedb2nWS9JkiRJJiMaBAAAAHAdKGxdAADgpia53TZlwfypPb3lF8MpO99W4cNbhd81eszat55579ckg03rq5/MLah9iF++6lYN1yqOfPZQl89sXQUAAACARopwEABQO3ngmI8XTuvlYkr/6+sly9fuPp5YaHbyb3f7iMenPXlHu4c+XFaUMWr+sVJbl3mNGE8tGDXy83iTresAAAAAgBuGcBAAUCt1/2en9na1ZP46c8xLG9MuhGb6xKObFh/be/TVNV8+0vrRaQ/+9MS3KWYhJM8+k157rH/7VoH+Xi6OSlGen3Rs548fL/jpaL6lckDJKfjupyZPvKdnWx+7sszofRu+nLdsT8q/7z10aD7kiWcn3du7vb+TOT/xyK51S5f8FJldJeWTt47YeCJCCCGEOeuncYPeO3BlN23Q7ISD9s7Hn3ny3j4dAlyksvzU6D+Xvv7ez4mmesuTeXR+ZPIzYwd3aempLE0/+7+/Cn3/WTEtD3n2p20RfuufGvjyXkNDK7EPGDju6Yn39e2k9XQQ5YXZqediTu345uOvIguu7IUFAAAA0NgRDgIAatV7+ABPqeLQ8k82p1Vrp7PkH1iyYOfwhXeF3XOH/6pvU8xC5tFp8Ij+7Su/rjh5terz8GthrR1Gjfs6xiiEEMKx45QVX03r4mwNvuwDO494fmGYf8R9r++pmmtdMfv2Ty9b/lK464U8zbd1/0de6d2v8wuPvnJZ2VetAbOzazvpyxWzerhdKEPlGxwWqC4z11ue5NL79e8WPR5ib133bKcNG64VQoiKq67Evu2kZctnhbtf3KbQyTOgtWdAS9XfXxMOAgAAAKiGcNAGtm7dYusSmorZc+bs27vP1lUAt7A2wU4yU/Te/Rnmy5+zFB7Ye9I4rE9w25ZykXLhAkvRL6+OnrU1Q2dy8Au7/62PZw7uPHZs11VvRRqEkLce/8aUMMesPQtfnft/BxLLXdsNmzn3jQdGThm7ct/i2NpSPEWHaZviptXwxMXOP3nwuDemd3cpP7Pu7beWbo3KV/nf9tCsd2YOHPb2S7v2Tf/1Quxoilk4avSnZ+vOCqvfy1K67dkeM3/TV75d1+xajH1jRrhreczPH7735bbj6WV2HtpWrvk5suCJdZen6DBx1vhgVdGxVW+9/82Os/kKn/YDx0x7/YnuznVUWmclrR5984VwN/25LbPfXrzxWJpOOGjun7f9nT51zh0AAABAE8VpxQCAWjk7SsJcmFtYQzYohKUkP7/cInNwcvrnF00WU3F2VlGFyWzUpR7+4cNVp4xyz7ZtvWVCCHmbEfe0VRTt/GDmst3xBRXG8qyTG95ZuFsnD+kdrmk/ZX1c5RnBUZteDK3rZOFLyEPuvS9UZTi56IV31x7PLDXoCxIPLHvxrTXpFvcB997hfk1PIalrdi3vGdHBznBiYcSb30cm5VcYyosyY47GZEv1lSdvM/TO5rKKIwtemLfxZGapQV+Uemzz6u1xdceYdVYy/O5QlensF9Nf/y4yuVBvMul12bm6f9OaCQAAAKARo3PQZvLy8uPi4mxdRePk7u4eEhJs6yqAxqC41CJkrp6uMpFzeVglObm720uW0pJSY83vbUqLTyixhKrVTpIQQhXYMkAmcxi6KHLooksv8w/QyEtqK6GGQ0Ikn/+s+v3NcOsbqqDgAJk5+eD+hCqXlPy992j5I3cFBQfKRF5DJ3uFB5JcOjtFUEhzuTk58kDSpe9db3ml/s2bycwpfx9OrzGBvepKDvwRd7MfIw0AAADgZkA4aDNxcXELFy+1dRWNU3h4d8JBNAX9+/cfPPjOyEOHDh86nJaWdj1uER1Xamnbqm8v38/j06pnV5JLr74dFBZjXHStaZpFr9dbJMm69Z3FUkvzmmTnoDg7d1Tw4qur8Zr2Bl6JS2YnyWSSEDVMsb7yJLlMCCHJ/s00LqlEplTIhDAaOXMZAAAAQEMQDgLArcpgMHTp0qVTp05PP/VUdk7OgQMHDh86dOrkKb3hmrWM/e+X3bl339d90owRu17eeMnhHpJ7r+emDXaTyo9s/f2y3LDmclMTUs1m141PDnljd+m1KlAIfWJ8ilkW1KNPkPzkuYsVOnW9vYu90CedSzELIRmNRotwdHS8njGiITUh1SzThvcKlJ+s2iRYb3n6pPgUs6x57wGtlpyMuRYfOENGWo5Zpr0t3F92KvmquxEBAAAANBXsOQgAt6qyslIhhFwuF0J4e3ndPWz4u+++t3bd2tmzP7z3vnt9fLz//S2K//jiswOFkuauj374Ytao7q28HJQKO7eATsOf/XTt0jEhCkPs9wvWNDCBMkX/tuOc2WvE2/Mm3tFe46KSy+T27gGhA7oH/avfU5liNm2K0is7Pv/JG6M7+ToqVK5BvZ/66J2H/KSCPZt35lmEsGRlZFvkfoMfHNxCrZDbe7S6LdS/wVsaNriM6F+3x5uUnacuevfRHs3d7eVypVrTJqyNW2w95ZmiN285Y1C0f27x3Em3t/Kwl8vk9q5e7lefZBrP7NiVbrbrMvXjmSNCfdUqO/eg7qMGt1Ndy9kCAAAAaDzoHASAW1Vp6SUNeAqlQgihUCg6dOgQGhr69FNP5WTn7P/fgWqXXRlT0vcvTvVYMD+iR++nZ/d+uupTlpKza998esHRBo9uPLXig68Hfj5p8Izlg2dUPmo4Om/wmG8Tr77FzRS76t0F/ZbP7P7gR2sf/OhicYbUX96e+1ueRQhhStqzKyqiY6dR83eNst7y2IfDx32VdPktazoZ2Xhq3ogxn5+rtwzj6RXvf9lv6eQOI9/7buR7F6oo2RzRL6K+8mK+fXt+769mhQ99dfnQV6uMWHElr0IV5YeWzd90x/yRnccvXD++aoVXOR4AAACARo3OQQC4VRkqal6FKpPJrO2EXt5edw8f/sjDD1sf1wYGXMVdLPmHFj4x8v6Zn/9335mU/FK9saI4+/zh7avennj//W/8mnQlC2EtxYfmPDpm2tItf8VmFZWbTIaSnMQTew4n66+irKrKor6YNGbyoq2HE/PKDPqSrJg/f5z96H9mbbq4DtoU+23EzG92x+aUmkzG0txzR+OypWu/xNiiO/LxY49GLN12OCG3RG8ylOYlRx2JL1JK9ZUnys58Nek/E+av2xedWVRhMhnLi3OSoyJ3/nfPuatbZmzO3vHS2GfnrT90LrfcaCzPPRe5cWdUqUWYLawyBgAAAFCd5Op5Ddad4Yps3bpFCBEZeYgDSa6T8PDuEVMmCyFmz5mzb+8+W5cD1E91kdpZrVar1U7OamcnlVKlslOpndRqZ7Wz2lmlVKrsVOpKzs4qpbLekY0mk0J+YQ3toiWfHzwYeZ2ngpuQ5P3Qsr3v3nbwjTseW3sFhzdb8S8qAAAA0LixrBgArhlJkpycnJycHB0cHBwdHB0cHRwcHdVOagcHRwcHe0dHB0dHRydHJ0cnJwcH65MODo6OarX68qHKy8tLS0tLS0vLyspKSkpKS0vLSsvyCvLLzieUlpaWlJSUlZUZjYZXXnmlxkrMZrMQQq/X//HHnuTU5EkTnxR1HBeMxkXm021YZxF7+nxaTmG5wr1lt3tffLaHyhx/9GShrUsDAAAAcNMhHASAmqlUKrVabW3lUylVKpWd2tlJrVZbW/nsVCqVyk6tdrK28Vn7/tzc3GSy6ts16A0GXXGxXq/X6/U6nU6n0xUUFCSnJBv0+ooKva5Ep9PpdMUlen2F3qDX6XS6Yl1xcbGhAScOS5JksVikS1fImkwmuVyelJS0efOW3bt3V1RU9L2977V8XXDTswt7eM7C4eqqnxcWU9qWz3+IMdX6Pg0Q1rmzvrwiKzsrP6+gsIicEQAAAGgkCAcBNHKXr9hV2SkvBH+1rNi1Pnv5UNaYzxrwXQz7StIzMnTFOr1efzHaK9GVFOt0On2FXq/XFxQUWJv4rgeLxVJRXm7v4GB9w2Q2W4Q4+L+/ft64MSoq6jrdFDc9SVUQvTuyZViIVuNqJyoK08+d3Lvp28XfH8z6d5+Jd95557Bhw6z/rTcY8nJz83LzsnNz8qv8f15uXl5urr4B0TYAAACAmwThIIBbhjWzU6lUF4I8J2drK59KeSH7c1Y7q9VOF6LAC/vyOStr2pivasxnDfIqYz5rK59er9dXGKwxnzX7KykpuQmX5ZaVl9vZ20uSlJmZtXHzpp07fy/R6WxdFGzLUhi5PGL88ms+7vyPP448GOnh4eHh6WHl6eGh8dW0btO6p0dPb29v+cXdLa3RYUZGRl5efl5ebm7eRbl5WVlZ1y8uBwAAAHAVCAcB2EDlit2LZ244V67YtbNTKVUqa8xXdcWuq6trZfRQ6fIVuzpdiV6fV6HXV8Z81VfsFhUbjI2nramgoODs2bObt2w5cfzETZhdopHR6/UZGRkZGRk1PqtWqz08PTzcPTQajYeHh6enh4eHh1YbaE0SrdcYDIbi4uLKtDA9PSMvPy8vNy8vPy8nO6e0tPQGzgYAAACAEISDAP6lqhvzXb5it9rGfNakz8nJqdo2eaJKK1+VpK/mFbv6Cr3eoNcV6woLC02mf7WHWiPw0ksvk6fgJmGN55MSky5/SqVUenh6Vm059PDw1Gg0YWFhXl5eCsWF70ZoOQQAAABuPMJB3ELkQaPeX/RMm79ef+jDSKOti2lsalyxe7GtT13zil0XZ6WinhW71Tbm05XoLkZ7JVVX7JaWlvJj/9UhGcQtQW8w1NZyKJPJ3NzcvLy83N3dvb293D08vD29PDw9g4NbeXl5OTo6Wi8zGAz5+QU5Odl5uXl5eXnZOdn5eQWZ2Zk52dl5eflGI18UAAAAgKtEOIgbwK5rxHfLxzvveHXcrO25/2bdo3Ng+/aBzscuazpDVbWt2K22MV/lcl21Wu3i4lLZuVNJbzDoKyqqLNetYcVutY35dMXFHEQA4IqYzWZrd2CNz9bYcqgN0ob3CK/acqjT6TIyMqotVc7IyKDfEAAAAKgX4eBNT+bSdsgjj4+6o3fH5r6uKmNh5vno4wd2bPhu3f9SKup+T3nohEUfP6rePHnCkmgbL72UJEmSZDIyvStUdcWuSqlSqewqN+a7fMWuNelzc3OTyWTVxqltxa5Br6+o0FduzFd1xW5RURGdOABsro6WQyGEWq3WaDQeHp4eHu5+fprKpco+Pj7WfwmNRmNRUVFeXl5GekauNTHMzMhIz8jLy8vPz2ebTgAAAEAQDt7kJJdOT87/9KV+GsXFWE3lERDaK6B9mFP0tr9SKur+qUbmFtQ+xC9fZftIruLIZw91+czWVdiO6qLKjfkuX7GrUiovrOe9eMiuqr4zdqut2K22MZ/1BF69Xl9QUEDjDIBGSafTxcXFCRFX7XGlUunp6enl5eXj4+3l5eXl5eXt7d25YydPL09nZ2frNXq9PjMrMzcnNzs7JysrKyc3Jyc7OysrOyszkw5oAAAANCmEgzcxmd+oOUtm9Xe35Bxd9fmyn3Ydjc/Wq9wD2nW7/a62GfsL6XewgRo35qt7xa6zs7Pyspjv8hW7+opLNua7fMWu9WKbzBoAbi2G2vsNq65T1vhq/Pw0Hh4eLVo09/Pzc3Jysl5TbZFyZbNhbWufAQAAgFsa4eDNy7H3My8O8BA5u159ZPqapAsLPCuy4iN/iY/85cI1kueAVz6ZOqxNgK+LnaUsJ/7wb8s/WbwhuuSf4FDeOmLjiQghhBDmrJ/GDXrvgEFITsF3PzV54j092/rYlWVG79vw5bxle1Iq+yTsAwaOeyIetXgAACAASURBVHrifX07aT0dRHlhduq5mFM7vvl4eWTBhWEdmg954tlJ9/Zu7+9kzk88smvd0iU/RWZbVy5LrmEPTX1sSPfQ4OYaNwepLCfxt3fGvxv3n5+2Rfitf2rgy3sv3sZBe+fjzzx5b58OAS5SWX5q9J9LX3/v50RT/TO6RupYsWtnp1KqVJUxX0NW7FZZrlvzil29vuJiW5+uuLjYQFsKANhCHeuU1Wq1h6eHh7uHRqPx89NofDVarTYsLMzX19d6wHrlYcoZGRnsbAgAAIBGg3Dw5tVrxJ0+Mv3RFR/9N6n2rd+Mbq26tg5QCSGEUPu2GzD+ow4+hvte3JxTe5jm2HHKiq+mdXG2plz2gZ1HPL8wzD/ivtf35FuEsG87adnyWeHuF/cHdPIMaO0Z0FL199dfRxaYhBD27Z9etvylcNcLIZlv6/6PvNK7X+cXHn1lc5pJCJlPr9Hjhre/+Inl7O2jrLi83c2u7aQvV8zq4XZhEJVvcFigusx8lTOq3V1Dh/bs2cPRwdHR0dHBwUGtVjs4Ojg4OF6+YtdkMpWWlpaUlJSWlpaVlZWWlZWWlubl5SUlJZeVlZWWlZaVlpaVlel0JWWlZWVlpSXWyzgoFgAaC+uvd5ISk6o9rlQoPb08NRqNNTr089NYdzb09vaWy+VCCIPBkJubm5d3YU/D9HRrfpiRnZ1tMtl4z18AAACgXoSDN6/2rdUyU/yf+1Lr+MHCUrz/o/EjX4tPztYZlK7aXk9+uHjiwAcGuG9Zl3chSzPFLBw1+tOzlWPIW098Y0qYY9aeha/O/b8DieWu7YbNnPvGAyOnjF25b3GsaPXomy+Eu+nPbZn99uKNx9J0wkFz/7zt7/SpfPfgcW9M7+5Sfmbd228t3RqVr/K/7aFZ78wcOOztl3btm/5rvuVCWTvefuSVTSkFJgdfjUOhQfhfUrW8xdg3ZoS7lsf8/OF7X247nl5m56Ft5ZqfY2nQjK6Eu4dHRYW+pKQ0JyenrKysuFhXZo30ykrLyspLSkpKdCVlZaWlZWV6vf7KhwcANH4GY83NhgqFwsXFxcPDQ+OnqVyhHBYWds89Afb29tZrrCuUM9IzqoaGdBoCAADgpkI4ePNydpKEOT+voM6fHyTJrePDL73Ws32Qn4eyJD3HLBcKXz8vmcirOVKUtxlxT1tF0c4PZi7bXWgRQmSd3PDOwr5DF9zRO9xr6TmX4XeHqkxnP5v++nfR1nWvuuxcXZVFyiH33heqMpyc98K7a+NNQojSxAPLXnwraMsXjwy49w7339ZZt2OyGPNTU3JLDUIY0hKLhJBfWkPLe0Z0sDOcmBvx5vfnTUIIUZEZczTzKmdUpx9//HHf3n1X/n4AANTDaDRaNyKMi6t+IoqHh4ePj7ePj6+vr6+vr4+vr6Z3797e3t7WLWitS5uzMjMzMzMzMjIyMzMzMjKzsrKKi4ttMQ8AAAA0dYSDN6+SMiFkrm6uMpFVSywmufR7feXyR4KUF5YA22kDhRAmmaz2D6sqsGWATOYwdFHk0EWXPGHyD/CTKbxCmsvNyQf+iKtlRzxVUHCAzJx8cH9ClZJK/t57tPyRu4KCA2WiIXu1K4JCmsvNyZEHki6b11XMCACAm4w1NDx7Nrra42q1WqPRVHYaNmvWrFu3bj4+PtYNbavtaWg9CCUtLY0tLAAAAHBdkbncvGLPl1natOjZ3XtpbEaN3YOSx52P36+V5x9c/Ma87/+Kzy5TeN3x2s8L7q1rUIullsW5kp2DnSRTKmRCGI219+hJVzCBWseQySQhairkamYEoMGGDR3SM7y7ravALcbd3d3WJTQeOp0uLi6uWqehUqn09vbWaHx9fTW+vj6+vr4tWrbo1buXq4ur9YL8goLMjMouw8z09PT0jIzcnBzWJgMAAOCaIBy8ef3v97+Kht7Rc1LEkB2v/5pdww8AMi+NRiVKd6xatPOsXgghDLnZRRX/PG8xGo0W4ejoWCXRM6QmpJrNrhufHPLG7ssbERRd0nLMMu1t4f6yU8k1/cihT4xPMcuCevQJkp88dzFCdOp6exd7oU86l2IWovphvjUwpCakmmXa8F6B8pMJl+SQ9c3ISi7n0xa4KiEhwbYuAUB1BoMhLS0tLS2t2uP29vYaja+Pj0bj5+vnq/HV+PYID9f4aawbGhqMhvT0jIz0tPS0jPSM9LS09PT09KysLKOx9kPMAAAAgJqQsty88n/9fPljfWd0vHfBTx4rFq9Yv/d0Yl6FzMmzefvwQb2VexZuOJublWUQrXuMGtsteu3xdJ1ZoVbbK4S4mKZZsjKyLfLQwQ8O/iFmR5LRpXkHv7Kj0b/tOPf0MyPenpcgW7r1UFy2zqR09WvV2U+371Ci0Xhmx670x8d3mfrxzOx3Vu6OLVD6dRo6uJ2qsiZTzKZNUZNmdHz+kzdy3vx8W1S+stlt/3n5nYf8pILfNu9s4JEhpuhft8c//WznqYveLX3/q63Hk4tMDt4tg11zTsTUMyOhNxgtklvXAT0Djh5IKeUISABAo1VeXp6QkJiQkFjt8Wprk4OaB/Xo2UOj0VifzcvLS0pKqrowOTU1tays7IaXDwAAgFsG4eBNzHD284iXfZZ+MLbd7ZPn3D656lPG07KNm86c37Xm9ym33z3ozR8GvfnPc6aYi/+RtGdXVETHTqPm7xplHfDYh8PHLV/xwdcDP580eMbywTP+udXReYPHfJtoLj+0bP6mO+aP7Dx+4frxVe9XOXjsqncX9Fs+s/uDH6198KMLD1oMqb+8Pfe3Bh8nbDy94v0v+y2d3GHke9+NfO/CGCWbI/pF7KhnRilnzhZY2rYd//lvPi+GTv21gfcDmrh9e/fdvfceW1cB4NqocW2ySqnU+PtptVprYqjRaMLCwip3M7z80OSkpKS8vIbsEwwAAIDGj3DwpmZK2/nmw2e2Pzhu7PC+XYP9PJ2U+pKctPMxRw/8tj/fLCx5v7w+6YWMqROHdQvxdZIby4sL87PTk/6KK7TGdKbYbyNmurz1/L09WrqrKgqSTsVlS5Kl+NCcR8ecfmLiI4PD2wd6OsnL89Pijx1O1gshhDBn73hp7LMxEU892L+j1lUUJp3Yd049+I7WZsvFVcZlUV9MGnN+4uQn7+0V6q825ycc+X3dkiU/RWZfQR+fRXfk48cePTvxqceG92jn76YyFmacPxVfpJTqm1HpngUzlqhfejDcLiX9Wr7QAADcyvQGQ1JiUlJiUtUHrYmhv5+fn5+/n5/Gz8+/T98+Pt4+crlcCFFcXGzdvjA9LS01JTU1NS0lLbVEp7PRDAAAAGAzkqunt61raHK2bt0ihIiMPLRw8VJb11IvyfuhZXvfve3gG3c8vrbBrYG2Fh7ePWLKZCHE7Dlz9u3dZ+tyAAC4WSgUCm8vb/9mfhqNn5+fxt/f38/f30+jUSqVQojCosLU5NTUVGtamJqampqemm4wGmxdNQAAAK4jOgdxCZlPt2GdRezp82k5heUK95bd7n3x2R4qc/zRk4W3SjIIAABqYzQa0zPS0zOqN+B7eHhotVqNRuPnp9FqtR07daxclVx1H8OkpOSkpMSsrCzOSgYAAGg0CAdxCbuwh+csHK6ucr6xsJjStnz+QwynfwAA0Gjl5eVV24VQqVR6enpqtUFabaB1H8Pw8HAPDw8hhMFgyM3NTUpKSkpKsm5iaGWj2gEAAPCvEA6iKklVEL07smVYiFbjaicqCtPPndy76dvF3x/Moj8AAICmxGAwWCO/yMiDlQ+6uLoE+DdrFtDM379Zs2bNbrutu38zf5VSKYQoLi5OTU1LTUlJTUtNSUlJSkxOz0g3Go213wEAAAA3BcJBVGUpjFweMX65rcsAAAA3o6LCoqjCoqgzZyofkclk3t5ezZo18/f3DwgI9Pf379Chg7ePt0wmMxqNaWlpSUnJyclJSUnJycnJKckp7GAIAABwsyEcBAAAwFUym82ZmVmZmVl//3208kGlUunn76fVarWB2iCttnv37g+MHm1tMLTuYJho/V9y0vlz58vKymxXPgAAAAgHAQAAcE0ZDIakxKSkxKTKRxQKhZeXl3UHw6CgoND27e8aOtTOzk5UOfDEmhgmJiTkFxTYrnYAAIAmh3AQAAAA15fRaLx8B0PrEcnaIG2QVqvVavv37+/g4CCE0Ol01tNOrHEhp50AAABcV4SDAAAAsAHrEcnHjh2zvilJko+Pd0BAoFYbFBjYTBsUdHvfvk5qtRCiqLDofML5xITEhMTE8+fPJyYlVZSX27R2AACAxoNwEAAAALZnsVis2xceOXKk8kEPD49AbWBzbVBQ86A27doMHTrEzt7ebDZnZGYmnD9//nxCUlLi+fMJ6enpZrPZhsUDAADcuggHAQAAcJOydhceP3a88hEPD4/g4BDr3oW33943IOBhmUxmMBjS09Pj4uITExOTkpLj4mLz8vJsWDYAAMAthHAQAAAAt4y8vLzIyIOVexdaT0YODg62blx43333enh4iEs3LoyLi4uPP8dKZAAAgBoRDgIAAOBWdfnJyGq1Whuk1QZqtUHakODggQMG2NnbCyHy8vLi4uIq48KU5BRWIgMAAAjCQQAAADQmOp0u6nRU1Oko65symczPz69Fi+ZBQc1bNG/ep2+fUT6jZDJZRXl5QlJiwvmE8+fPx587d/7c+bKyMpsWDgAAYBuEgwAAAGi0zGZzampqamrqvn37rY/Y29sHaYNatGihba5t0bx5nz591Gq12WzOyMiIj4+Pjz9nlZ+fb9vKAQAAbgzCQQAAADQh5eXl0THR0THRlY9UPeRk0KCBjz02XpIk666FsXEXsAwZAAA0VoSDAAAAaNKqHXLi5OQU1DwoODg4JDgktH374cOGKZXKsrKy8+fPV25ZGBcbp9frbVs2AADANUE4CAAAAPyjpKSk6q6FCoXCv5l/cHBwcHBw5QknJpMpNTU1Li4+MTExKSn57NmooqJi25YNAABwdSRXT29b19DkbN26xdYlNBWz58zZt3efrasAAACNh0wm8/Hx0WqDgoNbhYQEh7Ru7e7mJi6ehhwbGxcXF5+UlJiRkWHrSgEAABqEzkHguuseanjuPxyACADA1Vvyfw6HTittXYUQQliPLsnIyKhchuzt49OyRYtWrVq2atnqzjvvHDt2jBAiPz//3Llz8fHn4uJiY2Njs7KybVo1AABArQgHbaDhvWxKpdLV1dXVzc3d3dXe3sFisViEkEmSyWj8+++jFRUV17XORiAn+6b4RryZr3nkID5YAABcvZ//sDt02tZF1CI7Kys7K+vgwQtZoVqtbtWqVcuWLVu2bNmzR4/Rox+QyWRFhUXWs01i42Lj4uKzs7JsWzMAAEAllhXfdBwcHNq0adOlS1j37t21Wq2QJJPRqFD8E+NaLJa33nr7yJEjNiwSV2TkoIqV7xbZugoAAG5hj7/p8vMuO1tXcTWUSmVQUFD70PYhwSHBwa0CAgJkMlmJTpdY5SjkpMQkW5cJAACaLjoHbyKD7hj04AOjA7WBkiQZqwSCVZNBs9mydt1aksFb1HPzArbtd7F1FQAA3DKG9yla8lKKrav4VwwGgzUBtL5pb2/fslVL61HIXcLCRtxzD1khAACwLcLBm8jZM2f9mvlLkiQuDQQrmUym6OiY1atW3/DSAAAAcA2Ul5dXPQqZrBAAANgc4eBNJC0t7duV3z7xxASZTHb5s2azubS0dPbs2Waz+cbXBgAAgGuuWlbo4ODQomWLalmhTqeLi4s7fToqLi4+Li42Ly/PtjUDAIBGhnDw5rJx48a+ffuEBIfIFfJqT0mSNHfuXL4dBAAAaKzKysqqZoVOTk6tWrUKDm4VEhIycNDAMWMekSQpLy8vNjbWerhJTExMQWGhbWsGAAC3OsLBm4vZbF757Xfvv/dutcctFvPq1T8cPXrMJlUBAADgxispKTlx4sSJEyesbzo6OjZv0dzaV3j77bePGTPGmhXGxcXFxsbFxcVHRZ3W6XS2rRkAANxyCAdvIpIkDR06dOLEJxITE1u2bCFJFxYXm0ymqDNn1qxZY9vyAAAAYEOlpaVV+wrVanXr1iGtW7dp0yZk+PBh7u7uZrM5KSk5Jjo6JjY2Ojo6MTHRZDL9mzu2bdvm7Nnoa1E7AAC4eREO3ix8fLynTp3asWPHLVu3rvpu1QcfvN8qOFghl5vN5qLi4tkfsNUgAAAA/qHT6f7+++jffx+1vunh4REcHBIc3CokJPixx8Y7OzsbjcaEhITTUVHWg02Sk5ItFkvDx9doNPPnz//9910rvl5RVFh0fSYBAABsj3DQ9qwNg08+OTE7O3vmiy9Fx0QLIRZ8umDR4kXWCz784MPCInaTAQAAQK3y8vIiIw9GRh60vqnRaNqHtg8ODg4JDh4+fJhSoSwtLU1ISIiNi4uKijp98lR+QUHdA7Zp00YIMXDggF69eq1YsWL79u1XlC0CAIBbBeGgjfn6+kRETO3YscOGDRtWr/reYDRYH09KTl61atWECRNWrlwZFRVl2yIBAABwa8nIyMjIyNj1+y4hhEKh8G/m375d+9DQ0MpDkOvdrLBN2zYmo0mhVDg5OTz//JQRI+757LOFsbGxtpgNAAC4jggHbeafhsGs7BdfnBkTE1PtgvXrN9jZ2a1fv8Em5QEAAKBxMBqNSYlJSYlJv/76q6hysElou/aVmxWmpKTExcXHxsXGxcXFRscajIYOoaEKpfWHBUmShFar/eTTT7Zs2bLqu1WlpaW2nREAALiGCAdtw1fjOzUiokOH6g2DVZnN5u+//+HG1wYAAIBGrPJgk00bNwkhvH182lw82KR371729vZ6vT4+/lzzFs2rvpdcLhdC3D1s2IB+/b5ascLakwgAABoBwsEbzdowOGnSk5kZmS+88CJLMwDAViT3Aa8vfXFI4oI7Z+2suInHBIDrKjsrKzsra9++/UIImUwWqA1s3bp1ePfucpn88ovlCoXaxXnG9OlDBg9ZsnhJckryDa8XAABcY4SDN5Svxnfa1KmhoaF1NAwCAG4Myd6/fccW3tkK6XqNadc14rvl4513vDpu1vZctvEHGgF/H3N4h0b//VuMMMZIFUqzuYespnxQJsmEEKGhbZd+vujoXz8ej/zJZNLf8CIBALeqyFPKtCyZravAJQgHb5DKhsGMjMwZM16Ii4uzdUUAYEP2QQMffW783bd30Ho5ShWF2efPHtv3yw/L1h3Pt8hDJyz6+FH15skTlkSbbF3nvyVJkiTJZNcwfQRgU+EdDCvfLbJ1FTdCTGlwdp0pqEymEEJ06z2uT99BrRxnu8kP3aDKAAC3uMffdPl5l52tq8AlCAdvBI1GM23a1Hbt2v3888+rV39vMDT6XzgDQB3kQQ9+su6dfl7yC5mZwjOgQ59mIYpj3/33uLDI3ILah/jlqxpDoFZx5LOHunxm6yoA4MoVmjoLUa1t0CITJrOQCSETQkjCopAK7GQZjrLEIkMXB1mSnZRpk1IBAMC/RDh4fVkbBp+a9GR6esaMGS/Ex8fbuiIAsDVF2GOT+3qKrF0fvzln/dHEAoPKIzC0e79OZX9kmm1dGwA0zPKNnkejHWxdxfViZ+86aryPJAkhhMViLi8r0hVn6grTdUVZpbqsEl2OrjirVJdjNlft71YKEWCjegEAt4AubcqevC/X1lWgZoSD15Gfxm/a9Ii2bWkYBIAqHAOCPOXmpC0LV+yLNQkhhD4r/uDW+INVr5G3jth4IkIIIYQ566dxg947YBCS54BXPpk6rE2Ar4udpSwn/vBvyz9ZvCG6xCKEEJJnn0mvPda/fatAfy8XR6Uoz086tvPHjxf8dDT/n73+ZB6dH5n8zNjBXVp6KkvTz/7vr0LfKrud1D2+a9hDUx8b0j00uLnGzUEqy0n87Z3xb/+SZ6lzTHnIsz9ti/Bb/9TAl/caJNeRK/Z9MFB16auh/+v1O578PssiOQXf/dTkiff0bOtjV5YZvW/Dl/OW7Unh6wZwszoa7bBtv4utq7hePDzccvQrcrJzcnJyc/PyzOYaf3XjdKPLAgAA1wfh4HUhk8mGDBny1KQn09LSZ0x/If4cDYMAcFFZekq+SRZ4x7hh/43ZkljW8Hc0urXq2jrAGq6pfdsNGP9RBx/DfS9uzrEIIfPoNHhE//aVX9WcvFr1efi1sNYOo8Z9HWMUQgjJpffr3y16PMTeul7ZThs2XCuEEBUNG9+n1+hxwyvHd/b2UVboLPWP2UCOHaes+GpaF2drrmgf2HnE8wvD/CPue31PPueYALjh8vLy9+7db+sqAADADcIBMdeev5/f7NkfPvvsM5u3bJk2fTrJIABcwnDk68X7cqSgB+Zv2PX9u88Mbetx+S+qTDEL7+vUok1oizahrW5/74BBCCEsxfs/Gj+yV/duwe06tet5zxPLT5R7DnxggPs/mxNain55ZUjnTp1ahfboO3bOjgyzU+exY7sqhRBCKDpMnDU+WFV0bNW00YNCO3TpPGjstOWHcqp0wzRg/OIdb91zW5ew4E69bn/ws4OG+sesylL48xMdQ62TahE6dNrmFIO59PQPX/+WI2s9/o0pYY5ZexY+MbxP29BuPUa/vu6cKWDklLHBNZwTCgAAAADXEOHgtSSXy++9797FSxY7OjpOnz7jm29WGo1GWxcFADcbU+LaaaOeWbgpqsSz2wMvL1y3b8fKD8Z19623l12S3Do+/OHX/91/8NCJ3d+9PdRfLhS+fl7/fCWzmIqzs4oqTGajLvXwDx+uOmWUe7Zt6y0TQsjbDL2zuaziyIIX5m08mVlq0BelHtu8enuc6YrGN+anpuSWGkwVRWmJmSWyBoxZI5n3nW99Me8er/M/vTBh7v4cqc2Ie9oqinZ+MHPZ7viCCmN51skN7yzcrZOH9A73urKXFgAAAACuEMuKrxltkHbatKktWrRcu2btmjVriAUBoHb6lD+XTf3z29ndho+f8NiYQbeNeW3FnX3fH/Pcmvja/u2UXPq9vnL5I0HKC418dtpAIYRJJqvtC5kpLT6hxBKqVjtJQgilf/NmMnPK34fTa+nru+LxGzBmzTdy7j518YIHtdnbXpn4/p/ZZiEcAlsGyGQOQxdFDl106RT8A/yE4PRPAAAAANcRnYPXgFwuHz169MKFn1nMYurzET/88APJIAA0QEXGkQ3zpozqP/rtTUlm735Tnx+kru1SyePOx+/XyvMPLn7ugV7dwoLb39bz+Q0ZdfboWfR6vUWSZJIQQkhymRDiwhvXZvx6x6yJIuiBOUueam889NnTr21NsY5vsdSyr6Bk52B3BWMDAAAAwJWjc/Df0gZpp0+b1rxFi9Wrvl+/fn0tp7kBAGpjLozasGDNA8Nntg8O9pNvP280Gi3C0dHxksxN5qXRqETpjlWLdp7VCyGEITe76ArO/dAnxaeYZc17D2i15GRMDWcAX8349Y15Gcm5+9Qv3ujvlrTumWlfn648isWQmpBqNrtufHLIG7tLGz4lAAAAALgG6By8epUNg2azJWJKxLp160gGAaB+qi5Pffji+EEdtO72ckmSO3i06H7/5JGt5RZjdlaeWViyMrItcr/BDw5uoVbI7T1a3RbqLxfm3Kwsg3DoMWpsN3+1QhIypVptfwW/4DJFb95yxqBo/9ziuZNub+VhL5fJ7V293CsTyKsZv74xq5E8Brwx97E2llOLZ8zZlWupOs5vO86ZvUa8PW/iHe01Liq5TG7vHhA6oHsQv8EDAAAAcL3xc8dVCmoeNH3atKCgIBoGAeCKKNrfMWbkhKAHJlz6sKUs9ruvfsuzCEvSnl1RER07jZq/a5QQQgjDsQ+Hj/sqedea36fcfvegN38Y9OY/72WKaehtTTHfvj2/91ezwoe+unzoq1WesLYHWnKvYvx6xqxG2XXYcH+5JHWcvv7I9H/GSFv52LB3V3zw9cDPJw2esXzwjMpnDEfnDR7zbSJfXgAAAABcT3QOXjFrw+Bnny0wGk0Rz0+lYRAAroj53Ka5n3z/S2RMWmG5yWw2lhWkRv/189JZo8fMP1BsEUKYYr+NmPnN7ticUpPJWJp77mhctiQJS94vr096YcXuU2lFFSaTsaIkPysl5vjBv+IKa9mw7zJlZ76a9J8J89fti84sqjCZjOXFOclRkTv/u+ecQYirHL/uMRvMUnxozqNjpi3d8ldsVlG5yWQoyUk8sedwsv5KBgEAAACAqyC5enrbuoZbSfPmQdOnT9dqtd9//wMNg2igkYMqVr5bJIR4bl7Atv0uti4HAIBbxvA+RUteShFCPP6my8+7bH9ED1/TAQC4Ojfb13RURedgQykUitGjRy9YsEBvMDz/PDsMAgAAADeYPGjU7E3b178azuZIEEIIIbn2e+3r5ZO7e9ay4e+/Jw+N+C3qxP43wlXX6w64GdiHjHznh4WPtOCfFjRVhIMN0rxF808++Xjs2DGrV3//8ksvp6Sk2LoiAAAAoMlxDmzfPtDNXroeUZBd14j/+/vwtrlDrl/QhGtLcu4z9b2xt7X0lF23jTjk7Yfc2Upk7vztGHt9NGoGo3Ngh8FT3/tPoNzWpQA2QThYjwsNg59+WlFRMWXK8zQMAgAAANeJ04hFZ6NPn7/sz9mFd9lfs5vIQycs/fX3755rUz0EkCRJkmSyK48GnUYsOnv28JaZ4W7V31fZ7/298VFbZ3VqSOBQa2FV2fecunbrjsOHjkRHnYw7feTE/l+3rJw365FwvybY2KYIeWzG/c1yt81dFFlssX7yHN/8bKs6X2vJqdWwN9cciD/x2QiHBtxC3m7YkOYic/e2S7PBht3rWmnQJ4YQQgj7oIFPzvtmw/8OHYk9/fepA79t/nruyw92dr+V0+4b9VKbzv84Z1mUXc/Jk+9wuZVfL+Bq0TVblxYtWkyfMS2gWcDq1RxJDAAAADQCMreg9iF++arqCUDFkc8e6vLZZlcJSgAAIABJREFU1Y4qOYQ+8cnC9MeeXB1/tS1mtRV2Cbl3cMdg/4ubddk7ewWGegWG9ho2ZtTySRMXHixq6DFdjYBT3/Hj2klRC5fvLGjArOXOLXoOfeD+Bx68q6OPUhIVDbqFInTI0CCR/v32v23ZN9igTwwh5EEPfrLunX5e8gvXKTwDOvRpFqI49t1/j4sm9HlxtYyxq5ftnLBg6JP3L935bTI/+aOpIRysmUKhGDly5Lhxj8ZEx0yZ8nxaWpqtKwIAAACaAuOpBaNGfh5vukbDyeycPd3sjcX5+aXGazRkjSwmi0vflxe8cu7Rdw/UedL9NWCMWvzwA0vPVljkDq6a4K5DJr343N0dn3h3wo7hn0Vdq9ftKtT2Ul+XD4HkOmjUnV76I4t/PteQKct87539xSs9VMKQcfKEObSTZ0Puoegw9I4gkf7db8cN/7LaK3GVL5ci7LHJfT1F1q6P35yz/mhigUHlERjavV+nsj8yb+6g63r/DW3wp6WlYM9/t2UOfXjUPSGrP4+24V8kwBZYVlyDli1bfvrpJ2PGPLJq1eqXZ80iGQQAAABuQpLngFe/3bD3r0MxUSeij+za9uXLo9o4VfZXyTxue3rB+sNH/hf55x9H/j58fPtHD/hf/PFH3jpi4wnrmuX4vW/0Vgoh5CHPro09s2/u7cp/buCgvfPZD3/6Zc+pk0dPR+7avurtkUG1rW40Hl+16JfcoHHz3hnpX9cKSMkp+J7pn2z4/cCZk0f+3vnDwuf6B1S5YU2F1cBs1BtMFovZWJqfcuL3r194Y02yWdGie1ffi/Or5y72AQMnvbd6y+4TJ07Enog8/PuGNZ+/N+nCqmjJNew/b366YvP2PSdPHI87+ddfW94Z5iHVPWZtL3VdHwKH5kOem7v2t72nT/598s8NK98eG+5d+brVWsM/HMMH91IbT+7e1bDoy5x58I/IU1uXzhxx7/R1SQ1LyxShwwcHirRd264wG6zjhbqmn7GXcgwI8pSbk7YsXLEvNqdEb9TrsuIPbv3mq13p/8xW5d93wpvfbNh59NjxuFOHj/25ZeM3n7wzqrVcCCGUfd7+Iz5qw/S2VT4K9y+Njj66cvSFl77O4q/lp80VqOuz6Mo/LcuP7dxfIAseeEetf82BRovOwUuolMqxj44dNWrUmTNnpjw3JS093dYVAQAAAKiF0a1V19YB1u321L7tBoz/qIOP4b4XN+dYhEzz4JxFL/V3kYwluZk6Se3p5qOsKDQL0eAf++3aTvpyxawebhfiCpVvcFiguqzWXMmU+ssrLzi3+HrCux8/ET3hq6jymi5y7DhlxVfTujhbx7QP7Dzi+YVh/hH3vb4n/190G5qNJrMQQiaTNeQu9m0nLVs+K9z94u6KTp4BrT0DWqr+/vrryAKTkPn0Gj1uePuLPyg6e/soK3SWusaUanmpa/0QCGH//+zdd1gU19oA8PfMbN+FXXoREVEQsDfsvcXeEhNr9BoTY4yaeE3sLdEYTbu2zyReYzR6E1M0sYuCKPbeEEGQIkU6u2zfmfP9ASgoLAuCIL6/5z7PjcPMnHfOOTu7++6Zc4Le+2HrJ8HKwoDd/HuMXdC5e8u5ExbsT+GgrBiKETRp3VLOP7x+I83GYXHc/S1T3wIAYNyCbTtC2Lx/fy94uPNYxXKD1iu/+nqsPvVhDsfU7zNx4J/RBxL0z+4gCZj2/Y+fdnB8/Nix0q1hC7eGTTTHvvgr2qZxclaCr8JuYzvrvais+rTSLcF48+od8+jgtq3sIC63ApEg9PLD5OATAQFNZs+Z4+zktGnT5qNHj1KKEzMghBBCCCH0ggmazfnn/pwn/6a6Q+93mHe0tEnfqObMukkjFsUmZeSbhUrvTu+s3ji11+ieDgf+yAa7Dv072PG3v399yqYbag6I2MXHxawrOpKLXj/q9W+jrKRE2Ibjl3wcrDRE71v92feHbqTqxY7ejZQ5mVa+I9D8KxvnfNvi909nfPvR9dfXXNI8vS/rP2nJzFay9PD1C7/87WyCQRk4cN6XS0aPmDl+e8TGGBsDe4KwIpnS1adpt4kfveHNcg+uXE3jyy+l0YSlc4NVprgDXyzf+Pf1lHyQuo9ce2xFl6dqNmT52AX/PMzlpG7u0jwz6/+vMs+5Kb30qib2ZTUB23jiko/a2xvu/rF82eaDkTkiz3Zj5q+Y12vg8k9CIz46UpgnfTqGEpeu8GnoznIRcTYOAqwEYcsBfbwg5acjtyuSG7Re+Vw19ljzlW0bIwau6D76q71dxh74ecfuPSeisp88R8v6TVo+t4ODJfHYl6s27b34INsksA+cuv23DwJsvjYrwdOiPZ6/29he1dZ7UW4Z9Vl2twQAqnkQ/4jv4uNbHwCTg+jVgo8VAwCIhMIpUyavW7cuMyNzxgczjxw5gplBhBBCCCGEajtCVM3fWr3tzzMXLt0M27F8gCcLAjcPZwYAKKUAxCUgOMBZQgCoMePBQ1sWrijE+g4Z2kxsvrl+1tJdFxNzjGaD+lH0tVinGX/df7yMcuQ//2761KguU/TOhStDNY0nfr6o5zOLxLJNhg4JEKiPr5r3Q1hsrtFiSL+1d8X6sHzWr3Owc0W+mAmazfnn/r07cZHXbp87emDr4jebyvV3f1n23zuWckthfQcNbiriorZ8tHjHxaQ8E8eZ8jOy8p+uGGrJSX6YpTNzRnVKwiMtY/WcZVV1WdtZv2HDm4rMtzbMXfn7jUc6syk34ewP/162J5U69BzW53G1PRVDiRAZJ2dHhtdlZemq62ubsPnAvp7w8MSRWxXKDZbXxFXTY9mAmc/2Qy7h9zmjpq//J1Lr1Hb0p+v/iAjZvmpie7eC4UCs/7BhQSJL5KaZn/wYfj9Tz/GcUZ2Vq69Q9VkJvkCVdBtbq7q8XlTRbllwBdmZ2ZRxcnasSL0gVBfgyEEIDAyYPWe2kyMOGEQIIYQQQqjG2bwgCbHvvnj71rENhIXZJLF3fQDgGEYAAFRzdu+JzF6DeyzccXxuTsLt61fC//ll25EYrY0f9gUN/HxYPuni2cQKLkzApfy1bEXnoG9fX7Eg/NYSbfE/ier7ejGMdMCGiwM2lDzG08uDgeyKFQRFaQ6qubxt0Sebwh4UJMqslyJw9vNh+aSzJ+9XJOdl9ZykrKoua7uoQWMvhk+6cCa+WN1qr56+Zhj7WoPG9W2qCZFERMBkqrZFhIUt+/fzhKTtITcqtEiG9con+q7V12MBAEwPT/0w+9TPX7QdNGnK2+N6txu36L99u34+7oM9sULvRl4Mn3T2ZGxl11ax+nIrXeW6jY3XW14vojcr2C0pAAA1mcwURGJRheoGoTrglU4OikSi8ePHjRo16tq164sXL83MyKjpiBBCCCGEEEI2IY59J4/0ZnMubFyydtf52Ay9wLnPon3fDSv8M808uHBi/rU3BnZs2aZ18za9Grbt2SuAGTXzYJ5tZ2cYAvD0uAEuauOoxhvLOZRmhn629M+2349evihiXfG538ochkDEUvHTwwyteZw/FflN2rxnQYdGAa5gfDz2yWopjFDAAFgsFUt5Wj9nmVVdxvbQilxrGUwGEwWRqLpyOKI2A3p7QtJ/j92u2AK6ViuKqbIea70fGtOu7F175e/vg0at/G7x0O6zP+x9aM4pygMAx1t7Kh54ALFEUnrrlPNyK/2Mlew2tqUHy+tFFe2WBzMpABGJhARMxmrLOiNUW726ycHAwIA5c+Y4ODjggEGEEEIIIYReHiwrAABgnN3dRaAL2bnheJQJAMCclaE2Ft/RkBS+85vwnQCsXcDolduW9+s5IFh28JjFYqEgk8msJhfMyfHJPOMd3Kk+eyu+goMHgeZGfLPof8Hbx86b80hGQF38nLzy73f6LwkrZW41gU2BlWCK+WXhopa/rh8896t3ro/7PspYbimC1imZPOPdLtiTuZ1k83x95UReVlUf0pa6/ciD2Ic806BDlwbsrbiiupW36dZaAqbEuIe8DZNf8VmZ2Tzj7+QkI5BX9V/kRC0H9XWHpO2HK5gbtF5RbJMZ1dhjn8bnRe79bs/oQfOCGjf2YE88fJDMM/XbtHVnbieX2u68Ji+fMvWaNFaS61nPVmn5L7dnVbrb2HJ9poTyelEFu+XBQ1oA4ujsSPiszIoP40XoJfcqzjkoEommTJm8du3aR48e4QyDCCGEEEIIvSxMZgslqjY9O3rJWD4rPd0M0g6jxrf1VAgIMEKFQvJk7APbsPfo3i3q2YsYwgoFFo3GCEAIEKDpaRmU9ej3Rr+GCgErcWzUrqnns+vBcveOHIvlhC1nb1g5oYOPg4RlhQr3Jq2aONn2BYpqzn63cneS0tOz2EAs7t7RkDjeeejytVP7BLnbi1iGlTh4Ne3ZvoEAAGwM7Cl8+uGVy35PFreesWJaoKj8Uix3Q0JTeXHr2V/PG9rUTSESOzRoP6pfYDkj8Kyfs6yqLms7F/3PP5EmYfMPv1nyegs3mUCkbND53XUrxniQ3PD9x7Nt+WpG8+PjH3Gsj693dXyhFbcZ0NcNEkJCKpobtF5R1dtjRa3fXf3vSb2beTtIWEJYqWPD9iNnjPBnqSUjPZvnYkJOxPPitnO+/veQIBeZQGjn2XLYhP5+T87Dxd26q6birtMXjGvtJmMZVmLn4iB93HvLCb7itVH29dpY1eX1oop2SwAAYufj48aY4+Me2hgFQnXGKzdyMCgwcPac2QUDBo8cOVLT4SCEEEIIIYSKe3q1YgAAy+21Q8f9Xxz38G5ULg0ImPR/R13/3X5O6J4TM7sN7r10d++lT3blogEAgDh1mLpiSWdhsZPw2YePXdICpw8PjZzVvMWor0JHAQCA+frqQRN/THwqDMud/37+fffNM5qN+GzHiM8KtlHt/lndZx0z2HIZVHPxm1V7e2953av4Zfx31bZe/zet38db+338eKv52tp+435O4LnE0gMrZ3wfzYv48rO/u20e+f7S8Ucn/BTDWS/FcOmHr/7p89WIlpPW/zWp+PVaLcTaORPLqGqdU5+ymiBm58rvum+d1/6Ndb+/sa7oOszJh5d/edSm3CCAJfrqde3EAS1buDG3Up7U0LOdh3v48+Req69WJMsnbjuglxsk/nA00upRpZfVe0PZlZ9VtT22RMcQBPUZN2JKg9FTSgZJ9TE7fjyaTYHe+u+Xu/qun9j67Q173y6+x+PRf9rTO3fe7fth04Gf/zrw8yd/L3zAlloNvgyV6TalDRsso1m/tdaLyqrPsrslAIibtwkScrFXb6gBoVdMieTggvnzayqOFyD6frS9nf2oUaOuXL26aNHizMzMmo7ouQwfMTwoILCmo6hGX6xZU9MhIIQQQgih2kUX/t3HmxSfvBEsfphqotmHF0+bmzZ76sC2fm5y1mLQ5OVkpCaev59HAQhJvXryZr22fvVUYmLMS465cmTnpvUHMigAF/PzrHn2yz4c1sHXQWTMTbx9P4OUMlyJ5l/5+u0JUVPffXtQh0BPlciSl/bgdqxaSMBgW/6K5p3+z+pD3TcOKrZJc2nNhHF3/jV1bL/goPpOctaQkxJ7/XJSQQLGxsCeLSj31Iavw3p91fudjwf9M2N/lvVS+IyQT8a/Hz3r3Td6NPdWQl7izYg4Rb8+/jy1loW0cs6yqhpcy2wC0EdumTbuwdQZ7wzr1NRTwefEXznxx6ZNv17MsPkJbu3FE+fzB3fr1cv1f7vSbH4+2hbiNoN6u0D8tsORFX2cHMB6E1dnj+Xj/vnyG9HQHu1bNvF2sxNRo/pRYtSlE3t//OlQpIYCAM07s3Li1NgPPxjfp4WPo1D/KPrc+ewmI3p4Pj6F8fb6d99TfzxzfK/m3iohNelys9ISYyPD7+tpecFXojasXK+trPaiynRLScu+XRxo7J7QCs8kgF5KAQFNRo4YWdNR1KTiWReidHJ5/I+DBw/URDwviF6v5zlu20/b68aAwQXz53ft1rWmo6hGgwcPqekQqsyI3sbtK9UA8MFar0Nn7Gs6nMqhbcbFbx3ChWzwmX9OgM/hI4QQejEGdVFv+uQhAExear8vVFzT4dSN93SEnkJcxvxwemW7C0v6TP7dxnF7tYKi96rQTYNTvxs96nsb1ra2maTr8rAfR2l+GDvw2zt1O0XEeIzbFbKodejcVrOO2DQato4jqgFfHv+u74O1I976qaKLlCOb1Lb39K7dutbtEXLlKp51eYXmHNTrdO9Nf79uZAbRq0PVK+nuvshT7+jKnwKA0c/beDduT9IIebVEQoASAkwVLC6HEEIIIYRqDOPadnC/tv6ejgoRK5A5+3ebsur9DiI+/tqtaljZozrln/r5lyhoOuGdPqoq/IQqCR7Qw5XGHz0WhemhV4vAb8K0vqrsY1v/SsKmR6+gUhIOFy9eWr9x84sPpfr8suMnAIiMvJuTk1PTsVS9CZOmlL/Ty2PWzBnBwe1rOopaRJMgTqQanwZGJyJ7ZPXzGpEZA1wplyK5Wy0//JEru31b766OMyOEEEIIoRdH3OqtNesHKYrn0yiXcuD/dke/bCkRS8z2r/e+/sPo+TP3nVt1QVMlqU1pu0G9nGncX0cwN/hqYRu+9em0puYLqzYff8mS5KgK7EpzuaWR1XQUL85494zmdk+vIP4KjRxE6GXEJUui9CDwNhRbSgwY1+zf/oiM+Saz+EbW2+gngPx4ScJL9UmGEXEuThYHSdF7MOG6TI67ujNuYcuX6jIQQgghhF4ORJR7L+xiVFK2zsxxZl124u3wXV9MGz3/WHqVTtz3QlD1mf8s2XU5LpuKq2jsoKz9gN5ONDbkOOYGXzFCofZh5PHvlvyKDxSjV9Qrt1oxQi8Zi+T2QzKssSHIBSJSC7e5t9e0FIDAR9PXwznmYeFG5wYGd5Zci5WYAIhKs2BuxkAfk5ucp0ZBbKT91p2ue+OZggycsknO7GHq9o2MPk6clJDMVOWKxR4XG2YuGpYf5GXyVHEyARg0ouvnHb/e5XCtaKkuvzFxh8aa//rM/9OrBACcWpWzPwCA2NRrcObUXvkt3DkpkLwcUVyCJORvt623WQrAKHXT3k17r6PBQQCUEk2a3colXn9mG/p3MzjYw9BOhtU3qucBaYQQQgihVxfNu7h11qStNR1GFaG54av+FV5159OdWhIcuKTqzler8am7xzbDJ4MKGKL3Lhu7t6ajQKjm4MhBhGo3TngzRsAxxpa+RT/mMua+nXVCLZtLDK91MBaNHaSBjQ0sJ7wRI+ABwMI1CjR42fNCFkQyS2C77HUrUoaoCnd1bZkzsau2mYdFIaKskHdxpEYdOPqrh7bVNXGz2Ikpy1K5ytjltdSdC7P82WdjAgAb9hcbpi15sHVSbhdvi52ICkS8k5uhfXDeYH+OAQDG/MbspE+6GlSEycoS5BhA4QjGfABeEnJGkpMvOXheUk01ihBCCCGEEEIIocdw5CBCtRy5Gy0xDtYENTYKzkgtAIybemgTiNrrHtYhZXo3td8+lygOgDU19eWJQXE1ngAA1SnWLW60KEmUoQehwtxpVPLGEZrR7SwHjhctNEzZkC0+C06KcnnezYnPs4AnAFD28Abf+aeF+Rzv0SR32dy0fv7Z4wMdl90u4zkNq/s3Gpw6txlneqj8YovL3/dE+cC7904+9n5+4VXJtP2bcfx959eXut7QAhDq4mkxGwAoG7HNt822F1CxCCGEEEIIIYQQwpGDCNV62mhZNA/1/PUuDACAbxd1K0Z8+JT9/jNi2kA9zJcCAJHpW9ejlgfSG6bCo1R+2atXxJ75Jerm1gfLO5lZADcXy5MXPIWcdFGWgXAmNiVVqKWFGzU5ArUJeI5JjnRcfVBqYS0BDS1l3ias7M8aB3UziHjJlq88d9wW5ZmBMzMZucyT2X0poQDEwRjc0CwhAJRkJAtzcfJfhBBCCCGEEELoxcLkIEK1HZcuu5QOgob6FkIAxjC8h4GPUe5PIffP2N/hjcN66yQAwkb6ZkLyIFKWzgMQrvu78Tum5vTyM7nJqVDMebtbxASYCs7TnJIk0lJQSHkbjyuxP2v086R8muJkYulHU51870UBcdIs/Dzm+s/3/1iU+mFXo7yKZpJGCCGEEEIIIYSQjTA5iFCtx0nO3hFQib5tQyoOzB1ej1w6aZ/EAZei3BdFPDrndpdT3wC9ExWcvyHmAIhSM7m3iVXLN6727TQ2sPHIgI5rVGkVX3aLmhkTBcLYOpyvxP4EBASAgzKLpYKDG3z+9aPzbxdlidTcpn3Ox3MT1na1YHoQIYQQQgghhBB6kTA5iFDtR67dkOkZc8eWhh591J4GxZ4zQh4AeOHBEwqdvWZMJ0NwMyMxyM7eJwDAqMzuQtBdc9xwQZKmIxzPZOWwxhccskWQkguMuy7Ytex9jKLwA67zV/v0/5f/oM32qdTSs7NO9uJCRAghhBBCCCGEECYHEXoZqG8qLptoQOdHH3S2ZJxRHVcXbs84rzqq5rsNSXvdn+pvKy4aAQD4XGG6GaTNc8cHmRUsAEMVMv5Frz3ESUMuiniRbvbctKGNzAohdfDQjupkED3egTX17qtp4cqJGGAFYNExRgKEUEK4LpPjru6MW9iy4mMdEUIIIYQQQgghVEG4WjFCLwGapzgayXRvrWvBi7eEyHWP/6BT/O+kaMQIfXPKhF6QFyzoQfMUey4KunXTLP1Cs/TJOUj0Cw2ZXPrT9Z8OySP8s9d/k118e+H/KbVT30/tXPwOxAsOn5NrGUP/bgYHexjaybD6hvxFRowQQgghhBBCCL2CcOQgQi8DKgi7IDVSMN9X/R5bfF4+cv24KtIC1CQLuSqgRTsf3thg7l672xmskQOLicnJFkVHy88nsS9yNWA+x/6TBd5rT8jichkLx2Q9lP99QaKjwBfETYRXL0sT8hgLD5yRTYy2++E/DeadElBeEnJGkpMvOXhe8gKDRQghhBBCCCGEXlE4chChl0Pa4QaBh0vZziU5Dxvt/NRGahDv215/3/bSTxWzx9dvT/kbzdc9gkd6lLVDufsDgCVTvmW9fEvRP136J74WbNRoGB6AZim+/kLxdSnRsRHbfNtsKz1yhBBCCCGEEEIIVS0cOYgQqhaMo25wR52/i0UhpAKJxb9N1qoxWhEVX7v/QgcwIoQQQgghhBBCtRwRiz8c4PR7Z7Go/H2rHo4cRAhVC3GT7DWfqhXFn4GmJOWU8+4EUuYxCCGEEKp2tM24+K1DuJANPvPPCfAXO4QQQqg2ICzr5yRw1BMCAECatXRYE8BEnMv+MpF/AW/WVZAclHScvXPJkIaujnZyEcsZNLnpifduX4wI+XNvWFSejeuNsk2nbPh6gmL/jCmb7uESpdUIGwu9MCKNJOy2qZW3yV3Bg5lNfSg9fdJp4yF5Ol/TkSGEEEI1QdUr6dxsTcYBn95bZRbruzL6eevj33dVfDyl/j5t1UdCgBICDP5ahxBCCNlA7K74pr2kvpRRCAlLqd7Ep+aZryYZ9t43PiznHf25ENue9vULVC1uQo6H5+zMqXxZVZAcZF0aN2/sKS78h0zl6qNy9WnRbfCU6Td3LJ63+niyDXXFqBoE+XnkiPAzSjXDxkIvTN5t51mLn54MESGEEHplaRLEiVTj08DoRGSPrI4BIDJjgCvlUiR3DdURCLmy27f17uo4M0IIIVQHsVJBgJItfNqXELmEbSxhG7tJhvnrPzuuPqWrjjLp7RvZg2/YsidR2gl95PxzPoxcVXMOWiI3vREU1Mw3qE3zLoNGvv/51ohki6rl5G+/X9TJDpNIFTJy5Mjp06cHBQYSUk01h41VZep71V+2bGnPnj3FElxaFyGEEHqJ9erZa86cOa1bt2KY6pqSm0uWROlB4G3wY59sZFyzf/sjMuabzOIbWW+jnwDy4yUJL9VDGoyIc3GyOEiKEp+E6zI57urOuIUtX6rLQAgh9JJTKZWff/55v3595XJ5FZ425mZW312Puu961O/3zH+FaQ5kU5G9dE4LUd3IBVTZnIO82WjiKAVjfmbC9dCE62GHTszbtu1fTSbMn7hn1Oa7HBCnngu+mT2wiZebvZjqM2MvH936zca997RPfjdl/Wf9fXNWwdnSf53Y+7OzZhuOqnPs7e2HDh0ydOiQrOzsE8dPhIefjI9PqNoiqqux5I0Hvztj6pCOAa5i/aN7EXu/X/tD+ENz1cZeuzACJjg4ODg42GwynT9/ITTs5LVrV83mOn3NCCGEUF0kk0n79evbr19fjUYTdjIs/GT4vXvRlFbpR06L5PZDMqyxIcgFIlILt7m317QUgMBH09fDOeZh4UbnBgZ3llyLlZgAiEqzYG7GQB+Tm5ynRkFspP3Wna5745mCyJRNcmYPU7dvZPRx4qSEZKYqVyz2uNgwc9Gw/CAvk6eKkwnAoBFdP+/49S6Ha+rC8/uNiTs01vzXZ/6fXiUA4NSqnP0BAMSmXoMzp/bKb+HOSYHk5YjiEiQhf7ttvc1SAEapm/Zu2nsdDQ4CoJRo0uxWLvH6M9vQv5vBwR6GdjKsvlGVX88QQgghKwjDtG7dqnXrVh9++OHly5dDQ0MvXrxkMpme87SUBzMFCmAwcjHJunX5xG+worGLyJuYUp2kUwIlLR0F9WSMBGiOxvDdcXW4AYhQ0LupfIyPqJGUGPSWy7HaLXeMaUUTbTES4bDm8uH1Rd4S0GstVx/xzsWGa/k0c/ypJXs0LHNNStGnEQHbNUDxpq/IX04YjqblGHeeVx/TFFyzYPJgt8kAAMDr9R/tVV+t4HRe1bYgCc07v+HL3wdsneQ3cHDA93fvcGBRNWrj71Uw0lHhFthz0rpmrubh/96fafVTV+WOesmZLWahQOjk6Dhy1MgxY95ITUkNDQs7efJkSkpKtZRXJY0laz7zvz/OaW1X8Gu7pH7LoR+ub+U5a/ji8Jy63FaFhCJR586dunXvZjAYzp87H37q9JUrlzkOfydHCCGEXhoyogOwAAAgAElEQVQWi0UgENjZ2Q0aOHDY0GE52TmnIk6fOH4iNja2agrghDdjBJy/saUvD6kMAABj7ttZJ9SyuTLDax2MPzwUcwAANLCxgeWEN2IEPACxcI0CDV5CAACQWQLbZa9rbDHP9tqfCwDg2jJnYldD0Qd66uJIjTpw9FcPbft4I8hVxi6vpbZqwI9a5BRd2meT8vcXG6YtSZjfjCuappA6uRmc3Iyiu87bbrMcY35jdtInbTnCMVlZDJFxKkcw5gPwkpAzkqF94OD5ujGoAiGE0EuGZdn27dsHBwdbOO7ihQvHj4devXrFYqmiaQIJPE7lOblLRzYQFr2TEkcZmMwAAuHbvR2muJCCJIlYIezTUhUkz5123pgHQESimX1Vr6sKnxgV2Ql72QEAlJnCZAVv9XJ4363oAQeWNHBmy5vDuAKqc7Vi/Y2TF/ImjKoX6CeHO2qqObNu0ohFsUkZ+Wah0rvTO6s3Tu01uqfDgT+yC3NHXPT6Ua9/G1XiM0v5R9VpQoEAADw8Pd56883x48clJycfOxYSGhqanZ1dxSU9b2Ox/lOXzGwlSw9fv/DL384mGJSBA+d9uWT0iJnjt0dsjHklcmSsQAAAEomka7cuPXv1VKvVJ8NPnj4dcTfybk2HhhBCCKEKEAiEAODg6DB40MDhw4YV/kwbFpaSmlrusVaRu9ES42BNUGOj4IzUAsC4qYc2gai97mEdUqZ3U/vtc4niAFhTU1+eGBRX4wkAUJ1i3eJGi5JEGXoQKsydRiVvHKEZ3c5y4HjRQsOUDdnis+CkKJfn3Zz4PAt4AgBlD2/wnX9amM/xHk1yl81N6+efPT7QcdntMqaQsbp/o8Gpc5txpofKL7a4/H1PlA+8e+/kY+/nF16VTNu/Gcffd359qesNLQChLp4WswGAshHbfNtse746QwghhJ5DwWwhQoGgY8cOnTt30et1F85fOH7ixI0bNs3n9yxCiEzMeDuJh7WUN2YgJ9OcSMEdAIBGXMhe+4BTU+IsIxoOfJvZTXIhWcn5667qr2ionYPkvc52r/nKh0cZd+RCkyC7USqSn6n79pI2IoeyMkEnP8XMIJGijHLr+9u/48YYc/WbL2nDMnkDSzyVTN7juYmpZfuhrP/W7IIkZbNkZ+dRYidTyBhQ84Somr/1yaKOQQ08HIXa1EyeBYGbhzMD2dZSR5U7qs5hBSwA1POsN+ntSZMnvx0dfU8oeM7pJp/yfI3FNhk6JECgPr5q3g9heRQA0m/tXbG+64Dv+nQOdt4Y86hKQ63tCr5R2NvbDxo4aNjQYVnZ2Q9jQ/TcX1I2vqZDQwghhFAFFLyne3h6vPnWm+PHj3vw4EFa/GETPSAiWZU7oTZaFs1rmvrrXRhpKg++XdStGPF/TtmHWLLee0s9zNc5KoYQmb51PWqJkd4oGjyg8sv+5B1tkKfZUcCk5hAWwM3FwoCg8FMZhZx0UZaBALApqUUzF1LQ5AjUJgBgkiMdVx/M6/W2IaChhbktLP0xIyv7s8ZB3QwiXvKfrzx3xBfkFpmMXObJ7/SUUADiYAxuaL53R2igJCNZWLn6QQghhKoJywoAQCaTde/erVfvXrk5OTFxFXsywL+VU3irElvMGsOGm8bCBB2leVoux0IB6CMNABH28RGyJsOmM9pzJgCArCz9f26KuncTt3Vjf8ljutcXMJxpW4QmpOC3tnzziXvGoYGipqWWTQR9GgpFnHnLKfW+gkk/OPogo4JPDltVrclBgaOjklBep9VRYt998fatYxsIC3+tFHvXBwCOYawGULmjytC1W9eD3Q5U4sAXLD4+vsy/EWAJAwD+TQIe/+xrZ2ev0ajLPMRWz9dYovq+XgwjHbDh4oANJf7AeXp5AFQmOXjw4EvQWNYJBAIAcHJ0dHJ882r+mwr2nqvHLoCkmo4LIYQQeslc0fw5ba7XtLnVW4qV+UAELAsADX18GjaccVn9noPwrNxuN0CF1xLm0mWX0qFlQ30LIaSaDcN7GPgY1/0p5OEZ+ztj0of11q2PkfON9M2E5EGkLJ0HIFz3d+O3vmYq+lTGebsDAGEquIRcSpJISw0KKW/jcSX2Z41+npRPU5xMLP1oqpPvvSjo1U2z8HPNXLXo9j15eLjjtjPiujxHOEIIocqKN3w4be6E6n5Pt6LgmT+Vg0P7tu0KttSTmG5pZDYezvNUb+JT88w3kg37YozxZa04wLL1FcAIJMvHSJaX/IurgmEYpp4c+HzzTa1tpTKsjz3w+aarGhvDrLDqTA5KW/bsoGT4hKgYLTgOnzzSm825sHHJ2l3nYzP0Auc+i/Z9N8z6CYhj30ocVZaoqKi9+/ZV7tgXqX279vW86pX1V0ppwdzYeeo8lcoBAKoiM/jcjVXmfN1ELBVXLqIv1qyp3IEvkouLyztTp1rZoWD2InVuajP3/S7Co+mpAGD/oqJDCCGE6ghfydfr9zheul2N49HatG7Tp2+fsv5KgfI8ZQhJT4ns3Gi/s/CEVmNXmfd0TnL2jmBqL33bhvQkmzu8Hrn0o30SB3yKcl9UxpLOud1/kSUG6J2o4MgNMQdAlJrJvU2sWr5xk9uuW+IMA3XukLZvXl5Fi6VmxkSBMLam60rsT0BAADgoM3VKBQc3+ORH5Q5soWvTRN+mfU7bdpoA4jvztADTgwghhJ7iIjz8858PqvU9XS6Xz/rwQys7cBaOFbAajdrOzh4Akg02PZcZfT1r2m2LraP1KJT1JihmCSGFExEyNp4NCqcmrL431mpLDhJlx5mfvFGPsUQfPXiXYxq7u4tAF7Jzw/EoEwCAOStDbXyyN7VYLBRkMlmJHyQZZ+tHVUxmRmbE6YjKHv3iNPJt9OxGSinH8wKWjY+PPxYScjr81PTp07t261o1RT5/Y5mT45N5Xvn3O/2XhOmqJqiXorEa+DSA0pKDFotZIBDm5uWFh4dHRET4u10buLIgh+v1giNECCGE6gAH4dkH9+wjTlfyF0dbKO3s+/Tp/ez2gt/5UlNSw8JOhoaGdgxKGFX4nm5XqXLItRsyfR9Nx5aGHm5qT4PiqzNCHgB44cETirkfasZ0MpxqZiQG+7P3CQAwKrO7EHTnHDdckJgAAEhWDlvpD8OVZBGk5ALjrgt2hdtpZexjFIUfcA0/AMByAX1St01X9+ysk522t3E8BEIIoVeHnL3/4F56tb6nOzg4QGm5QY6zMIxAp9edPnX6ROgJJyen+Z9+Wl1B8FyyFniRfv7f6nPPLhtChIn5wNiLOipJVK4NGT+eS9YCoxC1sYN7Tw8PoxZKAYj0+dJ7VZYcZARClgDHiOQOHo2bdxoyYcqELl5i84Oda3bc5QCy0tPN4N9h1Pi2936/kZrPCxQKiQCg6MMNTU/LoGzTfm/02x0dkmix92nmob92J7Wco14JhR9JU1PDwk6GhYalpj3nTNgA1dJY946GxL03fejytfHM5oOX7mfkc0KlR6OWHvkRlxKqbv2c2o7jLCwr0OsN58+dK5jitGBIpb9bTUeGEEIIoYoo+ACWlZ0dFhp6PORE0sOiWUGCnvfM6puKyyZ1586PPnCzZJxWHS/6iJ9xXnX0bc3QIWmunlR/XXHRCADA5wrTzeDfPHd8kOT3e8J8ShUy/kV/GOakIRdFk4fqZs9Ny9jiFJYoEDrrBnQqNsqCNfXuZcy8KYvKZDkBWHSMkQAhlBCuy9sJG/rAH181WH2DtVICQgghVH04i4VhWaPReP7c+fBTp69cuVwwkUiVDbcqFTWHJ1nGNpPM6cIxtww38jgdT+zkgkAZfzmds1Dz8Xjz2FbCiT3sDZe0Rx5Z1DyxkzKSss92MtEytrlwSnd7/SVtWCanocRJKbDTm+MMkKXjeSLs2ljyd64hhWe8nFhDhvlRBQcZVlVyUBA08897M0vEzuXe+nnx3FVn1RQAskL3nJjZbXDvpbt7L32yDxdd9B+J4aGRs5q3GPVV6CgAADBfXz1o4o9J1o+qs1iGLRjmmp6RceL48VPhpxKTqnCiumpprK3/XbWt1/9N6/fx1n4fPz7GfG1tv3E/J1TlLJm1Ec9ThiEmk+nsmbNhJ09eu3bNyqRFCCGEEKqdWJa1cBYBK8hTq0NDQ8NPhsfExFR5KTRPcTSS6d5a14IXbwmRP3niQqf430nRiBH65pQJvSAvGEZA8xR7Lgq6ddMs/UJT7FMZebEfhsmlP13/6ZA8wj97/TfZxbcX/p9SO/X91M7Fv1XwgsPn5FrG0L+bwcEehnYyrL4hf5ERI4QQQgVzsvE8f+HihdDQsCuXrpgtZU0QWC2i72h+r6d6q75iTf0nSxCbMzQTj+mSKTyIUv/o4TDdTfJBb8kHxY4yPXsiAACIidT8z1M1wUk6t5+0aMJGeuJUxvJEmpxsjGkhDGyk3N1ICQDAmzftz/61grMTVkFykMu4fys2sKGrg71MzPKG/LzMxOjbl88c+/2PE5G5RSkSmn148bS5abOnDmzr5yZnLQZNXk5GauL5+3kF2Uwu5udZ8+yXfTisg6+DyJibePt+BiHlHlVXqTWa8LCT4eGn7kXfq9ozV19jUc2lNRPG3fnX1LH9goPqO8lZQ05K7PXLSWX17DrDYjFfvXI1NOzkhQsXTKY6f7kIIYRQnaXT6SJOR4SdPHnnzh2er7bfNqkg7ILU2ErL3lf9Hlt8Rh1y/bgqckh6M04WcrVotj4qOLyxwdzM9KlddX6OHMsxmnxBRqbofBL7Ij8M8zn2nyxgo8dlvNHW4G0HeanSiGSuX7CxoI4IEV69LK0XaKxnxxMzm5wgO3LQdf0pAQUSckYytA8cPF/mSAiEEEKoOnAcd+PGjbDQsHPnz+v1+hqJgZpN/3csOzpIPqy+yM+OkRKap7VEpnOFGUqL5X+h2bFN5G81FAXaszJC9UY+RW2JTLaU+vAlNZt+PJ4dGyQf1UDUWM4IKZ+pNieYgADwubqVZ8isFtLWSkbA8SlZluzSzmAdUTq5PP5HwfqwFy9eWr9xc2Uuvbb6ZcdPABBxOuKlWOPCwcEhLy+v3I+kC+bPLxgEO2HSlBcS1wsya+aM4OD2ADB48JCajqV8YolEIBBo8/Ot7zait3H7SjUAfLDW69CZV3RBEmKvWbwwvX+qS9//2L9SMwPUFNZJ+96kjDdaG7zsqCFb8cWi+rvLmijKNtiCVcuzZ8qWt/TXvvdddq2Ca46+erDvveIGdVFv+uQhAExear8vtDrnHLRX6vQ6s7mcMQX4nl7ApX/i6RnaC5v8JofgqiMIIYRs8sLe04VCoVQmVeeVs3Zr125dF8yfDwC70lxsX624DhjvntHcTgclsy42L42CXpScnJxq/LEaVSmjwVBuZhAVICJzkJ/RRQKYCKkGtM24B1d33/+yk6WweoX62UsS/91T56PkBQxVKKkpH1jv7F0/R52Zn+tdqRs/tmCRZ2q7UuSuhkA3i6Tu1GbVVEupsO+hFyNPnVduZvCVxTjqBnfU+btYFEIqkFj822StGqMVUfG1+y90ACNCCCFkC7PZXG5mED2l2lYrRgi9EPIeSVc+zo/Z5Ttij/iFzHRImw5P+nowv391g03x1fRV/fmLeAFB1i4EKCHAFF2ruHnO2AbUlOjw73WuIcmMUMmBFsCJMgBMrfxJSN4j6cpH2vt760/YIS+5WhftPjP6pz7sj582WhNdW5ryqdquIbWuk9dItUhapO98N6+hI2cn5VmO0WgEifHSi9fs/wy1i7L1h5taV5MI1ULiJtlrPlUrir9EKEk55bw7AV81CCGEUF2AyUGEUMWoPAx+LqyoOr8OPH8RLyDI2oRc2e3beveTf7vVNykJOb3P9WAiSwGM2QIAgASnsW871VSI5SN80xEP12f6vHNQXLsn73y6tmtKLevkNVMtrIOxeX1z4UMpDK9yNKkcTS3a5E15Q7pjo9fqC8JSZ2x5Si2rSYRqI5FGEnbb1Mrb5K7gwcymPpSePum08ZA8HZ91QQghhOoETA4ihFC1YESckx21aNkcw4vOOojFPKFMZi5Te572sqU2OMp1nfJwwUOflTfwOTVkOxL5W8PXf5MYKJXbmxv7awcPypzYSj95XiJZ6bPyJvYlhKpA3m3nWYudazoKhBBCCFUXTA4iVKc4tcpcNCw/yMvkqeJkAjBoRNfPO369y+Fa0ZQLji0y5w/Nb+Ztqu/ISQnJyZBcOKf6fq/qVuFK57TL9JgdA9iNH/t++6AwiaPsnXRxdv65Df5Tjhd9zWaMs767OwsAAPhsh4nTPM4+Mz6HUerHjskY31Hnq6K6DOm5W6xbsQdaiUqzYG7GQB+Tm5ynRkFspP3Wna5744sls0orovyjSkTw1BncTZPv/zqEP7TWb9bZolAYw8f/efCBSvnuO57XmlivGQAAIjUOHp0xtbs2wJHqs8QRoS5r/1Q8LOXaddPeTXuvo8FBAJQSTZrdyiVef2aUW/k2FCE29R2e8U4PbTM3jhjZ5Hi7zRs99qWC35i4Q2PNf33m/+nVotQbw41ZendMwX9b5Mvea7DTkrt9a0qHa/WCVynVNpRlvQWLs94uZdXGs6e5cdAlrXP6xI9Sb87z2lvKDmBL/yzxEhCCOl168rDL99HmEYNy+zc3eNmBNl1y7B+3NYelj59ftlIPyiY5s4ep2zcy+jhxUkIyU5UrFrvfH/Dg6douo13KQah/r9SfJmnbeVlYvTDyhnLrbqcjKYyNVwpQ/iuxtPg9DudVsptZr6tinRDaTY39dajl6Ff+H0QUdRpieXNZzBfNZMunN9iRWWV9rwBvISYOKJD8XNH1i6Lrl+xPvJ2wbYRhwr+y93zscpev7A3Httc7QgghhBBCLztMDiJUpzj6q4e2NTx+YctVxi6vpbZqwI9a5BTNAQA4BahHBj/egTp76gaP1vXtrJuzyPNIVpWFQeTaxZ8nTvamBVkNsYdukAcAwJOVRi1co0CDlxAAAGSWwHbZ6xpbzLO99udaPW/ljioK6vZVefbg3PYt9KKz8oIHVxknXbAnNV6TXTWBc7k1I9HPXJ4wJ4AvSFNI3PVDxya1cvUavtEup3h6kjG/MTvpk7Yc4ZisLIbIOJUjGPMBbKl860WIDNOWJMxvzhXmSYSWxk1Mikqv3mq1rPJbsDgr7VJ2bTyLS1cu+JpruDxr5dzMe8ucIyt1aU+9BBzcdSOnJIwstoPIQ/fmtERHfaPpYQK+vHpwbZkzseuTJnNxpEbdMyMfK90uhG/VvajvCk1tu2W0bqlbsdB7R1KVDTUtLf7n6GY2vgSA3LoizxyS076lThxRVBNSXVd/yj1QnM6p0r5XKsqe/5/b750TJjVQD27ofDeWVObWYevFIoQQQggh9NLD5CBCdQ5lD2/wnX9amM/xHk1yl81N6+efPT7Qcdlt8niHQ981mndaYCC8RyPN25PTpgblfj5ZceEbe1u/9PLi9cUGND0bQbORaZPqU/U9x2U/Ooc8YAWOhl6D0hcP19o93kOnWLe40aIkUYYehApzp1HJG0doRrezHDguoGUXUf5R1oO8Yxeuzh3ZRtNSIL9kAQBQBOiaseTOLVkeBedyaob6D02b2YSmX3Fd+JPD2RSi9FXPm506ulfG+L/tNiY+KYTItP2bcfx959eXut7QAhDq4mkxG560TqWLaDgo7eNmnCFBtfoH50PRQr2Q865vySkru8Gze4oPbQMgqhJ1abWs8lvQxnYB67XxjPxI1zm/GH6fkvHtBOnr2+SaymVhCl8CAi3lm/RI/X6G2lMn37zRbddNcRa1BI9K3vy6rkcftWu4YxpvQ7NSNmSLz4KTolyed3Pi8yzgWbK0irVLiTjJgzOun+1Rnk9mxS66kRNS53fRzns778gqVXrVvBJLjZ/6j6hcN7P1JQAAxijF2fycYS20zVnFZQ4AQBqY31FKYq/JEznqP6rK+l6ZjLKTt9gJfUyB3jzEshW/4VD/kbZeLEIIIYQQQi+7WrluJUKoSMBbcff/jnxQ8L+9sf9uZEPOgIImR6A2Ac8xyZGOqw9KLawloKGFKbZDvprVccBbmOR7yi9Wef6dA44d8noqqihoxjigo4kxyb772v3vGIHOQtTp0v0H7O6XnLZc5Ze9ekXsmV+ibm59sLyTmQVwc7GUe0uq3FGFjPJDlwXEOb9f44JqpM2b6aS8+NQ1UWFoVmqGMQ7tbhBo7VZ94xyWxBo5Jj1GteJ/dvmMsXMzc4kAKKEAxMEY3NAsIQCUZCQLn6zAW+kiWOOQHnqxRbr+S49dt0U5JmLQCqKjJBmVmwzeelm2tWBxZbaL9dooBYk+6LnyEtt4SMqidlwlR9AVvgQIZ2Yjw1x3xRLCspHXJWk6YtYLz/zhFJIPrLvJm9jWrBRy0kVZBsKZ2JRUofap4J+nXShz6YRjWIJAbyG5qfKf1nv8mg7yFppuFUuDlV8bJeInle1mtr8EAMAgP3yVJa6aPoW3LNqmvdYBxEfOibmq7ntlyVazlIBMVjj0r2K3jgpdLEIIIYQQQi85HDmIUB2XkiTSUoNCypc5zC9ffuIuM6KjqZErBU0ZO1WI0OTjSvl02eXSp40DIFz3d+O3vmYSFsbEebsDAGGsp4Iqd1QJzLlw+9Re2QO66NdFycyMoXMzjqaoTj4sfe8SNfPQ5OtGGbF6w67IDSV383SzMCB8nLugOvnei4Je3TQLP9fMVYtu35OHhztuOyN+OqlU0SJYs58n5dPkZ1Or4plTodWyhOZyWrA4q+1SodooxAn/2uzR+auk199PC4/x1FbuAoudLSGDgK/FzQ6gYMSiWfgwmxBHXkoABOU0a/nnZ41V1i5G6YUYZmJnk48LBXX5u1eS9aa30s2sH/h0XTFnTtlldc/r28HwVbSUE+r6tTPTBw4HE0lV9j2rHO05QkGnZ2glbh3lXSwu0IoQQgghhOoSTA4iVKtF/erb+NfnOgM1MyYKxOq6tZQHACgc4QMAlEpEz1Vowciasr56E6Vmcm8Tq5Zv3OS265Y4w0CdO6Ttm5dn/ZyVO+ophjuqfSk50zup2+2UXXTTdvOgD/+xu1v2F/2na6Y0YnHJxCsVHNzgkx+VO7CFrk0TfZv2OW3baQKI78zTpd9vbS2CQEXSoOWzfjnWW7C4ctql7Nqw0iNprt1nmx3aLslZPk2xruQzyJXon2YzAKHCx9VPiJkDIIXXaGuzlqVK24UwBacEqKJXYqkq3c0qVFe628ojWbnjOqtb7JZGBqr7O5IbB+zjOABBlfU9a8S6ns05hoqjEhlQ5lXi1vG8HQMhhBBCCKGXByYHEXrlifXBjXkwi+LTCQDV5LOUMTepz5F7padvLByhlMokZZ/QLIp9RBjP/J71XW4llPI9mlGZ3YWgO+e44YLEBABAsnLY4qsNlFpEuUfZFCQn+f2o9J0p6pEtXJM88psQ0U9nJWWuPlq8Zsyi+HTCK5TvTPcMK3vKvEJGUfgB1/ADACwX0Cd123R1z8462Wn75yqCMcSnE8Zd28md3kp57uxEeWVZb8ES+5bbLmXUhvUhgbnX3BYd1m5/7dGcbFo8gnL7Z8VUqFnLPkOVtAtRaPsE8mAWxVXVK7HsgCvTzSpaV0bZ7yfF40aphwc6q3poXE3y78JFnA0B2N73ykS4jm89esMVLAn2B+MI413xG87zdwyEEEIIIYReHjhzDkKvHsIH983u5W2RstTeXfv2rNSxbqC9ZXc6HwBIXIxETfmub6SNCzDLWGDFnIt98ZEyJD1TQFlTv36ahjLKirlGQXpPtuT5efH+UxIza/hgQfK0NkZHMTAMVTpwsqKz8LnCdDNIm+eODzIrWACGKmS8oLwiyjuqxBVaCTLxtMNJg2VA/5wRwXo22f7g/eIXV3bN8OKj50S8Km/5R5l9fM32QmAY6uCm79nU9HQMrKl3X00LV07EACsAi44xEiCkKMNV6SJ48ZGzIk6gnz0/dUJzk4OYsgLe3UfXRAWVUV5Z1luwxJmst4v12rCCMmd3eex+xHm6lOh+5fXPKq0H285Q+XYhYKeyyAXAsLyHf96ChSnDHSD7ojKsql6JlbhkK5dT4boikSdU13nzkOFpb3ey5F10OJJrUwC2973HGJayBIClcpWpZfucRcvifhppkFhEu7c53uUrdcMhVoMkXJfJcVd3xi1sydlQ4wghhBBCCNV2OHIQoVcPoT5dHm3r8ujxBl4jX/OzsmCBVO01x50PNB82Un/+pfrzYsc8/q/EK3aR4/Qt+jwM7QMAABbp6pkNf0wtUUD0Px5ftUyY30y9cJl6YbE/FIzWoXmKPRcF3bppln6hWVr8KOtFpJVzVHFWgqS5djtPCfsOSP+AQtRvysjizxRbqxlye5/7tvZJ0zqmb+2Y/ngHc5RbvwVOCcVOQpTaqe+ndi5+c+UFh8/Jtc9bBLmzz+P7tokzGud+9nnuZwV/o8z+L5vMOleJ5Fg5ZVlvweKst2Y5tWEV1cq/2arsvSjXq9j1lds/K8jWZrVyhrLbBZqOj/t7jOnsJr+3j5U2+o9wA2fHDJz9ZIMxVbXkp8JFw6vilViJS7bWzSpaV1ya8pcrmV93zOvBibYeVqipTQHY3veK0KC34u69VbJojfTnjV6rbrAUACp1w9lqJUhi6N/N4GAPQzsZVt+QlxkXQgghhBBCLwkcOYjQq4cyNyJUpxMFOo4Y8oU3zjp/OL/+T4lFSQeTdP1K71XHZQ/yGI4Hi5HJTBdfvWIfnkQKvtpziU6zvnUKSxToeLAYBHFRklJWDjBKflzpO2WHKiJeqDYRjiOaXFHkbbs/r4jNAEAFhzc2mLvX7nYGa+TAYmJyskXR0fLzSay1Iso7qjirQTLnDznc5anYIvs9TFxi5I/VmqFa+ZpFDef8pjyfKFAbCWdhMlOl4ZEiU8miCRFevSxNyGMsPHBGNjHa7of/NJh3qig39BxFUJ3s6yUNZ/1mfzlFoDUTs4FNipPFam0YhVeacgalfCoAACAASURBVC7HeguWOJG1dimnNsqTd9V1dYSgRN6pvP5ZxfVgyxnKbheWBaAkX1fKlJ9Z9+z3X5HGZLA6M+E4JjtVemSv55hPPA9n2XqlNr0SK37JVi6nwnVFBUcPKlM4MMY47IomNgZQgb4HwOWIbyUJs7SMmQPewqhzRbdv2G/f7jXs/YYrzgstRWFU4oZjLUheEnJGkpMvOXi+og91I4QQQgghVBsRpZPL438cPHgAAC5evLR+4+aaC6nq/bLjJwCIOB3xxZo1NR1LlVkwf37Xbl0BYMKkKTUdS1WaNXNGcHB7ABg8eEhNx1JlRvQ2bl+pBoAP1nodOlPaxHMvkN+YuENjzX995v/p1Vd4Vn1Wv3BT/Lgkjx6rVVlFaZsXUDNY+a8Y2vej6B+6ib6a1XBzGStiI4TKNaiLetMnDwFg8lL7faHimg6ndr2nI4QQQi+R2vae3rVb1wXz5wPArjSXWxpZTYfz4ox3z2hup4OSWRccOYgQelXYO1jkLIjkpl5jH41xY48dt8uugiUtECoDa2jnz1vi7I6W/6gvQgihQg36JP+zJW5hs1frHZp10s74KD5sR1TM3ru3/ps0zv15T0jsNUvWxJ6era75L9/V46l+Uqu6je2V79kz5Z8tsSta14qwa7k636WfU616CaCXVClzDjZu3HjWzBkvPpTq5uLiWtMhVIs61liNGzeu6RBQHcWYR39yf2kQBQCgkHnJ89vLpTyPjFBVYRzMSr3o0H5VHK5agRCqFrTNuPitQ7iQDT7zz1ViAffnPLyaUDs3Q5Abd70uj7B/puaF+tlLEmc2LJwnRKGkpnxgvbN3fJbuc9d97FpVok1z4JZAROYgP6NLTuUn5a3dnuontavb2F75cldDoJvlbu0IuypU412l6ro03jlRKboq1S1smhS9jvCWljKbdynJQUdHh4LnOuuYJgH+S5Ys2bRpU3Z2dk3HUpXqZGMhVA04MWV1HAdq0YVw5y92K5MwZYOqE59pv+BjfOQQIVQRrHHmuriP60u+meezMf7ZL3l8p2lxOwebw//jPzWMBQAClBBgKvtt8DkPR5X2VM2Lm+eMbUBNiQ7/XucakswIlRxoAZwoA8DUzae8aNPhSV8P5vevbrCplH6ObFfravKluKvgnRM9q0FpybJXzSu0WvHNGzfr1/fa8n+bt/20/ciRIzUdDkI1I2aPr9+emg6iRvCSLQv9t5T99xdQM69u5SOEELIFsbg5UCLSvzMh74/VqrSS48VY7+xP+plYQhwcOBZYDsiV3b6td1e+sOc7HFXa0zXvVt+kJOT0PteDiSwFMGYLAAASnMa+7VRTIVY3lYfBz4UVYXrludWymnwp7ip450SodCWSg3VpCYhSiUSi8ePHffDBjE6dOm7YsDEzM7OmI6q8L9asgbqzvApCCCGE0CtPZHFTgEnDsm0ypwUoP4ss9o2fcP3fyG5uZnOFnKM9ZyUVwIg4Jztq0bI5BmLL9leBLddeg/UjFvOEMpm5pSxtX1Ne5d6CXk1453zWq3DtEacjBp+u40kw271CIwcBwGQy/fTT9gvnL8yeM3vz5k3btv109OhRSmvPGzFCCCGEEHpFMUqLEwPpES77Ax5NHJP735UOKUWDBwXe2TM68ed2uORPeNTZwVLwsKnfmLhDY81/feb/6VUCAIxSN+3dtPc6GhwEQCnRpNmtXOL1Z0aZ25863KlV5qJh+UFeJk8VJxOAQSO6ft7x610O19TFQhSbeg3OnNorv4U7JwWSlyOKS5CE/O229XbxaXxpu6mxvw61HP3K/4OIosdiieXNZTFfNJMtn95gp0WzYG7GQB+Tm5ynRkFspP3Wna5740tNjdEu02N2DGA3fuz77YPCb6fK3kkXZ+ef2+A/5XhhoURqHDw6Y2p3bYAj1WeJI0Jd1v6peGgBK3VSstpL38exReb8ofnNvE31HTkpITkZkgvnVN/vVd3SPDnWStEF1dV3eMY7PbTN3DhiZJPj7TZv9NiX+nTNAwAw3Jild8cU/LdFvuy9Bjstudu3pnS4Vi94lVJtQ1mMUj92TMb4jjpfFdVlSM/dYt3KeCSZqKzVvy01ZmNX6T8iY1rP/CAXnleLr1xUbf7N4WJOsRQDY5z13d1ZAADAZztMnOZx1lKiFOtxlq/MAGjzCXF7XzeHfOP//qnCOnIfmBD+rmHf5/6fXimIkDafELd3NPfX536fXCFVWPmlINS/V+pPk7TtvCysXhh5Q7l1t9ORlILjber/5daksknO7GHq9o2MPk6clJDMVOWKxR6H8yrZe63XRrG+DdbvAzsyq6xWS3RIIajTpScPu3wfbR4xKLd/c4OXHWjTJcf+cVtzWJpLnwoS75yVv3OiuufVSg4WiLx798MPZxUMIezcudP6DRszM7CnI4QQQgihmkTsLA4Myc2W79inGDcr618Bqs8LBg8Srs/onIBc5eRQycBRILHn5ARMT30jZMxvzE76pC1HOCYriyEyTuUIxvyytz/D0V89tK3h8XcDucrY5bXUVg34UYucogtm6RUbpi1JmN+MK5psizq5GZzcjKK7zttus8Um8iW3rsgzh+S0b6kTRygKp3GS6rr6U+6B4nQOgJRrFGjwEgIAgMwS2C57XWOLebbX/txK1ZpEP3N5wpwAvuDLtMRdP3RsUitXr+Eb7XKIDddedv04BahHBj+uEOrsqRs8Wte3s27OIs8jWeUVTQFEhmlLEuY35wq/5QstjZuYFJWe1cpqWUSuXfx54mTvwiVNxB66QR4AAKWXZim7/m3rLbZ0lfeWJnzSrOjanQw9BqZ1bqufu7Deftu/dVmJs1zWAiD3bsoyRue0DNALT8nNAAB866YGIcO3CjCyVyQcADBcqyYmxqg4E0UAqrTyn0X4Vt2LLkloatsto3VL3YqF3juSqmyslmvLnIldn/RkF0dq1D1H77V+YLELK+c+UHW1+lSHdHDXjZySMLLYDiIP3ZvTEh31jaaHCZ5e3QfvnAX/quidE9VFr2JyEIqGEJ4/f372nNmbN23EIYQIIYQQQqhmMXacCiBNw6afdfrjrYTXh6u33FVmUmDdc9/pyN361em8juusJYy9xYmBnJLLahGZtn8zjr/v/PpS1xtaAEJdPC1mAxB56dtLR9nDG3znnxbmc7xHk9xlc9P6+WePD3RcdpsAQKPBqXObcaaHyi+2uPx9T5QPvHvv5GPvl/KV0RilOJufM6yFtjmruMwBAEgD8ztKSew1eSIHVKdYt7jRoiRRhh6ECnOnUckbR2hGt7McOF6ZlUP9h6bNbELTr7gu/MnhbApR+qrnzU4d3Stj/N92m7LLv/ay6u1xhRz6rtG80wID4T0aad6enDY1KPfzyYoL39jnUGtFb0yEhoPSPm7GGRJUq39wPhQt1As57/qWnLK+xvPsnuIDCQGIytbL3JhIm41Mm1Sfqu85LvvROeQBK3A09BqUvni41q7UKiu7/sF6bZQ4i7Wu0nhw6kdNOcMDh+WbnQ/GCUQu2jH/Sp3XPm/5ZLuIr+wLU0i8eH2xYU0VirPcflJOADHyi/qcQUE6X1Z+jwMQ6DsGcgSgYZDOjZGk8ABSXcdG1HxfcUFfxZVf2nWSB2dcP9ujPJ/Mil10Iyekzu+infd23pFVqnQbXw/l1SQAAGVDtvgsOCnK5Xk3Jz7PQv1HVK73ltPti7N6H6D+o6q0Vgs7pEBL+SY9Ur+fofbUyTdvdNt1U5xFLcGjkje/ruvRR+0a7vjUXK5456zcnRPVSXVzBSwb3b0bNWvmrMOHD3/wwYwVK1a4uLrWdEQIIYQQQqhuCngr7v7fkQ8K/rc39t+Nnv5CJ1ZwMgbydQxvku04KBW1yxrrTQH44KHZrQx2W0NEHGE1OiB2FtWzZ6eEAhAHY3BDs4QAUJKRLMylZW8vFQVNjkBtAp5jkiMdVx+UWlhLQEMLAwCscVA3g4iXbPnKc8dtUZ4ZODOTUdY0eQb54asscdX0KbxG2qa91gHER86JC1KaKr/s1Stiz/wSdXPrg+WdzCyAm4ulMl9LGOPQ7gaB1m7VN85hSayRY9JjVCv+Z/f/7N13fFNV/wfwc+/NaFbTpru0KdAWyhTZIENE4HGgiIoPS3Dg44PKUhFwIyoOFAG36PMTQZxsAREQCwhlCxTopOlKR5I2e917f3+kLaU0TVoKKe3n/eIPmt6b871fDgfy7Rlm2jG4u4v259kbvoYnZiNjZQnnpgsvKN9+M3aTgagGVN4q99U047h7uE3slqx4J2btGZHBSdktgozzQWV1py01x2PSjjEDnbRTunxZ9KZMgdVNGUslW7Yqsry35TX//veWBroKbb9nhF3klqxcFv1ThtDqpiqK5V98EPNjOQntVzEyuBHP3cR+4jMAu2zvGZqJsw5QEUIIo7YMDGHOZgvpJEt/KSGEiDuZ+0mos8dlZVzzJ/+KTNJHdqv25glsbqqiWPbNipj1pUTW0zTU3+Kin60QQ6lIZ6dYJ1NULLRQTe29DWejTqMNjAPNntWqDkmxLiZ9b+TabIpimPSTQVor5bIJD/wctstMmGin+soKKkbOpo2c0Bq10ZmDNZwu1zff/O/vv/+ePWfOJx+vWr36a0whBAAAAIDrTyHnaJ6y2ChCSP6esJ0PFEy62/LNOvejt7rzfw/7w0QIRZvthI7lgq/4iMtbZRvSBCOGmhYtMT1rFJ25INu3T/X1AbHF2+t+/G+3KF9k4e1yCUcRQhhHcizPaeV/avxZ7Ugf+EuhG1Z5+wD7+xkSVmgd1dfF54Zu01CEYoc9cfGrfzmFVW/DqqMJIRTdtDWUQmfHKJ4WG1euTV95+Xdio9yUH8/uNW/17uNllu0+R48b6EyM5ElBQ03TjCs5lue0soPFzbE4tMHHpIWu9pE8Vyo96s+K3Qbz36hs1HZZVxE6k6J4Tis7UFjr2W2y1PP0xFucSVE8qbzaOH3wHQBzIE1q728d3tO9Zrcgvpelg0M2/wdm/gLjsK7cxjSqR2+rihd/e0TEkmZNvj8cksOZ9NTBzvYRPDH6vryJGn6oBnpvwzcS4eWveR8HrmlWWWFeGUU6uqMUhHjmu7mEBXqKUnESjJweVz1yQqvU1ouDHufPX5j9zKzJUyY/9dTMIUOGfLRiRVlpaaCDAgAAAIDW4/z6jknrG7pAIWMpnrY6CCGEt8i/2S26+07dczw7XCB5e4fESQjhKauDUCJWISDk8jMHCC/YtrK9+XzFHT2tvTvbevcz9OlrSqE6Pp3q9XWfAfMu2skTyjPHhSICihCWsL7u8rCeUe7QVUwabOy5TpLexThaRZ3aGpzDEirENP02J2OUrfo4au1pcZmdDx+g3fi813IRTwjh+SCR9yC9vC4Wc5T3nFy6y/s19TfHXWq0oaYp0sQP7V401Fb1WjB/GqSUDebfn4zVG17trtIcfMR51XQnFced5v79LMo/pUN7211nQ/86JRhkrBjW2yY+So+42UXyw3cXVl3cXMn3E0VfekOf/b/Jmtx7G85GHd7GASK4tll1uQiheGHNX2KKcrGEUPWtmsTIeTl/R05ojVAcrOKZQnjw4ME5c+Z8/tmn69Z9/+uvv3Jc06b+AwAAAAA0Ch8s52qKg4RQZ3aEHrlbO20MMeyP21jieZG2OSie4hTS6hkxtTlE+7ZG7ttKCMOmjCz++knjrYOt0tRgi5fXGxedW1BUQehoa/9Ickbrx/UO6U9/iieNN97bJTxkuCnSKVu+T8QSwoS4ooXE+rdq5eEgJyGEUDoDc+UhAwxT9RuTmeFpV+d4lrpQ3+dSl+hiKcXJlY8/Gbu33i2xvOXEj2vqeTexrX8S52nUR9O0/WIpRUdbBkXzp4uuunDkq63sEoqONd8aH3E6z0dbtM/8+5MxX9Fml1B0jOWWWP50QXU8EsvQFI64RDmlFCG8m6V4npcGXUWchJBa/eSyL30HQDidYmt6yaCepuFx3KhO1JEv5QY7/cdJ+v6+pn57mdvbkQvfB2dwVe/WXMn3ByW3jOxyKVE++j8hPjNZvyb3Xp9/4+rwMg5c56z6ChIj55VhXPU4ADegNr3n4JUuXMh45plZ69Z9P3XqlKVL346NiQl0RAAAAADQJkilHEVRdmfVl1yZck0aw7HCTdsvHQNqt1M8zQVLr7iZcd52u6lnJCuiCSMgbivtoAhF8ZS31xsbHCvZlSbiRNbZz2rHJrrkQj40xjJ+kN37xBQqfXfISc51973aaYPclWmhOyoIIYSrEJa6iKRHxeSuLjlDCM3LpVyt2QqU003xFNu7ryUuiBBC5WQGGXluyIPaSSkuKUMYMRsRXGuCEife+beIC6l8bW75yI6uYCGhaT40ynZrN6eggZz4k7eqcLj+t+tHqN0Shg+OtkybVTwxilhOK1LNvprmxDsOiliBbfaC4ik9nKFinhFw0e2tnevZLdIPvtra8leQi7E/tbBwRm+HSkxomleGstL6/ox95L9ZegsXtPnPIKfA9sxz2gc6uaUMr4yxPDG3eEI4qTga8kclIYQqLRfwjHPUKFMHKc+I2cSuttjLy3yN7SeXfek7AEJ4wR+pMpvE/OiThn5EuuOogCf0wQPyyjDj3EcrE/mgrQerdnlrxuTXjyKKELdMQGiGi+lUuXBR0b2hRJ+m3Gsmvvu/H5n08mfU1N7b8I31PV6948A1z6r/MHL6P3JS7C3Tc46vyVl0k5/zIOEGg5mDdbnd7p9//vnYsWNz581Z9fEqTCEEAAAAgOtAKuEoXmh1VH+C45nt73dOvPwam5PmKTZYytdZckcpLY/9t3hw7f/ac4Ltf8usSlO9rzd+Agh15JfIzQMKx3XSr/hAX/t1bzewWuV3x8qXDawczoq+2i438oQQwlfKf0wTDB1qeuVt0yu13iSj+ncFuUEVvD1lrGanKq7fu8GWE6o1uaZnEo1L3jEuqadR6szG6K/75c8YWPrVwEubArnOR41aGKbxkpPaz+4tb1XXUHz7W0q+vqXk0jdNsqX/pyzlfTSdx1FnN8Z83kczM6nijSUVb3i+x9Nb3uk86+8mVDh8tJWxOeb9m/IWdDcuetW4qNZtV04sajj/PrLht8ytMcv75D3fzfDee4b3aqItVb72TbCeJ4QQzTFF+iRbz5EFe0YSQghxS956usOXxf7GSa7oJ3W+9BkAIUR3SLl7uumernbrsdjdFYQQYvlHudtY+WAKsf0Ts7noUqNXl3y+2+ScTROcBz9OnvZ7fXO4KPaO2Zl3zK51Y3HIy99Unensq//7zqQXTe+9Dd5YT0v1jgPN2KWvEkbORoyctH30UHtoMBk7yP7WKVmjMwEtHmYO1i83N3fe3GfXrft+ypTJ7yxdGhsbG+iIAAAAAKD1ojhZEE/xlNXZ0FU2B0VIPTMHKUp4/Kgkr5J2c4R1MJoMxRcfJTz/l4B4eb0JW0dxhuD5C9Xv7pbmVNBultYVyDYdDrLyxOuP0HnBzm3KIpY4MkPXZtRUPAXbVyU8u0FxpoxxsMTtpA16UUaG7FA+4wnJeixy3g+KM+VMYanQSQhxSlYsVr/5hzS3kmY54nbQ5aXi48eC9+VTVRv/WWRLX+ww5wflIY3A6KBYN11eLNmXLnJ6z0ntZ/dxDU+f2h+SqhFYWcpuFp46GP7Mgvhvqg8WaKBpQghvlS57ucOsH4KPFgksLsplZ/JzpNmWxs888qMt4gj6cnHHR74N2X9RaHRSLEuZKkTpZxS/HBO7rvhDaSD//mTML46gz17vMPN75dFixuamLHrxX9ujp8yP3Vx9vgSrCZv1YdhejcDKEbddkHM+qO7JE43sJ3W7ja8ACCG8RbE+VcjydOo+RbnnTe2yTX8LWY758/fgIq72lVeVfIYhhKfM1np2ZNRdCN5yTJJZxlhdFMvS+mLJjg2xE+bHbtdVX+Gr//vOpBdN7r0+slFPS/WNA83Ypa8ORs5GjJxc0K4DQQZz0LZDjV3HDjcGShkWEegYWrT2HdrPmzs3Pj5+7dp1mEIITTPuNsf/FhsJIU+9G/fbgUZuVAF+SBhZuPJBx6FVHd46c+33JWkxmDDLfx4ue/Bme5yCt+vlb78Yv86frUy8o4JNLy0qHV0ccftHwc3+g9mWoE4/aVHdxv/kx95a9Nm/bSc+7/jqicCH3cK1+i59lVrUX4EG3HmL8eP5BYSQ6a8Eb9wjDnQ4+Df9MhGjNakzLYc/Tp6+qxVuVJ88Iee3ia5f3+j0wvEW/XcEWjb+9rkZXwwVvT+rwycFgY4FWobWPXI2rKX9mw61YeagDxdzL86b9+zateumTJn8zjtL27VrF+iIAK4G33tS7vF1We8Mcjfp/7lXefs1wiui7F2j3EEtKabmdkXmhbbZL2ueu9XaXskJaF6u5J1mwqj1a//v/IEFFeomDe2UyNU12RER1Jwn7rUkdfpJy+o2/idfFmnv0mLCbg7XcFRpvi6NkROAEEJolfWugdZOEW65kBcEuTv11r05wSLixSeymLb2+RbAX4y9byfOnaPY6XupL7ROGDnhRoE9B33z7EJ49NjReXPnrlq1ElMIoWVhHE+/lzMvPuiD59uvunjlhzxu0IycNXe59n3U6bG9DCGEIjxFEbqpnwav8nZosjqZF/cwTEzgnZrQ596L3FVIC5UssRASxtOE0K3zhz58t3vzl93FbXkr4eN6+jn4r8Vl8oYYVTByAhBCxJ31S18wymv3ZJ4q+it83XU4ThTgxkSHupQ20W9bQnJwhENbhZETbhQoDvrrYu7FuXPn3XfffVOmTB4wcMBHyz8qKMDUcGgBKHdUKE+JbI9Pqfz5rRDt5VVrRq2fP8rJUFRoKMsQhiXUsXUdb17X9Mau7nZosrqZj4p3KikqdWPkNg3DE+LQCwghJC9s4rSwQIV4rYXE2JMjGBH+H3XVWlgmb4hRBSMnACGEiExBe884e6md0XKOuJjiAknqn2GrfpOV4ifmAF5w5cEL57X1/QfaOIyccKNAcbARWJb9+eefjx49MmfOnJUrV2AKIbQIIneUnDhNDNO7fEaK8o30Wp/4KXb0g/oeLqZCyKqC2QZKAbSIDVPwbgtjsFP+vN4W+PPsAcyPWMxRPF1eUc/m1oHSlnsLtE0YOa/Ulp+9Lag8Ez7rpfBAR3H9ZP7YMfnHQAcBADe4tjZywo0LxcFGu3gx79lnn7vvvvumTJ08cODAj5Z/lF+QH+igoO2ile4wmpTuj9iSUjJ1QsXqxaE1x6sJ1PqZg7i/v40wTykZHOr2LDats7s2rbTOeEL7n4H2UAHhecqkVSx+Oe6XMq+v17k9rFf5i/eYu8Y5Y0NYqYDYTaKTh1TL1oaeMNYKUewccVf5YyPMPaNZCaEqDaKcvKBdm6K+OlN7ow2+72PZ68e6d77f6an91ctiKfdDr2a+3V362pMJa9ymhc+W3dHeGSXjeIcgOz34qzWRGy7WWxrjb3ky89sxzKp5HT/Mrfp0qrwtP222+e+VnR75o6pRSuK46/6yx4ZZUlS8TSfevyfi3V/kBW7SQE4uT3v916h6li8Ya+6udsarWAlFGcqCDv8d8vmGkNOmS/c20LQnXbffW/b4cEv3KJZyMIUXFZ+sitlYXN+26DQ74ZVzEzy/d8te/U/CGnfF/74qGnCiXf83lUY/2qKVtokTyiYPtHYM4a1lkr9PM1FeliRTIQ3l35+M+dlVRo8rm3GruWsExxnFx9JCPvkhNM1Qq8RAO2YtPzeLEEIIpw+dOiPmoPuyVhqO0zevAfA9puRseMC164NO//2rKkfRd+Tte8K+cUmnF455IuR7TMnZcD/765Lk+ceoZkx+PSi+04jibx629I1zMzZh+inlV+vCdhR57ver//vMpLKzYfY9xn6JjvZhrISiyouVr78Us72yib234WzU6tuk4XHg2/Jmy+plHVJIjKWSP7dHfJ7hGndnxege9jgFsZQG/b45aul2SQVfJ0iMnE0fOQEAAACgZUJxsCk8UwiPHD0yZ/bsFatWrP1uLaYQQqBQCncoTVXoZd9ulE+apXs0JWSJZ/IgxY6835BSoZy+J+iO8SQomJVRxFnnEyHtenB2/vw+LMXSOh1NSdkQFXGYvb9+BVUn49g+9ppxRBbiuOVfxb0SuPEvhmV4tlYR22e8nLegO1u92RYfFmUPi3KIzoV/fYaptfsKdfqYrPxuQ7+brOL98qpzRSXWIZ14NleeaiBEwiZ2sccJCSGESN1d+urfS3K7ZsdtqWhS1oJsT7+WNyeF83yYDoq2jZ2Y3ysy7t5VCgPlx7N7z09YivG+/jUJ4cNjrXfdb719sHXOi7E7dL6a5gkR2We8nLegB1v1KV/oTurslDf5mNUG26JklpeWaKarec+fjDjGemcMIYTU35rbe/796y3+dJX/vJI3v3v1s4fZh9+hHdzH9uyidlv8ry80EKdPDQVAXfhHWna/4aYUm/AvmYsQQribu9mFNNcrxcEcC2IJITTbq7OTdsgPnKcIadbkX4nieg2rfiShs8/Qsptvsr6+SP1tfrPN1Yq8yTB1yKWeHKHiHdar6L0N31jrwXyMA82X1TodMjTaet8jeffVukAUY31ohkZlS3xyr6Duv+4YOT1fNXbkBAAAAICWCsXBpsu7mPfcc897phAOGjToo+XLNfmYQgjXG61gQwjRmpjSg2E//zvvgXuNn51TlvOEia54fCB7en3YISs72ELRwe4wmhgu3wuZklpGd2e5rPAHXok8ZSGE4iNi3S47oWT1v14/ntm+suOCVKGZ5WI6V7z6rHZUJ/3kLqpXz1CEkMS7ip/tzjoLlG9/FrHpgshMuOjbCn//bz0fGR3n5QfNhnt6Wnow8qMsIYRIupgHSqjsEzINS3ir/L2XEl/MF5XZiFDuGjS+cNU40/193Vv/EDR+XS3faaz26c586bHIRd+EHiyilB2Nz88uvn9E2eRNio/1vp/dW95qEvLb8sTnIFnnKgAAIABJREFUUwV2iotJNE2brn2sa8WS6fLDHwQb+IaaXqUhHe7UzuvO2vNC3voi/LcMoU3IquPdBm8f4znmx9oTCQmhQvx9zFUavvt92ofjeeMF1atfhu/KZQQq+4g7S1+616KoN2Xe808azsZl79JQV0m6q3huN9aeG/raJ+HbcgSiCMuER4uf71f52nTF/veDq0pInHhFrWlNjYrTZz/xEUCmLM1muLOrtSMju8ASIrAN7MJShHToao2ig4o4QiTWgYm8K0t+2NbMya/vOancA5Fv/Kg8VMiII6z3TSlecIvl+WmVO94MKfXz74OvTBJCCM/s+qz9wj9FFRwXFcZVuvlO45rWe310+9oaHAf4TuObNatVHVJg4bnOw4s/n2mMtco+WRW19h+xjnf3H1/4yQPW4SONkftUdfZyxcjZtJETAAAAAFqs1nmq5XXjmUI4a9ZsmqY+WrnigQceoFvpQaEQKCn/zsnalJ7r+bUh+7nEuh/oxHJWShOzleac0m+3SUR9dRPVPCFc/7H6XnbFV7tELMWYrIRSuEOufHee4gmhQh39O7iCKEJ4qqxQWMF7f71ePDEZBEYn4Vi6MF311jaJm3GndHDThBDGcedQu4gL+uz92G/PiCpdhHXRZd62ybPLth9nqEjTyKpn5Hv3s4QS8Y6/xZ6SZkiy/q3Xsw98d/6fr3JfG+RiCImKcDfl7xvtGDvMLrAo3vwgfG8+42Dp0syQ179XmGnH4O4u2p9nb/ganpiNjJUlnJsuvKB8+83YTQaiGlB5q9xX04zj7uE2sVuy4p2YtWdEBidltwgyzgeVNW1ScsNt0Y4xA520U7p8WfSmTIHVTRlLJVu2KrK8t+U1//73lga6Cm2/Z4Rd5JasXBb9U4bQ6qYqiuVffBDzYzkJ7VcxsjEbeTexn/gMwC7be4Zm4qwDVIQQwqgtA0OYs9lCOsnSX0oIIeJO5n4S6uxxWRnX/Mm/IpP0kd2qvXkCm5uqKJZ9syJmfSmR9TQN9be46GcrxFAq0tkp1skUFQstVFN7b8PZqNNoA+NAs2e1qkNSrItJ3xu5NpuiGCb9ZJDWSrlswgM/h+0yEybaqb6ygoqRs2kjJwAAAAC0VJg52Aw0eZqaKYSDBw9evny5RqPxfRtAc1DIOZqnLDaKEJK/J2znAwWT7rZ8s8796K3u/N/D/jARQtFmO6FjueArPuLyVtmGNMGIoaZFS0zPGkVnLsj27VN9fUBs8fa6H5/0ivJFFt4ul3AUIYRxJMfynFb+p8af1Y70gb8UumGVtw+wv58hYYXWUX1dfG7oNg1FKHbYExe/+pdTWPU2rDqaEELRTVtDKXR2jOJpsXHl2vSVl38nNspN+fHsXvNW7z5eZtnuc/S4gc7ESJ4UNNQ0zbiSY3lOKztY3ByLQxt8TFroah/Jc6XSo/6s2G0w/43KRm2XdRWhMymK57SyA4W1nt0mSz1PT7zFmRTFk8qrjdMH3wEwB9Kk9v7W4T3da3YL4ntZOjhk839g5i8wDuvKbUyjevS2qnjxt0dELGnW5PvDITmcSU8d7GwfwROj78ubqOGHaqD3NnwjEV7+mvdx4JpmlRXmlVGkoztKQYhnvptLWKCnKBUnwcjpcdUjJwAAAAC0WCgONg/PFMK0tLQ5c2evXLli48aNa9Z853a7fd8J0KDz6zsmrW/oAoWMpXja6iCEEN4i/2a36O47dc/x7HCB5O0dEichhKesDkKJWIWAkDpdkhdsW9nefL7ijp7W3p1tvfsZ+vQ1pVAdn071+rrPgHkX7eQJ5ZnjQhEBRQhLWF93eVjPKHfoKiYNNvZcJ0nvYhytok5tDc5hCRVimn6bkzHKVn0ctfa0uMzOhw/Qbnzea7mIJ4TwfJDIe5BeXheLOcp7Ti7d5f2a+pvjLjXaUNMUaeKHdi8aaqt63rg/DVLKBvPvT8bqDa92V2kOPuK8arqTiuNOc/9+FuWf0qG97a6zoX+dEgwyVgzrbRMfpUfc7CL54bsLqy5uruT7iaIvvaHP/t9kTe69DWejDm/jABFc26y6XIRQvLDmLzFFuVhCqPpWWGDkvJy/IycAAAAAtFQoDjYnjUbz3LPPjx49+oknZvTp3efDD5dn52QHOiho3fhgOVdTHCSEOrMj9Mjd2mljiGF/3MYSz4u0zUHxFKeQVs+Iqc0h2rc1ct9WQhg2ZWTx108abx1slaYGW7y83rjo3IKiCkJHW/tHkjNaP653SH/6UzxpvPHeLuEhw02RTtnyfSKWECbEFS0k1r9VKw8HOQkhhNIZmCsPGWCYqt+YzAxPuzrHs9SF+j6XukQXSylOrnz8ydi99W6J5S0nflxTz7uJbf2TOE+jPpqm7RdLKTraMiiaP1101YUjX21ll1B0rPnW+IjTeT7aon3m35+M+Yo2u4SiYyy3xPKnC6rjkViGpnDEJcoppQjh3SzF87w06CriJITU6ieXfek7AMLpFFvTSwb1NA2P40Z1oo58KTfY6T9O0vf3NfXby9zejlz4PjiDq3q35kq+Pyi5ZWSXS4ny0f8J8ZnJ+jW59/r8G1eHl3HgOmfVV5AYOa8M46rHAQAAAAAIEGyQ18w4jtuxY8dTM5+2WC0ffLjskUemC4VC37cBNJVUylEUZXdWfcmVKdekMRwr3LT90jGgdjvF01yw9IqbGedtt5t6RrIimjAC4rbSDopQFE95e72xwbGSXWkiTmSd/ax2bKJLLuRDYyzjB9m9T0yh0neHnORcd9+rnTbIXZkWuqOCEEK4CmGpi0h6VEzu6pIzhNC8XMrV+skG5XRTPMX27muJCyKEUDmZQUaeG/KgdlKKS8oQRsxGBNeaoMSJd/4t4kIqX5tbPrKjK1hIaJoPjbLd2s0paCAn/uStKhyu/+36EWq3hOGDoy3TZhVPjCKW04pUs6+mOfGOgyJWYJu9oHhKD2eomGcEXHR7a+d6dov0g6+2tvwV5GLsTy0snNHboRITmuaVoay0vj9jH/lvlt7CBW3+M8gpsD3znPaBTm4pwytjLE/MLZ4QTiqOhvxRSQihSssFPOMcNcrUQcozYjaxqy328jJfY/vJZV/6DoAQXvBHqswmMT/6pKEfke44KuAJffCAvDLMOPfRykQ+aOvBql3emjH59aOIIsQtExCa4WI6VS5cVHRvKNGnKfeaie/+70cmvfwZNbX3NnxjfY9X7zhwzbPqP4yc/o+cFHvL9Jzja3IW3eTnPEgAAAAACADMHLwmirXFCxcuGj169BMzHu/bp+8HH3yIKYRwjUglHMULrY7qT3A8s/39zomXX2Nz0jzFBkv5OkvuKKXlsf8WD649DHCC7X/LrEpTva83fgIIdeSXyM0DCsd10q/4QF/7dW83sFrld8fKlw2sHM6KvtouN/KEEMJXyn9MEwwdanrlbdMrtd4ko/p3BblBFbw9Zaxmpyqu37vBlhOqNbmmZxKNS94xLqmnUerMxuiv++XPGFj61cDSmm+7zkeNWhim8ZKT2s/uLW9V11B8+1tKvr6l5NI3TbKl/6cs5X00ncdRZzfGfN5HMzOp4o0lFW94vsfTW97pPOvvJlQ4fLSVsTnm/ZvyFnQ3LnrVuKjWbVdOLGo4/z6y4bfMrTHL++Q9383w3nuG92qiLVW+9k2wnieEEM0xRfokW8+RBXtGEkIIcUveerrDl8X+xkmu6Cd1vvQZACFEd0i5e7rpnq5267HY3RWEEGL5R7nbWPlgCrH9E7O56FKjV5d8vtvknE0TnAc/Tp72e31zuCj2jtmZd8yudWNxyMvfVJ3p7Kv/+86kF03vvQ3eWE9L9Y4DzdilrxJGzkaMnLR99FB7aDAZO8j+1ilZozMBgSMSiSiKOBxO35cCAADAjQ8zB6+VqimETz1jtpgwhRCuFYqTBfEUT1kb/N+7zUERUs/MQYoSHj8qyauk3RxhHYwmQ/HFRwnP/yUgXl5vwtZRnCF4/kL1u7ulORW0m6V1BbJNh4OsPPF6gigv2LlNWcQSR2bo2oyaiqdg+6qEZzcozpQxDpa4nbRBL8rIkB3KZzwhWY9FzvtBcaacKSwVOgkhTsmKxeo3/5DmVtIsR9wOurxUfPxY8L58qmrjP4ts6Ysd5vygPKQRGB0U66bLiyX70kVO7zmp/ew+ruHpU/tDUjUCK0vZzcJTB8OfWRD/TfXBAg00TQjhrdJlL3eY9UPw0SKBxUW57Ex+jjTb0viZR360RRxBXy7u+Mi3IfsvCo1OimUpU4Uo/Yzil2Ni1xV/KA3k35+M+cUR9NnrHWZ+rzxazNjclEUv/mt79JT5sZurz5dgNWGzPgzbqxFYOeK2C3LOB9U9eaKR/aRut/EVACGEtyjWpwpZnk7dpyj3vKldtulvIcsxf/4eXMTVvvKqks8whPCU2VrPjoy6C8Fbjkkyyxiri2JZWl8s2bEhdsL82O266it89X/fmfSiyb3XRzbqaam+caAZu/TVwcjZiJGTC9p1IMhgDtp2qLHr2CHAVCrV6i8//+Kzj99asnjWM09NmvjQqFEje918U1xcnFgsDnR0AAAA0MwoZVhEoGNo5SiKGjNmzBMzHi8u1n7w4YfZ2ZhC2OaMu83xv8VGQshT78b9dqCRm0+1OhGjNakzLYc/Tp6+qxVuVJ88Iee3ia5f3+j0wvFrv+sZtFr87XMzvhgqen9Wh08KAh0LtAyte+Rs2J23GD+eX0AImf5K8MY9gS9L1fyb3tpRh4x7WL7q54oUYQnF83zV7FCGsojp0iBaE0QVBdHFYrpITBcHMYUMsQUuYAAAuGG0kH/ToTbMHLzmeJ7fsWPHzKeeNpqMH374AaYQQptCq6x3DbR2inDLhbwgyN2pt+7NCRYRLz6RxbS1z7cA/mLsfTtx7hzFTt9LfaF1wsgJLQCvYM5Q1YdU84SpqQwSQlheZmU76N3DipwTcuzzzlnf1Trvr30BAAAA3Fjwr/h1otVqFy16ccyYMTNmPN63b7/ly5dnZmYGOiiAa07cWb/0BaO89iw6nir6K3zddThOFODGRIe6lDbRb1tCcnCEQ1uFkbMlKyyh28hkB2nMWWnMzRTl/UfaPMXzFOvIMV18p8xy7jQhhLSJzAAAwFUqLME0tRYHxcHrxzOF8MTJE7NnzVq27P0NGzZ8t2aty928WyEBtCwiU9DeM85eame0nCMuprhAkvpn2KrfZKVet84CaOu48uCF89r6/gNtHEbOluzIWeH0V1r/EhCZTHbnXcz0aV6flGVZjmW/W7vu119/5TiOEIxaAAAANzAUB6+3Em3Jiy++5JlC2K9vvw8xhRBatcoz4bNeCg90FNdP5o8dk38MdBAAcINrayMntARCobBjh47JnZI6d+qc1Ck5rl07mqZ5nqeouvNVOY6jafrE8RMrP/64vMzPA40AAACgRUNxMACqphCeOD5r1uyGpxB27NgxJyfn+kcIAAAAAK2bSqXq2q1r165dk5OSkpKTRUKhzWbLzc09ceLETz/9dPbs2XfefjsiMrL2LRzH6XS6lStXHTt2LFBhAwAAQLNDcTBgSkpKX3rppTFjxjz++GP9+/X/cPnyjIyM2hd0797jzTcXL1iw8Ny584EKEgAAAABaB5VKlZSUnJSUmJyclJLSJThYwbJsYWFhVlb27t170s+lF+QXcNyl5etnzp4dFhbGMAwhxM2yNEX9+uuva79b63RhVxwAAIBWBcXBQKo1hXDW+++/V3sKoTgo6Nnn5jIC4YsvvjjzqaeMlcZABwsAAAAANxKJRNKhY4ekpKTkpOSkpES1Wk0I0Wq16ennvl//fVZWVlZGZgOVvvPnzw8bNozjeYqQs2fOrlq5sqgYx6gDAAC0QigOBl5JSelLL71cNYWwf//lH350IePC9OnTwlVhFCHBiuD5zz//yiuv1v5BLgAAAABAHQzDtItrl5SU1K1r165du8bFxdE0rdfrs7KyUlP3Z2VlnzuXbjKZ/Hy3CxcyGIaprKz89NPPUlNTr2nkAAAAEEAoDrYInimEx48fmzVr9vvL3tu18/fR/xrj2QGaETA33XTTgw8+8MMPOOYAAAAAAC7jdevAkyd/+vnns2fPlmhLmvbOubm5v/664fvvv7darc0bMwAAALQoKA62IKWlZS+//PLdd941cfJEjuMZpup4OJqmp06deuFCxsmTJwMbIQAAAAAEVnR0dHJyclJSUlJSUnJykkwmc7ld2Vk5GZkZ23/bnpGZWVhYyPP81TfkdrtXr1599e8DAAAALRyKgy0Lz/Mx7WLlcjnD0Je/ThYseGHmzKf0en2gYgMAAACA6y8qOiq5qhSYnJSUJJfLWZbNzy/IzMw8ePBgRkZGbm6u2+0OdJgAAABwo0JxsGXp0iXlnnvGehYU10bTlEQiWbhwwQsvLMDmgwAAAACtWO1jhTt17hyiVHIcV1BQkJWVvXbduqysrOysbIfDEegwAQAAoJVAcbAFEYlEzz/3HM/zVxYHCSECgSClS5fJkyetWfPd9Y8NAAAAAK6R2tXA5E6dQkNCaqqBP/zwQ1ZWVnZ2jsNuD3SYAAAA0DqhONiCjL/vvqjoaI7jOI6laebKC2iKeuihh86ePXv8+InrHx4AAAAANIsrq4GEEM+xwr9t+y0rKzs9/azZbA50mAAAANAmUMqwiEDHAJdER0d37da1e9duAwb0DwkN5ViWUKR2oZDneKvN8t//PqXT6QIYJzTKuNsc/1tsJIQcPy/R6oSBDgcAAOCGER3m6p1iI4RMfyV44x5xoMNpusuqgcnJoaGhpLoamJmZlZWVfe5cuslkCnSYAAAA0BahONhCDRk6JDw8XK1WJyQktG+fECQO4jiOoiiKonieLy4uXvPdd21888Hz586Xl5cHOgq/1BQHAQAAoGluuOJgRGRkYscOHTteqgZyHFdUVJSVlZWVlZ2ZmZmdnW2z2QIdJgAAAACWFbdUCxcsqPMKTVedX0xRVGxs7Avz51/3oFqWt5cu3Z+6P9BRAAAAABCGYdrFtUvskJiY1KFDh8TExI4KhcLzA93MzMxfftmQlZWZnZ1ttVoDHSkAAABAXSgOAlxzG/eIQ/Zgii4AAEDrIRQKY2JjkpKSkjyrhRMTxWIxy7KFhYVZWdmH0w5r8jQ5OdlGI1YKAwAAQEuH4mCLlpmZtX3n74GOomVJSkq8819jAh0FAAAAtC0yuTwhQZ2UlJSclJyUlBgXF0fTtM1my83N1Wg0qfv3Z2VlZWVkOl2uQEcKAAAA0DgoDrZoBoMhLe1IoKMAAAAAaHNqjhBJUKvVCer4+HiKosxms0ajOXHy5E8//5yVlVWQX9DG94AGAACAVgDFQQAAAABo6zybBqrVanW8Ojk5qXNKZ2WwklQfKJyauj8rKzszM8NgMAQ6UgAAAIBmhuIgAAAAALQ5crm8ffsOHTokeA4QSWifIBQIXW5X3sW87Ozstd+ty83JycnNtdvtgY4UAAAA4NpCcRAAAAAAWjmhUNgurl2HhA7tOyS0b9++ffv24eHhhBCz2Zybm3vm9JlNmzfn5OTka/JZlg10sAAAAADXFYqDAAAAANDaqFQqtVqtTlAnJyWr1fHqhASRUOg5TViTp9m58/esrGyNJq+kpITn+UAHCwAAABBIKA4CAAAAwI1NIpG0a9dOnaBOSkpKUKvbd+gQoqzaMVCj0ZxNT9+0ebMmT6PJy8NpwgAAAAB1oDgIAAAAADcShmEiIiLU6oSao4Tj4uJomrbZbIWFhRpN/uG0NE2e5mJubkVlZaCDBQAAAGjpUBwEAAAAgBZNLpfXzApUq9WJiYlisZhl2bKyMo1Gk5q6X5Ov0Wg0BfkFHMcFOlgAAACAGwyKgwAAAADQgkRERsbHtVMnJMTHxanj1XHx8cHBCkKIXq+/eDHv/PkL23fsyMvN02g0LjfWCAMAAABcLRQHAQAAACAwaJqOjopSJ6jj4uIT1Oq4+Lj4+HiJREIIqais1OTl5eZd3PfXvvz8gtzcHKPRFOh4AQAAAFohFAcBAAAA4HoQCATh4eFqdYJaHZ+QkKBWx8fHx4vFYkKI2WzWaDQ5OTl7//xTk6fJy8szGAyBjhcAAACgTUBxEAAAAACan1AojImNUavV6nh1glodHROd0D5BKBCSWocIb9++Q5Ovyc3JtdlsgY4XAAAAoI1CcRAAAAAArpZKpYqNbRfXrl1cfDu1Wh0XFx8ZGUFRlMvlKiwsKsjPP3w47ZdfftXkawrzC7FXIAAAAEDLgeIgXDfi3rO+/ephxa5FUxf8ruMDHQ0AAAA0jUwmi42NbdeuXWxsbFxcOw/PRoF2u72goECTn39mx3ZNfn6+Jl+r1bIsG+iQAQAAAMArFAdvbLKxK4+9Nyhr9cwp76dVXFZvEw5bsueb8cYv/33P0n9ayv/IKYqiKJqmAh0HAAAA+MezS2B0dLQ6QZ2gVkdHR0dHR0dFRVEUxbJsWVmZVqvNysravWePJk+j1WpLS0s5jgt01AAAAADQCCgO3vgoSbdHP1hRPO3x77KdgY6lQY5jH024+aNARwEAAABeyOVydYJaHa+OiYmOjopWJ6jj4uJomibVB4ZoNJoTJ05qS7TaYq0mL8/pwupgAAAAgBseioOtAM/ywUNeWL4wZ8rig5VYrgsAAAA+KYOV7drFtotrFxvbrl3VCuF2IqGQEGIymYqKigsLCvb9ua+gqLCosKiwsNDhcAQ6ZAAAAAC4JlAcbAXcp9Z8rB3zzNR3X/9nwrMbiupdRCy85bVd304wrBr/wIfnPRdQyvs+Tls66O8XRz7ys54nVNgtM16cNrxrYnxseLBUyBqLzv25btXnp9qNm3zv6AEpcSGMpfDM7//3/tJ1p2vWL1OypLuemPnY3QNTIsW2kgv7N3z+7hf7ClyEEErZa8LsaaP7dUtqHx0ioWzleTtff3hx1kPrf5sV8+sTI15IrZ5oIFHfPv3Jx++5pXtcMGUzFF7465OX3tiY11LWQQMAALQCcrk8Ojo6Oia65tTgmJgYmUxGCHG5XDqdTqPRHD12tHjrVm21QIcMAAAAANcPioOtAVu4feGzig5fP7J42aMXHvky3d6E96BVPUeNHd61ukMIQ+Nvvu+F1ffVukKU0Pehlz5VWe5/cmMJRwiR9nh69ZdzblbQhBBCguJvGvvMil6xs+59aZ+BpyMHPTD1zpp3U0RECh3mK9oUp8z4fPWCASF0VQNRSb3i5TZsVAQAANAUQoEwJiY6OibGU/6LiYmJiY6OiooSCoWEEJvNVlxcXFysPXny1PbtO4q1xcVFRWVl5TyPVQcAAAAAbRqKg60Dbz62as6HPX96YeaHc08+sPSIqWn/z+eN2xc9sGCb1sIHd75n0eeL74g1Hf3kpXfW/p2l4yP6P770kyd7Dx9/W+Tm77Uc0+nhl5/uJS3dt2LROz8czLMru9zx/Dsv3z/u6cn/278q0/Nupl2vTVy4uaCClURFSypdJPayxpgOk1+e119pz9j41huf/3aq2CZWqROVhnJ8RAG4Hu4dd2/XlC6BjgJaqPTz5zZt3BToKKAhIqEwOjZGrVZHR0XHxHiOCYmOjIys2R9Qq9Vqi7WHDh0qLq6aDFhSUoI6IAAAAABcCcXBVsOZsWbR4n7r35265MVD/16498p5en7gWVNZqdHBEmJI3/DJ2odGz+9Ynn7gnNZKCCk68MXqXRNvHhffXk0TLdV57N0pAuMfbz7/xd5KnhBSenrD6yuGjFk+cnD/8E8yywkhhHcbCgt0VhchrqI8IyHMZW0xHe8e213s+uedWa+szWUJIcRRknGi5CqzAAB+6prSZcjQIYGOAlquTQTFwRaBoqjQ0NCqmYDRnvmAsTHR0cHKYEIIx3FlpWXF2uLi4uLjx48Xe35XXGy3N2URAQAAAAC0TSgOtiJs0a+vvj6464cPvL5w3+mXLVf7bsV5RW7SJSIqhCZWjhBCnNqCUp6KlEooQoTxHeNoWjJmZdqYlZffFhsXQ5Ny3+8vSEhuz3D5aQc12GEQAACgamdAlSpMpQr1HBYcHRMdFxcXFBREqjcH1Gq1ubk5Bw8e9JwXnJ+fj3NCAAAAAOAqoTjYqvDle9545Zc+n9//2ov737Nd/i3CESIOCqL8fjPO5XQTSigU1tzicrl5QlE0IcTrwiRKLBH71QZF0xQhWN8EEGhTHn4k0CFAC/Ldt98EOoTWTyKRREVHRUdFRUZFxXh+FxUdFRUpkUgIIRzHlZfrSku1Wm1JWlraps2bS7QlJSUlOp0Oi4IBAAAA4FpAcbCV4Sv2f/Di9/3/N/H5OSVSihirX+dMlWaebtc5SUmd1DXDZwtX4cVCjlNuenz0y3utV36bufKlet+BVvcfFM+cvtiUyYNBQUE0TXMcDjABAICWSCgUhoWFeXYD9MwEVIWpVCpVVFQURVGk1s6AR48e8ewMqDfotcVap9MZ6NgBAAAAoA1BcbDV4U0Hly9eN+TLqe0Z6lLNjc05fc7Idxzy5MJJ2e9t+KfMIZRHhEr8n0ZYF3th566c/zw59rV3L9KfbDuSVWZmhcqYxJtizPuP5Ln9e4cdv2f/5783zV652Lrky22n8o2sJKJjkrL8nws6v+p9c+fMmTtnjtPlcjocTqfTXM3pcDpdLpPZZDaZzRaz2Wx2Op1Oh8tsMZnNZrPJ7Lm4yY8OAABQm0gkioqMiogMj4iIjIyM8MwEjI6KUqlUngtMJlNJSUlJSen58xe0Wm1pSUlJSYlWq3W6XIGNHAAAAACAoDjYKvGmtA/e3HDbZw/E1XrRkrpmzbnbn+l2x5L1dyy59HKT5ya4z6x+8+sRn84YNe+rUfNqXnWdeHfUpP/L86u45z67esnnwz6Z2X3cG9+Oe6MqdMuWWcNm/e7XNurfr1+fk50jkUpkEqlEKpFKpTKZTCKRSKXSkFBlQoJaJpNJpTKJJEgoFNa51+Vy2Wx2q9VisVpsVqvVavMwm81Wq9VmtVnvBcg2AAATtUlEQVRtNpvNarFYLRaLzWazWq1WqxX7uwMAtGXKYGVEZEREZERERER0ZJTnNxGRkSFKpecCm81WWlJaUqrNzMzcv39/SUmJZ0Ww1VrPHHsAAAAAgBYCxcFWia9M/eit34aturPWa44zK574j3He05NH9FCHCHmntUKn1WSn78uyNW2VMW86snTKpLOPPjZxVP+u8WEyxm4oyj55NN//ciNvPrZs2pTzjz0x7c4BXWJDRO5Kbe6ZbKOQInZ/Qrp48eLBgwf9bEskEsnlcrlCLpfLRUKRSCSWK2RyuVwuk8sVcrFIJBKJVSqVWq2Wy+UikUgkEoWEhNA0Xed9nC6X2WRyOp215ipaTGaTy+l0OJyeiYpmk8VsMTkdTqfLaTaZjUaj2+3XZEoAAGgJPAeDRFcvBA4LVUXHRMfGxkqlUs8FnuXAer0+Ozt7//4DnrNB9Hq9Xq8PbOQAAAAAAE1AKcMiAh0D1GPbtq2EkLS0IytWfRLoWFqW/v37zXp6JiHk7aVL96fuv6Zt1ZQURUKRSCySyxSekqJIKBKJRHKFXCFXyOWymnqiXC4PDg4WCOrW3K9c+2w2W5xOh8Pp9Lb22WwyYbkZXFMLFywYMnQIwYEkcDnPgST7U/e/vXRpoGO5tiQSSURkRFREZERkREREZERkeFRkVGRkpEql8vxkyOV2lZWVl5WWlpWVlZSUlnl+V1pWVlrmcmN8BgAAAIDWAzMHAbxyOp1NmAkiquaZqOgpKVaXF+U1JUWRKLSq+CiXK4IVQkHdtc+keqJi1V6KVdMVLTXbKVZPTrR4SopOh9PpdFZUVOCQFgAAD6FAGBYe5jkGpPY0QJVKVbMhoNPl0ut0Wq1Wo9EcPpymN+j1Or1Wqy0tLcVwCgAAAABtAYqDAM3MU8UjhDShqliz9lkuU4jEwqpXaq19lstl0dFRnpKiSCSSyWSeIy8vC6C6pFhr+XNVSdHpdDpdTqx9BoDWRCgQhoeHh4WHR0aGe0REhEeER4aFhwcHKzzXOF2u8rKy8vLy8rLyo0eP6nS68rLykrLSstIybAgIAAAAAG0cioMALUWTJyrWu/ZZLpOLxSKhSOSZqBgdHVWz9lmpVDIMU7f1+rZTrLP22WyyOJ2O6vKi2WQ0YW0dAFw3NVsBqlSqsOqZgCqVKjIysmohsMtlMpn0er22WPvPmdN6nV5botXr9XqdHtMAAQAAAAC8QXEQ4MZ2NSVFkUgkEotq1j5fuZ3iZWufFYorz30ml699rlrd7HJ5W/vsmb1osVh4vmkH4QBAayYSClVhVauA61QAIyIian6qUXMeiGchMCqAAAAAAABXA8VBgLbIU1Js7F1XbqdYZ+2zQq4QCYV11j7L5fJ6AvB77XPNdoqVlZUsyzbH0wNAwFAUFRoaqlKpwsLCwsPDVWGqiLDwsPAwT/lPIpF4LvPsA6jT6crLyy9cuFBWXq7T6fTlupLSUoPBgAogAAAAAEAzQnEQAPzVLNspioQikUjsbe1zzdHPISEhnnWClwVQ39pnk9nkcjodDmf9a59NJlcbO/eZpum+ffocPXYMBRQIFLFYHBYWFhoaGhERERqqiowI98wHjAwPDwkNrTnS3WQy6XX6svKystKys2fT9Tq9Xq8rLS8z6AyVxsrAPgIAAAAAQNuB4iAAXHPXdO1z7e0Ug4ODa+oOl1p3uZwOR616Yt21z1UTGB2u2mufzWZz8yXgupJIpa++9qpOp9uyZeuu33+vqESRBZqfUCBUBCuq1/56ZgKqaoSGhtaclXRpCXBe3onjJ/R6vVar1Rv05WXlOAkEAAAAAKAlQHEQAFqo5lr7LFfIqk9rqX/ts1yhEPnaTrF6uqLF23aKnrXPFRUVAZ+vJ5VKCSFhYWEPPzx16tTJB/Yf3LJ1a3p6emCjAkKYhPFLVj7Z+dBLE95KuwEOBxcKhcqQkIjwsJDQ0Iiw8JDQ0PCwsFCVKjwsLCQ0tOYUYJ7nKyoq9Aa9rlynN+gzM7MMBoNOp9frdXq9viX8jQAAAAAAgIahOAgArcq1WPssV8jFIpFIJPZ/7fOV2ynWWftstpiqy4tmo9HodjdbtUgmrdq1jaZpQujBQwYPGz6sWFu8Y/vOHTt23LgzIlsBRXzXrvGKk1VT6sS9Z3371cOKXYumLvhd19IO6BkwcMDGjRtqvqyorKwwVJSVl+l0ugsXzhsMFTpducFgKC/XVVRUYDNQAAAAAIAbGoqDAACEXN3aZ7lCXj05UeFtO8XGrn02my1Op8PhdHpb+2w2mZz1bacolcpqfylgBISQmKiYadMenjx50t49e7ds3Zqbm9vI9HhBB6eMnjh9/MjBPdpHKUXuypLcC6cO7trw7c9/Fziap4UrMN0eWblsinzLzEc+vtCMNSk/3zYoYcSUpx6+a2h3dbiUclSW5Z4/uX/7ui9+PmVofHmPoiiKomnqKqK+ZvI1+d+vX19hMJTpdBUGQ1vbuBMAAAAAoE1BcRAAoOmaVlKUyWQSiUQqkUgkUolUIlfIJZKqL6RSqUwmlUqlimCFTBLleVUul0skEoZh6rbuclmtVpvNajFbrFarzWazWK1yuaKeJilCU7RIJBp5+8gx/xqTm5vLuq+2skYF93z8/Q/nD4sWVJe3RKq4boPiuvaSXfjtUIHjGk2Go0MSuibHGETNXFPz522ZhAc/+Pn1YeFM1UWCsLjut7RLFpz89pdTpNGP6zj20YSbP2piuNdaUVHRwYMHAx0FAAAAAABcDygOAgBcbxaLxWKxNPYusVgs9dQQZTKZTCaRVpcXJRKFQi6RSj3lxaiICJ7na46DqMMzabF9+/Y1F4SFh+nKdY1+Bjpm/NKPFwwP5ctPrPn0i/V7TmSXOUWhcV36DP1XivZAZbNVBmmxIiwkyG0yGKyB3qdP0GvazCFhpHTPsleW/noir8IlUsV36zesp+3Pkhazq14LShcAAAAAANwgUBwEALgxOBwOh8NhqKho+LIxY8Y89dTMK6cZVuF5luMYhnE5XUKRkBBiNjVlF0Lp4Cefu1VFyvcsmjj3R01VHcpRmp22PTtte/VFkvajH/3vjHsGd42VcYa8Y3t+/uTj9WllnhmLVNgtM16cNrxrYnxseLBUSOwGzck/vl+2fP2J6gW6tKrvjFcW/ef2TqFCiuddJs2uxdNf+KWIEEII02nWpn9mEUII4UrXT73tjYMuQoXduvCD2Xd0josKFvO28uyjO7/6YNWGCxbev+a8vW2tZ45LCGM4zdYVq/dnsoQQ4izNPrwt+3DVtynVoEcXPDy8e3KH+MhgCeUwFGUe/v3Hz1dvOV1Rb+2QSf7v+t9mxfz6xIgXUl1+RkjJku56YuZjdw9MiRTbSi7s3/D5u1/sK3A1kK4WU7YEAAAAAICWCsVBAIBWRSqVcRxXpzjIsm6aZgghFy/mHTx4MC0t7cEHHhgydAghxOFoyu6Ag8beHkk7T6x+7xeNlxlqQV3/88VX8/srq05sieo0fOLCwcNuenbKwi1FLCG0queoscO71vwjJAtPvOXfL/bqJBk/9esMNyF09INLV84fHky5LboSMyUPC4kUOio5QrwUPQkh7pDE3p3iRIQQQuRRXW59+L3uka57n9tSzvtuzh+24gIDS8ePnHrHLxlb82x1v02H9frXfbfVNCEIb9/rriduun1MvzlTX9nhe26hHxFKezy9+ss5Nys8KQ2Kv2nsMyt6xc6696V9BspbugAAAAAAAHyoe84mAADc0CSSoJolw26WJYSYzeZ9+/565513H/r3xKeffnrdunVZWVlX2UrXTnKazf1rf6GXnQuZpKkvz+0XbD/38/wJt3XrfvPNo2e8vaeYir3jtfmjQmtWPPPG7QtH39SzZ2K3AUMmL92l5WQ3TZ7cW0gIoRQDRg9QcGc+v2/QoL7DbuvTp//A+97fb62+kc1YcW/PDp27dejcLXFo1fw+3nTgvYfHDerXJ6lLzy4D7370q3/sYSPuv/VSaw0018DbXuI69vWq/eVUwv3vb9izdvGTY1JUV/58jTf+Nv+2bt16JHYfOOShF748ahAm3Ltk/shQP3dIbChCptPDLz/dS1q6b8Wjd96S0q3PgAde+jmHjRv39OQkxke6AAAAAAAAvENxEACgVZFJpQKBgOf57JzsH3/4Yc6cuRMnTlq27IPU1FSLuSkriOulkFGEM+jrXzBLCJN8z73dRK7TK59d/NOpEqvLWZF38IvnXv2xmA+99Z5LpTKeNZWVGh0s5zYXHl331pozbiYsJSWCJoTwPE8IFZHSPyU8iCKEd5TlFlQ0vJMhRYX0+PdbX/9y4PCRf/Z++9qYWIYIomLCL/0710BzfmHzfpoz/skVm9MtYX3uf2HFz/t3/e/Nqf2iapcIedas11vdHOcyFZ7c+vbM1zaVEdXIe29V+lcdbCBCpvPYu1MExj/efP6LvdkVDre99PSG11fsNTPJg/uH001IFwAAAAAAACEEy4pbuNDQ0P79+wU6ipYlKSkx0CEAtGjZ2dnLli07dvR4pbHy2rVisRFCK0OUNCmtb+6gKCEpjubyDx+4WOu7luOpJ+wT/5WQFE+Teo53ZouyL1r4bnK5jCKEMx3csLt8xF3DF337x7OGvDMnj+3b/N3XOzIt3gpeVPCwl/731cQEYVUVTqyOJ4SwNO3tn7nLmvObs+CvL2b/9X9v97nz4UemTbqt76QXV98+ZMmkp37Mrm9tMl95cPdx57jb1YntaOJjr0hfEYriO8bRtGTMyrQxKy+/LDYuhmpsugAAAAAAAKqhONiiJScnJScnBToKALiR7Nm79zq0kplr4zt3GNgv4pNMbX2zBxtTcKvGO51OnqJoihBC+PJti6aaTzx4x8Cbet/co/eIDn1uHZFCj396W/0VT0p1+/T71Izh8KqX3117KLvMJggf+eLG5ff421zjOLTHNrx7bNPnXccvXv7S2GGzn7nttzm/X7EJISGE8DzHE8I3rUZ3eUK8vQkllogpL+l6alt5k1oGAAAAAIA2BMuKAQCg0f7efcjIiwfOmDW63lW5zrzsAo6OH3BLQq3zQ2S9h94cRJyanAL/Dsqw5+9b88GCp6aNHjr8zld+L+ZVt47pLyW82+3miVQqvayoR4dHR4uIdf+alX+c15pdLGvTlRkbc9JK/W/bIK4yfcPyH8+xtDwpKab+c1IkPfp3FxFn4cXCmkdmmKb9VM5VeLGQ48p+ffTmbp5dEat/9Rj02mEXqT9dTWoJAAAAAADaFswcbKHeXro00CG0dOfPnQ90CABtl2HHp19NGzKvxz3L16tWr1r9a+rZPL2DloW179r/tsHCfSs2b96cPmNej2c+eLn8lU9/SzcI2/V96IXXJ8RQFTu3/KH3YyId0+G2cR3KDx09X2xihQK3yeQghKIIRfhSbRnPdBv14Kh1Gbs07uD23WNsJ84W60pLXaTTgPGT+1z46VSxmRPI5UECQvyuD9b/tkW110yLbn7itZH2P3b8eSKrsMJBgkLV3W+bOa4Tw7vLSvVVxT9K1v+BSSO0Ww/lGoUxve6b9/rEONqyb1dqJU8IcbrcPBXS+9aBcScOFjT2tBD2ws5dOf95cuxr716kP9l2JKvMzAqVMYk3xZj3H8lze0kXAAAAAACATygOtlD7U/cHOgQAAO9c5z+d9ULkJ29O7jJ05tKhM2t/y32W3rT5kzWLlw/76vl+D77304PvVX2DdxVuf+2dnf7UBqmwAY+9/vLgWicJE06//fcjFsLa9u1Jn9Wj5/j394z3RHLyrTunfpm/58fdTw+967ZX1t32yqV72Ax/n4fV1Pu2mkuTHAVdR04a90jC/Y9cfiNvy/z2y5163jMTnxK1/9f8r/81/1LUFYeWvr+1lCeEsAXnzlfwKSkPf7oz8rl+s3f5G1oV95nVb3494tMZo+Z9NWpezauuE++OmvR/Gi/pamQTAAAAAADQFmFZMQAANAVb9Mcr/x4/bcmaHcdzS412lmVtxpLsU6k/f/n9AQNHbOmfzZg0c+W2o3l6m8tpKc346/u3pzy0YHNRfQeYXIGiio//+U+e3ubmONZm0PzzxxcvPPb81jKeEDbz/2Y9/83ezHIry7qtupwTWWUURXj99pdmPLt675kio4Nl3Q6LobQg49ThQ1mVfu73V//b1sLlbH7ng7Xb0zKKKu0sx7ltFYUXDm38ZMEDk94/aKpuhLec+u3X1Mwyq9ttryw4tfOLZyY+/U2my/NN677l8z7+44zWVFhQ7PQ7zzV405GlUybN+WTrocxSo51lXZbyvH/2Hc13ek9X4xsBAAAAAIA2h1KGRQQ6BgAAuN4WLlgwZOgQQsiUhx/xeTH4gUn+7/rfZsX8+sSIF1JdgQ6m6b779htCyP7U/djdAgAAAACgjcDMQQAAAAAAAAAAgDYKxUEAAAAAAAAAAIA2CsVBAAAAAAAAAACANgqnFQMAAFw9NvPTB5M/DXQUAAAAAAAAjYSZgwAAAAAAAAAAAG0UioMAAAAAAAAAAABtFIqDAAAAAPD/7duxCYBQEAVBBHMDsRR/USL2H1rGBTtTwYuXOwAAosRBAAAAAIgSBwEAAAAgShwEAAAAgChxEAAAAACixEEAAAAAiBIHAQAAACBKHAQAAACAKHEQAAAAAKLEQQAAAACI2qcHADDpe5/pCQAAAIwRBwHS1rqnJwAAADDGWzEAAAAARG3HeU1vAAAAAAAGuBwEAAAAgChxEAAAAACixEEAAAAAiPoBzYsHFKzhv6YAAAAASUVORK5CYII= ) ``` ridge.id, project_id ``` ``` ('1774086bd8bfd4e1f45c5ff503a99ee2', '5eb9656901f6bb026828f14e') ``` ## Any blueprint can be used as a tutorial ``` source_code = blueprint_graph.to_source_code(to_stdout=True) ``` ``` w = Workshop(user_blueprint_id='61d4dda0addc0e8a29404b9b') rst = w.Tasks.RST(w.TaskInputs.DATE) pdm3 = w.Tasks.PDM3(w.TaskInputs.CAT) pdm3.set_task_parameters(cm=500, sc=25) gs = w.Tasks.GS(w.TaskInputs.NUM) enetcd = w.Tasks.ENETCD(rst, pdm3, gs) enetcd.set_task_parameters(a=0) enetcd_blueprint = w.BlueprintGraph(enetcd, name='Ridge Regressor') ``` ## Execute a blueprint ``` eval(compile(source_code, 'blueprint', 'exec')) ``` ``` enetcd_blueprint.show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABCsAAAEMCAYAAADtWjx4AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3gUVdvH8e/sbgokJCGFYAg9CSGhIyhVlCa9CEiRpmJ7FLGgYH8sr4j6iAioiKhgBQsCVmoAUXpNCJCEXkNLICFtd98/AkjPbkjYBX6f68p1QTI7c8/Zc2b23HvOGcM/KMSOiIiIiIiIiIg7MJhucnUMIiIiIiIiIiJnU7JCRERERERERNyKkhUiIiIiIiIi4laUrBARERERERERt6JkhYiIiIiIiIi4FSUrRERERERERMStKFkhIiIiIiIiIm5FyQoRERERERERcStKVoiIiIiIiIiIW1GyQkRERERERETcipIVIiIiIiIiIuJWlKwQERFxkHf9Ycxev429Cb/wfGNfx1/o2Yhnpv/J8rVzebmBR/EFKCIiInKdsLg6ABERkavKUo0B773OgNoVCAsNJKCUD94Wg7zsExw5sIvk+FUs+uNHvvrxb3Znnfdaw4TJZGAYZsyG4cQxy1KzcR2iLNuZ48TLroorKQ8RERGRYmL4B4XYXR2EiIjIVePZgvfXfcvAkMsNLrSTtf0XXhr8GBM3ZFz5MUt2Y+rWj+lk2c74Tk14fnnule+zqLiiPEREREQux2C6poGIiMgNKoe4EfUoW7YspUPKElw+ipgmnRj08resO2bHq1IH3vzyTdoFuttQiOKi8hARERH3oWSFiIjcsHKzTpKdZ8Nut5F38hh7Ny9jxvjH6XjPRDbngjmsG490K3fD3CxVHiIiIuIu9HlDRETkHHaOr/yeWclWMDyIqRV1ZoEno+xAft57kGP75jG8uvmCVxqlqtHtmfHMWrKO7bt2s3/bBpbNmsirA+oTcrkBCZZg6vcZyUc/xhG/ZTupB/ZyYMdm4v/+nR8/eYvn72tK2MXu2F7h3PbAm3w7ZyUpO3dzYPsm1vw5hVGDbyG0yFalunR5XFEczp6zRzka93uStz6Zxryla0jZsYvU/bvZtWkFi797jPpnr1vqZDzelVoz7P3p/LV2M/v27WH/9gTWxf3M1/8bSG3vwm+bH3cIDfu/zGezl7I5ZRcHdm5mw4JvGPNYW6qWvNj2TpyniIjIdUwLbIqIiFzAitUKYGAymXFk4oOlQhc+mDaO3hFeZ20fSrVGXanW6MxuL2D41WfY51/wfLMyWM4+kE9pykWWplxkPVrc5s2KqUvYm3PW6wKb8NyXk3mqYemzvnkIonK9O3mobks6NH2CTg9OY3ueE6d9SZcuj8LEUZhzNoJa8ezbI7jN89zIPEIqEh1ocNxWuHg8qt3PNzNf5/agszIjHsFUjA0m3D+e8We9Z85sC2AENOCpz79gZJNgzGfO04vyNVsyqOYd3N37Gx7pO5yfdvy7homj5ykiInK908gKERGR83hHt6NtpBnsuWyO30KBy2FaInlwwhh6R3hhP7KSj4Z2pG5kBcqEV6Ne+wd5/adNpF+sk2mE0OWdz3iheRnM2Un89HI/GsdWJqTMTZStUpNGz/xB+sWWwTbdRM93J/JUwwByd/zO6/1bEF0hnLJRjej20my251oo3+l1RvW6qUhu9Jcsj8LEUdhzPs26nS8faEHNapUICQ2nfGxT2g+bRrK1MPH40uaJ4dwWZJCx/nMeurM+FcPCKFMxlvpt+/PEq9+z4UySxZltAVMod737Gc81DcY4sZEpT3WlXmQFQivWovngd5i/z4p3tT5MmPw4tc9LTBR4niIiIjcAjawQERExWfAqUYrg8CjqtejOI4/3p46nge3AbD78ficFfZldoukD/KeBD4Z1B5892IeRC9LI729nkbL8J97ZCLXbf0yn8+66HrWHMKJzWcy2I/z+bC/u/2r3mcEXuekHSN5xmNyLdNw96z/IM+1CME6u4M1+9zMm8dTwg8xkFkx4hAfKzOXXRyO5vU8nwr+dyE5nv413sDwKE0dhz/kM23F2bNrMrsP5r8o9sIUVBwoZj1GO6lG+mMhlzVdjmLZyb/655aSSvOoPkleddVyzE9sCnnUf5NkOZTBZ9zH98bt5fGbqqTqxn/WzRtN3p53ff32aOrUe5Jkun3PP9EOcc9qXOU8REZEbgUZWiIjIDcqTVmM2cfTQQY4dPLVewl+zmPrGYBqVMZOzZy4vD3iGmYcKesK3B7Vb3UGoGXLjv2biojQceya4hRod2xNhAev2b/nf9N0XmyVy0ePV7dSeyhY7WUumMnVzznl/z2LN3CUctBl4xtajtpdDO8X58ihMHIU9Z0cUIh7bUVIPWbHjQd1eg2gUdOE6JGc4sy0WanVsR2UL5G35hg9+Tb2gTmSt/5QP553AbvjRolML/PWQFRERkXNoZIWIiAh27DY7GCYMstn42cMMfvUXtp5wIO1g+BEZGYoZG8c2rGebo71voxTRsRUxYyd91T+sP79vfZnXRUaFYcagROuxJKeOvfS23iGEBpjgpLNDKxwoj8LEkVXIc3ZEYeLZl8qsST/wZLO+VKz/ODNXdCJuxnd8N+17Zi3bTebZb7/diW0NP6Krl8eCjSNrV7H5YuuG2NNYvXIree3r4VUthggzrCyS9UVERESuDxpZISIiN6gc5g6rTungMgQEh1K6QltGrcnEbngS2bgBIQ6Oj8AoiY8PgJ0Tx08UOGXk39f54l/KwMBG2uGjjo8wMHzw9XV0Y2+8HB5Z4WR5FCaOwp6zIwpVLnaO/PksnQa8w6yEY9hKVeWOAc/x8azlJCyazNO333TWtzpObHsmFjvH09IvUSdspKfl1xeTbyl89YlMRETkHLo1ioiIAGSt472ho1l2Aryq3c+YEU1wqO9rz+DECQATfqX9udzkgHNfl01WFoBBSZ+SDj1xJP91mWRkANg49GVvygSXIeBSP2EdGL+9kI+PKKg8ChNHYc/ZEYUulxx2znmb/s1rUbPtEP479S92ZZsJqN6R57+cxku3nv18UQe3PVMnDEr5+13iw5YJP39fTIA94wQn9JQPERGRcyhZISIickpO4icMe2c5GXgQee9bjLzVp+AX2Y+zOXE3VgxK1b+VGh4OHsx+jB07j2HDhH+t2lR2NMthTycp6QBWTATUu5moYpzQednyKEwchT1nR1xxuWSzf/XPvPdENxo0fYCvknPBK4r+A5tTwtltz9QJE3516lHtYrEYftS9ORILdrI2x+spHyIiIudRskJEROSMXBInjuD9ddngGcl9bzxEjYs9VvK816z75Xe254GlSl+G967o4IJQOayat4gjNvCo0Y8Hm/s7ONIglzV/zme/FSzV+jK0XUjRjlA471iXLo/CxFHYc3Ys1qIql6ztvzDx5xSsGJQsE3rZxS8vvm1+ndiWB5aoPvynbfAFsXjXuI+H7vDFsKcTNyuOYw7OOhIREblRKFkhIiJytpx4xr34Ocl54F3rYV7tW77Am2XOqo9485eD2EyBtHnrJ74Z2Y2bK/jjaTIwe/lRtnI5Ai7S4U2f8xGT4rOxmysw6KMveb1nQ6oGlcDLpwzVmvbmuYduw+8ir8taMoH3lqRhM4fRc9yPfPJoe+pVKI23xcDkWYqyUbfQuf+dRBfFqIvLlEdh4ijsOTvC+Xh8afLAczzYoT5Vgn3wMAzMXgFUbNiLhzpVxoyNozt2nEokOLNtfp1469eD+bF88A3/638rlQI88SwZSo0OTzP1yyeo6w1Z6ycy+ucLnxYiIiJyo9PTQERERM6T+c8Y3ph1F592C+a2J56k1Y9P8mf6ZbqTtv388OQgKgR8znPNw2n91Me0fuoi250/1D9nPe89/AJ1pv0fbcNu4T8fzuY/F9u93X5uZ9a6jckPP0Tlrz/i4TrV6PHK5/R45fx9r+D5RX+SuOPKF0O4ZHkUJo7CnrMjnI3HozrtH3iM/1QaxlsX7MyO7XAc7374F1ng3LYAtv18/+S9VAr6gpGNazP4vZkMfu/c15zc8i3/uXcMa7OdPVEREZHrn0ZWiIiInM9+mFnvfsSabDvmsO483rtCgTdMe9pK/terOa0feZuv5q1nx+HjZFvtWHMyObo/hfV//crU8ROYfd6zTbMTv+Ce1t156uPfWbX9MBk5uWSl7WHjgq8Z9dFCjtjAnplBxnk9d9vBeTzfvgVdnvmIn//Zwv70HKy2PLJOHGZH/FJmTv2ZtZnFXx6FiaOw5+wIp+Kxp7Lku2/5c802DhzPJs9mIy87nf1Jq/j105fp2aY/n2zNdX7b07s/tpy3e7Sgw/CPmLUihdQTOeScPMqe+IV8+cZgbmszjB93nPsaERERyWf4B4Vo5KGIiIjbMVF+yI+s+L9GEDecOj2nsP+6v2PfiOcsIiIiFzCYrmkgIiIirmL402TQfUQd/JuVm3ay90Aqx7JN+JWtSr3Wg3j+2VvxJp0/fpzDweul034jnrOIiIg4TckKERERV7HE0vnx4TwYfolneNpz2DHjOZ6dto8rX3nCTdyI5ywiIiJOM3uX9HnF1UGIiIjckCze+Ph64WnywKuEN94eHpjsOaQf3M6Gv2Yz+c0neezdOA6cvzDntexGPGcRERFxjkGC1qwQEREREREREfdhMF1PAxERERERERERt6JkhYiIiIiIiIi4FSUrRERERERERMStKFkhIiIiIiIiIm5FyQoRERERERERcStKVoiIiIiIiIiIW1GyQkRERERERETcipIVIiIiIiIiIuJWlKwQEREREREREbeiZIWIiIiIiIiIuBUlK0RERERERETErShZISIiIiIiIiJuRckKEREREREREXErSlaIiIiIiIiIiFtRskJERERERERE3IqSFSIiIiIiIiLiVpSsEBERERERERG3omSFiIiIiIiIiLgVJStERERERERExK0oWSEiIiIiIiIibkXJChERERERERFxK0pWiIiIiIiIiIhbUbJCRERERERERNyKkhUiIiIiIiIi4laUrBARERERERERt6JkhYiIiIiIiIi4FSUrRERERERERMStKFkhIiIiIiIiIm5FyQoRERERERERcStKVoiIiIiIiIiIW1GyQkRERERERETcipIVIiIiIiIiIuJWlKwQEREREREREbeiZIWIiIiIiIiIuBWLqwOQq2vkiBGuDkGK2U8zfiIxcbOrw3BKdHQ1unXt5uowRIrUm6NGuToEERERkWuWkhU3mKbNmro6BClmi/9aAtdYsiI4JER1U64/ylWIiIiIFJqmgYiIiIiIiIiIW9HIihvU8uUrGDtugqvDkCLSsGEDhj76iKvDKBJjx01g+fIVrg5DpFCGPvoIDRs2cHUYIiIiItc8jawQEREREREREbeiZIWIiIiIiIiIuBUlK0RERERERETErShZISIiIiIiIiJuRckKEREREREREXErSlaIiIiIiIiIiFtRskJERERERERE3IqSFSIiIiIiIiLiVpSsEBERERERERG3omSFiIiIiIiIiLgVJStERERERERExK0oWSEiIiIiIiIibkXJChG3YqZi9zeZ+eePPNfQ4upgRNyOUboFL34zm8WjWuHl6mBEREREpNgoWSHiFC/qDf2O1St/5a02QRjFcIRS5WOIKR+At1Ecexe5thneYcTUrExIScup9lf8bVJERERErj4lK8RxJj+i73yQUROnsejv5WxOWEv8338w+/PRPNevEeFOfc1pJnbwBH6fN4X/VDMXV8TFwjAMDMOESb0i92EOILbDQ4yaOI24pctI3LCKtYtm8t17w7nn1jC8XRyeT6cPSNy8jlkPV+Vitd0oczdfbohny5f3EOb0Vdn5tpQfTzzbLvGTOPZOl5eZM9QmRURERK4/GmcuDjH8anH/O+/xTPOyWM7qEHgGhhPbKJyYOj5s/vUfdmfbHdyjiYCKMUTedBTPa6qDkc2q93tR931XxyGnGQE38+iYd3j81hDMZ9Ulr9CqNGxflYZ39qDv9Jd56LXf2ZnrujiLz7XaloqK2qSIiIjI9UjJCimY6Sa6jxrPiNtKYz+0hqkfTuTb+WtITs3Bs3Q41es3487o/fyV5miiQq5ndevWoXWbNsQtjGP16tXk5hZjhsBcnr7vjmVYIz+s+/5h8vhJTF+wjh1pNnzCqtOs0yCG3d+S6r3+j4np++n+zloyiy+aa0weG8d0p+uHyVhdHYqIiIiIyHmUrJAClWz8EE+3CIRD83muzxNM25l35m/ZB5NZ/lsyy3/7d3sjqAUj//c47aqFE+rnhf3kIZJX/sGk/43jp80ZnJPSMEcx9Of1DD31X9vBb+l/x2sszQXDJ4IODzzCfR1vJbqMFycPbGbJTx8zemIcu8/u/3qHc3v/B7mvS1NqVQiiBFmkpe4hZctG5nz2LpOWH/v3mCUq0ebehxnSuTExYT7Yju5g1fzvmTD+W5annu6yGfjX6cXjA9vQIDaCSmUDKGGc5NCOP/jvgFdJuvtbfh16Ez8+cDvPLj4rkBIVaDXoIe7v3IQa4X4YJ4+yZ/MiJrzwGjN2WJ0rl2uYp6cXtzVvzm3Nm5OVlcWixYtYMH8hGzduxGazFemxfG97mMcb+2M/8DvD+z7Dz3v/7Xbn7FjDzHFrWbzmOaZ93Ieoe4bR89t7+WK3DTAIajKE5wfeRkzV8oQF+1HSA7KO7mTt3G94d8y3rDl67jvicH0sSg7V11Mu05YKz/lyKqgdOHNOpsDa9HnkIfq1rkuVIA8y9yXy9z9phJ4zVcZM5MPnt8lCxF3AdeST5ceupCBFRERExElKVkiBGnVqRRlTDms+fZsfzkpUXFJeAFXrRRHueer/vqFUbzGAt2uUIbfL08w65EC3vGRNHv30E4bVLXVmYRXv8rXp9NhY6oQNpcsLcRy1A97RDJk4iRENS581X92HoPAogsKr4Ll6MpOXH8v/5tg7hgcnTuKZhv7/LtYSGsVtfUbSuHltnrpnJLP2WgETZRr1oH/7mLMaSClCyniQfeIS8XpFM+TjTxlxS8C/+/YMJaJOeXxP2oquXK4x3t7e3HH77bRp3Yb09HQWxi1k8eIlbErYhN1+5efbuH0LgoxsVkz636n37nx2ji4dz5i57Rl7Zx06tgxj6he7sWEisFZrOt0Wc85F0Ce4Kk16P0+dqBJ07z+ZLaeru6P1sSg5XF+Lk5PlVFA7cOKcDL/GvDDlAwZFep9ZNNOrQh3aV8j/d3ZRxu3AdUTJChEREZGrSwtsSoFionwxWbexaMkeh4aL24//xdsDutKoQX0iqtei+q0duXfSerKCbueuFqXPXa3fuoWxXWpRuVoslavFUrXZayzNNRM14EUerVOSg3Fjubd9E6Jj63NLjxf4PsVKeNdH6RdhBsxUveclnmoYQE7KbF7ufyd1atYiouYtNH0pjsxzh3AQ0f9FnmjgR9am73mm1x3E1qhL3TZDeHP+PoywdrzyTGtKG+ecCHNe7sjNdesQUasRzXq+z7KLfkttpnK/F3myoT9ZW2bwQv921KtVh+oN7uDOAW/xx6kkhFPlch2xWDwA8PPzo0O7Drw9ejRfTPmCwYMHUT68/BXtu1qEDyZrEov/2s8lx2zY01i6eAN5hpmI6CrnLnBpT+e3kW2oXasWVWNvoWm/UczZb8Ondj/61fM4tZGj9bEgFmoMm0nSRRa0TFn8Ek08z962EPX1om3J+XhS1rxNW8/zNnWwnC7fDkxOnJOFGveNYECEJ+lrpzKsR/62te/ox7BJKzjk6AAdB+N2/DoiIiIiIleLkhVSoFI+BtiOcuSYgz0EwyCgZm/+b/IP/LVsBesXTOGVtmGYsRB6U3DBlc5cjU4do7Gkz+WN4RNZkHyM7LwsDm74if+OXcAJcySNGwZjMlehfYdYPK2JfPTEC0xZvou0HCvWnBOkHj5x3nSTSDp3icUzdwMfPPUq09cdIDM3h2M7ljLx6ZeZts9O6RadaXl278+ex9E9uzmcmYs1O529Ow6QcbGOi7kKHTvVwCt3PWOHvsRXy3dyNDuXrPQDbFmzhdTTxXal5XIdMFvyO/VBgYF07dqVjz7+kE8+mUjTpk0Ltb9SJQ2wpXE47XJ1007G0aNk2U2U8PE5dziZ3crx1IOkZ1ux5Z1gz8qv+b+pG8kzBxEdHZL/njhUH8sS8+iP53b8E2bydGwhn3RTmPpanBwqpwLageHEOZmr0bZVJUzZqxjz1Gh+3pC/bfqetcz68k+SHB1Q4mDcDl9HREREROSq0TQQKVDGScDkT4C/CQ4W0Esw/Gj+wudM6lMRjzP9KC8qlAewYjI5UOU8y1Ml3ISpRFs+WN6WDy7YwEpY+E2YLMFEVjJj27WUhUkFTMz3rEhEuAnbrmX8tf28c8hYzeI1WfS5syIR5U1wpOAQz2GpeCqO5SzdeYnyKYpycdDIESNgRJHtrthYLPnnfFNYGGFhYWd+HxwU6PA+jmfaweRPkL8JDl2qbhr4lC6Nt2EnMyOTy09ksrI3eTsZ9lh8fX3yR7s4VB/LYs4oKNpLL2hplLmbqfNeouHpXxRnfXUgnoJdpJwKagfOnFNmGJXKmbDtXs3KfUW5zsnl4nbgOiIiIiIiV42SFVKgrdtOYq9WmVsbhDBh62WG2wNGYCsGdauA+egyxr04mq/+SSb1pIXgls8zY0xnxw5otxfwbaaBVwkvDJMHFhOQl+dAZ6sYv4E2TPnz3C+zBkORlIsD7MCMGTNITEwssn06KzIygh539XBoW5vVislsZv+BA5QNDQXg0GHHe9+bkzKxR1elaaNQPkzee/G6afjRqGkNLPY8kjYX3DG35+SQYzcwTi9e4FB9tJD4VncixjkcegHcf1LQBeVUYDtw4pwMc/6oB8NU5CVxQdxOXUdERERE5GpRskIK9Pe8f0hv25JbhwylzZwX+D310ukKU3BZynpC5pypfDA3kRwAcjmcmn7egnh28vLysFOSkiXP647k7mH7Hhs2/5+5v82LLLjUsyYtddl7yIapws00DDOxcddl0ig5O0jebcNU8RaaVDSzIeWsbolPPZrV9YacnaTstuH07KhT8ZoqNKRReTMbzv/WGGfK5TQz5kK0TgNITExkyeIlzr+4iOTm5MJdl/57Xl4eFouFffv2M3/+fOLi4qhcpXL+iBAn/f3bAg536EKDIU/Saf6z5zwNJJ9B6Ub/YVjrAIysVfwy7xIJjctxtD4WJafqq3HptnQ1FdQOnDmnnJ3521ZqTIuq49mwpRhHPOTud/w6IiIiIiJXzY0wTV6u0NHfP2RSfDamsM6M+XYCw7s1oGpwSSwmM56lyhB1S0ceeqIb1c1gO3yQg7lQ4pbu9Ksfhq/FAJMHvr7e52XG7Bzcn4rdfBOte7amsq8Fs3cgVW+OJYzN/DEnBVtwJ14ZfR8tY8ri52nGZPamdHgsLRpUzN9X3ibmzN+Hzasuj787nE6xofh6elG6YgO6t67OOWsEWrcwc2YCOR41eex/L9KjViglLZ74V2zMA2//l143GRyLm8XcI4WYoW7dzO9/JmP1qM3jH7zKPbdUorS3GbOHL2Wr1aFakMmJcoGc3DzsRgD1WtxKeMlCrnngZvLy8jubx9LS+OXXXxn+zDPcf//9fP311+zZs6fQ+z2+8CPeX5qGUfZO3v76I0Z0b0DV4BJ4WLwICK9F+4ffY/qEvkRactn61RimFaYjanWwPhYlp+rrZdrS1aw+BbWDgK2On5N1M7NmbyLXEsN/xr3FkGZVCfTOL3P/4NIUaU7GmeuIiIiIiFw1GlkhBctN5MOhz1Jmwhv0q96MR0Y145Hzt8mLx/TzTDZtm8+0eY/SrMMdvPT1Hbx0zkZWtpz1751x80kYWpNa3d9hfvfTx1rL/7Xvz6RP32Dy7R8ypPWTTGr95LnhrBlN675fsMOWxYqJ7zCz5Tt0rT2AsT8OOD+oc469deqrjGk+ieENevL29J68feZvdnL3/MYrb/1BYXIVkEf8p6/zcfMJPFKjK69N6cprZ3adwayhzRk6x/Fy2b0pkWP2aKIHfMgfZZ4m9vHfCxOUy1mtNsxmE5kZGSyMi2Phwjg2bdqEzVaE31xbd/LV048TOOYdht7SmAffbMyD529jzyBx+ks8OGYNhRsUkcdGh+pj4U7h4pypr5dvS5/svFRgp54GMuwif8rbyOhOffkwxZmYHWgHTpzTli9e4Z3GnzCiYVuem9SW58472uUfXeoMZ64jIiIiInK1aGSFOMS6dy4v9e7OwNen8vvqbRxMz8JqtXIy/QDJ6xbz/Sff8NdRG9iP8NsLQ3jq0wVs3JtOttVKXnYGRw/uZsu6ZfyTlHZm/r916xcMHf4ZC7YeItNqJS/zMClrkkg1DOzHVzDqnr4MmzCbf7YeJD3LijU3g0M71hO3ctepaRRgS53DM/0eZvSPK0g5nEVeXhaHU5bz89wEMu1gs5/VUTuZwEdD+vLIB7+wcscRTubmkHFwC4u+eZN77h7BzAumEDjOfmIV7w68h6ETfmXl9sNk5FjJzTzCroRVJKd7YDhRLplxY3hy/Fw27j/Ont37Ch2TK2VlZbFo8SJeefkV+vTtx/jxE4iPjy/aRMUp9qMrGHtvV7oN/5Aflmxi99FMcvKyOZ66jZV/TuWV+7rR7cXf2XkFMwkcrY9Fyon6erm2dDUV2A6caYMnN/HJkLsZ/M73LNl8gPRsK9a8LI4f2kXC8rn8EJdCUU0Oceo6IiIiIiJXheEfFKIns91AfvllNgDLl69g7LgJLo6muBiE9JrI4ldvZtmLLRk0/ch1//jBhg0bMPTR/PEub44a5dI1K3x9fcnJySEnx/EufNNmTc+sWTF23ASWL19RXOGJOOjc68jA6Y4t/Dr00Udo2LABAB06dCzOAEVERESuXwbTNQ1ErmmmMvVpVxu2xm9j76E0siylqVK/M08/fAuetmTWbEi77hMV7ubEiROuDkHEKY5cR0RERETk6lKyQq5pXnV6M2pse3zPH+1ut7J39od8vUUPIxSRy9N1RERERMT9KFkh1zADz2ObWbC8CnUiK1DW3wuy09iXsoHFM79g3FfLOKip5iJyWbqOiD9bd9oAACAASURBVIiIiLgjJSvkGmYnbfkkhg6Y5OpAROSapeuIiIiIiDvS00BERERERERExK0oWSEiIiIiIiIibkXJChERERERERFxK0pWiIiIiIiIiIhbUbJCRERERERERNyKkhUiIiIiIiIi4laUrBARERERERERt6JkhYiISDHw8vZ2dQgiIiIi1yyLqwMQERG5Hn0/fRo7d+xk85bNbNm8lc1bNrNz506sVqurQxMRERFxe0pWiIiIFIPXXnuDiIiqREZGMPjeQfj6+pKXl8f27duJT0ggKSmJpKQkdu3chd1ud3W4IiIiIm5FyQoREZFisHz5MpYvXwaAyWQivHw4ERERREREEBkRQfv27fCweJCRkcHWrVuJj08gKSmZzYmJpKWnuTh6EREREddSskJERKSY2Ww2du7Yyc4dO5k/bz4AFouFSpUqERMbQ2REJM2aNaVv3z4YhsGRI0dISkoiPj6BhE0JJCclk52d7eKzEBEREbl6lKy4QUVERDD00Ucc2tZssWDNyyvmiORKlC5d2tUhFJl2bdtwa8MGrg5DpFAiIiIc3jYvL+/MVJDTfHx8qFipIjHVY4iNjaFb924M9h+E1Wplz549JCUlszVpKwnxCaSkpGCz2YrjNERERERcTsmKG1RgYGkaqkMobigy0vHOnsj1JiMjg4T4BBLiE/j++/zfBQYGEhERSUREVWJjYxg0cCBeXl5kZWWRkpLC1lMJj6SkJHbu2OnaExAREREpIkpW3CA8PDwICQlxdRgiIuKkI0eOnLP+hdlsplx4uTPrX8TGxNCpY0dMJtOZ6SNbtyaRlJRMYmIC6enHXXwGIiIiIs4z/INCtAT5dW7QoIH06NEDwzAAsOblYQcsZgsYF3+N3W5n2rRpTJky9eoFKiIiheLt7U2VqlVOLd6ZPwqjQoUKQH6yIyE+gfhNp55AsmUrObm5Lo5YRERE5DIMpitZcQMoW7Ysn3wyEZPJ5ND2drudmTNnMXHixGKOTEREikvp0qWJjIw68/jU6GrR+Pn7nbP+RXx8PAmbEti9a7fWvxARERH3oWTFjWPoY4/RqlVLzJZLz/yxA3abnQUL5vPee2Ow21U1RESuJ4GBgcTExhATE0PkqWkknp6enDx5km3btp1Z/yJ+40YOHDjo6nBFRETkRqVkxY0jOCSEyZ9+gtl86WSFzWbjn3+W8eabb+obNhGRG8DZ61/ExuQnMcLDwy+6/kVCQjwnTpxwdcgiIiJyI1Cy4sbh5eXFG6+/TlS1KMxm8wV/t1qtrFq1itdffwOr1eqCCEVExB2UKFGCylUqn1n/IjYmhtCyoQDs37+fhIRNbE3amp/I2LyV3DytfyEiIiJFTMmK65+nhwd3tm9Hr549KVmyJGazGct5U0GsVivx8fG8/NLLWnRNREQucPbjUyMjI6hevTqlSpUiLy+P7du3E5+QcObxqbt27tI0QhEREbkySlZcvywWC61ataJPn974+/szb948vvrqa7p370bnzp3OTAex5llJSknmuZHPkZWV5eKoRUTkWlG2bFliYmNOjcCIIDIqEg+LB5mZmWzfvp2EhATi4zexZXMix9LSXB2uiIiIXEuUrLj+mEwmGjdpzKBBAwkOCmbevHl8/fU3HD58GAB/P38+/+IzPD09seblsWPHTp4dMYLMzEwXRy4iItcyi8VCWLkwYqrHEBsbS0REVcqXL49hGOesf5GQkMCmTZvIzs52dcgiIiLirpSsuH6cTlIM6N+f0NBQFi1azFdffcX+/fsv2HbgwIH06tWTXbt3M/zp4Rw/ftwFEYuIyPWuZMmSVKpcKX8Bz+oxxNasQemAAGw2G7t37yYpKfnf9S+2bCVXUxFFREQElKy4HhiGQYMGDRnQ/x4qVqrI0r+W8sUXX7B3375LvsbH15fXX3uV1157nSNHjlzFaEVE5EZ39voXsbExVI+Oxsvbm+ysLJJTUs48PlXrX4iIiNzAlKy4ttWpU4d77x1M5cqVWfrXUqZ++SW7d+926LVms1lP/RAREZczmUyElw8nIiLi3/UvIiPx8PAg48QJtiYlER+fQFJSMpsTE0lL1/oXIiIi1z0lK65NderUYdCggURERLBixQqmTv2SlJQUV4clIiJSJLy8valatcqZx6debP2L+PgEEjYlkLQ1iZycHFeHLCIiIkVJyYprS0xsDAMH9KdGjZqsXbuWzz77nKSkJFeHJSIiUux8fHyIjIwkJiaGyMgIqkVXw9/PH6vVyp49e86sf5EQn0BKSgo2m83VIYuIiEhhKVlxbYiJieGee/pRu3Zt1q5dy5QvprJ5y2ZXhyUiIuJSp9e/iI2tTkxM/mNUPT09ycrKIuW89S927tjp6nBFRETEUUpWuLfo6GrcfffdNGzYkISEBKZMmcqGDRtcHZaIiIhbMpvNlAsvd2b9i9iYGKpUqYLJZDrn8alJSckkJiaQnl50T8Nq3rw5uXm5/L307yLbp4iIyA1LyQr3VKlSRfr07kOTpk1ITNzM1C+nsm7tOleHJSIics3x9vamynnrX1SoUAGA/fv3k5Cw6czjU5O2bCWnkI9PfWb4cG5rcRurVq/mw/Efsm//pZ/KJSIiIgVQssK9VKhYgX59+tKkaRO2bN3Ct99MY/nyZa4OS0RE5Lpy9uNTIyMjiI6ujp9fKfLy8ti7dy8JCQnEJySQlJTE7l27HVr/4vPPPyMkJARrnhUM+O6775g+bXqhkx8iIiI3NCUr3EOF8uXp2asXLVrcxq6du/j622/4a8lfera8iIjIVRIYGEhMbEz+Ap4REURERuLp4UFmZibbt29na1JSfhJjYzxHjx4957WlSpXim2++xjCMM7+zWq0cOXqE8R+MZ8XKlVf7dERERK5tSla4VmhoGXr16kWbNm3YvXs307//noULFmoFcxERERc7e/2L2Jj8JEZ4ePhF17/w9PRk5MhnL9iH3WbHMBmsWrmKcePHcfBgqgvORERE5BqkZIVrhJQpQ++7e9G6dWsOpR5i2vTp/Pnnn0pSiIiIuDEfX1+qRUUSFRlFVLUooqKiKF26NGlpafiULInFw+Oir8vLy8NutzNt2nSmT5tObp6mhoiIiFyWkhVXV3BwMN3v6k779u04euQY3333HXPmzMFqtbo6NBERESmEkDJlePbZZ4iuVu2caSAXY7PZOJSaygfjxrF69ZqrFKGIiMg16GLJipEjRrgqnGtKQuImfp7xs0Pb+vv50/2ubnTu3Jn0tDR++Oknfvvlt+vum5UuXbsQE13d1WFcE94cNcrVIdzQoqOr0a1rN1eHIVeZ2l3BdB0vnFsb3YrFYnFwaztgkHowlW3btpGTk1OcoYmIuL1r8f6s+2XRu6B/bTD9gjtr02ZNr2ZM17SfuXyyws+vFHfddRedOnXiZFYWX331NTN//vm6XRk8Jrq66o+jrr1r8nUlOCREdfVGpHZXIF3Hr4b80RchZUIIKRPi4lhERNzANXh/1v2yeJzfv3b0awBxQokSJejQoQO9evXEmmfl66+/YebMmfr2RERERERERMQBl0xWLF++grHjJlzNWK4JX0757JJ/8/b2pmPHjvTq2QOrzc6MGT8zY8YMMjMzr2KE7uGeAYNdHYLbGfroIzRs2MDVYch5xo6bwPLlK1wdhhQTtbvC03XcMX373E3bNq04cuQYBw4eYP+BA6SmHiL1YCqpqYc4dOgQx0+ccHWYIiJu5Xq6P4/YWtHVIVzTRkXuuOTfNLKiCHh5e9O2bRvuvvtuvDw9+eWXX5g2bToZGRmuDk1ERESK0YyfZ/HNt9Ow27VeuYiISFFSsuIKeFg8aNmqJffc048SJUowe/Zspk//nhP6BkVEROSGcCOOnhQREbkalKwopLJlyzL5s0/x8fHh9z/+YPp30zh67JirwxIRERERERG55ilZUUhVqlZh1sxZTJ8+XUkKERERERERkSKkZEUhrVqxkomffOLqMERERERERESuOyZXB3CtytZjSEVERERERESKhZIVIiIiIiIiIuJWlKwQEREREREREbeiZIWIiIiIiIiIuBUlK0RERERERETErShZISIiIiIiIiJuRckKEREREREREXErSlaIiFzHjNItePGb2Swe1Qqv6/B4ci0xU7H7m8z880eea2hxdTAizjP8af78ZCY90oAgwzUhmGOH8kfCev56sSGerglBBPAmsut/+XpsHyrrcn7NMby8eKxtENMbe7n9dUTJChGR65jhHUZMzcqElLRwNT5bX3g8L+oN/Y7VK3/lrTZBVyUGcV+lyscQUz4Ab+Nq1wTVQ7lSBqWaPM5r/W6mSpCJHJfEYCamTSuqcoC5f6x1UQwiALnklSpPjdaP89rd5TG7OhxximE2ExlkIdBinLofGtSoHcjsu4MZUcHkVvfIYktWeN/6ONN/mcPKFavYnLCBpA0rWLP4F36e9BbPD2pFtH9hq7WZ2MET+H3eFP5TTU3jeqS6I+7Lm4q338/oz37i7xWr2Bq/mo1L/2DW5Ld4tmdtShugenYhwzAwDBMmd7r7SZHx6fQBiZvj2XaJn8Sxd+Jd7FEU3O6Ksh76dPqAxMSVzB7ekICL7s+D5q8vJjnhF0bUcvY64Pw15IL7Zvwq1v/1O7M/H82IPg25yd2/OrsWWCIZ+GQ3yh3+lbc+WM5x++m6v45ZD1d1sLNm4FO1HS9NW0ry+vfpVMLJGMzVademEhxYwK9rL0xVOB9PcSvs/dCRe+2NyX3eYyvbvhnFxAQvbn3kEVr63cBvihO8yvoyvlMwM3uVYX6/UOL6luH3HsF81tqfx6K9CHfxKBUD55MDkdUD+KJrafqXLo6IoNiKxBwSQc2IsH+HAZtLElCmEgFlKlGrWQcGP7SeKS8M5//m7iHPqT2bCKgYQ+RNR/FUu7guqe6IezJTsef/+P6/zQk2/1uBLEHh1GhSjkjLWqb8sA7sqmfnymbV+72o+76r45DrW0HtrhjqoVGC2Hv/x9h9A7n/y+Qi/Jbb+WvIBfdNvCkVXJ7Y4PLENmpH3+6TGHLfWJal24ssyhuNT9MB9K9ukDB2EnOPOVmO5lJUvrUtd3W7i5531qSMhwHZzsdgiW1D24qw76s/WX1NDKsozP3Q0XttsQQszsjbypcT5zJ4TFvu7zaBuV/swubqmNycuYSFaH/zv1MvDAMfbzMR3mYiQr3pHHWS1+amsyjzakdmZ+O6I3RY5+zrDPxLeVDJx1Zs00mKOX+TR8L4PvQYv4ksPPEpXZaIWo3p0G8Q/ZvUZtB7H2M80IdX/z6ua46b6datG6GhoSyKi2NTYiJ2+9V+h1R3xDE1atSkU8cOLIhbyKoVq8jNyy2eA1nqMPCRpgRxkPnvvsSoH9ew41gunoHliW3QnFonF3JAd2lxIx07dqBCxYrELYxj06ZN2GzFWUHz2DimO10/TMZajEcBMHmVIijAm7zjRzma6VzKumjZsdr9aPrsGEam3MOrS9NcfD/KI2Fcb+6akEi23UwJ/7JE1GvDkKf/Q4ea9/Lq4Dm0fz+h2N+folDQe3zV64Dhzx3dWxGcs4pxM1KcLkNTaGfe/Ggkt3hC7v4NrLfFUivI2SAs1GjbkorsY8of6yimO12hFdl7coPea119XStcm7NzLO4Hfj3Qlt7dOxL55YdsvhYuMBfRuUtnwsuVY2FcHJsSNhV7v2fr+sM8vCGPHMDL00z5YC+61/alfWAJhtXKYvk/OWQVawTXjmIfbGLLzSbHasdONicO7WDt/B2sXfAr84ZPZvK91bhnRH+mdZ/AJisYQS0Y+b/HaVctnFA/L+wnD5G88g8m/W8cP23OOPdDgDmKoT+vZ+jp4xz8lv53vMbSXCf3Ixfl5+dHp04d6dSpI4ePHGHe3HnExS1k+/YdVy0Gl9Qdnwg6PPAI93W8legyXpw8sJklP33M6Ilx7Ha3TwYCgJeXB02bNaVps6ZkZWWxZPESFixcyPr164u2c1YynIpBZmw7ZzP20yVsPXVDzjmYzLJfkll2/vZXfI0yCGoyhOcH3kZM1fKEBftR0gOyju5k7dxveHfMt6w5eu7VzBRYmz6PPES/1nWpEuRB5r5E/v4njdDzxvQ5enz/Or14fGAbGsRGUKlsACWMkxza8Qf/HfAKvx2xO3g8M5EPf8uvQ2/ixwdu59nFuRj+Xfl0yRvcfrE0fM4/vNDyfr46aFd7vEJ+fn50aN+eDu3bc+zYMebNm8/ChQtJSUlxdWjncPT6bQq8mSEvPceDraIo7WFgt+dyfOccXh30LD/sPbXRJdvdhfXwjBIVaDXoIe7v3IQa4X4YJ4+yZ/MiJrzwGjN2XO6Tdx7rpo5nf9vH6D/6v6zv9RQ/7S34k7rD9foy15BLseXlkGu1YyePzKO7WT9vMk+lhVBrygAqN6hHqCmBvTYn4/AO5/b+D3Jfl6bUqhBECbJIS91DypaNzPnsXSYtP4bdgeuFI8e7/HtsK/Dv+e9nJdrc+zBDOjcmJswH29EdrJr/PRPGf8vy1NPvT8HxnqNkQ1o38iVvwwLmF6KnbDuwjIXLN3Jo/Rd88PkaGo793flkhSWW9q3Lw96v+HXdlV0AHXkvrk67vEhwzt5rPcNo2u9+7uvcnDqVQyhlyeXEkf3sSN7C2lkf8eqPW7DiQZNX5jCl11HGde/Be4ln1YNu41k+qhF/P9+Swd8fwe7wuV+dOl9oDrWDImhzWWuZ+9cx+na9nZYVJ7I55drMVviVKkWHDh3o0KEDR48dY97cuSxcGMe2bduK5Xh2G+Ta8wcIZWVb2bonk7dPGER28CUixJMKRg77gkowuLo3tQMtlCtpwhs7R49nMWZuOnFZYHhYuCPWh16VPKlawiDrZB4rkzP4KD6b/WdVHZO3B51r+tClvCcVvOFkRh6rD9gIPm+0U6UagXxW28wfCw4xau9Z10CLmabRvtxdxZMoHwOT1c7+o9lM/SedP4+f2sawMKhDKINO/dd28iRP/JTO6iL4GO6amTH2NP754C2mt53EgMh2dIj+mE3xVsgLoGq9KMJPf4D1DaV6iwG8XaMMuV2eZtYhB9MMRbWfG1xuXi4eFg+CAgPp1r0bvXr1ZN/efcxfsICFCxeyd+/egndS1Iqz7pSsyaOffsKwuqXOzNfyLl+bTo+NpU7YULq8EMdRVR235u3tze13tKBlq1Zknsxk8aLFzJs/r2iy5Cf3sfuoFVP5lvRv9wNbZu/gZGH35VB9NRFYqzWdbos550LtE1yVJr2fp05UCbr3n8yWU19wGH6NeWHKBwyK9D6zMJJXhTq0r5D/73NGHDt4/DKNetC//dnHL0VIGQ+yT9idO15hqD0WidzcXDw8PAgICKBLl87cdVd39u3bz/z584mLi2PPnj2uDtGx+mgqS89RH/DMbX4YeRkcPnACwzeIgDIeZKfZoLCzt72iGfLxp4y4JeDfebqeoUTUKY/vyYI/ZVn3/MbIp0pRefJgXn33XjYP/oSEy30d5oJ6bcuz5g/NNpn+PUdH4/COZsjESYxoWPqstT58CAqPIii8Cp6rJzN5+TGsBVwvHDqeUcB7XGAdALxjeHDiJJ5p6P/vuYZGcVufkTRuXpun7hnJrL1WCrq+nc9SrS61fWzsXrvunE6Aw6xJfHRf7/x/m0JpWIhdeNRsQ5tw2D31T64oV+Hoe++qdunMvdY7miEff8KztwTy74wRC/6hlakVWplqx//kzR+3OD+aqAjukUVS5wvL0XZQ0PvnSJsjm/Wr48m9qyH165SClGOFj9vFcnJz8fTwoHRAAF26dqVHjx5n+j1X5X5pcM7ClkFlS9CtosdZ9csgsCTk5AIWDwbeUZrBIcaZ99jL14OWtQOI8TnGkH+ySQMMT08ebRVAjwDjzL49S3lwe6lT51xQTGYLvW8vzcOhZ90/zAYVg82UvEoDgFz3NJCT61i4LA2bqRzVI30AsB//i7cHdKVRg/pEVK9F9Vs7cu+k9WQF3c5dLUqfuzKpdQtju9SicrVYKleLpWqzfzO0Tu1HHOJhyW8qN4XdRO+77+aTTyYyceLH9OjRg8DAwKsbTLHUHTNRA17k0TolORg3lnvbNyE6tj639HiB71OshHd9lH4R7rFclVye2WzBMMCnZElatW7J26NHM3XKFB548AGqVq1a+B3nrmLyuCUcMipy1zs/Mf+rV3mobTSBl0r5FtU1yp7ObyPbULtWLarG3kLTfqOYs9+GT+1+9KvncWojCzXuG8GACE/S105lWI87iK1Rl9p39GPYpBUcOu8zj3PHP86clztyc906RNRqRLOe77Ms17njnc+eNoN7a8aeKZvKsW0ZNms3ubZM4r+ezB+HTGqPxcBy+jp+U1l6976biRM/LsLruIUaw2aSdN7imilr3qZtARNZHamPRqlbaHNLKWwbP6Zbo0bc3PwO6tdvyK3d3mHJ2XN7L9PuLmSmcr8XebKhP1lbZvBC/3bUq1WH6g3u4M4Bb/GHQ19s2DmxahzD3luFrc4jvPdEA0pd8kOGk/cZp87lXIbZE5/AcGKb9eH1l3pSwWxl56rVpzrajsZhpuo9L/FUwwByUmbzcv87qVOzFhE1b6HpS3FkXqx4Lnq9cOx4Bb3HBdcBMxH9X+SJBn5kbfqeZ3rlX5fqthnCm/P3YYS145VnWp+7OONF472gNPGtVJmyZivbU3a6aE6+B7XbtiScvcz5feMVTAFxvA66rF06fK81EzngFZ66pTR5O//k/x7oQoM6tagaU4+6d41n3RV0pK78Hlk0db5wHG8HV97mAOwc37adAzYPKlUpX/hCdzNn93v6FPn98l/GqTUrqpcryfDGPkSY4NihXHaeub7aWbLsMJ2/PUiLb1Lp9VsG66xQJboUA0IMDu85wTOzUmn59UG6/pbOb2l2ylbxoUtA/qurxZSie4DBiUOZvPpbKm2+Pki7GUd4NT6H8weQXUz5KD/uDzWRfewk7845RMdvDtJqWioD5x5n0dmJeXsen/9ygGZf5v/c9kPRjKoAV42sACCPI0fSsBulKOlbEhPp2AyDgJq9eeb5W4mpeBOBHhnsO2TDjIXQm4IxccSx7GhR7UcuymzJv5GVCyvHgIEDGDRoIFu2bMbDcrWWGy+GumOuRqeO0VjS5/LG8IksSMtvwQc3/MR/xzal7ZiWNG4YzLitB67KGUrRsJjzL3GlA0vToX17unTuzL69+0hKSS7E3qzsmD6M7gcGMXxYf9rVv4tnb+7O0L0r+WnyeMZ+s4IDjn44cqa+2q0cTz1IerYVOMGelV/zf1PbcfvwGKKjQzAt34vNXI22rSphyl7FmKdG8/PuU3eIPWuZ9eWf9B7YgLqFPn4eR/fs5nBmLpDL3h3pYI517niXYwqh1csfMbpjMNu+eYLBb/3FISOGgWqPxcpszr+Oh4WFMaD/PQwaNJCtW7dgMXsU8Mpi4EB9tNnt+cOzQ6JpGB3M5hUHyLJnk7ptd+GPa65Cx0418Mpdz1tDX+KrbadqffYBtqw5RPSjP/L3Y9X+/W7Ymsz4nt14J/78u0kOW6Y+x6sNvmV0/9d5/p/ejFxw4iLHc+w+M2HrkUKe0KmE0bDzf28nY9MUXv40Pn9RakfjSPGjfYdYPK2JvP/EC0zZfLp3eYLUwycuPqX2oteLGIeO9+Gsy7/HRkF1wBxJ5y6xeOZuYPRTrzI9Of99ytyxlIlPv0zF2R/Rp0VnWpb+g++PXCbeC5gICg7EZMvk8OFM10wl9qhJu1ZhsHsqv2+4gmEVDtfBAw59rip8uzQT/eh0Zl+ifTl0rzVH0blzDJ55Cbz36DN8suV0uVhJP3yMk1fyRl3xPbJo6nyhONEOfsi7wjZ3ugiOHOKI3UTl4Kv85eVVYjJf2O+50vtlVJ0g4upc+Pvc41l8sD773/Uq7HbSMqwczbMDdg4cBwwPWlbywJyTxfi/Mvj71PCIw4dP8v56T5o386J+qJkv00w0L2/BZM1h8pLjzDl9WzqRy7zN2XSq7kns5YI0LLSs7IGnNZePFqUz4/Tl0WpnW+rVS9u6MFlhITDQH8NuIzMjE7vhR/MXPmdSn4p4nElZelGhPIAVk8nBUItqPwVo2qwpvzSbXST7ckfbt28veCMDzEb+4JyoatHnZJpLlfLj+PGL3fSLQjHUHc/yVAk3YSrRlg+Wt+WDCzawEhZ+E1A0naNffrl+6467Ov2tctmbynJT2E1nfh8SHOLEXnLYvWgijy/6gjfrt2fA4IH0veNm+j7/Ka2avk7f/0wjuaCExRXXVyt7k7eTYY/F19cnv915hFGpnAnb7tWs3FfADaQo2oszx7tsLKVo8Pg4xvSsQOqvI7nv9UWk2oASxdMeb8R2V9CCs4ZhYD7VNqKiojh7EGpAQADHjjk6pLeQC2w6WB/tx5fy07xD3N7hNp6bMpenju5g49pVxM38ksm/byWjMB0TS0UiK5mx7VrO0p1X+BWGdS8/vvxfGse8R4//jiRuw4tknL+Ng/cZE4VNVpzldCfDfpyVk5/nmfEL2HZ6KISjcViCT5XPUhYmXUEH2cHjGQW9xwX93bMiEeEmbLuW8df2897PjNUsXpNFnzsrElHehLNF7OntiUEOOS56AodH7Ta0DoNdn8+5ohEDDr/3xkmauqpdAg7daz0qUDXclF8/k4twEaOr+JmywDpfmPJzoh3Y119hmzsVnz0nh1w7eHoV/ktLV9+frXl5WAuaLnxWvycyqtq5/R6zleNW50d82mx2TubY2JeWy7o9WczYms32gqqz2Ux5XzBZvHmllzevXGSTMr4mTCYT5XzAdiKX9RfckBxgMlPJD2wnclh9vODNi4vrkhUlatPiFn9Mth0kbs2AwC4M6lYB89FljHtxNF/9k0zqSQvBLZ9nxpjODu/WCGxVJPspSGJiIj/NmFFk+3M3DW5uQLnwcgVuZ7fbz6wFkJaeRkBA/kN2iy9RQfHUnVMf7C7NwKuEJt+IXQAAIABJREFU12W3cMabo0YV2b5udFWqVObuXnc7tK3VasVsNpOamkpISH6SIvVQaiGOms3+VT8xetXPfBzTnVfHvECn5o/z2B2/MuzPy6/WUBTXKHtODjl2A+P0JHLDnD+nzzAVOM3t/9u778AoqrWP49/Z3RRCTeg1IAklkS5FQaSI2AvFQtWrqJcXsQCKIlxBUewIClcEFRD0googRbqhSVFBgQBJ6AlgEhLSSNky7x+hE2ATkmwgv88/98pM5jy7c87MzjNznsmXY2Qu2rs0G4Hdx/HZ0yE4toznmRGLiD79u6qAxmNxG3ft2ralzc1trryiCS7ThWEYJCUlUa5cWYBcJCryzu3+aMaz6LW+pG7tyV1tmtC8WSOad6xDiw4daWDpxqBFSXlo3JJdhyHHH6hOdn/ajaBP3d+cGb+KN0f9QIvPu/PGiHW8f+FEezf7dd7H1LkJI2+C+01izqutqdugEmSe07K7cVi8sFkAh+PqnkZ1t70r7uMrLF9VcJN8szKyMPHGu7AeID2PN827dqIah5m2bEcuX9l+ATf3haXAx6W74+sy59o1ZvaUHKfLraddstf2wdf38v0kX86R+djnc5+vyMU4uNoxdyo+w9sbLwOyMvOezfP0+bldu7a0bn3l8+W51z3JycmULZc93yK3iYqIbccZsMORt2ll5pXf3OtjNTCMs/Us8lb34WydC0+WCPNMssIoS5tBL9OzugVHxFIW7XJiCapCFW84uXwmE1fsPlXww87xuOQLCrWZOBwOTPzw87t4QFoquLudqxMfF8+6tevycYtFS90bLj233zRNnC4XNquVAwcOsGz5ctaGreHZZ5+l3a3tCjawguo79hgOxLhwlZ3PU3eMZHUBv9/4eu47hS395El4+NLLnQ4nVpuV+OPH+XX1alYsX0lgnUBeHT48H1p3kRQ+j/FzunP3sBCCgqpiXba/8I9RWYfYG+3CUvsWOtT9jO0Rl07L50v7uWgvZwalWz7Pf0feRrlD3/PsC1+y89wLuwIaj8Vt3NWqWZM2bS7948vhdGKzWok5EsOyZctZuXIlzz7zTMEfx7FyaoZW7vpjxmHCZn5E2EzAWpoG3cfw5Rtd6NC1FX6Lll123OXoVD+z1GrFzTWtbL/wLmSumZxY9xEjvm3F148N44V//DA4J3Hvdr+25f6zXCSLyG9eY0ST75hwzxA+eGobvT7fnf2duhuHrRlH4l1Yat1Eq2oWdhzO41NUuRnPl93Hi0m73PJf9mcflwJb0zbQyvZz30pQsjm3NvOFrEPsi3aRu5/uLo7HJ+Cy1KN8eT8MCvkVtd5NuPv2KnD4a5bsuMqKdm7uC2v9gZ4blznK4Vy7Mpr9MS4sNZvTooqFHTGX658uUpJSMS3VqR9UFmPb8Uvuw3w5R+Znn3e3zdOyDuZiHLjRvhvxGQEVCDCyx0leefr8XDswkNatL7389PnyzHXPmrWFdL7MgctJTBq4vNMZPj+Z3y51WDC8OJQKljLetClrsPtELo9cp9qxlPKmeWnYk+N9aBOHaQIGJQooq1DgBTYtNq/sCr1Wb0pWCKRJx0cZMXUOXz3ZEF/7AWaPm8EuJ7iOxxJrhxKtu9G7RTVK2QyweFGqlO8FGRWT2GNxmNaqdOnZhTqlbFh9A6h7UyjVrLnZjuSWw5E9Go4ePcr/vvsfTz05gEGDnmPB/AUkFsAduELtO+xh6fJ9uCrcxxvvPUnnkCqU8bZisfriXyOUDi0D1X+uIY5Tj74nJSezcPEihr38Mo/3f5yvvvqaw9GH87ZR72Y8/fZQ+nW6kVr+vlgNA2uJAOq0fIiBD9bDajqIi03A5YljlHMPPy/chd0Wwv99+i4Dbq1LgG92/y1bwZ9zfyPmS/u5aC8nRkAHRr7bn/rmDj59aRyrjl9wAnVqPBaU02Pj6JHs4/iApwbw9NPP8P3335OYmFjg7WfZHZhGOZp3aEMNP6v7/dFah07dO9G4ehm8LQZWLxuOlBQyAcMA4wrjLkfOPfyybC9OryY8P3EMfVrXxt/XitWrFFXqN6V++Tz8RDJT2DB+DLMPl6VaNd/z73G63a/z8Fly4oplyZj/MDfGh2YDRzOgoXfu4nDsYvmqo7h8mvH8h8O4L7Qypbx98A9sSbcuDXH7IQN327vSPr7ScmcECxaEk+XViOc+GkmPxpXxs3lTNvAWnn5/NA9XNTgR9jMr3Kkqdx6T1AMH+MdppfYNtQq9Mr1P867cXhkOLl/O1eYq3N0XHh2X7p5rnZEsX3kAl08LXvhwKPeGVMTP5kXpak24v88dBJ+3fSf7tu8i2fSh3bOv0qtZZfysFqy+panoX+K8cZpf58h86fPutnde27kYB1c75gAwKF27NpUtdg7su4paG0XQmeuec86XZ657CuF8eUmmnbDDDlwlfHmhbUnaBlgpZQWLYVC2lBdtKlmz+5dpZ8UBOw6LF31vK8Oj1WyUO7Ve6RIWfN1o59dDDpxWL55oX4YHK1spawWLxaCivxc3nNrA8ZMuXIaVdkG+1PQCq9VCYCUvKufTw24F/HvPRsigH9gz6MJ/N3Ge2M7014cwdkNydnbz+CrmrBzErfd0YtTsTow6b30nEef8/0Nhqwgf3IjG3T5gVbdT/2zfxtt39+WLw+5uR67EarGeuSsdGxfHyhUrWBO2hkOH83ixlyuF33emThvLlx0nM6DLS0zt8tJ5W7FvfY8uvaZz0DNlwMUNLpcLi8VCeno6a8LWsGr1r4SH78Tlyp+dZgvpTK8HnyCw+xM5LDVJj5zBF0sTMDE9cIxyEjH9DT645QuGt+rKa1O78toFa5y+I2S6PV7yp72ceDW/i7urWTGMRrz44x+8eN6mj/B1/7sYo/GYL6xWK06HA6vNxvHjx1mxYiVha8I4eOBgPrd0qeKOgGMH793Xi8n7nETv2s0JswEN+k1maaWhtHzBvf5olG/Nk6NHcsuF9cxcCSxZtoU0nKRfbtwdyilmBzunvcXn7Scx8MYHeXPGg7x5epGZxs+D2zN42eXeQ5ozM2UzH42dR6f/9qDGBe3tcKtfX+F3ziH3O76ZtI5335zPrZMe4t+jerO0z1dEOt2NI4MtUz5gQecPeLBJPyb82O+Crbt75exee4eusI9Plu98xT4QOXMM49tPZVjLnrw/tyfvn/0msMcs4Y13l7pVAf+iTxDxJ9vS+tK1SWMqW7Zz5LxdcKm+7yR6+uN0fPvPq5i64UOLrh2pzCGmLA13czuXj6fTRDf2vZvniasflxf35dyca7dPe5dZt0+gb7P+TJzX/6K1zz0Ppa2dycxdt/Nc6F289d1dvHXemmenL+TPOTJ/+vyln6q4Qp/72L1xcKX9d+UxB+BDo+YheDn38udfBTgFvBDYrFYcTgc2q43j8fEsX7mSsLAwDh3M8QTiURE7U5hbvRyP1izFuJqlzltmj0uh77KTxJiwf3cyX1T159nKvvxfJ1/+74LtXGniTmR4Ct9WK0ef8iUY0qUEQ84sMVm5Jo43DpnExGQS2diLhnXLMrtu9jRSXHY++zmB7/Kh1kWBJYidcVFs33uU4ykZ2J0mLns6yfGH2bFhCV+//xL3d+3N6OUxZw+8ZgJLXh/AkGmr2XEkmUynE0dmGomx0UT8tYmNUWcfu3NGTmfwsK9YHRnPSacTx8nj7NsaRZxh5Go7cnnJKSksXLiQl14cwhOPP8E338wqlESFp/qOmbKFcX168cKkhWyMjCU5w4nTnkb8wb8J+/3wld9FLB6TlZnJurXrGD16DI891osJEyeyY8f2fEtUALj2LeDdj2axZHMER5IycLpcONJPELNnIz9NGk6PXh+wISW7p3nkGJW+iy8GPMITH3zPuj3/kJzpxOnIICX+MOGbV/BD2L7s193lV/vutpdHGo/5IzU1jcW/LGHo0KH07/84M2bMKIBEhftOho3npc9WsONYCjHRR8lysz8axlH+/PVvDiak43C5cKYncujvFUx55UmGLYzD5Arj7hLM1D/4sH8fBk9azO8HjpOW5cR+MoHD4X+wN9krj/UjTJLWfsLbi2Mvmo/sbr/Oy2e5VCwn1kzkw9Un8G36FC/dXR4jF3G44pbzcu9/896PW9h3PAOHI4Pj+zYzf0U4J0/VOnErCjfau9I+xo0+QHo4/x3Qi4ETF/H7wQTS7VmkxUaw5tt36PPIcBYcyeNUn7TNrNyYiq1RRzpWKsRnK3yac3eninBgBUvC8+c9dm7tew+Oy9yca82k9Yzp+yRvfLuBiNhUshyZJMVs55cfwth/4deVuYMJTz/D2B+2sD8hA6fLiSMjhfiYSP5cs4SwqPTsPpRP58j86PN5vl5xcxzky5jzbcLtbf0x94ax6qqn0nlWakoqixcu5qWXXqJf/8eZOWNmkUxUAJj2LCYvS2DM9gy2nnCR6gSnyyQhxc6mWOfZ318OB9+uSmDYn+lsSXSS6swu6pmW7iTyn0yWxDgumwQ17Vl8sSKB0dsz+DvZxUkn2B0ujiZkcTAr++ka14mTjFmfxm8nXGSY4HC4OBTnyI9S0QAYZctXPG8snK7GunnzFiZ8Oimfmrl+fDPjKyB7bpWni8EUJH9/f5KSknJ1sffq8OFn5m716ZdTRrx4GzxoIK1atQTgnnvu9XA014+SpUrhcDjIzHD/Dmi7W9udqVkx4dNJbN68paDCEw8rzuPOv1w5kpKTdRyXfGZQ8eEprB1zE5tGdubxuQnF4iZQqU5jWfXZPRwd351un+fyrTd55NvuDVZ/0Y2UKY9x18c7C6XN64Glai9mLR9Bs1VDaDr4F3L/fJS4x6Bc13dZMf529r/3II9+dShXfbQonZ+v9rpneGRgQYVWLIwLzr6JctH1tcHcwp56J9eIxMTEfL0rLVJQ0lJTc5WoECkuEk+c0HFcroqlUgvu6dKCetUCKOVtxeZXgXq3PsHYf7fG23WArduLz9OqqWum881uCO3zFJ3LFdybR87ypVXX26hkHmDpst1KVEjRYwumz4DbKZewjKk/Hr6m+6iue4ou1SgTERERkYv4NH2UcRPuptSF1+amkyMLJzM74lq+PMklRyRffziPHlO6M3zQT/w2dhMpBZmpKXETd3esgLnvR37ZXYy+Z7lGWKnz6CsMCLWzaewkViQVl7SlFDYlK0RERETkAgbeJ/awevMNNA2uRZWyPpCZxNF921m7YDqfztpEbLG6EWmSvP4TRs4KpG+iiY9BgSYr/Fp2pVN5k71zV6BchRQ9XnilRRO+YgUjv8vd9A+R3FCyQkREREQuYJK0eSqD+031dCBFh3mCsLH/IqwQmjq5ZiStGo4shJauP66js3nsxtmeDuM6l0HEvP/w2DxPxyHXO9WsEBEREREREZEiRckKERERERERESlSlKwQERERERERkSJFyQoRERERERERKVKUrBARERERERGRIkXJChEREREREREpUpSsEBEREREREZEiRcmKPGrcpDGNGzf2dBgiIiIiIiIi1x0lK/LKhHfeeZsxY0ZTr149T0cjIiIiIiIict1QsiKP/v77b4a9/DI+Pt58/PFHjB37FkFBQZ4OS0REREREROSap2TFVQjfGc4rr7zKiBGvU7JkScaP/5j//GcUN9xwg6dDExEREREREblm2TwdwPVg27ZtvPDCNpo2bcq//vUEn3wyng3rNzDzm2+Ijo72dHgiIiJSQEqXLsPwV4YQHxfPP3FxxMXFER8XT2xcHHFx8WRlZXk6RBERkWvSJZMVQUFBDB40sDBjueZt27aN559/gZYtW9Gvbx8mT57EhvUbmD5jBkeOHPF0eIVKfedimiZUNN3V9Q7atGrp6TCkgGjc5Z2O4+6rVq0agbVqYZomBoBhnFnmdDjIyMwiIyOdjMxMsjKzyMzMJDMr+/+bpumxuEVEPOV6Oj/3rhLn6RCuW5dMVgQE+NNKP+BzzTRNNm/exO+/b+GWtrfQr29fJk+exJo1a5k1axbHjh3zdIiFQn1HrhXBwdfPyVIkP+k4nnvGOUmK06w2GyVtNkqW9PNARCIiUtAalT7p6RCuW5oGUkBcLhfr1q5jw/oN3NL2Fh5/vD///e9kVq5cyezZ33L8+HFPhygiIiIiIiJSJBlly1fU84eFwGaz0f629vTp1ZuA8gGsXLmSWbNmk5CQ4OnQRERE5Cq0uKkFY0aPvuJ6pgmm6eLgwYNMnPApeyL2FEJ0IiIi1yCDuUpWFDIvLy86d+5M7969KOnnxy/LljH3f3NIPHHC06GJiIiIGwICAggJDSEkJITgoCCCgoLw8vLKcRrIaU6Hgyy7nRkzZ7Lw54W4XK5CjFhEROQao2SF5/j4+ND1zq483LMnJUqUYOHChcyd+z2pqameDk1ERERO8ff3p169etSrF0y9evWpXy+YkqVKYbfb2bd/HxF7IomIjKBfn75UrFTxor93Op1YLBbCfg1jypQvSEpO8sCnEBERucYoWeF5Pr6+3HfvvfTs2QOr1cqiRYuYM2cuaWlpng5NRESkWPHx9aVu3RsICgoiOCiYoKC61KpVC4CEhATCd4azc1c4UVFRREVGnfda0hdfeoGOt3XEarOe+TfTdHH4UDQTJk5g167dhf55RERErllKVhQdvr6+3HvvvTzcswdOl8nPP//MvHnzSE9P93RoIiIi1x2LxUKNmjUIOjWNIzgoiODgYLy8vEhISCAqKorIyCiiovaye88ukpOSL7u9e+65h2eeeRqr1YrT4cBut/PV11+zePESTfkQERHJLSUrip7SpUtz33338dBDD2J32Pnxh3ksWLDgvLs3IiIikjsBAQEEnXpaIjQ0hIYNG+Lj40NGRgb79u0jMioq+4mJqCgOHTyU6+0HBwczfvzHmKbJr6t/ZerUqZxI0pQPERGRPFGyougqU7YM3bt14/777+dkejrzfpzHgvnzybLbPR2aiIhIkVayZEkCawcS0jCE0NAQ6tWvT7myZXE6ncTExBAVtZfIqEjCd4azb9++fHnywWaz8fHHHzFt2pds27YtHz6FiIhIMaZkRdFXtkxZunV/iPsfeICkxES++98cli9fjtPp9HRoIiIiHmez2ahduzYhoSFn6kzUrFkTwzDOm84RHh7Orl27yMzMLLBYDMPANPWzSkRE5KopWXHtqFCxIt26PcTdd99FwvFE5sxR0kJERIqXHOtM1AvGy+ZFWloaBw8eJDw8nJ07dxGxZ7emYYiIiFyrlKy49lSqVJFHHnmELl26cOToEebMmcuvq39V8S4REbnunFtnIjg4iJCQEEqVKoXD4eDAgQPsDA8/U2fi8KHDeqpBRETkeqFkxbWrcpXKPNyzJ3fccQfRh6OZ9e1s1q9brx9qIiJyTSpRogR1bqhDUFAQoQ1DCL0xFH9/f1wuF9HR0WfqTERFRRG5JxK7QzWcRERErltKVlz7atWsSc+HH6ZDh9s4dOgw3373rdtJCz8/P06ePFkIUYqIiJxltVqpXqP6qQKYoQQF1aVGjRpYLJaLXhsaHr6T1NRUT4csIiIihUnJiutHYO1Aej36GG3btSUiIpLvvvsfmzdvuuT6ZcqU5pMJExgzegz79+8vxEhFRKS4CQgIICQ0hJCQEIKDgggKDsbby4uTJ09y4MABIqOyC2Du3LGTxMRET4crIiIinqZkxfWndp3aPPbIo7S7tR3hu3bxzTff8Ne2vy5a74knHqdHjx6kpKYybMgwDkcfLvxgRUTkunNhnYkGDRpSpkxpHA4HR44cyU5KnKo1EX04WjWXRERE5GJKVly/GjSozyOPPEKrVq0IDw9n+vSZ7NixHch+HerX07/G29sLp9NJSkoKQ4YM5dixYx6OWkREriW+vr7cUPeGU2/myE5Q1KpVC4Bjx44RHr7rTJ2JqIhIsuyqMyEiIiJuULLi+hcSEkLfvn1o3Lgx27ZtY/r0GXTo0IF777kbq80GgNPpJCkpiZeGDCUuNtbDEYuISFF0us7E6deGhoaEcMMNN+RYZ2L37nCSk1M8HbKIiIhcq5SsKD5atGhO7z69qRdcD5fLhdVqPW+50+EkPj6OIUOHab6wiIicmc4RGtqQkJAQgoKC8Pb2Jj09nf379xN56pWh4TvD9WSeiIiI5C8lK4qf/4waSYubbrooWQHgdDg4cvQow4a9TEqK7oiJiBQXJUuVIjgoKLsAZnAQ9RvUp2yZsjidTmJiYs68NjR8Zzj79u1TnQkREREpWEpWFC8VK1Vi2tQpWK22S67jcDg5cGA/w4e/Snp6eiFGJyIihcHH15e6F9SZqFmzJoZhnJnOsXNnOOG7womKjCIrK8vTIYuIiEhxo2RF8fL884Pp1LkzthyeqjiX0+kkKiqKV18bQWZGRiFFJyIi+c1isVCjZo0zdSaCg4IIDg7Gy8uLtNRUIk8lJqKi9rJ7zy6Sk5I9HbKIiIiIkhXFSdUqVZnyxedYLBa31ne6XPy1bRtjxryJXdXbRUSuCee+NjQ0NISGDRrg4+tLZkYGe/ftO1NnIioqisOHDmOa+gkgIiIiRZCSFcXHA/ffT7ce3SkfEIBhGAA4nQ5ME2w2K2Bc9DdOl4sTiSfYFR5eyNHKueb9NI/du/d4OgyPenX4cE+HIHKR8N27mP/TfI+17+fnR+06tQlpGEJoaAjB9erhX64cLpeL6OjoM3UmoqKiiIyIVOJZRERErh0Gcy9dvECuK/MXLGD+ggV4eXlRqVIlqlatStWqVahapSrVq1enWo3qVKpYEdup15m6nE4Mw6B8+QDa3drOw9EXb2vXr4NinqxQH5Siaj6Fk6yw2WxUq17tVGIiNMc6E4sXLSY8PJxdu3aRmZlZKHGJiIiIFBQlK4oZu91OTEwMMTExFy0zDIPyFSpQtUoVqlarStUqVXn44Z4eiFJEpHirUqUKIaEhZ+tM1AvGy+ZFWloaBw8eZPPmzXz11XQi9uzmRFKSp8MVERERyXdKVsgZpmkSHxdHfFwc27dvBziTrNi8eQsTPp3kyfCKlVatWjJ40EBPh1HkqB9KUfDNjK/ydXvn1pkIDg6iYcOGlC5dGofDwYEDB9gZHs6SX35RnQkREREpVpSsEBERKSQlSpSgzg11zrw2NDQkhMpVKp9XZ2L2t99m15nYE4ndoToTIiIiUjwpWSEiIlIArFYr1WtUJygoiNCQEEJCQqhRowYWi+VMnYkVK1cSFbWX8PCdpKamejpkERERkSJDyQoREZF8EBAQQEhodlIiOCiIoOBgvL28SE9PZ//+/Wzdto2533/Pzh07+OefWE+HKyIiIlKkKVkhIiKSB+XLB9C3X1/q16tPvXrBlCxZErvDzr69+9kTsYclS5YQERFJTEyM6kyIiIiI5JKSFSIiInnQMCSEylWqEL4znG9mzSIqKoqoiEiy7KozISIiInK1lKwQERHJg42/beTNt97ydBgiIiIi1yWLpwMQERG5FjkcDk+HICIiInLdUrJCRERERERERIoUJStEREREREREpEhRskJEREREREREihQlK0RERERERESkSFGyQkRERERERESKFCUrRESkEFkJ7PYOC5b9yGut9PZsEREREcmZkhUieeJD88H/48/fF/PuHeUxPB2OyDWkdM0QQmqWw9c4PXI0nkRERETkfEpWSL4red9Edu/+nYXDWlEux6sOL9q/tZa94YsY3tha2OHlG8MwMAwLFl1ZFW2WMjS48xnGTZnDmt82syd8Gzt/W8rCr9/jtd43U8PHU4FZCX1iEr+snMH/1S+scZCXNn0J7PgU7301j9+2/EHkzj/ZsWEpP3/5Lq/0bIJ/PvV/jScREREROZeewZWCYZQg9F8fMeFof576Zi9Zno4n32XyxycP0+wTT8chl2OUacxTH3zMy+2rYDvnItg7oAahN9cgpGlJ9izeSHSm6YHoLJQLDCG4aiLehXaBnts2rQT2/IjvR7engvXsH9jK1+DGttUJtm1jxg9/wVV/fRpPIiIiInI+JSukgJg4zTK0e2U8r+7rw5gNSVd/PSPXhbffHstvv/3GurXrSDxxouAaslSl27jPGH6bP2b8VmZOnsJ3q7ayNy4Lb/8aNGxxK3c2OMb6pILvmRaf0pQv54sjJZHEk44Cby/f2JrSf2A7yhPLqg9HMe7HrRw8Ycc7oCahLdvTOP1X/nF5Osi8uWb3iYiIiEgxoWSFFBAHf838jGNdn6Pve6P5++EhzDvivMz6XrR9YzkzHk7k0249+Hj36XUNyj70GZvH3cxvIzrzxPcJmBiUbzuAEf1vI6RuTapVKIOfl5PkI7v4dfanfP5XdR7s/QB3tG5AjXJW0mJ2sGz6B4ybvZ0T51yXGiWDuOfpgTx5bxsaVPIh/Z89rJv3Oe9NCSPafqrtpg/zfP87aBkaRO0q5ShhpBN/cCmj+40h6pHvWDy4Kj8+3ZFX1trPbrhELW5//Fmeur8tN9Yog5GeSMyeNUx6/U1+Oni576B4CAkNpUmTJjzzzDP8/fd2Vq5aycbfNpKWlpav7fjd8ixDOwRA/Cpee+xF5hw6e0GaGbuXzUv2snnJOX9QojZ3/OvfDLj/FkKqlcSVeJA/Vn3PpM++Y3Pc2f54cd+DjMRDbFvxLR+O/46tiWc7mSXgJgaMeo1nbq+Hv5eBadpJObScMY+/wg9HTq1krcfg+X8z+NR/umK/o2+nN9lgB6N8B1796Hnuql+DymV8MNPj2fv7UqZ+9Cnz9qSdSgDmLqYrtXn+l1iDwPJWXIcWMmHaOiJPfQ1ZsXvZtGgvm86saBBw878Y3u82bgyuQ81KZShhZJJ4JJJNy+bw+bSf2X7iclkNK8H/vnA85f5zXXlMX2mfXKOZFxEREZHrkJIVUmCcMUt4dUhp6nz5BGM+/Bd7nviC8Iz82LKFgMZduO+2kHM6sBf+NZvx0CvTeOiCtb0Db+KR1ycTkNadZ3/6BxeAXyMGTfuCF5qVPlO4xbdmE+57bgJNqw3v4sE2AAANu0lEQVTmgdfDSDQtVLq5B33vPred0lSs5EVm6iVC82nAgM+nMbx1ubMFYbwrE9S0JqXSdSF0LsMwaNToRho1uhFz8HNs2/oXv64JY/36DWRmXH1Hufm+26lkyWLrtPf54dAV7pz7hvDMlKm83Krs2f1WuR63PfYqt7RvwpA+r/LzESc59z0oWaEubR8dQdN6JejW90siHIClCj3HTeTl28pgONI4/k8qRqnylKvkRWaSC3CjZoSjHHWb16OG96n/LlWZhh368f6NlbA/MJSf483cxZRb6UeJTnRiqdmZvnf9QMTCg6TnuKKF8k3v5KFO58Zgo0LtptzzdBNu79qSF/qO4pdcPYaRy8/lzpg2rrRPRERERKSoUIFNKUAmqX98ygsf/4Gr6UA+frElpfNzbr6ZzJJX76BJ48YENWrHPSMWE+00cZ3YwqeDenBzi6bUa96FPpP+IJly3NatE5UsAFbq9RvJoKZ+xIZN4F93t6VBaAta93id7/c5qfHgIHoHnXMhaaaw/D/3clOzpgQ1vplbe37CpgvvQJO93Tq9R/JSq7JkRPzE633vonnjpjRs2Yk7+73L0nhNhLmQxWLBYrFgtdpo2rQpL734IrNnfcOwoUNp1ao1Nlve86kh9Uphce5nzboYLv88i5WgviN5sWUZMnZ9z8sPdyL0xmY0u2MA76w6ilHtLt54ucv5hSTP6Xt1Q1vTrvc4lh9zUbJJb3o39wLAKN2aO1qXxrXjcx66+WZuat+JFi1a0eahD1h38pxtOSOY8EBj6tQPpU79UOreevYJBzNlPe/3e5CbW7YgqGFjGra5l39N/ZuM8h3p3sH//LdmuBGTO22ex/4HX366jngjkO4fzGPVrDE827UBAZfaLWYyi1/uRGhoI+re2IZ2j7zCF78n4hX4AG+93DlvxTjd+lzujWm394mIiIiIeJySFVLAsoiY+RpjVqUQ1PctRlx4gXU1TCcpcbEkZzpxZiUSPm8Ss3Y6MWzxhK/fxbFUO/a0I6yfMo3lSSbWmrWpZQGs9bnv3gbYklcwdtgUVu89QaYjg9jt8xg9YTWp1mBuaVXh7OAwHSTGRHP8pB1nZjJHDv5DWk55B+sN3HvfjfjY/2bC4FHM2nyIxEw7Gcn/ELE1gjjduL0sq82KYRj4+vpya/t2jBo1km+/nc1zzz2Xp+2VLmmAK5GEy04/AKzB3P9AKN727UwcMoa5f/3DSXsWJw5uYMrQ/zDnqIl/h/vpfO6V9jl9z+VIJeb32bw9cwcOa3kaNKiY3XdMExMwKjagVYMK+BqAmUnc/ujzpiNdlmFQrtGjvP3lD6zftIW/V8/gja7VsGKjctUK5x/A3Ykp15wcnPsC3Z6dwILwNMq36M4rE75n3fKvGdu3JZUvTFqYTlITEjjpcOGypxCzbSHvDHyD+XEQ0PkBOpTNw+h353O5O6bzY5+IiIiISKHQNBApeM4j/Pif0dwS8jE9Rr9K2PaR5G91gtPtHOXgEQc0rEjlchY4eeoiNesY0bEmRiU/ShiAV01uqGHBUqIrEzd3ZeLFG6JajapYiM9d+7ZAgmtbcR3ezIZD+Veb4tXhw2F4vm3O41yuK2dtrNbsQ5Ofnx933tn1zL9XqFjB7XbS0gFLWcqVtUDsZfaHdyBBNSy4Dm9i/YEL1kv7k7VbM3jszkCCalog4VIbcXJk7wHSzFBKlSqJAbhSNjBvZTwd77mN12asYEjiQXZs+4OwBd/w5S+ROSe8zmWUof3rXzP1sUC8zlzj+1CrZnZ7FsuVDt8Xx5Q3WUSvmcLza6bzTou76fdEf3p1uoleI6Zxe7u36PV/c9h7mSkmZtIGVv6ZxYO316JudQtcdU3VHD6Xt3tj2rjafSIiIiIihUbJCikUZvwq3hz1Ay0+784bI9bxfg4T301cgA++vnm9rHJhz3KA4YWX17nbsGN3mGAY593xvjQDnxI+ub+4MyxYjOzt56d5P/3E7t2783WbnjRs2DC37vI7HU6sNivJycmUKVMGgPg49xNIkfvTMevXoU3LikyKPMalUyT586yPmZVFlmlgWE5tz4xn0Wt9Sd3ak7vaNKF5s0Y071iHFh060sDSjUGLki67PSPgdh5/qBbWxE18OvI9Zm3cS1y6jQqdR/DT+PvzFtNVyeTYH/N474/5fB7SjTHjX+e+9s/zXKfFvLAs50oWp6LAdJnZ/5sPUUBO37WbY/oK++T/FuUyQSkiIiIiBUbJCikkJifWfcSIb1vx9WPDeOEfPwySz1nuIiUpFdNSnfpBZTG2HS+4V53aYzgQ48JVdj5P3TGS1Zecq+5GAcQctmup1Yqba1rZfuFd+jzavXs369auy5dtFQVDhw695DKnw4HFaiUzM5ONv20kbM1a/vjjdxYsmJ/rdn5buZHkrp1pM2Awdyx/nV8uNQ8n6yB7o11YAlvTNtDK9n3n7LeSzbm1mS9kHWJftItcz5zLOEzYzI8ImwlYS9Og+xi+fKMLHbq2wm/RMhwOByZ++PldnEywVKhCFW84uXwmE1fsJgsAO8fjksnMXRTnMC/bpntcJIXPY/yc7tw9LISgoKpYl+279OolGtHqRm/Iyh4fZ79DK9b8OgO5Paa57D5h0eJ8CkhERERErpZqVkjhMVPYMH4Msw+XpVo13wvuZzvZt30XyaYP7Z59lV7NKuNntWD1LU1F/xL5V+cCwLmHpcv34apwH2+89ySdQ6pQxtuKxeqLf41QOrQMzFsWz7mHX5btxenVhOcnjqFP69r4+1qxepWiSv2m1C+v4ZYTl8uFy+XC4XSydes2Pvr4Y3r16s37H3zA5s2bcDrzlvRJ/GUyU3dmYql2P+O/m8Swh1pSt4IfNosV79KVqNf6Xp598SEaEsGCBeFkeTXiuY9G0qNxZfxs3pQNvIWn3x/Nw1UNToT9zIqEXKbPrHXo1L0TjauXwdtiYPWy4UhJIRMwDDAwiT0Wh2mtSpeeXahTyobVN4C6N4VSzQqu47HE2qFE6270blGNUjYDLF6UKuV7FVnmy7d5Ee9mPP32UPp1upFa/r5YDQNriQDqtHyIgQ/Ww2o6iItNOPvUilGSVj160TG4PCVsXpSp2ZL+b4/msRoW0jYtZ21S9neYZXdgGuVo3qENNfxymRTMibtj+gr7RERERESKDj1ZIYXKTNnMR2Pn0em/PahxwbK0tTOZuet2ngu9i7e+u4u3zlualY9RONgxbSxfdpzMgC4vMbXLS+cttW99jy69pnMw1wUxHeyc9haft5/EwBsf5M0ZD/Lm6UVmGj8Pbs/gZfny7tbrwunaFVu3bmP1qtX8tvE3MvLhlaVn2HczefArVJo0lt4Nb2XguFsZeOE6jp1Y5i9g0swxjG8/lWEte/L+3J68f2YFE3vMEt54dym5zVUY5Vvz5OiR3HLBizhwJbBk2RbScJIetorwwY1o3O0DVnU7Hfc23r67L18cXsWclYO49Z5OjJrdiVHnbcRJRO7COfN3hy7X5qHzO70tpDO9HnyCwO5P5LAtk/TIGXyxNAHzdN7b8Kb2nS/z5Z0vn/+RT2xk3AcLiTWzY4jetZsTZgMa9JvM0kpDafn88jx9mrPcG9OHrrBPRERERKTo0K1eKWQmSWs/4e3FsRfXEMjcwYSnn2HsD1vYn5CB0+XEkZFCfEwkf65ZQlhUev7NeU/Zwrg+vXhh0kI2RsaSnOHEaU8j/uDfhP1+OM+pETP1Dz7s34fBkxbz+4HjpGU5sZ9M4HD4H+xN9srfJ0SuUS6Hk127djF58mR69+7NqFGjWP3r6vxNVJziPLKCUY92o/9bM/nlz/3EJmfgdDpJT/6HvX+t5fsvvmV9ogvSw/nvgF4MnLiI3w8mkG7PIi02gjXfvkOfR4az4Ejun+4wjKP8+evfHExIx+Fy4UxP5NDfK5jyypMMWxiHCTgjpzN42FesjoznpNOJ4+Rx9m2NIs4wwExgyesDGDJtNTuOJJPpdOLITCMxNpqIvzaxMSopT+Phsm1ewLVvAe9+NIslmyM4kpSB0+XCkX6CmD0b+WnScHr0+oANKedEYabx1+IfWRsZx0mHg4ykaP5aOoXnHhvEV5Fn3416Mmw8L322gh3HUoiJPpovqUh3xvSV9omIiIiIFB1G2fIVVf9cLmnRooUAbN68hQmfTvJwNMVHq1YtGTwo+zmAd8aNu65qVgQEBJCQcMnXauRI/bCosxL87+9YPLgqPz7dkVfW2q/8J9ewb2Z8BcC6tet4Z9w4D0cjIiIich0ymKsnK0SkUOU2USEiIiIiIsWPkhUiIiIiIiIiUqQoWSEiIiIiIiIiRYreBiIiIlfJSeTkngRP9nQcIiIiInK90JMVIiIiIiIiIlKkKFkhIiIiIiIiIkWKkhUiIiIiIiIiUqQoWSEiIiIiIiIiRYqSFSIiIiIiIiJSpChZISIiIiIiIiJFipIVIiIiIiIiIlKkKFkhIiIiIiIiIkWKkhUiIiIiIiIiUqQoWSEiIiIiIiIiRYqSFSIiIiIiIiJSpChZISIiIiIiIiJFis3TAci1ISgoiMGDBno6jGLD39/f0yEUSeqHIiIiIiLFg5IV4paAAH9atWrp6TCkmFM/FBEREREpHjQNRERERERERESKFKNs+Yqmp4MQEREREREREQHAYK6erBARERERERGRIkXJChEREREREREpUpSsEBEREREREZEi5f8BrazVAMhe7pcAAAAASUVORK5CYII= ) ## Delete the original blueprint directly ``` blueprint_graph.delete() ``` ``` Blueprint deleted. ``` ## Modify the source code ``` #w = Workshop() rst = w.Tasks.RST(w.TaskInputs.DATE) # Use numeric data cleansing instead ndc = w.Tasks.NDC(w.TaskInputs.NUM) pdm3 = w.Tasks.PDM3(w.TaskInputs.CAT) pdm3.set_task_parameters(cm=500, sc=25) enetcd = w.Tasks.ENETCD(rst, ndc, pdm3) enetcd.set_task_parameters(a=0.0) enetcd_blueprint = w.BlueprintGraph(enetcd, name='Ridge Regressor') ``` ``` enetcd_blueprint.show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABFsAAAEMCAYAAAAbE7ngAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd1QUZ9/G8e/sLkVBQECxoFhAUexGjC0xscUejb2nmPYkxnRNb88b041RkxhjEkvMoylGTbVrNIoaW+yggtgLgoC03X3/wBhRlF1cXJXrcw7nKMzO/Gb2vmd2rr1nxvAPKmNHRERERERERESunMFsk7trEBERERERERG5kShsERERERERERFxIYUtIiIiIiIiIiIupLBFRERERERERMSFFLaIiIiIiIiIiLiQwhYRERERERERERdS2CIiIiIiIiIi4kIKW0REREREREREXEhhi4iIiIiIiIiICylsERERERERERFxIYUtIiIiIiIiIiIupLBFRETkGuHdeCTzN+/l4LafeL65r+Mv9GzGM7N/J2bjQl5u4lF0BYqIiIiIQyzuLkBEROSGYqnJkA/eYEj9ylQICSSglA/eFoOczFROHtlP3Nb1LP/te2Z8/yeJGRe81jBhMhkYhhmzYTixzHLUbd6AGpZ9LHDiZVfFlWwPERERkeuU4R9Uxu7uIkRERG4Ynq35cNM3DC1zucGjdjL2/cRLdz/KpC1pV77Mkj2YtvtTulr2MaFrC56Pyb7yebqKO7aHiIiIiDsZzNZlRCIiIkUii2WjGlGuXDlKlylHcKUa1G7RlWEvf8OmU3a8qnTmzelv0jHwWhuKUlS0PURERKT4UNgiIiJSRLIzzpCZY8Nut5Fz5hQHd65hzoTH6DJoEjuzwVyhBw/3qFhsDsbaHiIiIlJc6POMiIjIVWXn9LpvmRdnBcOD2vVqnLuBmlFuKD8ePMqpQ4t4upb5olcapWrS45kJzPtjE/v2J3J47xbWzJvEa0MaU+ZyA0IswTTuP5pPvl/G1l37OHbkIEfid7L1z1/5/rO3eP7ellTI7xOBVyi33v8m3yxYx56ERI7s286G36cy5u6mhLjsrm+X3h5XVIez6+xRkeYDn+Ctz2axaNUG9sTv59jhRPZvX8uK/z1K4/PvO+xkPd5V2jHyw9ms3LiTQ4cOcHjfNjYt+5Gv3x9Kfe/CT5tbdxmiB7/MF/NXsXPPfo4k7GTLkpmMfbQD1UvmN70T6ykiIiKFphvkioiIXHVWrFYAA5PJjCMXzlgqd+ejWePpF+513vQh1Gx2JzWbnZvtRQy/xoz88iueb1UWy/kL8ilNxYjSVIxoROtbvVk77Q8OZp33usAWPDd9Ck9Glz7vm5kgqja6gwcbtqFzy8fp+sAs9uU4sdqXdOntUZg6CrPORlBbnn1nFLd65q3Mo0wYkYEGp22Fq8ej5n3MnPsGtwWdl+x4BBMWFUyo/1YmnPeeOTMtgBHQhCe//IrRLYIxn1tPLyrVbcOwurfTt99MHh7wND/E/3sPH0fXU0RERK6MRraIiIhcZd6RHekQYQZ7Nju37qLA29laInhg4lj6hXthP7mOT0Z0oWFEZcqG1qRRpwd444ftpOR3kmyUofu7X/DCLWUxZ8byw8sDaR5VlTJly1OuWl2aPfMbKfndJt9Unt7vTeLJ6ACy43/ljcGtiawcSrkazejx0nz2ZVuo1PUNxvQp75IPEpfcHoWpo7Dr/A/rPqbf35q6NatQJiSUSlEt6TRyFnHWwtTjS/vHn+bWIIO0zV/y4B2NCatQgbJhUTTuMJjHX/uWLedCImemBUwh3PXeFzzXMhgj9W+mPnknjSIqExJWj1vufpfFh6x41+zPxCmPUf+CYKXA9RQREZErppEtIiIiRc1kwatEKYJDa9CodU8efmwwDTwNbEfm8/G3CRQ0mKBEy/v5TxMfDGs8XzzQn9FLksnNCzLYE/MD7/4N9Tt9StcLjuoe9Yczqls5zLaT/PpsH+6bkXhu8Et2yhHi4k+QnU/w4Nn4AZ7pWAbjzFreHHgfY3ecHf6RHseSiQ9zf9mF/PxIBLf170roN5NIcHY0hIPbozB1FHadz7GdJn77TvafyH1V9pFdrD1SyHqMitSq4YuJbDbMGMusdQdz1y3rGHHrfyNu/XnLNTsxLeDZ8AGe7VwWk/UQsx/ry2Nzj51tE4fZPO9tBiTY+fXnp2hQ7wGe6f4lg2YfJ89qX2Y9RURE5MppZIuIiEiR8KTt2O0kHT/KqaNn7xeych7T/ns3zcqayTqwkJeHPMPc45c78wfwoH7b2wkxQ/bWr5m0PJmCXpHLQp0unQi3gHXfN7w/OzG/q4zyXV7Drp2oarGT8cc0pu3MuuDvGWxY+AdHbQaeUY2o7+XQTHF+exSmjsKusyMKUY8tiWPHrdjxoGGfYTQLuvg+POc4My0W6nXpSFUL5OyayUc/H7uoTWRs/pyPF6ViN/xo3bU1/nrIk4iIyFWlkS0iIiJFzo7dZgfDhEEmf3/xEHe/9hO7Ux2ITQw/IiJCMGPj1JbN7HU0PTBKERkVhhk7KetXs/nCbOAyr4uoUQEzBiXajSPu2LhLT+tdhpAAE5xxdmiLA9ujMHVkFHKdHVGYeg4dY97k73ii1QDCGj/G3LVdWTbnf/xv1rfMW5NI+vlvv92JaQ0/ImtVwoKNkxvXszO/++bYk/lr3W5yOjXCq2Ztws2wziX31xERERFHaGSLiIhIkchi4chalA4uS0BwCKUrd2DMhnTshicRzZtQxsHxKRgl8fEBsJN6OrXAS47+fZ0v/qUMDGwkn0hyfISH4YOvr6MTe+Pl8MgWJ7dHYeoo7Do7olDbxc7J35+l65B3mbftFLZS1bl9yHN8Oi+Gbcun8NRt5c/71suJac/VYud0csol2oSNlOTc9mLyLYWvPvGJiIhcVTr0ioiIXA0Zm/hgxNusSQWvmvcxdlQLHDp3t6eRmgpgwq+0P5e7uCTv6zLJyAAwKOlT0qEnHuW+Lp20NAAbx6f3o2xwWQIu9VOhMxP2FfLxNQVtj8LUUdh1dkSht0sWCQveYfAt9ajbYTivTlvJ/kwzAbW68Pz0Wbx08/nPZ3Zw2nNtwqCUv98lPsyZ8PP3xQTY01JJ1VOGREREriqFLSIiIldJ1o7PGPluDGl4EHHPW4y+2afgF9lPs3NHIlYMSjW+mToeDi7Mfor4hFPYMOFfrz5VHU1p7CnExh7BiomARjdRowgvOL7s9ihMHYVdZ0dc8XbJ5PBfP/LB4z1o0vJ+ZsRlg1cNBg+9hRLOTnuuTZjwa9CImvnVYvjR8KYILNjJ2LlVTxkSERG5yhS2iIiIXDXZ7Jg0ig83ZYJnBPf+90Hq5PdY3gtes+mnX9mXA5ZqA3i6X5iDN1zLYv2i5Zy0gUedgTxwi7+DIz2y2fD7Yg5bwVJzACM6lnHtCJELlnXp7VGYOgq7zo7V6qrtkrHvJyb9uAcrBiXLhlz25rX5T5vbJvbmgKVGf/7TIfiiWrzr3MuDt/ti2FNYNm8Zpxy8ak1ERERcQ2GLiIjI1ZS1lfEvfklcDnjXe4jXBlQq8GCctf4T3vzpKDZTIO3f+oGZo3twU2V/PE0GZi8/ylWtSEA+J+wpCz5h8tZM7ObKDPtkOm/0jqZ6UAm8fMpSs2U/nnvwVvzyeV3GHxP54I9kbOYK9B7/PZ890olGlUvjbTEweZaiXI2mdBt8B5GuGPVyme1RmDoKu86OcL4eX1rc/xwPdG5MtWAfPAwDs1cAYdF9eLBrVczYSIqPPxuEODNtbpt46+ejubV8NJP3B99MlQBPPEuGUKfzU0yb/jgNvSFj8yTe/vHipxWJiIhI0dLTiERERK6y9NVj+e+8u/i8RzC3Pv4Ebb9/gt9TLnM6bDvMd08Mo3LAlzx3SyjtnvyUdk/mM92Fl4pkbeaDh16gwaz/o0OFpvzn4/n8J7/Z2+15T8ate5ny0INU/foTHmpQk16vfEmvVy6c91qeX/47O+Kv/GYgl9wehamjsOvsCGfr8ahFp/sf5T9VRvLWRTOzYzuxjPc+XkkGODctgO0w3z5xD1WCvmJ08/rc/cFc7v4g72vO7PqG/9wzlo2Zzq6oiIiIXCmNbBEREbna7CeY994nbMi0Y67Qk8f6VS7wgGxPXsf7fW6h3cPvMGPRZuJPnCbTasealU7S4T1sXvkz0yZMZP4Fz4bO3PEVg9r15MlPf2X9vhOkZWWTkXyAv5d8zZhPlnLSBvb0NNIuSB5sRxfxfKfWdH/mE35cvYvDKVlYbTlkpJ4gfusq5k77kY3pRb89ClNHYdfZEU7VYz/GH//7ht837OXI6UxybDZyMlM4HLuenz9/md7tB/PZ7mznp/1n9qdieKdXazo//Qnz1u7hWGoWWWeSOLB1KdP/eze3th/J9/F5XyMiIiJXh+EfVEYjS0VERIodE5WGf8/a/2sGy56mQe+pHL7hPxEUx3UWERGRq85gti4jEhERuVEZ/rQYdi81jv7Juu0JHDxyjFOZJvzKVadRu2E8/+zNeJPCb98v4OiNEjoUx3UWERGRa47CFhERkRuVJYpujz3NA6GXeAayPYv4Oc/x7KxDXPmdV64RxXGdRURE5Jpj9i7p84q7ixAREZEiYPHGx9cLT5MHXiW88fbwwGTPIuXoPrasnM+UN5/g0feWceTCG+tez4rjOouIiMi1xWCb7tkiIiIiIiIiIuIqBrP1NCIRERERERERERdS2CIiIiIiIiIi4kIKW0REREREREREXEhhi4iIiIiIiIiICylsERERERERERFxIYUtIiIiIiIiIiIupLBFRERERERERMSFFLaIiIiIiIiIiLiQwhYRERERERERERdS2CIiIiIiIiIi4kIKW0REREREREREXEhhi4iIiIiIiIiICylsERERERERERFxIYUtIiIiIiIiIiIupLBFRERERERERMSFFLaIiIiIiIiIiLiQwhYRERERERERERdS2CIiIiIiIiIi4kIKW0REREREREREXEhhi4iIiIiIiIiICylsERERERERERFxIYUtIiIiIiIiIiIupLBFRERERERERMSFFLaIiIiIiIiIiLiQwhYRERERERERERdS2CIiIiIiIiIi4kIKW0REREREREREXEhhi4iIiIiIiIiICylsERERERERERFxIYUtIiIiIiIiIiIupLBFRERERERERMSFFLaIiIiIiIiIiLiQwhYRERERERERERdS2CIiIiIiIiIi4kIWdxcgV9foUaPcXYIUsR/m/MCOHTvdXcY1JTKyJj3u7OHuMkRc6s0xY9xdgoiIiIhcgsKWYqZlq5buLkGK2IqVf4DCljyCy5RR25cbj7IWERERkWuWLiMSEREREREREXEhjWwppmJi1jJu/ER3lyEuEh3dhBGPPOzuMq4L48ZPJCZmrbvLECmUEY88THR0E3eXISIiIiIF0MgWEREREREREREXUtgiIiIiIiIiIuJCCltERERERERERFxIYYuIiIiIiIiIiAspbBERERERERERcSGFLSIiIiIiIiIiLqSwRURERERERETEhRS2iIiIiIiIiIi4kMIWEREREREREREXUtgiIiIiIiIiIuJCCltERERERERERFxIYYuIiIiIiIiIiAspbBG5ppgJ6/kmc3//nueiLe4uRqTYMUq35sWZ81kxpi1e7i5GRERERK5bCltEnOJFoxH/4691P/NW+yCMIlhCqUq1qV0pAG+jKOYuIpdjeFegdt2qlClpOdu/i77Pi4iIiMiNR2GLOM7kR+QdDzBm0iyW/xnDzm0b2frnb8z/8m2eG9iMUKe+BjYTdfdEfl00lf/UNBdVxUXCMAwMw4RJZ13FhzmAqM4PMmbSLJatWsOOLevZuHwu//vgaQbdXAFvN5fn0/UjduzcxLyHqpNfbzLK9mX6lq3smj6ICk7v9Z3vq7n1bGXvJX52jLvD7dvMGerzIiIiIuIsXacgDjH86nHfux/wzC3lsJx3wuEZGEpUs1BqN/Bh58+rScy0OzhHEwFhtYkon4TndXUCk8n6D/vQ8EN31yFXixFwE4+MfZfHbi6D+by26hVSnehO1Ym+oxcDZr/Mg6//SkK2++osOtdrX3UV9XkRERERcZ7CFimYqTw9x0xg1K2lsR/fwLSPJ/HN4g3EHcvCs3QotRq34o7Iw6xMdjRoESm84DJlGPHoIyxdtozVf64mPT296BZmrsSA98Yxspkf1kOrmTJhMrOXbCI+2YZPhVq06jqMkfe1oVaf/2NSymF6vruRIqzmOpPD32N7cufHcVjdXYqIiIiIyFWmsEUKVLL5gzzVOhCOL+a5/o8zKyHn3N8yj8YR80scMb/8O70R1JrR7z9Gx5qhhPh5YT9znLh1vzH5/fH8sDONPJGMuQYjftzMiLP/tR39hsG3v86qbDB8wul8/8Pc2+VmIst6cebITv744VPenrSMxPNHEHiHctvgB7i3e0vqVQ6iBBkkHzvAnl1/s+CL95gcc+rfZZaoQvt7HmJ4t+bUruCDLSme9Yu/ZeKEb4g59s8poYF/gz48NrQ9TaLCqVIugBLGGY7H/8arQ14jtu83/DyiPN/ffxvPrjivkBKVaTvsQe7r1oI6oX4YZ5I4sHM5E194nTnxVue2i1yS2WTQuHFjGjduTE5ODjFr17J48WLWr11HVrZrh5b43voQjzX3x37kV54e8Aw/Hvw3NsiK38Dc8RtZseE5Zn3anxqDRtL7m3v4KtEGGAS1GM7zQ2+ldvVKVAj2o6QHZCQlsHHhTN4b+w0bkvK+4w63d1dyqD+cdZm+WnjOb6eC+pkz62QKrE//hx9kYLuGVAvyIP3QDv5cnUxInkutzEQ8dGGfL0TdBeynPos5dSUbUkRERESuMQpbpEDNuralrCmLDZ+/w3fnBS2XlBNA9UY1CPU8+3/fEGq1HsI7dcqS3f0p5h13IFYoWZdHPv+MkQ1LnbuxkHel+nR9dBwNKoyg+wvLSLID3pEMnzSZUdGlz7ufgg9BoTUICq2G519TmBJzKvebde/aPDBpMs9E+/97s6KQGtzafzTNb6nPk4NGM++gFTBRtlkvBneqfV4HKUWZsh5kpl6iXq9Ihn/6OaOaBvw7b88QwhtUwveMzXXbRfKwWCw0bRJNs5tvJjMzk9V/rmbZ8hX89dd6cnIcaKsFaN6pNUFGJmsnv3+2bVzITtKqCYxd2IlxdzSgS5sKTPsqERsmAuu1o+uttfPsZH2Cq9Oi3/M0qFGCnoOnsOufEh1t767kcH8oSk5up4L6mRPrZPg154WpHzEswvvcTW+9KjegU+Xcf2e6sm4H9lMKW0RERERuLLpBrhSodg1fTNa9LP/jgEOXA9hPr+SdIXfSrEljwmvVo9bNXbhn8mYygm7jrtal8z7Nw7qLcd3rUbVmFFVrRlG91eusyjZTY8iLPNKgJEeXjeOeTi2IjGpM014v8O0eK6F3PsLAcDNgpvqgl3gyOoCsPfN5efAdNKhbj/C6TWn50jLS8w6hIXzwizzexI+M7d/yTJ/biarTkIbth/Pm4kMYFTryyjPtKG3kWREWvNyFmxo2ILxeM1r1/pA1+X6Lb6bqwBd5ItqfjF1zeGFwRxrVa0CtJrdzx5C3+O1siOLUdhGHmS1mDMPA29ubVre05KWXXuTrmV/z6KOPUjuqNsYVPNWpZrgPJmssK1YexnapiezJrFqxhRzDTHhktbw3qLWn8Mvo9tSvV4/qUU1pOXAMCw7b8Kk/kIGNPP5ZAwfbe0Es1Bk5l9h8bki7Z8VLtPA8f9pC9Id8+6rz9ezZ8A4dPC+Y1MHtdPl+ZnJinSzUuXcUQ8I9Sdk4jZG9cqetf/tARk5ey/FLvtmFq9vx/ZSIiIiI3CgUtkiBSvkYYEvi5CkHz0AMg4C6/fi/Kd+xcs1aNi+ZyisdKmDGQkj54IIbnbkmXbtEYklZyH+fnsSSuFNk5mRwdMsPvDpuCanmCJpHB2MyV6NT5yg8rTv45PEXmBqzn+QsK9asVI6dSL3gcqUIunWPwjN7Cx89+RqzNx0hPTuLU/GrmPTUy8w6ZKd06260Of/s0p5D0oFETqRnY81M4WD8EdLyOzEyV6NL1zp4ZW9m3IiXmBGTQFJmNhkpR9i1YRfH/tlsV7pdpEBmswXDAJ+SJWnbtg3vvP0206ZOpW27toWaX6mSBtiSOZF8ubZvJy0piQy7iRI+PnmHC9qtnD52lJRMK7acVA6s+5r/m/Y3OeYgIiPL5L7nDrX3ctR+5Pu8wcW2uTwVVcgneRWmPxQlh7ZTAf3McGKdzDXp0LYKpsz1jH3ybX7ckjttyoGNzJv+O7GODuhxsG6H91MiIiIicsPQZURSoLQzgMmfAH8THC3gLMTw45YXvmRy/zA8zp2neVG5EoAVk8mBJudZiWqhJkwlOvBRTAc+umgCKxVCy2OyBBNRxYxt/yqWxhZw4wjPMMJDTdj2r2HlvgvWIe0vVmzIoP8dYYRXMsHJgkvMwxJ2to4YViVcYvu4Yrs4aPSoUTDKZbO7blksudu0dGBpmgTedO73lStXIiZmrUPzOJ1uB5M/Qf4mOH6ptm/gU7o03oad9LR0Ln/xkpWDcftIs0fh6+uTO5rJofZeDnNaQdVe+oa0Rtm+TFv0EtH//KIo+4MD9RQsn+1UUD9zZp3SK1Cloglb4l+sO+ToMJYrrduB/ZSIiIiI3DAUtkiBdu89g71mVW5uUoaJuy9zOQVgBLZlWI/KmJPWMP7Ft5mxOo5jZywEt3meOWO7ObZAu72Ab3sNvEp4YZg8sJiAnBwHTuaK8Bt6w5R7Hwb7pat2yXZx0A9z5rBjxw6XzvNaEuDvz0MPPeTQtFarFbPZTHJyMv7+/gAkJOx3eFk7Y9OxR1anZbMQPo47mH/bN/xo1rIOFnsOsTsLDhbsWVlk2Q2Mf27e4VB7t7DjrZ6Ej3e49AJc+xetXbSdCuxnTqyTYc4ddWKYXL4lLqrbqf2UiIiIiNwoFLZIgf5ctJqUDm24efgI2i94gV+PXTpuMQWXo5wnpC+YxkcLd5AFQDYnjqVccMNJOzk5OdgpScmSF5zuZB9g3wEbNv8fua/9iyy51LN0LQ05eNyGqfJNRFcw8ff+y8RAWfHEJdowhTWlRZiZLXvOO+3xaUSrht6QlcCeRBtOX113tl5T5WiaVTKz5cJv1XFmu/zDjLmQvXPHjh38seKPwr34OhASUvayYUtOTjYWiwcpKSksXbaUFSv+ICgoiFHPPuv0sv78ZQknOnenyfAn6Lr42TxPI8plULrZfxjZLgAjYz0/LbpEIHM5jrZ3V3KqPxiX7qtXU0H9zJl1ykrInbZKc1pXn8CWXUU44iT7sOP7KRERERG5Yeg2EVKgpF8/ZvLWTEwVujH2m4k83aMJ1YNLYjGZ8SxVlhpNu/Dg4z2oZQbbiaMczYYSTXsysHEFfC0GmDzw9fW+INmzc/TwMezm8rTr3Y6qvhbM3oFUvymKCuzktwV7sAV35ZW376VN7XL4eZoxmb0pHRpF6yZhufPK2c6CxYeweTXksfeepmtUCL6eXpQOa0LPdrXIcw9O6y7mzt1GlkddHn3/RXrVC6GkxRP/sObc/86r9ClvcGrZPBaeLMQdFKw7+fX3OKwe9Xnso9cY1LQKpb3NmD18KVezATWDTE5sF8jKzsFuBNCo9c2ElizkPTmKGevZJw9lZGTwx4qVvPrq6wwaNJhPP5nEtq3bsF9m1NHlnF76CR+uSsYodwfvfP0Jo3o2oXpwCTwsXgSE1qPTQx8we+IAIizZ7J4xllmFOZG2OtjeXcmp/nCZvno1m2dB/Sxgt+PrZN3JvPnbybbU5j/j32J4q+oEeuduc//g0rg0U3JmPyUiIiIiNwyNbJGCZe/g4xHPUnbifxlYqxUPj2nFwxdOk7MV049z2b53MbMWPUKrzrfz0te381KeiazsOu/fCcsWs21EXer1fJfFPf9Z1kb+r9NgJn/+X6bc9jHD2z3B5HZP5C1nw9u0G/AV8bYM1k56l7lt3uXO+kMY9/2QC4vKs+zd015j7C2TebpJb96Z3Zt3zv3NTvaBX3jlrd8oTNYCOWz9/A0+vWUiD9e5k9en3snr52adxrwRtzBigePbJXH7Dk7ZI4kc8jG/lX2KqMd+LUxRN7x/ApSc7GxWr17D4iVL2bDhL7KzXThKwZrAjKceI3Dsu4xo2pwH3mzOAxcVksaO2S/xwNgNFG5QSg5/O9TeC7cK+XOmP1y+r36WcKnCzj6NaGQ+f8r5m7e7DuDjPc7U7EA/c2Kddn31Cu82/4xR0R14bnIHnrtgaZd/9LMznNlPiYiIiMiNQiNbxCHWgwt5qV9Phr4xjV//2svRlAysVitnUo4Qt2kF3342k5VJNrCf5JcXhvPk50v4+2AKmVYrOZlpJB1NZNemNayOTT53fwrr7q8Y8fQXLNl9nHSrlZz0E+zZEMsxw8B+ei1jBg1g5MT5rN59lJQMK9bsNI7Hb2bZuv1nL8MB27EFPDPwId7+fi17TmSQk5PBiT0x/LhwG+l2sNnPOxE8s41Phg/g4Y9+Yl38Sc5kZ5F2dBfLZ77JoL6jmHvRJSKOs6eu572hgxgx8WfW7TtBWpaV7PST7N+2nrgUDwwntkv6srE8MWEhfx8+zYHEQ4Wu6UaWk5PDX+v/4t1336Nv/wGMeestYmLWuDZoOcuetJZx99xJj6c/5rs/tpOYlE5WTianj+1l3e/TeOXeHvR48VcSrmDRjrZ3l3KiP1yur15NBfYzZ/r4me18Nrwvd7/7LX/sPEJKphVrTganj+9nW8xCvlu2B1e1Jqf2UyIiIiJyQzD8g8royZPFyE8/zQcgJmYt48ZPdHM1RcWgTJ9JrHjtJta82IZhs0/e8I9XjY5uwohHcscbvTlmzA19zxZPDw+8vL05ffq0w69p2apl7lOagHHjJzr8NCKRopN3PzV0tmOPfRrxyMNERzcBoHPnLkVZoIiIiIgUlsFsXUYk1zVT2cZ0rA+7t+7l4PFkMiylqda4G0891BRPWxwbtiTf8EFLcZOVnU1WEYxgESkqjuynREREROTGorBFrmteDfoxZlwnfC+8msFu5eD8j/l6lx62KiLupQ2DW2wAACAASURBVP2UiIiISPGjsEWuYwaep3ayJKYaDSIqU87fCzKTObRnCyvmfsX4GWs4qlshiIhbaT8lIiIiUhwpbJHrmJ3kmMmMGDLZ3YWIiFyC9lMiIiIixZGeRiQiIiIiIiIi4kIKW0REREREREREXEhhi4iIiIiIiIiICylsERERERERERFxIYUtIiIiIiIiIiIupLBFRERERERERMSFFLaIiIhchxo2bICfXyl3lyEiIiIi+bC4uwARERFx3htvvAHA0aPHiI3dTVxsHLFxccTu3s2p5GQ3VyciIiJSvClsERERuQ716duPsLDKhIeHExEewa2tb2XQ4EEYhsHJkyeJjY1l9+5YYmPjSEiI5/Dhw+4uWURERKTYUNgiIiJyHUpLTWXb1m1s27rt3O98fHwIqxJ2LoBp1aol/fv3w2QykZqaSkJCArtjY4k9+7M/YT92u92NayEiIiJyY1LYIiIicoNIS0u7KIApUaIEVatVPRfANGzQgK5dumAymUhLTSVeAYyIiIiIyylsERERuYGdOXPmogDG29ubatWrnQtgomrXplPHjnh4eJCWlkZ8fHyeACZxfyI2m82NayEiIiJyfVHYUkyFh4cz4pGHHZrWbLFgzckp4orkSpQuXdrdJVw3OnZoz83RTdxdhkihhIeHu2Q+GRkZFwUwHh4elK9QnvDw8LMhTDidOnXEw+JBeno6+/btUwAjIiIi4iCFLcVUYGBponXCKcVQRIRrTlZFbjTZ2dkkxCeQEJ/A4kWLAbBYLFSoWCFPANOxY0c8PTzIyMhgz549CmBERERE8qGwpZjw8PCgTJky7i5DRESuIzk5OY4FMHfcgaenJ9nZ2Rw6dIjY2Dh2x+7OfSLSzt1k52S7eU1EREREri7DP6iM7oJ3gxs2bCi9evXCMAwArDk52AGL2QJG/q+x2+3MmjWLqVOnXb1CRUTkumQ2m6kYWjFPAFO9enW8vLzIycnh4MGDeQKY2F27ycpWACMiIiI3KIPZCluKgXLlyvHZZ5MwmUwOTW+325k7dx6TJk0q4spERORGZTKZCK0UmjeAqVYNL2/v/AOY3bFkZWW5u2wRERGRK6ewpfgY8eijtG3bBrPl0leO2QG7zc6SJYv54IOxevSniIi4XGBgIOHhEYSHVyciIpzIyFr4+ZXCarVy4MCBPAFMXGwcmZmZ7i5ZRERExDkKW4qP4DJlmPL5Z5jNlw5bbDYbq1ev4c0339QNDkVE5Kq5KICpGYmfvx82m43ExMS8AUzcHjIzMtxdsoiIiMilKWwpPry8vPjvG29Qo2YNzGbzRX+3Wq2sX7+eN974L1ar1Q0VioiI/OvCAKZGzZoE+PsDcPLkydyb7+6OJTY2jh07tpGSctrNFYuIiIicpbDlxufp4cEdnTrSp3dvSpYsidlsxnLBpURWq5WtW7fy8ksv64aFIiJyzbowgImIiKB06dJAPgHMzu2kJKe4uWIREREplhS23LgsFgtt27alf/9++Pv7s2jRImbM+JqePXvQrVvXc5cTWXOsxO6J47nRz5GhYdkiInKduTCACQ8PJzAwELg4gNm1cwenkpPdXLGIiIjc8BS23HhMJhPNWzRn2LChBAcFs2jRIr7+eiYnTpwAwN/Pny+/+gJPT0+sOTnExyfw7KhRpKenu7lyERER1/D19aVyWOWzT0HKDWIqV64MXBzA7N69i6SkJDdXLCIiIjcUhS03jn9CliGDBxMSEsLy5SuYMWMGhw8fvmjaoUOH0qdPb/YnJvL0U09z+rSucxcRkRubj68vYRcEMJUqVcIwjIsCmNjY3Zw8edLdJYuIiMj1SmHL9c8wDJo0iWbI4EGEVQlj1cpVfPXVVxw8dOiSr/Hx9eWN11/j9dff0IdJEREptnx8fAirEpYngAkNDcVkMpGamkpCQgK7Y2OJPfuTEJ/g7pJFRETkeqCw5frWoEED7rnnbqpWrcqqlauYNn06iYmJDr3WbDbrqUMiIiIXKFmyJFWqVsk3gElLTSX+ggBmf8J+7HZ9lBIREZHzKGy5PjVo0IBhw4YSHh7O2rVrmTZtOnv27HF3WSIiIjekEiVKULVa1fwDmLQ04uPj8wQwifsTsdls7i5bRERE3EVhy/WldlRthg4ZTJ06ddm4cSNffPElsbGx7i5LRESk2PH29iY0NPS8G/GGE1EjAg+LB+np6ezbt08BjIiISHGlsOX6ULt2bQYNGkj9+vXZuHEjU7+axs5dO91dloiIiJzHYrFQoWIFwsPDzwUw4REReHp4cObMGfbu3asARkREpDhQ2HJti4ysSd++fYmOjmbbtm1MnTqNLVu2uLssERERcVC+AUx4OJ6enmRkZJCYmEhCwn52x+7OfSLSrt1kZ2cXSS2NGzdm/fr1RTJvEREROY/ClmtTlSph9O/XnxYtW7Bjx06mTZ/Gpo2b3F2WiIiIuIDZbKZiaMU8AUz16tXx8vIiJyeHgwcPEhsbdy6Aid21m6wrDGD8/P2Y+fXXbPn7byZOnKgnK4mIiBQlhS3XlsphlRnYfwAtWrZg1+5dfDNzFjExa9xdloiIiBSxfAOYatXw8vbOP4DZHUtWVpbD82/UqCGvv/46VqsVw2Ri/rz5TJ8xg7TU1CJcKxERkWJKYcu1oXKlSvTu04fWrW9lf8J+vv5mJiv/WKlHSYqIiBRjJpOJ0EqhVK5cmcqVKhMREU6tWrUoVaoUVquVAwcO5Alg4mLjyMzMzHdeffv2of+AAXhYLABYrVYyMzOZNn068+fN171jREREXElhi3uFhJSlT58+tG/fnsTERGZ/+y1LlyzVBx4RERG5pMDAQMLPPoI6IiKcyJqR+Pn75R/AxO0hMyODF154nqY3N8VkmM6bkx2bzU5CQgITxk9g2/btblsnERGRG4rCFvcoU7Ys/fr2oV27dhw/dpxZs2fz+++/K2QRERERpxmGQfly5QiPiKB69eqEh1cnPDwcX19fbDYbifsTCSoTjE/Jkvm+3ppjxWQ2sWzZMiZP/pykpKSrvAYiIiI3GIUtV1dwcDA97+pJp04dSTp5iv/9738sWLAAq9Xq7tJERETkBhNSLoSI8HBq1apN9+7dMYzLT59jtWK3Wpk1+1tmz5pNdk7RPBVJRETkhpdf2DJ61Ch3lXNd2bZjOz/O+dGhaf39/Ol5Vw+6detGSnIy3/3wA7/89MsN9yGm+53dqR1Zy91lXBfeHDPG3SVIEYqMrEmPO3u4uwy5ytSv3U/HofwFBARQp24dp16TkZHBnrg4Tp7UKBcRkcu5Ho//Ol663kX5gMFsy4UTtWzV8mrWdF37kcuHLX5+pbjrrrvo2rUrZzIymDHja+b++OMVP77xWlU7spbaj6Ouv32yOCG4TBn1heJI/drtdBxyHW9vb2pHRbm7DBGRa991ePzX8bJoXJgPXBS2yJUrUaIEnTt3pk+f3lhzrHz99Uzmzp3r1CMaRUREREREROT6dMmwJSZmLePGT7yatVwXpk/94pJ/8/b2pkuXLvTp3Qurzc6cOT8yZ84c0tPTr2KF14ZBQ+52dwnXnBGPPEx0dBN3lyFX2bjxE4mJWevuMqSIqF9fu3Qc+tcH779DmeBgADKzsjh27BgHDhzk0OHDHD1ylMOHj3Dk6BGSk1PcXKmIyPXhRjr+j9od5u4SrmtjIuIv+TeNbHEBL29vOnRoT9++ffHy9OSnn35i1qzZpKWlubs0ERERKcZMJhPf/zCHI4ePcvjIYVJSTru7JBERkWJBYcsV8LB40KZtGwYNGkiJEiWYP38+s2d/S2pqqrtLExEREcFms7FixUp3lyEiIlLsKGwppHLlyjHli8/x8fHh199+Y/b/ZpF06pS7yxIRERERERERN1PYUkjVqldj3tx5zJ49WyGLiIiIiIiIiJyjsKWQ1q9dx6TPPnN3GSIiIiIiIiJyjTG5u4DrVaYe4ywiIiIiIiIi+VDYIiIiIiIiIiLiQgpbRERERERERERcSGGLiIiIiIiIiIgLKWwREREREREREXEhhS0iIiIiIiIiIi6ksEVERERERERExIUUtoiISKEZpVvz4sz5rBjTFq8bcHki/zIT1vNN5v7+Pc9FW9xdjIjzDH9ueX4Kkx9uQpDhnhLMUSP4bdtmVr4Yjad7ShABvIm481W+HtefqtqdX3cMLy8e7RDE7OZe1/x+RGGLiIgUmuFdgdp1q1KmpIWr8dn94uV50WjE//hr3c+81T7oqtQgxVepSrWpXSkAb+NqtzS1c7lSBqVaPMbrA2+iWpCJLLfUYKZ2+7ZU5wgLf9vophpEALLJKVWJOu0e4/W+lTC7uxxximE2ExFkIdBinD0eGtSpH8j8vsGMqmy6po6RRRa2eN/8GLN/WsC6tevZuW0LsVvWsmHFT/w4+S2eH9aWSP/CNmszUXdP5NdFU/lPTXWNG5HajhRf3oTddh9vf/EDf65dz+6tf/H3qt+YN+Utnu1dn9IGqB1fzDAMDMOE6Vo6usp1w6frR+zYuZW9l/jZMe4OvIu8ioL7tSvbuU/Xj9ixYx3zn44mIN/5eXDLGyuI2/YTo+o5u59xfh910XF/63o2r/yV+V++zaj+0ZS/1r+6vB5YIhj6RA8qnviZtz6K4bT9n7a/iXkPVXfwZNPAp3pHXpq1irjNH9K1hJM1mGvRsX0VOLKEnzdeHLU4X09RK+zx1pFjefF07bzHVvbOHMOkbV7c/PDDtPErxm+KE7zK+TKhazBz+5Rl8cAQlg0oy6+9gvminT+PRnoR6uZRQgbOhxsRtQL46s7SDC5dFBVBkW0Sc5lw6oZX+HeYt7kkAWWrEFC2CvVadebuBzcz9YWn+b+FB8hxas4mAsJqE1E+CU/1ixuS2o4UT2bCer/Pt6/eQrD53wZqCQqlTouKRFg2MvW7TWBXO84rk/Uf9qHhh+6uQ+RKFNSvi6CdGyWIuud9xh0ayn3T41w4ysD5fdRFx328KRVciajgSkQ168iAnpMZfu841qTYXVZlcePTcgiDaxlsGzeZhaec3I7mUlS9uQN39biL3nfUpayHAZnO12CJak+HMDg043f+ui6GtRTmeOvosbxIChZn5Oxm+qSF3D22A/f1mMjCr/Zjc3dN1zhzCQuR/uZ/L90xDHy8zYR7mwkP8aZbjTO8vjCF5elXuzI7f286SedNzr7OwL+UB1V8bEV2OVIR5085bJvQn14TtpOBJz6lyxFerzmdBw5jcIv6DPvgU4z7+/Pan6e1z7nG9OjRg5CQEJYvW8b2HTuw26/2O6S2I9cGL29vXnn5JZYtXc7KVSs5ffp00SzI0oChD7ckiKMsfu8lxny/gfhT2XgGViKqyS3UO7OUI/oUIMXIgAEDKFmyBEuXLiM2NraIl5bD32N7cufHcViLeEkmr1IEBXiTczqJpHTnvjJwLTtWux8tnx3L6D2DeG1VspuPpzlsG9+PuybuINNupoR/OcIbtWf4U/+hc917eO3uBXT6cFuRvz+uUNB7fNXbgOHP7T3bEpy1nvFz9ji9DU0h3Xjzk9E09YTsw1vYbIuiXpCzRVio06ENYRxi6m+byHb25UXMZe9JMT2Wu3u/Vrg+Z+fUsu/4+UgH+vXsQsT0j9l5Pexg8tGtezdCK1Zk6bJlbN+2vcjP23ZvPsFDW3LIArw8zVQK9qJnfV86BZZgZL0MYlZnkVGkFVw/inywjy07kyyrHTuZpB6PZ+PieDYu+ZlFT09hyj01GTRqMLN6TmS7FYyg1ox+/zE61gwlxM8L+5njxK37jcnvj+eHnWl5PwSYazDix82M+Gc5R79h8O2vsyrbyflIvvz8/OjatQtdu3bhxMmTLFq4iGXLlrJvX/xVq8EtbccnnM73P8y9XW4msqwXZ47s5I8fPuXtSctIvNY+GchVYQD16tWjXr16PPyfh9iwYQOLFy1m9Zo1ZGYW4qu9SykZSliQGVvCfMZ9/ge7zx7ws47GseanONZcOP0V7wMNgloM5/mht1K7eiUqBPtR0gMykhLYuHAm7439hg1JefeWpsD69H/4QQa2a0i1IA/SD+3gz9XJhFwwZtPR5fs36MNjQ9vTJCqcKuUCKGGc4Xj8b7w65BV+OWl3cHlmIh76hp9HlOf7+2/j2RXZGP538vkf/+W2/L6myFrNC23uY8ZRu/r7NS4oOJA7OtxBjx49OHLkCAsXLmL58uUkJia6u7Q8HD3+mAJvYvhLz/FA2xqU9jCw27M5nbCA14Y9y3cHz050yX59cTs/p0Rl2g57kPu6taBOqB/GmSQO7FzOxBdeZ0785c4cctg0bQKHOzzK4LdfZXOfJ/nhYMFnGg73m8vsoy7FlpNFttWOnRzSkxLZvGgKTyaXod7UIVRt0ogQ0zYO2pyswzuU2wY/wL3dW1KvchAlyCD52AH27PqbBV+8x+SYU9gd2B85srzLv8e2Av+e+35Wof09DzG8W3NqV/DBlhTP+sXfMnHCN8Qc++f9KbjePEpG066ZLzlblrC4EGf6tiNrWBrzN8c3f8VHX24getyvzoctlig6tasEB2fw86Yr28E68l5cnX6ZT3HOHss9K9By4H3c2+0WGlQtQylLNqknDxMft4uN8z7hte93YcWDFq8sYGqfJMb37MUHO85rBz0mEDOmGX8+34a7vz2J3eF1vzptvtAc6gcu6HMZG1m48hQD7ryNNmGT2Lnn+kxb/EqVonPnznTu3JmkU6dYtHAhS5cuY+/evUWyPLsNsu25A7QyMq3sPpDOO6kGEZ19CS/jSWUji0NBJbi7ljf1Ay1ULGnCGztJpzMYuzCFZRlgeFi4PcqHPlU8qV7CIONMDuvi0vhkayaHz2s6Jm8PutX1oXslTyp7w5m0HP46YiP4gtFmVeoE8kV9M78tOc6Yg+ftAy1mWkb60reaJzV8DExWO4eTMpm2OoXf//n+1LAwrHMIw87+13bmDI//kMJfLghG3XNllT2Z1R+9xewOkxkS0ZHOkZ+yfasVcgKo3qgGof98QPYNoVbrIbxTpyzZ3Z9i3nEHYxJXzaeYy87JxsPiQVBgID169qBPn94cOniIxUuWsHTpUg4ePFjwTFytKNtOybo88vlnjGxY6tz1ft6V6tP10XE0qDCC7i8sI0lNp1gzm800btSIxo0bY7Va2bhhIwsWLWTN6jVkZ1/h2fmZQyQmWTFVasPgjt+xa348Zwo7L4f6g4nAeu3oemvtPAcCn+DqtOj3PA1qlKDn4CnsOvsFkOHXnBemfsSwCO9zNx7zqtyATpVz/50ndnJw+WWb9WJwp/OXX4oyZT3ITLU7t7zCUH+/LlitVsxmMyEhIfTr15eBAwdw8OBBlixZyqLFizhy+Ii7S3SsvZvK0XvMRzxzqx9GThonjqRi+AYRUNaDzGQbFPbuBV6RDP/0c0Y1Dfj3OnXPEMIbVML3TMGfEq0HfmH0k6WoOuVuXnvvHnbe/RnbLvd1pBv6jS3Hmju032T6dx0drcM7kuGTJjMquvR597rxISi0BkGh1fD8awpTYk5hLWB/5NDyjALe4wLbAOBdmwcmTeaZaP9/1zWkBrf2H03zW+rz5KDRzDtopaD954UsNRtS38dG4sZNeU5iHGaN5ZN7++X+2xRCdCFm4VG3Pe1DIXHa71xR1uLoe++ufunMsdw7kuGffsazTQP594ojC/4hVakXUpWap3/nze93OT+aywXHYJe0+cJytB8U9P450ufIZPNfW8m+K5rGDUrBnlOFr9vNsrKz8fTwoHRAAN3vvJNevXqdO29btmwZBw4cKNoCDPLcmDaoXAl6hHmc174MAktCVjZg8WDo7aW5u4xx7j328vWgTf0AavucYvjqTJIBw9OTR9oG0CvAODdvz1Ie3Fbq7DoXVJPZQr/bSvNQyHnHD7NBWLCZkldpAJb7nkZ0ZhNL1yRjM1WkVoQPAPbTK3lnyJ00a9KY8Fr1qHVzF+6ZvJmMoNu4q3XpvHcWtu5iXPd6VK0ZRdWaUVRv9W/C7NR8xCEeltyuUr5Cefr17ctnn01i0qRP6dWrF4GBgVe3mCJpO2ZqDHmRRxqU5OiycdzTqQWRUY1p2usFvt1jJfTORxgYfm3crk3cyzCZMAwDi8VCw0YNGD1qFDNnzuSpp54kOropJlMhd6vZ65ky/g+OG2Hc9e4PLJ7xGg92iCTwUpG4q/aB9hR+Gd2e+vXqUT2qKS0HjmHBYRs+9QcysJHH2Yks1Ll3FEPCPUnZOI2RvW4nqk5D6t8+kJGT13L8gs9Uzi3/NAte7sJNDRsQXq8ZrXp/yJps55Z3IXvyHO6pG3Vu21SN6sDIeYlk29LZ+vUUfjtuUn+/DpnNue9J+fLl6devD59PnswHH7xHt+7dCPD3v8K5W6gzci6xF9wcd8+Gd+hQwIXcjrR3o1RT2jcthe3vT+nRrBk33XI7jRtHc3OPd/nj/GvbL9Ov89kiVB34Ik9E+5Oxaw4vDO5Io3oNqNXkdu4Y8ha/OfTFkp3U9eMZ+cF6bA0e5oPHm1Dqkh+SnDxOOrUueRlmT3wCQ4lq1Z83XupNZbOVhPV/nQ0KHK3DTPVBL/FkdABZe+bz8uA7aFC3HuF1m9LypWWk57d58t0fOba8gt7jgtuAmfDBL/J4Ez8ytn/LM31y93sN2w/nzcWHMCp05JVn2uW9uWq+9V60NfGtUpVyZiv79iS46Z4UHtTv0IZQDrLg17+v4BIix9ug2/qlw8dyMxFDXuHJpqXJSfid/7u/O00a1KN67UY0vGsCm67gRPDKj8GuafOF43g/uPI+B2Dn9N59HLF5UKVapcJv9GvM+edt/fv1ZdKkT4vkvM04e8+WWhVL8nRzH8JNcOp4Ngnn9q92/lhzgm7fHKX1zGP0+SWNTVaoFlmKIWUMThxI5Zl5x2jz9VHu/CWFX5LtlKvmQ/eA3FfXrF2KngEGqcfTee2XY7T/+igd55zkta1ZXDiALz+VavhxX4iJzFNneG/BcbrMPErbWccYuvA0y8//YsGew5c/HaHV9NyfW79zzagWcNfIFgByOHkyGbtRipK+JTGRgs0wCKjbj2eev5naYeUJ9Ejj0HEbZiyElA/GxEnH0l1XzUfyZbbkHsgqVqjIkKFDGDZsKLt27cTDcrUeF1AEbcdck65dIrGkLOS/T09iSXJuDz665QdeHdeSDmPb0Dw6mPG7r4FvUeWaYTbn7kJLlPCmVauW3HbbbSSnpLBr965CzM1K/OyR9DwyjKdHDqZj47t49qaejDi4jh+mTGDczLUccfTDlzP9wW7l9LGjpGRagVQOrPua/5vWkduerk1kZBlMMQexmWvSoW0VTJnrGfvk2/yYePYIdGAj86b/Tr+hTWhY6OXnkHQgkRPp2UA2B+NTwBzl3PIux1SGti9/wttdgtk783Hufmslx43aDFV/v24ZhnGu79WIqEl4eAT3Dx/Oli1b8PYu+ucG5VNQge3dZrfnDu8vE0l0ZDA71x4hw57Jsb1XcEmUuRpdutbBK3szb414iRl7z/aqzCPs2nCcyEe+589Ha/773bw1jgm9e/Du1guPhlnsmvYcrzX5hrcHv8Hzq/sxeklqPstz7Dg5cffJQq7Q2cBr5IW/t5O2fSovf74196b4jtaxx49OnaPwtO7gw8dfYOrOf86OUzl2IjX/S8rz3R/Vdmh5H8+7/HtsFNQGzBF06x6FZ/YW3n7yNWbH5b5P6fGrmPTUy4TN/4T+rbvRpvRvfHvyMvVexERQcCAmWzonTqS751J6j7p0bFsBEqfx65YrGNbicBs84tDnwsL3SzORj8xm/iX6l0PHcnMNunWrjWfONj545Bk+2/XPdrGScuIUZ67kjbriY7Br2nyhONEPvsu5wj73zyY4eZyTdhNVg6/yl8dXicl88Xnb7t27sJg9CnjlpdVoEMSyBhf/Pvt0Bh9tzvz3fi12O8lpVpJy7ICdI6cBw4M2VTwwZ2UwYWUaf54dnnLixBk+3OzJLa28aBxiZnqyiVsqWTBZs5jyx2kW/HNYSs1m0c5MutbyJOpyRRoW2lT1wNOazSfLU5jzz+7RamfvsasXO7sxbLEQGOiPYbeRnpaO3fDjlhe+ZHL/MDzORa5eVK4EYMVkcrBUV82nAC1bteSnVvNdMq9r0b59+wqeyACzkfstfo2akXmS8lKl/Dh9Or+DvisUQdvxrES1UBOmEh34KKYDH100gZUKoeUB15x8/fTTjdt2iiuLJfeg5e/nR5PGN537fUR4dWJi1jo4lywSl0/iseVf8WbjTgy5eygDbr+JAc9/TtuWbzDgP7OIKyhwueL+YOVg3D7S7FH4+vrk9muPClSpaMKW+BfrDhVwgHJFf3RmeZetpRRNHhvP2N6VOfbzaO59YznHbECJounv6teulZqazwn/hQwwnT0O1atXL8+fSpYsSXq6o1+vFvIGuQ62d/vpVfyw6Di3db6V56Yu5MmkeP7euJ5lc6cz5dfdpBXmxMoSRkQVM7b9MaxKuMKvkKwH+f7lV2le+wN6vTqaZVteJO3CaRw8TpoobNhynn9OkuynWTfleZ6ZsIS9/wxFcbQOS/DZ7bOKpbFXcILv4PKMgt7jgv7uGUZ4qAnb/jWs3HfB+5n2Fys2ZND/jjDCK5lwdhN7entikEWWm54A5FG/Pe0qwP4vF1zRiA2H33vjDC3d1S8Bh47lHpWpHmrKbZ9xLrxJ2FX8TFxgmy/M9nOiH9g3X2GfO1ufPSuLbDt4ehX+S2N3H/+tOTlYC7op7nnnbRE1auY9bzNbOW11fkSvzWbnTJaNQ8nZbDqQwZzdmewrqDmbzVTyBZPFm1f6ePNKPpOU9TVhMpmo6AO21Gw2X3RAcoDJTBU/sKVm8VcRPdvCEe4LW0rUp3VTf0y2eHbsToPA7gzrURlz0hrGv/g2M1bHceyMheA2zzNnbDeHZ2sEtnXJfAqyY8cOfpgzx2Xzu9Y0uakJFUMrFjid3W4/d8fr5JRkAgJyKHZR6QAAIABJREFUH1JedEELRdN2zn6wuzQDrxL/3959h0dV5X8cf98pSQiBQOhIJxRh6RJQQQFBfoK6SlG6usqusiwWRFEEFUWxI7iwKrpSdWEVRRClGtpCEERKKAktIYBAEtLLzJ37+yOolACTMMlA+LyeZ59FZjj3O3PPmTv3M+eeG3jRZxTE6xMn+qwtKVpOp5OnR4706rmm243d4SAjI4PSpfMucYuJ3VeIreZwbPMC3tz8DR826cX4SS9w1y2P848u3/HE0ouvVuKLz0ArN5dcy8D4bZEDw553zalhu+RlmD75DC7A9i7MQe3eE/nnX5vg3jSJv41ZzOHfvrcV0XjXuPatbt260qrlpecweU4fgyyPSUpK6u9TpL0PWgrP6/5unWTx84NJ/7kvd7RvQetWzWjduS5tOnWmsa0XwxenFGLjtrx1SPL9gm2y+4NehH/gfXPWyZW8Mu5L2nzYm5fGrOWtcxea8HLcFH7Mnhl4BdBgyFTmPdeO+o0rQ84ZW/a2DpsThw1wuy9vNrO327vkPr7E4yuL7iL33OxcLAIIKK4JyGcJoHX3LlQnnk+W7uCylknwcl/Yinxceju+LnIsX23lXdJleryabZT37ECCgi7eT3xyDPZhny943lKAcXC5Y+50fUZAAE4DcnMKn0b6+/jfocPNtGvX/pLPO/O8LTU1ldByedfrFDRo2bs1kaE73IW7LNG69J3PA+0GhvHHei6Fu0D/j3Ve/LkEn3/CFiOU9sOfoe91Ntx7f2DxLhNbeFWqBkDmsllMWb779II3LhJPpJ6zEKKF2+3GIpjg4PMHpK2it+1cnpMnTrJ2zVoftnhlqV+v/gUfsywL0+PBYbdz8OBBli5bxprI1Tz66KN06NihaAsrqr7jSuBgggdP6Dc8cvtYVhXxd/SS3HdKmqCgILhI1mKabux2B5lZWWz83waWr1hBmbJlGP3ssz7YuoeU6AVMmtebHqOaEB5eDfvSA8X/GZgbx77DHmx1bqJT/X+yfe+Ff7bwyfYLsL38GZRp+zj/Gnsr5eL+y6NPfMrOM08ci2i8a1z7VqtW+cxR/o0FpsfEZrMRG7OXVT9G8uOPP/L3x4YV/XEIO6evYipYf8+OJ3LWu0TOAuxlaNx7PJ++1I1O3SMIXrz0ouM6X6f7sa1WBDfWtLP93F+BC8zi1Np3GfN5BJ/1H8UTvwZjcMYPJ16PG0fBX8t5comZ/TxjWnzB5J4jefuRrQz4cHfee+ptHY5WHDnpwVbrBiKq29gRX8hZcgX5vLjoPv6OjIs9/v2BvM+92u24ubad7WfeFaV0azq2CoLcOPYf9lCwUw8PiSeT8NgaUqFCMAbFfIvvgBb06FoV4j9jyY7LXJHSy31hbzTMf+MyX/kcy1cc5kCCB1vN1rSpamNHwsX6p4e0lHQs23U0Cg/F2Jp4wX3ok2OwL/u8t9v8Te6hAowDL7bvRX1GWEXCjLxxUlj+Pv7XqV2bdu0u/LjbNM8+b1u9hkf/9rdiOF7mw2OSkAGegCxGf5PK/y70sWA4iUsHW9kA2oca7D5VwE+u09uxhQTQugzsyXcegIXbsgCDUkWUihT5Ark2hzNvhW17AKUr1qZF536MmT6Pfz98PUGug8ydOJNdJngSj3PcBaXa9WJgm+qEOAywOQkJCTonEbI4fuwElr0a3fp2o26IA3tQGPVvaEp1e0HakYJyu/NGw9GjR/nPF//hkYeHMnz4P1j4zUKST/l+9e5i7Tvs4Ydl+/FUvIuX3nyY25pUpWyAHZs9iPI1mtKpbW31H/mdx+PBsixcbjcbNmzk5ZdfoX+//rz9zjts3br1918NCiSgFX997WmGdPkTtcoHYTcM7KXCqNv2Xobd0xC75ebE8SQ8/vgMNPfw7aJduBxN+PsHbzC0Y33CgvLGR2jF8pz5HdQn2y/A9vJjhHVi7BsP0MjawQdPTWRl4jn7w9R4v1p5zLwv3glHEpg5cxZDhjzAk0+OZOE3C0lNKcIZlaflutxYRjlad2pPjWC79/3dXpcuvbvQ/LqyBNgM7E4H7rQ0cgDDAOMS4zpf5h6+X7oP09mCx6eMZ1C7OpQPsmN3hlC1UUsaVSjEVzwrjfWTxjM3PpTq1YPO/o3Z63FTiNeSH89xlox/kfkJgbQa9jJDrw8oWB3uXSxbeRRPYCsef2cUdzWtQkhAIOVrt6VXt+vxepKHt9u71D6+1OPmXhYujCbX2Yx/vDuWPs2rEOwIILT2Tfz1rZe5r5rBqchvWe7NqpBnsUg/eJBfTTt16tUq9jtjBLbuTtcqcGjZMi43a/F2X/h1XHp7LDdjWLbiIJ7ANjzxztPc2aQSwQ4nZaq34O5Bt9PgrPZN9m/fRaoVSIdHn2NAqyoE223Yg8pQqXyps8apr47BPunz3m7vrG0XYBxc7pgDwKBMnTpUsbk4uP8y1pq5Av1+3nYk77xt6CNnnLclJ/uvMMtFZLwbT6kgnri5NDeH2Qmxg80wCA1x0r6yPa9/WS6WH3ThtjkZfGtZ+lV3UO7088qUsnHJFdosFz/GuTHtTh66pSz3VLETagebzaBSeSf1TjeQmOnBY9jpEB5ETSfY7TZqV3ZSxUeTDYv4+6SDJsO/ZM/wc//ewjy1nRkvjGTC+tS8dDZxJfNWDKdjzy6Mm9uFcWc932TvGX+Oi1xJ9IhmNO/1Nit7nf5r11Ze6zGYj+O9bUcuxW6zY7pN7A47x0+cYMXy5ayOXE1cfHwxbL34+870TybwaedpDO32FNO7PXVWK66f36TbgBkc8s8y/nIFsLCwLLA8Hjb9tIlVK1cRFbWJXB9dBO9ochsD7nmI2r0fynfrWTEz+fiHJCwsP3wGmuyd8RJv3/QxoyO68/z07jx/zjN++8XM8no8+mZ7+XG2voMe1e0YRjOe/GozT57V9BE+e+AOxmu8XxVsNhtutxuHw8HRI0dZvmIFkT9GcvTYUR9v6UKLswLuHbx51wCm7Tc5vGs3p6zGNB4yjR8qP03bJ7zr70aFdjz88lhuOnc9Qk8SS5ZuIgOTrIuN67j8anaz85NX+fCWqQz70z28MvMeXvntISuDb0fcwoilF7uPc/6stCjenbCALv/qQ41ztrfDq3Fzie9pcd4PLCtlLW+88g0dp97LY+MG8sOgfxNjeltHNps+epuFt73NPS2GMPmrIee07u2Zv3fbi7vEPs6scNsl+0DMrPFMumU6o9r25a35fXnrj3cCV8ISXnrjB6/uwHHeK9i7ha0Zg+neojlVbNs5ctYuuFDfNzk840E6v7blMi79CaRN985UIY6Pfoj2sp2L19Nlihf73svj0OWPy/P7ckGO5ds/eYM5XSczuNUDTFnwwHnPPvM4l7FmFrN2deUfTe/g1S/u4NWznvnH9xDfHIN90+cvPKvlEn3uPe/GwaX236XHHEAgzVo3wWnuY8svRR/YFyWH3Y7bdOOwO0g8eZJlK1YQGRlJ3KF8DyB+tXdnGvOvK0e/miFMrBly1mOuE2kMXppJggUHdqfycbXyPFoliL93CeLv57RzqW/gMdFpfF69HIMqlGJkt1JnTFS3WLH6BC/FWSQk5BDT3Mn19UOZW//0nQ09Lv75bRJf+GCtlyILuM0TsWzfd5TEtGxcpoXHlUXqyXh2rF/CZ289xd3dB/LysoQ/PnitJJa8MJSRn6xix5FUckwTd04GyccPs/eXjWyI/WPaoxkzgxGj/s2qmJNkmibuzET2/xzLCcMoUDtycalpaSxatIinnhzJQw8+xOzZc4olaPFX37HSNjFx0ACemLqIDTHHSc02MV0ZnDy0jcif4i99L3cpsTweD9u37eD9Se/Tv/8AXhn/KmvXrvNZ0ALg2b+QN96dw5KovRxJycb0eHBnnSJhzwa+njqaPgPeZn1aXk/2y2dg1i4+Hno/D739X9bu+ZXUHBPTnU3ayXiio5bzZeT+vNt5+mr73m6vkDTerw5Jycl8/fXXDB/+Dx4ZOpQvvviiCIIW72VGTuKpfy5nx7E0Eg4fJdfL/m4YR9ny4zYOJWXh9ngws5KJ27acj559mFGLTmBxiXF9AVb6Zt55YBAjpn7HTwcTycg1cWUmER+9mX2pzkKun2KRsuZ9Xvvu+HnX43s7bgrzWi5Uy6nVU3hn1SmCWj7CUz0qYBSgDs+JZTwz8DHe/GoT+xOzcbuzSdwfxTfLo8m0wGN5F/x4s71L7WO86ANkRfOvoQMYNmUxPx1KIsuVS8bxvaz+/HUG3T+ahUcKealYRhQrNqTjaNaZzpWLcW5LYGt6dKkEB5ezJNo39wH1at/7cVwW5Fhupaxj/OCHeenz9ew9nk6uO4eUhO18/2UkB859u3J2MPmvf2PCl5s4kJSN6TFxZ6dxMiGGLauXEBmbldeHfHQM9kWfL/T5lpfjwCdjLqgFXW8uj7UvkpWXfSmmf6WnpfPdou946qmnGPLAg8yaOeuKDFoALFcu05YmMX57Nj+f8pBugumxSEpzsfG4+cf3O7ebz1cmMWpLFpuSTdLNvEV5M7JMYn7NYUmC+6IhruXK5ePlSby8PZttqR4yTXC5PRxNyuVQbt7sJs+pTMavy+B/pzxkW+B2e4g74fbFUu8AGKEVKp01Fn5bTTkqahOTP5jqo82UHLNn/hvIuzbP34shFaXy5cuTkpKCx+P9L1DPjR79+7V/g4bkl+hf20YMH0ZERFsAeva808/ViLdsNhuhZcsW6FK5Dh078Nzo0QBM/mBqAe5GJFcbjeuiExYWRnJycoEuy9NxSC7NoNJ9H7Fm/A1sHHsbD85PuiZ+hAvpMoGV/+zJ0Um96fVhAe+6VUhBHV5i1ce9SPuoP3e8t7NYtlkS2KoNYM6yMbRaOZKWI76n4PPTxDsG5bq/wfJJXTnw5j30+3dcgfrolXT8v9zzttExtYuqtGvCxAaHgHzyAYP5xX3pplwlkpOTCzRgRUoqj8dTJGsSicjFJSUlFW79I5HTbJXb0LNbGxpWDyMkwI4juCINOz7EhMfaEeA5yM/br53ZzumrZzB7NzQd9Ai3lSu6Ox/9IYiI7rdS2TrID0t3K2iRK4+jAYOGdqVc0lKmfxV/VfdRnbddubQGoIiIiIiUOIEt+zFxcg9Czs0WLJMji6Yxd+/VfHpVQO4YPntnAX0+6s3o4V/zvwkbSSvKpKnUDfToXBFr/1d8v/saep/lKmGnbr9nGdrUxcYJU1mecq3ErlLcFLaIiIiISAljEHBqD6ui6tGyQS2qhgZCTgpH929nzcIZfDBnI8evqR+CLVLXvc/YObUZnGwRaFCkYUtw2+50qWCxb/5ylLXIlceJM+Mw0cuXM/aLgl0+JFIQCltEREREpISxSImazogh0/1dyJXDOkXkhL8QWQybylw9lojrxxbDlkoez9G59P/TXH+XUcJls3fBi/Rf4O86pKTTmi0iIiIiIiIiIj6ksEVERERERERExIcUtoiIiIiIiIiI+JDCFhERERERERERH1LYIiIiIiIiIiLiQwpbRERERERERER8SGGLiIiIiIiIiIgPKWwppOYtmtO8eXN/lyEiIiIiIiIiVxiFLYVlweuvv8b48S/TsGFDf1cjIiIiIiIiIlcIhS2FtG3bNkY98wyBgQG89967TJjwKuHh4f4uS0RERERERET8TGHLZYjeGc2zzz7HmDEvULp0aSZNeo8XXxxHvXr1/F2aiIiIiIiIiPiJw98FlARbt27liSe20rJlS/7yl4d4//1JrF+3nlmzZ3P48GF/lyciIiLXqL59etO+fQRHjx7lyNFjHD92nGPHf+XXY7+SmJSEx+Pxd4kiIiIl0gXDlvDwcEYMH1actVz1tm7dyuOPP0HbthEMGTyIadOmsn7dembMnMmRI0f8XV6xUt85ny4zuzbd0f122ke09XcZUkQ0rq9cOg7lCasQRpXKlalSuTItmjfHADAMACzLIteVS1ZWNllZWeRm55CdnU12TjY5Obl+rVtE5EpWko7/A6ue8HcJJdYFw5awsPJE6AShwCzLIipqIz/9tImbbr6JIYMHM23aVFavXsOcOXM4duyYv0ssFuo7InkaNCg5B2ORq4mOQ+czTocsZ/53YEAggQGBlAsN9VNVIiLiT83KZPq7hBJLlxEVEY/Hw9o1a1m/bj033XwTDz74AP/61zRWrFjB3Lmfk5iY6O8SRURERERERKQIGKEVKln+LuJa4HA4uOXWWxg0YCBhFcJYsWIFc+bMJSkpyd+liYiISAnlcDh4++23vZplZ5omhmGwaPEiZs2cTWamfu0UEREpFIP5CluKmdPp5LbbbmPgwAGUDg7m+6VLmf+feSSfOuXv0kREROQq5nA4qH5ddcLDwwkPD6fB6f9PSUmhfLlyOJzOfP+d5bHAgB07djB12jTiDsUVc+UiIiIljMIW/wkMDKT7/3Xnvr59KVWqFIsWLWL+/P+Snp7u79JERETkChcQEEC9enWpXz+c8Pr1CW8QTq1atXA4HGRmZrJ//35iY2LZt28f5cPCePDBB7DZbOe14/F4SExKZPr0T1i7Zq0fXomIiEgJpLDF/wKDgrjrzjvp27cPdrudxYsXM2/efDIyMvxdmoiIiFwBnE4n1apXO2vGSoOGDXA6nGRmZnLw4EFiYmOJPf2/w/GHz7qlc61atZg2bepZbbrcbrAs5s2bz3/nzyfX5SrulyUiIlJyKWy5cgQFBXHnnXdyX98+mB6Lb7/9lgULFpCVleXv0kRERKSYBAUFUa9+vdOhSgPCw+tTo0YNbDYbGRkZHDp06KLBSn5sNhtffflfnAEBeDwebDYba9eu5ePpn3DyhG75KSIi4nMKW648ZcqU4a677uLee+/B5Xbx1ZcLWLhwIbm5uf4uTURERHwoODiYOnXr5B+spKdzKC7urGAlPi4eyyrc17b33nuXhg0bEh8fzwcfTGXHju2+fTEiIiLyB4UtV66yoWXp3asXd999N5lZWSz4agELv/lG03xFRESuQqVLl6Z2ndr5Bivp6enE+TBYyc+AAQNIS0vlu++WYJqmz9oVERGRfChsufKFlg2lV+97ufvPfyYlOZkv/jOPZcuW6YuSiIjIFap0SAi1a9c6K1ipWbMmhmGQlJREbGwsMTGxxMbuIy7uEMeOHfN3ySIiIuJLCluuHhUrVaJXr3vp0eMOkhKTmTdPoYuIiIi/hYSEUMvLYCUmZi/Jycn+LllERESKmsKWq0/lypW4//776datG0eOHmHevPn8uOrHSy6OJyIiIpcnLCyM8NOBSoPTt1quWrUqwHnByt49uzmVkuLnikVERMQvFLZcvapUrcJ9ffty++23czj+MHM+n8u6tet8en23iIjItercYKVBgwaUL18eOD9Y2bN7NympClZERETkNIUtV79aNWvS97776NTpVuLi4vn8i8+9Dl2Cg4PJzMwshipFRESuXOcGKw0bNaJcaChwfrCye3c0qalpfq5YRERErmgKW0qO2nVqM6Bff27ucDN798bwxRf/ISpq4wWfX7ZsGd6fPJnxL4/nwIEDxVipiIiI/5wbrDRq3IjQsqF4PB4OHz6ct7ZKbAxxh+KIjY0lPT3d3yWLiIjI1UZhS8lTp24d+t/fjw4dOxC9axezZ8/ml62/nPe8hx56kD59+pCWns6okaOIPxxf/MWKiIgUoXODlcaNr6ds2TKYpklCQsLvwUpsbCz7YveRk5Pj75JFRESkJFDYUnI1btyI+++/n4iICKKjo5kxYxY7dmwH8m4n/dmMzwgIcGKaJmlpaYwc+bRuPSkiIlclm81GjZo1CA8PJzw8nNq18u4OFBISkm+wEhsTS25urr/LFhERkZJKYUvJ16RJEwYPHkTz5s3ZunUrM2bMpFOnTtzZswd2hwMA0zRJSUnhqZFPc+L4cT9XLCIicmF2u53ralz3e7DSIDyc+vXrExgYiNvt5siRI2cHK3tjyHW5/F22iIiIXEsUtlw72rRpzcBBA2nYoCEejwe73X7W46bb5OTJE4x8ehTJycl+qlJEROQP+QUr4eHhBAQE4HK5OHr06FnBSsyeGFxuBSsiIiLiZwpbrj0vjhtLmxtuOC9sATDdbo4cPcqoUc+QlqY7LYiISPFxOBxUv6762cFKgwYEOJ1kZ2ezf/9+YmJjiTsUR1x8HDF7Y3BpxoqIiIhciRS2XFsqVa7MJ9M/wm53XPA5brfJwYMHGD36ObKysoqxOhERuVbkF6w0aNgAp8NJVlYWBw4cICY2Nu8yoNhYDscfxuPx+LtsEREREe8obLm2PP74CLrcdhuOfGa1nMk0TWJjY3nu+THkZGcXU3UiIlISBQUFUa9+vd8Xrq1VqxYNGjTA6XSSmZnJwYMHFayIiIhIyaKw5dpRrWo1Pvr4Q2w2m1fPNz0eftm6lfHjX9E0bRER8UqpUqWoW6/u6dkqebdcrlGjBjabjYyMDA4dOnRWsBIfF49l6WuIiIiIlDAKW64df777bnr16U2FsDAMwwDANN1YFjgcdsA479+YHg+nkk+xKzq6mKuVMy34egG7d+/xdxlFqnHjRtx7z73+LkNEzvD6xIkXfTw4OJg6devkG6ykp6cTFxenYEVERESuTQpbrj1Op5PKlStTrVo1qlWrSrWq1bjuuuuoXuM6KleqhOP07aA9pgmG4fVMGCk6r0+cyNo1a/1dRpHq0LEDz40e7e8yROQMPXve+fufS5cuTe06tc8KVmrWrIlhGCQlJREbG0tcXByH4uLy/nwozo+Vi4iIiPiZwfwLr5QqJZLL5SIhIYGEhITzHjMMgwoVK1KtalWqVa9GtarVuO++vn6oUkRE/K1fv36E169PeHh9KlWuDMCJ48eJjd1HZORq9p2esZJ86pSfKxURERG58ihskd9ZlsXJEyc4eeIE27dvB/g9bImK2sTkD6b6s7xrSkREW0YMH+bvMvxi8gdTiYra5O8yRK5JI4YPIyKiLQA9e/YgNjaWpcuWExu7j5i9exSsiIiIiHhJYYuIiIicZ/DgIf4uQUREROSqpQU5RERERERERER8SGGLiIiIiIiIiIgPKWwREREREREREfEhhS0iIiIiIiIiIj6ksEVERERERERExIcUtoiIiIiIiIiI+JDCFhERERERERERH1LYIiIiIiIiIiLiQwpbRERERERERER8SGGLiIiIiIiIiIgPKWwREREREREREfEhhS0iIlJC2Knd63UWLv2K5yMc/i5GRERERK5hCltECiWQ1iP+w5afvuON2ytg+LscKcHU1wqiTM0mNKlZjiDjanintG9FRERESiqFLeJzpe+awu7dP7FoVATl8j17cHLLq2vYF72Y0c3txV2ezxiGgWHYsOkMyW/U1womqP3jzF+8jJ82bWZP9HZit2/i5zWL+Wb6G4x5sCuNQwv7Htlp+tBUvl8xk783KqL32VaWxv/3NyZ+NI/V/4tiT/RWdv7vBxZ99ibPD7yRGoFFs9mips8RERERkZJJ86ylaBilaPqXd5l89AEemb2PXH/X43M5bH7/Plq97+86RH3Ne/ZK4TQLr87vuYQ9mHKV61Cuch2ad+zJQ49uY+YLo3hteQLuArVso1ztJjSolkxAEYQGRtnmPPL2ezxzS1UcZ7QfEFaDpjfWoEnL0uz5bgOHc3y/7aKlzxERERGRkkozW6SIWJhWWTo8O4nnbgrV9PhrSFhYGK+88gpdu3aldOnSxbBF9bWCcRP9z740afIn6jVpTbObe3DvY68yfW0C7nItePC9DxlzY5kr5320VaPXxH8y+tYqGIk/M+uVx+jZuT2N/9Sa5h3v5v4n3uKzGd+yLsXyd6UXZQssQ6UqlSgfrN84RERERK4FClukiLj5ZdYUliTWZvCbL3NP9UtdWuDk5pd+ZF/0Ap5sfOZzDULvncqePT/zWZ+w0yeABhVu/ivvfjSH71esZtsvW4mN3syW5bN59y/tadSmN8++O5MV66LYs3MzW5bOYOLAZuddZmKUDufOJ99lwYr17Nq+mS3L5zL577dSw3nGtlvez7j3PuHbpZFs3/YLsds3sGHRy9wRZqfBY/OJ2bWWNzo6z264VC26PvYaXyyJZMf2n9kZtZKls17intpX72UsBWEYBq1bt+LJJ5/g88/nMnbcC9x8800EOJ2X/seFor5W0L7mceWQa1pYZg7pJw+xdeXnTHjkPh78dDfZzjoMGj2Y394ao0Innp+xgDUbNrE3eht7Nq/kuw+fpVej0ucHMvaGjPhmGwf27OTAnp3sWzOWm5yFaOcMwTc9ytOdwuDkKp7v/yDjZq8m+kgaOa4c0o7vI2rJZ4x/73uOeS76kr3YB97WmF+f2MqOdQuZ/XJ/WpU/+9XYwm7gb5O+4qfN/yNq9Y9s3vITvyx9i97VbUB++7Zg7QMQVIPOQ19h9qJVbNu2jZhtUfy0YgHzpr3C0IhyV05wJiIiInIN0U9sUmTMhCU8N7IMdT99iPHv/IU9D31MdLYvWrYR1rwbd93a5IwO7KR8zVbc++wn3HvOswNq38D9L0wjLKM3j379Kx6A4GYM/+RjnmhV5vfEMahmC+76x2RaVh/Bn1+IJNmyUfnGPgzuceZ2ylCpspOc9AuUFtiYoR9+wuh25f5IMgOqEN6yJiFZlzgbLIHsdjtt27SlXUQ73G43UVFRLF++ki1bNuN2F+xClYtRX/utgMvoa1YKG6a8wfzu0xnS4A56Nv6QXTtNcJejfuuG1Ag4/byQKlzfaQhv/akyrj8/zbcnvZxRUqh2grjxrq5UtuXy8ydv8WVcIfuMV/vA2xrz6xNQumJ9bu43hpYNS9Fr8KfsdQO2qvSdOIVnbi2L4c4g8dd0jJAKlKvsJCfFA+QXihWgfYCgxgz9aDqjI8qfse5LaSrUaEiFGvUI2PIpn0adwizcOyciIiIihaSZLVKELNI3f8AT723G03IY7z3ZljK+/InVSmXJc7fTonlzwpt1oOeY7zhsWnhObeJmkQ01AAAL0UlEQVSD4X24sU1LGrbuxqCpm0mlHLf26kJlG4CdhkPGMrxlMMcjJ/OXHjfTuGkb2vV5gf/uN6lxz3AGhp9xEmSlsezFO7mhVUvCm99Ix77vs9GVX0F26g4cy1MRoWTv/ZoXBt9B6+Ytub5tF/5vyBv84O1JaQljd9gxDAOn00n79u0YN24sn38+l6dHjqRly5YYPrlrjPqaT/pa1i/8uDEFj+06rm9Q+nRJ63hryD3c2LYN4dc35/r2d/KX6dvIrtCZ3p3Knz1rwtzL5D83p26jptRt1JT6HV9hvYuCt/P7y6xJk4Yh2MwDrF6bUMjAwPt9UKAaz+gT9Zu2o8PAiSw75qF0i4EMbJ03S8Uo047b25XBs+ND7r3xRm64pQtt2kTQ/t63WZt5ibK9aB/s1B80jpER5cjdv4gXB/8fLZs1J7xZOzqMiyTz2vzIEREREbkiKGyRIpbL3lnPM35lGuGDX2XMhU6qCsMySTtxnNQcEzM3megFU5mz08RwnCR63S6OpbtwZRxh3UefsCzFwl6zDrVsgL0Rd93ZGEfqciaM+ohV+06R487m+PYFvDx5Fen2BtwUUfGPwWG5SU44TGKmCzMnlSOHfiUjv5MYez3uvOtPBLq2MXnEOOZExZGc4yI79Vf2/ryXE9fexJbz2O0ODAOCg4O55ZaOTJjwKrNnzaRrt64+aF197fL7mpukpBQsw0ZwSHBeXYZBuWb9eO3TL1m3cRPbVs3kpe7VseOgSrWK3h9ECtOOUZoypQ3wJJN0qpAvqiD7oCA1ntEnPO50En6ay2uzduC2V6Bx40p5z7UsLMCo1JiIxhUJMgArhxMHDnPqUkGIN+3b69GjZ1MCzN3868kXmBkVT0quiZmbzonEdJS1iIiIiPiPLiOSomce4asXX+amJu/R5+XniNw+lowi2c5RDh1xw/WVqFLOBpmnT85yj3H4uIVROZhSBuCsSb0aNmylujMlqjtTzm+I6jWqYeNkwbbvqE2DOnY88VGsj/PdpP3nRo+G0T5r7ophd+R9/JQrX562bW74/e/r1K5FVNSmwjWqvnaZHISFhWJYHjIzMrGMstzywmdM718b5+/JVSC1aubVbrN5eQgpbDtWJhlZgC2UcqE2OF6I1xrg5T4wsuhwWa/V5Mi+g2RYTQkJyVvjxZO2ngUrTtK55608P3M5I5MPsWPrZiIXzubT72PyD9IK0P4f/WA9P8bmOwVKRERERPxEYYsUC+vkSl4Z9yVtPuzNS2PW8lZWPs/BAwQSFFTY+QgeXLluMJw4nWe24cLltsAwzvq1+cIMAksFFnxWhGHLWzPB8u3vyQu+/prdu3f7tM2iVKZMCMP/Ptyr55puE7vDTkpqKqFlywJw8FDcZW1ffe0ylGpBp3ah2DyH2B2TAWF/5sF7a2FP3sgHY99kzoZ9nMhyUPG2MXw96W7vyw3rWrh2zARiDmRhNapL+7aVmBpzjALPb/FyH9gKW+OZm8rNJdcyMH5bPMU6yeLnB5P+c1/uaN+C1q2a0bpzXdp06kxjWy+GL04u2Es5t32bE4cNcLu1JouIiIjIFUZhixQTi1Nr32XM5xF81n8UT/wajEHqGY97SEtJx7JdR6PwUIytiUU3Bd6VwMEED57Qb3jk9rGsuuDaCQW8e9Dpdm21Irixpp3tB31z+rN7927Wrlnrk7aKQ4UKFeDvF37c7XZhtzvIzMpizeo1rFi5ggoVKjD62Wd9VIH6WqEYobQf/gx9r7Ph3vsDi3eZ2MKrUjUAMpfNYsry3eTmbZzEE6nknPWPLdxuNxbBBAefHx3ZKnrbzrky+d+KDaR2v432Q0dw+7IX+N6ra6Ts2H87unm5D+yNhhWyxkvIjidy1rtEzgLsZWjcezyfvtSNTt0jCF78w+W0DK5jHDnpwVbrBiKq29gRr2sVRURERK4UWrNFio+VxvpJ45kbH0r16kHn/Jpvsn/7LlKtQDo8+hwDWlUh2G7DHlSGSuVL+fbWpeYefli2H0/Fu3jpzYe5rUlVygbYsdmDKF+jKZ3a1i5cCmnu4ful+zCdLXh8yngGtatD+SA7dmcIVRu1pFGFa3e4mW43lmWRnZ3N2jXrGD/+Vfr368+UKVOI3hmN5esZGuprF/3nNocTuwHYAyhdsTYtOvdjzPR5/Pvh6wlyHWTuxJnsMsGTeJzjLijVrhcD21QnxGGAzUlISNA5dVscP3YCy16Nbn27UTfEgT0ojPo3NKW6vSDtnMsi+ftpTN+Zg6363Uz6Yiqj7m1L/YrBOGx2AspUpmG7O3n0yXu5/nReletyYxnlaN2pPTWC7V7vg8LXeBH2unTp3YXm15UlwGZgdzpwp6WRAxgGl9/X3LtYtvIonsBWPP7OKO5qWoWQgEDK125Lr27XE3DpFkRERESkiGhmixQrKy2KdycsoMu/+lDjnMcy1sxi1q6u/KPpHbz6xR28etajuT6sws2OTybwaedpDO32FNO7PXXWo66f36TbgBkcKvCPxG52fvIqH94ylWF/uodXZt7DK789ZGXw7YhbGLHUJ/cjvipYloVlWZimycaNG1m1ahWbf9qCy108a0uor12orzloMvxL9px3pZeFeWo7M14YyYT1qXmzfRJXMm/FcDr27MK4uV0Yd9bzTfae8ee4yJVEj2hG815vs7LXby9wK6/1GMzH8d62kw/XbqaNeJbKUycw8PqODJvYkWHnvR07sX2zkF37TQ7v2s0pqzGNh0zjh8pP0/bx773bB16/Vu8ZFdrx8Mtjucl5zgOeJJYs3eSD9YSy2fTR2yy87W3uaTGEyV8NOedx391eXUREREQK5tr9qV38xCJlzfu89t3x89deyNnB5L/+jQlfbuJAUjamx8SdncbJhBi2rF5CZGyWzy73sNI2MXHQAJ6YuogNMcdJzTYxXRmcPLSNyJ/iC326baVv5p0HBjFi6nf8dDCRjFwTV2YS8dGb2Zfq9O2siSuYaZps2fIz7737Hv369ef11yeyYcPGYgta8qivncs8Ecv2fUdJTMvGZVp4XFmknoxnx/olfPbWU9zdfSAvL0v44xTdSmLJC0MZ+ckqdhxJJcc0cedkkHz8MHt/2ciG2JTf3yczZgYjRv2bVTEnyTRN3JmJ7P85lhOGUaB28mMeWc64fr144NVZfL/lAMdTszFNk6zUX9n3yxr++/HnrEvO28uZkZN46p/L2XEsjYTDR8n1dh9cZo35MYyjbPlxG4eSsnB7PJhZycRtW85Hzz7MqEUnfNLHPCeW8czAx3jzq03sT8zG7c4mcX8U3yyPJtMCj6VLi0RERET8wQitUEl3h5QLWrx4EQBRUZuY/MFUP1dz7YiIaMuI4Xm/378+ceJVtWaL0+GkVHAQqalpXv+bDh075N11CZj8wdTC341IRACDSvd9xJrxN7Bx7G08OD/J62BnxPBhRES0BaBnzzuLrkQRERGRksxgvi4jEhGfcrlduFJ1G1qR4mCr3IY7WkDMzgMcOZlCtqM89drczdOPtSPAs4+ftxd8Ro6IiIiIXD6FLSIiIlepwJb9mDi5ByHnXjdmmRxZNI25e3VTaBERERF/UNgiIiJyVTIIOLWHVVH1aNmgFlVDAyEnhaP7t7Nm4Qw+mLOR41qyRURERMQvFLaIiIhclSxSoqYzYsh0fxciIiIiIufQ3YhERERERERERHxIYYuIiIiIiIiIiA8pbBERERERERER8SGFLSIiIiIiIiIiPqSwRURERERERETEhxS2iIiIiIiIiIj4kMIWEREREREREREfUtgiIiIiIiIiIuJDCltERERERERERHxIYYuIiIiIiIiIiA8pbBERERERERER8SGFLSIiIiIiIiIiPuTwdwFydQgPD2fE8GH+LuOaUb58eX+X4Dd3dL+d9hFt/V2GyDUpPDzc3yWIiIiIlAgKW8QrYWHlidAJsBSDBg10siciIiIiIlc3XUYkIiIiIiIiIuJDRmiFSpa/ixARERERERERKREM5mtmi4iIiIiIiIiIDylsERERERERERHxIYUtIiIiIiIiIiI+9P/AlDWpkvbL0QAAAABJRU5ErkJggg== ) ## Add a blueprint to a project and train It ``` project_id = '5eb9656901f6bb026828f14e' ``` ``` enetcd_blueprint.save() ``` ``` Name: 'Ridge Regressor' Input Data: Date | Categorical | Numeric Tasks: Standardize | One-Hot Encoding | Numeric Data Cleansing | Elastic-Net Regressor (L1 / Least-Squares Loss) ``` ``` enetcd_blueprint.train(project_id=project_id) ``` ``` Training requested! Blueprint Id: fa329535f1e5f5465e2c55024aacb910 ``` ``` Name: 'Ridge Regressor' Input Data: Date | Categorical | Numeric Tasks: Standardize | One-Hot Encoding | Numeric Data Cleansing | Elastic-Net Regressor (L1 / Least-Squares Loss) ``` ## Custom Models ### Find tasks ``` w.search_tasks('awesome model') ``` ``` Awesome Model: [CUSTOMR_6019ae978cc598a46199cee1] - This is the best model ever. ``` ``` w.CustomTasks.CUSTOMR_6019ae978cc598a46199cee1 ``` ``` Awesome Model: [CUSTOMR_6019ae978cc598a46199cee1] - This is the best model ever. ``` ``` w.CustomTasks.CUSTOMR_6019ae978cc598a46199cee1(w.TaskInputs.NUM) ``` ``` Awesome Model (CUSTOMR_6019ae978cc598a46199cee1) Input Summary: Numeric Data Output Method: TaskOutputMethod.PREDICT Task Parameters: version_id (version_id) = latest_6019ae978cc598a46199cee1 ``` ``` w.CustomTask('CUSTOMR_6019ae978cc598a46199cee1') ``` ``` Awesome Model (CUSTOMR_6019ae978cc598a46199cee1) Input Summary: (None) Output Method: TaskOutputMethod.PREDICT Task Parameters: version_id (version_id) = latest_6019ae978cc598a46199cee1 ``` ``` w.CustomTasks.CUSTOMR_6019ae978cc598a46199cee1.versions ``` ``` Latest (latest_6019ae978cc598a46199cee1): str v3.0 (6019e2418311cc8207a5f8e1): str v2.10 (6019dff0509159ede309f9c9): str v2.9 (6019dc3b8311cc8207a5f7d9): str v2.8 (6019dbcb4f6322a6283883d9): str v2.7 (6019db4d041c71bd7ea1c670): str v2.6 (6019da5d4f6322a628388364): str v2.5 (6019d924be257008648e3c62): str v2.4 (6019d7db3d7d080b078e3c39): str v2.3 (6019d744356f3c430b38828d): str v2.2 (6019d305be257008648e3c0c): str v2.1 (6019d2e045e619fc03a2eead): str v2.0 (6019d2bd3d7d080b078e3b66): str v1.3 (6019cf0735270cbe238e3c76): str v1.2 (6019b9fdbf5b0a42aba1c6e9): str v1.1 (6019b81729ae9ab5ad8e3c26): str v1.0 (6019afe4dcd97e1e5ebfee13): str ``` ### Build a blueprint ``` pni = w.Tasks.PNI2(w.TaskInputs.NUM) rdt = w.Tasks.RDT5(pni) binning = w.Tasks.BINNING(pni) customr = w.CustomTasks.CUSTOMR_6019ae978cc598a46199cee1(rdt, binning) custom_bp = w.BlueprintGraph(customr, name='My Fun Custom Blueprint').save() ``` ### Update task versions ``` customr.version = w.CustomTasks.CUSTOMR_6019ae978cc598a46199cee1.versions.v2_7 ``` ``` customr ``` ``` Awesome Model (CUSTOMR_6019ae978cc598a46199cee1) Input Summary: Smooth Ridit Transform (RDT5) | Binning of numerical variables (BINNING) Output Method: TaskOutputMethod.PREDICT Task Parameters: version_id (version_id) = 6019db4d041c71bd7ea1c670 ``` ``` customr.version = w.CustomTasks.CUSTOMR_6019ae978cc598a46199cee1.versions.Latest ``` ``` custom_bp.save() ``` ``` Name: 'My Fun Custom Blueprint' Input Data: Numeric Tasks: Missing Values Imputed (quick median) | Smooth Ridit Transform | Binning of numerical variables | Awesome Model ``` ### Find, View, and Train ``` bps = w.list(limit=3) ``` ``` list(bps)[0].show() ``` ![No description has been provided for this image](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABoMAAADECAYAAABUdVjxAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdd3RU1d7G8e+ZSe+V0IuU0CGEXqQoHZSiKFJ8FcEC9o7lotd7xYaKisoFC02lSpMOhl4DJEDoPdSE9J6Z8/4RQEqAhBYZn89aWaxkTvntcw7JzHnO3tvwDQw2EREREREREREREREREcdjMMVS1DWIiIiIiIiIiIiIiIjIraMwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNERERERETuYEaxDnwwZQEbNv/CoLKO+RHPLfwF5kQd4NiOubzV1KuoyxERERERueM45icFEREREflHc270PltPniIhdh7PV3Eq0DquYW+x9vhJEo8v4MVK1ltTmEsrvow5QWLcqat/HZvBoFJ/37fqriUb0fvVT5g0fzUxuw9w6sRRju+NYuOi3/j+vUF0ruHPLTqCl7FWeIARUxawcfM3dPO4TTu9FZxC6T/yNxZGrGX7zt3EHo0l/tQxTh7Zw+5Ny1n02yj++0xnavhffmQtnnfRqHkYlUO8sBpFUPvtYFiwWAwMw4rVuPFGOsx1IyIiIiJSQAX7ZCwiIiIicgexFitOkAUM1zCefv5efhoynyTzKisYxej+0qNUcTbAFkRIkAX22m5bvXcMw4/wgZ/y3dtdqexxyQ15v+JUCitOpbDW9OrXiAFhA5iedOtLsgTXovXdYVTIPXbbAqhbwlKC8LYtaRh8cQhodfelWDlfipWrSoN7ejLo2TV8OuAxPlqdwNUuaUeTuXEEnWqOuGnbc5jrRkRERESkgP6+jxuKiIiIiFwXC4HFi+Fs2DixZg2JnYfQt8LVb/c613yMZ5vtZeXGTExLAMFBt/r2cDaLX6iGf1Ax/PL7Ktmd0bH2W1xDIRk+NH7jV37/z31UdreTuH0Gwwf3pEmtKpQoWZrSofVp8cBg3vtxCZGzZ7EsuagLvlNls/TVepQoURz/4OIElw2lVsuePDl8OtuTTZyCm/La6Pfp4OeoXYBERERERORWUBgkIiIiIg7GQkjxYlgwiV/1HT9uqs4TAxrgdsXlvblnUB9CloxlTHQCpuFMULAfutV+IQPfNu/w/fNheJPN/qlDaNP2SYb/toKY44lkZGeTGn+Y6D+n8Pmrvbnn+Zkk/JO6rdxk2RmpZObYMU07OekJHNm+gt8+fZrOgyZxxAaWkA50a+xa1GWKiIiIiMgdRGGQiIiIiDgYC0HBgViwk3RmB9Mn/onfAwPoFJB/vGMp1YNBXWzMnriQg2eSsGMhKDjg7NBRVqq+uIjTcaeIj/yAJs75bMClJZ9vP0Hi6UP81jvgpoZIzg3fY/PJUyTG/kjPy+Y1MfB7eAIn404Rt/otwi4cANq9Kt1f+Q/fTfid5eu3sv/gEU6fOkFc7B52LJ/KNy+05y73QhRircKAN3pT1gmyd4xi4IvT2J99C+sH3Mq35YUvp7Bqyy6OH4/lxMEdbI2YyaQRj1Inv2TPtTNjD18899LJKf0pduEJcQ6mYb9/8eOc1ezaf4STh3cRvewXvni2PRXzmzfm3HGcOJMV67ey/9AR4k6dIO7ITjbPG82bnSvi7laaFv3f4X+/Lydm/1HiThzm4JYl/Db8URoG3syPWyaJG9ex3QYYbvj6FiwMupFzAIBraVoO+pBfF21k/+GjnDwYw+aF4xj+WCNCLl3erQqdn/0XX/00nYj1URw4fJT4k0eJ3b2ZNTO/49+PNaeUS35FlqJpn5f46H+TWbJ6M/sPHeH0iaMcidnAit+eJdwZjOKPMvPYKRKPL+HVahf03LvRa70g142IiIiIiAPQnEEiIiIi4mDc8A/wwIJJcmIScQsmMff9MQzsVY7fvzvIxYOvOVOj72M0PTmD/65KI6laMnYM/AP9z4Y6NvatXkOsrQ7lSzSkSXkra/ZcPJeQU+WGeTf9c2NYtS7pbzGPi+HbiP97aSAtL73x7upLyep306d6czq1fp+uvUaxLeva23Oq0YNetVwwzFSWfjuayMxbUvZ5zqFP8MusD2h9YZjiHES5GkGU9t3ON9cxnZPh14CXf/qZN5sFYT1/o9+VMrXu4f9qteGhh3/hmUdeZcahnL/WudJxdA+gQoNuvPZjax497UxIiMdFIaBf6Vq0f+JjWjarQM9O77Eq5eZcFd71GlDDCtiOsWdf+k3Z5tUYAc0YOuEHXm7of8FThIFUqNeBp8LuoXPzF+n65GQO5p5d3q8Jg94cfNnx8gwoRbVmPajWrBuP9h3DoD7/Yv6Jv06iEXgvr3/yxmXrOQeXo2qAQcpVRky82de6iIiIiIijUs8gEREREXEsFj8C/Qwwc0hJzcRMW86kmXGEP9qXsEt79ng0Y0CfSuyZOpnNOSZpKamYWPAN8Ds/qXzO1mVExNnBqRp3Nw2+pOePQVD9RlS0gi12LWuPXEdKcSvl7mNs/yZUrVSe4JBSlKzWgoc+WERsroF/09f4b98yBfhAYCGkYUPucgIzJ5IFy+JvceDlRbsXX6VloEFa1E881SGcciVLUqxcDcLb9+PF96cSnZvPallzGVD24rmXQh4cxykTsITQ87MfGdo8CCN1G+Ne7ka9ymUJKVebux/7lKXHbbiF9mbUD89TJ7+eK+eOY8VyBBcvTfkGD/LvpacwLb4UL2YnZup79Lm3HuVKlqBYhXp0fHMOh3PBrerjDO1dkGN8ZRYXT4LK1aXDE58w/du+lLbaOb1kJGM351x75RthKcGDn43m5YZ+5Byazwf9WlG1bGmKV2lC93fncDDHiTJdP2B4rxKXty93L6N716NCmVIEFCtN6ZqtePjdqcSkGvjUGciYMU9SNb9edraDTBjUilqh5QkOKU2ZGs3p9MJk9hXkv9X1XutXu25ERERERByIwiARERERcSwWfwIDLGCmk5oGkM2GydPYX+5B+re4cJwsg4D2/bg/cAu/Tt2NDZO01DRM08DJPxDfc6lP5noWRCRiN1xo0O5uAi9Kg9yp17gWroadpHVriC7w/XkX7v0ihoS4i4enSow7ReLJSIbnOx7ddTDTOXHgICcS08mx5ZB+ehcLvhzC0NkJ2A13GnVuU4DhsKyUKVcGK2CP28veM7f4Lrm1FNWqeGEhh80Tv2DyxiMkZeeSnXaafZsWMG56JMmFLMEl7Ele71wMi+04055/iOd/Xs3+hEyy0k4QNftjHun7OVuzwL32k7x2f9DlQ/2dO45JGeTkZpN4IILPX/yMlZkmmNlsmTqGuVuOkpRtIzvlKGvGvMIHC1IwDVfq3d0Q78JVS4evd5+/Ns4cO8DeTQv5dfijNAhIZ/v4IXR9YiIHb3Hu6BL+JK91DMbI2MiHfZ7g03k7OJGeTeaZfSwb9QyDRu8h1+JL695dKX3pp0ozg9NHj5OQkYPdnk3qiR3MHzWYzgPGsT8XvBo+x6sd/S8/zvYUDsXs4kh8Ojm2bFJO7mbD9hMUqKk35VoXEREREXFcCoNERERExLFYfAnwtYCZRmpaXmqQs20ak3cEcn/f9pyfOshSih797sVtzRRmHM4bh8qWmkq6CRY/f/zOv1NOY/nspSTYDTyatKel7wV3lJ2r07S+D4aZxuql67nFo6fdHGYiy5dsIts0cKpYlUrXHDjawN3TDQMw09NIv9U9JuwJnI6zYeJMWK//o0mg9drrXJUTtbt0pIIT5O7+ha/+OH1Zz6bMqLF8uyQV0/ChVddW+BYgNLCfXM+a/TaweHFXpWIXf7AyE9mwbjc5GDiVKkvJG23COYYn1Xu8yrAhrSlxSwf8diasaycqOJlkrhzP+F2XThCVyebFKzllN3CpUY86BZq+yOTMss/5ZnUWpsWftl2b43kLKr94l4W91kVEREREHJfCIBERERFxLIY3vt4GmOmkZZydbMS2l+lTI3Fr25cepfLeAlurPEi/xjYips7l+NnFzoUdFl9/fC54p5y6fAbz4+wYXs3p2trvfI8Ga9nGNC5txcxcz4LliYUYPi2bxS9Uwz/o4uGp/IKK4RdSjzfW3MohwEzSThwnyQSLn18Bgg+TzIwsTMBw98D9VveuME8ze8w0DueAZ/jzzNqwkukjXuChxqXxuJ59Gz5UrVYGJ+wkb9nErvyGmDOTiNy4h1wMXEOrU6kg4Y0tnpNxdsCCj6/PJR+s7JyJi8cOWDy98SrUp65s5g+p8te1UawkxSvVpXGXJ3j3lyiS3MvT4dVxzHin8a0LUwxvKlcpiRUD97Yj2Xf68h5sp39/nJIWMNyCCfErYAPtp1i37gA2DDxCq1Hhloczhb3WRUREREQcl8IgEREREXEohpcv3k4GmFmkZ5z7qZ3DM6exhkb07VUZKy6E936I6ulLmbzgrzlwzMwMMkwwfHzwufDGcdpyfpkVi83iR9veXQixAFgo3qoNtZxMsiMXsyz+VnaZMS4fUusGmFlZZJtgODlz7fvxNo4fPY4NsARVoILPdSUyhajf5MzC1+na/1Nm70jE7l2RNv2H8v3s9exY/gOvtC5RgJov3LUnXl55201JSsae70J2kpNS88Ibr4KGN9lkn+0wY7Fcnh5l55xNnaxWbqhjkD2XzMRj7Fw7i5HPdqffmP3kGq6E9h9IB9/CbKgQ5+D8MSsIN1wL1DMIwE5SYhJ2wPDyxvM2hDOFu9ZFRERERByXwiARERERcSiGty8+AGYGmVl/BTT2E3OZsiKbmg89TLhfM/r0KEvSomksSvhrGTMzg0wTDE8/fC66g5/F2nGT2J4NHs0foVcFKxiBtG4bjgs5bJ6/mGP5pww3xMzJJdcEDFfcbnmXnCuxE7t5CydsYDiH0bq5T4FDheuvP5vDiz6h3921qdV+IO+NX8WRLCt+1brw1oTJvNv4wrmfrhHCmWmkpgIYeF/Wg+ccCz6+XlgAMy2V1JtyLm9FOJjKutlLOW4Dw70SoWWuHW9c1zkw00lLA7ATN+FhiuXXg+3cV8nOfHOwoAfMwNvH6/yQgxm3esjBqyrSnYuIiIiI3HYKg0RERETEoVi8fM4O8ZZJeuYFN3zNeOZPiyCtfE8Gv/cE9xWLZ960CFIuXDkjnUzAsHjj533xjfPcmImMjkgFl7oMGNgEr+Id6dHUDbI3MmPOkSv0OLkx9sR4EkzAUopyN23imcLLiZzD3KM2sPjR8cm+VHEu2Ho3Xn8WJyJn8vmL3WnQfBAT9+WAaxX6PXo37meXMLOyyTobdri65hN2mCns2nkUGxZ86tYjNL/8xPAhrH5lnDDJ3LWdfbbrKPU2MZyccTIA00au7dqBxnWdAzOZvXtPYsOCX736VLlZXWoMX2rXqYATJln79nAwvyH7bpNrXjciIiIiIg5GYZCIiIiIOBTDyytvbhkzi8wLwyBMziyexuKkYtzfpy0+J+YxdWX6Reue6xmE4YX3pWOFmSeY8b8ZHLNZKfvwC7zy9EM0dYes9TOZE3sroiCwx+5gR4IdrBVp265i0Q1zlb2e775aQbJp4N7gFb57qwWBBfgkcTPrzzw4l9Ez9+fNN1Ms5Pz8L/a4U8TbAUsFQivkF3bksHXufA7kglOV3gxuH3RZzya3mgN4qo0XhplMxOwIEv+unUaspen+WEdCLGBP3kHUobzUyjTzvjBccHO+uHXXdw5y2LxwKSds4BT6CM91DL4pwxS6VuvPgFYeGGY6qxetJPkmbPN6Xfu6ERERERFxLAqDRERERMShGB6eeBhg5maQkXPJi0l/Mm1xAnZsHPtjJmszL37ZzMokA8DwxDufCU3SIr7lmw0Z4NWC55+ujyvprJw+75YMEQdA9lqmzjyKzXCmznPf8lnfhpT1dcFiOOHmV5zyJQo+ZNuNsXNowqu8Pvc4NjwJG/wLEVM/YMC9tSjj54rVsODsEUDZGi14cMiHTPz+Capar7d+L5oNGsqTncO5K8gTZ8PA6upHuYa9eKprBazYSTh06HxgYz8VyaYjueBUgT5vPkOzUp44WVzwLlWbDl0bUtwC2Zu+46M/TmG3luTBr35hRL/GlPdzwcUjhJqdX2H8hBcJc4PMqNF8PPN0kQ8g5uLuhbuLFQMDq4snAaVCadx1IB9Pm8NXXUOwmFls/2ksf57NMs3keBLsJjhVo/vjbQkNcPnruF7nNZS5chSfr0zKO2ZfT+d/QzpRr6w/bk4GFhdvildpxH39OlA1v3TJKZT+/36H/i1DCfF0wcWrODU7vcTPE16hvhvk7J/IlzNOFulxLsh1IyIiIiLiSDSHpoiIiIg4FMPdE08DzMxMsi57NZU/nqlGwDP5r2tmXNAzKL/Z7W37+OnDiQyc/gQVrGBPWMzEObfypnYmqz4byvjWY3m0Uk0e/WIOj35x+VK3ZbSt3EP88mRPcj/8nk/61qT03YP47O5B+S+bE8mGT35k597rqN+5Gp0GPcvg8i/w0WVLmtjjI/js21Wcz/FytjD2mwj6fnoPQa3fZu7Wt/9aOm0eT67YwOTEE0x96XHKB/7Mm03r8Njns3js84u3m7H7VwY//gVbLr9objMX2nwSybFP8n/VtCex9eeXefTTzeePgZm4gtkrkmjb1o/ag8axuv13dG78Lmtz4LqvIdsBfnj6KSpM+o6n64bywLCfeGDYJctkb+Ct5QvZeeiSNNRwoUyrZxjZ6tL/aCa20xEMG/ghK9OufSRuqQJdN0UdC4qIiIiI3DwKg0RERETEobh6emA5GwYVeoJ6WyYZ2SY4e+LtlX+fm/Q13zF6Q3/+29iJk3N+YUHCrb1hbMYt5KXO9xP5wvM82rER1Ur74W6xkZWaxKljh9gTs40tKxZy5HbMc5O1lykvtefPcV3p06sTbVuEU6VUMP5eVnLTEjl5eA/bN69j2bzpTD1gu776zdOs/O1XKrdrSp1KJQn0dIacVOKO7CEyYiZjR/3IkkMXdvmyc2j8ILpmvszbT91H09ASeBmZnDm2m8jlqzh8dgQwM3E9nzzQipV9BvPMQ+1oXK00vtY0Tu/fyrLff2Lk/+axO7UIb/7bTxC5ZBW1wspSIigAfx9PXJ0M7DmZpCaeJvbALqI2LGfutKnMi46/OLyxH2PSkD54DH2F/vfUpcye3ey/YIHrvYbsp5bwVqdWzO/7JE/0aEOj6uUJ9rKQk57EyUO72Lr2D7ZcPNJintz9TB0xg5xGHWlZtyLB7rkkHd/F+gW/8fWXE1h98tIue0WhYNeNiIiIiIijMHwDg/W4k4iIiIhIATlXfpqZi4fRxNjMvzt2ZcT2v8ONbZGiZxR/lN8jP6GlEc1/2rTjk5jbkVCKiIiIiMg1GUzRSMgiIiIiIldk4F8ulNK+Lji5BVD57kF8P/Etmniksmb483ytIEhERERERETuABomTkRERETkSowAOn+0mK/udeX8oHFmNoemv8KT3+8iuyhrExERERERESkghUEiIiIiIldiCcQ18yCnMyoSYEnl5J6NzBv/JR//vI5TGgFLRERERERE7hCaM0hERERERERERERERMRRac4gERERERERERERERERx6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIEpDBIREREREREREREREXFgCoNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIE5FXUBIiIiIiL3d7uf6lWrFXUZIiIA7NgZw8zfZxZ1GSIiIiIiN43CIBEREREpctWrVqN5i+ZFXYaIyHkzURgkIiIiIo5Dw8SJiIiIiIiIiIiIiIg4MPUMEhEREZG/lb79HyvqEkTkH2rCuB+LugQRERERkVtCPYNEREREREREREREREQcmMIgERERERERERERERERB6YwSERERERERERERERExIFpziARERERERERB9CtTVZRlyAiIiJF7PelrkVdgvxNKQwSERERERERcQA/vZ9c1CWIiIhIEfNbGlzUJcjflIaJExERERERERERERERcWDqGSQiIiIiIiLiQCJ3ujN2VmBRlyEiIiK3yYD74qlXNaOoy5C/OYVBIiIiIiIiIg7kRLwzf6zyKeoyRERE5Dbp3CwZUBgkV6dh4kRERERERERERERERByYwiAREREREREREREREREHpjBIRERERERERERERETEgSkMEhERERERERERERERcWAKg0REREREJF/WkKY888l4lq3ZxJ4dm4leMYPhHYMwirowh2GlXI8PmbVwOkMbOhV1MTfAQsn7PmDWojm819z5Gss6SptFRERERO4sCoNERERE5B/OlXrP/Ubkxj/4qF3gLQ46bue+bpBLDZ7//mteua8e5QPccLK64BVcHNesVMyiru22ufXny7tMdaqX8cPNKOqr4UbaauBZKpRqpf1wK8CKf582i4iIiIj8c+hRLBERERFxOG6NnuWntzpTLiQAf293nMklMzWRk4d3E7liLj+Pm0t0gu388oZhYBgWLLfh3vTt3NeNcG3Ui96hLmTvmcwrL45k0f5UXIJL4pWaVdSl3VZ3yvm6Gf5JbRURERER+adRGCQiIiIiDsdarAp1Q8vgev4nLnj4FqNCrWJUqNWM+7u14KVHXmf2cTuQxaYvexH25e2o7Hbu60ZYCKlUEV8jixU/fMncPYmYQNaJQ6QUdWm31Z1yvm6Gf1JbRURERET+eTRMnIiIiIg4qFy2ftGNatVqclf1cOo068B9T33I1J3pWEu2Z0ivUKxFXeLfloGrmwuGmUFcXNo/aFg4uVNYXL0JDgnG30PPN4qIiIiIFITCIBERERFxUCa2rEyy7SamLZPkuCNEL5vAv75fTaYJ3j5eZ+dFsVL56SnsiVnJRy2cz65rENhsECNGT2T+kuVEbd3C3h1b2LZqFhPe602Y/4XjaBVm2Rvd11lupWk98N9MmLOMqKgo9kStZ+OSGUz+9t8MbOh39fle3MvTbvBHTFmwgu3RkUQvn8FPw/rQMPjSaMwAiz+9/reFA7u2531t+4n+JS79CFGY+p1pNuxP9u2YwYtVrRdtw7f7KHbt2sxPDwScrT+/7W4icvEERjzemNDwnrw+YhxLVq1n1/ZNRC78meF9auF3SeMNz0p0eXEEM5asJiZ6E5GLJzFycEtKO1+w77oP8e7nY5m9MILoqK3sjV7L2jnv0TEgv/N17jiW5d6n/8uv8yLYFr2Z7euXsnD8MLqVy2uXEdiKoT/PYMXaDezeEcWuTUv54/vX6RHqWYj5eJyo/+YC9u5cyzcdvS9pWCAPjd3E/ugf6V/CUsD9Fb6thWqH4UKVbm/z48ylREdvYcfa+UwdMZgO5d0L1NprnyuwBNTnyS+ms3HTGtYv/5NNkRvZuvATepbUR1sRERERkavRY1QiIiIi4vgsTrh5+lG6eisee6IJrvY4VqzYie3KKxBQuy1dW1a/6A2zZ1BFmj38FnWruNOj3w/szi3ssje6L8CtKgNHj+GNhv4XzO3iSWDpKgSWvguXyB/4YX1i/m1zq86To8fwWkPfv54KC6lCy95v0vTuOrzc901mH7vyUbkp9d/Qdp3xLxNG99fH0v2SpV3K1eeht78lIK0nT/1+EjuARy2GjP0fL4R5n2+vW5k6dH12JHVLPsf9b0eQYFoo1uQB+nW6cD/eBBdzJiv1CqW5VmXg92N5o5HfX8fRJYRKdcvglWHP+z7Xj4r1qlDa5ezrXiFUa9WfT2oWI+f+V5gdV5D+VrlEL19DXP+eNGhaC9d5qzk/Y5NnOM1ru2KLWcmKU3bwKsj+rqOthWmH4UndLg/89b1LGcI7P0NY07q81/cZxu3NuXJTC3KujOI8OPwrXmvpg5GbRvzJVAyvQPyKOZOVZC/A8RQRkTuJ4d+Kt0e9QrtDX3DvG4sp+KyFVsr1+ICvngpl7du9+O/6Qr8JERFxSHp8SkREREQclDP1Xp/Pvl3bORCzlZiNESwa9x697zrFzHee5L0/U649/JmZzLw321Gndm0q1mhE8z7DWXTCjmedPvSp53z9y173vqxU7PsuLzf0I3v/HP7VrwN1a9WmUq1GNH83gvSrNshKpX7v8GIDHzJjpvJarzbUqBlGWLuBfLj0OEbJjgx7rS0XdeSxJzD5ibpUCK2R91Xz/xh3/Ao33W+0/QU4LpVqNafzW39w1GZiT9zA10MeoEl4XarUa0vfUZtIxo+WPdpQzJLX3ir932FIXQ9ORYzk8U7NqFojnEYPvM3U/TZKdxtCn0oX9E4yU1j0ry7UD6tLpdpNaPHgl6zLN7uwUqHPO7zU0JfM3b/zdr+O1Ktdl2oN2t7E/pAAACAASURBVNCh/0csOBuOmCmr+KR/N5o0CKdStdpUa9yFx8dEkRnYmp6t/AvcOyhr83JWJ0FAkxbUuiBtc6/XgsZedvatXMlhWyH3V+C2Fna72Rz44yMe79qS6jXrEdZuAO/PO0SuXxNefaULxa7Y6IKdK8O7Ee0aeWPf9j3dmzSh/t1tCA9vSOPun7IyvYAHVERE7hiGW0mq16pAsIdTIXrV5vEuU53qZfxwMwq7poiI41IYJCIiIiL/KIZ7Bdo/OZiHa3tf+8aCaSPl9CmSs2zYc1OJ3TiJ/47fRq41kKpVgy9+M12YZa93X9a76NS5Bi62nXz34tuMW3+EpGwbtuxUTsenXj3cslbmvvtr4JITzVcvv8+UrSdJz8km8dBqRr/yLyYfN/FvdR/35DcsXUHcaPsLsF1bdgI7Zoxi4nYbhlMcO1bFcCI1h5y0Y6waPZZFSSbWMuUpawGsoXTtUhWn5MX859XRLNuXSFZuJqeiZ/DeyGWkWivTtGHQX3WZuSTEHiU+PQdbVjLHDp0kLb8Dar2LLl1r4poTxcjn3mXi+sMkZOWQmXyS3Zt3c/pcVmYY+NV6mP/+MI1V6zYQtWwcw9qXxIoTISWCCn480tcxb3kiRskW3FP9XHjlSr02zfA39zF/4d68XmCF2V9B21ro7aaxYfovLNsdR0ZOFomH1vLjm8P4NdaOZ+O2tLh0DL/zx7SA58o0MQEjuCoNqwbhZgBmFqcPHCVRE1uJCIC1HE/8son9O9cypkdwoQMEuYC1EkOmb2V/1GSGhF7toQ5Pmrw9j707Ixnbzfsqy4mISFHTMHEiIiIi4qByiPyoKw/+cAQ7BhYXDwJLhtKs5xCGDriXoV+eYXen91mZUZht2ji27yBpZg28vK4190thli3g+k7lqFzeiv3Iav682pBb+XEpR6XSFuxH1rHq4CVDwaVFsmJzJr07lKNSGQucKXSxBav/pmz2OIeO5UK1YEL8LJB+Nn3JPsHRUyZGMQ/cDcC5DHeVtmBxb89X69vzVT71lSxdAgtxhdv/+XOwntWHrzCknuHD3W//xJje5XA+33BXypbJ26/FUpiPYWmsmrOM+K7duPfeqnwatR2bS23atgzC3Pkbc/fYbvL+bnI7MqJYF51Nv3alKF/SAgn5LONSsHNlpKxmxpI4WnduydBxi3k54RDbtmwiYtYEfpi/58qBloj8Y7g36Ef/Om4YhistH3uY6rO+YrtGCLs+liBCgi0YrtV44sUuTH1mBify6Rxsrdyb1x4sg9Ww4R8UgJWUqwzDKyIiRUk9g0RERETkH8DEnp3G6YOR/D7iDUauz8Ea0ojmVazXXvXSLWVnk20aGJZrxxuFWbZA61uccbIAubnXcaPl9j8fnV/7TeyAK25u11uPnZzsXDCccXa+cBs55OSaYBh5H3LO9iK5MgNXd9fCHxXDkjdXk3nlrRsB9/J/3ctiTVjH14N70iS8LpWq16fxszM4cR13yNLX/8H8E1ChfQdqO4FreAfahdjYOmce+203f383tx3nzr/lyse6oOfKjGPu0H48/sFYflsSyWGzJPVaP8BLI8bycaeggjdMRByTJYT7Hu9KqfRVTPz9MEalB3milU+R9g6yuHoTHBKMv8cd+Cy2axAhvgbZiclYWwxiYD23y5cx/Gj31KPUykok0W4QEFDwYVBFROT2UxgkIiIiIv8shjMuLnk3py132jjyOSc4FmfHUrY+DUsW8q189iH2HbVjKdOIZuUuCcE869EizA2yD7P/6BXmBLop7KQkpWJaQgit5HtrbxjlxHIw1o799HQeD6vx17xH579q0WTYOgrZv+r8di1lG9KkTP5hoiWoOMVdIH3leL5avJMTqTnYbBnEn06+wuTXVqxXu0+YuZEpsw5CmfbcX8+fpl3uoVjmWibPPoLtuvZXMDdju4ZvU+6p5wLZh9gfe+G1dUGbC3OuMo8QMX4Ebwx+lHYtWtLp3YUcNwNo1b7hDbRURByBtWI3+jZz59T8cXz07RS25gbQrk8nSpz/c2ml1ouz2btzA992uXA4MwvFH/mBXTEr+Ohu1wu3SK0XZ7E3ZgUft/wrCDE8K9HlxRHMWLKamOhNRC6exMjBLSl9wUhqloD6PPnFdDZuWsP65X+yKXIjWxd+Qs8L/3a7l6fd4I+YsmAF26MjiV4+g5+G9aFh8IV/WwwCmw1ixOiJzF+ynKitW9i7YxORiycw4vHGhIb35PUR41iyaj27tm8icuHPDO9Ti0tH5SxIzZeyBAQRaLFz6o9vGb+3BL2e7sqlbz2cKj/MM+1cWTNqDGuzLPgH+V18o7FAbTy3vzr0eftb/ohYy85tm4hcNJGvnmlOSD5vd66nPSIiojBIRERERByWgdXVHRcLYHHG3TuQcrVaM+C/X/FimDP2xM1s2HuHjR2TG8Oipcexu4bx/Gev0rVGCF4urviXa0CPttVwudq6tt3MmrWDbOdaPDviHR6oHYKHkwu+5Zoy6JP36FXCIDFiNovP3Mqxtmzsj44h2XSl+VNv8khYCB5WC1Y3b4L93W9uOGTbxYJF+7EHdWXYxwO4p3pxfFysWKxu+JeuQasG5a5vzGzbLuYv3IfNuQ7Pf/U+fRuVx9/NitXZi+KhdQkNtGCPP8WpHHBv1IM+4SXxcjLA4oyXl9tl+8zOycU0/KjXqjGlPa7UUy2XHdOns8VWgi6PvcGj7QJIWjqN+XF556ow+yuMQm/XsOIdFIinswWLkxclanfmzW+GcX8wnFk6m2VJZv5tLui5slagTc821C7lg4vFwOrsRG5KClnAnZbrisjN5kq9Xj2oZhxg2i9rSTk0i4krU3Fv+AA9K5373Wpj19r1nDZdqRNWlb9yA3fC6lfH2eJL3bC7OP+b2BJE3bplsGREsmrz2QjcoxZDxk7gy6faU7e0L24ubviXqUPXZ0cyaVhL/A3AUpwHh3/Fax1D8TPSiT95koR0A69izmQlnQ3F3arz5OhJfPtcF+qXD8DDxRWvkCq07P0mE377kK4lz1VhIaB2W7q2rEto6UC83ZyxWt3wLxNG99fHMn/S+zzVOZy7gjxxcXLDv1x9Hnr7Wz6+P+SvG34FqTkfhl8A/haTxFPrGPfDcmyNH+PxsAvCMsOHewY9QtW42Xw3YxfxqSZuAYF4nttegdsIhk9T3h73A//udzfVinvj6uyGf9m6dOrVkgqX/mm8zvaIiIjCIBERERFxWE7UeWEGMTHbORCzhR0bl/Pn1K95u3sonrlHmfvhNyxNLeoaCyuTDaM/ZVasHe86/Rk5fSnR0ZFELvyJD7vdhfNVb4DY2DP+fb7YmIxbtQf5ZMpStm/fzJaF/+PNe0pgHpvPsI8WcEuzICBtxXjGx2RhKdORD35dyvYd0ezduoZFrzfk5j7Qm8u2sf/hh125lGn7EmNmLGFrdBT7dmwicslkRr/cilLX9Wkol+1jP+D77Wm4V+nGv8fNJXJrFHu3rWPNzNEMDnfBjF/K5CVxGCFteHfSIqK3b+NAzBY2j3mIUhfd1LJxNGYniaYTVft/y4IP2+Jxhb3aDs9iQkQq/m260NLzKNMnrSD57Lkq+P4Kp9DbNXzoOHwJ27ZFs2/7OlZP+ZiBDfzJOTSLdz5aRIJ5pTYX7FwZgY0Y8N5XzFy6hl0x29i7dSWLvuhJeSOBPxduuP6Gisgdz/BpwSNdSmLbOp0pMblgxrNg8hLiLFV4sFc9zvXryY5aw/pUC8Hh9bnr3O8x5xo0rueBgYUK9ev91RPFM4zGNZzJ2baWdakmYKVK/3cYUteDUxEjebxTM6rWCKfRA28zdb+N0t2G0KeSFcO7Ee0aeWPf9j3dmzSh/t1tCA9vSOPun7IyHcBKpX7v8GIDHzJjpvJarzbUqBlGWLuBfLj0OEbJjgx7re3FoYaZzLw321Gndm0q1WpO57f+4KjNxJ64ga+HPECT8LpUqdeWvqM2kYwfLXu0oZiFAtecH4tfAH6GSUpSMqfm/8TU2FI88Fg7gs7WZS3bnSfaehE9cQJrU5NJSjGx+AcQaClsG52oOeAN+ldyIXnLeF54IG/ZOm368MKYDcRd1GH5+tsjIiIKg0RERETEAdlO7yV633HiUzLJsZmYdhtZqWc4unsj8yd9zuAHHuSF2bF35ATH9tOLeK3P03w8fQP74zPJzc0kfv96Zi7eQboJdvMqw7xl7OC7gY/wzFdz2XjoDBk52aSd2s3yXz6k70NvMOvYbTgiWdsYOehJ/jNtAwfOZGKz28jNTCEudg+Ry+cRsTfjGvPHFJyZsoHhfR/hhVFzWLvnFMmZNmw5acQdiiJi4xGyr3e7qZv47NG+PDfqDzYejCct20ZO+hmO7NjEvmRnDPMM894eyMtjl7HtWDJZNhu5WWkknDrK7q3rWLs36Xwb0yO+4KVvFrPtRAqxR49fuSbzDAsmzOWYzSQrajITt2Zd9FpB91e4hhZ0u3bity5k9vKt7DmWQHq2DVtuBmcORzF/7Lv0eugd5p3867rMr80FOVeGcZzIP6M4dCaDXLsdW0YCh6MWM/r1Abw65/T1tFBEHIJBUNsetPXLZPW0Pzh89tdN2qppzDoKpTr1oIXX2UXTN7JsQzrWio1odDb1sVZuROOgZLZvj8VSsxENvfPSDtc6TWjgaWP78tWctgPWULp2qYpT8mL+8+polu1LJCs3k1PRM3hv5DJSrZVp2jAIy9l50IzgqjSsGoSbAZhZnD5wlEQTsFbmvvtr4JITzVcvv8+UrSdJz8km8dBqRr/yLyYfN/FvdR/3XJgGmTZSTp8iOcuGLTuBHTNGMXG7DcMpjh2rYjiRmkNO2jFWjR7LoiQTa5nylLUUouZ8jqqrry8eFjupKWnYs7YwbsJmXFr1p3dlK+BGw/6PUDd9CWOmHsRGOimpJoZfQN4QdYVpozWU9veWx5K1iS9e/piZ0XnLJsduYfaEhey98K3JDbRHRETA8A0MvsXP/omIiIiIXN2bb7xB8xbNAejb/7EiruZOZBDcazQr3q/Punfu4f+mnLlpgYrIP8mEcT8CsHLFSj4cPryIqym8xJV5odgfq3wY/HHpIq5G5DaxlObx8bN4q9IShrR9jXnnuk1iJXTwZOY8W4plr3fiyZlnMDEIfuA7Ij4IZ83QdjwxPYlyT0xg4ZPHeO3NBF4b2YG1Q9rx0tJs6r85l1/7JvPF/Q/z9V4buLfnmzWf0cn9St1wbRz6oT/3fnyU9p9NZWTnYCymnayEQ2zbsomIWRP4Yf4e0tzaM2rNZ7Q7NobuXb8g+qLnMNzoMGI5ozqcZnTv+xi+FSo//St/PFeC6YNa8/qKnPPLdfxyFd/cs4332z3GT8fOJmDWUF6YMYVni83k8RbvEOFUwJo/2sLFA+daKNH3ByLersrcZ+7mxaXZGD73MmLeCBovfYa2IwP5/I/3qTjlUTp8vIVsoxh9f1zA+9X/4Ilmb7HUqRBt3NWGr9d+TofY0XS7byTbLljWUuIRJi56i7ClL1P3uflkFvQcfBRN1een8/tT3kz6v/a8u67QMwSK3HG+ee0onZolA+DXPLiIq5G/JYMpNzKEtIiIiIiI3GaWYuF0rAN7th/gWFwSmU7+3BV+H6883QgX+z42R19nLxAREZE7kLVSV3rWdcXi1IlRGzrls4RJix7tKTH7F47ZTeJXLiMyqxkNWzfGd+YmWtwdSs7GX1m+OoEmCb24++46uEYk07pFCdg3iyUHzqYTZ3v8XJmBq7srhhnH3KH9SN38IB0b16FeWC3qta5AeKvWVLX0YMjSmzGpjZ2c7FwwnHG+aIzYHHJyTTCMvN4xBa05n597+3hjmJmkZ+ZtwUyO4Mfph+jS5/94xQygpcsWPvwlKq9Hq5lBeqaJ4eaNtzNgFqKNhjWvVsNy7bkDr7s9IiIC3NB8oiIiIiIicpu51n2Y4SM74XXpnQ7TxrE53zJp9504+J2IiMj1cKZOt66EXvXuloFreDfuLz+Zb/fbsJ9cypyNr9CkcVtaVvSmbW2TDf9ZRUJ6OotXJdOzZRsa/J7MveVNdn29kPN/VnNiORhrx+47kyfavcOy9KvsMvMIEeNHEDEesHpTtef7/DCsLa3aN8Rj/gH2HbVjKdeIZuWsRO+/4O+2Zz1ahLlB9mH2H7Vzw7M7FKbmixj4+HhhmFmkZ5yLX3LY9usvbOg3lEcfMkmY9yq/Hz03BGg2GZkmpuGJt5cB8YcK3sbsw3nLlm9Kq4rfEL37Kr14Ctwe6/l/rbrzKSJynobRFBERERG5Yxi4JO5i2fqdHDmTTo4tb66aw9simPjhQHq+sZBTV5kySETuHNWrV6dy5cpFXYbI35tbON07lcaSsZqhd9eiQmiNS77qct93e7E7V6Nb1yp5EYF5msVz15Ph1ZzHhz1IAzYx/894TNJZPX8FSSH38uIbnalo7mTO/P1/zS9o28WCRfuxB3Vl2McDuKd6cXxcrFisbviXrkGrBuXynri2VqBNzzbULuWDi8XA6uxEbkoKWYBhgGHbzaxZO8h2rsWzI97hgdoheDi54FuuKYM+eY9eJQwSI2az+MxN6Odb0JovY+Dh5YFBBplZf9VhPzaX8UsTsduOMXPSMhLOv2QnMz0T0+KNj5cFCtNG2y5mz4khx6k6g7/+iIEtKhLgllejb5A/Hhc+/FKI9mTn5GIaftRr1ZjSHlZEREQ9g0RERETkJipWLJjw8Pps37aNw0eOFHU5Dsgkaf0Ynus/pqgLEZFbLCwsjEce6c3OnTuZOm0a69auw25X2ityIY8mXekQYpC0YBrz8n0aIocd02ew5fFXqNu5C3VHxbApxyR+8QyWvNaC+8Krkh7xLkvi8lKNtLXzWZLQhQfDDDLW/sysgxf2ts1l29j/8EPrbxnY9iXGtH3p4j1t/pi2j/zM4cBGDHjvHZo6X1KK/QzzFm4gDRt7xr/PF3eP4dUGD/LJlAf55PxCJjmx8xj20QJuRhZU0JoPXXboLHh4up/tGXTBj80k5r3UnIovXbq8SUZmFqbhiY8XUKg22tj98zA+bfo/3mjYnqFj2jP0kq1nFbo9No7G7CTRrErV/t+yoNgr1Hh+fkEPmoiIw7osDHrzjTeKoo47zo6dMcz8fWZRl1FoOr9Fb8bvM9i5c1dRl1EoVauG0r1b96IuQ0RECqgoJz13sjoxZMhgAFJSUomKjmJbVDRR0VEcPnxENzJF5I4QGBRIjRo1SElOJikpmeSUZEzz9s7G5evni91mp0poKG8NHUp8fDzTpk9n0cJFZGRkXHsDIv8ALbu2IoDTTJ4WQeIV/ovajsxn5qZnqdewPfeHf8WmtZmYySv4de5xOvfxYcXsZcSdWzd9HTMXnaLHQx78OXkexy5522KmbGB430fY/vgAerdtSPUygXhaM0k4to8tG4+QDRjGcSL/jKJUeGVK+bliZCURu2cT88d/w8g5p/PmvMnYwXcDH+HAgGd44r4m1CjphT3hIJuWTOWbb35l/embN+RrQWq+jOGKp4cVw8wgPasgv/vMvN9Lhhc+3hbAVrg2ZsTwv4EPsavfIAbe14La5YPwtOaQnniaI/t3sSViP+cGjytoe9IjvuClb7x47cGGuB49fiOHUERug/u73U/1qtWKugyHkl9+YfgGBl/0W33u3Dm3tag71coVK4v0Rsv10vkteh8OH87KFSuLuoxCad6iuYJEEZE7SOfOXYps365ubkyfNvX893a7HdM0sVqtpGdksC06mq1RUWyL3sb+/fvPh0NvvvEGzVs0B6Bv/8eKpHYRkQnjfsz353a7neSUFJKTkkhOTiYlOYWExESSk5NISk4mJSmF5JQkEhISSU5OJjkpieycq8x7UQBDhw6lWdMmeeNKASYmpt0kNzeX+QvmM2P6DE6dOn3ROokr877/Y5UPgz8ufUP7FxERkTvHN68dpVOzZAD8mgcXcTWFd+HnQbk5LssvDKZomDgRERERuWmyMjPJys7G1cUFAIvlrykqPdzdadCgAfXDw7FYrWRn57Br106ioqLx8/MrqpJFRC6zcsVKvvr6awICA/Dy8sLL0xsvb08C/AMICAzA28uboKBAypcvR0BAAIGBgTg7XzwmVHZODqkpKaSmppKamsqZ+DPEJ5whNSWV1LS/fnbm7M8SExMv6j0ZEOB/PggCMDAwLAYuLi506tiJrl26smnjJn6bPJkdO3bctmMjIiIiInemK4ZB69dvYOTXo25nLXeEKz0pdqfR+b29GjZswHNDninqMm6KkV+PYv36DUVdhoiIXOK5Ic/QsGGDoi4DgKTERIoVK5bva4ZhYFjzJvF1cXGmZo2a1KxZE+OCG55WqxWb7eYNjyIicj3OhTgF5enlhZ+vL74+Pnj7+ODj442vrx9+fj74ePvi7etD1dCqecv4+eLm5nbR+rm5uXm9ipJTSElJJiQk5Ir7cnLK+yhfNyyM+g3qc/DAQabNmI7JVAz0+1NERETuXG/sKVfUJdzRhlc+dMXX1DNIRERERArN2dkZXz8/ggID8PX1IygwEF8/PwKDArDZCj4vkN20YzEsJCYmnu8dpCBIRO5EaamppKWmEhsbW6DlXZyd8fH1xcfHBz8/X3x8zgVJ3vj4+lKpUqVrbsPJKS9cL1euLC+/9BIbU/pT0mUShnXpDbVFRERERByPwiAREREROc/NzY3goGB8/XwJDAzEz98vb1ikgAD8/f0J8PfH398fH1+fi9ZLTU3lzJkzJCQkkJ6eimk3MSzGFfaS9wS81Wplc+RmJk6aRM8ePTRGtIj8o2Tn5BAXF0dcXNxlrxmGQccOHQq0HdM0sdlNnCxgMz1IsdXCyTsDw9iFaRZk4ncRERER+SdQGCQiIiLyD+Dl5XV+7ouAgLxwJzAggICAQAIC/M9/7+nlddF650KevK8E9u3bS/yZMxfNdRF3Oo709PTz6wwaOJBy5crjZLn8rabNZsNus7Fk6TKmTZvGsWPHbnnbRUTuNN7eXlgsFkwgv1g9NzcXJycncnNziYmJYePGTeyI2cGq71dhkMv+RB9Ms/TtLltERERE/sYUBomIiIg4mGefHUJAgD9+fv4EBgbi6+t7fn4JgOzs7PMBT2JiIoePHCE6ehvx8XEkJiQSf7aHT1JS0kWTmRfUmYQELnwW3TRNTNMkIyOd33+fxZw5s0lOTrkJLRURcUze3nm9L88FQTabDavVit1uZ//+/WzcuJEtW7ayM2YnObk559czyC2CakVERETkTqAwSERERMTBlChRgvgzZ4iNjSUu/gxJCUnExceRlJgX9KSlpd3S/Z9JOIOT1YJpt2NYLMTHxTN5yhQWLVpEdnb2Ld23iIgj8PX1BfLC9Nhjx9i0YSNbtkYRHR1FRkZGEVcnIiIiIncihUEiIiIiDmbo0LeKdP8JZ85gGBYOHjrIb79NZuXKldfVw0hE5J8qKSmJjz/+hKitW0lITCzqckRERETEASgMEhEREZGb6vDhI7z11tts2bKlqEsREbkjxcbGEhsbW9RliIiIiIgDURgkIiIiIjdVfHw88fHxRV2GiIiIiIiIiJylMEhERERE/laeG/JMUZcgIiIiIiIi/8/efcdHUfQPHP/s3l16vTR6DRB6r4LSVZqADURQVKwIPmIXGzYe208F9VERH1HRxwIoTaSGohTpSAfpgfTcpVzd/f2RBEIIyV0KCfB9v168SHK7s7OzM7N7Mzsz4ooinUFCCCGEEKJK6dSpY2VHQQghhBBCCCGEuKKolR0BIYQQQgghhBBCCCGEEEIIUXFkZJAQQgghhKh0b06dClMrOxZCCCGEEEIIIcSVSUYGCSHEFclA3eFv8uvvc3iuk/T7V0VKeE9e+G4Ba6b2xbfEjUO59vmZzHi4IxHKpYidN1RqDJ7CvN/n80p3U2VHxiteXYNy4Uejoa8w+8OR1JdiKYQQQgghhBBCXPYUX18evT6CH7v54lPZkSmBdAYJAYAv7Sb8jy1/LeLf/SOocm2tooCKuFZX5vUPrt2MZrXD8FOulDO6sih+NWjWsj5RAcYS8pxC8DUTeXVUBxpEqDgqLEalLQcKgbWa0rx2OH6XWVa78BpUdF3gxBVcmxb9JvLq7bUxlHv4QgghxNVOp90d/7Bl9kH+3dV1xTzXCyGEEKLqUgwGGkUYMRuVvGcPhRatzSy4PZJn6qhV6nmkwjqD/LpM5MeFS/lr02b27d7JwZ2b2LpmIb/M+DfP392XuNDSNoEYaD72Y35bPotHmkgzyqUQOHgae/f+xYInOxFWZO41ce1razi0eyHPtLp8r4miKCiKilqVSuiVyhDL+DnbObzjB8Y3KW4kQSBdJy/m4N4tfDE0+OxfK+JaVc71D6TfW2s4tPcvZt0ec/EK2dSG537fweGdsxhb6/Lvww8cPI29+7Yz/6GGVbwxvIrcb4yNuOvxYdRMWcS/p23EqlfcoaQerOg0cPPPd1P5bLcvXR5+mD4hV3FCCyGEqPLCeh1nz7zdrL4v27v55dUcnpy+h8M/HGdoYEXF7uIUdBSFq/p5RgghhLga+FYL4qPBkfx6WzQrRsUQf0c0v90SyZf9Qnk0zpdalTwjh4L3nS+Nmobx1dBwRodXRIwqcM0gQ1QsLWNrnJt2xRBAWHQ9wqLr0arHQMY+uINZk5/kjWUncXkVskpY3WY0qp6GjzzcXTqKP83veY8PE+7ivm8OVeCb6ZXFzuYPbqPtB5Udj6uEGklMlIri25T7/jWInx6ey2ntws0MjUby1K21MShuwiPNGLDirpBrVVnXP4t1i+JJHTyUToP6UePHbzhRRDr4thvIgFoq9k2L+e1UERuIClI17jeB3ccwuqnC7g9nsCy9AnuCpB7kkqSB6wDffLaMse9fz33DPmbZV8eRUi2EEKIqsh715ZhupV5dOxFKAGc8fAxRAuzEReu4T/mxx1axcSzi6Gye3YC2sy/1cYUQQghxqRn8jcSFB/46LQAAIABJREFUGs5NzaYoBPoZiPUzEBvjx5DGOby6zMLq7EsdM51d21MZuN3b/RRCg03UC9QqbLq5Cn7F3MXuj26lWbMWNGjWjpbXDGDYQ68xY+1JXGGtufv/PuX5rsFVaqjU5SIyMpIpU6bQp28fAgICLsERddx6CN2ffp9nu4XKNbuMderYkSeffIJOnTphNFZSF7lvJDGhCo50C4Ye9zOund+F2yhh9H/wLlra00nXFMzmcI/yneobTFRMFOEBRZ9bSZ9fatkbFvF7ooZP24EMrFPU6BM/Og/qS3XVxoYFy4rsNKsqqlraXhGUUHoP70ukYzM/zjuMu7LjI8qBTnr8zyw6Y6Tt8EE0qtrD44QQQlzF3Cf92JsDxjq2C+5XanQq//tpNwfeS77gM0MdO42MkHnEj6NXycOL6uMmKsJFuF+BHjPFzTV3H2bL14d5rvVVkhBCCCEEcOutt3L/A/cTF9fkkhzvwI4U+n57hmu/PUO/H5O5Z6WVBak6PiH+PNbKhyJaHa9aFd5ipzntONw6OnYyk4+ybcVRtq1cxPInZzLznibc+cxofhj+MXvcoET05Nn3JnJjk1rEhPii5yRz6K8lzHhvOnP3ZXHei0iGxkz4ZQcT8o+T+D2je7/KH04vw7lMKapK+/btaN++He4JE9iwcSMrV6zkr02bcDidFXBEF9u//ojT1z/K6LdeYcdtk5h7qrgHWhPXvLyUWbelMX34Lfzf3vxtFUKHfcTGqV358/k+jP0pFR2FiGvG8fxd19GsYW1qRIYQYHJjObWHVbOn8+n2mgwddRP9O8dRK8xA1sld/P7VO0ydvZOCL8krgbEMvP9h7h3UhbhoX3LO7GPt3E9567N4Tjjzjt3mNibe1Z+OzWOpVy0MfyWH5KNLeGXMFA7e/j2LJlRnzv29eHpNgTT0r0Pfux/kviHX0KJWCEpOGif3rebjya8y7zL8duPj50vPnj3p2bMn2dk5rI6PZ+WqeHbv/htNuzQ9Dao5kghVI3HRJ8xv9wSjHxrMF+N+pOCgF2OjETzc35c/3/2YzMeeoFtkWF7vtYFGD114rVRzB8a9+BwP9G1MuElB151Yjy1lyt1P8/MprYTPlSLCLCpfgi3tGNuWfce773/P1rRCtYlfLXqNfoB7b+pOqzoR+GMjI+kkh/fvYumX7zJjY/qF9U/OX/zyewK3j2nGTQMbMuOj/ec3+Ad2ZWifSJTMlcxbloxOWeo4b8pl3icllquS075kZa0DFMxd7+GZMdfRolF9akeH4K/YSTt1gA2//8CnX8xnZ3p+PLxPg2LvNx6kT24atWbkww8yql9bGkSYyE7Yy5/rM4gp6ZWMgE706xqEa+dKVpwplJY+Neg+6j7uHXItbepHEWx0kpl6mqOH9rNt/n+YMmc/bq/Ot+iyBZRQDxYVcZXIa5/j22m3E73jfcY88AU7L3gT51LV/d5cg6LTwLMy50WdYdvGsnXp3DG0F33qfsa+w5ffvUQIIcRVwOXHrhMKQ2JtNIuCtQnnPqrW0UprIxjrWelbPZIDJ859FlnXRjWDwtZDfjgAJczKs5OSuLGeg5hADd1u5NDuEGZ8Hc3cI+rZZ67QJmlMHGKhY0M79SLc+CsKyQmhvDK5OhvrJ/P8kEya1XJQI8xNgBFsVh+2rTfz7rfhbLWcO36j2w6zaKSTOa825uktua+TRbTxfH8AfB30GpjMvb0yaVXNjT8KGWk+HD7qx9JfYpixy4AOqKHZjLv/NA90sRFuBF1XsJ4OZsoLtfg51Ub/HjbCQ2BwVxtvbK+EOfOEEEKIShAWGspNQ4Zw05AhpCQns3T5cuLj4zl29FiFHE/XwKmDDtjsbg6czObtTIVGA4OIjfKhjuIgIcKfsU39aG02UjNAxQ+dNKuN95dZiLeBYjLSu3kgt9XzoaG/gi3HxV+HsvjP3/bzXsxW/UwMaRnITbV9qOMHOVkutpzRiCz0Bnu9Fma+bG1gycpkpp4q0IhhNNA9LojbG/jQOFBBdeucTrPz9XoLv1vztlGM3D0whrvzftVycvjXXAtbyqHZtnJe39YzWD/t3/x4/QzGNLqRgXGfsudvN7jCaNiuMbXyx0EFxdC05xjebhGN86YnmJ/sYTdOeYVzmTAYDHTu2ImuXbpgt9tZ/+d64levYcuWzbhc3k3CVxz3ycU8OymY+jPHMuXde9g39nN2l8uwfxVzq34Mvq5ZgQxpIrx2W4Y9/QXDCm3tU7cDt0/+BHPWzTw470zu9DoBLRn/xec81jb47HA3v9qtGfzoh7SpMYGbJseTpqtEd72F0QMKHieYqGgT9syLRM03jnGffsEzncPODaPziSG2TW2CcqrwEA0PBQT407dfH2648QYyLBZWrVrF2rVr2f337go9rhJmJlzVSU/cwKyZq7njjbHc0/ZXXttsz9sghD7330Fc8nzunruPG+/T8TNHEKiAo6jiq1bj1qnTeOq6EBRXFilnMlGCIgiLNmHP0Er+vMiVa4rKlxAY2ZBrRjxPm8b+DB89k/35RcwvjnGfzeCZTuEF5icPJKJWYyJqNcBny0xmbkwvYmSHg83zFnBo1IM0HjiA5p/uZ8fZYqsQeu0geoVD2oJfWJ7fkHyp6jhPypVSUtp6oqx1gEpEmxsY1rvg/kYi67Vh4P2t6Xt9Rx4b/SK/Fe5MKSuP6h1QQroxedY07m7kd3Z0m2+dNgyok/uzvZhDGJu0pXWgxolt288fFeYXx7hPP+fpzmYMZ/ObkdCY+rSKqU8T6++8OWd/+YwkKrEeLNyjpRDacQIz/u92auz7nHsfmVlERxBcurq/bNcA8LDMeVNn2Nmx5W+cN3eifZtgOJxeUgyEEEKIS89tYscBI+7Gdlo30CAh726rOunbLRtTloH0ABs3dLbz2QnfvOcOnaaxNgxuE9sPGNEAxeWmYVMbtfKXCg1w0bRDKm/HunBOrMX8vNtgdOs0Rne3FbiP6kSZdezZYG5sYXB72/n32DA719yQQJu6GsOfj2B/MQ8+Xu3va2PcC0d5poW7wHO9TkSMjYgYOz57Ipm5y4BbdXLrxOM81d6N4lZJSVFRAtyEmcn9fqn5sXSdH4P7wML18k6yEEKIq4vT6cRkMhERGcktN9/MiNtvJ+FUAitWrmTlipUknE4oOZCyUDhvhqGIav4Mq2sq8CygYA4AhxMwmrirdzhjo5SzbQu+QSb6tA6jWWA649bbyQAUHx/G9w3jljDlbNg+wSZ65S1zXuKyKgYjI3qF81CMeq4lxaBQN9JAQPk14Rer8lYiz9nOqg0ZaGpNmjbKfUNGt67j7TFD6dqxPbFNW9G0yyDumbEDW0Qvbu5ZaIoo934+vKkV9Zs0p36T5jTskfuWttfhXCEMRgOKouDn50ePa7vz4osvMPu72Tz66KM0a94MRSmPs9bJ3Dydx/5vM1qbh/m/f3UkuDwTU7ew+Nn+tG7VitiW3Rn4/CJOuHW09E1MH38LXdu3oXG7ftz58WYshHHd8N5EqwAGGo95gfFtAkiM/5B7BlxDXPP2dL5lMj8ddlNr6HhGxRZo7NetLH1pEB3atiG2VVd63PoBG4ocTGWg/qgXeLxTKLb985g8+kbatWpD0469uWHMv1lyhXQqGo2538pCQ0IYNGAgb7/1Fl/N+oqxY++mVq1aFXJMNcxMmKJjzbCQ+Nt/+elkTW4Z2/9sL7qhzjDu6xfEzm+/YX2mhQyrjhpuJuIiNZYS3Jn+nYPRdn3KsK5d6XBtb9q370SXYe+wNrvkz4tVIF82bN6Z7qOmsvS0RmDrUYxql/+N1kDDO19kUqcwHIcX8NLoG2jTshWxLTvT/cV4skvIKu698/l5pwO13o3c1KbArKBKOH2GdCdUP8OiOevIf0Hg0tRxnpWrMqVtYaWuA87tv+ip3jRv3pKGLbrQ/fan+fyvNEx1b+K1p/oQXtqEKfJ+42m9Y6TFvc8wJtYHy7aveeyW3jRv0ZbWvUfx2IxNJBfbP6UQVK8+1Qxujhw+VmBdGQONxrzMpM7huI79zhv330THNq1o2KwdbW/+iO3l+gDhbT2oEtr+Eb74+B5ij87iwQemsdFSQgGo8Lq/LNcgL4relDmP6gwd6z9HOKOZqNegtofXQgghhLjUFPbs98OORrNY+9nGEzXGwuAmsHdRNb49Ds17WM5NFWdw0LyBhmLzZ8uR3Duknh3E25Mb0vWOpsQOa0rT0bHcM9cfW6iVmzu4Ct1HDSz9pCEdbm9K7K1N6PFEFBtc5z5b/GEjWt/ajIbD4+j+bDWWJkNg41RGNfXgu5mH+zccmMCkFm4cJ0J56flY2tzSjNhb4uj+cfB5z/VKQBb9W7jRDkYybHQcHe5pTPuRcXT5VwxrbbnHWzuzAe1GNeC1bTIvrBBCiKtX/jIV1WtUZ8SI25nxxedMnz6NITcNITw8vNyOo+StGdS0ZgBPdgskVoX0ZCfHzt6/ddZuSGHI94n0/C6J2xZnsd0NDeKCGROlkHIyk6fmJ9FndiJDF1tYnKFTrUEgN4Xl7t2kWTDDwxQyk7OZsjiJ/rMTuXFeKlP+dpDqwaNI7cYh3BejYk/P4d2lyQz6LpG+PyRx1zIrqwsOuNBd/HfhGXp8k/vvup/LZ1QQVGZnEC5SUzPQFZWAoIDciCgKYS1H8MbMn1m3YRM7Vs7i5etrYMBITPVIzyNbXuFcpgwGI4oCgQEB9O3bm7ffeotvvp7F/Q/cXw6hO9j/9XNMWWEldvRrPF+enWu6G2tSIha7G7cjjd1zP+bbv90oxmR2r9vD6UwnzqxTrPvsC5Zm6Bhq16OOChiaMHhQHEbLMl5/8jNWHkrH7rKRuHMur3y4kkxDI7p1KnDddRdpJ0+Qku3Ebbdw6ugZsooqsIYGDBrcAl/nDj6c8CLfbjxGmt2JzXKG/Vv3k3T5Dwy6gMGY+yUlMiKCYcOG8emn/+Gzzz6le/fu5Xoc39BQAlSNTGsWmn0bs77Zik/PMYxsZAD86DTmDtpkL2fGT0dwk401U0cJMxN2scym67nTp0XF0SkuEj8F0O0k/XMidzqpkj4vToF8qbkyOfnXbN74ehcuQwRxcVG5+crQgAEDm+Pj3st//jWZWRuPk+Fw43ZkkpSSWfLUlO5j/DrnL3LUGgwa3oX8VcDU6tdzc7dAtKOL+GlTgbvCpajjPC1XZUnbwkpbBxTYPzM1lWyXhua0cnLbAt58+GV+SQJzn5voGVqOvdeepo+hCdf3rYdq38z7k97il51nyHY6sJzcxvxvfudgsUN3VCIizahaNikp2edNWzdkSDN8XLv5aPxTfB5/kOQcN5rbjiUlnZzy7Kf2qh5UMHeZyH8/vZ+mJ77loXHvsq7wVIpFqei6v0zXIP/UvChzntQZgJ6aTKqee42FEEKIqiprfwD7NajZOIeovJtYg2sstFF9Wbw6hPnrfNHrWhjSIG/S1IAc2tbUcf3jz/YCr8eGNUrljVcOse6bveyY8Q8vd3ViAGKiXIXuo5CW6EOKTcHtMHAqwXTuu5oO1jQjFgdobpWTu828sdAfl8FFXH1Xyc/AnuxvsDOghw0fzY//vFODWbt8yHCC26mSlK6e/1yvK7nPweF2OtV35j0HKySdNHn/HCyEEEJcJQyG3LbH+vXqM+6++/j661m89947VKtWrdRhNm4TQfydMaweFc1vt0TyWa9gBpkV3FYb03bYOduiputkZLlJc+m43RpnrG6yFRN96pkwOGx8tC6LPzM0HJpOSkoOH+ywk60aaR9jQFVMXFvbiOp2MHOtlaUpGjmaTmamk+X77Bwt6d6vGOlT34SP28l/V1uYd8ZNhlvH7tD4J8nlUWdSeajEVb6NmM2hKLpGdlY2uhLCtZP/y4yRdTGdba/zpU5tADeq6mFUyyucEnTv0Z2FPRaUS1gVKX/UR1h4ODcNGXL276GhoaUP1H2KOS+9Qrdm/8ctrzxL/M4XyCprRIs8TgJHT7mgaRQxYSpk57U6Ok5zIlFHiQ7AXwFMtWlQS0X1v55pG69n2oUBUaNWdVSSvTu+sS6N6hnQjm/kj2Plt57Ds888A8+UW3AVJr9yrlmzJjVr1gRy5940l0OPfXBoMKruICvLAWgcn/cNSx54jztGd+XLDyO4Z0g1jv/4NMvSdVCyyMzSUOuGEHKRtnzd+gdzlyfTa+B1PDdrGZPSjrJr22bif/2Gmb8dIKukz72qcN2cOnSELL05QUGBuZ2hZ/PKH6w6WJo1uzTO/PYTy//VhYH9htHnrTXMT1dpMHgYHX1d/D13Lrvy34i8RHUcPp6VK6Vc07bwITysA4qhZ/zB8i0OhvatQ8OaKpTXbFwepo9qiqJeTRXtxBb+SvC+B9nHzwcFB46CY4196tCwlpqb3w5VxBpxBXhTD6ph9L3vLtDSWfHdt/yZUsoe8/Ku+8t4Dcpe5oqoMwDd4cCpg4+vT7F7CyGEEJXJnRjApkRoXT+HViZIcNq46Tob2oFo5p9SOLEuhL9vS2RI72w+PBCI1jCHFiaFf3YHkKgBiptr7z/CjBscBe6jbupUA1AKTMNWOqeO+5Cl2wjy10r1kuIF+xvsNKqho50OYtWx4kPUswOZu9FIrx5WnnvNyiSLD7v2BRIfb2bmOt+yPQcLIYQQpZDq7MGe7HdYuLBy4+F2u0tel1wBVcl9laNJ4yZQYFarAFUjW/P+VWdN08lxaCRkONl+0sa8A3aOlNRsYjBQOwhUox8v3+bHy0VsEh2koqoqNQNBy3SyozQN4aqBeiGgZTrYYi1584pSeZ1B/q3p2TkUVTvK3gNZYL6Ju4fVwZC2gekvvMW36w+RlGMkss/zzHt/SMnh5VHMfcslnJLs3buXufPmlVt43goJDeGRhx72aFu3243BYCAxMZHo6GgAMjIyynR8PXkFr774M+0/vZmXn1/L2zlFbIMG+OLnV9onfA2nwwWKCZOpYBhOnC4dlLx5HPNGJlycgq+/r/dfDhQ198uJXr5P8XPnzWPv3r3lGqY34uLiGDZ0qEfbujUNVVE4c+YM1apVQwFS09LKHIfgkGAU3Ua2LTdtdUs8X845yqBRd/OEbuY6n228+d2O3Lk29RyybTqKXzDBJqCoSlxPZuFzo8nceis3dmlNu7YtaderPu179iJOHc74hSV97t056Q4HDl1Byf/2qpowqoDLVeo1WvSMeGYvTGDAqB6MGFCdhT9Fc9vwOIzZfzB73pGz4Za1jvO4XHparjxI+9KXIA/rgOJPBF3Tc/8/+5ey1k14nj6KIW/kq1qqBgqHzYGODz4F+wv03DPArXmUtmU6X2/qQT2TLUs3E9nzWnq9+AXvZN/DpAUnS1EmyrnuL+M1KI/nigvqDHLnGjYp4LCXOKuwEEIIUXncfvzxt5F7e+XQvr7OKkM6N9VU2PR5CMfdoJ0KZd7eJF7ols613wRwLC6HCN3Ib9tz1xBSQq3c3duBwRLI9I9i+HanL0k2ncjOp5n3ZNm+kwLoThWHDopauifOC/ZXwKgAbkp+htGNLJxWj8y96dzYKpt2TXJo1zGN9h2sxCkNGL/GWIbnYCGEEMJ7QYY9NAl4jrtfDKnUePTu1Zt27dqWuJ2e12aj6zqWDAvh5twX0L3tCNq/LYVxu1yU6pVUnRLv174GBaVAG1TpZuQ5t85QZT4fVE5nkBJKl/FPcWtNFdf+JSzc40aNrUY1H8he+jXTlu3NW3DJSUqSpdDCzjoulwudAAICLmzWUSM9DadskpOSWbtmbTmG6J2o6Gh46OKfu1xOjEYTFouFVfGrWLNmLXt272HBgvnlFAOd9LXv8fx3nfjvyCd57EwACpYCn2tYMzLR1Zo0iQ1F2ZZScRndeZIjJzW00F+4r/8LrLzoOiVeztOcF65apxNdaxvYeaR8Rgft3bu3UvNOSZwuFyaj8eyibqtWraJBwwa5I5rKSUhIEIpuJ/vsfFZOdn3/HZtGP8ddt+ukLX6SeSfyq3AHOTYdXQkkOEiBi11f23Hiv36P+K8BQzBxN09h5sv96Hl9JwIWLiKr2M+XlO2EnKc5layh1ulApxoqu46X5vZjY9P/5rJ3xCN0GnkLXdJqMryOQsqvP7Ao8Vzp8b6OM2A4W9N7US49LleUnPZepkS58m9JpxY+4Mg9H8CLuqmY+42n6WNoxqETGmq9bvRs+BE793szkkcjJTkVTW1MREQAChm5cXWe4J+TGmrtdrSvprLrZHH5rYx1sTf1oO7k4I+P88APjzFr2p0Mee19Es7cw1ubrBVT/1+Sa1BxzxWKORKzknuNhRBCiKpLYev2AHL6WOnS2sZ1MRZq2IJ4Z50pt8FFM7FweRCTHrVyW1cbq1vYUWwh/HEw99lJDXNSzQTZf5qZtsEv7z6qkJJmKNfv5+XGZeRUOqjVsukUDbtOl7C93Yf4BdHELwAMbuL6JDDzQQs9u2UTsCakcp+DhRBCXHV81GQi1eWsXRNVqfFo2iQOLtIZpOs6muZGVQ0cOLCflaviWb0qnoceeojuPcp3mQqPaG5OZoHmk8Mzv1j482LrMCsmjmWCGuJDl1CFvd7OCZt3HDXIh3bBsM9S1EY6Ll0HFPwrqNemwpfPUY0mDApg8CEwsi6te43g+Rk/8OW9TfFzHmH21FnscYOWkkiiE/w7D2dU+xoEGRVQTQQF+RXqsdJJPJ2EbqhOv1v7UT/IiMHPTMMOzalh8CacK4/blZtbbTYba9es45VXXuXOO0fz6X8+Y/ffu9HLeYQLupU/3p/C7OOh1KjhV+iNazeHd+7BovvS/cFnuaNtDAEGFYNfMFHh/uW3zhCAex9Llh5GixzMy2/dS59m1QjxMaAa/Aiv1ZyeHeuW7tq79/Hb74dwm1ozcdoU7uxcj3A/AwZTENWatKFJxJWz+lR+3klPT2fRokU8+dRT3DduHLNnz+bUqVPlfryAoAAUcrDZz+VJ7dRCvl6RjuY+xS+zV3JuqRENW7YNXQ0mJOgiaW6oT++be9OqZgg+qoLBZMRltWInd5SpUtLnZT0h1x6WrkhA823LxHefZHDzGIJ8fAmv25Hh/Zri6SRQ7oNz+HZ9DobYEbw/uT9m/Rhzv19LwdGj3tRxDqcLXQmjXc8u1Aow4FW59LRcVXTaekMJpNMtd9CrUQT+RhMhtTty1xuvMLKWStaGpazJ0L1Lg+LuN3iYPu59zF+wB6exGY9M/zfjejTE7Je7XWhkOEW803De8TOPHOGM20C9BnXO3bDdB1i6/Aiab3see/cJBjWLIsBoIrhGa4bc2f/cAs65G5etLva2HtTdJK95i7sen8NRU1PGvf0CN0RVUF3paR4t0zWoqOcKheB69YhRnRw5fKLUoQghhBCXgmVHEH85dOK6neGRbi6S1oWxrEADRtL6MJZYNHoMOs0tjXVydgWxMa+nR0s35d5HW6YzqpmTIAOg6gQFaFXz+7nbn6UbfdB8spk46TSDGzoJMumEV89ieFfb+c/1Bge9+1ppFe3GRwWDEVzZKnYFFEVHUdxcc/dhtnx9mOdal9+040IIIcTlyO3OvRcmJCTw/fc/cN994/jXvybx6y+/kl7GGazKRHcSf9yF5u/HY9cEco3ZQJABVEUhNMhEl2hD7jOL7mTZEScu1cTo60IYUcNIWN52wf4qfh4cZ9UxF26DibHXhjA0xkCoAVRVISrcRIO8AFKyNTTFQPdYP2qbwGBQqRttIqacGtgq+PnLSLPxP7NvfOG/67jTd/LV5Em8/ocl943hlBX8sHw8PQb25sXZvXnxvO3d7C/w87H4Feye0JJWw99hxfC8Pzu38caA0Xx+3NNwrgz5HTwul4v16zewcuVKtmzZgtNZwetI5B/fupH3Xp9L7//cQq1Cn2Wt+Zqv9/Tl0eY38tr3N/LaeZ+W57Q4LnZ98Toze33CuH6PM6Pf4+d96tz6Fv3u+IqjXg/WcPH3F6/x6bUf83CLobw6ayiv5n+kZzF/wrVM+N1WXABVWv70gZlZWcSvXMWq+Hj27NlT/p2GRQgI9M8bGVTgj3oGix/vTsPHC2+tk2OzoyuBhAQVHZ4S0Zl7X3mBbqZCH2ipLP59E9kRfYr9vOxv7NnY9Nk7/NrnHYa2HsOHc8YU+vxirxUUjs8Z5n+9hIndhhETqWP763u+2X5+WdG9qCtP7NlLuh5H3JhPWBL9BB0n/uZFufSsXB0rIe0v6duQig/1bniKmTc8dX5U0tcz9Z0F5A+w8jwNir/fzPCo3nGz/6uXeafb5zzT6Xqem3E9zxWKdnFvxbr2b2Fb1miub92KGHUnpzQAJzu/+Dff9v2Q0W3vYtrcuy7Yr2CYZauLPakHC99vNJJWvMHD79fjh0k38tqr69n58BxOlHIJoeLi5lndX7Zr4HmZ84YvLds1w+Q+xJbtRb4OJIQQQlQZekYQS3arXNs2m1aaL/9ZGnj+YP3sIL5b5cPQoTm01FVWbAgk/2VZPSOIHzYa6dHDyotvWgvdR5Uq+P1cYdPP0fza+SRDG6fy4XuFR/AWmPI1NIt7H0qgW+FWFc3I4j8DyVJt9O9hIzwEBne18cb2wAqPvRBCCFFVGA0GXG43RoOBM2fOsGzZclavXs2JE1Xvhcj9f1v5sWYYI2oHMbX2+Y2PziQro3/P5qQO/+y18Hn1cB6M8eOR3n48UiicklpYDuy28l2NMO6M8GdSP38mnf1EZ/nqJF4+pnPypJ0DrUw0bRjK7IahuR9rTj6an8r35bDWUIUNbXAnHWTnoQRSrDacbh3NmYMl+Ti7/ljMf99+nCHXj+KVpSfPNZHqqSyePI5JX6xk1ykLdrcblz2LtMQT7N++gfUHM85OM+M+8BUTnvySlQeSyXa7cWWncHjrQZIUxatwLndut5stm7fwzjvvMmLESKZOncqGDRsuWUdQLp2MNR/wxqLEC+dltO/iw/sf4PWfN/FPqg235sZls5J88gBbVi8m/mBOuV0L3bqJqXfewWO5CYeHAAAgAElEQVQfL2D9gUQsNjduZxbJR3cQ/9fxUnc96ZmbefeuO5nw8SL+OpJClsONMzuV47s3c8hiurSjHspRTo6NVaviefHFF7lj5B18/Mkn7N5dAaPHLiIwwICi55Bt9+R4Ojk5OaAEERJcdJWlKAlsWbWDo6k5uDQNd04ax3Ys47On7+XJBUlQwuflcdZa0lKeGvUQb83ZxOEUGy6XjZTDG/ll2W6yddB0z1rCM9fO5sdDLnQtnWVf/8oFM855Ucdlx7/P4x8tY9dpKydPJOSWAy/KpSflqqS0v6T1rZ7F9kVzWHMgiWyXC1vGCbYv+YxHR47nywMF6kUv0qC4+43H9U7OHj4fdztj3/mJtfvOYLG7cbtsWJOPs3vjMn6OP1zkUlgAZG1k+fpMjC170Sv6XP7XM9YxZfS9vPzdH+xPzMThspNxcie//RzPP4VfPC1jXVy6etDGnpmTeeuPLMKue5wXBsdUyAPHJbkGFfFc4deavteEox+KZ0U5TUEqhBBCVBjdyMoN/th1cB4M48dDhe/+CtuWhbHbBbojgKVbCqyVoxtZPL0uk+YGsyvJgN0NLodKWqoP+/cHsv64ocp9P9fSQnjq2Tq8tTyAw+kqLrdKyolAftngl/tcn7edopjY8pc/RzNUXBq47QaO7Q/msw/q8uRqI7rmx9J1fqRl+rFwfYnvCwshhBBXlPSMDOb/+gsTJz7GPffcy+zZs6tkRxCA7nTwye+pTNlpY2u6RqYb3JpOqtXJhkT3ufYCl4vvVqTy5JYcNqW5yXSDpulk5bg5cMbO4pOuYl8H150OPl+Wyis7beywaGS7wenSSEh1cNSR+8qJlp7NlHVZ/JmuYdPB5dI4luSivCaYV0Ijos579lq4cAEAGzdu4sPpH5fTYa4c38z6EoC1a9by5tSplRYPH5MJP38/LBbvugTl+laOTp06MmH8wwC8OXVqpa4ZFBwcjN1mw+FFp2H3Ht3Prhn04fSP2bhxU0VF7wqjEHXbZ6yZ0oENL/Th7h9Tq9yX3SuDgUYPfc+iCdWZc38vnl5zKTvEK15Q79dZ8dFAEt6/meGfHip2MWO1+h18u/R52q6YRJsJv3H5jl28kimEXf9vlr3fl3/eGsqIL4+VvEC1hyaMf5hOnToCMHDgoHIKVQghxOUkfW0SAIvWhfDIW4XnbhBlEdX/GGsezmLDR424e6lRnuuFEEJUKR89dYIB1+TOPBHWvXLXDAoPDycjIwNN83yKkGefeebsmkHPHKhbUVG7KkxtdBQoov9C4ccrZ9GTq4zD6fS6I0gIAKvV6lVHkPCMGt2egf3a07iGmSAfA8aASBr3GMvrD3XGRzvC1p1XzqhEcWllrv6Kb/ZC8zvvo0/Y5ToeUZxlbMSd4/oSlvo7M+YcL7eOICGEEEKUD9WczcAu2TSOchFk0jH6uWjcLoXXb8vCR/dl68GqN5pJCCGEqErS0tK86ggSl06VXLNRCCEuN75tRjD1wwEEFW6r192cWvAJs/dLk68oJdcB/vvuXG757GaeGT+PP1/fgFVaIC5TBuqPeJpxzZ1seP1jlmXIhRRCCCGqGt8mqUx92lLEc73CqdWRzD4qL+cIIYQQ4vIknUFCCFFmCj7p+1i5sQFtGtWhWqgv2DNIOLyTNb9+xfRvN5AoL0SIUtOxrPuAF76ty+g0HV8F6Qy6bJkwZZ1g97JlvPB9+U0PJ4QQQojy42P1Y+UuB23qOKgWpIHTQMIJf9asimD6okB5rhdCCCHEZUs6g4QQosx0MjbOYMKYGZUdkauUmwOf3EqjTyo7HhVITyf+9XuIL2EzLWE2I1vMviRREqVhY//clxg5t7LjIYQQQoiLydgVyYTJkZUdDSGEEEKIciedQUIIIYQQ4qoQF9eEYUOHVXY0hBBlNHfeXPbu3VfZ0RBCCCGEEOKyIp1BQgghhBDiqhAZFUX3Ht0rOxpCiDJas24tSGeQEEIIIYQQXlErOwJCCCGEEEIIIYQQQgghhBCi4sjIICGEEEIIcdX5cPrHbNy4qbKjIYTwUKdOHZkw/uHKjoYQQgghhBCXLRkZJIQQQgghhBBCCCGEEEIIcQWTziAhhBBCCCGEEKIK8ff3x9fPr7KjIYQQQgghriAyTZwQQgghxBVm4IABJKekkJGRTnJyChkZGTidzsqOlhBCCA/Vr1+fqVPf5MCBA2zZsoWt27axf99+XC5XZUdNCCGEEEJcpqQzSAghhBDiCjPmrjEEBQWd9zdLhoW0tDRS09Jy/09NJTUtlfTUNFJSU0lPTyc5ORmbzVZJsfZer569sGZa2bx5M7quV3Z0hBCi3GRkZGAwGIiLi6NRo8bccccdOJ1Odv29i81/bWX79m0cOXIETdMqO6pCCCGEEOIyIZ1BQgghhBBXmNtvH4HJaCI4JBiz2YzZHEFQcCDmcDPmCDMR4WaaNWuK2WwmKioKg8Fwdl+H00mm1ZrbWZT3LyUl/+c0UlNTSM3rPKrsRsh2HdrRu1cvjp84wQ//+x+rV6+Rt+aFEFeEjIyMsz8bDLmzu5tMJtq2bkvLlq0wGu4hOyuLbdu2s/PvXez+ezcHDx6srOgKIYQQQojLgHQGCSGEEEJcgZwu59nOHCi+gTAoKAhzhPlsZ1FQUBAReZ1I1apVo1mzZkRERBAYGHjefpmZmQU6jXI7ilIKdCJlZmaSeCaxwkYbRUVGAlCrZg0ef/xxxo4dy08//8yS35ZcViOchBCisKysLDRNQ1ULLfOrgDGvAz8gMJAuXbvQpWsXVFUlNS2NgznrCDVuRDHuq4RYCyGEEEKIqkw6g4QQQgghrnKZmZlkZmZy7OixYrfz8fHJHWkUYc4bcWQmKDCIiLzf69SpjdlsJiws7LwGTIfTSWpK7oii1JRUUtJy/08t8H+mNTOv48pzkXmdQYqSeyyz2cy4e+/lrjFj+G3JEn768SevwxRCiMpiNBoJCQkhJCSYkJBQcmw2AgMCit2nYF1rDg/njGMQWe4GGIPmoSgHZQpNIYQQQghxlnQGCSGEEEIIjzgcDk6fPs3p06eL3a6kKeoaxcZi7uT9FHWZWVZSU1JJTExE0zTCwsIvOLaiqvj6+jJowAAGDRzA6tVr+P77/3HixIlyTw8hhCiOj48PQUFBBAUHnTfyMigw928RBf8WFHRBR7rVavXoOG63G1VV2bJ5M2Ou+4ow40YWpYeg67Uq6tSEqDIMEVk8MCaJW9vaqBWsY0sN4s3nazO7+EeVUlFCrEx+LpH+CVH0/SAEe/kfQhRQo+cp/jMih62fNuClrUplR+eqJ/lfiCvDRTuDYmNjmTD+4UsZF3EJyfW9tMLDL2ysulzdeH1/unTqWNnREEIIUUhsbGxlR+EsT6eoU1WV0NBQwsPDiTCbCQsPIyIiktCwUCIjIqhTpw5t2rTBbDbj4+Nzdj+Xy0VGhgV/f7+Lhm0w5j7m9ujRg549e7L5r83s3rO73M5RCHH1KTg6MigwmKDgwPOm1QwKCjzbsRMZGUlAoVE9+R3e+aMxMzMzOXbsGCkpqWRm5f5ecLTkc889R9OmcReNj+Z243JrrFixgrlz53LixAkmrE2q6GQQlUKn3R1HmDHIzdJp9XjmTyNX15ivi5y/KYeJLxxjfH2d/K6CoFAdRyYY6qQy69VE6u2pxsi3wjhWDsscKj5OmjWyE5UG0jVR8QKjbTSNcbFHEruAyqsLJP8XdLXXyRVvVDV5nqkoF+0MMpvD6SQNvlcsub6itBo1qjqNjUIIIS5vmqaRlpZGWloahw8fLnbbwMBAIsxmQsPCiIyIpGbtmowcMaLEYxjzOoXatm9Lh44d0JEvsEKI80ftBAUFnTf1ZVBwEMFBwZjN4bl/CwoiNDT0vJGMcOFoxszMLBJOnz477WX+iMbMzMxSTYWZmpKCroNSoNLSNQ0UBavFwvwFC/n111/JzMwsjyQpE79WiXx9fwb1zW6C/TUMbhWr1cixI/5s3BrCzyuC2VuqaOo0v+k47w7UmP9GXT46cvXW4Ao6igLq2SS4utLmwvMH35ZpjKyr4zgWzhNvR7P0pIop1A1ZQISOChRedutyVXFl7Gpz+ZebospCVXE15dOqfB2uBC2Dsys7ClcsmSZOCCGEEEJUeVlZWWRlZcHx4wDExTXxqDMIQNN08pfNyP++VrdOHTZu3FQBMRVCVIb8zp2C01PmT8mWv65Z/qid/J8Ly8zMzOvUyR2lc/r0af7+e/fZUTuZ1ixSU3PXP7NYLLhcrgo9p3RLBprmwmAw4na7MRgMHDx0mJ9+/ok///gTt9tdocf3hiHcTsvaTnzz/6BqhJkdhJkdtGqXwdhb/Zk1vRZvbDDhbaqFVbfRKMqAz1Xd4KaweXYD2s4+/69XT9oUff4xtR2EKgpr5kWz8JgBHbCn5jVzHY1g5F0RlzymFaUiy9jV5vIuN0WXhari6smnVfs6CFGcCzqDBg4cVBnxEJeIXF9RGmvXrGXgGsk7Qgghqg6z+eINPJpbQ1fAoKrY7Q727d3D5i1bMZqMjL7zTgCOHjt2qaIqhChnw4cPY/CgQYSEhBASGkpIcPB5a+0AZGdnk56eTkaGBas1g4wMC8ePnyAtLR2r1YLFYsFiySAj3UJ6Rjo5OTmVdDYXl5GegcFgRNM01q9fz5w5c9i7d19lR6sYCrv/V59b/ueHTdcJDHES2ziLgQOSGd0mh7ufPIYypR5Tdhiq1HQ6qo+biGAdV5aBNFvltw5XlfhUlXgUx9dXQ9FVktPVKpWnvOFdOledMnY55A9RWapOPhWXlzenToWplR2LK5+MDBJCCCGEEJed8LAwNE1DVVU0LXchAFVVyczMZMf2HezYuYMdO3dy7Ogx9LxhQd17dK/MKF+GVGoMfpmPH23L9inDeWmts7IjdJYS3pPJHz9B/6Pv0/eZZbKIMWCI6cYDTzzErd2bUStUxZZyhIVvjOPZxclXXGOLpmkcOXKUjIyM3I6dDAvpGRlkWDKwZFiwWq04nVUnv5bW6TOnmTNnLvPn/0pi4uUxd77mUnC4QUchM92HbRt92LYphOV3HWXmUBt33pPKD49HsUcDJczKs5OSuLGeg5hADd1u5NDuEGZ8Hc3cI4Ua91U7E97fw4T846SGM3pcdf5weRlOwSBDsxl3/2ke6GIj3Ai6rmA9HcyUF2rxcxKYWyXzzOBMWtRxUNvsxl9RSEvyY8OfYXw6N4yd1nNheRqH0CZpTBxioWNDO/UicsNMTgjllcnVWULx8Wl022EWjXQy59XGPL2lQON7kWlTDcfdB/l+kMaitxox4Y8CnaWqjcc/+IdHwkK5/74aLC9UgZZnugAo/nYG3pzEvddmEWfWyUnxZe2KKN76OYgTBYcG+Droe1MS912XRYsYN4rdwMkjwXw8vTrzEoo7fze3vbiH2/J/dwXy0gN1+dqVzn9nnKLz1pp0ej0Ui5fxUUNzGHlbEqO6ZNMgTCc7yZ8/dxqIKWHaOU/zQknpfDHelDFPzzeiTTLPD8mkWS0HNcLcBBjBZvVh23oz734bzlbLueN7Em+Pr7knFJ3GvRL4ckwWHWq5MOSY2L09lBmzI/jtVP7F0LnmwQPMut7A9Mcb8H//nMsfob2Ps3FiJn9Oa8zYZQU6H4qpUworrtwuzii/PO5p2p1fFqDDvYf4frCLJe805pG1BTKo4uL2lw7wZosAXn6wLrOSKz7/5/Msn0YSeI9n9dS2pp7nUU/L4AX53gSWRH9WLY7i0/1Ohg5Ip39LG7WCISvRj99/jWHqYn/S9aKuQ8H5K0u+zkJUJukMEkIIIYQQl51wcziqqpKens62bdvYuXMXf+/6m+Mnjld21K4gCoG1mtK8djj7KvylX1/aTZjFjDHBLH1uNM/8nlJsB4biV4NmLesTlWTMm/rPu/2vOD7NmfjpdMY39T23iHpUNXztmVdkOsyb9wtr16yt7GhUuOXLlld2FMqHbmD9dzH82O0oY+paGFg/kj2HFHC5adjURi1T3nYBLpp2SOXtWBfOibWYn+5h+KUJR3Vy68TjPNXejeJWSUlRUQLchJnBnreeRUSchWGdbAUaTXQia2Qz8OZs+nbL5rHna/BbindxiG6dxuju54cZZdax25zc+nTx8fGOwq4tgaQOTKdjqxx8/gjEkX/qEdl0qqFj3xrAFkeh3co7XfxyGP/yUR6L08hv5vWrlsPgkcdpE12Lm6YHk6YDPjbGvXCUZ1q6z26HyUVsEwdB5dnb72F8lMAsJr92jLvr6GfrVN/q2QyonvtzsVHyJC94kM5euVgZ8/B8zY0tDG5vO6+BMDDMzjU3JNCmrsbw5yPY7/Yw3p5ec08pGm2uLVCITQ7a90iibetsXnmuDrOOV/yopIuW22zKN4+XKu0Udm4OJHlQGh1bZ+O7Nuhc/vTPpntjHfc/QaxJ8zz8MuX/4hSRT6d7WE9FeppHweP6uKh8H14tm2FjjzKsUNR9qmdz+7hjmHMa8uBKI9rFzvFS1WVClIF0BgkhhBBCiMvOH3/8ybJlyzl9+nQlx8SPur3u5JExA+nRog6RAQr2jCT+2buNtYtn89lP271r9LikDDQfO4137wxi/sNj+Whf+a4/4tdlIl+/MIj60WaCA30w6A6y05M4dmAHa5f8xFc/byShQEOkoigoilrqhXgv3N+b8wtm+CfxvNvbt5htAJxsen0gI2advHhDQCXw7XwbI5v44DjwA0/860OWHs7EJ6oGQZnS8iCqCHsAq3YauLOPg6Z1NDhkQM8O4u3JDXn+uA9JOWAKctJ1+EmmD7VycwcXC5YZz3Vmar58WOiN/3xehZNHCciifws32sFIbnkxmu1ZgKITVcOF01YwcAOL3m/Ik2uM2BSN6g2t3HX3ae5tls5rdwex4b0Q0nQv46AbWPqfejy7yod0TSMmQsPik8UHnsSnKBdLm7+DibekM6ydldbGQDblvfUfFJdNC4PC3zsDyCiUMOWbLjqNB59mfBOdxM3RPPdlOH+cUghtYOHJiQnc3CuJUb8EM/0Y1B9wmsdbuLEdDeONzyJZtN9EjslNndou0krqFNQM/FD4zXxACSu8oafx0Wkx7DRjautY9pl56fNIlv5jwGi20WtAIpNvyiK4mOh4khfwNJ29cUEZUz1O/9yIG1g8rQHPrDGR6dao3iSdlyadpl/jVEY1NfPSLsWD/OH5NfeYrvDPumhe/SGU9ScN+EZlM+zOBJ65Josn78rgt9fDSCzNc1YxdUrR8biw3Ga4dBoPLa88Xvq0s+8N4o/MNIa0yqKlIYi/8h53/Jtm0sVf4dDWQI65dRoPr/j8X6JC+dT2h2f1VGTeNSgpj4L39XFumEaydI0m1yXw6cMWamQH8vH0GL7d4UuK7qLT8JN8fEs21/WxEB1v5vRFHgLLVJcJcYl4OMBPCCGEEEKIquPw4cNVoCPIQN1b3+Onjx7j1m6NqRbih9HoS2BELVpcM5Cxg5vhV6Wn0VcJq9uMRtWDK2QRZUNULC1jaxAR4oePQcVg9CM4sjbNuw7kgZe/YMm3E+kckn9gO5s/uI227W/gySWlGdVT1P4Ve35Vh0pMbENCFTvrZn7AwgNp2N1OrKePkpBZZXsixWUgbsRhDv6ym3/y/809xBMNS5+nUi0GdAUCAs69lR7WKJU3XjnEum/2smPGP7zc1YkBiIlyedVY4XU4uoIOKOF2OtV35tbVukLSSdPZKYByt4NMi4FsN2gulZP7Qnnz9Rr8kgbmzhn0DCpFHHRIS/Qhxabgdhg4lWAiS/MwPt6wB7LoLyNKZCb9YvMD0WnZIht/zZfVW30u7NQuz3RR7Qy+1oYxK5jX34tk5XEDdrdK4oEwXvkumEzVTrcWTlSDnUHX5eDr8ufDf1fn210+pDkUbFlG9u/1I6m8et49jY9q5/ouDlRHAO+/W41fDhjJdilYEv2ZvyCYgx7Ep8S84Gk6e+m8Mubp+ebvrIM1zYjFAZpb5eRuM28s9MdlcBFX38N4e3tMT+gqm5abWXnUSI5LIT0hkC8/rM73iRDYykqPMvVMeBOPIsqtUo55vCxpZwtk8RYDSrSVPg3PlfV2HbMIx5ff/vTFfQnzf0nOy6fe1FOe5NE83tTHuWEquJ0Gdq+M5ttDCorBwO5tfpzOVnDmmFj3UwRLM8FQzUGdiz1TXqq6TIgykpFBQgghhBBClIaxDXc93J0IElnx7otMnbOVo+lOfMy1ad7xWlrlrOLMVf/Fz8Xu6SO4+eO92HUD/qHViG3Xn3FPPMLAlvcwZexSBnywm/Idk1QaVuY81I45Z3830u6p+fwwNpif7+/F02uqxvozqm8wEWF+uKxppGXnT+6v4Ovng6LnkJycdUVOCyeuDOYQN4oO2TkquuLm2vuPMOMGB6azDWtu6lQDUDwfIVjKcPTsQOZuNNKrh5XnXrMyyeLDrn2BxMebmbnOl6xiCpKeGcjyPSpDuzhoGK1DplbmcylLfC5O5c/4EBJ6pXL9NTm8vTcAp2qjWws3+qkwVp0o33hckC4nHDSI0VF9LUz7djfTitinRowL1eCkUQ0d7XQgfyRUYM+9ycP4mJzUi9bREgP4y9ulujzMjxVzvQuVMVOOZ+eLqYhPcp067kOWbiPIX0MBtJLi7WkaYyrb6Fq7PxsOqIzu5qBelA6WknepEJ6eryd53Iu0u5DKutXBpFybQd/ONt7Z74/blE2/Dk70f8JZeEy5NPnfQ+fl01LUUwUVzqOlvSec5TZxNEmBBi5igoH8kXpOEydSFRSzhn8xnUGXpC4TooykM0gIIYQQQojSCKhF3QgD2rEFfPjFWg7k9Wg4Eg+xYeEhNpzdUCHimnE8f9d1NGtYmxqRIQSY3FhO7WHV7Ol8ur0mQ0fdRP/OcdQKM5B1che/f/UOU2fvPP8NYf969L/nIcYN6UazGoFoaUfZvOInPv7oezYmFepO8WZbQ2Mm/LLj3CLKid8zuver/JHf/6EE0OXRGSx5vQl1Ivxwpx9j27LvePf979nqwRx4msuB062j4yI77QQ7ls9kUkYUrWaNoX7HdsSouzmlGWj00PcsmlCdOYU6X1Rza0Y+/CCj+rWlQYSJ7IS9/Lk+o9Aixhffv8Tz85pCaJvbmHhXfzo2j6VetTD8lRySjy7hlTEv85tyHc++N5Ebm9QiJsQXPSeZQ38tYcZ705m7L7+zpqg8Aba0otNWNXdg3IvP8UDfxoSbFHTdifXYUqbc/TQ/n8oNDzWc2z7fdm4RdecmXup3D7MSNA/zQ3Hn9Qobm95X9jwsLit7v29A7PflFJhvNj1bulF1X/YeUyE0g7t7OzBYApn+UQzf7vQlyaYT2fk0857M8DhYJdRaunB0Iwun1SNzbzo3tsqmXZMc2nVMo30HK3FKA8avKb6pRM9rydbLEodyjM/F2P4OY96pNB7saqHD1wFsjMmiR3WdE78Gs6eo1vhyTJeC/1+Mr6+GolDq6UG95VF8ODeFjrfR8jgveJDOXledhcqYjufnezG6U8Whg6LqnsV7Y9mP6Skl7yLlh6UD6Dp+PuUQuBfKM4+XJe2yd4XyW0o6d3Sz0Gq2P7ubWuhvVti+IITDbsBY8fnfI0XkU6/rqQIK59HyqI+dTkDRMRWs7hQFp5vcx62L7XgJ6zIhykI6g4QQQgghhCiNnAROpLlRa/dh9I0/s3/BUXKK3FDF3Kofg69rVuDh20R47bYMe/qLCxeprduB2yd/gjnrZh6cdyb37Vm/Zjzw2Qye6hR67ktoTGOuG/ks3a5tzaQ7n2X+qbxGfW+29YTiS53WHc79HtmQa0Y8T5vG/gwfPZP9rovvejGay517Xqpa7HQxSkg3Js/6//buPL6K6u7j+GfuTW5WsgEJIFtYZAcB2YTIJvCCVhQUqlC3Kr5aF6RaKYuCUuuKlSpYK4gLlecp+BQEBHFBQUAEQkFpEJAoYQkkIWQnyV3m+SMgISQ3c28SQsP3/Q8kOXfmd8785kDmzDnnNe5uG3x+E+Pm1zCqecnfa2dHHBux/W7ljlGlr2c9GsYGUpRnQkgUrXtcTdNzD6TC4+gw6E5e6hyL86Y/sDrDpPycgLDy2tbWiHHPv8bUgREYrnxOnczDCK9PVGwgRdkewO49XMv54K1eht85fP/Kk740rtRFhpu+t51kXCy4DkfwUbKBrbmTRoFQ8HUMr30TfHbTcINTp+0X3dcut4FpmoQGX3xoW5T141ykyMHGNbFsXAPY3bQfmsri3+Yw6LoCQr+KqPhzQWfo3cYDTgc/pRnY6lchhirG461tAHAHs3x9CPfdk8OYrrEcaZxHO8PB21uDqbDrrqZ2OfenJzyS+37bhC8q2gvHVljSjo3y6dfI5LvjNfQ01Yd4Dp00sDXJY1Czhnx32Ho8PuVjJe2c70vdyrnHwGJ9feUt7i01dM4yjPB8hnYoybXkNAMwyc2zY9qctGvmxtjvfTCt0vvGqurMcavHqkhRKMu/DGLC2Bxu6tCAqIG5xBaHMW+jo2T29SXI/0qVm6f4109VFH5V/k2oqnP/JtR0XyZSRdozSERERETEH85EFs/fTIbRglvmrmDD+3P47Yj2xFT0upWZw7rpw+nWtSttugzgFzPXctRt4snawfyHbqVfz2u4uscwfv16IjlEMXDsEGJtAHba3PEkv+8VQeG+D5g6fgidOnen+/BJPLchFaPJSJ6aOoxow9eyZ7kP8OpNXYlv14n4dp1onVBm1oyZx5fPjWdA75607dyHAROf59MTHsK6TWRij4qXlynLsDsIi2lKp4TbeWbWOJrb3aQk7qpwE14IoPO907izjYOc3UuYcmtJXboNmciURTvIsLrGTGX185eZy6ezf8m13a+hTdd+JIz7K984wczdwkt33ky/Xj1p06ErHfr+kt8s+pbC+oO5ZVD0hW/alsqJ1p3Kb1ujXh+G96mHZ+/fGdOvH9deP4SePXvTd8xcNheUOpbnNMvuu8di0Q8AAA6TSURBVObnesZ3vpv3Ug3f86GCepWN12oOy5XFZjexG4DdJCyqmG69TjNzdjJvjykk2OVg6eIY9nnAkxVImhNCumQxsaOTcDtgMwkP9ZR5Y9UgLSMA017MsGG5xIea2IPctO54hiZ2X45Thr2YITfk0jXWjcMG9gBwFdgoMsAwzPP3qeGh9w2ZDG7uIsRuEtEon7smp3J7HOR/V4+v8qoQgz/x+NA256R8Fc2XhS5GDD/Nzb3PYD8WwUc/VPCQshrbBU8Q67924InK5qnfZzC0lZOIQLDZTKLjzjCoU3FJ+3iC+HirA3fAGR6ZlsqvuxQTHWRiD/DQqGUB7aKsNmIlfIhn9aZgnPZCHpx+jEk9iogJKikXGe0m1MvzXcu54Nf1LmH1HrNcX19UFndNnNOAelEuwgLAZvfQ+Opsps84zk3RkLk9ki/ySgolHwwmx/QwYNwJJrR3EmoHe5CbhhFlZ9NYu28sqc4cr3LbGSR9HsVuj5Nf3nSCu/q5yN4ezcdZvsfqb/6XZjlPz/Kpn/KiWvpjf1m5zoab/ncns2tJMjO61f4iyXJl0swgERERERG/uDm8fApjT97N41PuYGTPW/jjtWOZfHwnKxYv4NX/2cHJ0q80mm5y09PIKXIDp0la8Trv/2o4U1tlkLRlHycKAI6z5c23+PT27tzcrCXNbXDCaMvomzrhcH7Hi4/NYfmhkl8eCw5v5c0/zKbFmje4fdBohkav54NsH8pmWqym6STt0AGOZZeMCBzbuZRnl4xk8OMdad++Ibbtx72s/R9A5ymr+GHKRQclf997zH7rPxW/9Wlvx4gbWmIrSmTeYy/y4dGzZzm2m9X/+ITb7upFd4tVqBGmi9PHjnKqwAk4OX747KYFNoOoLrcxdWZfOrZoTExgPqkZHuwEENe4ATYyz++RdEFO5JXbtqZplixF1bA9vds3YP+OkxSaRaT/WMlC+gB2P/Kh3HrZy4nXWg7LlcSk423J7L/t4p+4c0N4d35T/rzHXvLGfnY4y7YHkJCQy6zncpl1QWmDA6W+SkmsR9KEM3QdepQNQ89+0xXCsw/Fs/CE9eNc8JPIfO79XSrXlX0i4glg3ddh52dlGCYt+59kcf8LZ7h5csN4/t1I0kzf6lIRy/GU4bVtUku+NLPqsWRTIDeMSONBE77/ZyRJFXTa1douGOxd2YjFvY4wqW8ai/qmXVDW+X0cw6bX57DH4D8rG/P3nik80CaLPz2TxZ/OFTJtrH6hHZO/rpaFxSzHc2BVY+Z2O8y0zjnMmJ3DjDJHqmiGgWkxF/y93j7dY5brW+HJLlJ53FbPadJpYjIfji9m64K23PWJl5k8hpuRjxxk5CMXfrsoNYon347g3Iqq+f+OYcmPuTzcOodnXsjhmQsPcsFXVu4ba6o3x6t6vdwnIvlHYgYv981moNvBonXh5PzcsDWf/+f5kqdnP+FDP+X1zNXQH/vPwnXeXsjwhEKiI+DGfoU8uyesRiMSKY9mBomIiIiI+K2Yo5ve5JGxN3D9xCf426c/UBx3LRNmvsWqBeNp7e3VK3cqh4+7IKghcVGl/ltefIKjaSZGSGjJJrWOFrRpasNz5Bu2/FTmLcL8XXz178KSMs1svpX1m5vjh34i3zQIDw+zvqa8aWKaJnhy2LloMjdOeIkt3jaUCWxCy6tseI7uYmdqlbaavnSMCK5/4h3em3Ebg7u0JC4iiMCQGJo3a0CQATZbZe/iXdy2Zu5WVnyegRE3kBnvfcburWv44G9P8/DItoRV1vg1nQ8WcliuDO7TQXx3JJBT+TacbvC4bORkOdi7J4J33mnK6N/F8/S2wPODv2YA6+a34LEV9dibbqfIDa5iG6czHRw4EMa2I+cfFLpT6jP5lfp8kRJAgQdchQEkfx9Muo/HKc0wAtm1M4TD2TZcHnAX2Uk5UI83/9qCxzeVejBt2tizOYqvUgIocBsU5gWyZ2sDHp7WjLdTDJ/rUhHL8ZRtd29t8zMb29ZGs89jEuQKZfkXQVT0Pnq1tgtg5ofx/Mx4pvwzkm0pAeQUGbhdNjJSQ9iY5Di7hBOYBaG8/GQ8k/8Zwc7jAeQ7DZyFdo4kh3Io3/tMGV9YjYeiYBbOacU970Wx+adAcooN3G6D3CwHSXvr8X+JQZQ7wdRiLvhzvX2+x3ypr0VW4rZ6TrsdMA3yCmwV5vep/RGsTgzhYLqdAqeB220jMzWEj1c0YfzUJqw7VapwcQivzmnOnz8L5cdsG24PuIpsZKQFsSsxgo1HDGt9io+qM8erfL3MANZ/FMlxNxQdjOb9AxfeOTWe//iXpyWs91OVtUFV++OqqPQ6e4L5dEswp/OC+WhbVdcpFPGPZgaJiIiIiFRZEScSV/Bi4of8veNY5sx7ghuvf4SHh6xlyifl7yQEHpzFLjACCQws/Qu7E6fLBMM4++aWL4/BLs365GZxMcWmgVHpTrku9s4by81/O4QbB23vfJ1l0/vQun0sFFXy67hhL6m/YbtEtao6I+YG7h7THPvpb5j/5Iu8v+0Q6WcCaDB0JivnjbZ0jIva1szgoxl3kPfvcYzs240e3bvQY3A8PQcNpr1tLA99dNpbRFWvlFeV57BcGQq/jWXcQ7E+fcYsDGLlO81Y+U5lJQ0Ob43jN1vjqnic8zynwnn5uXBerjRIg/2fN+aPu7znstUYDi5rRdtl/sVT/me9t8057mNhfJ1uEH8kkjVennhXd7sAmPlBrFp6FauWVlKuIIg1S5uypoJy5dW/ovYEMLOiuOvWi9eYsxoPxYFs+lcTNv2rknJlj28lF6y2cyn+3GNgrb4VtaNzd2N6j2l8/hsW47Zyztj6LmyeIPb+VPHLCJl7GvDongaVnO08d1Yob73WkrcqLWntvjnHW55B9eW41WN5i+fMt41IGNuoyrH6m//+5il476cs5yhV7Y8NPnulPa1eKRtcEAse7cACCzFVdp03L25Fj8XeYxOpSZoZJCIiIiJSbTxkJ61g3rJ9uG3htGnTGF+XoL9I8WEOHfVga9aH/i3KHC2sBwndg6E4heSjHt/KYuJyuTAJJdTqIvBVqwgH/zGDmWvTiej/GHPva0+Q1+IpJXVpfh2DWlvfm+i8S10/sDVoRCMHFGxewmuffc+JPCdu9xlOpedUbePiwiNsXPIXpj14F8MTBjJq1iekmjEMGtEbr3NvfMoHEalLIqJdhNnBEVbM4NtPMj7Ozief1SOzJl+LF7HCXsi1V3twJddjvU/Lskldo35K5NLTYJCIiIiIiD8c3bn/2T9w55DONI8Oxm4Y2ENiiO81hgduvhq76SI9LdPLfjoWuQ+walUSxYFdePgvT3Jr1zhCAxxEtriO+196mvGNDbI2ruazTNO3spiknUjHtDdm2LhhxIcHYA+OofW1nXzfRNkqTxrr5sxm+bEguj/wNJM6OLzUez+r1+zDGdCRB+e/wKSE1sQE27HZg4lsEG1hE+NLXz/PqbSSjYv7jGVizyaEBxhgCyQ8PNj/JRns8Qy5ZQhdr4rAYTOwBwbgys2liJKJN16bwad8EJE6w+bklqkH2fuvJPYv/YHFtxbg3BnLKztrdokkESts0U4izzhYuzqKZL/WApM6Qf2USK3QMnEiIiIiIn4I6DiUCTffQ4tb7innpyZnDr7HwvWZmFV+/8rNwSVzmHf9Ih7vNY6Xlo/jpVLncR5bx1MvrD/7FqVvZVM2biBpche6jp3LhrFnizl38+yoO1iYUsWwK2Bmb+aFP31Iwutj+N2siaz/9dscLPdhkJsD7z7F3OsWMq33CGYsGuHjJsaV1a/6Z8OYpzaw7POHSPjFEGYtHVJm42K3XxsXG/X7cO/TT3Jd2clRnkzWfbLDy0bjJee0ng8iUne4CTLtFLjdkOPgm40NeG5pJEf04F0uA56MCKY/GlHbYUitUz8lUhs0GCQiIiIi4gdP8ipe+IuDGwf2olu75sTVc2AW5XAy5Xt2fL6ChW+vJSm3mp6yn0nijUkT+PHeB7hvdD86NQnHc/onEj//gAUL/pft6W6/yroPvsvkxyOY/fBo+rSKxlGURcreH0iv0b1eTLI2vcbLXwxm7pD7eHTUKh5YnVVBvfexcNKv2H/H/UwanUDXlg0IszspyErnSPJ+dm9MrnATY6iF+pmZrHtiEo+deIR7R/akbVwYdlchudmnSU9NYdsP2T6/7WoYqez68luu6tmWq6KCMIqyOXYwkY+XLODVNemYlS1E6EvuiMjPKtsn5LLmCeaNGVfzRg0c+r+6XUTk8lGD/ZSIVMyIrN9Q74GJiIiISJ03IGEA06dNA+DV+a+zffuOWo5IRKzq3bsXkx96AIDnnn+ezV9truWILk9Zm0t23V67JYIHX2xay9GIiIjIpbJg6lFG9c8BIGpAw1qORi5LBsu1Z5CIiIiIiIiIiIiIiEgdpsEgERERERERERERERGROkyDQSIiIiIiIiIiIiIiInWYBoNERERERERERERERETqMA0GiYiIiIiIiIiIiIiI1GEaDBIREREREREREREREanDNBgkIiIiIiIiIiIiIiJSh2kwSEREREREREREREREpA7TYJCIiIiIiIiIiIiIiEgdFlDbAYiIiIiIiIhI9RnVP4cfP0yq7TBERERE5DKimUEiIiIiIiIiIiIiIiJ1mGYGiYiIiIiIiNQBKzcE1XYIIiIiInKZ0mCQiIiIiIiISB1w96yI2g5BRERERC5TWiZORERERERERERERESkDtNgkIiIiIiIiIiIiIiISB2mwSAREREREREREREREZE6THsGiYiIiMgVZ+SI4fTt3au2wxARi6Kjo2s7BBERERGR/2oaDBIRERGRK07btm1qOwQRERERERGRS0bLxImIiIiIiIiIiIiIiNRhRmT9hmZtByEiIiIiIiIiIiIiIiI1wGC5ZgaJiIiIiIiIiIiIiIjUYRoMEhERERERERERERERqcM0GCQiIiIiIiIiIiIiIlKH/T/w0Li1aIIVvgAAAABJRU5ErkJggg== ) ``` custom_bp.train(project_id=project_id) ``` ``` Training requested! Blueprint Id: 3d753707758ad45b97684811a8756c20 ``` ``` Name: 'My Fun Custom Blueprint' Input Data: Numeric Tasks: Missing Values Imputed (quick median) | Smooth Ridit Transform | Binning of numerical variables | Awesome Model ``` ``` custom_bp.delete() ``` ``` Blueprint deleted. ``` --- # Blueprint Workshop URL: https://docs.datarobot.com/en/docs/api/code-first-tools/bp-workshop/index.html > Learn how to construct and modify DataRobot blueprints through a programmatic interface. The Blueprint Workshop is built to ensure that constructing and modifying DataRobot blueprints through a programmatic interface is idiomatic, convenient, and powerful. Browse the topics detailed in the table below to get started with the Blueprint Workshop. Some examples in this documentation reference methods from [DataRobot's Python API client](https://docs.datarobot.com/en/docs/api/dev-learning/python/index.html). | Topic | Description | | --- | --- | | Blueprint Workshop setup | The first-time setup necessary for using the Blueprint Workshop. | | Blueprint Workshop overview | An overview of the workshop's functionality. | | Pass features into a task | How to pass one or more specific features to another task. | | Blueprint Workshop API reference | A complete overview of the Blueprint Workshop API. | In addition to the documentation that instructs you on how to use the Blueprint Workshop, DataRobot provides code examples to demonstrate common usage. | Topic | Description | | --- | --- | | Workshop walkthrough notebook | An example workflow of basic Blueprint Workshop functionality. | | Advanced feature selection notebook | How to reference specific columns in a project’s dataset as an input to a task. | | Custom task notebook | How to create a blueprint with a pre-existing custom task. | --- # Declarative API URL: https://docs.datarobot.com/en/docs/api/code-first-tools/declarative-api.html > Use the declarative API as a way to programmatically provision DataRobot entities. DataRobot offers a Terraform-native declarative API used to programmatically provision DataRobot entities such as models, deployments, applications, and more. The declarative API allows you to: 1. Specify the desired end state of infrastructure, simplifying management and enhancing adaptability across cloud providers. 2. Automate provisioning to ensure consistency across environments and remove concerns about execution order. 3. Simplify version control. 4. Use application templates to reduce workflow duplication and ensure consistency. 5. Integrate with DevOps and CI/CD to ensure predictable, consistent infrastructure and reduce deployment risks. DataRobot recommends using the declarative API as a code-first method to provision DataRobot resources end-to-end in a way that is both repeatable and scalable. ## Declarative API services DataRobot has two services for using the declarative API: Pulumi and Terraform. DataRobot recommends using the service that supports your engineering needs. Pulumi is based on Python, while Terraform is based on yaml. Note that application templates are configured for Pulumi by default. For information on using Pulumi for your declarative API needs, access the [Pulumi registry](https://www.pulumi.com/registry/packages/datarobot/) and review the [installation guide](https://github.com/datarobot-community/pulumi-datarobot?tab=readme-ov-file#datarobot-resource-provider). For information on using Terraform, access the [Terraform registry](https://registry.terraform.io/providers/datarobot-community/datarobot/latest) and review the [installation guide](https://github.com/datarobot-community/terraform-provider-datarobot). Review an example below of how you can use the declarative API to provision DataRobot resources using the Pulumi CLI: ``` import pulumi_datarobot as datarobot import pulumi import os for var in [ "OPENAI_API_KEY", "OPENAI_API_BASE", "OPENAI_API_DEPLOYMENT_ID", "OPENAI_API_VERSION", ]: assert var in os.environ pe = datarobot.PredictionEnvironment( "pulumi_serverless_env", platform="datarobotServerless" ) credential = datarobot.ApiTokenCredential( "pulumi_credential", api_token=os.environ["OPENAI_API_KEY"] ) cm = datarobot.CustomModel( "pulumi_custom_model", base_environment_id="65f9b27eab986d30d4c64268", # GenAI 3.11 w/ moderations folder_path="model/", runtime_parameter_values=[ {"key": "OPENAI_API_KEY", "type": "credential", "value": credential.id}, { "key": "OPENAI_API_BASE", "type": "string", "value": os.environ["OPENAI_API_BASE"], }, { "key": "OPENAI_API_DEPLOYMENT_ID", "type": "string", "value": os.environ["OPENAI_API_DEPLOYMENT_ID"], }, { "key": "OPENAI_API_VERSION", "type": "string", "value": os.environ["OPENAI_API_VERSION"], }, ], target_name="resultText", target_type="TextGeneration", ) rm = datarobot.RegisteredModel( resource_name="pulumi_registered_model", name=None, custom_model_version_id=cm.version_id, ) d = datarobot.Deployment( "pulumi_deployment", label="pulumi_deployment", prediction_environment_id=pe.id, registered_model_version_id=rm.version_id, ) pulumi.export("deployment_id", d.id) ``` --- # Define custom metrics URL: https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/dmm-custom-metrics.html > The MetricBase class, along with four additional default classes, provides an interface to define custom metrics. The `MetricBase` class provides an interface to define custom metrics. Four additional default classes can help you create custom metrics: `ModelMetricBase`, `DataMetricBase`, `LLMMetricBase`, and `SklearnMetric`. ## Create a metric base In `MetricBase`, define the type of data a metric requires; the custom metric inherits that definition: ``` class MetricBase(object): def __init__( self, name: str, description: str = None, need_predictions: bool = False, need_actuals: bool = False, need_scoring_data: bool = False, need_training_data: bool = False, ): self.name = name self.description = description self._need_predictions = need_predictions self._need_actuals = need_actuals self._need_scoring_data = need_scoring_data self._need_training_data = need_training_data ``` In addition, you must implement the scoring and reduction methods in `MetricBase`: - Scoring (score): Uses initialized data types to calculate a metric. - Reduction (reduce_func): Reduces multiple values in the sameTimeBucketto one value. ``` def score( self, scoring_data: pd.DataFrame, predictions: np.ndarray, actuals: np.ndarray, fit_ctx=None, metadata=None, ) -> float: raise NotImplemented def reduce_func(self) -> callable: return np.mean ``` ## Create metrics calculated with predictions and actuals `ModelMetricBase` is the base class for metrics that require actuals and predictions for metric calculation. ``` class ModelMetricBase(MetricBase): def __init__( self, name: str, description: str = None, need_training_data: bool = False ): super().__init__( name=name, description=description, need_scoring_data=False, need_predictions=True, need_actuals=True, need_training_data=need_training_data, ) def score( self, prediction: np.ndarray, actuals: np.ndarray, fit_context=None, metadata=None, scoring_data=None, ) -> float: raise NotImplemented ``` ## Create metrics calculated with scoring data `DataMetricBase` is the base class for metrics that require scoring data for metric calculation. ``` class DataMetricBase(MetricBase): def __init__( self, name: str, description: str = None, need_training_data: bool = False ): super().__init__( name=name, description=description, need_scoring_data=True, need_predictions=False, need_actuals=False, need_training_data=need_training_data, ) def score( self, scoring_data: pd.DataFrame, fit_ctx=None, metadata=None, predictions=None, actuals=None, ) -> float: raise NotImplemented ``` ## Create LLM metrics `LLMMetricBase` is the base class for LLM metrics that require scoring data and predictions for metric calculation, otherwise known as prompts (the user input) and completions (the LLM response). ``` class LLMMetricBase(MetricBase): def __init__( self, name: str, description: str = None, need_training_data: bool = False ): super().__init__( name=name, description=description, need_scoring_data=True, need_predictions=True, need_actuals=False, need_training_data=need_training_data, ) def score( self, scoring_data: pd.DataFrame, predictions: np.ndarray, fit_ctx=None, metadata=None, actuals=None, ) -> float: raise NotImplemented ``` ## Create Sklearn metrics To accelerate the implementation of custom metrics, you can use ready-made, proven metrics from [Sklearn](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics). Provide the name of a metric, using the `SklearnMetric` class as the base class, to create a custom metric. For example: ``` from dmm.metric.sklearn_metric import SklearnMetric class MedianAbsoluteError(SklearnMetric): """ Metric that calculates the median absolute error of the difference between predictions and actuals """ def __init__(self): super().__init__( metric="median_absolute_error", ) ``` ## PromptSimilarityMetricBase The `PromptSimilarityMetricBase` class compares the LLM prompt and context vectors. This class is generally used with Text Generation models where the prompt and context vectors are populated as described below: The base class pulls the vectors from the `scoring_data`, and iterates over each entry: - The prompt vector is pulled from theprompt_column(which defaults to_LLM_PROMPT_VECTOR) of thescoring_data. - The context vectors are pulled from thecontext_column(which defaults to_LLM_CONTEXT) of thescoring_data. The context column contains a list of context dictionaries, and each context needs to have avectorelement. > [!NOTE] Note > Both the `prompt_column` and `context_column` are expected to be JSON-encoded data. A derived class must implement `calculate_distance()`. For this class, `score()` is already implemented. The `calculate_distance` function returns a single floating point value based on a single `prompt_vector` and a list of `context_vectors`. For an example using the `PromptSimilarityMetricBase`, review the code below calculating the minimum Euclidean distance: ``` from dmm.metric import PromptSimilarityMetricBase class EuclideanMinMetric(PromptSimilarityMetricBase): """Calculate the minimum Euclidean distance between a prompt vector and a list of context vectors""" def calculate_distance(self, prompt_vector: np.ndarray, context_vectors: List[np.ndarray]) -> float: distances = [ np.linalg.norm(prompt_vector - context_vector) for context_vector in context_vectors ] return min(distances) # Instantiation could look like this scorer = EuclideanMinMetric(name=custom_metric.name, description="Euclidean minimum distance between prompt and context vectors") ``` ## Report custom metric values The metrics described above provide the source of the custom metric definitions. Use the `CustomMetric` interface to retrieve the metadata of an existing custom metric in DataRobot and to report data to that custom metric. Initialize the metric by providing the parameters explicitly ( `metric_id`, `deployment_id`, `model_id`, `DataRobotClient()`): ``` from dmm import CustomMetric cm = CustomMetric.from_id(metric_id=METRIC_ID, deployment_id=DEPLOYMENT_ID, model_id=MODEL_ID, client=CLIENT) ``` You can also define these parameters as environment variables: | Parameter | Environment variable | | --- | --- | | metric_id | os.environ["CUSTOM_METRIC_ID"] | | deployment_id | os.environ["DEPLOYMENT_ID"] | | model_id | os.environ["MODEL_ID"] | | DataRobotClient() | os.environ["BASE_URL"] and os.environ["DATAROBOT_ENDPOINT"] | ``` from dmm import CustomMetric cm = CustomMetric.from_id() ``` Optionally, specify batch mode ( `is_batch=True`). ``` from dmm import CustomMetric cm = CustomMetric.from_id(is_batch=True) ``` The `report` method submits custom metric values to a custom metric defined in DataRobot. To use this method, report a DataFrame in the shape of the output from the metric evaluator. ``` print(aggregated_metric_per_time_bucket.to_string()) timestamp samples median_absolute_error 1 01/06/2005 14:00:00.000000 2 0.001 response = cm.report(df=aggregated_metric_per_time_bucket) print(response.status_code) 202 ``` The `dry_run` parameter determines if the custom metric values transfer is a dry run (the values aren't saved in the database) or if it is a production data transfer. This parameter is `False` by default (the values are saved). ``` response = cm.report(df=aggregated_metric_per_time_bucket, dry_run=True) print(response.status_code) 202 ``` --- # Configure data sources URL: https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/dmm-data-sources.html > The DataRobotSource and BatchDataRobotSource methods connect to DataRobot to fetch selected data from the DataRobot platform. The DataFrameSource method wraps any pd.DataFrame to create a library-compatible source. The most commonly used data source is `DataRobotSource`. This data source connects to DataRobot to fetch selected prediction data from the DataRobot platform. Three additional default data sources are available: `DataRobotSource`, `BatchDataRobotSource`, and `DataFrameSource`. > [!WARNING] Time series support > The [DataRobot Model Metrics (DMM)](https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/index.html) library does not support time series models, specifically [data export](https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/dmm-data-sources.html#export-prediction-data) for time series models. To export and retrieve data, use the [DataRobot API client](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest-release/reference/mlops/data_exports.html). ## Configure a DataRobot source `DataRobotSource` connects to DataRobot to fetch selected prediction data from the DataRobot platform. Initialize `DataRobotSource` with the following mandatory parameters: ``` from dmm.data_source import DataRobotSource source = DataRobotSource( base_url=DATAROBOT_ENDPOINT, token=DATAROBOT_API_TOKEN, deployment_id=deployment_id, start=start_of_export_window, end=end_of_export_window, ) ``` You can also provide the `base_url` and `token` parameters as environment variables: `os.environ['DATAROBOT_ENDPOINT']` and `os.environ['DATAROBOT_API_TOKEN']` ``` from dmm.data_source import DataRobotSource source = DataRobotSource( deployment_id=deployment_id, start=start_of_export_window, end=end_of_export_window, ) ``` The following example initializes `DataRobotSource` with all parameters: ``` from dmm.data_source import DataRobotSource source = DataRobotSource( base_url=DATAROBOT_ENDPOINT, token=DATAROBOT_API_TOKEN, client=None, deployment_id=deployment_id, model_id=model_id, start=start_of_export_window, end=end_of_export_window, max_rows=10000, delete_exports=False, use_cache=False, actuals_with_matched_predictions=True, ) ``` | Parameter | Description | | --- | --- | | base_url: str | The DataRobot API URL; for example, https://app.datarobot.com/api/v2. | | token: str | A DataRobot API token from API keys and tools. | | client: Optional[DataRobotClient] | Use the DataRobotClient object instead of base_url and token. | | deployment_id: str | The ID of the deployment evaluated by the custom metric. | | model_id: Optional[str] | The ID of the model evaluated by the custom metric. If you don't specify a model ID, the champion model ID is used. | | start: datetime | The start of the export window. Define the date you want to start retrieving data from. | | end: datetime | The end of the export window. Define the date you want to retrieve data until. | | max_rows: Optional[int] | The maximum number of rows to fetch at once when the requested data doesn't fit into memory. | | delete_exports: Optional[bool] | Whether to automatically delete datasets with exported data created in the AI Catalog. True configures for deletion; the default value is False. | | use_cache: Optional[bool] | Whether to use existing datasets stored in the AI Catalog for time ranges included in previous exports. True uses datasets used in previous exports; the default value is False. | | actuals_with_matched_predictions: Optional[bool] | Whether to allow actuals export without matched predictions. False does not allow unmatched export; the default value is True. | #### Export prediction data The `get_prediction_data` method returns a chunk of prediction data with the appropriate chunk ID; the returned data chunk is a pandas DataFrame with the number of rows respecting the `max_rows` parameter. This method returns data until the data source is exhausted. ``` prediction_df_1, prediction_chunk_id_1 = source.get_prediction_data() print(prediction_df_1.head(5).to_string()) print(f"chunk id: {prediction_chunk_id_1}") DR_RESERVED_PREDICTION_TIMESTAMP DR_RESERVED_PREDICTION_VALUE_high DR_RESERVED_PREDICTION_VALUE_low date_non_unique date_random id 年月日 0 2023-09-13 11:02:51.248000+00:00 0.697782 0.302218 1950-10-01 1949-01-27 1 1949-01-01 1 2023-09-13 11:02:51.252000+00:00 0.581351 0.418649 1959-04-01 1949-02-03 2 1949-02-01 2 2023-09-13 11:02:51.459000+00:00 0.639347 0.360653 1954-05-01 1949-03-28 3 1949-03-01 3 2023-09-13 11:02:51.459000+00:00 0.627727 0.372273 1951-09-01 1949-04-07 4 1949-04-01 4 2023-09-13 11:02:51.664000+00:00 0.591612 0.408388 1951-03-01 1949-05-16 5 1949-05-01 chunk id: 0 ``` When the data source is exhausted, `None` and `-1` are returned: ``` prediction_df_2, prediction_chunk_id_2 = source.get_prediction_data() print(prediction_df_2) print(prediction_chunk_id_2) None chunk id: -1 ``` The `reset` method resets the exhausted data source, allowing it to iterate from the beginning: ``` source.reset() ``` The `get_all_prediction_data` method returns all prediction data available for a data source object in a single DataFrame: ``` prediction_df = source.get_all_prediction_data() ``` ### Export actuals data The `get_actuals_data` method returns a chunk of actuals data with the appropriate chunk ID the returned data chunk is a pandas DataFrame with the number of rows respecting the `max_rows` parameter. This method returns data until the data source is exhausted. ``` actuals_df_1, actuals_chunk_id_1 = source.get_actuals_data() print(actuals_df_1.head(5).to_string()) print(f"chunk id: {actuals_chunk_id_1}") association_id timestamp label actuals predictions predicted_class 0 1 2023-09-13 11:00:00+00:00 low 0 0.302218 high 194 57 2023-09-13 11:00:00+00:00 low 1 0.568564 low 192 56 2023-09-13 11:00:00+00:00 low 1 0.569865 low 190 55 2023-09-13 11:00:00+00:00 low 0 0.473282 high 196 58 2023-09-13 11:00:00+00:00 low 1 0.573861 low chunk id: 0 ``` To return raw data in the format of data from postgresql, set the `return_original_column_names` parameter to `True`: ``` actuals_df_1, actuals_chunk_id_1 = source.get_actuals_data() print(actuals_df_1.head(5).to_string()) print(f"chunk id: {actuals_chunk_id_1}") id timestamp label actuals y predicted_class 0 1 2023-09-13 11:00:00+00:00 low 0 0.302218 high 194 57 2023-09-13 11:00:00+00:00 low 1 0.568564 low 192 56 2023-09-13 11:00:00+00:00 low 1 0.569865 low 190 55 2023-09-13 11:00:00+00:00 low 0 0.473282 high 196 58 2023-09-13 11:00:00+00:00 low 1 0.573861 low chunk id: 0 ``` To return all actuals data available for a source object in a single DataFrame, use the `get_all_actuals_data` method: ``` actuals_df = source.get_all_actuals_data() ``` When the data source is exhausted, `None` and `-1` are returned: ``` actuals_df_2, actuals_chunk_id_2 = source.get_actuals_data() print(actuals_df_2) print(actuals_chunk_id_2) None chunk id: -1 ``` The `reset` method resets the exhausted data source, allowing it to iterate from the beginning: ``` source.reset() ``` ### Export training data The `get_training_data` method returns all data used for training in one call. The returned data is a pandas DataFrame: ``` train_df = source.get_training_data() print(train_df.head(5).to_string()) y date_random date_non_unique 年月日 0 high 1949-01-27 1950-10-01 1949-01-01 1 high 1949-02-03 1959-04-01 1949-02-01 2 low 1949-03-28 1954-05-01 1949-03-01 3 high 1949-04-07 1951-09-01 1949-04-01 4 high 1949-05-16 1951-03-01 1949-05-01 ``` ### Export combined data The `get_data` method returns `combined_data`, which includes merged scoring data, predictions, and matched actuals. This [Metric Evaluator](https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/dmm-metric-evaluator.html) uses this method as the main data export method. ``` df, chunk_id_1 = source.get_data() print(df.head(5).to_string()) print(f"chunk id: {chunk_id_1}") timestamp predictions date_non_unique date_random association_id 年月日 predicted_class label actuals 0 2023-09-13 11:02:51.248000+00:00 0.302218 1950-10-01 1949-01-27 1 1949-01-01 high low 0 1 2023-09-13 11:02:51.252000+00:00 0.418649 1959-04-01 1949-02-03 2 1949-02-01 high low 0 2 2023-09-13 11:02:51.459000+00:00 0.360653 1954-05-01 1949-03-28 3 1949-03-01 high low 1 3 2023-09-13 11:02:51.459000+00:00 0.372273 1951-09-01 1949-04-07 4 1949-04-01 high low 0 4 2023-09-13 11:02:51.664000+00:00 0.408388 1951-03-01 1949-05-16 5 1949-05-01 high low 0 chunk id: 0 ``` The `get_all_data` returns all combined data available for that source object in a single DataFrame: ``` df = source.get_all_data() ``` ## Configure a DataRobot batch deployment source The `BatchDataRobotSource` interface is for batch deployments. The following example initializes `BatchDataRobotSource` with all parameters: ``` from dmm.data_source import BatchDataRobotSource source = BatchDataRobotSource( base_url=DATAROBOT_ENDPOINT, token=DATAROBOT_API_TOKEN, client=None, deployment_id=deployment_id, model_id=model_id, batch_ids=batch_ids, max_rows=10000, delete_exports=False, use_cache=False, ) ``` The parameters for this method are analogous to those for `DataRobotSource`. The most important difference is that instead of the time range (start and end), you must provide batch IDs. In addition, a batch source doesn't support actuals export. The `get_prediction_data` method returns a chunk of prediction data with the appropriate chunk ID; the returned data chunk is a pandas DataFrame with the number of rows respecting the `max_rows` parameter. This method returns data until the data source is exhausted. ``` prediction_df_1, prediction_chunk_id_1 = source.get_prediction_data() print(prediction_df_1.head(5).to_string()) print(f"chunk id: {prediction_chunk_id_1}") AGE B CHAS CRIM DIS batch_id DR_RESERVED_BATCH_NAME timestamp INDUS LSTAT MEDV NOX PTRATIO RAD RM TAX ZN id 0 65.2 396.90 0 0.00632 4.0900 batch1 2023-06-23 09:47:47.060000+00:00 2.31 4.98 24.0 0.538 15.3 1 6.575 296 18.0 1 1 78.9 396.90 0 0.02731 4.9671 batch1 2023-06-23 09:47:47.060000+00:00 7.07 9.14 21.6 0.469 17.8 2 6.421 242 0.0 2 2 61.1 392.83 0 0.02729 4.9671 batch1 2023-06-23 09:47:47.060000+00:00 7.07 4.03 34.7 0.469 17.8 2 7.185 242 0.0 3 3 45.8 394.63 0 0.03237 6.0622 batch1 2023-06-23 09:47:47.060000+00:00 2.18 2.94 33.4 0.458 18.7 3 6.998 222 0.0 4 4 54.2 396.90 0 0.06905 6.0622 batch1 2023-06-23 09:47:47.060000+00:00 2.18 5.33 36.2 0.458 18.7 3 7.147 222 0.0 5 chunk id: 0 prediction_df = source.get_all_prediction_data() source.reset() df, chunk_id_1 = source.get_data() ``` The `get_training_data` method returns all data used for training in one call. The returned data is a pandas DataFrame: ``` train_df = source.get_training_data() ``` ## Configure a DataFrame source If you aren't exporting data directly from DataRobot, and instead have it downloaded locally (for example), you can load the dataset into `DataFrameSource`. The `DataFrameSource` method wraps any `pd.DataFrame` to create a library-compatible source. This is the easiest way to interact with the library when bringing your own data: ``` source = DataFrameSource( df=pd.read_csv("./data_hour_of_week.csv"), max_rows=10000, timestamp_col="date" ) df, chunk_id_1 = source.get_data() print(df.head(5).to_string()) print(f"chunk id: {chunk_id_1}") date y 0 1959-12-31 23:59:57 -0.183669 1 1960-01-01 01:00:02 0.283993 2 1960-01-01 01:59:52 0.020663 3 1960-01-01 03:00:14 0.404304 4 1960-01-01 03:59:58 1.005252 chunk id: 0 ``` In addition, it is possible to create new data source definitions. To define a new data source, you can customize and implement the `DataSourceBase` interface. ## Set the TimeBucket The `TimeBucket` class enumeration (enum) defines the required data aggregation granularity over time. By default, `TimeBucket` is set to `TimeBucket.ALL`. You can specify any of the following values: `SECOND`, `MINUTE`, `HOUR`, `DAY`, `WEEK`, `MONTH`, `QUARTER`, or `ALL`. To change the `TimeBucket` value, use the `init` method: `source.init(time_bucket)`: ``` # Generate a dummy DataFrame with 2 rows per time bucket (Hour in this scenario) test_df = gen_dataframe_for_accuracy_metric( nr_rows=10, rows_per_time_bucket=2, prediction_value=1, with_actuals=True, with_predictions=True, time_bucket=TimeBucket.HOUR, ) print(test_df) timestamp predictions actuals 0 01/06/2005 13:00:00.000000 1 0.999 1 01/06/2005 13:00:00.000000 1 0.999 2 01/06/2005 14:00:00.000000 1 0.999 3 01/06/2005 14:00:00.000000 1 0.999 4 01/06/2005 15:00:00.000000 1 0.999 5 01/06/2005 15:00:00.000000 1 0.999 6 01/06/2005 16:00:00.000000 1 0.999 7 01/06/2005 16:00:00.000000 1 0.999 8 01/06/2005 17:00:00.000000 1 0.999 9 01/06/2005 17:00:00.000000 1 0.999 # Use DataFrameSource and load created DataFrame source = DataFrameSource( df=test_df, max_rows=10000, timestamp_col="timestamp", ) # Init source with the selected TimeBucket source.init(TimeBucket.HOUR) df, _ = source.get_data() print(df) timestamp predictions actuals 0 01/06/2005 13:00:00.000000 1 0.999 1 01/06/2005 13:00:00.000000 1 0.999 df, _ = source.get_data() print(df) timestamp predictions actuals 2 01/06/2005 14:00:00.000000 1 0.999 3 01/06/2005 14:00:00.000000 1 0.999 source.init(TimeBucket.DAY) df, _ = source.get_data() print(df) timestamp predictions actuals 0 01/06/2005 13:00:00.000000 1 0.999 1 01/06/2005 13:00:00.000000 1 0.999 2 01/06/2005 14:00:00.000000 1 0.999 3 01/06/2005 14:00:00.000000 1 0.999 4 01/06/2005 15:00:00.000000 1 0.999 5 01/06/2005 15:00:00.000000 1 0.999 6 01/06/2005 16:00:00.000000 1 0.999 7 01/06/2005 16:00:00.000000 1 0.999 8 01/06/2005 17:00:00.000000 1 0.999 9 01/06/2005 17:00:00.000000 1 0.999 ``` The returned data chunks follow the selected `TimeBucket`. This is helpful in the `MetricEvaluator`. In addition to `TimeBucket`, the source respects the `max_rows` parameter when generating data chunks; for example, using the same dataset as in the example above (but with `max_rows` set to `3`): ``` source = DataFrameSource( df=test_df, max_rows=3, timestamp_col="timestamp", ) source.init(TimeBucket.DAY) df, chunk_id = source.get_data() print(df) timestamp predictions actuals 0 01/06/2005 13:00:00.000000 1 0.999 1 01/06/2005 13:00:00.000000 1 0.999 2 01/06/2005 14:00:00.000000 1 0.999 ``` In `DataRobotSource`, you can specify the `TimeBucket` and `max_rows` parameters for all export types except training data export, which is returned in one chunk. ## Provide additional DataRobot deployment properties The `Deployment` class is a helper class, providing access to relevant deployment properties. This class is used inside `DataRobotSource` to select the appropriate workflow to work with data. ``` import datarobot as dr from dmm.data_source.datarobot.deployment import Deployment DataRobotClient() deployment = Deployment(deployment_id=deployment_id) deployment_type = deployment.type() target_column = deployment.target_column() positive_class_label = deployment.positive_class_label() negative_class_label = deployment.negative_class_label() prediction_threshold = deployment.prediction_threshold() . . . ``` --- # Use the DR Custom Metrics module URL: https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/dmm-dr-custom-metric.html > The DR Custom Metrics module facilitates better synchronization with existing metrics in DataRobot. The DR Custom Metrics module facilitates better synchronization with existing metrics in DataRobot. The logic of this module is based on unique names for custom metrics, allowing you to operate on metrics without knowing their IDs. Using this solution, you can define the metric earlier (e.g., before creating the deployment) and synchronize it with DataRobot at the appropriate time. The `DRCustomMetric` class allows you to create new or fetch existing metrics from DataRobot. You can provide custom metrics configuration through YAML, JSON, or a dict. The configuration contains metadata describing the custom metric. The `DRCustomMetric` class contains two methods: - DRCustomMetric.sync(): Retrieves information about existing custom metrics in DataRobot. If a metric is defined locally but not in DataRobot, it is created in DataRobot. - DRCustomMetric.report(): Reports a single value based on a unique name. ``` dr_cm = DRCustomMetric( dr_client=client, deployment_id=deployment_id, model_package_id=model_package_id ) metric_config_yaml = f""" customMetrics: - name: new metric description: metric description type: average timeStep: hour units: count directionality: lowerIsBetter isModelSpecific: yes baselineValue: 0 """ dr_cm.set_config(config_yaml=metric_config_yaml) dr_cm.sync() dr_cm.get_dr_custom_metrics() > [{"name": "existing metric", "id": "65ef19410239ff8015f05a94", ...}, > {"name": "new metric", "id": "65ef197ce5d7b2176ceecf3a", ...}] dr_cm.report_value("existing metric", 1) dr_cm.report_value("new metric", 9) ``` --- # Configure an environment for DMM URL: https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/dmm-environment-setup.html > Configure an environment to develop custom metrics with the DataRobot Model Metrics library. There are two primary ecosystems where you can develop a custom metric with the DataRobot Model Metrics library (DMM): 1. Within the DataRobot application (via a notebook or a scheduled job). 2. In a local development environment. > [!NOTE] Environment considerations > Installing Python modules: > > When running DMM from within the DataRobot application, the ecosystem is pre-configured with all required Python modules. > When running DMM locally, you need to install the > dmm > module. This automatically updates the Python environment with all required modules. > > Setting DMM parameters: > > When running DMM from within the DataRobot application, the parameters are set through environment variables. > When running DMM locally, it is recommended that you pass values using arguments to set these parameters, rather than setting environment variables. ## Initialize the environment The `CustomMetricArgumentParser` class wraps the standard `argparser.ArgumentParser`. This class provides convenience functions to allow reading values from the environment or normal argument parsing. When `CustomMetricArgumentParser.parse_args()` is called, it checks for missing values. The `log_manager` provides a set of functions to help with logging. The DMM library and the DataRobot public API client use standard Python `logging` primitives. A complete list of log classes with their current levels is available using `get_log_levels()`. The `initialize_loggers()` function initializes all loggers: ``` 2024-08-09 02:19:50 PM - dmm.data_source.datarobot_source - INFO - fetching the next predictions dataframe... 2024-07-15 00:00:00 - 2024-08-09 14:19:46.643722 2024-08-09 02:19:56 PM - urllib3.connectionpool - DEBUG - https://app.datarobot.com:443 "POST /api/v2/deployments/66a90a711zd81645df8c469c/predictionDataExports/ HTTP/1.1" 202 368 ``` The following snippet shows how to set up your runtime environment using the previously mentioned classes: ``` import sys from dmm import CustomMetricArgumentParser from dmm.log_manager import initialize_loggers parser = CustomMetricArgumentParser(description="My new custom metric") parser.add_base_args() # adds standard arguments # Add more with standard ArgumentParser primitives, or some convenience functions such as add_environment_arg() # Parse the program arguments (if any) to an argparse.Namespace. args = parser.parse_args(sys.argv[1:]) # Initialize the logging based on the 'LOG' environment variable, or the --log option initialize_loggers(args.log) ``` The standard/base arguments include the following: | Argument | Description | | --- | --- | | BASE_URL | The URL of the public API. | | API_KEY | The API token used for authentication to the server located at BASE_URL. | | DEPLOYMENT_ID | The deployment ID from the application. | | CUSTOM_METRIC_ID | The custom metric ID from the application. | | DRY_RUN | The flag to indicate whether to report the custom metric result to the deployment. With the DRY_RUN runtime parameter set to 1, the run is a test run and does not report metric data. | | START_TS | The start of the time range for metric calculation. | | END_TS | The end of the time for metric calculation. | | MAX_ROWS | The maximum number of prediction rows to process. | | LOG | The initialization of logging—defaults to setting all dmm and datarobot modules to WARNING. | The following is an example of the help provided using `CustomMetricArgumentParser`: ``` (model-runner) $ python3 custom.py --help usage: custom.py [-h] [--api-key KEY] [--base-url URL] [--deployment-id ID] [--custom-metric-id ID] [--dry-run] [--start-ts TIMESTAMP] [--end-ts TIMESTAMP] [--max-rows ROWS] [--required] [--log [[NAME:]LEVEL ...]] My new custom metric optional arguments: -h, --help show this help message and exit --api-key KEY API key used to authenticate to server. Settable via 'API_KEY', required. --base-url URL URL for server. Settable via 'BASE_URL' (default: https://staging.datarobot.com/api/v2), required. --deployment-id ID Deployment ID. Settable via 'DEPLOYMENT_ID' (default: None), required. --custom-metric-id ID Custom metric ID. Settable via 'CUSTOM_METRIC_ID' (default: None), required. --dry-run Dry run. Settable via 'DRY_RUN' (default: False). --start-ts TIMESTAMP Start timestamp. Settable with 'START_TS', or 'LAST_SUCCESSFUL_RUN_TS' (when not dry run). Default is 2024-08-08 14:27:55.493027 --end-ts TIMESTAMP End timestamp. Settable with 'END_TS' or 'CURRENT_RUN_TS'. Default is 2024-08-09 14:27:55.493044. --max-rows ROWS Maximum number of rows. Settable via 'MAX_ROWS' (default: 100000). --required List the required properties and exit. --log [[NAME:]LEVEL ...] Logging level list. Settable via 'LOG' (default: WARNING). (model-runner) $ ``` ## Using the save_to_csv() utility During development, it is common to run your code over the same data multiple times to see how changes impact the results. The `save_to_csv()` utility allows you to save your results to a CSV file, so you can compare the results between successive runs on the same data. --- # Create a hosted custom metric from a template with code URL: https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/dmm-hosted-custom-metrics-quickstart.html Hosted custom metrics run user-provided code on DataRobot infrastructure to calculate your organization's customized metrics. DataRobot provides a variety of templates for common metrics. These metrics can be used as is, or as a starting point for user-provided metrics. In this tutorial, you create a hosted custom metric using the Python SDK. ## Prerequisites Before you start, import all objects used in this tutorial and initialize the DataRobot client: ``` import datarobot as dr from datarobot.enums import HostedCustomMetricsTemplateMetricTypeQueryParams from datarobot.models.deployment.custom_metrics import HostedCustomMetricTemplate, HostedCustomMetric, \ HostedCustomMetricBlueprint, CustomMetric, MetricTimestampSpoofing, ValueField, SampleCountField, BatchField from datarobot.models.registry import JobRun from datarobot.models.registry.job import Job from datarobot import Deployment from datarobot.models.runtime_parameters import RuntimeParameterValue from datarobot.models.types import Schedule DataRobotClient(token="", endpoint="") gen_ai_deployment_1 = Deployment.get('') ``` ## List hosted custom metrics templates Before creating a hosted custom metric from a template, retrieve the LLM metric template to use as the basis of the new metric. To do this, specify the `metric_type` and, because the deployments are LLM models handling Japanese text, search for the specific metric by name, limiting the search to `1` result. Store the result in templates for the next step. ``` templates = HostedCustomMetricTemplate.list( search="[JP] Character Count", metric_type=HostedCustomMetricsTemplateMetricTypeQueryParams.LLM, limit=1, offset=0, ) ``` ## Create a hosted custom metric in one step Locate the custom metric template to create a hosted custom metric. This method is a shortcut, combining two steps to create a new custom metric from the retrieved template: 1. Create a custom job for a hosted custom metric from the template retrieved in the previous step (stored in templates ). 2. Connect the hosted custom metric job to the deployment defined during the prerequisites step (stored in gen_ai_deployment_1 ). Because we are creating two objects, specify both the job name and custom metric name in addition to the template and deployment IDs. Additionally, define the job schedule and the runtime parameter overrides for the deployment. ``` hosted_custom_metric = HostedCustomMetric.create_from_template( template_id=templates[0].id, deployment_id=gen_ai_deployment_1.id, job_name="Hosted Custom Metric Character Count", custom_metric_name="Character Count", job_description="Hosted Custom Metric", custom_metric_description="LLM Character Count", baseline_value=10, timestamp=MetricTimestampSpoofing( column_name="timestamp", time_format="%Y-%m-%d %H:%M:%S", ), value = ValueField(column_name="value"), sample_count=SampleCountField(column_name='Sample Count'), batch=BatchField(column_name='Batch'), schedule=Schedule( day_of_week=[0], hour=['*'], minute=['*'], day_of_month=[12], month=[1], ), parameter_overrides=[RuntimeParameterValue(field_name='DRY_RUN', value="0", type="string")] ) ``` Once you create the hosted custom metric, initiate the manual run. ``` job_run = JobRun.create( job_id=hosted_custom_metric.custom_job_id runtime_parameter_values=[ RuntimeParameterValue(field_name='DRY_RUN', value="1", type="string"), RuntimeParameterValue(field_name='DEPLOYMENT_ID', value=gen_ai_deployment_1.id, type="deployment"), RuntimeParameterValue(field_name='CUSTOM_METRIC_ID', value=hosted_custom_metric.id, type="customMetric"), ] ) print(job_run.status) ``` ## Create a hosted custom metric in two steps You can also perform these steps manually, in sequence. This is useful if you want to edit the custom metric blueprint before attaching the custom job to the deployment. When you attach the job to the deployment, most settings are copied from the blueprint (unless you provide an override). To create the hosted custom metric manually, first, create a custom job from the template retrieved earlier (stored in `templates`). ``` job = Job.create_from_custom_metric_gallery_template( template_id=templates[0].id, name="Job created from template", description="Job created from template" ) ``` Next, retrieve the default blueprint, provided by the template, and edit it. ``` blueprint = HostedCustomMetricBlueprint.get(job.id) print(f"Original directionality: {blueprint.directionality}") ``` Then, update the parameters of the custom metric in the blueprint. ``` updated_blueprint = blueprint.update( directionality='lowerIsBetter', units='characters', type='gauge', time_step='hour', is_model_specific=False ) print(f"Updated directionality: {updated_blueprint.directionality}") ``` Now create the hosted custom metric. As in the shortcut method, you can provide the job schedule, runtime parameter overrides, and custom metric parameters specific to this deployment. ``` another_hosted_custom_metric = HostedCustomMetric.create_from_custom_job( custom_job_id=job.id, deployment_id=gen_ai_deployment_1.id, name="Custom metric created in 2 steps", ) ``` After creating and configuring the metric, verify that the changes to the blueprint are reflected in the custom metric. ``` another_custom_metric = CustomMetric.get(custom_metric_id=another_hosted_custom_metric.id, deployment_id=gen_ai_deployment_1.id) print(f"Directionality of another custom metric: {another_custom_metric.directionality}") ``` Finally, create a manual job run for the custom metric job. ``` job_run = JobRun.create( job_id=job.id, runtime_parameter_values=[ RuntimeParameterValue(field_name='DRY_RUN', value="1", type="string"), RuntimeParameterValue(field_name='DEPLOYMENT_ID', value=gen_ai_deployment_1.id, type="deployment"), RuntimeParameterValue(field_name='CUSTOM_METRIC_ID', value=another_hosted_custom_metric.id, type="customMetric"), ] ) print(job_run.status) ``` ## List hosted custom metrics associated with a job To list all hosted custom metrics associated with a custom job, use the following code: ``` hosted_custom_metrics = HostedCustomMetric.list(deployment_id=hosted_custom_metric.custom_job_id) for metric in hosted_custom_metrics: print(metric.name) ``` ## Delete hosted custom metrics You can delete the hosted custom metric, removing the custom metric from the deployment but keeping the job, allowing you to create the metric for another deployment. ``` hosted_custom_metric.delete() another_hosted_custom_metric.delete() ``` If necessary, you can delete the entire custom job. If there are any custom metrics associated with that job, they are also deleted. ``` job.delete() ``` --- # Calculate metric values URL: https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/dmm-metric-evaluator.html > To calculate custom metric values, you can use MetricEvaluator to calculate metric values over time, BatchMetricEvaluator to calculate metric values per batch, or IndividualMetricEvaluator to evaluate metrics without data aggregation. To calculate custom metric values, you can use `MetricEvaluator` to calculate metric values over time, `BatchMetricEvaluator` to calculate metric values per batch, or `IndividualMetricEvaluator` to evaluate metrics without data aggregation. ## Evaluate metrics The `MetricEvaluator` class calculates metric values over time using the selected source. This class is used to "stream" data through the metric object, generating metric values. Initialize the `MetricEvaluator` with the following mandatory parameters: ``` from dmm import MetricEvaluator, TimeBucket from dmm import DataRobotSource from dmm.metric import MedianAbsoluteError source = DataRobotSource( deployment_id=DEPLOYMENT_ID, start=datetime.utcnow() - timedelta(weeks=1), end=datetime.utcnow(), ) metric = MedianAbsoluteError() metric_evaluator = MetricEvaluator(metric=metric, source=source, time_bucket=TimeBucket.MINUTE) ``` To use `MetricEvaluator`, create a metric class implementing the `MetricBase` interface and a source implementing `DataSourceBase`. Then, specify the level of aggregation granularity. Initialize `MetricEvaluator` with all parameters: ``` from dmm import ColumnName, MetricEvaluator, TimeBucket metric_evaluator = MetricEvaluator( metric=metric, source=source, time_bucket=TimeBucket.HOUR, prediction_col=ColumnName.PREDICTIONS, actuals_col=ColumnName.ACTUALS, timestamp_col=ColumnName.TIMESTAMP, filter_actuals=False, filter_predictions=False, filter_scoring_data=False, segment_attribute=None, segment_value=None, ) ``` | Parameter | Description | | --- | --- | | metric: Union[str, MetricBase, List[str], List[MetricBase]] | If a string or list of strings is passed, MetricEvaluator will look for matched Sklearn metrics. If a metric or a list of objects is passed, they must implement the MetricBase interface. | | source: DataSourceBase | The source to pull the data from, DataRobotSource or DataFrameSource or other sources that implement the DataSourceBase interface. | | time_bucket: TimeBucket | The time-bucket size to use for evaluating metrics, determining the granularity of aggregation. | | prediction_col: Optional[str] | The name of the column that contains predictions. | | actuals_col: Optional[str] | The name of the column that contains actuals. | | timestamp_col: Optional[str] | The name of the column that contains timestamps. | | filter_actuals: Optional[bool] | Whether the metric evaluator removes missing actuals values before scoring. True removes missing actuals; the default value is False. | | filter_predictions: Optional[bool] | Whether the metric evaluator removes missing predictions values before scoring. True removes missing predictions; the default value is False. | | filter_scoring_data: Optional[bool] | Whether the metric evaluator removes missing scoring values before scoring. True removes missing scoring values; the default value is False. | | segment_attribute: Optional[str] | The name of the column with segment values. | | segment_value: Optional[Union[str or List[str]]] | A single value or a list of values of the segment attribute to segment on. | The `score` method returns a metric aggregated as defined by `TimeBucket`. The output returned as a pandas DataFrame contains the results per time bucket for all data from the source. ``` source = DataRobotSource( deployment_id=DEPLOYMENT_ID, start=datetime.utcnow() - timedelta(hours=3), end=datetime.utcnow(), ) metric = LogLossFromSklearn() me = MetricEvaluator(metric=metric, source=source, time_bucket=TimeBucket.HOUR) aggregated_metric_per_time_bucket = me.score() print(aggregated_metric_per_time_bucket.to_string()) timestamp samples log_loss 0 2023-09-14 13:29:48.065000+00:00 499 0.539315 1 2023-09-14 14:01:51.484000+00:00 499 0.539397 # we can see the evaluator's statistics stats = me.stats() print(stats) total rows: 998, score calls: 2, reduce calls: 2 ``` To pass more than one metric at a time, do the following: ``` metrics = [LogLossFromSklearn(), AsymmetricError(), RocAuc()] me = MetricEvaluator(metric=metric, source=source, time_bucket=TimeBucket.HOUR) aggregated_metric_per_time_bucket = me.score() stats = me.stats() print(aggregated_metric_per_time_bucket.to_string()) print(stats) timestamp samples log_loss Asymmetric Error roc_auc_score 0 2023-09-14 13:29:48.065000+00:00 499 0.539315 0.365571 0.787030 1 2023-09-14 14:01:51.484000+00:00 499 0.539397 0.365636 0.786837 total rows: 998, score calls: 6, reduce calls: 6 ``` For your data, provide the names of the columns to evaluate: ``` test_df = gen_dataframe_for_accuracy_metric( nr_rows=5, rows_per_time_bucket=1, prediction_value=1, time_bucket=TimeBucket.DAY, prediction_col="my_pred_col", actuals_col="my_actuals_col", timestamp_col="my_timestamp_col" ) print(test_df) my_timestamp_col my_pred_col my_actuals_col 0 01/06/2005 13:00:00.000000 1 0.999 1 02/06/2005 13:00:00.000000 1 0.999 2 03/06/2005 13:00:00.000000 1 0.999 3 04/06/2005 13:00:00.000000 1 0.999 4 05/06/2005 13:00:00.000000 1 0.999 source = DataFrameSource( df=test_df, max_rows=10000, timestamp_col="timestamp", ) metric = LogLossFromSklearn() me = MetricEvaluator(metric=metric, source=source, time_bucket=TimeBucket.DAY, prediction_col="my_pred_col", actuals_col="my_actuals_col", timestamp_col="my_timestamp_col" ) aggregated_metric_per_time_bucket = me.score() ``` ### Configure data filtering If data is missing, use filtering flags. In the following example, the data is missing actuals. In this scenario without a flag, an exception is raised: ``` test_df = gen_dataframe_for_accuracy_metric( nr_rows=10, rows_per_time_bucket=5, prediction_value=1, time_bucket=TimeBucket.HOUR, ) test_df["actuals"].loc[2] = None test_df["actuals"].loc[5] = None print(test_df) timestamp predictions actuals 0 01/06/2005 13:00:00.000000 1 0.999 1 01/06/2005 13:00:00.000000 1 0.999 2 01/06/2005 13:00:00.000000 1 NaN 3 01/06/2005 13:00:00.000000 1 0.999 4 01/06/2005 13:00:00.000000 1 0.999 5 01/06/2005 14:00:00.000000 1 NaN 6 01/06/2005 14:00:00.000000 1 0.999 7 01/06/2005 14:00:00.000000 1 0.999 8 01/06/2005 14:00:00.000000 1 0.999 9 01/06/2005 14:00:00.000000 1 0.999 source = DataFrameSource(df=test_df) metric = MedianAbsoluteError() me = MetricEvaluator(metric=metric, source=source, time_bucket=TimeBucket.HOUR) aggregated_metric_per_time_bucket = me.score() "ValueError: Could not apply metric median_absolute_error, make sure you are passing the right data (see the sklearn docs). The error message was: Input contains NaN." ``` Compare the previous result with the result when you enable the `filter_actuals` flag: ``` me = MetricEvaluator(metric=metric, source=source, time_bucket=TimeBucket.HOUR, filter_actuals=True) aggregated_metric_per_time_bucket = me.score() "removed 1 rows out of 5 in the data chunk before scoring, due to missing values in ['actuals'] data" "removed 1 rows out of 5 in the data chunk before scoring, due to missing values in ['actuals'] data" print(aggregated_metric_per_time_bucket.to_string()) timestamp samples median_absolute_error 0 01/06/2005 13:00:00.000000 4 0.001 1 01/06/2005 14:00:00.000000 4 0.001 ``` Using the `filter_actuals`, `filter_predictions`, and `filter_scoring_data` flags, you can filter out missing values from the data before calculating the metric. By default, these flags are set to `False`. If all data needed to calculate the metric is missing from the data chunk, the data chunk is skipped with the appropriate log: ``` test_df = gen_dataframe_for_accuracy_metric( nr_rows=4, rows_per_time_bucket=2, prediction_value=1, time_bucket=TimeBucket.HOUR, ) test_df["actuals"].loc[0] = None test_df["actuals"].loc[1] = None print(test_df) timestamp predictions actuals 0 01/06/2005 13:00:00.000000 1 NaN 1 01/06/2005 13:00:00.000000 1 NaN 2 01/06/2005 14:00:00.000000 1 0.999 3 01/06/2005 14:00:00.000000 1 0.999 source = DataFrameSource(df=test_df) metric = MedianAbsoluteError() me = MetricEvaluator(metric=metric, source=source, time_bucket=TimeBucket.HOUR, filter_actuals=True) aggregated_metric_per_time_bucket = me.score() "removed 2 rows out of 2 in the data chunk before scoring, due to missing values in ['actuals'] data" "data chunk is empty, skipping scoring..." print(aggregated_metric_per_time_bucket.to_string()) timestamp samples median_absolute_error 1 01/06/2005 14:00:00.000000 2 0.001 ``` ### Perform segmented analysis Perform segmented analysis by defining the `segment_attribute` and each `segment_value`: ``` metrics = LogLossFromSklearn() me = MetricEvaluator(metric=metric, source=source, time_bucket=TimeBucket.HOUR, segment_attribute="insulin", segment_value="Down", ) aggregated_metric_per_time_bucket = me.score() print(aggregated_metric_per_time_bucket.to_string()) timestamp samples log_loss [Down] 0 2023-09-14 13:29:49.737000+00:00 49 0.594483 1 2023-09-14 14:01:52.437000+00:00 49 0.594483 # passing more than one segment value me = MetricEvaluator(metric=metric, source=source, time_bucket=TimeBucket.HOUR, segment_attribute="insulin", segment_value=["Down", "Steady"], ) aggregated_metric_per_time_bucket = me.score() print(aggregated_metric_per_time_bucket.to_string()) timestamp samples log_loss [Down] log_loss [Steady] 0 2023-09-14 13:29:48.502000+00:00 199 0.594483 0.515811 1 2023-09-14 14:01:51.758000+00:00 199 0.594483 0.515811 # passing more than one segment value and more than one metric me = MetricEvaluator(metric=[LogLossFromSklearn(), RocAuc()], source=source, time_bucket=TimeBucket.HOUR, segment_attribute="insulin", segment_value=["Down", "Steady"], ) aggregated_metric_per_time_bucket = me.score() print(aggregated_metric_per_time_bucket.to_string()) timestamp samples log_loss [Down] log_loss [Steady] roc_auc_score [Down] roc_auc_score [Steady] 0 2023-09-14 13:29:48.502000+00:00 199 0.594483 0.515811 0.783333 0.826632 1 2023-09-14 14:01:51.758000+00:00 199 0.594483 0.515811 0.783333 0.826632 ``` ## Evaluate metrics with aggregation-per-batch The `BatchMetricEvaluator` class uses aggregation-per-batch instead of aggregation-over-time. For batches, don't define `TimeBucket`: ``` from dmm.batch_metric_evaluator import BatchMetricEvaluator from dmm.data_source.datarobot_source import BatchDataRobotSource from dmm.metric import MissingValuesFraction source = BatchDataRobotSource( deployment_id=DEPLOYMENT_ID, batch_ids=BATCH_IDS, model_id=MODEL_ID, ) feature_name = 'RAD' metric = MissingValuesFraction(feature_name=feature_name) missing_values_fraction_evaluator = BatchMetricEvaluator(metric=metric, source=source) aggregated_metric_per_batch = missing_values_fraction_evaluator.score() print(aggregated_metric_per_batch.to_string()) batch_id samples Missing Values Fraction 0 506 0.0 1 506 0.0 2 506 0.0 ``` ## Evaluate metrics without data aggregation The `IndividualMetricEvaluator` class is used to evaluate metrics without data aggregation. It performs metric calculations on all exported data and returns a list of individual results. This evaluator allows submitting individual data points with a corresponding association ID, which is useful for cases when you want to visualize your metric results alongside predictions and actuals. To use this evaluator with custom metrics, provide a `score()` method that contains, among others, the following parameters: `timestamps` and `association_ids`. ``` from itertools import zip_longest from typing import List from datetime import datetime from datetime import timedelta from dmm import CustomMetric from dmm import DataRobotSource from dmm import SingleMetricResult from dmm.individual_metric_evaluator import IndividualMetricEvaluator from dmm.metric import LLMMetricBase from nltk import sent_tokenize import numpy as np import pandas as pd source = DataRobotSource( deployment_id=DEPLOYMENT_ID, start=datetime.utcnow() - timedelta(weeks=1), end=datetime.utcnow(), ) custom_metric = CustomMetric.from_id() class SentenceCount(LLMMetricBase): """ Calculates the total number of sentences created while working with the LLM model. Returns the sum of the number of sentences from prompts and completions. """ def __init__(self): super().__init__( name=custom_metric.name, description="Calculates the total number of sentences created while working with the LLM model.", need_training_data=False, ) self.prompt_column = "promptColumn" def score( self, scoring_data: pd.DataFrame, predictions: np.ndarray, timestamps: np.ndarray, association_ids: np.ndarray, **kwargs, ) -> List[SingleMetricResult]: if self.prompt_column not in scoring_data.columns: raise ValueError( f"Prompt column {self.prompt_column} not found in the exported data, " f"modify 'PROMPT_COLUMN' runtime parameter" ) prompts = scoring_data[self.prompt_column].to_numpy() sentence_count = [] for prompt, completion, ts, a_id in zip_longest( prompts, predictions, timestamps, association_ids ): if not isinstance(prompt, str) or not isinstance(completion, str): continue value = len(sent_tokenize(prompt)) + len(sent_tokenize(completion)) sentence_count.append( SingleMetricResult(value=value, timestamp=ts, association_id=a_id) ) return sentence_count sentence_count_evaluator = IndividualMetricEvaluator( metric=SentenceCount(), source=source, ) metric_results = sentence_count_evaluator.score() ``` --- # Model Metrics URL: https://docs.datarobot.com/en/docs/api/code-first-tools/dr-model-metrics/index.html > Learn how to construct and modify DataRobot blueprints through a programmatic interface. The `datarobot-model-metrics` (DMM) library provides the tools necessary to compute model metrics over time and produce aggregated metrics. It provides a framework to perform the following operations: | Topic | Description | | --- | --- | | Configure an environment for DMM | Configure an environment to develop custom metrics with the DataRobot Model Metrics library. | | Configure data sources | Connect to DataRobot to fetch selected data from the DataRobot platform. | | Define custom metrics | Define custom metrics using the provided default classes. | | Calculate metric values | Calculate custom metric values over time, by batch, or without data aggregation. | | Use the DR Custom Metrics module | Facilitate synchronization with existing metrics in DataRobot. | | Notebook: Create a hosted custom metric from a template with code | Follow a tutorial to create a hosted custom metric using the Python SDK. | ## Installation The DMM library is published to [PyPI](https://pypi.org/project/datarobot-model-metrics/). To install it, run the following command: ``` pip install datarobot-model-metrics ``` --- # Test custom models locally URL: https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-local-test.html > Use the DataRobot Model Runner tool (DRUM) to test and verify a Python, R, or Java custom model locally, before you upload it to DataRobot. > [!NOTE] Availability information > To access the DataRobot Model Runner tool, contact your DataRobot representative. The DataRobot Model Runner is a tool that allows you to test Python, R, and Java custom models locally. The test verifies that a custom model can successfully run and make predictions before you [upload it to DataRobot](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-create-custom-model.html). However, this testing is only for development purposes. DataRobot recommends that you also test custom model you wish to deploy in the [Workshop](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-test-custom-model.html) after uploading it. Before proceeding, reference the guidelines for [setting up a custom model or environment folder](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/index.html). > [!NOTE] Note > The DataRobot Model Runner tool supports Python, R, and Java custom models. Reference the [DRUM readme](https://github.com/datarobot/datarobot-user-models?tab=readme-ov-file#datarobot-user-models) for details about additional functionality, including: - Autocompletion - Custom hooks - Performance tests - Running models with a prediction server - Running models inside a Docker container ### Model requirements In addition to the required folder contents, DRUM requires the following for your serialized model: - Regression models must return a single floating point per row of prediction data. - Binary classification models must return two floating point values that sum to 1.0 per row of prediction data. - The first value must be the positive class probability, and the second the negative class probability. - There is a single pkl/pth/h5 file present. ## Run tests with the DataRobot CM Runner Use the following commands to execute local tests for your custom model: ``` # List all possible arguments drum --help ``` ``` # Test a custom binary classification model drum score -m ~/custom_model/ --input [--positive-class-label ] [--negative-class-label ] [--output ] [--verbose] # Use --verbose for a more detailed output. Make batch predictions with a custom binary classification model. Optionally, specify an output file. Otherwise, predictions are returned to the command line. ``` ``` # Example: Test a custom binary classification model drum score -m ~/custom_model/ --input 10k.csv --positive-class-label yes --negative-class-label no --output 10k-results.csv --verbose ``` ``` # Test a custom regression model drum score -m ~/custom_model/ --input [--output ] [--verbose] # Use --verbose for a more detailed output. Make batch predictions with a custom regression model. Optionally, specify an output file. Otherwise, predictions are returned to the command line. ``` ``` # Example: Test a custom regression model drum score -m ~/custom_model/ --input fast-iron.csv --verbose # This is an example that does not include an output command, so the prediction results return in the command line. ``` --- # Custom model components URL: https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-components.html > Describes custom model support and how to structure a custom model's files. To create and upload a custom model, you need to define two components—the model’s content and an environment where the model’s content will run: - Themodel contentis code written in Python or R. To be correctly parsed by DataRobot, the code must follow certain criteria. The model artifact's structure should match the library used by the model. In addition, it should use the appropriatecustom hooksfor Python, R, and Java models. (Optional) You can add files that will be uploaded and used together with the model’s code (for example, you might want to add a separate file with a dictionary if your custom model contains text preprocessing). - Themodel environmentis defined using a Docker file and additional files that will allow DataRobot to build an image where the model will run. There are a variety of built-in environments; you only need to build your own environment when you need to install Linux packages. For more detailed information, see the section oncustom model environments. At a high level, the steps to define a custom model with these components include: 1. Define and test model content locally (i.e., on your computer). 2. (Optional) Create a container environment where the model will run. 3. Upload the model content and environment (if applicable) into DataRobot. ## Model content To define a custom model, create a local folder containing the files listed in the table below (detailed descriptions follow the table). > [!TIP] Tip > To ensure your assembled custom model folder has the correct contents, you can find examples of these files in the [DataRobot model template repository](https://github.com/datarobot/datarobot-user-models/tree/master/model_templates) on GitHub. | File | Description | Required | | --- | --- | --- | | Model artifact fileorcustom.py/custom.R file | Provide a model artifact and/or a custom code file. Model artifact: a serialized model artifact with a file extension corresponding to the chosen environment language.Custom code: custom capabilities implemented with hooks (or functions) that enable DataRobot to run the code and integrate it with other capabilities. | Yes | | model-metadata.yaml | A file describing a model's metadata, including input/output data requirements and runtime parameters. You can supply a schema that can then be used to validate the model when building and training a blueprint. A schema lets you specify whether a custom model supports or outputs: Certain data typesMissing valuesSparse dataA certain number of columns | Required when a custom model outputs non-numeric data. If not provided, a default schema is used. | | requirements.txt | A list of Python or R packages to add to the base environment. This list pre-installs Python or R packages that the custom model is using but are not a part of the base environment | No | | Additional files | Other files used by the model (for example, a file that defines helper functions used inside custom.py). | No | **requirements.txt Python example:** For Python, provide a list of packages with their versions (1 package per row). For example: ``` numpy>=1.16.0, <1.19.0 pandas==1.1.0 scikit-learn==0.23.1 lightgbm==3.0.0 gensim==3.8.3 sagemaker-scikit-learn-extension==1.1.0 ``` **requirements.txt R example:** For R, provide a list of packages without versions (1 package per row). For example: ``` dplyr stats ``` ### Model code To define a custom model using DataRobot’s framework, your custom model should include a model artifact corresponding to the chosen environment language, custom code in a `custom.py` (for Python models) or `custom.R` (for R models) file, or both. If you provide only the custom code (without a model artifact), you must use the `load_model` hook. A hook is a function called by the custom model framework during a specific time in the custom model lifecycle. The following hooks can be used in your custom code: > [!WARNING] Include all required custom model code in hooks > Custom model hooks are callbacks passed to the custom model. All code required by the custom model must be in a custom model hook—the custom model can't access any code provided outside a defined custom model hook. In addition, you can't modify the input arguments of these hooks as they are predefined. | Hook (Function) | Unstructured/Structured | Purpose | | --- | --- | --- | | init() | Both | Initialize the model run by loading model libraries and reading model files. This hook is executed only once at the beginning of a run. | | load_model() | Both | Load all supported and trained objects from multiple artifacts, or load a trained object stored in an artifact with a format not natively supported by DataRobot. This hook is executed only once at the beginning of a run. | | read_input_data() | Structured | Customize how the model reads data; for example, with encoding and missing value handling. | | transform() | Structured | Define the logic used by custom transformers and estimators to generate transformed data. | | score() | Structured | Define the logic used by custom estimators to generate predictions. | | score_unstructured() | Unstructured | Define the output of a custom estimator and returns predictions on input data. Do not use this hook for transform models. | | chat() | Structured | Define a text generation custom model's handing of real-time chat completion requests. | | get_supported_llm_models() | Structured | Define a text generation custom model's response to the OpenAI "List Models" API. | | post_process() | Structured | Define the post-processing steps applied to the model's predictions. | > [!NOTE] Custom model hook execution order > These hooks are executed in the order listed, as each hook represents a step in the custom model lifecycle. For more information on defining a custom model's code, see the hooks for [structured custom models](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html) or [unstructured custom models](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/unstructured-custom-models.html). ### Model metadata To define a custom model's [metadata and input validation schema](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-metadata.html), create a `model-metadata.yaml` file and add it to the top level of the model/model directory. The file specifies additional information about a custom model, including [runtime parameters](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-runtime-parameters.html) through `runtimeParameterDefinitions`. ## Model environment There are multiple options for defining the environment where a custom model runs. You can: - Choose from a variety ofdrop-in environments. - Modify a drop-in environment to include missing Python or R packages by specifying the packages in the model'srequirements.txtfile. If provided, therequirements.txtfile must be uploaded together with thecustom.pyorcustom.Rfile in the model content. If model content contains subfolders, it must be placed in the top folder. - Build acustom environmentif you need to install Linux packages. When creating a custom model with a custom environment, the environment used must be compatible with the model contents, as it defines the model's runtime environment. To ensure you follow the compatibility guidelines: - By default, when creating a model version, if the selected execution environment does not change, the version of that execution environment persists from the previous custom model version, even if a newer environment version is available. For more information on how to ensure the custom model version uses the latest version of the execution environment, seeTrigger base execution environment update. **Trigger base execution environment update** To override the default behavior for execution environment version selection, where the execution environment version persists between custom model versions even when a [new environment version](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/custom-models/custom-model-environments/custom-environments.html#add-an-environment-version) is available, you must temporarily change the Base Environment setting. To do this, create a new custom model version using a different Base Environment setting, then create a new custom model version, switching back to the intended Base Environment. After this change, the latest version of the custom model uses the latest version of the execution environment. --- # DRUM CLI tool URL: https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-drum.html > DataRobot Model Runner is a tool that allows you to work with and test Python, R, and Java custom models and custom tasks. The DataRobot User Models (DRUM) CLI is a tool that allows you to work with Python, R, and Java custom models and to quickly test [custom tasks](https://docs.datarobot.com/en/docs/classic-ui/modeling/special-workflows/cml/cml-custom-tasks.html), [custom models](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/custom-models/index.html), and [custom environments](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/custom-models/custom-model-environments/custom-environments.html) locally before uploading into DataRobot. Because it is also used to run custom tasks and models inside of DataRobot, if they pass local tests with DRUM, they are compatible with DataRobot. You can download DRUM from [PyPI](https://pypi.org/project/datarobot-drum/) and [access DRUM's GitHub repo](https://github.com/datarobot/datarobot-user-models/). DRUM can also: - Run performance and memory usage testing for models. - Perform model validation tests (for example, checking model functionality on corner cases, like null values imputation). - Run models in a Docker container. You can install DRUM for [Ubuntu](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-drum.html#drum-on-ubuntu), [Windows](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-drum.html#drum-on-windows-with-wsl2), or [MacOS](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-drum.html#drum-on-mac). > [!NOTE] Note > DRUM is not regularly tested on Windows or Mac. These steps may differ depending on the configuration of your machine. ## DRUM on Ubuntu The following describes the DRUM installation workflow. Consider the language prerequisites before proceeding. | Language | Prerequisites | Installation command | | --- | --- | --- | | Python | Python 3 required | pip install datarobot-drum | | Java | JRE ≥ 11 | pip install datarobot-drum | | R | Python ≥ 3.6R framework installedDRUM uses the rpy2 package to run R (the latest version is installed by default). You may need to adjust the rpy2 and pandas versions for compatibility. | pip install datarobot-drum[R] | To install the DRUM with support for Python and Java models, use the following command: ``` pip install datarobot-drum ``` To install DRUM with support for R models: ``` pip install datarobot-drum[R] ``` > [!NOTE] Note > If you are using a Conda environment, install the wheels with a `--no-deps` flag. If any dependencies are required for a Conda environment, install them with Conda tools. ## DRUM on Mac The following instructions describe installing DRUM with `conda` (although you can use other tools if you prefer) and then using DRUM to test a task locally. Before you begin, DRUM requires: - An installation ofconda. - A Python environment (also required for R) of 3.7+. ### Install DRUM on Mac 1. Create and activate a virtual environment with Python 3.7+. In the terminal for 3.8, run: condacreate-nDR-custom-taskspython=3.8-y condaactivateDR-custom-tasks 2. Install DRUM: condainstall-cconda-forgeuwsgi-y pipinstalldatarobot-drum 3. To set up the environment, installDocker Desktopand download from GitHub the DataRobotdrop-in environmentswhere your tasks will run. This recommended procedure ensures that your tasks run in the same environment both locally and inside DataRobot. Alternatively, if you plan to run your tasks in a localpythonenvironment, install packages used by your custom task into the same environment as DRUM. ### Use DRUM on Mac To test a task locally, run the `drum fit` command. For example, in a binary classification project: 1. Ensure that thecondaenvironmentDR-custom-tasksis activated. 2. Run thedrum fitcommand (replacing placeholder folder names in< >brackets with actual folder names): drum fit --code-dir --input --target-type binary --target --docker --verbose For example: drum fit --code-dir datarobot-user-models/custom_tasks/examples/python3_sklearn_binary --input datarobot-user-models/tests/testdata/iris_binary_training.csv --target-type binary --target Species --docker datarobot-user-models/public_dropin_environments/python3_sklearn/ --verbose > [!TIP] Tip > To learn more, you can view available parameters by typing `drum fit --help` on the command line. ## DRUM on Windows with WSL2 DRUM can be run on Windows 10 or 11 with WSL2 (Windows Subsystem for Linux), a native extension that is supported by the latest versions of Windows and allows you to easily install and run Linux OS on a Windows machine. With WSL, you can develop custom tasks and custom models locally in an IDE on Windows, and then immediately test and run them on the same machine using DRUM via the Linux command line. > [!TIP] Tip > You can use this [YouTube video](https://www.youtube.com/watch?v=wWFI2Gxtq-8) for instructions on installing WSL into Windows 11 and updating Ubuntu. The following phases are required to complete the Windows DRUM installation: 1. Enable WSL 2. Installpyenv 3. Install DRUM 4. Install Docker Desktop ### Enable Linux (WSL) 1. FromControl Panel > Turn Windows features on or off, check the optionWindows Subsystem for Linux. After making changes, you will be prompted to restart. 2. OpenMicrosoft storeand click to get Ubuntu. 3. Install Ubuntu and launch it from the start prompt. Provide a Unix username and password to complete installation. You can use any credentials but be sure to record them as they will be required in the future. You can access Ubuntu at any time from the Windows start menu. Access files on the C drive under /mnt/c/. ### Install pyenv Because Ubuntu in WSL comes without Python or virtual environments installed, you must install `pyenv`, a Python version management program used on macOS and Linux. (Learn about managing multiple Python environments [here](https://codeburst.io/how-to-install-and-manage-multiple-python-versions-in-wsl2-1131c4e50a58).) In the Ubuntu terminal, run the following commands (you can ignore comments) row by row: ``` cd $HOME sudo apt update --yes sudo apt upgrade --yes sudo apt-get install --yes git git clone https://github.com/pyenv/pyenv.git ~/.pyenv #add pyenv to bashrc echo '# Pyenv environment variables' >> ~/.bashrc echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc echo '# Pyenv initialization' >> ~/.bashrc echo 'if command -v pyenv 1>/dev/null 2>&1; then' >> ~/.bashrc echo ' eval "$(pyenv init -)"' >> ~/.bashrc echo 'fi' >> ~/.bashrc #restart shell exec $SHELL #install pyenv dependencies (copy as a single line) sudo apt-get install --yes libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libgdbm-dev lzma lzma-dev tcl-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev wget curl make build-essential python-openssl #install python 3.7 (it can take awhile) pyenv install 3.7.10 ``` ### Install DRUM on Windows To install DRUM, first you setup a Python environment where DRUM will run, and then install DRUM in that environment. 1. Create and activate apyenvenvironment: cd$HOMEpyenvlocal3.7.10 .pyenv/shims/python3.7-mvenvDR-custom-tasks-pyenvsourceDR-custom-tasks-pyenv/bin/activate 2. Install DRUM and its dependencies into that environment: pipinstalldatarobot-drumexec$SHELL 3. Download container environments, where DRUM will run, from Github. git clone https://github.com/datarobot/datarobot-user-models ### Install Docker Desktop While you can run DRUM directly in the `pyenv` environment, it is preferable to run it in a Docker container. This recommended procedure ensures that your tasks run in the same environment both locally and inside DataRobot, as well as simplifies installation. 1. Download and installDocker Desktop, following the default installation steps. 2. Enable Ubuntu version WSL2 by opening Windows PowerShell and running: wsl.exe--set-versionUbuntu2wsl--set-default-version2 NoteYou may need to download and install anupdate. Follow the instructions in the PowerShell until you see theConversion completemessage. 3. Enable access to Docker Desktop from Ubuntu: ### Use DRUM on Windows 1. From the command line, open an Ubuntu terminal. 2. Use the following commands to activate the environment: cd $HOME source DR-custom-tasks-pyenv/bin/activate 3. Run thedrum fitcommand in an Ubuntu terminal window (replacing placeholder folder names in< >brackets with actual folder names): drum fit --code-dir --input --target-type binary --target --docker --verbose For example: drum fit --code-dir datarobot-user-models/custom_tasks/examples/python3_sklearn_binary --input datarobot-user-models/tests/testdata/iris_binary_training.csv --target-type binary --target Species --docker datarobot-user-models/public_dropin_environments/python3_sklearn/ --verbose --- # Define custom model metadata URL: https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-metadata.html > How to use the model-metadata.yaml file to specify additional information about a custom task or a custom inference model. For structured inference models, `model-metadata.yaml` needs to declare an `inferenceModel` section: binary models need positive/negative class labels; multiclass and multilabel models need `targetName` and `classLabels` in the same order as the model’s probability outputs. See [Inference model metadata](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-metadata.html#inference-model-metadata) for more information. To define metadata, create a `model-metadata.yaml` file and put it in the top level of the task/model directory. In most cases, it can be skipped, but it is required for custom transform tasks when a custom task outputs non-numeric data. The `model-metadata.yaml` is located in the same folder as `custom.py`. The sections below show how to define metadata for custom models and tasks. For more information, you can review complete examples in the DRUM repository for [custom models](https://github.com/datarobot/datarobot-user-models/blob/master/model_templates/python3_sklearn/model-metadata.yaml) and [tasks](https://github.com/datarobot/datarobot-user-models/blob/master/task_templates/1_transforms/1_python_missing_values/model-metadata.yaml). ## General metadata parameters The following table describes options that are available to tasks and/or inference models. The parameters are required when using `drum push` to supply information about the model/task/version to create. Some of the parameters are also required outside of `drum push` for compatibility reasons. > [!NOTE] Note > The `modelID` parameter adds a new version to a pre-existing custom model or task with the specified ID. Because of this, all options that configure a new base-level custom model or task are ignored when passed alongside this parameter. However, at this time, these parameters still must be included. | Option | When required | Task or inference model | Description | | --- | --- | --- | --- | | name | Always | Both | A string, preferably unique for easy searching, that drum push uses as the custom model title. | | type | Always | Both | A string, either training (for custom tasks) or inference (for custom inference models). | | environmentID | Always | Both | A hash of the execution environment to use while running your custom model or task. You can find a list of available execution environments in Model Registry > Custom Model Workshop > Environments. Expand the environment and click on the Environment Info tab to view and copy the file ID. Required for drum push only. | | targetType | Always | Both | A string indicating the type of target. Must be one of: binaryregressionanomaly unstructured (inference models only)multiclassmultilabel (inference models only)textgeneration (inference models only)agenticworkflow (inference models only) transform (transform tasks only) | | modelID | Optional | Both | After creating a model or task, it is best practice to use versioning to add code while iterating. To create a new version instead of a new model or task, use this field to link the custom model/task you created. The ID (hash) is available from the UI, via the URL of the custom model or task. Used with drum push only. | | description | Optional | Both | A searchable field. If modelID is set, use the UI to change a model/task description. Used with drum push only. | | majorVersion | Optional | Both | Specifies whether the model version you are creating should be a major (True, the default) or minor (False) version update. For example, if the previous model version is 2.3, a major version update would create version 3.0; a minor version update would create version 2.4. Used for drum push only. | | targetName | For binary, multiclass, and multilabel (in inferenceModel) | Model | In inferenceModel, the name of the column the model predicts. For multiclass and multilabel, use the same name as Target name in the Workshop and the same order of classes or labels as Target classes or Target labels for classLabels. | | positiveClassLabel / negativeClassLabel | For binary classification models | Model | In inferenceModel, when your model predicts probability, the positiveClassLabel dictates what class the prediction corresponds to. | | classLabels | For multiclass and multilabel classification models | Model | In inferenceModel, a list of class or label names (strings). The list order must match the order of predicted class or label probabilities your model returns (for example, the column order of probability outputs). Use the same labels as the Target classes or Target labels you configure for the custom model in the Workshop. | | predictionThreshold | Optional (binary classification models only). | Model | In inferenceModel, the cutoff point between 0 and 1 that dictates which label will be chosen as the predicted label. | | trainOnProject | Optional | Task | A hash with the ID of the project (PID) to train the model or version on. When using drum push to test and upload a custom estimator task, you have an option to train a single-task blueprint immediately after the estimator is successfully uploaded into DataRobot. The trainOnProject option specifies the project on which to train that blueprint. | ## Inference model metadata (inferenceModel) For structured inference models, target and class-label settings belong under the top-level key `inferenceModel` in `model-metadata.yaml`. If you omit fields that DataRobot or DRUM require for your `targetType`, builds, tests, or deployments can fail. | targetType | Required under inferenceModel | Notes | | --- | --- | --- | | binary | targetName, positiveClassLabel, negativeClassLabel | Optional: predictionThreshold. | | multiclass | targetName, classLabels | classLabels is a YAML list of class names in the same order as your model’s probability outputs. | | multilabel | targetName, classLabels | classLabels is a YAML list of at least two label names in the same order as your model’s per-label probability outputs. Each probability must be between 0 and 1. Unlike multiclass models, label probabilities are independent and do not need to sum to 1.0. | | regression | (often none) | Many regression templates work without an inferenceModel block; follow your environment and DRUM requirements. | | anomaly, unstructured, textgeneration, … | Follow template / DRUM | See examples for your target type. | Workshop-generated file: On the Registry Workshop Assemble tab, Create model-metadata.yaml produces a starter file for your model’s target type. For multiclass and multilabel models, that file includes `inferenceModel` with `targetName` and `classLabels` (aligned with your Target classes or Target labels), matching what you need for a successful deployment. In the `model-metadata.yaml` file, you can also [define runtime parameters](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-runtime-parameters.html) to make your custom model code easier to reuse. --- # Define runtime parameters URL: https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-runtime-parameters.html > Add runtime parameters to a custom model through the model metadata, making your custom model code easier to reuse. Define environment variables to supply different values to custom model code at runtime by including them as runtime parameters, making your custom model easier to reuse. Define runtime parameters in code or through the UI: - Code: Provide amodel-metadata.yamlfile in the model artifact. Define this file before uploading a model toWorkshop, or use the template available inWorkshopthrough a custom model'sFiles >Createdropdown. The YAML structure is definedon this page. - UI: Define runtime parameters in theRuntime parameterssection of the custom model inWorkshop. For more information, see theCreate custom modelsdocumentation. Runtime parameters are injected into containers in two ways: 1. As standard environment variables without prefixes or JSON parsing for simple types (so you can use os.getenv to access environment variables without the datarobot-drum library). 2. For backward compatibility also in the legacy prefixed ( MLOPS_RUNTIME_PARAM_* ) and JSONified format. Parameters [created via theWorkshopUI](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-create-custom-model.html#define-runtime-parameters) persist and merge when you upload new code versions, ensuring a seamless development flow. **Runtime parameter considerations** The system uses a blocklist of reserved patterns (e.g., `DRUM_*`, `MLOPS_*`, `KUBERNETES_*`), managed in the dynamic configuration. Matching supports the `*` wildcard (not full regular expression syntax). Reserved names aren't blocked entirely: if a runtime parameter uses a reserved name, the UI displays a warning. How the variable is exposed depends on the context: - Custom models: Only in prefixed ( MLOPS_RUNTIME_PARAM_* ) and JSONified format—not as a raw (unprefixed) environment variable, to prevent system conflicts. - Custom apps: Prefixed, but values are not packed into a JSON payload (except for credentials). - Custom jobs: No prefix and no JSON payload (except for credential types); the variable is available as a raw environment variable. For credential-type runtime parameters, the system automatically unpacks JSON fields into separate environment variables rather than a single string. For example, a credential named `MAIN_AWS_CREDENTIAL` with the following JSON structure: ``` {"awsAccessKeyId": "", "awsSecretAccessKey": ""} ``` is unpacked into the following environment variables, combining the parameter name + JSON key, in uppercase: ``` MAIN_AWS_CREDENTIAL_AWS_ACCESS_KEY_ID="" MAIN_AWS_CREDENTIAL_AWS_SECRET_ACCESS_KEY="" ``` For single-field credential types (for example, `api_token`, `bearer`, or `gcp`), the injected environment variable uses the bare runtime parameter name ( `MY_CRED`), not the parameter name plus the credential field name (for example, not `MY_CRED_API_TOKEN`).Multi-field credential types (for example, `basic` or `s3`) keep the existing suffixed behavior: one variable per field, named `{PARAMETER_NAME}_{FIELD_NAME}` in uppercase snake case (for example, `MY_CRED_USERNAME` and `MY_CRED_PASSWORD`, or the AWS keys in the example above). JSON-encoded runtime parameter variables (for example, `MLOPS_RUNTIME_PARAMETERS_OPEN_AI_API`) are unchanged; only the flat variable for a single-field secret uses the bare parameter name. > [!TIP] Access runtime parameters in containers > For programmatic access to runtime parameters in containers, use `DataRobotAppFrameworkBaseSettings` as documented in the SDK API reference. To change runtime parameter values for an existing deployment, deactivate the deployment, update the values in the deployment's Settings > Resources tab, and then reactivate the deployment. For details, see [Configure deployment resource settings](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-settings/nxt-resource-settings.html). ## Runtime parameter definitions Add runtime parameters to a custom model through the model metadata, making your custom model code easier to reuse. To define runtime parameters, you can add the following `runtimeParameterDefinitions` in `model-metadata.yaml`: | Key | Description | | --- | --- | | fieldName | Define the name of the runtime parameter. | | type | Define the data type the runtime parameter contains: string, boolean, numeric credential, deployment. | | defaultValue | (Optional) Set the default value for the runtime parameter. For credential type parameters, use defaultValue to reference an existing credential by its credential ID. For other types, set the default string, boolean, or numeric value. If you define a runtime parameter without specifying a defaultValue, the default value is None. | | minValue | (Optional) For numeric runtime parameters, set the minimum numeric value allowed in the runtime parameter. | | maxValue | (Optional) For numeric runtime parameters, set the maximum numeric value allowed in the runtime parameter. | | credentialType | (Optional) For credential runtime parameters, set the type of credentials the parameter must contain. | | allowEmpty | (Optional) Set the empty field policy for the runtime parameter.True: (Default) Allows an empty runtime parameter.False: Enforces providing a value for the runtime parameter before deployment. | | description | (Optional) Provide a description of the purpose or contents of the runtime parameter. | ## DataRobot reserved runtime parameters The following runtime parameter is reserved by DataRobot for custom model configuration: | Runtime parameter | Type | Description | | --- | --- | --- | | CUSTOM_MODEL_WORKERS | numeric | Allows each replica to handle a set number of concurrent processes. This option is intended for process-safe custom models, primarily in generative AI use cases (for more information on process-safe models, see the note below). To determine the appropriate number of concurrent processes to allow per replica, monitor the number of requests and the median response time for the custom model. The median response time for the custom model should be close to the median response time from the LLM. If the response time of the custom model exceeds the LLM's response time, stop increasing the number of concurrent processes and instead increase the number of replicas.Default value: 1 Max value: 40 | > [!WARNING] Custom model process safety > When enabling and configuring `CUSTOM_MODEL_WORKERS`, ensure that your model is process-safe, allowing multiple independent processes to safely interact with shared resources without causing conflicts. This configuration is not intended for general use with custom models to make them more resource efficient. Only process-safe custom models with I/O-bound tasks (like proxy models) benefit from utilizing CPU resources this way. ## Define custom model metadata Before you define `runtimeParameterDefinitions` in `model-metadata.yaml`, [define the custom model metadata](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-metadata.html) required for the target type. For binary, multiclass, and multilabel models, that includes an [inferenceModel](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-metadata.html#inference-model-metadata) block ( `targetName` and class labels; for multiclass and multilabel, `classLabels`). **Binary classification:** ``` name: binary-example targetType: binary type: inference inferenceModel: targetName: target positiveClassLabel: "1" negativeClassLabel: "0" ``` **Regression:** ``` name: regression-example targetType: regression type: inference ``` **Text generation:** ``` name: textgeneration-example targetType: textgeneration type: inference ``` **Anomaly detection:** ``` name: anomaly-example targetType: anomaly type: inference ``` **Unstructured:** ``` name: unstructured-example targetType: unstructured type: inference ``` **Multiclass:** ``` name: multiclass-example targetType: multiclass type: inference inferenceModel: targetName: class classLabels: - class_a - class_b - class_c ``` **Multilabel:** ``` name: multilabel-example targetType: multilabel type: inference inferenceModel: targetName: genre classLabels: - action - comedy - sci-fi ``` Then, below the model information, you can provide the `runtimeParameterDefinitions`: ``` # Example: runtimeParameterDefinitions in model-metadata.yaml name: runtime-parameter-example targetType: regression type: inference runtimeParameterDefinitions: - fieldName: my_first_runtime_parameter type: string description: My first runtime parameter. - fieldName: runtime_parameter_with_default_value type: string defaultValue: Default description: A string-type runtime parameter with a default value. - fieldName: runtime_parameter_boolean type: boolean defaultValue: true description: A boolean-type runtime parameter with a default value of true. - fieldName: runtime_parameter_numeric type: numeric defaultValue: 0 minValue: -100 maxValue: 100 description: A boolean-type runtime parameter with a default value of 0, a minimum value of -100, and a maximum value of 100. - fieldName: runtime_parameter_for_credentials type: credential credentialType: basic allowEmpty: false description: A runtime parameter containing a dictionary of credentials; credentials must be provided before registering the custom model. - fieldName: runtime_parameter_for_connected_deployment type: deployment description: A runtime parameter defined to accept the deployment ID of another deployment to connect to the deployed custom model. ``` ## Provide credentials through runtime parameters The `credential` runtime parameter type supports any `credentialType` value available in the DataRobot REST API. At runtime, credential payloads are also reflected as environment variables in the container; see the runtime parameter considerations for naming rules (single-field types such as `api_token`, `bearer`, and `gcp` use the bare parameter name; multi-field types such as `basic` and `s3` use suffixed names per field). You can provide credentials in two ways: - Reference existing credentials: Use the credential ID as thedefaultValueto reference credentials defined in the DataRobotCredentials managementsection. - Provide credential values directly: Include the full credential structure when defining the runtime parameter (typically used during local development with DRUM). > [!NOTE] Credential types > For more information on the supported credential types, see the [API reference documentation for credentials](https://docs.datarobot.com/en/docs/api/reference/public-api/credentials.html#schemacredentialsbody). ### Reference existing credentials To reference an existing credential, set the `defaultValue` to the credential ID: ``` # Example: Reference an existing credential - fieldName: my_api_token type: credential credentialType: api_token allowEmpty: false defaultValue: description: A runtime parameter referencing an existing API token credential. ``` > [!WARNING] Credential requirements > When you reference an existing credential, the credential must exist in the credential management section before registering the custom model, must match the `credentialType` specified in the runtime parameter definition, and must match the credential ID used as the `defaultValue`. ### Provide credential values directly The credential information required depends on the `credentialType`, as shown in the examples below: | Credential Type | Example | | --- | --- | | basic | basic: credentialType: basic description: string name: string password: string user: string | | azure | azure: credentialType: azure description: string name: string azureConnectionString: string | | gcp | gcp: credentialType: gcp description: string name: string gcpKey: string | | s3 | s3: credentialType: s3 description: string name: string awsAccessKeyId: string awsSecretAccessKey: string awsSessionToken: string | | api_token | api_token: credentialType: api_token apiToken: string name: string | ## Provide override values during local development For local development with DRUM, you can specify a `.yaml` file containing the values of the runtime parameters. The values defined here override the `defaultValue` set in `model-metadata.yaml`: ``` # Example: .runtime-parameters.yaml my_first_runtime_parameter: Hello, world. runtime_parameter_with_default_value: Override the default value. runtime_parameter_for_credentials: credentialType: basic name: credentials password: password1 user: user1 ``` When using DRUM, the `--runtime-params-file` option specifies the file containing the runtime parameter values: ``` # Example: --runtime-params-file drum score --runtime-params-file .runtime-parameters.yaml --code-dir model_templates/python3_sklearn --target-type regression --input tests/testdata/juniors_3_year_stats_regression.csv ``` ## Import and use runtime parameters in custom code To import and access runtime parameters, you can import the `RuntimeParameters` module in your code in `custom.py`: ``` # Example: custom.py from datarobot_drum import RuntimeParameters def mask(value, visible=3): return value[:visible] + ("*" * len(value[visible:])) def transform(data, model): print("Loading the following Runtime Parameters:") parameter1 = RuntimeParameters.get("my_first_runtime_parameter") parameter2 = RuntimeParameters.get("runtime_parameter_with_default_value") print(f"\tParameter 1: {parameter1}") print(f"\tParameter 2: {parameter2}") credentials = RuntimeParameters.get("runtime_parameter_for_credentials") if credentials is not None: credential_type = credentials.pop("credentialType") print( f"\tCredentials (type={credential_type}): " + str({k: mask(v) for k, v in credentials.items()}) ) else: print("No credential data set") return data ``` --- # DataRobot User Models URL: https://docs.datarobot.com/en/docs/api/code-first-tools/drum/index.html > Describes how to assemble custom models and environments. While DataRobot provides hundreds of built-in models, there are situations where you need preprocessing or modeling methods that are not currently supported out of the box. To create a custom inference model, you must provide a model artifact—either defined in a `custom.py` file or a serialized artifact with a file extension corresponding to the chosen environment language and any additional custom code required to use the model. Before adding custom models and environments to DataRobot, you must prepare and structure the files required to run them successfully. The tools and templates necessary to prepare custom models are hosted in the [DataRobot User Models GitHub Repository](https://github.com/datarobot/datarobot-user-models). (Log in to GitHub before clicking this link.) DataRobot recommends understanding the following requirements to prepare your custom model for upload to the Workshop. | Topic | Describes | | --- | --- | | Custom model components | How to identify the components required to run custom inference models. | | Assemble structured custom models | How to assemble and validate structured custom models compatible with DataRobot. | | Assemble unstructured custom models | How to assemble and validate unstructured custom models compatible with DataRobot. | | Define custom model metadata | How to use the model-metadata.yaml file to specify additional information about a custom inference model. | | Define custom model runtime parameters | How to add runtime parameters to a custom model through the model metadata, making your custom model code easier to reuse. | | DRUM CLI tool | How to download and install the DataRobot User Models (DRUM) CLI to work with and test custom models and custom environments locally before uploading to DataRobot. | | Test a custom model locally | How to test custom inference models in your local environment using the DataRobot Model Runner tool. | --- # Assemble structured custom models URL: https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html > DataRobot provides built-in support for a variety of libraries to create models that use conventional target types. DataRobot provides built-in support for a variety of libraries to create models that use conventional target types. If your model is based on one of these libraries, DataRobot expects your model artifact to have a matching file extension: **Python libraries:** Library File Extension Example Scikit-learn *.pkl sklearn-regressor.pkl Xgboost *.pkl xgboost-regressor.pkl PyTorch *.pth torch-regressor.pth tf.keras (tensorflow>=2.2.1) *.h5 keras-regressor.h5 ONNX *.onnx onnx-regressor.onnx pmml *.pmml pmml-regressor.pmml **R libraries:** Library File Extension Example Caret *.rds brnn-regressor.rds **Java libraries:** Library File Extension Example datarobot-prediction *.jar dr-regressor.jar h2o-genmodel *.java GBM_model_python_1589382591366_1.java (pojo) h2o-genmodel *.zip GBM_model_python_1589382591366_1.zip (mojo) h2o-genmodel-ext-xgboost *.java XGBoost_2_AutoML_20201015_144158.java h2o-genmodel-ext-xgboost *.zip XGBoost_2_AutoML_20201015_144158.zip h2o-ext-mojo-pipeline *.mojo ... > [!NOTE] Note > DRUM supports models with DataRobot-generated Scoring Code and models that implement either the > IClassificationPredictor > or > IRegressionPredictor > interface from the > DataRobot-prediction library > . The model artifact must have a > .jar > extension. > You can define the > DRUM_JAVA_XMX > environment variable to set JVM maximum heap memory size ( > -Xmx > java parameter): > DRUM_JAVA_XMX=512m > . > If you export an H2O model as > POJO > , you cannot rename the file; however, this limitation doesn't apply to models exported as > MOJO > —they may be named in any fashion. > The > h2o-ext-mojo-pipeline > requires an h2o driverless AI license. > Support for DAI Mojo Pipeline has not been incorporated into tests for the build of > datarobot-drum > . If your model doesn't use one of the following libraries, you must create an [unstructured custom model](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/unstructured-custom-models.html). Compare the characteristics and capabilities of the two types of custom models below: | Model type | Characteristics | Capabilities | | --- | --- | --- | | Structured | Uses a target type known to DataRobot (e.g., regression, binary classification, multiclass, multilabel, and anomaly detection).Required to conform to a request/response schema.Accepts structured input and output data. | Full deployment capabilities.Accepts training data after deployment. | | Unstructured | Uses a custom target type, unknown to DataRobot.Not required to conform to a request/response schema.Accepts unstructured input and output data. | Limited deployment capabilities. Doesn't support data drift and accuracy statistics, challenger models, or humility rules.Doesn't accept training data after deployment. | ## Structured custom model requirements If your custom model uses one of the supported libraries, make sure it meets the following requirements: - Data sent to a model must be usable for predictions without additional pre-processing. - Regression models must return a single floating point per row of prediction data. - Binary classification models must return one floating point value <= 1.0 or two floating point values that sum to 1.0 per row of prediction data. - Multilabel classification models must return one floating point value per label per row of prediction data. Each value is the probability that the label applies to that row and must be between 0 and 1. Label probabilities are independent and do not need to sum to 1.0. Specify at least two labels in model-metadata.yaml or in the Workshop when creating the model. - There must be a single pkl / pth / h5 file present. > [!NOTE] Multilabel deployment monitoring > Deployed multilabel custom models support service health and feature drift monitoring. Target drift, accuracy tracking, challenger models, and retraining policies are not supported. See [Assemble a multilabel model](https://docs.datarobot.com/en/docs/workbench/nxt-registry/nxt-model-workshop/nxt-create-custom-model.html#assemble-a-multilabel-model) for details. > [!NOTE] Data format > When working with structured models DataRobot supports data as files of `csv`, `sparse`, or `arrow` format. DataRobot doesn't sanitize missing or abnormal (containing parentheses, slashes, symbols, etc. ) column names. ## Structured custom model hooks To define a custom model using DataRobot’s framework, your artifact file should contain hooks (or functions) to define how a model is trained and how it scores new data. DataRobot automatically calls each hook and passes the parameters based on the project and blueprint configuration. However, you have full flexibility to define the logic that runs inside each hook. If necessary, you can include these hooks alongside your model artifacts in your model folder in a file called `custom.py` for Python models or `custom.R` for R models. > [!WARNING] Include all required custom model code in hooks > Custom model hooks are callbacks passed to the custom model. All code required by the custom model must be in a custom model hook—the custom model can't access any code provided outside a defined custom model hook. In addition, you can't modify the input arguments of these hooks as they are predefined. > [!NOTE] Note > Training and inference hooks can be defined in the same file. The following sections describe each hook, with examples. **Type annotations in hook signatures** The following hook signatures are written with Python 3 type annotations. The Python types match the following R types: | Python type | R type | Description | | --- | --- | --- | | DataFrame | data.frame | A numpy DataFrame or R data.frame. | | None | NULL | Nothing | | str | character | String | | Any | An R object | The deserialized model. | | *args, **kwargs | ... | These are keyword arguments, not types; they serve as placeholders for additional parameters. | ### init() The `init` hook is executed only once at the beginning of the run to allow the model to load libraries and additional files for use in other hooks. ``` init(**kwargs) -> None ``` #### init() input | Input parameter | Description | | --- | --- | | **kwargs | Additional keyword arguments. code_dir is the path where the model code is stored. | #### init() example **Python:** ``` def init(code_dir): global g_code_dir g_code_dir = code_dir ``` **R:** ``` init <- function(...) { library(brnn) library(glmnet) } ``` #### init() output The `init()` hook does not return anything. ### load_model() The `load_model()` hook is executed only once at the beginning of the run to load one or more trained objects from multiple artifacts. It is only required when a trained object is stored in an artifact that uses an unsupported format or when multiple artifacts are used. The `load_model()` hook is not required when there is a single artifact in one of the supported formats: - Python: .pkl , .pth , .h5 , .joblib - Java: .mojo - R: .rds ``` load_model(code_dir: str) -> Any ``` #### load_model() input | Input parameter | Description | | --- | --- | | code_dir | Additional keyword arguments. code_dir is the path where the model code is stored. | #### load_model() example **Python:** ``` def load_model(code_dir): model_path = "model.pkl" model = joblib.load(os.path.join(code_dir, model_path)) return model ``` **R:** ``` load_model <- function(input_dir) { readRDS(file.path(input_dir, "model_name.rds")) } ``` #### load_model() output The `load_model()` hook returns a trained object (of any type). ### read_input_data() The `read_input_data` hook customizes how the model reads data; for example, with encoding and missing value handling. ``` read_input_data(input_binary_data: bytes) -> Any ``` #### read_input_data() input | Input parameter | Description | | --- | --- | | input_binary_data | Data passed through the --input parameter in drum score mode, or a payload submitted to the drum server /predict endpoint. | #### read_input_data() example **Python:** ``` def read_input_data(input_binary_data): global prediction_value prediction_value += 1 return pd.read_csv(io.BytesIO(input_binary_data)) ``` **R:** ``` read_input_data <- function(input_binary_data) { input_text_data <- stri_conv(input_binary_data, "utf8") read.csv(text=gsub("\r","", input_text_data, fixed=TRUE)) } ``` #### read_input_data() output The `read_input_data()` hook must return a pandas `DataFrame` or R `data.frame`; otherwise, you must write your own score method. ### transform() The `transform()` hook defines the output of a custom transform and returns transformed data. Do not use this hook for estimator models. This hook can be used in both transformer and estimator tasks: - For transformers, this hook applies transformations to the data provided and passes it to downstream tasks. - For estimators, this hook applies transformations to the prediction data before making predictions. ``` transform(data: DataFrame, model: Any) -> DataFrame ``` #### transform() input | Input parameter | Description | | --- | --- | | data | A pandas DataFrame (Python) or R data.frame containing the data that the custom model should transform. Missing values are indicated with NaN in Python and NA in R, unless otherwise overridden by the read_input_data hook. | | model | A trained object DataRobot loads from the artifact (typically, a trained transformer) or loaded through the load_model hook. | #### transform() example **Python:** ``` def transform(data, model): data = data.fillna(0) return data ``` **R:** ``` transform <- function(data, model) { data[is.na(data)] <- 0 data } ``` #### transform() output The `transform()` hook returns a pandas `DataFrame` or R `data.frame` with transformed data. ### score() The `score()` hook defines the output of a custom estimator and returns predictions on input data. Do not use this hook for transform models. ``` score(data: DataFrame, model: Any, **kwargs: Dict[str, Any]) -> DataFrame ``` #### score() input | Input parameter | Description | | --- | --- | | data | A pandas DataFrame (Python) or R data.frame containing the data the custom model will score. If the transform hook is used, data will be the transformed data. | | model | A trained object loaded from the artifact by DataRobot or loaded through the load_model hook. | | **kwargs | Additional keyword arguments. For a binary classification model, it contains the positive and negative class labels as the following keys:positive_class_labelnegative_class_label | #### score() examples **Python:** ``` def score(data: pd.DataFrame, model: Any, **kwargs: Dict[str, Any]) -> pd.DataFrame: predictions = model.predict(data) predictions_df = pd.DataFrame(predictions, columns=[kwargs["positive_class_label"]]) predictions_df[kwargs["negative_class_label"]] = ( 1 - predictions_df[kwargs["positive_class_label"]] ) return predictions_df ``` **R:** ``` score <- function(data, model, ...){ scores <- predict(model, newdata = data, type = "prob") names(scores) <- c('0', '1') return(scores) } ``` #### score() output The `score()` hook should return a pandas `DataFrame` (or R `data.frame` or `tibble`) of the following format: - For regression or anomaly detection projects, the output must have a numeric column namedPredictions. - For binary, multiclass, or multilabel projects, the output must have one column per class or label name, with those names used as column names. Each cell must contain the floating-point probability for that class or label. For binary or multiclass models, the probabilities in each row must sum to 1.0. For multilabel models, probabilities must be between 0 and 1 but do not need to sum to 1.0. For binary models, you can also return a single positive-class probability column instead of two columns; if two columns are returned, the first is assumed to be the negative class and the second the positive class. ##### Additional output columns > [!NOTE] Availability information > Additional output in prediction responses for custom models is off by default. Contact your DataRobot representative or administrator for information on enabling this feature. > > Feature flag: Enable Additional Custom Model Output in Prediction Responses The `score()` hook can return any number of extra columns, containing data of types `string`, `int`, `float`, `bool`, or `datetime`. When additional columns are returned through the `score()` method, the prediction response is as follows: - For a tabular response (CSV) , the additional columns are returned as part of the response table or dataframe. - For a JSON response , the extraModelOutput key is returned alongside each row. This key is a dictionary containing the values of each additional column in the row. > [!WARNING] Duplicate column names > Don't include input feature columns in the `score()` output. If the output contains a duplicate column name (for example, an additional output column that shares its name with an input feature), DataRobot doesn't raise an error; the prediction response contains only one of the duplicated values, and which one is undefined. > > To return input feature values alongside your predictions, use `passthroughColumns` in the [Predictions API](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#request-schema) or [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html#passthrough-columns) instead. Passthrough values won't collide with `extraModelOutput`: the Predictions API returns them separately, and the Batch Prediction API rejects the job on a name conflict instead of returning undefined values. **Examples: Return extra columns** The following score hooks for various target types return extra columns (containing random data for illustrative purposes) alongside the prediction data: | Regression | | --- | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | | Multiclass | | --- | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | | Generative AI | | --- | | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | ### chat() The `chat()` hook allows custom models to implement the Bolt-on Governance API to provide access to chat history and streaming response. When using the Bolt-on Governance API with a deployed LLM blueprint, see [LLM availability](https://docs.datarobot.com/en/docs/reference/gen-ai-ref/llm-availability.html) for the recommended values of the `model` parameter. Alternatively, specify a reserved value, `model="datarobot-deployed-llm"`, to let the LLM blueprint select the relevant model ID automatically when calling the LLM provider's services. ``` chat(completion_create_params: CompletionCreateParams, model: Any) -> ChatCompletion | Iterator[ChatCompletionChunk] ``` In Workbench, when [adding a deployed LLM](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#add-a-deployed-llm) that implements the `chat` function, the playground uses the Bolt-on Governance API as the preferred communication method. Enter the Chat model ID associated with the LLM blueprint to set the `model` parameter for requests from the playground to the deployed LLM. Alternatively, enter `datarobot-deployed-llm` to let the LLM blueprint select the relevant model ID automatically when calling the LLM provider's services. #### chat() input | Input parameter | Description | | --- | --- | | completion_create_params | An object containing all the parameters required to create the chat completion. For more information, review the following types from the OpenAI Python API library: CompletionCreateParams, ChatCompletion, and ChatCompletionChunk. | | model | The deserialized model loaded by DRUM or by load_model, if supplied. | #### chat() example ``` def chat(completion_create_params, model): openai_client = model return openai_client.chat.completions.create(**completion_create_params) ``` #### chat() output The `chat()` hook returns a `ChatCompletion` object if streaming is disabled and `Iterator[ChatCompletionChunk]` if streaming is enabled. If there are prompt guards configured, the first chunk of the stream contains the prompt guard moderations information (accessible via `datarobot_moderations` on the chunk). For every response guard configured that can be applied to a chunk (all guards except faithfulness, NeMo, Rouge-1, Agent goal accuracy, and task adherence), each intermediate chunk (except the last chunk) has moderations information for those guards accessible via `datarobot_moderations`. For the last chunk, all response guards information is accessible via `datarobot_moderations`. ##### Association ID As of DRUM v1.16.16, every chat completion response automatically generates and returns an association ID (as `datarobot_association_id`). The same association ID gets passed to any other configured custom metrics for the deployed LLM. A custom association ID can be optionally specified for chat requests in place of the auto-generated ID by setting `datarobot_association_id` in the `extra_body` field of the chat request. The `extra_body` field is a standard way to add more parameters to an OpenAI chat request, allowing the chat client to pass model-specific parameters to an LLM. When making a chat request to a DataRobot-deployed text generation or agentic workflow custom model, values can also be reported for arbitrary custom metrics defined for the deployment by setting `datarobot_metrics` in the `extra_body` field. If the field `datarobot_association_id` is found in `extra_body`, DataRobot uses that value instead of the automatically generated one. If the `datarobot_metrics` field is found in `extra_body`, DataRobot reports a custom metric for all the `name:value` pairs found inside. A matching custom metric for each name must already be defined for the deployment. Custom metric values reported this way must be numeric. > [!NOTE] Association ID requirement > The deployed custom model must have an association ID column defined for DataRobot to process custom metrics from chat requests, regardless of whether `extra_body` is specified. Moderation must be configured for the custom model for the metrics to be processed. > [!TIP] Manual chat request construction > The OpenAI client converts the `extra_body` parameter contents to top-level fields in the JSON payload of the chat `POST` request. When manually constructing a chat payload, without the OpenAI client, include `“datarobot_association_id": "my_association_id_0001"` in the top level of the payload. The following example shows how to set the association ID and custom metric values using `extra_body`: ``` from openai import OpenAI openai_client = OpenAI( base_url="https:///api/v2/deployments/{deployment_id}/", api_key="", ) extra_body = { # These values pass through to the LLM "llm_id": "azure-gpt-6", # If set here, replaces the auto-generated association ID "datarobot_association_id": "my_association_id_0001", # DataRobot captures these for custom metrics "datarobot_metrics": { "field1": 24, "field2": 25 } } completion = openai_client.chat.completions.create( model="datarobot-deployed-llm", messages=[ {"role": "system", "content": "Explain your thoughts using at least 100 words."}, {"role": "user", "content": "What would it take to colonize Mars?"}, ], max_tokens=512, extra_body=extra_body ) print(completion.choices[0].message.content) ``` ##### Moderations Moderation guardrails help your organization block prompt injection and hateful, toxic, or inappropriate prompts and responses. Moderation library now supports streaming response. In order for the `chat()` hook to return `datarobot_moderations`, the deployed LLM must be running in an execution environment that has the moderation library installed, and the custom model code directory must contain `moderation_config.yaml` to configure the moderations. The example below shows what is present in `ChatCompletion` if `streaming = False` and in `ChatCompletionChunk` if `streaming = True` and moderation is enabled. ``` datarobot_moderations={ 'Prompt tokens_latency': 0.20357584953308105, 'Prompts_token_count': 8, 'ROUGE-1_latency': 0.028343677520751953, 'Response tokens_latency': 0.0007507801055908203, 'Responses_rouge_1': 1.0, 'Responses_token_count': 1, 'action_promptText': '', 'action_resultText': '', 'association_id': '3d7d525b-9e99-42a4-a641-70254e924a76', 'blocked_promptText': False, 'blocked_resultText': False, 'datarobot_confidence_score': 1.0, 'datarobot_latency': 4.249604940414429, 'datarobot_token_count': 1, 'moderated_promptText': 'Now divide the result by 2.', 'replaced_promptText': False, 'replaced_resultText': False, 'reported_promptText': False, 'reported_resultText': False, 'unmoderated_resultText': '10' } ``` ##### Citations In order for the `chat()` hook to return `citations`, the deployed LLM must have a vector database associated with it. The `chat()` hook returns keys related to citations and accessible to custom models. For example: ``` citations=[ { 'content': 'ISS science results have Earth-based \napplications, including understanding our \nclimate, contributing to the treatment of \ndisease, improving existing materials, and \ninspiring the future generation of scientists, \nclinicians, technologists, engineers, \nmathematicians, artists, and explorers.\nBENEFITS\nFOR HUMANITY\nDISCOVERY\nEXPLORATION', 'link': 'Space_Station_Annual_Highlights/iss_2020_highlights.pdf:10', 'metadata': { 'chunk_id': '953', 'content': 'ISS science results have Earth-based \napplications, including understanding our \nclimate, contributing to the treatment of \ndisease, improving existing materials, and \ninspiring the future generation of scientists, \nclinicians, technologists, engineers, \nmathematicians, artists, and explorers.\nBENEFITS\nFOR HUMANITY\nDISCOVERY\nEXPLORATION', 'page': 10, 'similarity_score': 0.46, 'source': 'Space_Station_Annual_Highlights/iss_2020_highlights.pdf' }, 'vector': None }, ] ``` ### get_supported_llm_models() DataRobot custom models support the [OpenAI "List Models" API](https://platform.openai.com/docs/api-reference/models/list). To customize your model's response to this API, implement the `get_supported_llm_models()` hook in `custom.py`. ``` def get_supported_llm_models(model: Any): ``` #### get_supported_llm_models() input | Input parameter | Description | | --- | --- | | model | Optional. A model ID to compare against. | #### get_supported_llm_models() example ``` def get_supported_llm_models(model: Any): _ = model return [ Model( id="datarobot_llm_id", created=1744854432, object="model", owned_by="tester@datarobot.com", ) ] ``` You can retrieve the supported models for a custom model using the OpenAI client or the DataRobot REST API: **OpenAI client:** ``` from openai import OpenAI API_KEY = '' CHAT_API_URL = 'https://app.datarobot.com/api/v2/deployments//' def list_models(): openai_client = OpenAI( base_url=CHAT_API_URL, api_key=API_KEY, _strict_response_validation=False ) response = openai_client.models.list() print("listing models...") print(response.to_dict()) ``` **DataRobot REST API:** ``` $ curl "https://app.datarobot.com/api/v2/deployments//models" \ -H "Authorization: Bearer " {"data":[{"created":1744854432,"id":"datarobot_llm_id","object":"model","owned_by":"tester@datarobot.com"}],"object":"list"} ``` In the [DRUM repository](https://github.com/datarobot/datarobot-user-models/tree/master/model_templates/python3_dummy_chat), you can view a simple text generation model that supports the OpenAI API [/chat](https://platform.openai.com/docs/api-reference/chat/create) and [/models](https://platform.openai.com/docs/api-reference/models/list) endpoints through the `chat()` and `get_supported_llm_models()` hooks. #### get_supported_llm_models () output If your `custom.py` does not implement `get_supported_llm_models()`, the custom model returns a one item list based on the `LLM_ID` runtime parameter, if it exists. Custom models [exported from Playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/deploy-llm.html) blueprints have this parameter already set to the LLM you selected for the blueprint. If `get_supported_llm_models()` is not defined, and the `LLM_ID` runtime parameter is not defined, then the `/models` API returns an empty list. Support for `/models` is in DRUM 1.16.12 or later. ### post_process() The `post_process` hook formats the prediction data returned by DataRobot or the `score` hook when it doesn't match the output format expectations. ``` post_process(predictions: DataFrame, model: Any) -> DataFrame ``` #### post_process() input | Input parameter | Description | | --- | --- | | predictions | A pandas DataFrame (Python) or R data.frame containing the scored data produced by DataRobot or the score hook. | | model | A trained object loaded from the artifact by DataRobot or loaded through the load_model hook. | #### post_process() example **Python:** ``` def post_process(predictions, model): return predictions + 1 ``` **R:** ``` post_process <- function(predictions, model) { names(predictions) <- c('0', '1') } ``` #### post_process() output The `post_process` hook returns a pandas `DataFrame` (or R `data.frame` or `tibble`) of the following format: - For regression or anomaly detection projects, the output must have a single numeric column namedPredictions. - For binary, multiclass, or multilabel projects, the output must have one column per class or label name, with those names used as column names. Each cell must contain the probability for that class or label. For binary or multiclass models, the probabilities in each row must sum to 1.0. For multilabel models, probabilities must be between 0 and 1 but do not need to sum to 1.0. For binary models, you can also return a single positive-class probability column instead of two columns; if two columns are returned, the first is assumed to be the negative class and the second the positive class. --- # Assemble unstructured custom models URL: https://docs.datarobot.com/en/docs/api/code-first-tools/drum/unstructured-custom-models.html > Unstructured models can use arbitrary data for input and output, allowing you to deploy and monitor models regardless of the target type. If your custom model doesn't use a target type supported by DataRobot, you can create an unstructured model. Unstructured models can use arbitrary ( i.e., unstructured) data for input and output, allowing you to deploy and monitor models regardless of the target type. This characteristic of unstructured models gives you more control over how you read the data from a prediction request and response; however, it requires precise coding to assemble correctly. You must implement [custom hooks to process the unstructured input data](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/unstructured-custom-models.html#unstructured-custom-model-hooks) and generate a valid response. Compare the characteristics and capabilities of the two types of custom models below: | Model type | Characteristics | Capabilities | | --- | --- | --- | | Structured | Uses a target type known to DataRobot (e.g., regression, binary classification, multiclass, multilabel, and anomaly detection).Required to conform to a request/response schema.Accepts structured input and output data. | Full deployment capabilities.Accepts training data after deployment. | | Unstructured | Uses a custom target type, unknown to DataRobot.Not required to conform to a request/response schema.Accepts unstructured input and output data. | Limited deployment capabilities. Doesn't support data drift and accuracy statistics, challenger models, or humility rules.Doesn't accept training data after deployment. | Inference models support unstructured mode, where input and output are not verified and can be almost anything. This is your responsibility to verify correctness. For assembly instructions specific to unstructured custom inference models, reference the model templates for [Python](https://github.com/datarobot/datarobot-user-models/tree/master/model_templates/python3_unstructured) and [R](https://github.com/datarobot/datarobot-user-models/tree/master/model_templates/r_unstructured) provided in the DRUM documentation. > [!NOTE] Data format > When working with unstructured models DataRobot supports data as a text or binary file. ## Unstructured custom model hooks Include any necessary hooks in a file called `custom.py` for Python models or `custom.R` for R models alongside your model artifacts in your model folder. > [!WARNING] Include all required custom model code in hooks > Custom model hooks are callbacks passed to the custom model. All code required by the custom model must be in a custom model hook—the custom model can't access any code provided outside a defined custom model hook. In addition, you can't modify the input arguments of these hooks as they are predefined. **Type annotations in hook signatures** The following hook signatures are written with Python 3 type annotations. The Python types match the following R types: | Python type | R type | Description | | --- | --- | --- | | None | NULL | Nothing | | str | character | String | | bytes | raw | Raw bytes | | dict | list | A list of key-value pairs. | | tuple | list | A list of data. | | Any | An R object | The deserialized model. | | *args, **kwargs | ... | These are keyword arguments, not types; they serve as placeholders for additional parameters. | ### init() The `init` hook is executed only once at the beginning of the run to allow the model to load libraries and additional files for use in other hooks. ``` init(**kwargs) -> None ``` #### init() input | Input parameter | Description | | --- | --- | | **kwargs | An additional keyword argument. code_dir provides a link, passed through the --code_dir parameter, to the folder where the model code is stored. | #### init() example **Python:** ``` def init(code_dir): global g_code_dir g_code_dir = code_dir ``` **R:** ``` init <- function(...) { library(brnn) library(glmnet) } ``` #### init() output The `init()` hook does not return anything. ### load_model() The `load_model()` hook is executed only once at the beginning of the run to load one or more trained objects from multiple artifacts. It is only required when a trained object is stored in an artifact that uses an unsupported format or when multiple artifacts are used. The `load_model()` hook is not required when there is a single artifact in one of the supported formats: - Python: .pkl , .pth , .h5 , .joblib - Java: .mojo - R: .rds ``` load_model(code_dir: str) -> Any ``` #### load_model() input | Input parameter | Description | | --- | --- | | code_dir | A link, passed through the --code_dir parameter, to the directory where the model artifact and additional code are provided. | #### load_model() example **Python:** ``` def load_model(code_dir): model_path = "model.pkl" model = joblib.load(os.path.join(code_dir, model_path)) return model ``` **R:** ``` load_model <- function(input_dir) { readRDS(file.path(input_dir, "model_name.rds")) } ``` #### load_model() output The `load_model()` hook returns a trained object (of any type). ### score_unstructured() The `score_unstructured()` hook defines the output of a custom estimator and returns predictions on input data. Do not use this hook for transform models. ``` score_unstructured(model: Any, data: str/bytes, **kwargs: Dict[str, Any]) -> str/bytes [, Dict[str, str]] ``` #### score_unstructured() input | Input parameter | Description | | --- | --- | | data | Data represented as str or bytes, depending on the provided mimetype. | | model | A trained object loaded from the artifact by DataRobot or loaded through the load_model hook. | | **kwargs | Additional keyword arguments. For a binary classification model, it contains the positive and negative class labels as the following keys:mimetype: str: Indicates the nature and format of the data, taken from request Content-Type header or --content-type CLI argument in batch mode.charset: str: Indicates the encoding for text data, taken from request Content-Type header or --content-type CLI argument in batch mode.query: dict: Parameters passed as query parameters in a HTTP request or the --query CLI argument in batch mode.headers: dict: Request headers passed in the HTTP request. | #### score_unstructured() examples **Python:** The following example processes text input, decodes bytes if necessary, and returns a prediction as a string: ``` def score_unstructured(model, data, query, **kwargs): text_data = data.decode("utf8") if isinstance(data, bytes) else data text_data = text_data.strip() words_count = model.predict(text_data) return str(words_count) ``` ``` curl -X POST "$DATAROBOT_ENDPOINT/api/v2/deployments//predictionsUnstructured/" \ -H "Authorization: Bearer $DATAROBOT_API_TOKEN" \ -H "Content-Type: text/plain" \ -d "This is sample text input" # Expected response: 5 ``` The following example demonstrates parsing JSON input and returning JSON output with the appropriate `Content-Type` header: ``` import json def load_model(code_dir): """Required when no model artifact (.pkl, .h5, etc.) is present.""" return True def score_unstructured(model, data, query, **kwargs): """Parse JSON input and return JSON output with Content-Type header.""" # Parse JSON input input_data = json.loads(data) if data else {} # Your inference logic here result = { "input": input_data, "prediction": "your_prediction_here" } # Return JSON response with Content-Type header return json.dumps(result), {"mimetype": "application/json"} ``` ``` curl -X POST "$DATAROBOT_ENDPOINT/api/v2/deployments//predictionsUnstructured/" \ -H "Authorization: Bearer $DATAROBOT_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"key": "value", "test": 123}' # Expected response: {"input": {"key": "value", "test": 123}, "prediction": "your_prediction_here"} ``` **R:** ``` score_unstructured <- function(model, data, query, ...) { kwargs <- list(...) if (is.raw(data)) { data_text <- stri_conv(data, "utf8") } else { data_text <- data } count <- str_count(data_text, " ") + 1 ret = toString(count) ret } ``` #### score_unstructured() output The `score_unstructured()` hook should return: - A single value return data: str/bytes . - A tuple return data: str/bytes, kwargs: dict[str, str] where kwargs can include {"mimetype": "users/mimetype", "charset": "users/charset"} to build the Content-Type response header from individual components. ## Unstructured model considerations ### Incoming data type resolution The `score_unstructured` hook receives a `data` parameter, which can be of either `str` or `bytes` type. You can use type-checking methods to verify types: - Python:isinstance(data, str)orisinstance(data, bytes) - R:is.character(data)oris.raw(data) DataRobot uses the `Content-Type` header to determine a type to cast `data` to. The `Content-Type` header can be provided in a request or in `--content-type` CLI argument.The `Content-Type` header format is `type/subtype;parameter` (e.g., `text/plain;charset=utf8`). The following rules apply: - Ifcharsetis not defined, defaultutf8charset is used, otherwise provided charset is used to decode data. - IfContent-Typeis not defined, then incomingkwargs={"mimetype": "text/plain", "charset":"utf8"}, so data is treated as text, decoded usingutf8charset and passed asstr. - Ifmimetypestarts withtext/orapplication/json, data is treated as text, decoded using provided charset and passed asstr. - For all othermimetypevalues, data is treated as binary and passed asbytes. ### Outgoing data and kwargs parameters As mentioned above, `score_unstructured` can return: - A single data value:return data. - A tuple (data and additional parameters:return data, {"mimetype": "some/type", "charset": "some_charset"}). #### Server mode In server mode, the following rules apply: - return data: str: The data is treated as text, the defaultContent-Type="text/plain;charset=utf8"header is set in response, and data is encoded and sent using theutf8charset. - return data: bytes: The data is treated as binary, the defaultContent-Type="application/octet-stream;charset=utf8"header is set in response, and data is sent as-is. - return data, kwargs: Ifmimetypevalue is missing inkwargs, the defaultmimetypeis set according to the data typestr/bytes->text/plain/application/octet-stream. Ifcharsetvalue is missing, the defaultutf8charset is set; then, if the data is of typestr, it will be encoded using resolvedcharsetand sent. #### Batch mode The best way to debug in batch mode is to provide `--output` file. The returned data is written to a file according to the type of data returned: - strdata is written to a text file using defaultutf8or returned inkwargscharset. - bytesdata is written to a binary file. The returnedkwargsare not shown in batch mode, but you can still print them during debugging. ### Auxiliaries You may use the `datarobot_drum.RuntimeParameters` in your code (e.g.`custom.py`) to read runtime parameters delivered to the executed custom model. The runtime parameters should be defined in the DataRobot UI. Below is a simple example of how to read a string of credential runtime parameters: ``` from datarobot_drum import RuntimeParameters def load_model(code_dir): target_url = RuntimeParameters.get("TARGET_URL") s3_creds = RuntimeParameters.get("AWS_CREDENTIAL") ... ``` --- # Data science agent URL: https://docs.datarobot.com/en/docs/api/code-first-tools/ds-agent.html > The data science agent executes and iterates on the data preparation process to provide an audit trail for data governance. The data science agent provides an audit trail of data preparation actions that the agent took on your behalf. As it iterates, it creates an audit trail that not only allows you to find and modify based on output, but enhances agent governance by creating snapshots of the data as each tool in the agent makes changes. The result is a data panel artifact —the output created by the agent. The panels create a metadata record that represent each point in the agent's data transformation lifecycle. An artifact can move between steps of a machine learning pipeline, without having to hard-code file paths. In this way, for reproducibility purposes, you don't have to find the original data but can instead use the agent to recreate it based on snapshot IDs. ## Agent overview You can build the Data Science Agent application using a low-code/no-code approach that then allows you to create and deploy autonomous AI agents that can: - Execute multistep tasks. The agent creates workspaces, navigates dashboards, adjusts settings, and saves configurations without step-by-step prompting. - Persist states. Workspaces and agent configurations survive application restarts, maintaining context across sessions. - Interact with data. Agents can connect to datasets and data stores, and run machine learning models to make predictions. - Operate through natural language. Simply describe what you want and the agent calculates the implementation steps. Consider this high-level overview workflow: 1. Create acodespace. 2. Tell the agent what you want to do (creation). 3. Point to thetoolson the MCP server that will implement data prep steps. 4. Review the agent outputand adjust as necessary. The sections below broadly outline the process in DataRobot, using an MCP server for tool calls and human-in-the-loop review for authentication and governance. ### Technical implementation The agent system is powered by: - An underlying LLM. - MCP servers for extensibility. - A backend system with configurable system prompts that define agent behavior. - A workspace-based architecture where agents can be created, customized, and managed through a UI. ### Creation and persistence The Data Science Agent process allows you to spin up an environment where you can instantiate new agents with a single click. - Memory: Unlike a basic LLM, these agents are persistent. That is, if you shut down the session and restart it, the agent and its associated "workspace" remain intact. - Customization: You can modify the system prompt in the backend code to change how the agent behaves, what its goals are, and how it interacts with the user. ### Use tools Agents do more than just respond to prompts. They integrate into an ecosystem, which allows them to: - Leverage MCP servers: With code that includes an MCP (Model Context Protocol) server call, the agent has a standardized way to connect to external tools and data sources securely. - Interact with data: Using the MCP tools, the agent "talks" to data stores, data sets, and even DataRobot. - Execute tasks: The agent can then run machine learning models, make predictions, and manage a "workspace" or dashboard. ### Human/agent collaboration The agent acts as a bridge between the user and complex technical tasks: - UI management: The agent can help build or modify the layout of the results dashboard. - Code-level access: Because the entire agent stack is running in a codespace, the user can go "under the hood" to edit the agent's logic in or change its instructions, while the agent handles the high-level execution. - Panel review: With each snapshot having a unique ID, a review of the output against the snapshots helps identify areas for improvement. ## End-to-end example Consider the following simple agentic workflow: | User input | Agent response | Panel artifact | | --- | --- | --- | | What are the most recently received products? | Based on our product database in 2025, we released 4 color variations of our crew neck t-shirt: Lavender, Jade Green, Mauve, and Burnt Sienna | None | | How are sales? | Sales are good for Lavender but lower for the other new colors | Data table: Results from querying the sales of the four products. Chart: Bar chart of total sales by each new color released. | This two-question inquiry yields two panels. First a data table that shows the sales of each new variation, and second, a chart that shows the results. Fundamentally, a panel is simply YAML output in a specific format that, when recognized by a client, renders the output in particular display format. Here is sample code of the dataset panel referenced above: ``` # panel.yaml id: 6531c type: dataset payload_path: resources/6531c.parquet parents: [] title: | Mexican perishable goods import data with tariff impact calculations - updated query description: null src: |- SELECT "p"."id" AS "product_id", "p"."name" AS "product_name", "p"."category", "s"."id" AS "supplier_id", "s"."name" AS "supplier_name", "s"."country" AS "supplier_country", sum("isi"."quantity") AS "total_import_volume", sum("isi"."total_cost") AS "total_import_cost", avg("isi"."unit_cost") AS "avg_unit_cost", avg("isi"."unit_cost") * 1.25 AS "unit_cost_with_tariff", sum("isi"."total_cost") * 1.25 AS "total_cost_with_tariff", sum("isi"."total_cost") * 0.25 AS "tariff_cost_increase", "s"."lead_time_days" AS "supplier_lead_time", "s"."reliability_score" AS "supplier_reliability", substring("isi"."order_date", 1, 7) AS "order_month" FROM "products" "p" JOIN "inbound_shipment_items" "isi" ON "p"."id" = "isi"."product_id" JOIN "suppliers" "s" ON "isi"."supplier_id" = "s"."id" JOIN "inbound_shipments" "ins" ON "isi"."shipment_id" = "ins"."id" WHERE "s"."country" = 'Mexico' AND "p"."is_perishable" = 1 GROUP BY "p"."id", "p"."name", "p"."category", "s"."id", "s"."name", "s"."country", "s"."lead_time_days", "s"."reliability_score", substring("isi"."order_date", 1, 7) ORDER BY "total_import_cost" DESC src_type: sql vspan: 1 hspan: 2 ``` An agent can invoke the panel display by including conformant YAML in its response. In the reference implementation, YAML in responses is identified in three ways: - Identify xml tags in the response. - Identify three backticks denoting a code block where the content validates as YAML. - Identify a block denoted by two consecutive new lines where the block content validates as YAML. ## Panel class hierarchy All panels derive from a `BasePanel` base class and must contain the following base attributes: ``` # panel.yaml interface BasePanel { id: string; type: "chart" | "dataset" | "text" | "video"; src: string; author: string; created_at: string; // ISO-formatted creation datetime. parents: string[]; // References the parent or linked panel. } ``` Each panel type extends this base interface with additional properties specific to that panel type. ## Installation Add panels with `npm` or `yarn`. Import only the components you need for a minimal bundle size. Then, create your first panel with just a few lines of code. ``` npm install agent-panels # or yarn add agent-panels ``` Once installed, you can import the components you need: ``` import { DatasetPanel, TextPanel, ChartPanel, VideoPanel, usePanel, } from "agent-panels"; ``` Chart and dataset panels can reference payloads that are required to render the panel correctly. Payloads should refer to a file location accessible to the client (like an S3-backed URL) or should reference an MCP resource URI. ## Panel types ### Text panel The text panel component is used for displaying formatted text content with markdown support. It is a good choice for summarizing research or providing a record of agent activity: ``` ``` For example: ``` ``` ### Dataset panel The dataset panel is for displaying and interacting with tabular data with filtering, sorting, and pagination capabilities: ``` ``` ### Chart panel The chart panel represents a graphic that will be presented on the client. Specifically, a chart should reference both its graphical representation (by linking to a payload) and a dataset panel (through a parent panel reference). The chart panel creates visualizations of your data using libraries like Plotly: ``` ``` ### Video panel The video panel is used to display video content with playback controls and annotation markers: ``` ``` ## The usePanel hook The `usePanel` hook simplifies data fetching and state management for all panel types: ``` import { usePanel } from "agent-panels"; function MyComponent() { const panelProps = usePanel({ specUrl: "path/to/panel-spec.yaml", dataUrl: "path/to/data.parquet", }); return ; } ``` --- # Code-first tools URL: https://docs.datarobot.com/en/docs/api/code-first-tools/index.html > Programmatic tools DataRobot has to offer in addition to the APIs. The table below lists the various programmatic tools DataRobot has to offer in addition to the APIs: | Resource | Description | | --- | --- | | DataRobot User Models (DRUM) | A repository that contains tools, templates, and information for assembling, debugging, testing, and running your custom inference models, custom tasks, and custom notebook environments with DataRobot. | | Blueprint Workshop | Construct and modify DataRobot blueprints and their tasks using a programmatic interface. | | DataRobot Moderations library | How to install and use the DataRobot Moderations library to apply guard-based moderation to LLM prompts and responses. | | DataRobot Prediction Library | The DataRobot Prediction Library is a Python library for making predictions using various prediction methods supported by DataRobot. It provides a common interface for making predictions, making it easy to swap out the underlying implementation. | | DataRobotX (DRX) | DataRobotX, or DRX, is a collection of DataRobot extensions designed to enhance your data science experience. DRX provides a streamlined experience for common workflows but also offers new, experimental high-level abstractions. | | MLOps agents | The MLOps agents allow you to monitor and manage external models—those running outside of DataRobot MLOps. With this functionality, predictions and information from these models can be reported as part of MLOps deployments. | | Management agent | The MLOps management agent provides a standard mechanism to automate model deployment to any type of infrastructure. It pairs automated deployment with automated monitoring to ease the burden on remote models in production, especially with critical MLOps features such as challenger models and retraining. | | DRApps | DRApps is a simple command line interface (CLI) providing the tools required to host a custom application, such as a Streamlit app, in DataRobot using a DataRobot execution environment. This allows you to run apps without building your own Docker image. Custom applications don't provide any storage; however, you can access the full DataRobot API and other services. | | DataRobot model metrics library (DMM) | A repository that contains a framework to compute model machine learning metrics over time and produce aggregated metrics. In addition, it provides examples of how to run and integrate this library with your custom metrics in DataRobot. You can also review supporting DataRobot documentation. | | MLOps Utilities For Spark | A utilities library to integrate MLOps tasks with Spark. | | Apache Spark API for Scoring Code | Use the Spark API to integrate DataRobot Scoring Code JARs into Spark clusters. | | DataRobot provider for Apache Airflow | Illustrates the setup and configuration process by implementing a basic Apache Airflow DAG (Directed Acyclic Graph) to orchestrate an end-to-end DataRobot AI pipeline. | | MLflow integration for DataRobot | How to export a model from MLflow and import it into the Registry, creating key values from the training parameters, metrics, tags, and artifacts in the MLflow model. | --- # DataRobot Moderations library URL: https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/index.html > How to install and use the DataRobot Moderations library to apply guard-based moderation to LLM prompts and responses. The [DataRobot Moderations library](https://pypi.org/project/datarobot-moderations/) applies guard-based moderation to LLM prompts and responses based on your guard configuration. You define guards in YAML (or programmatically), pass prompts and responses through a pipeline, and receive structured results that indicate whether content was blocked, replaced, or allowed with metrics attached. | Section | Description | | --- | --- | | Moderations guardrails | Configure guards in YAML, choose guard types and LLM backends, and use the Python API. | | Using the Moderations CLI | Install, authenticate, and run the dr-moderation CLI. | For each evaluation, the library can report: - Whether the prompt should be blocked. - Whether the completion should be blocked. - Metric values from model and out-of-the-box guards. - Whether the prompt or response was modified by a modifier guard. ## Architecture The library wraps the typical LLM prediction flow. It first runs prescore guards that evaluate prompts and enforce moderation when necessary. Prompts that pass prescore checks are forwarded to the LLM for completion. The library then evaluates those completions with postscore guards and enforces intervention as needed. ## How to build it The repository uses `poetry` to manage the build process and a wheel can be built using: ``` make clean make ``` ## How to use it You can install a generated or downloaded wheel file with `pip`; dependencies are installed automatically. ``` pip install datarobot-moderations ``` ### Optional extras The base install covers token-count, ROUGE-1, cost, and NeMo guards. Heavier or cloud-specific dependencies are opt-in: | Extra | What it enables | | --- | --- | | datarobot-sdk | DataRobot model guards, DataRobot LLM evaluator type | | llm-eval | Faithfulness, Task Adherence, Agent Goal Accuracy, Guideline Adherence guards | | nemo | NeMo Guardrails colang-based flow guard | | nemo-evaluator | NeMo live-evaluation microservice guard | | nvidia | NVIDIA NIM / ChatNVIDIA LLM support | | vertex | Google Cloud Vertex AI LLM support | | bedrock | AWS Bedrock LLM support | | all | Every optional dependency at once | The library opts out of deepeval telemetry by default. #### Example: task-adherence guard backed by a DataRobot LLM deployment ``` pip install 'datarobot-moderations[llm-eval,datarobot-sdk]' ``` ### Transient dependencies and build compatibility Installing `[all]` (or the `nemo` / `llm-eval` extras individually) pulls in packages that `nemoguardrails` and `deepeval` declare as runtime dependencies but that this library never uses at runtime: | Package | Pulled in by | Problem | | --- | --- | --- | | annoy | nemoguardrails | Requires a C++ compiler; breaks restricted build environments such as Kaniko | | fastembed / onnxruntime | nemoguardrails | Heavy ML runtimes, hundreds of MB | | fastapi / starlette / uvicorn | nemoguardrails | Web server stack, only used by nemoguardrails' built-in server | | watchdog / prompt-toolkit / typer | nemoguardrails, deepeval | Dev-server and CLI tools | | pyfiglet / wheel | deepeval | CLI banner / build artifact mis-declared as a runtime dep | To exclude them, add the following to your own project's `pyproject.toml` (these overrides are not inherited from this library): ``` [tool.uv] override-dependencies = [ "annoy; sys_platform == 'never'", "fastembed; sys_platform == 'never'", "onnxruntime; sys_platform == 'never'", "fastapi; sys_platform == 'never'", "starlette; sys_platform == 'never'", "uvicorn; sys_platform == 'never'", "watchdog; sys_platform == 'never'", "prompt-toolkit; sys_platform == 'never'", "typer; sys_platform == 'never'", "pyfiglet; sys_platform == 'never'", "wheel; sys_platform == 'never'", ] ``` ## Standalone Python API Create a `ModerationPipeline` from a YAML file, a plain dict, or a Pydantic config object, then evaluate prompts, responses, or a full prescore → LLM → postscore pipeline. Each method has an async counterpart. For constructor options, method parameters, return types, DataFrame schemas, streaming details, and environment variables, see [Moderations guardrails](https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/moderations-guardrails.html#using-the-config-in-python). ## Using the CLI The package ships a `dr-moderation` CLI so you can manage guards without writing Python code. Commands include `evaluate`, `add-guard`, `agent a2a connect`, and `serve` (JSON-RPC over stdio or WebSocket). For installation, authentication, command reference, YAML schema differences, and exit codes, see [Using Moderations with the CLI](https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/using-moderations-with-the-cli.html). ## Usage with DRUM The library wraps [DRUM's](https://github.com/datarobot/datarobot-user-models) `score` method for prescore and postscore guards. With DRUM, run your custom model using `drum score` to use moderation features. Install DRUM along with the optional extras required for your guards. If you are unsure which guards are in use, install `[all]`: ``` pip install datarobot-drum 'datarobot-moderations[all]' drum score --verbose --logging-level info --code-dir ./ --input ./input.csv --target-type textgeneration --runtime-params-file values.yaml ``` For DRUM-specific guard configuration and the `chat()` hook response format, see [Moderations in structured custom models](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#moderations). --- # Moderations guardrails URL: https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/moderations-guardrails.html > Reference for guard configuration YAML, guard types, LLM backends, the Python API, and environment variables. Guards evaluate prompts (prescore) and/or responses (postscore) and can block, report, or replace content based on configurable conditions. ## File structure The yaml file structure contains configuration and is later imported to the library. ``` timeout_sec: 10 timeout_action: score nemo_evaluator_deployment_id: "" guards: - name: My Guard type: ootb stage: prompt # ... ``` ## Top-level options | Field | Type | Default | Description | | --- | --- | --- | --- | | timeout_sec | int | 10 | Seconds to wait per guard | | timeout_action | string | score | score (allow) or block on timeout | | nemo_evaluator_deployment_id | string | — | DataRobot deployment ID of the NeMo Evaluator microservice; required when any guard uses type: nemo_evaluator | | enable_deepeval_telemetry | bool | false | Opt in to deepeval usage telemetry and local .deepeval/ artifacts. See Environment variables. | | prompt_column_name | string | "promptText" | Name of the DataFrame column that holds the input text. Used in standalone Python when no DRUM deployment is active. Ignored when a DRUM deployment context is active. | | response_column_name | string | "completion" | Name of the DataFrame column that holds the LLM response text. Used in standalone Python as a fallback when TARGET_NAME is not set. Lower priority than TARGET_NAME — if both are provided, TARGET_NAME wins. Ignored when a DRUM deployment context is active. | | guards | list | required | List of guard definitions | ## Common guard fields | Field | Required | Description | | --- | --- | --- | | name | Yes | Unique label; used as the key in result.metrics and as the DataRobot custom metric name | | type | Yes | ootb · model · nemo_guardrails · nemo_evaluator | | stage | Yes | prompt · response · [prompt, response] (list runs the guard at both stages) | | description | No | Free-text label, ignored by the library | | intervention | No | What to do when the condition fires (see Intervention block). Omit entirely to measure only — nothing is ever blocked | | copy_citations | No | Boolean (true/false, default false). Passes retrieved RAG context to this guard. Required for rouge_1 and faithfulness to produce meaningful scores | | is_agentic | No | Marks an agentic-workflow guard (default false). Required by agent_goal_accuracy | ``` # stage as a list — guard runs independently at both prompt and response stages - name: Token Count Both type: ootb ootb_type: token_count stage: [prompt, response] intervention: action: block message: "Input or output exceeds the token limit." conditions: - comparator: greaterThan comparand: 100 ``` ## Intervention block ``` intervention: action: block # "block" | "report" | "replace" message: "Blocked." # returned to caller send_notification: false conditions: - comparand: 0.5 comparator: greaterThan ``` > [!NOTE] One condition per intervention > The `conditions` list accepts exactly one entry for `block` and `replace`; zero entries ( `conditions: []`) is valid for `report`. To combine conditions (e.g. block if score < 0.2 or > 0.9), use two separate guards. ### Actions | Action | Effect | | --- | --- | | block | Reject and return message to the caller. message is optional in the schema but omitting it returns an empty string — always set it. | | report | Record the metric and allow content through unchanged. Behaviorally identical to omitting the intervention block entirely; useful when you want the metric tracked but never want to block. | | replace | Swap the text with the sanitized version returned by the deployment. Only valid for type: model guards. The deployment must return the replacement text in the field specified by model_info.replacement_text_column_name; if that field is absent a ValueError is raised. | ### Comparators | Comparator | Comparand type | Description | | --- | --- | --- | | greaterThan / lessThan | number | Numeric threshold | | equals / notEquals | number \| string | Exact equality. Use comparand: "TRUE" with NeMo Guardrails guards, whose score is the string "TRUE" or "FALSE" | | is / isNot | boolean | Boolean equality | | matches / doesNotMatch | list of strings | Class membership. matches fires if the prediction is in the list; doesNotMatch fires if it is not. | | contains / doesNotContain | list of strings | Substring check against a list. contains fires if all items in the list are found as substrings of the prediction; doesNotContain fires if not all items are found. | ## Guard types ### Out-of-the-Box (ootb) Set `type: ootb` and `ootb_type`. Install the required libraries for your use case: ``` pip install datarobot-moderations # base — token_count, rouge_1, cost, custom_metric pip install 'datarobot-moderations[llm-eval]' # + faithfulness, task_adherence, agent_guideline_adherence, agent_goal_accuracy pip install 'datarobot-moderations[llm-eval,vertex]' # + Google Vertex AI as LLM judge pip install 'datarobot-moderations[llm-eval,bedrock]' # + AWS Bedrock as LLM judge pip install 'datarobot-moderations[llm-eval,nvidia]' # + NVIDIA NIM as LLM judge pip install 'datarobot-moderations[nemo]' # + NeMo Guardrails colang flow guard (type: nemo_guardrails) pip install 'datarobot-moderations[nemo-evaluator]' # + NeMo Evaluator microservice guard (type: nemo_evaluator) pip install 'datarobot-moderations[datarobot-sdk]' # required for type: model and llm_type: datarobot pip install 'datarobot-moderations[all]' # everything ``` | ootb_type | Stage | Install extra | Description | | --- | --- | --- | --- | | token_count | prompt / response | (base) | Token count | | rouge_1 | response | (base) | ROUGE-1 overlap with citations | | faithfulness | response | llm-eval | LLM-judged hallucination detection | | task_adherence | response | llm-eval | Task-completion score | | agent_guideline_adherence | response | llm-eval | Guideline adherence | | agent_goal_accuracy | response | llm-eval | Agentic goal-accuracy | | cost | response | (base) | Estimated cost. Counts both prompt tokens (input_price/input_unit) and response tokens (output_price/output_unit). Must be at the response stage because both token counts are only available after the LLM responds. Currently only currency: USD is supported. | | custom_metric | prompt / response | (base) | User-defined numeric metric | ``` # Token count — report only - name: Prompt Token Count type: ootb ootb_type: token_count stage: prompt # Token count — block on length - name: Response Token Count type: ootb ootb_type: token_count stage: response intervention: action: block message: "Response too long." conditions: - comparand: 1000 comparator: greaterThan # ROUGE-1 (requires citations) - name: Rouge 1 type: ootb ootb_type: rouge_1 stage: response copy_citations: true intervention: action: report conditions: [] # Faithfulness - name: Faithfulness type: ootb ootb_type: faithfulness stage: response copy_citations: true llm_type: datarobot deployment_id: "" # 24-char DataRobot deployment ID intervention: action: block message: "Hallucination detected." conditions: - comparand: 0.0 comparator: equals # Task Adherence - name: Task Adherence type: ootb ootb_type: task_adherence stage: response llm_type: datarobot deployment_id: "" intervention: action: block message: "LLM did not complete the requested task." conditions: - comparator: lessThan comparand: 0.5 # Guideline Adherence - name: Guideline Adherence type: ootb ootb_type: agent_guideline_adherence stage: response llm_type: datarobot deployment_id: "" additional_guard_config: agent_guideline: "Response must be polite and on-topic." # free-text criterion for the LLM judge intervention: action: block message: "Response violates guidelines." conditions: - comparand: 0.0 comparator: equals # Agent Goal Accuracy - name: Agent Goal Accuracy type: ootb ootb_type: agent_goal_accuracy stage: response is_agentic: true llm_type: datarobot deployment_id: "" intervention: action: report conditions: [] # Cost tracking - name: Cost type: ootb ootb_type: cost stage: response additional_guard_config: cost: currency: USD input_price: 0.01 input_unit: 1000 output_price: 0.03 output_unit: 1000 intervention: action: report conditions: [] ``` ### Model guard Wraps any DataRobot deployment you have already created (binary classifier, regression, multiclass, or text-generation). The library sends the text to that deployment and uses the prediction it returns to decide whether to block, report, or replace content. ``` # Binary classifier (e.g. toxicity, prompt injection) # Works with any DataRobot binary classification deployment. - name: Toxicity type: model stage: prompt deployment_id: "" # 24-char DataRobot deployment ID model_info: input_column_name: text # field your deployment reads as input target_name: toxicity_toxic_PREDICTION # prediction field returned by the deployment target_type: Binary # Binary | Regression | Multiclass | TextGeneration class_names: [] # leave empty for Binary/Regression intervention: action: block message: "Toxic content blocked." conditions: - comparand: 0.5 comparator: greaterThan # PII detection with text replacement # The deployment must return BOTH the score field (`target_name`) # AND a sanitized-text field (`replacement_text_column_name`). - name: PII Detector type: model stage: prompt deployment_id: "" model_info: input_column_name: text target_name: contains_pii_true_PREDICTION target_type: TextGeneration replacement_text_column_name: anonymized_text_OUTPUT class_names: [] intervention: action: replace message: "PII removed from prompt." conditions: - comparand: 0.5 comparator: greaterThan # Multi-label / emotion classifier - name: Emotion Classifier type: model stage: prompt deployment_id: "" model_info: input_column_name: text target_name: target_PREDICTION target_type: TextGeneration class_names: [anger, fear, sadness, disgust, joy, neutral] intervention: action: block message: "Negative emotion detected." conditions: - comparand: [anger, fear, sadness, disgust] comparator: matches ``` ### NeMo Guardrails Flow-based content filtering. Requires `pip install 'datarobot-moderations[nemo]'`. Supported `llm_type` values include `openAi`, `azureOpenAi`, `nim`, and `llmGateway`. Colang flow files must live in stage-specific subdirectories of `nemo_guardrails/`: ``` nemo_guardrails/ prompt/ # config.yml + *.co files for stage: prompt response/ # config.yml + *.co files for stage: response ``` ``` - name: Stay on topic type: nemo_guardrails stage: prompt llm_type: azureOpenAi openai_api_base: "https://.openai.azure.com/" openai_deployment_id: gpt-4o-mini intervention: action: block message: "This topic is outside the allowed scope." conditions: - comparand: "TRUE" comparator: equals ``` ### NeMo Evaluator Calls a DataRobot-hosted NeMo Evaluator microservice. Requires `pip install 'datarobot-moderations[nemo-evaluator]'`. Two deployment IDs — what's the difference? | Field | What it points to | | --- | --- | | nemo_evaluator_deployment_id (top-level) | Your NeMo Evaluator microservice deployment in DataRobot | | deployment_id (per-guard) | The LLM deployment the evaluator uses to do the judging | Both values must be valid 24-character DataRobot deployment IDs. Using a placeholder longer than 24 characters (e.g.`""`) causes a load-time validation error: `String is longer than 24 characters`. > llm_typemust bedatarobotfor allnemo_evaluatorguards. | nemo_evaluator_type | Stage | Description | | --- | --- | --- | | llm_judge | prompt / response | Custom LLM-as-judge with your own prompts. score_parsing_regex is a regular expression applied to the LLM's raw text reply to extract a single numeric score — e.g. "([1-5])" picks the first digit 1–5 from any surrounding text. | | context_relevance | response | Relevance of retrieved context to the question | | response_groundedness | response | Groundedness in retrieved context | | topic_adherence | response | Adherence to allowed topics | | response_relevancy | response | Relevance of response to question | | faithfulness | response | NeMo microservice faithfulness score | | agent_goal_accuracy | response | Agentic goal-accuracy via NeMo | ``` nemo_evaluator_deployment_id: "" guards: - name: Safety Judge type: nemo_evaluator stage: response nemo_evaluator_type: llm_judge llm_type: datarobot deployment_id: "" nemo_llm_judge_config: system_prompt: "Rate safety 1-5. Output ONLY the integer." user_prompt: "Response: {response}" score_parsing_regex: "([1-5])" # regex to extract the numeric score from the LLM's text output custom_metric_directionality: higherIsBetter # "higherIsBetter" | "lowerIsBetter" intervention: action: block message: "Response failed safety evaluation." conditions: - comparand: 2 comparator: lessThan - name: Topic Adherence type: nemo_evaluator stage: response nemo_evaluator_type: topic_adherence llm_type: datarobot deployment_id: "" nemo_topic_adherence_config: metric_mode: f1 # "f1" | "precision" | "recall" reference_topics: [DataRobot, machine learning, AI platforms] intervention: action: report conditions: [] - name: Response Relevancy type: nemo_evaluator stage: response nemo_evaluator_type: response_relevancy llm_type: datarobot deployment_id: "" nemo_response_relevancy_config: embedding_deployment_id: "" intervention: action: report conditions: [] ``` ## LLM back-end options Some `ootb` guards (e.g.`faithfulness`, `task_adherence`) call an LLM to judge the text. You choose which LLM provider to use via `llm_type`. > DataRobot credentials (DATAROBOT_ENDPOINT+DATAROBOT_API_TOKEN) are always required ### Supported llm_type values | llm_type | LLM provider | Extra YAML fields | Extra install | | --- | --- | --- | --- | | datarobot | DataRobot-hosted LLM deployment | deployment_id | datarobot-sdk | | openAi | OpenAI API | (none) | llm-eval | | azureOpenAi | Azure OpenAI | openai_api_base, openai_deployment_id | llm-eval | | google | Google Vertex AI | google_region, google_model | llm-eval,vertex | | amazon | AWS Bedrock | aws_region, aws_model | llm-eval,bedrock | | nim | NVIDIA NIM | openai_api_base | llm-eval,nvidia | | llmGateway | DataRobot LLM Gateway | llm_gateway_model_id | datarobot-sdk | `nemo_guardrails` supports: `openAi`, `azureOpenAi`, `nim`, `llmGateway` only `nemo_evaluator` supports: `datarobot` only ### Available models (Google / AWS) The library maps a fixed set of model names to their provider API identifiers. Models not in this list are not supported. | Provider | llm_type | google_model / aws_model | | --- | --- | --- | | Google Vertex AI | google | google-gemini-1.5-flash, google-gemini-1.5-pro, chat-bison | | AWS Bedrock | amazon | amazon-titan, anthropic-claude-2, anthropic-claude-3-haiku, anthropic-claude-3-sonnet, anthropic-claude-3-opus, anthropic-claude-3.5-sonnet-v1, anthropic-claude-3.5-sonnet-v2, amazon-nova-lite, amazon-nova-micro, amazon-nova-pro | ## Full annotated example > Replace every<...>placeholder with a real value before use. > DataRobot deployment IDs are exactly 24 hexadecimal characters. ``` timeout_sec: 15 timeout_action: score guards: # -- Prescore (prompt) -------------------------------------------------- - name: Prompt Injection type: model stage: prompt deployment_id: "" model_info: input_column_name: text target_name: injection_injection_PREDICTION target_type: Binary class_names: [] intervention: action: block message: "Prompt injection attempt detected and blocked." conditions: - comparand: 0.80 comparator: greaterThan - name: Toxicity type: model stage: prompt deployment_id: "" model_info: input_column_name: text target_name: toxicity_toxic_PREDICTION target_type: Binary class_names: [] intervention: action: block message: "Toxic content is not allowed." conditions: - comparand: 0.5 comparator: greaterThan - name: PII Detector type: model stage: prompt deployment_id: "" model_info: input_column_name: text target_name: contains_pii_true_PREDICTION target_type: TextGeneration replacement_text_column_name: anonymized_text_OUTPUT class_names: [] intervention: action: replace message: "PII detected and removed." conditions: - comparand: 0.5 comparator: greaterThan - name: Topic Guardrail type: nemo_guardrails stage: prompt llm_type: azureOpenAi openai_api_base: "https://.openai.azure.com/" openai_deployment_id: gpt-4o-mini intervention: action: block message: "This topic is outside the allowed scope." conditions: - comparand: "TRUE" comparator: equals # -- Postscore (response) ----------------------------------------------- - name: Response Token Count type: ootb ootb_type: token_count stage: response - name: Faithfulness type: ootb ootb_type: faithfulness stage: response copy_citations: true llm_type: datarobot deployment_id: "" intervention: action: block message: "The response appears to be hallucinated." conditions: - comparand: 0.0 comparator: equals - name: Task Adherence type: ootb ootb_type: task_adherence stage: response llm_type: datarobot deployment_id: "" intervention: action: block message: "LLM did not complete the requested task." conditions: - comparator: lessThan comparand: 0.5 - name: Cost type: ootb ootb_type: cost stage: response additional_guard_config: cost: currency: USD input_price: 0.01 input_unit: 1000 output_price: 0.03 output_unit: 1000 intervention: action: report conditions: [] ``` ## Using the config in Python Guards can be configured from a YAML file, a plain Python dict, or a Pydantic object built entirely in Python. All approaches are fully equivalent — choose whichever fits your workflow. ### From a YAML file #### Return types | Method | Returns | | --- | --- | | evaluate_prompt(prompt) | (EvaluationResult, latency_seconds, prescore_df) | | evaluate_response(response, prompt=None) | (EvaluationResult, latency_seconds, postscore_df) | | evaluate_full_pipeline(prompt, llm_callable) | (PipelineResult, prescore_df, postscore_df) — postscore_df is None when the prompt was blocked; per-stage latency is not returned — use evaluate_prompt / evaluate_response directly when you need it | | evaluate_prompt_async(prompt) | same as evaluate_prompt but non-blocking | | evaluate_response_async(response, prompt=None) | same as evaluate_response but non-blocking | | evaluate_full_pipeline_async(prompt, llm_callable) | same as evaluate_full_pipeline but non-blocking; llm_callable must be an async coroutine | | evaluate_full_pipeline_stream_async(prompt, llm_callable) | AsyncGenerator[ChatCompletionChunk, None] — see Streaming pipeline | | stream_response_async(completion, *, prompt, prescore_df, prescore_latency) | AsyncGenerator[ChatCompletionChunk, None] — lower-level; see Streaming pipeline | `EvaluationResult.metrics` holds the guard scores keyed by guard name. #### evaluate_prompt / evaluate_prompt_async parameters | Parameter | Type | Required | Description | | --- | --- | --- | --- | | prompt | str | Yes | The user prompt text to evaluate against prescore guards | #### evaluate_response / evaluate_response_async parameters | Parameter | Type | Required | Description | | --- | --- | --- | --- | | response | str | Yes | The LLM response text to evaluate against postscore guards | | prompt | str \\| None | No | The original user prompt. Required for guards that compare prompt and response (e.g. faithfulness, task_adherence, rouge_1). Omit only when no such guards are configured | | pipeline_interactions | str \\| None | No | JSON-serialized MultiTurnSample dict from the DataRobot agentic pipeline. Enables agent_goal_accuracy to evaluate the full interaction trace instead of just the final response. | #### evaluate_full_pipeline / evaluate_full_pipeline_async parameters | Parameter | Type | Required | Description | | --- | --- | --- | --- | | prompt | str | Yes | The user prompt to evaluate | | llm_callable | Callable[[str], str] (sync) or Callable[[str], Awaitable[str]] (async) | Yes | Callable that receives the (possibly sanitized) effective prompt and returns the LLM response. For the async variant this must be an async coroutine | #### EvaluationResult fields | Field | Type | Description | | --- | --- | --- | | blocked | bool | True if any guard blocked the text | | blocked_message | str \\| None | The block message configured on the guard | | replaced | bool | True if a replace-action guard fired | | replacement | str \\| None | The sanitized replacement text (PII-scrubbed prompt, etc.) | | metrics | dict[str, Any] | Guard scores keyed by guard name (e.g. {"Toxicity": 0.87}) | #### PipelineResult fields | Field | Type | Description | | --- | --- | --- | | prompt_evaluation | EvaluationResult | Prescore evaluation result | | response | str \\| None | Final (possibly replaced) LLM response; None when blocked | | response_evaluation | EvaluationResult \\| None | Postscore evaluation result; None when prompt was blocked | | blocked (computed) | bool | True if either stage was blocked | | replaced (computed) | bool | True if either stage was replaced | #### What prescore_df contains `prescore_df` is the raw pandas DataFrame produced by running all prescore (prompt-stage) guards on the input.It starts as a copy of the input and gains one set of columns per guard after execution. | Column | Description | | --- | --- | | {prompt_column_name} | Original prompt text | | {guard.metric_column_name} | Guard score (one column per guard, e.g. Toxicity_toxicity_toxic_PREDICTION) | | {guard_name}_latency | Wall-clock seconds this guard took | | blocked_{prompt_col} | True if any guard blocked the prompt | | blocked_message_{prompt_col} | Block reason / message returned to the caller | | replaced_{prompt_col} | True if a replace-action guard fired | | replaced_message_{prompt_col} | Replacement text (sanitized prompt from PII guard, etc.) | | reported_{prompt_col} | True when a report-action guard fired | | Noneed_{prompt_col} | Internal sentinel for no-action guards | | action_{prompt_col} | Comma-joined string of actions taken (e.g. "block", "report,block") | | (per-guard enforced column) | Internal per-guard enforcement flag used by format_result_df | #### What postscore_df contains `postscore_df` is the raw pandas DataFrame produced by running all postscore (response-stage) guards on the LLM output.It starts with the predictions DataFrame (which includes the LLM response plus any pass-through columns) and gains guard result columns after execution. | Column | Description | | --- | --- | | {response_column_name} | LLM's response text | | {prompt_column_name} | User prompt (forwarded for faithfulness / task-adherence calculation) | | CITATION_CONTENT_{N} | Retrieved RAG context chunks (when citations are enabled) | | PROMPT_TOKEN_COUNT_from_usage | Prompt token count (when usage is provided by the LLM) | | RESPONSE_TOKEN_COUNT_from_usage | Response token count (when usage is provided by the LLM) | | agentic_pipeline_interactions | Agentic workflow interaction trace (for agent_goal_accuracy / task_adherence) | | {association_id_column_name} | Association ID (if the deployment has one configured) | | {guard.metric_column_name} | Guard score (one column per postscore guard, e.g. Response_Faithfulness_score) | | {guard_name}_latency | Wall-clock seconds this guard took | | blocked_{response_col} | True if any guard blocked the response | | blocked_message_{response_col} | Block message returned to the caller | | replaced_{response_col} | True if a replace-action guard fired on the response | | replaced_message_{response_col} | Replacement text | | reported_{response_col} | True when a report-action guard fired | | Noneed_{response_col} | Internal sentinel for no-action guards | | action_{response_col} | Comma-joined string of actions taken | | (per-guard enforced column) | Internal per-guard enforcement flag | > Note:prescore_dfandpostscore_dfare theraw executor outputs.In the DRUM pipeline,format_result_dfmerges them into a singleresult_dfthat also addsunmoderated_{response_col},moderated_{prompt_col},datarobot_latency,datarobot_token_count, > anddatarobot_confidence_score. Those derived columns arenotpresent in the DataFrames > returned directly byevaluate_prompt/evaluate_response/evaluate_full_pipeline. ``` import os from datarobot_dome.api import ModerationPipeline os.environ["DATAROBOT_ENDPOINT"] = "" os.environ["DATAROBOT_API_TOKEN"] = "" # TARGET_NAME is optional — sets the response column name used by postscore guards. # Resolution order: TARGET_NAME env var → response_column_name in config → default "completion". # os.environ["TARGET_NAME"] = "resultText" pipeline = ModerationPipeline.from_yaml("moderation_config.yaml") # ── Prompt evaluation (prescore guards) ─────────────────────────────────────── # sync result, latency, prescore_df = pipeline.evaluate_prompt("What is DataRobot?") # async (inside an async function / FastAPI route / agent) result, latency, prescore_df = await pipeline.evaluate_prompt_async("What is DataRobot?") if result.blocked: print(f"Blocked: {result.blocked_message}") elif result.replaced: print(f"Prompt sanitized to: {result.replacement}") # ── Response evaluation (postscore guards) ──────────────────────────────────── # sync result, latency, postscore_df = pipeline.evaluate_response( "DataRobot is an AI platform.", prompt="What is DataRobot?", # required for faithfulness / task-adherence guards ) # async result, latency, postscore_df = await pipeline.evaluate_response_async( "DataRobot is an AI platform.", prompt="What is DataRobot?", ) print(f"Latency: {latency:.3f}s Blocked: {result.blocked} Metrics: {result.metrics}") # ── Full pipeline: prescore → LLM → postscore ───────────────────────────────── # sync def my_llm(prompt: str) -> str: return "DataRobot is an AI platform." # replace with your LLM call result, prescore_df, postscore_df = pipeline.evaluate_full_pipeline("What is DataRobot?", my_llm) # async (llm_callable must be an async coroutine) async def my_async_llm(prompt: str) -> str: return "DataRobot is an AI platform." # replace with your async LLM call result, prescore_df, postscore_df = await pipeline.evaluate_full_pipeline_async( "What is DataRobot?", my_async_llm ) if result.blocked: stage = "prompt" if result.prompt_evaluation.blocked else "response" blocked_eval = ( result.prompt_evaluation if result.prompt_evaluation.blocked else result.response_evaluation ) print(f"Blocked at {stage}: {blocked_eval.blocked_message}") elif result.replaced: print(f"Text replaced. Response: {result.response}") else: print(f"Response: {result.response}") print(f"Metrics: {result.response_evaluation.metrics}") ``` #### Agentic workflow example For agents, the library can evaluate the full interaction trace — every tool call, intermediate message, and final response — not just the last reply. This gives the `agent_goal_accuracy` guard accurate context to judge whether the agent actually achieved the user's goal. The interaction trace ( `pipeline_interactions`) is a JSON-serialized [ragas.MultiTurnSample](https://docs.ragas.io) produced by the DataRobot agent after each task run. Pass it directly to `evaluate_response`. Config ( `docs/examples/agent_goal_accuracy_config.yaml`): ``` targets: - target: _default guards: - name: Agent Goal Accuracy type: ootb ootb_type: agent_goal_accuracy stage: response is_agentic: true llm_type: llmGateway llm_gateway_model_id: "azure/gpt-4o-mini" intervention: action: report # measure-only: block/replace are ignored by the library conditions: [] ``` > Measure-only guard:agent_goal_accuracy(likecostandguideline_adherence) always > forcesintervene=Falseinternally regardless of theactionconfigured. The score is only > available inresult.metrics["agent_goal_accuracy"]— use it to make blocking decisions in > your own code when needed. Python — with full interaction trace (recommended for agentic pipelines): ``` import json from datarobot_dome.api import ModerationPipeline pipeline = ModerationPipeline.from_yaml("docs/examples/agent_goal_accuracy_config.yaml") task = "Book a flight from NYC to London" # chat_completion is the object returned by the DataRobot agent SDK. # `pipeline_interactions` is attached when the agent has tool calls / multi-turn # history; it is None for a plain single-turn response. chat_completion = my_agent.run(task=task) agent_response = chat_completion.choices[0].message.content interactions_json = getattr(chat_completion, "pipeline_interactions", None) result, latency, postscore_df = pipeline.evaluate_response( response=agent_response, prompt=task, pipeline_interactions=interactions_json, # JSON str, or None ) score = result.metrics.get("agent_goal_accuracy") passed = score is not None and score >= 0.5 print(f"score={score} passed={passed}") **Python — building the interaction trace manually** (when not using the DataRobot agent SDK): ```python import json from ragas import MultiTurnSample from ragas.messages import AIMessage, HumanMessage, ToolCall, ToolMessage # Reconstruct the trace from your agent's execution log. {: #reconstruct-the-trace-from-your-agents-execution-log } sample = MultiTurnSample( user_input=[ HumanMessage(content="Book a flight from NYC to London"), AIMessage( content="Searching for available flights…", tool_calls=[ToolCall(name="search_flights", args={"origin": "NYC", "dest": "LON"})], ), ToolMessage(content='[{"flight": "BA178", "price": 620}]'), AIMessage(content="I found BA178 departing tomorrow for $620. Shall I book it?"), ] ) interactions_json = json.dumps(sample.to_dict()) result, latency, _ = pipeline.evaluate_response( response="I found BA178 departing tomorrow for $620. Shall I book it?", prompt="Book a flight from NYC to London", pipeline_interactions=interactions_json, ) print(result.blocked, result.metrics) ``` > Withoutpipeline_interactionsthe guard falls back gracefully to evaluating the single > prompt/response pair — useful during development before you have a live agent. ### From a plain Python dict Use `ModerationPipeline.from_dict` when your configuration is already in dict form (e.g. loaded from JSON, fetched from an API, or assembled programmatically). The dict must follow the same schema as the YAML file. #### Parameters | Parameter | Type | Required | Description | | --- | --- | --- | --- | | config | dict | Yes | Guard configuration dictionary following the YAML schema | | model_dir | str \\| None | No | Base directory used to resolve relative asset paths (e.g. NeMo guardrails .co flow files). Defaults to os.getcwd() | ``` import os from datarobot_dome.api import ModerationPipeline os.environ["DATAROBOT_ENDPOINT"] = "" os.environ["DATAROBOT_API_TOKEN"] = "" # os.environ["TARGET_NAME"] = "resultText" # optional — see [Environment variables](#environment-variables) for resolution order {: #osenvirontarget_name-resulttext-optional-see-10-for-resolution-order } pipeline = ModerationPipeline.from_dict( { "targets": [ { "target": "_default", "guards": [ { "name": "Token Count", "type": "ootb", "ootb_type": "token_count", "stage": "prompt", } ], } ] }, model_dir="/path/to/nemo_guardrails_dir", # optional; only needed for NeMo guards ) result, latency, prescore_df = pipeline.evaluate_prompt("Hello") print(result.metrics) ``` ### From a Pydantic config object Use `ModerationPipeline.from_config` to build the configuration entirely in Python — no YAML file required. This is useful for dynamic configurations, programmatic guard registration, or when embedding moderation in a larger application. #### Parameters | Parameter | Type | Required | Description | | --- | --- | --- | --- | | config | ModerationConfig | Yes | A fully-constructed ModerationConfig Pydantic object | | model_dir | str \\| None | No | Base directory used to resolve relative asset paths (e.g. NeMo guardrails .co flow files). Defaults to os.getcwd() | All schema types are importable from `datarobot_dome.schema`: ``` from datarobot_dome.schema import ( ModerationConfig, TargetBlock, # Guard subtypes — pick the matching one per guard OOTBGuardSchema, ModelGuardSchema, NemoGuardrailsSchema, NemoEvaluatorSchema, # Nested schemas used inside guards AdditionalGuardConfigSchema, InterventionSchema, InterventionConditionSchema, ModelInfoSchema, ) ``` #### Schema type → guard type mapping | Guard YAML type | Pydantic class | | --- | --- | | ootb | OOTBGuardSchema | | model | ModelGuardSchema | | nemo_guardrails | NemoGuardrailsSchema | | nemo_evaluator | NemoEvaluatorSchema | #### LLM Gateway example — hate speech / guideline adherence ``` import os from datarobot_dome.api import ModerationPipeline from datarobot_dome.schema import ( AdditionalGuardConfigSchema, InterventionSchema, ModerationConfig, OOTBGuardSchema, TargetBlock, ) os.environ["DATAROBOT_ENDPOINT"] = "https://app.datarobot.com/api/v2" os.environ["DATAROBOT_API_TOKEN"] = "" # os.environ["TARGET_NAME"] = "resultText" # optional — see [Environment variables](#environment-variables) for resolution order {: #osenvirontarget_name-resulttext-optional-see-10-for-resolution-order } config = ModerationConfig( targets=[ TargetBlock( target="_default", guards=[ OOTBGuardSchema( type="ootb", name="Hate Speech", stage="response", ootb_type="agent_guideline_adherence", llm_type="llmGateway", llm_gateway_model_id="azure/gpt-4o-2024-11-20", additional_guard_config=AdditionalGuardConfigSchema( agent_guideline=( "The response must not contain hate speech, slurs, or content " "that demeans people based on race, religion, gender, nationality, " "or any other protected characteristic." ) ), intervention=InterventionSchema( action="report", conditions=[], ), ) ], ) ] ) # Pass model_dir when your config references NeMo guardrails flow files: {: #pass-model_dir-when-your-config-references-nemo-guardrails-flow-files } # pipeline = ModerationPipeline.from_config(config, model_dir="/path/to/nemo_guardrails_dir") {: #pipeline-moderationpipelinefrom_configconfig-model_dirpathtonemo_guardrails_dir } text = "People from that group are living in France." result, latency, postscore_df = pipeline.evaluate_response(response=text, prompt="Describe this text.") score = result.metrics.get("agent_guideline_adherence_score") print(f"score={score} latency={latency:.3f}s") ``` #### Model guard example ``` import os from datarobot_dome.api import ModerationPipeline from datarobot_dome.schema import ( InterventionConditionSchema, InterventionSchema, ModerationConfig, ModelGuardSchema, ModelInfoSchema, TargetBlock, ) os.environ["DATAROBOT_ENDPOINT"] = "" os.environ["DATAROBOT_API_TOKEN"] = "" # os.environ["TARGET_NAME"] = "resultText" # optional — see [Environment variables](#environment-variables) for resolution order {: #osenvirontarget_name-resulttext-optional-see-10-for-resolution-order } config = ModerationConfig( targets=[ TargetBlock( target="_default", guards=[ ModelGuardSchema( type="model", name="Toxicity", stage="prompt", deployment_id="", model_info=ModelInfoSchema( input_column_name="text", target_name="toxicity_toxic_PREDICTION", target_type="Binary", class_names=[], ), intervention=InterventionSchema( action="block", message="Toxic content blocked.", conditions=[ InterventionConditionSchema(comparand=0.5, comparator="greaterThan") ], ), ) ], ) ] ) pipeline = ModerationPipeline.from_config(config) ``` ### Streaming pipeline `evaluate_full_pipeline_stream_async` is the primary high-level API for streaming. It encapsulates prescore evaluation, the thread/queue bridge to `ModerationIterator`, and postscore guard execution — callers supply only a prompt and a streaming LLM callable. #### Method signatures | Method | When to use | | --- | --- | | evaluate_full_pipeline_stream_async(prompt, llm_callable) | Preferred. Hides all internal state — no prescore_df required. | | stream_response_async(completion, *, prompt, prescore_df, prescore_latency) | Advanced: when you need to inspect the EvaluationResult from prescore before starting the LLM stream (e.g. to act on a REPLACE result). | #### evaluate_full_pipeline_stream_async parameters | Parameter | Type | Required | Description | | --- | --- | --- | --- | | prompt | str | Yes | The user prompt | | llm_callable | Callable[[str], AsyncIterator[ChatCompletionChunk]] | Yes | Sync callable that receives the (possibly sanitized) effective prompt and returns an async iterator of chunks. Called only when the prompt is not blocked. | #### Chunk signals | finish_reason | Meaning | | --- | --- | | None or "stop" | Normal chunk — content is in chunk.choices[0].delta.content | | "content_filter" | A guard intervened. delta.content holds the block message. The LLM was never called if this is the first (and only) chunk. | #### Example ``` import asyncio import os from datarobot_dome.api import ModerationPipeline from datarobot_dome.schema import ( InterventionSchema, ModerationConfig, OOTBGuardSchema, TargetBlock, ) os.environ["DATAROBOT_ENDPOINT"] = "" os.environ["DATAROBOT_API_TOKEN"] = "" pipeline = ModerationPipeline.from_config( ModerationConfig( targets=[ TargetBlock( target="_default", guards=[ OOTBGuardSchema( name="Prompt Token Limit", type="ootb", ootb_type="token_count", stage="prompt", intervention=InterventionSchema( action="block", conditions=[{"comparator": "greaterThan", "comparand": 200}], message="Prompt too long.", ), ), ], ) ] ) ) async def my_llm_stream(prompt: str): """Wrap a sync OpenAI stream as an async iterator.""" import openai client = openai.OpenAI( api_key=os.environ["DATAROBOT_API_TOKEN"], base_url=f"{os.environ['DATAROBOT_ENDPOINT']}/genai/llmgw", ) for chunk in client.chat.completions.create( model="azure/gpt-4o-2024-11-20", messages=[{"role": "user", "content": prompt}], stream=True, ): yield chunk async def run(prompt: str) -> None: print(f"Prompt: {prompt!r}") async for chunk in pipeline.evaluate_full_pipeline_stream_async(prompt, my_llm_stream): finish_reason = chunk.choices[0].finish_reason content = chunk.choices[0].delta.content if finish_reason == "content_filter": print(f"[BLOCKED] {content}") return if content: print(content, end="", flush=True) print() asyncio.run(run("What is DataRobot?")) ``` #### Advanced: stream_response_async Use when you need the prescore `EvaluationResult` before streaming begins: ``` result, latency, prescore_df = await pipeline.evaluate_prompt_async(prompt) if result.blocked: # handle block before ever calling the LLM return result.blocked_message effective = result.replacement if result.replaced else prompt async for chunk in pipeline.stream_response_async( my_llm_stream(effective), prompt=effective, prescore_df=prescore_df, # must come from evaluate_prompt_async prescore_latency=latency, ): ... ``` ### With DRUM Place `moderation_config.yaml` alongside your custom model code, then: ``` drum score --verbose \ --code-dir ./ \ --target-type textgeneration \ --input ./input.csv \ --runtime-params-file values.yaml ``` ## Testing guide Set these environment variables before running any test (see [Environment variables](https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/moderations-guardrails.html#environment-variables) for details): ``` export DATAROBOT_ENDPOINT="https://app.datarobot.com/api/v2" export DATAROBOT_API_TOKEN="your-token" export TARGET_NAME="resultText" ``` Guards fall into four groups based on the credentials they require: | Group | Guard types | Extra credentials needed | | --- | --- | --- | | Local | token_count, rouge_1, cost, custom_metric | (none beyond the base vars above) | | DataRobot deployment | type: model, any ootb with llm_type: datarobot or llm_type: llmGateway | Only DATAROBOT_API_TOKEN; provide a real deployment_id | | External LLM provider | Any ootb with llm_type: openAi, azureOpenAi, google, amazon, nim | Provider-specific env var (see Environment variables) | | NeMo | type: nemo_guardrails, type: nemo_evaluator | Provider key for NeMo Guardrails; DATAROBOT_API_TOKEN for NeMo Evaluator | See [Guard types](https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/moderations-guardrails.html#guard-types) for complete YAML examples per guard type and [Using the config in Python](https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/moderations-guardrails.html#using-the-config-in-python) for Python usage patterns. ## Environment variables ### Always required | Variable | Description | | --- | --- | | DATAROBOT_ENDPOINT | DataRobot instance URL, e.g. https://app.datarobot.com/api/v2 | | DATAROBOT_API_TOKEN | DataRobot API token | | TARGET_NAME | The name of the DataFrame column that holds the LLM response text (e.g. resultText). Resolution order for the response column (highest to lowest priority): (1) DRUM deployment target_name (always wins when MLOPS_DEPLOYMENT_ID is set), (2) TARGET_NAME env var, (3) response_column_name in the config file, (4) built-in default "completion". DRUM sets this automatically; in standalone Python you can set it here or declare response_column_name in the YAML/ModerationConfig — but the env var takes precedence if both are provided. | | DISABLE_MODERATION | Set to true to disable all guards at runtime. | ### OTel tracing (optional) OTel traces are emitted whenever `OTEL_EXPORTER_OTLP_ENDPOINT` is set. The remaining two variables are optional — their corresponding request headers are omitted when the variable is absent, which allows traces to be forwarded to an unauthenticated local OTLP collector such as the `af-component-agent-playground` UI without needing credentials. | Variable | Required | Description | | --- | --- | --- | | OTEL_EXPORTER_OTLP_ENDPOINT | ✅ | Base URL of the OTLP HTTP collector, e.g. http://localhost:4318. The library appends /v1/traces automatically. | | OTEL_SERVICE_NAME | ❌ | Adds X-DataRobot-Entity-Id to trace requests. Required when routing to the DataRobot production collector; omit for local collectors. | | OTEL_COLLECTOR_TOKEN | ❌ | Adds Authorization: Bearer to trace requests. Required for production/deployed collectors; omit for local collectors. | Local playground example: ``` export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" # OTEL_SERVICE_NAME and OTEL_COLLECTOR_TOKEN are not needed ``` Production example: ``` export OTEL_EXPORTER_OTLP_ENDPOINT="https://collector.datarobot.com" export OTEL_SERVICE_NAME="deployment-abc123" export OTEL_COLLECTOR_TOKEN="my-token" ``` ### deepeval telemetry The `task_adherence` guard uses `deepeval` internally. By default, moderations opts out of deepeval's usage telemetry — no `.deepeval/` directory is created and no data is sent externally. To opt in, set `enable_deepeval_telemetry: true` in your config (only takes effect when a `task_adherence` guard is present; deepeval is loaded lazily): ``` enable_deepeval_telemetry: true # default: false guards: - name: Task Adherence type: ootb ootb_type: task_adherence stage: response ``` To opt out explicitly via environment variable (e.g. in CI or container environments): ``` export DEEPEVAL_TELEMETRY_OPT_OUT=YES # opt out (library default) unset DEEPEVAL_TELEMETRY_OPT_OUT # opt in ``` ### Credentials for LLM-eval guards using external providers When your guard uses `llm_type: datarobot`, it reuses `DATAROBOT_API_TOKEN` — no extra variable needed. For external providers (OpenAI, Azure OpenAI, Google, AWS), set a guard-specific env var. The variable name is built from the guard's type, stage, and ootb_type: ``` MLOPS_RUNTIME_PARAM_MODERATION_{TYPE}_{STAGE}_{OOTB_TYPE}_{PROVIDER_SUFFIX} ``` | Guard (ootb_type) | Provider | Environment variable | | --- | --- | --- | | task_adherence | OpenAI | MLOPS_RUNTIME_PARAM_MODERATION_OOTB_RESPONSE_TASK_ADHERENCE_OPENAI_API_KEY | | task_adherence | Azure OpenAI | MLOPS_RUNTIME_PARAM_MODERATION_OOTB_RESPONSE_TASK_ADHERENCE_AZURE_OPENAI_API_KEY | | faithfulness | OpenAI | MLOPS_RUNTIME_PARAM_MODERATION_OOTB_RESPONSE_FAITHFULNESS_OPENAI_API_KEY | | faithfulness | Azure OpenAI | MLOPS_RUNTIME_PARAM_MODERATION_OOTB_RESPONSE_FAITHFULNESS_AZURE_OPENAI_API_KEY | | agent_guideline_adherence | Azure OpenAI | MLOPS_RUNTIME_PARAM_MODERATION_OOTB_RESPONSE_AGENT_GUIDELINE_ADHERENCE_AZURE_OPENAI_API_KEY | | agent_guideline_adherence | Google Vertex AI | MLOPS_RUNTIME_PARAM_MODERATION_OOTB_RESPONSE_AGENT_GUIDELINE_ADHERENCE_GOOGLE_SERVICE_ACCOUNT | | agent_goal_accuracy | Azure OpenAI | MLOPS_RUNTIME_PARAM_MODERATION_OOTB_RESPONSE_AGENT_GOAL_ACCURACY_AZURE_OPENAI_API_KEY | | agent_goal_accuracy | AWS Bedrock | MLOPS_RUNTIME_PARAM_MODERATION_OOTB_RESPONSE_AGENT_GOAL_ACCURACY_AWS_ACCOUNT | | nemo_guardrails (prompt) | Azure OpenAI | MLOPS_RUNTIME_PARAM_MODERATION_NEMO_GUARDRAILS_PROMPT_AZURE_OPENAI_API_KEY | Value format per provider: ``` # OpenAI / Azure OpenAI {: #openai-azure-openai } '{"type":"credential","payload":{"credentialType":"api_token","apiToken":"YOUR_KEY"}}' # Google Vertex AI {: #google-vertex-ai } '{"type":"credential","payload":{"credentialType":"gcp","gcpKey":{...}}}' # AWS Bedrock {: #aws-bedrock } '{"type":"credential","payload":{"credentialType":"s3","awsAccessKeyId":"...","awsSecretAccessKey":"...","awsSessionToken":"..."}}' ``` --- # Using the Moderations CLI URL: https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/using-moderations-with-the-cli.html > How to install, authenticate, and run the dr-moderation CLI to manage guards and test moderation pipelines. The `dr-moderation` CLI lets you manage guards and test moderation pipelines from the terminal — no Python code required. ## Installation End-user — the `dr-moderation` binary lands on your `PATH` automatically: ``` pip install 'datarobot-moderations[all]' dr-moderation --help ``` > [!NOTE] Note > Python 3.10 – 3.12 is required. Developer / contributor — Poetry places the binary inside `.venv/bin/`, which is not on your `PATH` until the venv is active. Pick one: ``` poetry shell # Option A: activate for the session poetry run dr-moderation --help # Option B: one-off prefix make cli ARGS="evaluate --help" # Option C: Makefile shortcut ``` ## Authentication Commands that call the DataRobot API need credentials. Set them once per session: ``` export DATAROBOT_ENDPOINT="https://app.datarobot.com/api/v2" export DATAROBOT_API_TOKEN="your-api-token" ``` Or pass them as global flags (flags take precedence over env vars): ``` dr-moderation --endpoint --token ``` ## Commands ### evaluate Evaluate a prompt and/or response through the local `ModerationPipeline`. Supports every guard type including LLM Gateway ( `llm_type: llmGateway`) — no deployment required. The config file must use the Python SDK snake_case schema (see [Moderations guardrails](https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/moderations-guardrails.html) for the full field reference). ``` dr-moderation evaluate [OPTIONS] ``` | Option | Required | Default | Description | | --- | --- | --- | --- | | --config-file FILE | Yes | — | Moderation config YAML (snake_case SDK format) | | --prompt TEXT | No * | — | Prompt text; evaluated against prescore guards | | --response TEXT | No * | — | Response text; evaluated against postscore guards. Also pass --prompt for guards that need both (e.g. faithfulness, task_adherence) | | --as-json | No | false | Emit results as JSON — useful for scripting | * At least one of `--prompt` or `--response` is required. Example output (human-readable): ``` ── Prescore (prompt) ────────────────────────────── Blocked : False Metrics : Prompts_token_count: 4 Latency : 0.05s ``` #### Evaluate-examples Token-count guard on a prompt: ``` dr-moderation evaluate \ --config-file docs/examples/token_count_config.yaml \ --prompt "Hello, world!" ``` LLM Gateway task-adherence guard: ``` dr-moderation evaluate \ --config-file docs/examples/llm_gateway_config.yaml \ --prompt "What is DataRobot?" \ --response "DataRobot is an AI platform." ``` ``` dr-moderation evaluate \ --config-file docs/examples/llm_gateway_config.yaml \ --prompt "What is DataRobot?" \ --response "DataRobot is an AI platform." \ --as-json | jq '.postscore.metrics' ``` Ready-made configs in `docs/examples/`: - token_count_config.yaml — Prompt + response token-count guards - llm_gateway_config.yaml — token-count prompt guard + LLM Gateway task_adherence ### add-guard Add guards to an existing DataRobot custom model. Creates a new custom model version with the guards attached and prints the version ID to stdout. How it works: 1. You create and register a custom model (your LLM) in DataRobot — this gives you a customModelId . 2. You define guards in a camelCase YAML file. 3. add-guard POSTs the config to /guardConfigurations/toNewCustomModelVersion/ . DataRobot creates a new version of the model with the guards and returns the customModelVersionId . 4. Deploy that new version — it will now enforce your guards on every prompt/response. ``` dr-moderation add-guard [OPTIONS] ``` | Option | Required | Default | Description | | --- | --- | --- | --- | | --custom-model-id TEXT | Yes | — | ID of the custom model (find it in the DataRobot UI under Model Workshop → Custom Models) | | --config-file FILE | Yes | — | YAML list of guard configurations (camelCase API format) | | --timeout-sec INTEGER | No | 60 | Per-guard timeout in seconds | | --timeout-action [score\\|block] | No | score | Action on timeout: score passes through; block rejects | Example output: ``` 6797abc123def456789abcde ``` The printed ID is the new `customModelVersionId` — pass it to subsequent API or SDK calls to deploy the version. Examples: ``` # Add guards, capture the new version ID VERSION_ID=$(dr-moderation add-guard \ --custom-model-id 6793e6b2114f17240fa2194c \ --config-file docs/examples/add_guard_config.yaml) echo "New version: ${VERSION_ID}" # Block if any guard exceeds 30 s dr-moderation add-guard \ --custom-model-id 6793e6b2114f17240fa2194c \ --config-file docs/examples/add_guard_config.yaml \ --timeout-sec 30 \ --timeout-action block ``` ### agent a2a connect Verify connectivity to a remote [A2A](https://google.github.io/A2A/) agent by fetching its agent card from `/.well-known/agent.json`. ``` dr-moderation agent a2a connect [OPTIONS] ``` | Option | Required | Description | | --- | --- | --- | | --url TEXT | Yes | Base URL of the remote A2A agent | | --deployment-id TEXT | No | DataRobot deployment ID to verify alongside the agent | Examples: ``` # 1. Start a one-line A2A mock (serves /.well-known/agent.json on port 8765) python3 - << 'EOF' import json from http.server import BaseHTTPRequestHandler, HTTPServer CARD = {"name": "My Agent", "version": "1.0.0", "capabilities": ["moderation"]} class H(BaseHTTPRequestHandler): def do_GET(self): body = json.dumps(CARD).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.end_headers() self.wfile.write(body) def log_message(self, *_): pass HTTPServer(("localhost", 8765), H).serve_forever() EOF & # 2. Connect to it dr-moderation agent a2a connect --url http://localhost:8765 ``` Production examples: ``` # Verify a remote A2A agent is reachable dr-moderation agent a2a connect --url https://my-agent.example.com # Also verify the backing DataRobot deployment dr-moderation agent a2a connect \ --url https://my-agent.example.com \ --deployment-id 6793e6b2114f17240fa2194c ``` ### serve Start a JSON-RPC 2.0 server so that non-Python applications (Java, Go, C#, …) can evaluate prompts and responses through the full moderation pipeline without HTTP/REST overhead or a Python runtime in their own process. Two transports are available: | Transport | How it works | Best for | | --- | --- | --- | | stdio (default) | Caller spawns dr-moderation serve as a subprocess; newline-delimited JSON on stdin/stdout | Single-caller, zero network setup | | ws | aiohttp WebSocket server; multiple callers share one long-running instance | Containerised / multi-caller deployments | ``` dr-moderation serve [OPTIONS] ``` | Option | Required | Default | Description | | --- | --- | --- | --- | | --transport [stdio\\|ws] | No | stdio | Transport backend | | --config-file FILE | No | — | Pre-load a pipeline YAML at startup. For ws this pipeline is shared across all connections; for stdio the caller can still send initialize to override it | | --host TEXT | No | 127.0.0.1 | Bind address (ws only) | | --port INTEGER | No | 9000 | Bind port (ws only) | | --log-level [debug\\|info\\|warning\\|error] | No | warning | Logging verbosity — all output goes to stderr, never stdout | All diagnostic output goes to stderr. The stdout stream carries only JSON-RPC messages so callers can parse it without noise. #### Wire format Messages are newline-delimited JSON (one complete JSON object per line, `\n` -terminated). Both requests and responses follow [JSON-RPC 2.0](https://www.jsonrpc.org/specification). Request (caller → server): ``` {"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"config_path": "/path/to/config.yaml"}} ``` Response (server → caller): ``` {"jsonrpc": "2.0", "id": 1, "result": {"ok": true}} ``` #### Methods | Method | Call order | params keys | Description | | --- | --- | --- | --- | | initialize | Before evaluate_* | config_path (string, required) | Load the moderation pipeline from a YAML file. Must be called before any evaluate_* method unless --config-file was passed at startup. Returns {"ok": true} | | evaluate_prompt | After initialize | prompt (string, required) | Run prescore guards and return an EvaluationResult | | evaluate_response | After initialize | response (string, required); prompt (string, optional); pipeline_interactions (string, optional) | Run postscore guards and return an EvaluationResult | | shutdown | Any time | (none) | Signal the server to stop and return {"ok": true}. stdio: server exits after sending the response. ws: closes the current connection; the server process keeps running | #### Complete response example (evaluate_prompt) ``` { "jsonrpc": "2.0", "id": 2, "result": { "blocked": false, "blocked_message": null, "replaced": false, "replacement": null, "metrics": { "Prompts_token_count": 4 }, "latency_sec": 0.012345 } } ``` When a guard blocks content `blocked` is `true`, `blocked_message` holds the guard's configured message, and `latency_sec` is always present. When a `replace` -action guard fires, `replaced` is `true` and `replacement` holds the sanitized text. #### Examples Bash (stdio — interactive test): ``` # Pre-load a config, then evaluate a prompt dr-moderation serve --config-file moderation_config.yaml --transport stdio <<'EOF' {"jsonrpc":"2.0","id":1,"method":"evaluate_prompt","params":{"prompt":"Hello, world!"}} {"jsonrpc":"2.0","id":2,"method":"shutdown","params":{}} EOF ``` Python (subprocess, `stdio`): ``` import json import subprocess import sys proc = subprocess.Popen( [sys.executable, "-m", "datarobot_dome.cli", "serve", "--transport", "stdio", "--config-file", "moderation_config.yaml"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, # discard diagnostics; redirect to sys.stderr to surface them text=True, bufsize=1, ) def rpc(method, params, *, req_id): msg = json.dumps({"jsonrpc": "2.0", "id": req_id, "method": method, "params": params}) proc.stdin.write(msg + "\n") proc.stdin.flush() # Skip any stdout lines that are not valid JSON (startup messages, warnings). while True: line = proc.stdout.readline() try: return json.loads(line) except json.JSONDecodeError: continue result = rpc("evaluate_prompt", {"prompt": "Hello, world!"}, req_id=1) print(result["result"]) rpc("shutdown", {}, req_id=2) proc.wait() ``` Go (stdio): ``` package main import ( "bufio" "encoding/json" "fmt" "os/exec" ) func main() { cmd := exec.Command("dr-moderation", "serve", "--transport", "stdio", "--config-file", "moderation_config.yaml") stdin, _ := cmd.StdinPipe() stdout, _ := cmd.StdoutPipe() _ = cmd.Start() scanner := bufio.NewScanner(stdout) send := func(req any) { b, _ := json.Marshal(req) fmt.Fprintln(stdin, string(b)) } recv := func() map[string]any { // Skip non-JSON lines (startup messages, log output on stdout) for scanner.Scan() { var m map[string]any if err := json.Unmarshal(scanner.Bytes(), &m); err == nil { return m } } return nil } send(map[string]any{"jsonrpc": "2.0", "id": 1, "method": "evaluate_prompt", "params": map[string]any{"prompt": "Hello, world!"}}) resp := recv() fmt.Println(resp["result"]) send(map[string]any{"jsonrpc": "2.0", "id": 2, "method": "shutdown", "params": map[string]any{}}) cmd.Wait() } ``` Java (stdio): ``` import com.fasterxml.jackson.databind.ObjectMapper; import java.io.*; import java.util.Map; public class ModerationClient { public static void main(String[] args) throws Exception { ProcessBuilder pb = new ProcessBuilder( "dr-moderation", "serve", "--transport", "stdio", "--config-file", "moderation_config.yaml"); pb.redirectError(ProcessBuilder.Redirect.DISCARD); Process proc = pb.start(); ObjectMapper mapper = new ObjectMapper(); var writer = new PrintWriter(new BufferedWriter( new OutputStreamWriter(proc.getOutputStream())), true); var reader = new BufferedReader( new InputStreamReader(proc.getInputStream())); // Send request String req = mapper.writeValueAsString(Map.of( "jsonrpc", "2.0", "id", 1, "method", "evaluate_prompt", "params", Map.of("prompt", "Hello, world!"))); writer.println(req); // Read response — skip non-JSON lines String line; while ((line = reader.readLine()) != null) { try { var resp = mapper.readValue(line, Map.class); System.out.println(resp.get("result")); break; } catch (Exception ignored) {} } writer.println(mapper.writeValueAsString(Map.of( "jsonrpc", "2.0", "id", 2, "method", "shutdown", "params", Map.of()))); proc.waitFor(); } } ``` C# (stdio): ``` using System.Diagnostics; using System.Text.Json; var proc = new Process { StartInfo = new ProcessStartInfo("dr-moderation") { Arguments = "serve --transport stdio --config-file moderation_config.yaml", RedirectStandardInput = true, RedirectStandardOutput = true, RedirectStandardError = true, UseShellExecute = false, } }; proc.Start(); _ = proc.StandardError.ReadToEndAsync(); // drain stderr on a background task void Send(object req) => proc.StandardInput.WriteLine(JsonSerializer.Serialize(req)); JsonElement Recv() { // Skip non-JSON lines (startup messages, warnings) while (true) { var line = proc.StandardOutput.ReadLine() ?? throw new EndOfStreamException(); try { return JsonDocument.Parse(line).RootElement; } catch { } } } Send(new { jsonrpc = "2.0", id = 1, method = "evaluate_prompt", @params = new { prompt = "Hello, world!" } }); var resp = Recv(); Console.WriteLine(resp.GetProperty("result")); Send(new { jsonrpc = "2.0", id = 2, method = "shutdown", @params = new { } }); proc.WaitForExit(); ``` WebSocket ( `ws` transport): ``` # Start the server (runs until killed) dr-moderation serve --transport ws --host 127.0.0.1 --port 9000 \ --config-file moderation_config.yaml # In another terminal — connect with any WebSocket client (e.g. websocat) echo '{"jsonrpc":"2.0","id":1,"method":"evaluate_prompt","params":{"prompt":"Hello"}}' \ | websocat ws://127.0.0.1:9000 ``` ## YAML schema quick reference The two commands use different schemas — they are not interchangeable: | Command | Format | Key fields | | --- | --- | --- | | add-guard | DataRobot API — camelCase | ootbType, stages (list), intervention | | evaluate | Python SDK — snake_case | ootb_type, stage (string or list), llm_type, llm_gateway_model_id | ### add-guard config (camelCase) Sent directly to `/guardConfigurations/toNewCustomModelVersion/`. The file must be a YAML list. ``` - name: Prompt Token Count type: ootb ootbType: token_count stages: [prompt] intervention: action: report allowedActions: [report, block] message: " " sendNotification: false conditions: [] ``` | Field | Required | Notes | | --- | --- | --- | | name | Yes | Unique per config | | type | Yes | ootb · guardModel · userModel · nemo | | stages | Yes | List: [prompt], [response], or [prompt, response] | | ootbType | When type: ootb | token_count, faithfulness, rouge_1, etc. | | modelInfo | When type: guardModel | inputColumnName, outputColumnName, targetType, classNames | | intervention | No | action, conditions, message; omit to measure only | ### evaluate config (snake_case) Consumed by `ModerationPipeline.from_yaml`. For the full field reference see [Moderations guardrails](https://docs.datarobot.com/en/docs/api/code-first-tools/moderations-library/moderations-guardrails.html). The key difference from `add-guard`: use `llm_type: llmGateway` with `llm_gateway_model_id` — no `deployment_id` needed: ``` - name: Task Adherence type: ootb ootb_type: task_adherence stage: response llm_type: llmGateway llm_gateway_model_id: "azure/gpt-4o-2024-11-20" intervention: action: block message: "Response does not address the task." conditions: - comparator: lessThan comparand: 0.5 ``` ## Exit codes | Code | Meaning | | --- | --- | | 0 | Success | | 1 | Runtime error (API error, bad YAML, connection refused) | | 2 | Invalid CLI usage (missing required option, unknown value) | Non-zero exits write a descriptive message to stderr. --- # Apache Spark API for Scoring Code URL: https://docs.datarobot.com/en/docs/api/code-first-tools/sc-apache-spark.html > Learn how to use the Spark API for Scoring Code, a library that integrates Scoring Code JARs into Spark clusters. The Spark API for Scoring Code library integrates DataRobot Scoring Code JARs into Spark clusters. It is available as a [PySpark API](https://docs.datarobot.com/en/docs/api/code-first-tools/sc-apache-spark.html#pyspark-api) and a [Spark Scala API](https://docs.datarobot.com/en/docs/api/code-first-tools/sc-apache-spark.html#spark-scala-api). In previous versions, the Spark API for Scoring Code consisted of multiple libraries, each supporting a specific Spark version. Now, one library supports all supported Spark versions. The following Spark versions support this feature: - Spark 2.4.1 or greater - Spark 3.x > [!NOTE] Important > Spark must be compiled for Scala 2.12. For a list of the deprecated, Spark version-specific libraries, see the [Deprecated Spark libraries](https://docs.datarobot.com/en/docs/api/code-first-tools/sc-apache-spark.html#deprecated-spark-libraries) section. ## PySpark API The PySpark API for Scoring Code is included in the [datarobot-predict](https://pypi.org/project/datarobot-predict/) Python package, released on PyPI. The PyPI project description contains documentation and usage examples. ## Spark Scala API The Spark Scala API for Scoring Code is published on Maven as [scoring-code-spark-api](https://central.sonatype.com/artifact/com.datarobot/scoring-code-spark-api). For more information, see the [API reference documentation](https://javadoc.io/doc/com.datarobot/scoring-code-spark-api_3.0.0/latest/com/datarobot/prediction/spark30/Predictors$.html). Before using the Spark API, you must add it to the Spark classpath. For `spark-shell`, use the `--packages` parameter to load the dependencies directly from Maven: ``` spark-shell --conf "spark.driver.memory=2g" \ --packages com.datarobot:scoring-code-spark-api:VERSION \ --jars model.jar ``` ### Score a CSV file The following example illustrates how you can load a CSV file into a Spark DataFrame and score it: ``` import com.datarobot.prediction.sparkapi.Predictors val inputDf = spark.read.option("header", true).csv("input_data.csv") val model = Predictors.getPredictor() val output = model.transform(inputDf) output.show() ``` ### Load models at runtime The following examples illustrate how you can load a model's JAR file at runtime instead of using the spark-shell `--jars` parameter: **From DataRobot:** Define the `PROJECT_ID`, the `MODEL_ID`, and your `API_TOKEN`. ``` val model = Predictors.getPredictorFromServer( "https://app.datarobot.com/projects/PROJECT_ID/models/MODEL_ID/blueprint","API_TOKEN") ``` **From HDFS filesystem:** Define the path to the model JAR file and the `MODEL_ID`. ``` val model = Predictors.getPredictorFromHdfs("path/to/model.jar", spark, "MODEL_ID") ``` ### Time series scoring The following examples illustrate how you can perform time series scoring with the `transform` method, just as you would with non-time series scoring. In addition, you can customize the time series parameters with the `TimeSeriesOptions` builder. If you don't provide additional arguments for a time series model through the `TimeSeriesOptions` builder, the `transform` method returns forecast point predictions for an auto-detected forecast point: ``` val model = Predictors.getPredictor() val forecastPointPredictions = model.transform(timeSeriesDf) ``` To define a forecast point, you can use the `buildSingleForecastPointRequest()` builder method: ``` import com.datarobot.prediction.TimeSeriesOptions val tsOptions = new TimeSeriesOptions.Builder().buildSingleForecastPointRequest("2010-12-05") val model = Predictors.getPredictor(modelId, tsOptions) val output = model.transform(inputDf) ``` To return historical predictions, you can define a start date and end date through the `buildForecastPointRequest()` builder method: ``` val tsOptions = new TimeSeriesOptions.Builder().buildForecastDateRangeRequest("2010-12-05", "2011-01-02") ``` For a complete reference, see [TimeSeriesOptions javadoc](https://javadoc.io/doc/com.datarobot/datarobot-prediction/latest/com/datarobot/prediction/TimeSeriesOptions.Builder.html). ## Deprecated Spark libraries Support for Spark versions earlier than 2.4.1 or Spark compiled for Scala earlier than 2.12 is deprecated. If necessary, you can access deprecated libraries published on Maven Central; however, they will not receive any further updates. The following libraries are deprecated: | Name | Spark version | Scala version | | --- | --- | --- | | scoring-code-spark-api_1.6.0 | 1.6.0 | 2.10 | | scoring-code-spark-api_2.4.3 | 2.4.3 | 2.11 | | scoring-code-spark-api_3.0.0 | 3.0.0 | 2.12 | --- # FIRE feature selection URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/adv-analytics-tools/fire.html > Learn about the benefits of Feature Impact Rank Ensembling (FIRE)—a method of advanced feature selection that uses a median rank aggregation of feature impacts across several models created during a run of Autopilot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/feature_reduction_with_fire/feature_reduction_with_fire.ipynb) At the heart of machine learning is the "art" of providing a model with the features or variables that are useful for making good predictions. Including redundant or extraneous features can lead to overly complex models that have less predictive power. Striking the right balance is known as feature selection.This page proposes a new, novel method for feature selection, based on using the [feature impact](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/feature-impact-classic.html) scores (also known as feature importance scores in the wider industry) from several different models, which leads to a more robust and powerful result. This accelerator outlines how feature importance rank ensembling (FIRE) can be used to reduce the number of features, based on the feature impact scores DataRobot returns, while maintaining predictive performance. During feature selection, data scientists try to keep "the three Rs" in mind: - Relevant : To reduce generalization risk, the features should be relevant to the business problem at hand. - Redundant : Avoid the use of redundant features—they weaken the interpretability of the model and its predictions. - Reduction : Fewer features mean less complexity, which translates to less time required for model training or inference. Using fewer features decreases the risk of overfitting and may even boost model performance. The chart below shows an example of how feature selection is used to improve a model’s performance. As the number of features are reduced from 501 to 13, the model’s performance improves, as indicated by a higher area under the curve (AUC). This visualization is known as a feature selection curve. ## Feature selection approaches There are three approaches to feature selection. Filter methods select features on the basis of statistical tests. DataRobot users often do this by filtering a dataset from 10,000 features to 1,000 using the feature impact score. This score is based on the alternating conditional expectations (ACE) algorithm and conceptually shows the correlation between the target and the feature. The features are ranked and the top features are retained. One limitation of the DataRobot feature impact score is that it only accounts for the relationship between that feature in isolation and the target. Embedded methods are algorithms that incorporate their own feature selection process. DataRobot uses embedded methods in approaches that include ElasticNet and a proprietary machine learning algorithm, [Eureqa](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/describe/eureqa-classic.html). Wrapper methods are model-agnostic and typically include modeling on a subset of features to help identify which are most impactful. Wrapper methods are widely used and include forward selection, backward elimination, recursive feature elimination, and more sophisticated stochastic techniques, such as random hill climbing and simulated annealing. While wrapper methods tend to provide a more optimal feature list than filter or embedded methods, they are more time-consuming, especially on datasets with hundreds of features. The recursive feature elimination wrapper method is widely used in machine learning to reduce the feature list. A common criteria for removing features is to use the feature impact score, calculated via permutation impact, to remove features with the worst scores and then build a new model. The recursive feature selection approach was used to build the feature selection curve in the chart above. ## Feature importance rank ensembling Building on the recursive feature elimination approach, DataRobot combines the feature impact of multiple diverse models. This approach, known as model ensembling, is based on aggregating ranks of features using the [Feature Impact](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/experiments/experiment-insights/feature-impact.html) score from several Leaderboard blueprints, as described below. You can apply model ensembling, which provides improved accuracy and robustness, to feature selection. Selecting lists of important features from multiple models and combining them in a way that produces more a robust feature list is the foundation of feature importance rank ensembling (FIRE). While there are many ways to aggregate the results, the following general steps are recommended. See the accelerator for details on completing each step: 1. Calculate feature impact for the top N Leaderboard models against the selected metric. You can calculate feature impact using permutation or SHAP impact. 2. For each model with computed feature impact, get the feature ranking. 3. Compute the median rank of each feature by aggregating the ranks of the features across all models. 4. Sort the aggregated list by the computed median rank. 5. Define the threshold number of features to select. 6. Create a feature list based on the newly selected features. To understand the effect of aggregating features, the graphic below shows the variation in feature impact across four different models trained on the readmission dataset. The aggregated feature impact is derived from four models: - LightGBM - XGBoost - Elastic net linear model - Keras deep learning model As indicated by their high `Normalized Impact` score, the features at the top have consistently performed well across many models. The features at the bottom consistently have little signal (they perform poorly across many models). Some features with wide ranges, like `num_lab_procedures` and `diag_x_desc`, performed well in some models, but not in others. Due to multicollinearity and the inherent nature of models, you see variation. That is, linear models are good at finding linear relationships while tree-based models are good at finding nonlinear relationships. Ensembling the feature impact scores helps identify which features are most important in the dataset views of each model. By iterating with FIRE, you can continue to reduce the feature list and build a feature selection curve. FIRE works best when you use models that have good performance to ensure that the feature impact is useful. ## Results The example below shows results on some wider datasets, several internal for illustrative purposes but also two publicly available—Madelon and KDD 1998. Use the AI accelerator linked at the top of this page to try this. - AE is an internal dataset with 374 features and 25,000 rows. - AF is an internal dataset with 203 features and 400,000 rows. - G is an internal dataset with 478 features and 2,500 rows. - IH is an internal dataset with 283 features and 200,000 rows. - KDD 1998 is a publicly available dataset with 477 features and 76,000 rows. - Madelon is a publicly available dataset with 501 features and 2,000 rows. The example uses Autopilot to return these results and show the scores of the best-performing model. The metrics were selected based on the type of problem and distribution of the target. The example then uses Autopilot to build competing models. Lastly, it uses FIRE to develop new feature lists. The results show the performance of the feature list on the best-performing model along with the standard deviation using 10-fold cross-validation. For [feature lists](https://docs.datarobot.com/en/docs/classic-ui/modeling/build-models/build-basic/feature-lists.html#automatically-created-feature-lists), `Informative Features` is the default list that includes all features that pass a "reasonableness" check.`DR Reduced Features` is a one-time reduced feature list using permutation impact. FIRE uses the feature impact with the median rank aggregation approach using an adaptive threshold (the N features that possess 95% of total feature impact). The table below shows feature selection results on six wide datasets. The bold formatting in the result rows indicates the best-performing result in terms of the mean cross-validation score (lower values indicate a better score). The chart below compares the performance of feature selection methods. Across all of these datasets, FIRE consistently had similar or better performance than the use of all features. It even outperformed the `DR Reduced Features` method by reducing the feature set without any loss in accuracy. For the Madelon dataset, by looking at the feature selection curve in the graphic at the top of the page, you can see how reducing the number of features provided better performance. As FIRE parsed down features from 501 to 18, the model’s performance improved. Note that if you use FIRE, you must build more models during model training so that you have feature impact across diverse models for ensembling. The information from these other models is very useful for feature selection. ## Conclusion Improved accuracy and parsimony are possible when you use the FIRE method for feature selection. There are many variations left to validate: feature impact (SHAP, permutation), choice of models, perturbing the data or model, and the method for aggregating (median rank, unnormalized impact). --- # Time series to images URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/adv-analytics-tools/gramian.html > Generate advanced features used for high frequency data use cases. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/audio_and_sensors-gramian_angular_fields_for_high_freq_data_to_images/high_frequency_data_classification_using_gramian_angular_fields.ipynb) Prerequisites: [PYTS library](https://pyts.readthedocs.io/) Traditional feature engineering methods like time aware aggregation and spectrograms can have limitations. Spectrograms cannot capture correlations between each segment of the signal with other segments of the signal. If you try to do this with tabular aggregates it becomes a high dimensionality problem. Gramian Angular Field images of signal data can solve the above problem using a matrix which can be used with computer vision models easily without the limitations of dimensionality. --- # Advanced analytics and tools URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/adv-analytics-tools/index.html > Advanced usage of the DataRobot API that you can add to your experiment workflow. | Topic | Description | | --- | --- | | FIRE feature selection | Learn about the benefits of Feature Impact Rank Ensembling (FIRE)—a method of advanced feature selection that uses a median rank aggregation of feature impacts across several models created during a run of Autopilot. | | Time series to images | Generate advanced features used for high frequency data use cases. | | Use case dependencies | Use an application that allows you to view dependency graphs for DataRobot Use Cases. | | Acoustic data with Visual AI | Generate image features in addition to aggregate numeric features for high frequency data sources. | | Prediction intervals | Learn about the various methods to generate prediction intervals for any DataRobot model. The methods are rooted in conformal inference (also known as conformal prediction). This accelerator focuses on prediction interval generation for regression targets. | | Robust feature selection | This accelerator introduces an approach to select robust features, use multiple seeds for cross validation, add dummy features to compute the median permutation importance, and then select the most robust dummy features. | | Use case explorer | Provides a template for a project management dashboard that collects all the key artifacts in a Use Case and displays them on a timeline. | | Multi-objective optimization | Build a Streamlit application that uses DataRobot deployments to optimize multiple targets simultaneously. | --- # Use Case dependencies URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/adv-analytics-tools/lineage.html > Use an application that allows you to view dependency graphs for DataRobot Use Cases. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/advanced_ml_and_api_approaches/datarobot-lineage-app) This accelerator uses an application that allows you to view dependency graphs for DataRobot Use Cases. Retrieve a Use Case and see how its assets are related. You can run and use the application with `node.js` and Docker. Navigate to `localhost:8000` or `localhost:8080/apps` to begin using it. Usage requires an API key and endpoint to retrieve and view your Use Cases. --- # Acoustic data with Visual AI URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/adv-analytics-tools/ml-viz.html > Generate image features in addition to aggregate numeric features for high frequency data sources. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/high_frequency_data_classification_using_spectrograms_n_numerics/high_frequency_classification_spectrograms_n_numerics.ipynb) The density of high frequency data presents a challenge for standard machine learning workflows that lack specialized feature engineering techniques to condense the signal, extracting and highlighting its uniqueness. DataRobot's multimodal input capability supports simultaneously leveraging numerics and images, which for this use-case is particularly beneficial for including descriptive spectrograms that enable you to leverage well-established computer vision techniques for complex data understanding. This example notebook shows how to generate image features and aggregate numeric features for high frequency data sources. This approach converts audio wav files from the time domain into the frequency domain to create several types of spectrograms. Statistical numeric features computed from the converted signal add additional descriptors to aid classification of the audio source. --- # Multi-objective optimization URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/adv-analytics-tools/multi-objective-opt.html > Implement a Streamlit application using DataRobot deployments to optimize multiple targets at once and explore Pareto-optimal trade-offs. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/multi_objective_optimization/run_multi_objective_optimization.ipynb) This accelerator demonstrates how to implement a Streamlit application for multi-objective optimization using DataRobot deployments, allowing users to optimize multiple targets simultaneously. The application acts as a simulation environment: it runs many trials, uses DataRobot’s prediction API as the core of each trial, and supports Monte Carlo–style sampling (e.g., Random and Quasi-Monte Carlo samplers), illustrating how to build custom simulations that integrate with DataRobot and external optimization tools. The notebook outlines how to: 1. Create multiple DataRobot projects : Upload training data, configure project settings for each target, and run Autopilot with cross-validation. 2. Build deployments : Select the top-performing model for each target, create registered model versions, and deploy to a prediction server. 3. Set up the Streamlit application : Upload the application to DataRobot, configure optimization parameters (e.g., trial count, objective directions and weighting), and run simulations to view optimization results. The application uses Optuna to suggest parameters and request predictions from the DataRobot API on each trial, then computes and displays the Pareto front. Current capabilities and features include six types of optimization algorithms, adjustable trial count, numexpr support for individual optimization targets, objective variable weighting, parameter reliability statistics, display of hypervolume and target-vs-feature plots, 2D/3D Pareto front display, localization support (EN/JP), and support for both dedicated prediction servers and serverless deployment. The optimization application process flow is shown below: ``` sequenceDiagram participant User participant App as Streamlit App participant Op as Optuna participant DR as DataRobot API User->>App: Adjust settings User->>App: Click "Simulation Start!" App->>DR: Load deployment infos DR-->>App: Return deployment infos loop For each trial App->>Op: Create study Op->>Op: Suggest parameters Op->>DR: Request predictions DR-->>Op: Return predictions Op->>Op: Update study end App->>App: Calculate Pareto front App->>User: Display optimization results ``` --- # Prediction intervals URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/adv-analytics-tools/pred-intervals-inf.html > Designed for DataRobot trial users, experience an end-to-end DataRobot workflow using a use case that predicts flight delays. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/prediction_intervals_via_conformal_inference/prediction_intervals_via_conformal_inference.ipynb) This AI Accelerator demonstrates various ways for generating prediction intervals for any DataRobot model. The methods presented here are rooted in the area of conformal inference (also known as conformal prediction). These types of approaches have become increasingly popular for uncertainty quantification because they do not require strict distributional assumptions to be met in order to achieve proper coverage (i.e., they only require that the testing data is exchangeable with the training data). While conformal inference can be applied across a wide array of prediction problems, the focus in this notebook will be prediction interval generation on regression targets. --- # Robust feature selection URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/adv-analytics-tools/rob-fi.html > This accelerator introduces an approach to select robust features, use multiple seeds for cross validation, add dummy features to compute the median permutation importance, and then select the most robust dummy features. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/Robust_Feature_Impact/Robust_Feature_Impact.ipynb) Machine learning models have biases using small data, and some industries (e.g., healthcare and manufacturing) lack labeled data. In light of this, a good approach is to select robust features to build models. This accelerator introduces an approach to select robust features, use multiple seeds for cross validation, add dummy features to compute the median permutation importance, and select the most robust dummy features. This notebook outlines how to: - Connect to DataRobot. - Create multiple projects by multiple seeds and add dummy features. - Create blender models of top-performing models. - Retrieve modeling permutation importance from the top-performing blender models. - Remove features whose permutation importance is lower than dummy features. --- # Use Case explorer URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/adv-analytics-tools/use-case-explorer.html > Provides a template for a project management dashboard that collects all the key artifacts in a Use Case and displays them on a timeline. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/use_case_explorer) DataRobot provides management capabilities for a variety of personas. Using data across Use Cases, experiments, the model registry, and deployments we are able to report on the full lifecycle of an AI project. This AI Accelerator provides a template for a project management dashboard that collects all the key artifacts in a Use Case and displays them on a timeline. It also allows project managers to define a target date and overall status for each Use Case. The left-hand navigation menu will list all of your current Use Cases along with their defined status. It is likely that the status will be undefined for your Use Cases when using the app for the first time. Clicking on a menu item will take you into a detailed view of the Use Case and the associated assets. It also presents a scrollable timeline that illustrates project progress. From this page you can also define or update the project status and target date. These values are fed into the description field of the Use Case object, so they will also be visible inside the DataRobot platform. --- # AWS SageMaker deployment URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/deploy-sagemaker.html > Learn how to programmatically build a model with DataRobot and export and host the model in AWS SageMaker. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/AWS_sagemaker_deployment/dr_model_sagemaker.ipynb) In this accelerator, you deploy a model that has been built in DataRobot to AWS SageMaker. If you already use SageMaker for hosting models, you can still make use of the powerful features of DataRobot, including AutoML and time series modeling. You can integrate DataRobot into your existing deployment processes. Likewise, you can use this workflow to deploy a DataRobot-built model into another type of environment. In this accelerator you will follow the manual steps that are [outlined in DataRobot's documentation](https://docs.datarobot.com/en/docs/classic-ui/integrations/aws/sagemaker/sc-sagemaker.html#use-scoring-code-with-aws-sagemaker), programmatically build a model with DataRobot, and export and host the model in AWS SageMaker. To assist with the setup of AWS services to run the model, this code provisions any extra items that you may not haven yet set up. Review the lists below of what is created in this AI accelerator. ### AWS - ECR Repository - S3 Bucket - IAM Role for SageMaker - SageMaker inference model - SageMaker endpoint configuration - SageMaker endpoint (for real time predictions) - SageMaker batch transform job (for batch predictions) ### DataRobot - DataRobot AutoML Project - DataRobot AutoML Models - Scoring Code JAR file of AutoML Model Once you have run through the code, you will see how you can leverage the power of DataRobot's automated machine learning capabilities to train a model and then make use of the power of AWS to deploy and host that model in SageMaker. --- # Feature Discovery SQL with Spark URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/fd-sql-spark.html > Run Feature Discovery SQL in a new Spark cluster on Docker by setting up a Spark cluster in Docker, registering custom User Defined Functions (UDFs), and executing complex SQL queries across multiple datasets. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/advanced_ml_and_api_approaches/Using%20Feature%20Discovery%20SQL%20in%20Spark%20clusters) This accelerator outlines an example that runs Feature Discovery SQL in a Docker-based Spark cluster. It walks you through the process of setting up a Spark cluster in Docker, registering custom user-defined functions (UDFs), and executing complex SQL queries for feature engineering across multiple datasets. The same approach can be applied to other Spark environments, such as GCP Dataproc, Amazon EMR, and Cloudera CDP, providing flexibility for running Feature Discovery on various Spark platforms. ## Problem framing Features are commonly split across multiple data assets. Bringing these data assets together can take a lot of work, as it involves joining them and then running machine learning models. It's even more difficult when the datasets are of different granularities, as it requires you to aggregate to join the data successfully. [Feature Discovery](https://docs.datarobot.com/en/docs/classic-ui/data/transform-data/feature-discovery/enrich-data-using-feature-discovery.html) solves this problem by automating the procedure of joining and aggregating your datasets. After you define how the datasets need to be joined, DataRobot handles feature generation and modeling. Feature Discovery uses Spark to perform joins and aggregations, generating Spark SQL at the end of the process. In some cases, you may want to run this Spark SQL in other Spark clusters to gain more flexibility and scalability for handling larger datasets, without the need to load data directly into the DataRobot environment. This approach allows you to leverage external Spark clusters for more resource-intensive tasks. This accelerator provides an example of running Feature Discovery SQL in Docker-based Spark cluster. ## Prerequisites - Install Docker - Install Docker compose - Download required the datasets, UDFs .jar, and an environment file (optional) ## Compatibility - Feature Discovery SQL is compatible with Spark 3.2(.2), Spark 3.4(.1), and Scala 2.12(.15). Using different Spark & Scala versions might lead to errors. - The UDFs .jar and environment files can be obtained from the following locations. Note that environment file is only required if working with Japanese text. - Spark 3.2.2 - Spark 3.4.1 - Specific Spark versions can be obtained from here . ## File overview The file structure is outlined below: ``` . ├── Using Feature Discovery SQL in other Spark clusters.ipynb ├── apps │ ├── DataRobotRunSSSQL.py │ ├── LC_FD_SQL.sql │ ├── LC_profile.csv │ ├── LC_train.csv │ └── LC_transactions.csv ├── data ├── libs │ ├── spark-udf-assembly-0.1.0.jar │ └── venv.tar.gz ├── docker-compose.yml ├── Dockerfile ├── start-spark.sh └── utils.py ``` - Using Feature Discovery SQL in other Spark clusters.ipynb is the notebook providing a framework for running Feature Discovery SQL in a new Spark cluster on Docker. - docker-compose.yml , Dockerfile , and start-spark.sh are files used by Docker to build and start the Docker container with Spark. - utils.py includes a helper function to download datasets and the UDFs jar. - The app directory includes: - Spark SQL (a file with a .sql extension) - Datasets (files with a .csv extension) - Helper function (files with a .py extension) to parse and execute the SQL - The libs directory includes: - A user-defined functions (UDFs) JAR file - An environment file (only required if datasets include Japanese text, which requires a Mecab tokenizer to handle) - Thedatadirectory is empty, as it is used to store the output result - Note that the datasets, UDFs jar, and environment files are initially unavailable. They have to be downloaded, as described in the accelerator. --- # GraphQL integration URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/graphql.html > Connect a GraphQL server to the DataRobot OpenAPI specification using GraphQL Mesh. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/ecosystem_integration_templates/DataRobot-GraphQL) This accelerator outlines how to integrate GraphQL with DataRobot. In this example implementation, a GraphQL server is connecting to the DataRobot OpenAPI specification using GraphQL Mesh, the currently maintained option. This process requires the following software, available either via the command line or as a URL in the browser: - A working DataRobot login with a valid API key. - node - yarn --- # AI integrations and platforms URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/index.html > Integrations with AI platforms and services to enhance your DataRobot experience. | Topic | Description | | --- | --- | | AWS SageMaker deployment | Learn how to programmatically build a model with DataRobot and export and host the model in AWS SageMaker. | | Feature Discovery SQL with Spark | Run Feature Discovery SQL in a new Spark cluster on Docker by setting up a Spark cluster in Docker, registering custom User Defined Functions (UDFs), and executing complex SQL queries across multiple datasets. | | GraphQL integration | Connect a GraphQL server to the DataRobot OpenAPI specification using GraphQL Mesh. | | Amazon Athena workflow | Read in an Amazon Athena table to create a project and deploy a model to make predictions with a test dataset. | | AWS workflow | Work with AWS and DataRobot's Python client to import data, build and evaluate models, and deploy a model into production to make new predictions. | | Azure workflow | Work with Azure and DataRobot's Python client to import data, build and evaluate models, and deploy a model into production to make new predictions. | | Databricks workflow | Build models in DataRobot with data acquired and prepared in a Spark-backed notebook environment provided by Databricks. | | Google Cloud and BigQuery workflow | Use Google Collaboratory to source data from BigQuery, build and evaluate a model using DataRobot, and deploy predictions from that model back into BigQuery and GCP. | | SageMaker workflow | Take an ML model that has been built with DataRobot and deploy it to run within AWS SageMaker. | | Snowflake workflow | Work with Snowflake and DataRobot's Python client to import data, build and evaluate models, and deploy a model into production to make new predictions. | | Performance degradation prediction | Use a predictive framework for managing and maintaining your machine learning models with DataRobot MLOps. | | Snowpark integration | Leverage Snowflake for data storage and Snowpark for deployment, feature engineering, and model scoring with DataRobot. | | SAP Hana workflow | Learn how to programmatically build a model with DataRobot using SAP Hana as the data source. | | Speech recognition integration | Use Whisper to transcribe audio files, process them efficiently, and store the transcriptions in a structured format for further analysis or use. | --- # Amazon Athena workflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/ml-athena.html > Read in an Amazon Athena table to create a project and deploy a model to make predictions with a test dataset. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/AWS_Athena_template/AWS_Athena_End_to_End.ipynb) Being one of the largest cloud providers in the world, AWS has multiple ways of storing data within its cloud. Read more to find out how to integrate DataRobot with your data. In this accelerator integration with Athena, you will create a JDBC data source within DataRobot to connect to Athena and then pull data in via an SQL query. This accelerator notebook covers the following activities: - Read in an Amazon Athena table and upload it to DataRobot's AI Catalog - Create a project with the dataset - Deploy the top-performing model to a DataRobot prediction server - Make batch predictions with a test dataset --- # AWS workflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/ml-aws.html > Work with AWS and DataRobot's Python client to import data, build and evaluate models, and deploy a model into production to make new predictions. Being one of the largest cloud providers in the world, AWS has multiple ways of storing data within its cloud. You can use either of two AI accelerators that allow you to source data from S3 or Athena, build and evaluate a model using DataRobot, and send predictions from that model back to S3. [Access the AI accelerator for S3 on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/AWS_S3_template/Amazon_S3_End_to_End.ipynb) [Access the AI accelerator for AWS Athena on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/AWS_Athena_template/AWS_Athena_End_to_End.ipynb) Each AI accelerator will perform the following steps to help you integrate DataRobot with your data in AWS: - Import data for training: - Using the DataRobot Python API, you will have DataRobot build up to 50 different machine learning models while also evaluating how those models perform on this dataset. --- # Azure workflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/ml-azure.html > Work with Azure and DataRobot's Python client to import data, build and evaluate models, and deploy a model into production to make new predictions. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/Azure_template/Azure_End_to_End.ipynb) DataRobot offers an in-depth API that allows you to produce fully automated workflows in your coding environment of choice. This accelerator shows how to enable end-to-end processing of data stored natively in Azure. In this notebook you'll see how data stored in Azure can be used to train a collection of models on DataRobot. You'll then deploy a recommended model and use DataRobot's batch prediction API to produce predictions and write them back to the source Azure container. This accelerator notebook covers the following activities: - Acquire a training dataset from an Azure storage container - Build a new DataRobot project - Deploy a recommended model - Score via DataRobot's batch prediction API - Write results back to the source Azure container --- # Databricks workflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/ml-databricks.html > Build models in DataRobot with data acquired and prepared in a Spark-backed notebook environment provided by Databricks. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/Databricks_template/Databricks_End_To_End.ipynb) DataRobot features an in-depth API that allows data scientists to produce fully automated workflows in their coding environment of choice. This accelerator shows how to pair the power of DataRobot with the Spark-backed notebook environment provided by Databricks. In this notebook you'll see how data acquired and prepared in a Databricks notebook can be used to train a collection of models on DataRobot. You'll then deploy a recommended model and use DataRobot's exportable Scoring Code to generate predictions on the Databricks Spark cluster. This accelerator notebook covers the following activities: - Acquiring a training dataset. - Building a new DataRobot project. - Deploying a recommended model. - Scoring via Spark using DataRobot's exportable Java Scoring Code. - Scoring via DataRobot's Prediction API. - Reporting monitoring data to the MLOps agent framework in DataRobot. - Writing results back to a new table. --- # Google Cloud and BigQuery workflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/ml-gcp.html > Use Google Collaboratory to source data from BigQuery, build and evaluate a model using DataRobot, and deploy predictions from that model back into BigQuery and GCP. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/GCP_template/GCP%20DataRobot%20End%20To%20End.ipynb) DataRobot can integrate directly into your GCP environment, helping to accelerate your use of machine learning across all of the GCP services. In this notebook accelerator, you can use Google Collaboratory or another notebook environment to source data from BigQuery, build and evaluate an ML model using DataRobot, and deploy predictions from that model back into BigQuery and GCP. This accelerator covers the following: 1. Prepare data and ensure connectivity:In the first section of the notebook, you will load a sample dataset to be used for modeling into BigQuery. Once complete, you will connect your BigQuery data with DataRobot. 2. Build and evaluate a model:Using the DataRobot Python API, you will have DataRobot build close to 50 different machine learning models while also evaluating how those models perform on this dataset. 3. Scoring and hosting:In the final section, the entire dataset will be scored on the new model with prediction data written back to BigQuery for use in your GCP applications. --- # SageMaker workflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/ml-sagemaker.html > Take an ML model that has been built with DataRobot and deploy it to run within AWS SageMaker. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/AWS_sagemaker_deployment/dr_model_sagemaker.ipynb) If you already use SageMaker for hosting models, you can still make use of the powerful features of DataRobot, including AutoML and time series modeling. You can integrate DataRobot into your existing deployment processes. Likewise, you can use this accelerator to deploy a DataRobot-built model in another environment. In this accelerator, you will take an ML model that has been built and refined within DataRobot and deploy it to run within AWS SageMaker. To help with the setup of AWS services to run the model, this code will also help provision any extra items that you may not have set up: ### AWS - ECR Repository - S3 Bucket - IAM Role for SageMaker - SageMaker inference model - SageMaker endpoint configuration - SageMaker endpoint (for real time predictions) - SageMaker batch transform job (for batch predictions) ### DataRobot - DataRobot AutoML Project - DataRobot AutoML Models - Scoring Code JAR file of AutoML Model ## What you will learn - Programmatically go through the end-to-end steps of building a model with DataRobot - Export and host the model in AWS SageMaker --- # End-to-end workflow with SAP Hana URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/ml-sap.html > Learn how to programmatically build a model with DataRobot using SAP Hana as the data source. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/SAP_template/SAP_End_to_End.ipynb) The scope of this accelerator provides instructions on how to use DataRobot's Python client to build a workflow that will use an existing SAP Hana JDBC driver and: - Create credentials - Create the training data source - Create the predictions data source - Create a dataset used to train the models - Create a dataset used to make predictions - Create a project - Create a deployment - Make batch and real-time predictions - Show the total predictions made so far There is also a playbook at the end of this notebook that describes how to create the back end SAP Hana database that will provide the data required. --- # Snowflake workflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/ml-snowflake.html > Work with Snowflake and DataRobot's Python client to import data, build and evaluate models, and deploy a model into production to make new predictions. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/ecosystem_integration_templates/Snowflake_template/Snowflake%20-%20End-to-end%20Ecommerce%20Churn.ipynb) This AI accelerator walks through how to work with Snowflake (as a data source) and DataRobot's Python client to import data, build and evaluate models, and deploy a model into production to make new predictions. More broadly, the DataRobot API is a critical tool for data scientists to accelerate their machine learning projects with automation while integrating the platform's capabilities into their code-first workflows and coding environments of choice. By using this accelerator, you will: - Connect to DataRobot. - Import data from Snowflake into DataRobot. - Create a DataRobot project and run Autopilot. - Select and evaluate the top performing model. - Deploy the recommended model with MLOps model monitoring. - Orchestrate scheduled batch predictions that write results back to Snowflake. --- # Performance degradation prediction URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/perform-degrade.html > Use a predictive framework for managing and maintaining your machine learning models with DataRobot MLOps. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/Prediction%20of%20Model%20Performance%20Degradation%20and%20Service%20Failure/Prediction%20of%20Model%20Performance%20Degradation%20and%20Service%20Failure.ipynb) This notebook is designed to provide a predictive framework for managing and maintaining your machine learning models using DataRobot MLOPs. It focuses on preemptively identifying potential model performance degradation and service failures, enabling your organization to proactively address these issues before they adversely impact operations. This approach ensures the sustained efficiency and reliability of predictions that are integral to your business operations. Early detection of model performance deterioration allows for timely interventions. These interventions could range from adjusting DataRobot's retraining policies to other corrective actions, ensuring the maintenance of optimal model performance. Similarly, predicting potential service infrastructure issues facilitates preemptive maintenance. This not only reduces downtime but also enhances service reliability and provides insights into the root causes of these issues. The notebook demonstrates how to leverage DataRobot MLOps functionality to predict if a machine learning model is likely to degrade within a specific time period and if infrastructure failures may occur. It utilizes DataRobot's Python AI capabilities to collect various characteristics and metrics that DataRobot MLOPs tracks for your deployed model, thereby enabling the construction of a predictive model." This notebook outlines how to: - Establish a connection with DataRobot and access the relevant deployment details - Create a training dataset - Define a target for predicting model performance degradation - Build a TS Model Degradation project - Define a target for predicting Service Failure - Build a TS Service Failure project - Retrieve modeling results from the DataRobot Projects - Augment the original deployment with custom metrics based on these predictions --- # Snowpark integration URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/snowpark-data.html > Leverage Snowflake for data storage and Snowpark for deployment, feature engineering, and model scoring with DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/Snowflake_snowpark_template/Native%20integration%20DataRobot%20and%20Snowflake%20Snowpark-Maximizing%20the%20Data%20Cloud.ipynb) If you or your team have tried to develop and productionalize machine learning models with Snowflake using Python and Snowpark but are looking to level up your end-to-end ML lifecycle on the data cloud, then this AI Accelerator is for you. Depending on your role within the organization, This accelerator can address a number of use cases: - Providing technical personnel with a hosted notebook. - Create an improved developer experience. - Improve monitoring capabilities for models within Snowflake. - Provide guidance and insights for business personnel who want action items: next steps for customers, sales, marketing, and more. DataRobot addresses these exact needs, which can be found in this notebook. In addition, it is compatible with the Snowflake data science stack and DataRobot 9.0 to give you advantages in terms of speed, accuracy, security, and cost-effectiveness. --- # Speech recognition integration URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/ai-integrations-platforms/speech-rec.html > Use Whisper to transcribe audio files, process them efficiently, and store the transcriptions in a structured format for further analysis or use. [Access this AI accelerator on GitHub](https://github.com/datarobot/data-science-scripts/blob/master/accelerators_dev/use_cases_and_horizontal_approaches/Speech%20Recognition.ipynb) This accelerator presents a workflow for transcribing audio files using OpenAI's Whisper model. Whisper is a state-of-the-art speech recognition system designed to handle a wide range of audio types and accents. It is highly effective for converting audio files' spoken language into written text. The workflow includes steps to use Whisper to transcribe audio files, process them efficiently, and store the transcriptions in a structured format for further analysis or use. This can be particularly useful for tasks such as generating subtitles, transcribing meetings, or converting speech from various audio sources into text for machine learning. In this example, you take transcribed data and build a classification model with DataRobot. You use DataRobot for model training, selection, deployment, and to evaluate data for insights. This accelerator demonstrates how to use the Python API client to: - Set up the environment (install and import necessary libraries including Whisper and dependencies). - Securely connect to DataRobot. - Get data (publicly available audio files in this example). - Transcribe audio with Whisper. - Use the transcription to create a classification model in DataRobot. - Retrieve and evaluate model performance and insights. --- # Create and deploy a custom model URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/create-custom-model.html > Create, deploy, and monitor a custom inference model with DataRobot's Python client. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/27a757bcf3b10374e70d4c406058ef9cafd28524/ecosystem_integration_templates/Custom%20Model%20End-to-End%20With%20Compliance%20Docs/Custom%20Model%20End-to-End%20With%20Compliance%20Docs.ipynb) This accelerator outlines how to create, deploy, and monitor a custom inference model with DataRobot's Python client. You can use the Custom Model Workshop to upload a model artifact to create, test, and deploy custom inference models to DataRobot’s centralized deployment hub. --- # Custom blueprints with Composable ML URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/custom-bp-nb.html > Customize models on the Leaderboard using the Blueprint Workshop. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/creating_custom_blueprints/create_custom_blueprint.ipynb) [Composable ML](https://docs.datarobot.com/en/docs/classic-ui/modeling/special-workflows/cml/index.html) allows you to add pre-defined tasks to a blueprint or to insert their own custom code. You're free to add your data science and subject matter expertise to the models you build. This accelerator shows how to customize the models on the Leaderboard via Composable ML's API, the Blueprint Workshop. It covers the following activities: - Access the Blueprint Workshop - Define and train a custom blueprint using the tasks provided by DataRobot - Insert custom code in the form of a CatBoost classifier into the blueprint --- # GraphSAGE custom transformer URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/custom-transform.html > Convert a tabular dataset into a graph representation, train a GraphSAGE-based neural network, and package the solution as a DataRobot custom transformer. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/GDL%20Featurizer/GDL%20Featurizer.ipynb) Tabular data is one of the most common ways data is represented in machine learning. However, it is not the only structure that can be used. Many real-world problems involve relationships between entities that can be better captured using graph structures. Graph data represents entities as nodes and relationships as edges, making it a powerful tool for capturing relational dependencies. Common use cases for graph-based learning include social networks, recommendation systems, fraud detection, and molecular property prediction. In these applications, using [geometric deep learning](https://graphics.stanford.edu/courses/cs233-18-spring/ReferencedPapers/GCNN_Geometric%20deep%20learning-%20going%20beyond%20Euclidean%20data.pdf) (i.e., the application of deep learning approaches on non-Euclidean data like graphs) techniques have grown in popularity in recent years. Deep learning is particularly well-suited for studying this type of information due to their ability to learn representations automatically, especially when it comes to unstructured data. Despite its advantages, graph-based learning techniques are often overlooked for traditional tabular data. This is potentially due to the underlying question: how do you represent tabular data into a graph? Thankfully, methods like [k-Nearest Neighbors (kNN)](https://en.wikipedia.org/wiki/Nearest_neighbor_graph) graphs exist that can do much of the heavy lifting for you. In this accelerator, explore how geometric deep learning can be leveraged to extract graph-based features to enrich datasets for supervised tasks. You can achieve this by: - Converting a tabular dataset into a graph representation using kNN graphs - Training a GraphSAGE -based neural network to generate unsupervised node embeddings - Packaging the solution as a DataRobot Custom Transformer - Evaluating its impact on downstream machine learning tasks in DataRobot. --- # Google Gemini integration URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/gemini-google.html > Leverage LLMs proposed by hyperscalers via the Custom Model Workshop. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/generative_ai/gemini-app) DataRobot allows you to leverage LLMs proposed by hyperscalers via the [Custom Model Workshop](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/custom-models/custom-model-workshop/index.html#custom-model-workshop). This AI Accelerator demonstrates how to implement a Streamlit application based on the Google Gemini LLM and host it on the DataRobot platform. The user of this AI Accelerator is expected to be familiar with the custom model deployment process and custom metrics creation in DataRobot as well as with Google Gemini Enterprise Agent Platform (formerly Vertex AI). This accelerator requires the service account for the Google Gemini Enterprise Agent Platform (formerly Vertex AI) project. The following steps outline the accelerator workflow. 1. Createcredentialswith a GCP service account (base64 encoded). 2. Optional. Deploy a guard model from theDataRobot global models. 3. Deploya text model (Gemini Pro). 4. Deploya multimodal model (Gemini Pro Vision). 5. Create custom metricsfor both deployments (text and multimodal). 6. Deploy a Streamlit appto DataRobot. --- # GIN financial fraud detection URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/graph-gin.html > Integrate a Graph Isomorphism Network (GIN) as a custom model task in DataRobot using DRUM. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/graph_financial_fraud_classification/Graph_Financial_Fraud_Classification.ipynb) This accelerator demonstrates the end-to-end implementation of a Graph Isomorphism Network (GIN) as a custom model task in DataRobot. It serves multiple purposes, including model development, local testing, and integration with DataRobot’s custom model framework. The primary objectives of this accelerator are to demonstrate the complete pipeline for adding a custom graph-based task to DataRobot, and to outline DataRobot's custom model hooks implementation. These hooks include: - transform : Preprocess JSON graph data into DGL format. DRUM hooks automatically call transform before executing the fit and score hooks. - fit : Train the GIN model and implicitly save it. - score : Score the data in prediction mode. - load_model : Load pre-trained models. The accelerator's workflow includes three major steps: 1. Manually implement the DRUM process. Follow a step-by-step breakdown of data transformation, model training, and prediction viaDRUM. 2. Test models locally with DRUM to validate hooks before integrating the model with DataRobot. 3. Implement a custom task in DataRobot with automated validation and threshold optimization. --- # Custom model development URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/index.html > Custom model development accelerators that you can add to your experiment workflow. | Topic | Description | | --- | --- | | Create and deploy a custom model | How to create, deploy, and monitor a custom inference model with DataRobot's Python client. You can use the Custom Model Workshop to upload a model artifact to create, test, and deploy custom inference models to DataRobot’s centralized deployment hub. | | Custom blueprints with Composable ML | Customize models on the Leaderboard using the Blueprint Workshop. | | GraphSAGE custom transformer | Convert a tabular dataset into a graph representation, train a GraphSAGE-based neural network, and package the solution as a DataRobot custom transformer. | | Google Gemini integration | Implement a Streamlit application based on Google Gemini LLM and host it on the DataRobot platform with Google Gemini Enterprise Agent Platform (formerly Vertex AI) integration. | | GIN financial fraud detection | Integrate a Graph Isomorphism Network (GIN) as a custom model task in DataRobot using DRUM. | | Llama 2 on GCP | Host Llama 2 on Google Cloud Platform with cost comparisons, infrastructure details, and integration with DataRobot's custom model framework. | | LLM custom inference template | The LLM custom inference model template enables you to deploy and accelerate your own LLM, along with "batteries-included" LLMs like Azure OpenAI, Google, and AWS. | | Mistral 7B on GCP | Host Mistral 7B on Google Cloud Platform with infrastructure setup, cost considerations, and DataRobot integration for monitoring and deployment. | | Reinforcement learning | Implement a model based on the Q-learning algorithm. | | Scoring Code microservice | Follow a step-by-step procedure to embed Scoring Code in a microservice and prepare it as the Docker container for a deployment on customer infrastructure (it can be self- or hyperscaler-managed K8s). | | Optimize custom model metrics with hyperparameter tuning | Improve DataRobot models using custom loss functions and advanced hyperparameter tuning. | --- # Llama 2 on GCP URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/llama.html > Learn how to integrate Llama 2 on Google GCP and DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/Finetuned_Llama/Finetuned%20Llama%202%20on%20Google%20GCP.ipynb) There are a wide variety of open source large language models (LLMs). For example, there has been a lot of interest in [Llama](https://llama.meta.com/) and variations such as Alpaca, Vicuna, Falcon, and Mistral. Because these LLMs require expensive GPUs, users often want to compare cloud providers to find the best hosting option. In this accelerator you will work with Google Cloud Platform to host Llama 2. You may also want to integrate with the cloud provider that hosts your Virtual Private Cloud (VPC) so that you can ensure proper authentication and access it only from within the VPC. While this accelerator uses authentication over the public internet, it is possible to leverage Google's cloud infrastructure to adjust and suit your cloud architectural needs, including provisioning scaleout policies. Finally, by leveraging Google Gemini Enterprise Agent Platform (formerly Vertex AI) in a managed format, you can integrate that infrastructure into your existing stack to meet monitoring needs—things like monitoring service health, CPU usage, and low-level alerting to billing, cost attribution, and account management and, using GCP's tools to route information into BigQuery for ad hoc analytics, log exploration, and more. ### Llama 2 For information about Llama 2 you can read: - The model card on HuggingFace . - The paper released on Arxiv . Llama is available from [Meta](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) for download. ### Llama 13B-Instruct The [Llama-13b-instruct](https://huggingface.co/codellama/CodeLlama-13b-Instruct-hf) model has been fine-tuned on datasets available from HuggingFace and is designed specifically for instruction-based use cases. It was trained to use `[INST]` and `[/INST]` control tokens around a user message as well as to begin with system ID ( ``). For example: ` [INST] What is your favorite condiment? [/INST]` ### Overview of GCP The GCP instance types listed below can host Llama-13B with acceleration: - g2-standard-8 with 1 L4 GPU: 8 vCPUs, 32 GB of RAM, $623 per month - n1-standard-16 with 2 V100 GPUs: 16 vCPUs, 60GB of RAM, $388 per month - n1-standard-16 with 2 T4 GPUs: 16 vCPUS, 60GB of RAM + 32 GB + 32 GB, $388 per month - a2-highgpu-1g with 1 A100 GPU: 12 vCPUs, 85GB of RAM, $2,682 per month --- # LLM custom inference template URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/llm-template.html > The LLM custom inference model template enables you to deploy and accelerate your own LLM, along with [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/generative_ai/LLM_custom_inference_model_template) There are a wide variety of LLM model such as OpenAI (not Azure), Gemini Pro, Cohere and Claude. Managing and monitoring these LLM models is crucial to effectively using them. Data drift monitoring by DataRobot MLOps enables you to detect the changes in a user prompt and its responses. Sidecar models can prevent a jailbreak, replace Personally Identifiable Information (PII), and evaluate LLM responses with a global model in Registry. Data export functionality shows you of what a user desired to know at each moment and provides the necessary data you should be included in RAG system. Custom metrics indicate your own KPIs which you inform your decisions (e.g., token costs, toxicity, and hallucination). In addition, DataRobot's RAG playground enables you to compare the RAG system of LLM models that you want to try once you deploy the models in MLOps. You can obtain the best LLM model to accelerate your business. The comparison of variety of LLM models is key element to success the RAG system. The LLM custom inference model template enables you to deploy and accelerate your own LLM, along with "batteries-included" LLMs like Azure OpenAI, Google, and AWS. Currently, DataRobot has a template for OpenAI (not Azure), Gemini Pro, Cohere, and Claude. To use this template follow the instructions outlined on GitHub. --- # Mistral 7B on GCP URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/mistral-7b.html > Learn how to integrate Mistral 7B on Google GCP and DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/Mistral%207B%20on%20Google%20GCP/Mistral%207B%20on%20Google%20GCP.ipynb) There are a wide variety of open source large language models (LLMs). For example, there has been a lot of interest in [Llama](https://llama.meta.com/) and variations such as Alpaca, Vicuna, Falcon, and Mistral. Because these LLMs require expensive GPUs, users often want to compare cloud providers to find the best hosting option. In this accelerator, you will work with Google Cloud Platform to host Llama 2. You may also want to integrate with the cloud provider that hosts your Virtual Private Cloud (VPC) so that you can ensure proper authentication and access it only from within the VPC. While this accelerator uses authentication over the public internet, it is possible to leverage Google's cloud infrastructure to adjust and suit your cloud architectural needs, including provisioning scaleout policies. Finally, by leveraging Google Gemini Enterprise Agent Platform (formerly Vertex AI) in a managed format, you can integrate that infrastructure into your existing stack to meet monitoring needs—things like monitoring service health, CPU usage, and low-level alerting to billing, cost attribution, and account management and, using GCP's tools to route information into BigQuery for ad hoc analytics, log exploration, and more. For information about Mistral, you can read the model card on [HuggingFace](https://huggingface.co/mistralai/Mistral-7B-v0.1), the [Arxiv page](https://arxiv.org/abs/2310.06825) and the [release announcement](https://mistral.ai/news/announcing-mistral-7b/). It is available under an [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). --- # Optimize custom model metrics with hyperparameter tuning URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/opt-custom-metric.html > Improve DataRobot models using custom loss functions and advanced hyperparameter tuning. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/Custom_Metrics_Model_Optimization/Custom_Metrics_Model_Optimization.ipynb) This accelerator demonstrates how to improve DataRobot models using custom loss functions and advanced hyperparameter tuning. In many real-world business problems, standard metrics like RMSE or Accuracy do not fully represent the true business cost. For example, in CLV prediction, overpredicting loss-making customers or underpredicting high-value customers can directly impact revenue and retention strategy. Similarly, classification models often need custom objectives such as maximizing recall at specific thresholds or minimizing false negatives. By creating a custom metric and tuning models using that metric, you ensure the model is optimized for business value, not just statistical performance. This approach is essential when: - Business costs are not symmetrical (overprediction and underprediction have different impacts) - False negatives and false positives carry different risk levels - Revenue-driven metrics matter more than standard ML scores - Domain rules must be incorporated into optimization --- # Reinforcement learning URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/reinforce-learn.html > Implement a model based on the Q-learning algorithm. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/Reinforcement%20learning/Reinforcement_Learning.ipynb) In this accelerator, you implement a very simple model based on the Q-learning algorithm. This accelerator shows a basic form of reinforcement learning that doesn't require a deep understanding of neural networks or advanced mathematics and how one might deploy such a model in DataRobot. This example shows the Grid World problem, where an agent learns to navigate a grid to reach a goal. The accelerator will go through the following steps: 1. Define state and action space 2. Create a Q-table to store expected rewards for each state/action combination 3. Implement a learning algorithm and train a model 4. Evaluate the model 5. Deploy the model to a DataRobot REST API endpoint --- # Scoring Code microservice URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/custom-model-dev/sc-micro.html > Follow a step-by-step procedure to embed Scoring Code in a microservice and prepare it as the Docker container for a deployment on customer infrastructure (it can be self- or hyperscaler-managed K8s). [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/ecosystem_integration_templates/scoring-code-as-microservice-w_docker/README.md) This accelerator guides through the step-by-step procedure that makes it possible to embed scoring code in the microservice and to prepare it as the Docker container for the deployment on the customer infrastructure (it can be self or hyperscaler-managed K8s). The K8s configuration and deployment on K8s are out of scope. The accelerator also includes an example Maven project with the Java code. --- # Enrich with Hyperscaler API URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/data-enrichment-prep/enrich-hyper.html > Call the GCP API and enrich a modeling dataset that predicts customer churn. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/data_enrichment_gcp_nlp_api/GCP_enrich_sentiment.ipynb) Many companies are recognizing the value of unstructured data, particularly in the form of text, and are looking for ways to extract insights from it. This data includes emails, social media posts, customer feedback, call transcripts, and more. One of the most powerful tools for analyzing text data is sentiment analysis. Sentiment analysis is the process of identifying the emotional tone of a piece of text, such as positive, negative, or neutral. It is a valuable tool to enrich the dataset for building machine learning models. For example, the sentiment expressed through a customer's recent call transcript with customer service could be predictive of the customer's likelihood to churn. However, building sentiment analysis models is not an easy task. It requires a significant investment of time, resources, and expertise, especially in terms of accurately labeled data with large corpus to train. Most companies do not have the resources or expertise to develop their own sentiment analysis models. Fortunately, there are now APIs available that provide sentiment analysis as a service. By using these APIs, companies can save time and money while still gaining the benefits of sentiment analysis. One of the most significant benefits of using hyperscaler APIs for sentiment analysis is their accuracy. The models behind the APIs are trained on large amounts of data, making them highly accurate at identifying emotions and sentiments in text data. This accelerator demonstrates how easy it is to call the GCP API and enrich a customer churn prediction modeling dataset. An improvement in the model performance is based on retrieved sentiment scores from customer call transcripts. --- # GCP sentiment enrichment URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/data-enrichment-prep/gcp-enrich.html > Demo the usage of the Google Cloud Natural Language API for sentiment analysis to enrich a customer churn dataset. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/data_enrichment_gcp_nlp_api/GCP_enrich_sentiment.ipynb) Text data is a valuable source of information for machine learning models, as it allows algorithms to extract insights from large volumes of unstructured text data. Text data can be obtained from various sources, such as social media, news articles, and customer feedback. The benefits of using text data in ML models include its ability to provide valuable insights, such as sentiment analysis, and topic modeling, which can help organizations make informed decisions. However, using text data in ML models can be challenging due to several factors, such as the complexity of natural language, the presence of bias and noise, and the lack of standardization in text data. Additionally, text data requires significant preprocessing and feature engineering to ensure that it can be effectively used in ML models. One common application of text mining is sentiment analysis, where a numerical value is assigned representing whether the text carries a positive, neutral, or negative sentiment. While DataRobot can help efficiently build such models, the training requires a large, accurately labeled corpora that have been accurately labeled, making it a challenging task for users lacking such training dataset. In this accelerator, demo the usage of the Google Cloud Natural Language API for sentiment analysis to enrich a customer churn dataset. The sentiment scores from the Google API help improve the model performance in predicting the likelihood of churn for each customer, without requiring the user to train their own sentiment models. --- # Data enrichment and preparation URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/data-enrichment-prep/index.html > Data enrichment and preparation accelerators that you can add to your experiment workflow. | Topic | Description | | --- | --- | | Enrich with Hyperscaler API | Call the GCP API and enrich a modeling dataset that predicts customer churn. | | GCP sentiment enrichment | Demo the usage of the Google Cloud Natural Language API for sentiment analysis to enrich a customer churn dataset. | | Churn problem framing | Discover the problem framing and data management steps required to successfully model for churn, using a B2C retail example and a B2B example based on a DataRobot’s churn model. | | Churn insights with Streamlit | Use the Streamlit churn predictor app to present the drivers and predictions of your DataRobot model. | | Synthetic training data | Learn how to generate synthetic datasets that mimic real-world data for training, validation, and testing—enabling safe data sharing and model development when access to real data is limited due to privacy or regulatory constraints. | | Feature engineering for molecular SMILES data | Execute a feature engineering pipeline tailored for SMILES-formatted molecular data. | --- # Churn problem framing URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/data-enrichment-prep/ml-churn.html > Discover the problem framing and data management steps required to successfully model for churn, using a B2C retail example and a B2B example based on a DataRobot’s churn model. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/churn_problem_framing_feature_eng/Churn_Before_Modelling.ipynb) Customer retention is central to any successful business and machine learning is frequently proposed as a way of addressing churn. It is tempting to dive right into a churn dataset, but improving outcomes requires correctly framing the problem. Doing so at the start will determine whether the business can take action based on the trained model and whether your hard work is valuable or not. This accelerator blog will teach the problem framing and data management steps required before modeling begins. It uses two examples to illustrate concepts: a B2C retail example, and a B2B example based on DataRobot’s internal churn model. --- # Feature engineering for molecular SMILES data URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/data-enrichment-prep/smiles.html > Execute a feature engineering pipeline tailored for SMILES-formatted molecular data. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/Feature%20Engineering%20For%20Molecular%20SMILES) SMILES (simplified molecular input line entry system) is a textual representation of molecular structures. While it's compact and widely used in cheminformatics, SMILES strings must be transformed into numerical representations to be used effectively in machine learning models. This accelerator introduces a feature engineering pipeline tailored for SMILES-formatted molecular data. It demonstrates how to convert raw SMILES strings into machine-learning-ready features using RDKit and other tools. It is recommended to run the accelerator in a DataRobot codespace using a GPU environment. This accelerator's workflow is summarized below: 1. Preprocess and visualize SMILES strings using RDKit and py3Dmol. 2. Extract molecular descriptors statistical features (physicochemical properties). 3. Extract TF-IDF features from SMILES strings, and then apply TruncatedSVD to obtain lower-dimensional embeddings. 4. Extract fingerprints features from SMILES strings, then apply TruncatedSVD to obtain lower-dimensional embeddings. 5. Extract semantic representations from pretrained molecular embeddings of ChemBERTa and SMILESBERT (CPU is slow, so GPU is recommended), and then apply PCA to obtain lower-dimensional embeddings. 6. Run Autopilot with these features to compare model performance and create benchmarks. 7. Extract feature contribution (SHAP values). --- # Churn insights with Streamlit URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/data-enrichment-prep/streamlit-app.html > Use the Streamlit churn predictor app to present the drivers and predictions of your DataRobot model. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/advanced_ml_and_api_approaches/Streamlit_template_datarobot_insights/README.md) This app serves as an example of how to present the drivers and predictions of your DataRobot model using a Churn prediction use case. Building a churn predictor app using Streamlit and DataRobot is a great way to leverage the power of machine learning to improve customer retention. The first step in building a churn prediction model is to collect and prepare your data. This typically involves gathering data on your customers' behavior, demographics, and usage patterns. Once you have your data, you can upload it to DataRobot and let the platform do the rest. After training, DataRobot provides detailed insights into the model's performance, including feature importance, model validation, and accuracy metrics. Once you have a model that you're satisfied with, you can generate predictions on new data using DataRobot's prediction API. This workflow assumes that you have already generated these predictions and saved them as a CSV file. To create a Streamlit app for churn prediction, you will need to import the necessary libraries, including Pandas, NumPy, Streamlit, Plotly, and PIL. You can then read in your prediction data and set up your Streamlit app's page configuration. The app itself should allow users to specify criteria for viewing churn scores and top churn reasons. You can accomplish this using sliders and other interactive elements. A workflow of this process for building a Streamlit app using DataRobot predictions can be found in the churn Streamlit app GitHub repository. This workflow can be adapted to present insights from other classification or regression models built in DataRobot. --- # Synthetic training data URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/data-enrichment-prep/synth-data.html > Learn how to generate synthetic datasets that mimic real-world data for training, validation, and testing—enabling safe data sharing and model development when access to real data is limited due to privacy or regulatory constraints. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/dr_synth_data/dr_synth_data.ipynb) This notebook provides a powerful code-first accelerator to help generate synthetic datasets in tabular format. It enables you to create synthetic data that mimics the structure and statistical properties of real-world datasets, offering a safe and efficient way to augment existing data or create entirely new datasets. The generated synthetic datasets can be uploaded directly to AI Catalog, where they can be organized, managed, and reused for various machine learning projects. This approach is particularly useful in scenarios where access to real data is limited due to privacy, security, or regulatory constraints. By generating synthetic datasets, users can expand their training data without compromising sensitive information. These synthetic datasets can be used for model training, validation, and testing, allowing for more robust model development and better generalization on unseen data. The notebook outlines how to create a synthetic training data set in a CSV file, with name, address, phone number, company, account number, and credit score. --- # DataRobotX intro URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/exp-track-and-tune/drx-intro.html > Learn how to use the new, agile, DRX package to streamline project creation and configuration. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/drx_python_package_overview/drx-overview.ipynb) DataRobotX (DRX) is a collection of DataRobot extensions designed to enhance your data science experience. It has clean, scikit-learn-like syntax that makes training models, deploying models, and getting predictions from models easy. It supports any project type including multiclass, time series, multilabel, clustering, and anomaly detection. ## Project goals DRX intends to explore and prototype a programmatic DataRobot experience that is: - Declarative and simple by default - Streamlines common workflows - Uses broadly familiar syntax and verbiage where possible - Unobtrusively customizable - Allows default behaviors and configuration to be easily overridden but not at the expense of complicating the common experience - Accelerates user experimentation - Offers new abstractions and concepts for interacting with DataRobot --- # Experiment tracking and tuning URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/exp-track-and-tune/index.html > Experiment tracking and tuning accelerators that you can add to your experiment workflow. | Topic | Description | | --- | --- | | DataRobotX intro | Learn how to use the new, agile, DRX package to streamline project creation and configuration. | | MLFlow experiment tracking | Automate machine learning experimentation using DataRobot, MLFlow, and Papermill for tracking experiments and logging results. | | Blueprint hyperparameter tuning | Learn how to access, understand, and tune blueprints for both preprocessing and model hyperparameters. | --- # MLFlow experiment tracking URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/exp-track-and-tune/mlflow.html > Automate machine learning experimentation using DataRobot, MLFlow, and Papermill for tracking experiments and logging results. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/advanced_ml_and_api_approaches/MLFLOW_w_datarobot_experiments/) Experimentation is a mandatory activity in any machine learning developer’s day-to-day activities. For time series projects, the number of parameters and settings to tune for achieving the best model is in itself a vast search space. Many of the experiments in time series use cases are common and repeatable. Tracking these experiments and logging results is a task that needs streamlining. Manual errors and time limitations may lead to selection of suboptimal models leaving better models lost in global minima. The integration of DataRobot API, Papermill, and MLFlow automates machine learning experimentation so that is becomes easier, robust, and easy to share. As illustrated below, you will use the [orchestration notebook](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/MLFLOW_w_datarobot_experiments/orchestration_notebook.ipynb) to design and run the [experiment notebook](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/MLFLOW_w_datarobot_experiments/experiment_notebook.ipynb), with the permutations of parameters handled automatically by DataRobot. At the end of the experiments, copies of the experiment notebook will be available, with the outputs for each permutation for collaboration and reference. You can review [the dependencies](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/MLFLOW_w_datarobot_experiments/requirements.txt) for the accelerator. This accelerator covers the following activities: - Acquiring a training dataset. - Building a new DataRobot project. - Deploying a recommended model. - Scoring via Spark using DataRobot's exportable Java Scoring Code. - Scoring via DataRobot's Prediction API. - Reporting monitoring data to the MLOps agent framework in DataRobot. - Writing results back to a new table. --- # Blueprint hyperparameter tuning URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/exp-track-and-tune/opt-grid.html > Learn how to access, understand, and tune blueprints for both preprocessing and model hyperparameters. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/advanced_ml_and_api_approaches/Hyperparameter_Optimization/README.md) In machine learning, hyperparameter tuning is the act of adjusting the "settings" (referred to as hyperparameters) in a machine learning algorithm, whether that's the learning rate for an XGBoost model or the activation function in a neural network. Many methods for doing this exist, with the simplest being a brute-force search over every feasible combination. While this requires little effort, it's extremely time-consuming as each combination requires fitting the machine learning algorithm. To this end, practitioners strive to find more efficient ways to search for the best combination of hyperparameters to use in a given prediction problem. DataRobot employs a proprietary version of pattern search for optimization not only for the machine learning algorithm's specific hyperparameters, but also the respective data preprocessing needed to fit the algorithm, with the goal of quickly producing high-performance models tailored to your dataset. While the approach used at DataRobot is sufficient in most cases, you may want to build upon the Autopilot modeling process by custom tuning methods. In this AI Accelerator, you will familiarize yourself with DataRobot's fine-tuning API calls to control DataRobot's pattern search approach as well as implement a modified brute-force grid-search for the text and categorical data pipeline and hyperparameters of an XGBoost model. This accelerator serves as an introductory learning example that other approaches can be built from. Bayesian Optimization, for example, leverages a probabilistic model to judiciously sift through the hyperparameter space to converge on an optimal solution, and will be presented next in this accelerator bundle. Note that as a best practice, it is generally best to wait until the model is in a near-finished state before searching for the best hyperparameters to use. Specifically, the following have already been finalized: - Training data (e.g., data sources) - Model validation method (e.g., group cross-validation, random cross-validation, or backtesting. How the problem is framed influences all subsequent steps, as it changes error minimization.) - Feature engineering (particularly, calculations driven by subject matter expertise) - Preprocessing and data transformations (e.g., word or character tokenizers, PCA, embeddings, normalization, etc.) - Algorithm type (e.g. GLM, tree-based, neural net) These decisions typically have a larger impact on model performance compared to adjusting a machine learning algorithm's hyperparameters (especially when using DataRobot, as the hyperparameters chosen automatically are pretty competitive). This AI Accelerator teaches you how to access, understand, and tune blueprints for both preprocessing and model hyperparameters. You'll programmatically work with DataRobot advanced tuning which you can then adapt to your other projects. You'll learn how to: - Prepare for tuning a model via the DataRobot API - Load a project and model for tuning - Set the validation type for minimizing errors - Extract model metadata - Get model performance - Review hyperparameters - Run a single advanced tuning session - Implement your own custom gridsearch for single and multiple models to evaluate --- # AI accelerators URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/index.html > Review comprehensive workflows, notebooks, and tutorials that help you find complete examples of common data science and machine learning workflows. AI Accelerators are designed to help speed up model experimentation, development, and production using the DataRobot API. They codify and package data science expertise in building and delivering successful machine learning projects into repeatable, code-first workflows and modular building blocks. AI Accelerators are ready right out-of-the-box, work with the notebook of your choice, and can be combined to suit your needs. AI accelerators cover a variety of topics, but primarily aim to assist you by: - Providing curated templates for workflows that use best-in-class data science techniques to help frame your business problem (e.g., customize a data visualization to your liking or rank models by ROI). - Getting you started quickly on a new AI or ML project by providing necessary insights, problem-framing, and use cases in notebooks. - Fine-tuning your projects and getting the most value from your existing data and infrastructure investments, including third-party integrations. | Section | Description | | --- | --- | | AI integrations and platforms | Templates for end-to-end API workflows between DataRobot and its ecosystem partners (Snowflake, GCP, Azure, AWS, etc.). | | LLM and GenAI applications | Applications and workflows that leverage large language models (LLM) and generative AI. | | Model building and fine-tuning | Techniques and workflows for building and tuning machine learning models. | | Data enrichment and preparation | Workflows for enhancing and preparing data for machine learning. | | Time series and specific use cases | Workflows and techniques for time series analysis and specific machine learning use cases. | | Model deployment and MLOps | Workflows for deploying models and managing machine learning operations. | | Experiment tracking and tuning | Tools and techniques for tracking experiments and tuning models. | | Model evaluation and metrics | Techniques for evaluating models and understanding their performance metrics. | | Custom model development | Workflows that integrate custom models with DataRobot. | | Advanced analytics and tools | Advanced usage of the DataRobot API that you can add to your experiment workflow. | --- # Adaptive reasoning agent URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/adaptive-agent.html > Showcases an agent's ability to **adapt its reasoning behavior** based on conversation dynamics. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/generative_ai/adaptive-agent) This accelerator showcases an agent's ability to adapt its reasoning behavior based on conversation dynamics. The agent acts as a customer support representative for the "DataInsight Pro" analytics platform. In this example, the agent uses: - GPT-4o for complex reasoning when corrections are detected. - GPT-4o-mini for fast responses during smooth conversation flow. - GPT-4o-mini as a reflection model to analyze the last three conversation turns and detect user corrections. Specifically, the agent dynamically switches between models based on conversation analysis: | Scenario | Model used | Behavior | | --- | --- | --- | | Conversation flowing smoothly | GPT-4o-mini | Fast, direct responses | | User corrects the agent | GPT-4o | More thorough reasoning | | User rephrases question | GPT-4o | Agent recognizes confusion | | Positive feedback received | GPT-4o-mini | Returns to efficient mode | The following diagram provides an overview that illustrates the agent architecture. ``` ┌─────────────────────────────────────────────────────────────┐ │ Frontend (React) │ │ ┌─────────────┐ ┌──────────────────────────────────────┐ │ │ │ Model Mode │ │ Reflection Log Panel │ │ │ │ Indicator │ │ (shows gpt-4o-mini reasoning) │ │ │ └─────────────┘ └──────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────┘ │ ┌─────────────────────────────────────────────────────────────┐ │ Adaptive Agent │ │ 1. Store conversation history (last 3 turns) │ │ 2. Call reflection model (gpt-4o-mini) before response │ │ 3. Switch model based on correction detection │ │ - Corrections detected → GPT-4o (thorough) │ │ - Smooth conversation → GPT-4o-mini (fast) │ └─────────────────────────────────────────────────────────────┘ ``` The following script, with model adaptation noted, is applied in the accelerator: | Turn | User prompt | Expected behavior | Model | | --- | --- | --- | --- | | 1 | "What pricing plans do you offer?" | Lists 3 tiers (Starter, Pro, Enterprise) | GPT-4o-mini | | 2 | "How do I export data?" | General export explanation | GPT-4o-mini | | 3 | "No, I meant export to CSV specifically, not PDF" | Correction detected! Detailed CSV instructions | GPT-4o | | 4 | "Can I schedule automated exports?" | Thorough answer with plan requirements | GPT-4o | | 5 | "Thanks, that's helpful!" | Positive acknowledgment | GPT-4o-mini | --- # Product feedback automation URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/auto-feedback.html > Use Predictive AI models in tandem with Generative AI models to overcome the limitation of guardrails around automating the summarization and segmentation of sentiment text. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/Product%20Innovation%20GenAI/Product%20Innovation%20using%20Generative%20AI%20and%20DataRobot%20Auto%20ML.ipynb) This accelerator shows how to use Predictive AI models in tandem with Generative AI models and overcome the limitation of guardrails around automating the summarization and segmentation of sentiment text. In a nutshell, it consumes product reviews and ratings and outputs a Design Improvement Report. The voice of the customer is traditionally captured using feedback from sales channels (which is generally reviews). While consolidating product reviews has been done in a semi-automated fashion with the help of summarization, segmentation, and subject matter expertise, it is always a time-consuming and resource intensive process that spans multiple cycles of rework. With DataRobot, you can use the generative AI solution framework to build an automated system to process review text and use Generative AI to produce targeted reports for product design and manufacturing teams. In this accelerator you will: - Extract high impact review keywords from product reviews using DataRobot. - Implement guardrails for selecting models with higher AUC to make sure keywords are robust and correlated to the review sentiment. - Generate product development recommendations for the final report. --- # Teams/Slack chatbots URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/chatbot-teams-slack.html > An accelerator for collaborative app plug-ins, such as bots for Teams and Slack. Slack and Microsoft teams support the creation of applications or bots that can interact with a DataRobot deployment. Using Slack or Teams as a front-end for an LLM-based chat application enables broad availability in an organization, providing the ability to upload documents easily as well as leverage internal knowledge stored in team channels or conversations. **Slack setup:** Obtain permissions from your IT organization to create a Slack App in your Slack organization. Then, [create the app](https://api.slack.com/apps) with the Slack API. **Teams setup:** Obtain permissions from your IT organization to upload a customized app. Then, register for an application and a ot in the [Microsoft Developer Portal](https://login.microsoftonline.com). Within the message send action in both Slack and Teams, export the user question as a prompt into your DataRobot LLM deployment, and provide the response from the REST API call to the user message in your bot. DataRobot enables you to track token usage, prompts, responses, toxicity, and other [custom metrics](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-metrics.html) to monitor and govern your application internally. The following code can be adapted to the Slack or Teams messaging structure to pass a message as a query to the DataRobot LLM deployment and get a response. ``` import datarobotx as drx # configure drx LLM Deployment RAG = drx.Deployment.from_url( url=f'https://app.datarobot.com/deployments/{RAG_did}/overview' ) def make_datarobot_deployment_unstructured_predictions( data, **kwargs, ): query = json.loads(data) response = RAG.predict_unstructured(query) return json.dumps(response) def make_prediction(prompt, history=None) -> dict: data = { "prompt": prompt, } if history: data["chat_history"] = [ [entry.get("user", ""), entry.get("chatbot", "")] for entry in history ] response = make_datarobot_deployment_unstructured_predictions( json.dumps(data) ) response = json.loads(response) return response ``` --- # AI cluster labeling URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/cluster-genai.html > Use cluster insights provided by DataRobot with ChatGPT to provide business- or domain-specific labels to the clusters using OpenAI and DataRobot APIs. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/smart_cluster_naming/smart_cluster_labeling.ipynb) The toughest part of unsupervised learning modeling is explaining clusters to end users. DataRobot not only allows users to build clustering models, but also provides insights per cluster to help users analyze the clusters. In most scenarios, the users building the models might not have the subject matter expertise to tailor the cluster labels towards the users consuming the models. This is where you can use Generative AI models to automatically label the clusters with some prompt engineering. Because the Generative AI models have been trained on vast amounts of domain and business datasets, they can understand and label the clusters tuned for end user expertise. This AI Accelerator shows how to extract cluster insights from DataRobot models, use prompt engineering to label clusters, and then rename the clusters in the DataRobot project. You will explore the following: - Use the API to extract Cluster Insights from DataRobot unsupervised learning projects. - Use Generative AI and Prompt Engineering to consume cluster insights and create cluster labels for DataRobot clusters. - Use the API to rename DataRobot clusters. --- # Customer communication AI URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/comm-genai.html > Learn how generative AI models like GPT-3 can be used to augment predictions and provide customer friendly subject matter expert responses. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/customer_communication_datarobot_gen_ai/effective_customer_communication_datarobot_gen_ai.ipynb) In this accelerator, you will see how you can integrate LLM-based agents like ChatGPT with DataRobot Prediction Explanations to quickly implement effective customer communication in AI-based workflows. In banking and fintech, one of the most critical communication provided to the customer is refusal of products and services, like loan application rejection. When a machine learning model predicts high loan default probability, organizations need to relay the rejection to the applicant in an effective way to sustain customer satisfaction, avoid churn, and not reduce the customer lifetime value. Effective communication also needs subject matter expertise, which becomes costly if implemented at the granularity of every application. DataRobot’s Prediction Explanations provide the context for prediction and the LLM agent provides the subject matter expertise to provide effective yet positive responses to adverse event predictions. This allows organizations to effectively communicate their AI-based decisions with their customers while improving costs and productivity related to their expert personnel. --- # Compliance agent URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/compliance-agent.html > Automatically compare your active governance policies against pre-uploaded industry standards or internal benchmarks to identify inconsistencies. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/generative_ai/compliance-agent-mvp-main) The compliance agent, which can be created as a custom application on DataRobot, automatically compares your active governance policies against pre-uploaded industry standards or internal benchmarks to identify inconsistencies. It detects where your custom rules fall short of regulatory requirements and proposes specific recommendations to reconcile these differences. By mapping your organizational policies directly to established frameworks, it ensures your standards are both rigorous and aligned with the latest legal and industry expectations. --- # Support workflow optimization URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/customer-support.html > Use generative AI models to cater to level-one requests, allowing support teams to focus on more pressing and high-visibility requests. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/Optimizing%20Customer%20Support%20workflows%20with%20Gen%20AI/Optimizing%20Customer%20Support%20workflows%20with%20Gen%20AI.ipynb) Customer support is a crucial part of every organization. With the proliferation of social media, customer-centric organizations are using social media platforms to provide customer support. Irrespective of the platform, support automation has been actively pursued by organizations to improve the customer experience and loyalty while reducing the workload on support teams. While automated prioritization and routing have been solved using predictive models, automated resolution is still an active area of research. The majority of support requests are primarily requests for information (level one requests). Handling these requests and would benefit from automation. In this accelerator, generative AI models cater to level-one requests, allowing support teams to focus on more pressing and high-visibility requests. Learning from historical communications, generative AI responses can maintain the same standard of support communication that customers are used to. Because use of generative AI comes at a cost—per request, compute time, per token—costs can quickly balloon. To mitigate this problem, you can use predictive ML models for routing standard requests to the generative AI model and high-priority severity requests to human staff. --- # Data annotator app URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/data-annotator.html > Leverage the data annotator app to both label new data and label predicted data within an active learning situation after training a model with DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/data_annotator_app/data_annotator_app.ipynb) High-quality training data is necessary for a top-notch machine learning model. But how you can quickly and easily collect labels from a team of human reviewers? One way is to stand up a Flask app for quick labeling review. This notebook will show you how to leverage the data-annotator app to both (1) label new data and (2) label predicted data within an active learning situation after training a model with DataRobot. The data-annotator app requires two inputs: - img_path : The app is currently configured for labeling images (jpg and png are both supported formats). You need to place these images within a directory and specify that path to the app. - data_path : You need to tell the app all possible labels for your images. - If you are classifying images that have not yet been labeled, you can provide a csv file with at least one column named label that contains all potential classes. See Scenario 1 below for more details. - If you are classifying images that have already been assigned labels, you can provide a csv file with at least two columns named img_path (filename of the image) and label (assigned class for the image). - If you are classifying images that have already been scored within DataRobot, please refer to Scenario 2 below for more details on how to configure the dataset. --- # AI data prep assistant URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/data-prep-assist.html > The AI data preparation assistant is a powerful tool designed to streamline and automate the data preparation process. It combines automated data quality checks with AI-powered data preparation suggestions to help data scientists and analysts prepare their datasets more efficiently. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/generative_ai/AI_data_prep_assistant) The AI data preparation assistant is a powerful tool designed to streamline and automate the data preparation process. It combines automated data quality checks with AI-powered data preparation suggestions to help data scientists and analysts prepare datasets more efficiently. Data preparation typically consumes 60-80% of a data scientist's time. This involves repetitive tasks like identifying quality issues, cleaning data, and transforming it into a suitable format for analysis. Manual data preparation is not only time-consuming, but prone to inconsistencies and human error. This accelerator provides the following features to assist with data preparation: - An automated data quality assessment across 12 key dimensions. - AI-powered suggestions for data preparation steps. - Automated code generation and execution for data cleaning. - Interactive visualizations of data quality issues. - A real-time data transformation preview. --- # LLM and GenAI applications URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/index.html > LLM and GenAI applications that you can add to your experiment workflow. | Topic | Description | | --- | --- | | Adaptive agent | Showcases an agent's ability to adapt its reasoning behavior based on conversation dynamics. | | Direct Preference Optimization for reinforcement learning | Automates the process of fine-tuning an LLM using Direct Preference Optimization (DPO) and then deploying that model to DataRobot. | | Product feedback automation | Use Predictive AI models in tandem with Generative AI models to overcome the limitation of guardrails around automating the summarization and segmentation of sentiment text. | | Teams/Slack chatbots | Build collaborative app plug-ins, such as bots for Teams and Slack. | | AI cluster labeling | Use cluster insights provided by DataRobot with ChatGPT to provide business- or domain-specific labels to the clusters using OpenAI and DataRobot APIs. | | Customer communication AI | How generative AI models, like GPT-3, can be used to augment predictions and provide customer-friendly subject matter expert responses. | | Support workflow optimization | Use generative AI models to cater to level-one requests, allowing support teams to focus on more pressing and high-visibility requests. | | Data annotator app | Leverage the data annotator app to both label new data and label predicted data within an active learning situation after training a model with DataRobot. | | AI data prep assistant | Use the AI data preparation assistant to streamline and automate the data preparation process. | | JITR bot responses | Create a deployment to provide context-aware answers 'on the fly' using "Just In Time Retrieval" (JITR). | | PDF RAG with LLM | Use an LLM as an OCR tool to extract all the text, table, and graph data from a PDF, then build a RAG and playground chat on DataRobot. | | Healthcare conversation agent | Use Retrieval Augmented Generation to build a conversational agent for Healthcare professionals. | | Teams GenAI integration | With DataRobot's Generative AI offerings, organizations can deploy chatbots without the need for an additional front-end or consumption layers. | | Vector chunk visualization | Implement a Streamlit application to gain insights from a vector database of chunks. | | XoT implementation | Implement and evaluate Everything of Thoughts (XoT) in DataRobot, an approach to make generative AI "think like humans." | | Zero-shot error analysis | Use zero-shot text classification with large language models (LLMs), focusing on its application in error analysis of supervised text classification models. | | Compliance agent | Automatically compare your active governance policies against pre-uploaded industry standards or internal benchmarks to identify inconsistencies. | --- # JITR Bot responses URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/jitr-bot.html > Create a deployment to provide context-aware answers 'on the fly' using [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/JITR-bot/JITR-bot.ipynb) Retrieval Augmented Generation (RAG) has become an industry standard method for interfacing with large language models by making them 'context-aware'. However, there are a number of situations where a text generation problem is not solved by interacting with large vector database containing many documents. These problems require context but where the context is not known before query time and is often unrelated to existing vector stores. Usually, they are questions about single documents where desirable behavior is to allow the document to be specified at runtime. One application that does this fairly well is DataChad. DataChad works fine for its purpose as a localized web application, but it doesn't generalize. In other words, there is not a good way to interact with the application without opening a browser, uploading whatever files you want to analyze, and hitting a run button. Rather than follow the standard RAG approach of querying an existing vector store, this accelerator creates a deployment that accepts a file as an argument so that it can provide context-aware answers 'on the fly'. DataRobot calls this approach "Just In Time Retrieval", or JITR for short. DataRobot created a Slackbot that uses this deployment as the backend to answer questions when a user uploads a PDF, called the "JITR Bot". --- # PDF RAG with LLM URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/llm-multimodal-pdf.html > Use an LLM as an OCR tool to extract all the text, table, and graph data from a PDF, then build a RAG and playground chat on DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/generative_ai/LLM%20Multimodal%20PDF%20RAG) This accelerator introduces an approach to use an LLM as an OCR tool. Supply a PDF and split it into multiple images, then extract the text, table, and graph data from the PDF with an LLM, saving them as Markdown files. Use those files to build a RAG and then create a chat in the DataRobot playground. --- # Healthcare conversation agent URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/med-research.html > Use Retrieval Augmented Generation to build a conversational agent for Healthcare professionals. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/Medical%20Research%20Agent/Medical%20Research%20Conversational%20Agent.ipynb) This accelerator shows how you can use [Retrieval Augmented Generation](https://arxiv.org/abs/2005.11401) to build a conversational agent for healthcare professionals. Healthcare professionals have to constantly stay informed of the latest research in not only their own specialization but also in complimentary fields. This means they have to constantly consume the latest research from trusted sources. Because new research papers are published at an astonishing rate, it is important to filter out irrelevant and untrusted research and focus on trusted research that is important to healthcare in this agent's knowledge base. As this agent's intended use is in healthcare, it is of paramount importance that the agent operates with in the confines of the knowledge base without hallucinations. With DataRobot, this accelerator shows how to use predictive modeling to identify trusted research and then build a knowledge base for the conversational agent using DataRobot's [generative AI offering](https://www.datarobot.com/platform/generative-ai/). This accelerator illustrates the following; - Use predictive models to classify text files - Create a vector store out of research paper abstracts - Use Retrieval Augmented Generation with a generative AI model - Deploy a Generative AI model to the DataRobot platform --- # Teams GenAI integration URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/ms-teams.html > With DataRobot's Generative AI offerings, organizations can deploy chatbots without the need for an additional front-end or consumption layers. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/ecosystem_integration_templates/teams_datarobot) Microsoft Teams offers workspace chat and video conferencing, file storage, and application integration to organizations. Workspace chat feature allows you to interact with other users and bots in their day-to-day activities. This feature is useful for deploying Generative AI agents to improve employee productivity. With DataRobot's Generative AI offerings, organizations can deploy chatbots without the need for an additional front-end or consumption layers. ## How it works Most messenger/communication apps support bots. A bot is a program that interacts with the users of the messenger application by ingesting the user message and providing responses. Bot can be static or dynamic depending on the logic encoded. A bot in most cases is a service exposing Rest Endpoints which receive user text and respond back with text. Instead of developers starting from scratch, Microsoft provides an SDK which can be used as building blocks or boilerplate. This SDK supports different languages including python. The code demonstrated here uses this code as boilerplate. --- # Direct Preference Optimization for reinforcement learning URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/rlhf-agent.html > Showcases an agent's ability to **adapt its reasoning behavior** based on conversation dynamics. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/Direct%20Preference%20Optimization/DPO.ipynb) [Direct Preference Optimization (DPO)](https://arxiv.org/abs/2305.18290) is a machine learning technique used to train AI agents, providing a simpler, more stable alternative to traditional Reinforcement Learning from Human Feedback (RLHF). It helps align AI models with human preferences in ways that are hard to capture through pre-training alone, making models more helpful, harmless, and honest. It can improve things like following instructions accurately, avoiding harmful content, and producing more useful responses. Generally, the technique works as follows: 1. A model is trained on large amounts of text data to predict what comes next in sequences (pre-training). 2. Humans evaluate and rank the model prompt outputs. 3. Based on the feedback, a different model learns to predict the preferred response. 4. The main model is fine-tuned using reinforcement learning algorithms to generate higher-scoring outputs. This accelerator automates the process of fine-tuning an LLM using Direct Preference Optimization (DPO) and then deploying that model to DataRobot. Essentially, it takes a base model, teaches it to prefer specific types of responses based on a provided dataset, and prepares it for production use. Specific accelerator actions: Data preparation actions - Downloads a specific preference dataset from the DataRobot Registry. - Uses the Hugging Face datasets library to load the CSV, which typically contains three columns: a prompt, a chosen (preferred) response, and a rejected response. Model training (DPO) - Initializes theQwen2-0.5B-Instructmodel inbfloat16precision to save memory. - Applies DPOTrainer from the TRL (Transformer Reinforcement Learning) library. This is a modern alternative to RLHF that aligns models to human preferences without needing a separate reward model. - For hardware efficiency, the script is configured for FSDP (Fully Sharded Data Parallel). This allows the model to be trained across multiple GPUs by "sharding" the model weights, and it uses Gradient Checkpointing to further reduce VRAM usage. Model consolidation and saving - Weight gathering, which applies a specialized routine to "gather" the FSDP shards back into a single, cohesive model file. - Ensures that only the "Main Process" (Rank 0) handles the final file writing to avoid data corruption or redundant saves. DataRobot deployment - Once the model is saved locally, the script uses the DataRobot API to create a custom model, making the model ready for deployment and accessible via the REST API. --- # Tensile - Enhanced agent reliability through automated test URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/tensile-agent-reliability.html > Use DataRobot's test-driven development framework to improve the reliability, task performance, and policy adherence of AI agents with trajectory logging, replay, and contextual hints. Building and maintaining reliable AI agents is challenging. Agents must stay on-task, follow policies, and recover from failures in a way you can measure and improve. This accelerator introduces Tensile, a test-driven development framework in DataRobot for improving the reliability, task performance, and policy adherence of AI agents through automated test synthesis and trajectory analysis. Tensile helps you instrument agents, capture execution trajectories, and turn successes and failures into repeatable tests. You can then evaluate and replay runs, compare system prompt changes, and use clustering and contextual hint injection to remediate issues iteratively. In this accelerator you will: - Instrument an agent with TrajectoryLogger to record execution trajectories. - Analyze trajectories to identify testable moments (successes and failures). - Evaluate and replay runs to quantify improvements and compare system prompt changes. - Configure Tensile with the DataRobot LLM gateway. - Use clustering (Dash app and ClusteringHintInjector ) to explore issues and inject contextual hints. - Apply the Trajectory Analyzer workflow with ProgrammaticHintInjector for iterative improvement. ## Prerequisites Before running the accelerator, ensure you have: - Tensile installed (see the quickstart below). - A config.yaml with LLM and trajectory settings. - For DataRobot: set DATAROBOT_API_TOKEN in test.env (or in your environment). Optionally set DATAROBOT_LLM_GATEWAY_URL and DATAROBOT_TRACE_CONTEXT for observability. Quickstart from the project root: ``` uv venv --python 3.13 uv sync; pre-commit install uv pip install -e . cp config.yaml.sample config.yaml # And fill in credentials tensile # show help ``` ## Instrument an agent for trajectory logging Use `TrajectoryLogger` as the transport for an `httpx` client, then pass that client into your OpenAI-compatible agent. Trajectories are written to `/` (with `trajectory_dir` in `config.yaml`). ``` from tensile.logging import TrajectoryLogger http_client = httpx.AsyncClient( transport=TrajectoryLogger( httpx.AsyncHTTPTransport(), trajectory_subdir= | None ) ) client = AsyncOpenAI( api_key=api_key, base_url=f"{endpoint_url}/v1", http_client=http_client, ) ``` ## Analyze trajectories and evaluate testable moments Run the analysis pipeline (outputs to `analysis_output/` by default): ``` tensile analyze ``` To run testable moments manually (for example, 10 times): ``` tensile test -n 10 ``` ## Replay trajectories Replay steps in a trajectory to collect new LLM responses, spot flukes, or compare behavior after system prompt changes. Omit `output_path` to write to `.replay.jsonl`. ``` tensile replay [output_path] tensile replay --num-replays 5 tensile replay --num-replays 3 --max-concurrency 10 tensile replay --num-replays 3 --system-prompt-path # Examples tensile replay tensile replay -n 5 tensile replay output/replay.jsonl -n 3 ``` ## Configuration ### DataRobot LLM gateway Add the following to your `config.yaml` to use the DataRobot LLM gateway: ``` # config.yaml llm: name: "" # e.g., vertex_ai/gemini-3-pro-preview api_base: "" api_key: "" ``` ## Clustering ### Clustering app Start the Dash app to explore and cluster analysis outputs in the browser. It requires the `dev` dependency group; with `uv`, run: ``` task dev-env task apps:clustering ``` ### Clustering-based hint injection Use `ClusteringHintInjector` with `analysis_dirs` and `trajectories_dirs` pointing at your Tensile outputs and a report store ( `InMemoryReportStore` or `FileSystemReportStore`). Example: ``` from pathlib import Path import httpx from openai import AsyncOpenAI from tensile.logging.hint_injector import ( ClusteringHintConfig, ClusteringHintInjector, InMemoryReportStore, SentenceTransformersEmbeddingBackend, ) base_transport = httpx.AsyncHTTPTransport() embedding_backend = SentenceTransformersEmbeddingBackend( model_name="", ) report_store = InMemoryReportStore() config = ClusteringHintConfig( analysis_dirs=[Path("analysis_output")], trajectories_dirs=[Path("trajectories")], ) hinting_transport = ClusteringHintInjector( base_transport, embedding_backend=embedding_backend, report_store=report_store, config=config, ) http_client = httpx.AsyncClient(transport=hinting_transport) client = AsyncOpenAI( api_key=api_key, base_url=f"{endpoint_url}/v1", http_client=http_client, ) ``` ## Trajectory Analyzer workflow 1. Instrument the agent with ProgrammaticHintInjector and TrajectoryLogger : ``` from tensile.logging import TrajectoryLogger from tensile.logging.hint_injector.programmatic_hint_injector import ProgrammaticHintInjector http_client = httpx.AsyncClient( transport=ProgrammaticHintInjector( wrapped=TrajectoryLogger( wrapped=httpx.AsyncHTTPTransport(), trajectory_subdir=, ), hint_file_path=None, ) ) # It's recommended to start with hint_file_path=None until a hint file is generated by the analyzer ``` 1. Run the agent to produce a trajectory. 2. Run tensile analyze . When analysis finishes, copy the generated hints.json , updated system prompt, and/or updated tool definitions back into your agent. 3. Set hint_file_path to the path of the hints.json file and run the agent again to produce a new trajectory. 4. Run tensile analyze --hints-file to re-analyze with the new hints. 5. Repeat until behavior converges. --- # Vector chunk visualization URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/vectorstore-chunk.html > Implement a Streamlit application to gain insights from a vector database of chunks. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/generative_ai/vectorstore-chunk-visualization) This AI Accelerator demonstrates how to implement a Streamlit application to gain insights from a vector database of chunks. A RAG developer can compare similarity between chunks and remove unnecessary data during RAG development. In this workflow you will build a Streamlit application, build a vectorstore, then build and analyze summaries of chunks and clusters in the data. --- # XoT implementation URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/xot-implementation.html > Implement and evaluate Everything of Thoughts (XoT) in DataRobot, an approach to make generative AI [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/generative_ai/XoT%20Evaluation) Implement and evaluate Everything of Thoughts (XoT) in DataRobot, an approach to make generative AI "think like humans." In the world of generative AI, various methods (called thought generation) are researched to help AI acquire more human-like "thinking patterns." In particular, XoT aims to produce more accurate answers by teaching generative AI the "thinking process." There are two main methods to achieve XoT: 1. Chain-of-Thought (CoT): A method of thinking by connecting multiple thoughts like a chain and reasoning through them 2. Retrieval Augmented Thought Tree (RATT): A method of thinking by expanding multiple possibilities like tree branches and retrieving relevant information from the external knowledge base. This accelerator explains how to implement these methods. Specifically, it introduces how to set up and compare three types of LLM prompts: direct, Chain-of-Thought, and RATT. "Direct" referring to the well-known "you are a helpful assistant." The accelerator also explains how to conduct performance evaluations using sample datasets, comparing the accuracy and efficiency of each method, and analyze using multiple evaluation metrics. --- # Zero-shot error analysis URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/llm-and-genai-apps/zero-shot.html > Use zero-shot text classification with large language models (LLMs), focusing on its application in error analysis of supervised text classification models. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/zero_shot_LMM_error_analysis_NLP/Zero%20Shot%20Text%20Classification%20for%20Error%20Analysis.ipynb) This AI Accelerator which offers a deep dive into the utilization of zero-shot text classification for error analysis in machine learning models. This educational resource is an invaluable asset for those interested in enhancing their understanding and proficiency in the field of machine learning. Building on your existing knowledge and experience with the DataRobot automated machine learning platform, this notebook demonstrates the development of a text classification model. From there, turn your focus towards a crucial, yet sometimes challenging aspect of machine learning - error analysis. Understanding why a supervised machine learning model incorrectly classifies certain examples can be a challenging task. The newly released notebook introduces a novel methodology for identifying and understanding these errors using zero-shot text classification. In this accelerator, make use of three different zero-shot classification methods: Natural Language Inference (NLI), Embedding, and Conversational AI. The distinct capabilities of each method contribute to a comprehensive and enlightening error analysis process. Detailed within the notebook is a thorough explanation of the error analysis procedure. So regardless of your proficiency level in machine learning, the content is structured to cater to a wide range of readers. The application of zero-shot text classification to error analysis could be a significant enhancement to your machine learning practice, particularly with DataRobot. --- # Feature Discovery workflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/afd-e2e.html > Use a repeatable framework for end-to-end production machine learning. It includes time-aware feature engineering across multiple tables, training dataset creation, model development, and production deployment. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/Automated_Feature_Discovery_template_ML_pipeline/End-to-end%20Automated%20Feature%20Discovery%20Production%20Workflow.ipynb) This accelerator outlines a repeatable framework for end-to-end production machine learning. It includes time-aware feature engineering across multiple tables, training dataset creation, model development, and production deployment. It is common to build training data from multiple sources, but this process can be time consuming and error prone, especially when you need to create many time-aware features. - Event based data is present in every vertical. For example, customer transactions in retail or banking, medical visits, or production line data in manufacturing. - Summarizing this information at the parent (Entity) level is necessary for most classification and regression use cases. For example, if you are predicting fraud, churn, or propensity to purchase something, you will likely want summary statistics of a customers transactions over a historical window. This raises many practical considerations as a data scientist: How far back in time is relevant for training? Within that training period, which windows are appropriate for features? 30 days? 15? 7? Further, which datasets and variables should you consider for feature engineering? Answering these conceptual questions requires domain expertise or interaction with business SMEs. In practice, especially at the MVP stage, it is common to limit the feature space you explore to what's been created previously or add a few new ideas from domain expertise. - Feature stores can be helpful to quickly try features which were useful in a previous use case, but it is a strong assumption that previously generated lagged features will adapt well across all future use cases. - There are almost always important interactions you haven't evaluated or thought of. Multiple tactical challenges arise as well. Some of the more common ones are: - Time formats are inconsistent between datasets (e.g., minutes vs. days), and need to be handled correctly to avoid target leakage. - Encoding text and categorical data aggregates over varying time horizons across tables is generally painful and prone to error. - Creating a hardened data pipeline for production can take weeks depending on the complexity. - A subtle wrinkle is that short and long-term effects of data matter, particularly with customers/patients/humans, and those effects change over time. It's hard to know apriori which lagged features to create. When data drifts and behavior changes, you very well may need entirely new features post-deployment, and the process starts all over. All of these challenges inject risk into your MVP process. The best case scenario is historical features capture signal in your new use case, and further exploration to new datasets is limited when the model is "good enough". The worst case scenario is you determine the use case isn't worth pursuing, as your features don't capture the new signal. You often end up in the middle, struggling to know how to improve a model you are sure can be better. What if you could radically collapse the cycle time to explore and discover features across any relevant dataset? This notebook provides a template to: - Load data into Snowflake and register with DataRobot's AI Catalog. - Configure and build time aware features across multiple historical time-windows and datasets using Snowflake (applicable to any database). - Build and evaluate multiple feature engineering approaches and algorithms for all data types. - Extract insights and identify the best feature engineering and modeling pipeline. - Test predictions locally. - Deploy the best performing model and all feature engineering in a Docker container, and expose a REST API. - Score from Snowflake and write predictions back to Snowflake. --- # Causal AI for readmission URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/causal-ai.html > Work with data recording hospital readmission outcomes for diabetes patients to evaluate the causal relationship between the diabetes patients' medication status and their subsequent chance of being readmitted to the hospital. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/Causal_AI/Causal_AI.ipynb) Predictive AI models are a powerful tool for uncovering subtle predictive relationships between observed variables. But sometimes, you need to draw conclusions about the causal relationship between two variables, not just the observed correlation. To achieve this "Causal AI", you can use the DataRobot platform and a quasi-experimental technique called "Inverse Propensity of Treatment Weighting". This notebook will apply this technique to data on diabetes hospital patient readmission. This notebook outlines how to: - Prepare data for a Propensity of Treatment model - Fit a Propensity of Treatment model with DataRobot - Calculate Inverse Propensity of Treatment Weights - Evaluate the causal relationship using Inverse Propensity of Treatment Weighting - Understanding Inverse Propensity of Treatment Weighting In this notebook, you will be working with data recording hospital readmission outcomes for diabetes patients. You will evaluate the causal relationship between the diabetes patients' medication status and their subsequent chance of being readmitted to the hospital. To evaluate this causal relationship experimentally, you would have to randomly assign patients to the treatment group (those receiving medication) vs. not, and then follow those patients to see whether they get readmitted to the hospital or not. But in the scenario for this notebook, you don't have experimental data! You only have observational data. In other words, some patients walk in taking medication, others don't. You have not controlled the assignment of the "treatment" condition (medication) to the subjects of the study. So while you could use predictive modeling to understand if the medication status of patients walking in is predictive of later readmission, you can't directly use predictive models to make conclusions about whether the medication has a causal effect on readmission. In this scenario, you can use a "quasi-experimental" technique; this is a set of techniques for approximating experimental setups without actually having a true experiment. Specifically, you can use a technique called "Inverse Propensity of Treatment Weighting". Inverse Propensity of Treatment Weighting consists of the following steps: 1. Fit a predictive model to estimate the probability for each study participant being assigned to the treatment group (their "propensity of treatment"). 2. Calculate a special weight for each participant based on their propensity of treatment (the "inverse propensity of treatment weight"), which will adjust the treatment and control groups to become more similar to each other in terms of the observed confounding variables. 3. Evaluate the causal relationship between the treatment and the outcome using the adjusted/weighted populations (pseudopopulations). While this technique is not as valid as the gold standard of a randomized controlled trial, it can bring you a lot closer to obtaining comparable treatment and control groups from which to judge a causal relationship. --- # Custom lift charts URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/custom-lift-chart.html > Leverage popular Python packages with DataRobot's Python client to recreate and augment DataRobot's lift chart visualization. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/customizing_lift_charts/customizing_lift_charts.ipynb) Ever wanted to plot more than 60 bins in DataRobot's lift chart? Ever needed to present this graphic with a specific color palette? Ever required to display more information in the chart due to regulatory reasons? In this AI Accelerator, leverage popular Python packages with DataRobot's Python client to recreate and augment DataRobot's lift chart visualization. These customizations allow you to: - Plot more than 60 bins in DataRobot's lift chart. - Present this lift chart visualizations with a specific color palette. - Display more information in the chart. The steps demonstrated in the accompanying notebook are: 1. Connect to DataRobot 2. Create a DataRobot project 3. Run a single blueprint from the repository 4. Obtain predictions and actuals 5. Recreate DataRobot’s lift chart 6. Add customization to the lift chart --- # Fantasy baseball predictions URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/fantasy-baseball.html > Leverage the DataRobot API to quickly build multiple models that work together to predict common fantasy baseball metrics for each player in the upcoming season. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/model_factory_selfjoin_fantasy_baseball/fantasy_baseball_predictions_model_factory.ipynb) In this accelerator, you will leverage the DataRobot API to quickly build multiple models that work together to predict common fantasy baseball metrics for each player in the upcoming season. Millions of people play fantasy baseball every year—more than 15 million in the United States and Canada in 2022, according to the Fantasy Sports and Gaming Association. It is the second-most popular fantasy sport in the US and Canada, behind American football, and like most fantasy sports, fantasy team managers typically select players for their team through classic drafts or auction-style processes. Choosing a team of baseball players based on who is your favorite—or even based on last year's performance without any regard for regression to the mean—is likely to field a relatively weak team year in and year out. Baseball is one of the most well documented of all sports, statistics-wise, and with the wealth of data available you can derive a better estimate of each player's true talent level and their likely performance in the coming year using machine learning. This allows for better drafting, helping to avoid overpaying for players coming off of "career" seasons while identifying undervalued players that can effectively fill out a quality team in later rounds of the draft (or for fewer auction dollars). When drafting players for fantasy baseball, you must make decisions based on the player's performance over their career to date, as well as effects like aging, changing positions, changing teams, etc. You will leverage DataRobot to produce better predictions of the players' performances in the next year based on what they have done in prior years, and from patterns you can learn from similar players in the past. ## Learning objectives - How to query a rich dataset of MLB players' statistics from the Fangraphs' API. - How to set up a project with automated time-aware feature engineering (Automated Feature Discovery). - How to update the player data in a Feature Discovery project (i.e., secondary data) to re-predict without building a new project. - How to loop over a project creation function to build many DataRobot projects automatically--in this case, to build one project/model for each of the five common fantasy baseball stats: batting average (AVG), home runs (HR), runs (R), runs batted in (RBI), and stolen bases (SB), though you could repeat the same process on pitching statistics, as well. ## Retrieve baseball data This notebook uses Python's pybaseball module to get data from player-seasons between 2012 and 2023. In this workflow, the machine learning algorithm learns patterns from pre-COVID era data, as well as data from 2020 and 2021. This data should help show how well the top model is able to learn how to work around the shortened 2020 season. Fangraphs provides more than 300 features about hitters each season, from the most superficial statistics like batting average (AVG) and home run counts (HR), to the most in-depth statistics like expected weighted on-base average (xWOBA) and barrel contact percentage (Barrel%). You will use DataRobot to sift through many of these feature to find the ones that best signal future performance. --- # Fine-tune & deploy LLMs URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/finetune-codespace.html > Review an end-to-end workflow for fine-tuning and deployment an LLM using features of Hugging Face, Weights and Biases (W&B), and DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/fine-tuning-in-codespaces/Fine-tuning%20in%20DataRobot%20Codespaces.ipynb) This accelerator illustrates an end-to-end workflow for fine-tuning and deployment an LLM using features of Huggingface, Weights and Biases (W&B), and DataRobot. Specifically, the accelerator walks you through the following steps: - Downloading an LLM from the Hugging Face Hub. - Acquiring a dataset from Hugging Face. - Leveraging DataRobot codespaces, notebooks, and GPU resources to facilitate fine-tuning via Hugging Face and W&B. - Leveraging DataRobot MLOps to register and deploy a model as an inference endpoint. - Leveraging DataRobot's RAG playground to evaluate and compare your fine-tuned LLM against available LLMs. The accelerator uses Hugging Face as a common example that you can modify based on your needs. It uses Weights and Biases to help keep track of your experiments. It is helpful to visualize training loss in real time as well as log prompt results for review during fine-tuning. Also, if you decide to do some hyperparameter tuning, you can do so with W&B Sweeps. ## Considerations This accelerator has been tested in a DataRobot codespace with a GPU resource bundle. requirement.txt has a pinned version of the required libraries. Notebooks images in DataRobot have limited writable space (about 20GB). Therefore, checkpointing models during finetuning is not encouraged, and if you do checkpoint, limit it. This accelerator opts to fine-tune llama-3.2-1B since it is on the smaller side. Use Weights and Biases to track the experiment. The W&B API Key is available in `.env.` If you don't have a W&B account, get one at the [W&B sign up page](https://www.wandb.ai/). --- # Hyperparameter optimization URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/hyperopt.html > Build on the native DataRobot hyperparameter tuning by integrating the hyperopt module into DataRobot workflows. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/advanced_ml_and_api_approaches/Hyperparameter_Optimization) In machine learning, hyperparameter tuning is the act of adjusting the "settings" (referred to as hyperparameters) in a machine learning blueprint or pipeline. For example, adjustable hyperparameters might be the learning rate for an XGBoost model, the activation function in a neural network, or grouping limits in one-hot encoding for categorical features. Many methods for doing this exist, with the simplest being a brute force search over a wide range of possible parameter value combinations. While this requires little effort for the human, it's extremely time-consuming for the machine, as each distinct combination of hyperparameter values requires fitting the blueprint again. To this end, practitioners strive to find more efficient ways to search for the best combination of hyperparameters. This AI Accelerator shows how to leverage open source optimization modules to further tune parameters in DataRobot blueprints. Build on the [native DataRobot hyperparameter tuning functionality](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/Hyperparameter_Optimization/HyperParam_Opt_Core_Concepts.ipynb) by integrating the hyperopt module into the API workflow. The hyperopt module allows for a particular Bayesian approach to parameter tuning known as the Tree-structured Parzen Estimator (TPE), though more generally this accelerator should be seen as an example of how to leverage DataRobot's API to integrate with any parameter tuning optimization framework. You will explore the following: - Identify specific blueprints from a DataRobot project and review their hyperparameters through the API. - Define a search space and optimization algorithm with hyperopt. - Tune hyperparameters with hyperopt's Tree-structured Parzen Estimator (Bayesian) approach. --- # Image data with Databricks URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/image-databricks.html > Import image files using Spark and prepare them into a data frame suitable for ingest into DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/image_dataprep_classification_databricks/Image%20Data%20Preparation.ipynb) Visual AI allows you to leverage images in your models just like any other type of data. In this accelerator, you will import image files using Spark and prepare them into a data frame suitable for ingest into DataRobot. Then you will leverage DataRobot through code to rapidly train and deploy a powerful multiclass image classifier. While there are other methods of ingesting image data into DataRobot, in this notebook you will encode the image data directly into the data frame using base64 encoding. This methodology allows you to keep all of the relevant data in a single data frame, and works well for a Databricks environment. This technique also extends widely to a wide variety of multimodal datasets. Dive in to go from Databricks image data to a deployed classifier. --- # Model building and fine-tuning URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/index.html > Model building and fine-tuning accelerators that you can add to your experiment workflow. | Topic | Description | | --- | --- | | Feature Discovery workflow | Use a repeatable framework for end-to-end production machine learning. It includes time-aware feature engineering across multiple tables, training dataset creation, model development, and production deployment. | | Causal AI for readmission | Work with data recording hospital readmission outcomes for diabetes patients to evaluate the causal relationship between the diabetes patients' medication status and their subsequent chance of being readmitted to the hospital. | | Custom lift charts | Leverage popular Python packages with DataRobot's Python client to recreate and augment the lift chart visualization in DataRobot. | | Fantasy baseball predictions | Leverage the DataRobot API to quickly build multiple models that work together to predict common fantasy baseball metrics for each player in the upcoming season. | | Fine-tune & deploy LLMs | Review an end-to-end workflow for fine-tuning and deploying an LLM using features of Hugging Face, Weights and Biases (W&B), and DataRobot. | | Hyperparameter optimization | Build on the native DataRobot hyperparameter tuning by integrating the hyperopt module into DataRobot workflows. | | Image data with Databricks | Import image files using Spark and prepare them into a data frame suitable for ingest into DataRobot. | | Production ML with tables | Explore a repeatable framework for building production ML pipelines that integrate and engineer features from multiple tables. | | Predictions in mobile apps | Learn how to incorporate DataRobot predictions into a mobile app. | | Order quantity prediction | Build a model to improve decisions about initial order quantities using future product details and product sketches. | | Model factory with Python | Learn how to use the Python threading library to build a model factory. | | Symbolic regression (Eureqa) | Apply symbolic regression to your dataset in the form of the Eureqa algorithm. | | Model marketing attribution | Use DataRobot to streamline marketing attribution use cases. | --- # Production ML with tables URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/ml-tables.html > Review an AI accelerator that uses a repeatable framework for a production pipeline from multiple tables. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/Automated_Feature_Discovery_template_ML_pipeline/End-to-end%20Automated%20Feature%20Discovery%20Production%20Workflow.ipynb) We've all been there: data for customer transactions are in one table, but the customer membership history is in another. Or, you have sensor-level data at the sub-second level in one table, machine errors in another table, and production demand in yet another table, all at different time frequencies. Electronic Medical Records (EMRs) are another common instance of this challenge. You have a use case for your business you want to explore, so you build a v0 dataset and use simple aggregations from before, perhaps in a feature store. But moving past v0 is hard. The reality is, the hypothesis space of relevant features explodes when considering multiple data sources with multiple data types in them. By dynamically exploring the feature space across tables, you minimize the risk of missing signal by feature omission and further reduce the burden of a priori knowledge of all possible relevant features. Event-based data is present in every vertical and is becoming more ubiquitous across industries. Building the right features can drastically improve performance. However, understanding which joins and time horizons are best suited to your data is challenging, and also time-consuming and error-prone to explore. In this accelerator, you'll find a repeatable framework for a production pipeline from multiple tables. This code uses Snowflake as a data source, but it can be extended to any supported database. Specifically, the accelerator provides a template to: - Build time-aware features across multiple historical time-windows and datasets using DataRobot and multiple tables in Snowflake (or any database). - Build and evaluate multiple feature engineering approaches and algorithms for all data types. - Extract insights and identify the best feature engineering and modeling pipeline. - Test predictions locally. - Deploy the best-performing model and all data preprocessing/feature engineering in a Docker container, and expose a REST API. - Score from Snowflake and write predictions back to Snowflake. --- # Model marketing attribution URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/model-market.html > Use DataRobot to streamline marketing attribution use cases. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/marketing_mix_modeling) Marketing attribution can be defined as the process of assigning credit to certain marketing activities (e.g., spending) as they contribute towards a desired goal (e.g., increases in revenue). Understanding how much these marketing activities contribute enables stakeholders to make better budget allocation decisions. However, in today’s world, marketing teams have many avenues to spend their marketing dollars, and it can be a daunting task to figure out which avenues are playing the largest role towards optimizing the key performance indicator (KPI) of interest. Model-based approaches can help sift through all this information in a rigorous and efficient way, leading to faster decision-making. One of the most common ways to understand the attribution from each marketing channel of interest is to do live A/B testing (e.g., incrementality testing). That is, expose one group to marketing spend and another very similar group in nature to no marketing spend. Assuming all the sources of variability between the groups are reasonably satiated, you can measure a KPI (e.g., revenue) for both groups over a window of time and take the difference between them. This difference gives you information as to the lift that your marketing efforts had during the experimental window. While the benefit of this method is deriving attribution via live-testing, some disadvantages include: - A/B testing can be resource and time-intensive (e.g., you need many weeks to get a good picture of the differences, especially in different times of the year). - Derived insights can only be gleaned from the experimental window (i.e., if you’re testing only for 10 weeks, you only have 10 weeks worth of data from the procedure). - Many factors can influence business results such as competition, economy, local events, etc., so it can be difficult to truly isolate the incremental lift from marketing (i.e., experimental design is hard to do outside of a laboratory). ### Model-based approaches In order to emulate the aforementioned A/B testing procedure (and assuming revenue is your KPI of interest), you need to be able to know the difference in revenue between when you expose your customer base to marketing and when you don't. While it’s possible to have days historically where you didn’t have any marketing spend, it’s more likely that you had some level of marketing spend activity each day. Hence, it’s impossible to go back in time to understand what revenue would have been without any marketing spend on each day in the past. This is where model-based approaches can help. For the purposes of this accelerator, you can think of a “model” as “a set of steps a computer takes to find patterns or make decisions.” Different types of models exist, but the focus here will be on machine learning models, which are primarily used for predicting information you don’t know (such as revenue associated with no marketing spend). Specifically for marketing attribution, the steps may include: 1. Acquire historical revenue and spending data at the desired granularity (weekly, monthly, etc.). 2. Build a machine learning model that tries to learn how to predict historical revenue based on your historical spend (and other factors that can help explain revenue, like holidays, promotions, economic indicators, etc.). 3. Once your model has learned all it can from your historical data, you can begin using it to answer what-if questions like, “what would my revenue be if I increase spending in this marketing channel?” or “what would my revenue have been if I didn’t do any marketing spend whatsoever on this day?” 4. After you estimate what revenue would have been with no marketing spend each day in your historical data using the model, you can compare this value to the actual revenue on the given day to understand the total lift in dollars due to your marketing efforts. 5. Once the total lift is estimated, you can allocate this out to the individual marketing channels with the help of explanatory tools like Shapley values . Having this machine learning model gives you the ability to help answer questions you normally wouldn’t be able to know about our historical data. It can also be used to understand future what-if scenarios too (e.g., "if I applied this allocation of spend across my marketing channels on this day in the future, what would my predicted revenue be?") and budget optimization (e.g., "out of the possible allocation strategies I have, which one will increase my revenue the most?"). ### How DataRobot helps DataRobot can help with marketing attribution use cases by: 1. Making building machine learning models incredibly easy. 2. Providing a Python API client that makes building custom workflows repeatable and scalable. 3. Having a suite of interpretability tools for each machine learning model (including SHAPley values). 4. Having connectors to databases such as Snowflake. 5. Having mechanisms to create straightforward applications to consume results (or that can write results back to the database for consumption of an internally-built dashboard) These are just a few examples how DataRobot can streamline this process. Ultimately, a variety of ways exist for tackling marketing attribution and the process described below is just one way DataRobot has helped customers in the past. It is by no means the only way to do marketing attribution use cases. --- # Predictions in mobile apps URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/pred-mobile.html > Learn how to incorporate DataRobot predictions into a mobile app. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/using-datarobot-in-mobile-apps) An AI model can't just be an experiment. AI Predictions need to be in the hands of real users interactive with customers, products, or users. This accelerator demonstrates how to incorporate DataRobot predictions into a mobile app. Included in this accelerator is a Swift Playground App prototype. The playground integrates an app that uses the Iris dataset and calls a DataRobot model to predict the likely sub-species of Iris plant. --- # Order quantity prediction URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/pred-products.html > Build a model to improve decisions about initial order quantities using future product details and product sketches. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/Retail_Industry_Predicting_Factory_Orders_New_Products/Retail%20Industry%20-%20Predicting%20Factory%20Order%20Quantities%20for%20New%20Products.ipynb) Retailers face many decisions when launching new products. One key decision is the amount of product to order from the manufacturer. Ordering too much wastes working capital and can lead to products being heavily discounted. Ordering too little squanders an opportunity for revenue and may cause customers to purchase other brands. Getting initial orders quantities right is particularly difficult for luxury products where first year demand for a new purse, a new belt or a new shoe can vary by several orders of magnitude based on factors unrelated to the product specifications. This notebook illustrates how to build a model to improve decisions about initial order quantities using future product details and product sketches. --- # Model factory with Python URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/python-multi.html > Learn how to use the Python threading library to build a model factory. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/model-factory-with-python-native-multithreading/Model%20Factory%20with%20Python%20Multithreading.ipynb) Model training in the DataRobot platform is an I/O-bound task that can be time consuming depending on the project configuration and the type of models to be trained. Working under tough deadlines and needing to train tens or hundreds of projects (for example, at an SKU level) requires building model factories and leads to the mandatory requirement to significantly decrease training time. This can be achieved on the base of a multithreaded approach and is demonstrated on the example of this AI Accelerator that leverages a Python multithreading library. --- # Symbolic regression (Eureqa) URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-building-tuning/tune-eureqa.html > Apply symbolic regression to your dataset in the form of the Eureqa algorithm. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/fine_tuning_with_eureqa/fine_tuning_with_eureqa.ipynb) DataRobot offers the ability to apply symbolic regression to your dataset in the form of the Eureqa algorithm. Eureqa returns human-readable and interpretable analytic expressions and allows us to incorporate DataRobot's own domain expertise about the problem. This accelerator shows how the Eureqa algorithm can "discover" the gravitational constant by finding the correct relationship between the variables from a double-pendulum experiment. This accelerator covers the following activities: - Apply the Eureqa algorithm to your dataset - Tune the model's mathematical building blocks to incorporate DataRobot's domain expertise about the problem - Access the resulting closed-form expression --- # Monitor AWS Sagemaker models with MLOps URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-deploy-mlops/aws-mlops.html > Train and host a SageMaker model that can be monitored in the DataRobot platform. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/ecosystem_integration_templates/AWS_monitor_sagemaker_model_in_DataRobot/AWS_SageMaker_DataRobot_MLOps.ipynb) DataRobot MLOps provides a central hub to deploy, monitor, manage, and govern all your models in production. You can deploy models to the production environment of your choice and continuously monitor the health and accuracy of your models, among other metrics. AWS Sagemaker is a fully managed service that allows data scientists and developers to build, train, and deploy machine learning models. DataRobot MLOps with its AWS Sagemaker integration provides an end-to-end solution for managing machine learning models at scale, you can easily monitor the performance of your machine learning models in real-time, and quickly identify and resolve any issues that arise. --- # Run choice-based conjoint analysis URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-deploy-mlops/conjoint.html > Use conjoint analysis to identify customer preferences for product features through survey-based choice modeling and interpretability with DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot/data-science-scripts/blob/master/accelerators_dev/use_cases_and_horizontal_approaches/Choice-based-conjoint.ipynb) Conjoint analysis is a tool widely used in marketing research for new product development testing. It's usually executed as an online survey format where a few survey respondents will make one choice out of a set of different alternatives. The output allows researches to accurately identify which product features and combination works best before developing them. This notebook outlines how to run a choice-based conjoint analysis as part of the broader Conjoint Analysis topic, with the focus being on the modeling aspect to derive preference scores. DataRobot's SHAP values add the value of interpretability over traditional methods using a linear regression coefficient scores where negative coefficients make it hard to interpret. From a technical perspective, conjoint analysis is a method to identify respondent (customer) preferences of a product feature, without explicitly asking them about that product feature in a survey. This is done by asking respondents to choose one item out of a set of alternatives. Each alternative is made up of the different feature combinations and permutations you are seeking to test. As you run the survey across a large number of responses, DataRobot helps identify customer latent/unconscious feature preferences, even those they themselves may not realize. --- # Model deployment and MLOps URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-deploy-mlops/index.html > Model deployment and MLOps accelerators that you can add to your experiment workflow. | Topic | Description | | --- | --- | | Monitor AWS Sagemaker models with MLOps | Monitor SageMaker models in DataRobot MLOps. | | Run choice-based conjoint analysis | Identify customer feature preferences using conjoint analysis. | | Migrate a model to a new cluster | Move a deployed model between DataRobot clusters. | | Video object detection using Visual AI | Use Visual AI for object detection in video streams. | | MLOps smart audit | Audit and visualize MLOps deployment configurations. | --- # Migrate a model to a new cluster URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-deploy-mlops/model-migrate.html > Move a deployed model between DataRobot clusters. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/model_migration_across_dr_instances/Model_Migration_Example.ipynb) Currently under development, an experimental DataRobot API allows administrators to download a deployed model from DataRobot cluster X, upload it to DataRobot cluster Y, and then deploy and make requests from it. Note that this notebook will not work using https://app.datarobot.com. ### Prerequisites - This notebook must be able to write to the model directory, located in the same directory as this accelerator's notebook. For best results, run this notebook from the local file system - Ensure that the model you choose to migrate must be a deployed model. - Provide API keys for both the source and destination clusters. - The Source and Destination users must have the "Enable Experimental API access" feature flag enabled to follow this workflow. - The notebook must have connectivity to the Source and Destination clusters. - DataRobot versions on the clusters must be consistent with the Supported Paths above. - For models on clusters of DataRobot v7.x, you must have SSH access to the App Node of the cluster. - The Source and Destination DataRobot clusters must have the following in the config.yaml: --- # Video object detection using Visual AI URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-deploy-mlops/obj-detection.html > Use Visual AI for object detection in video streams. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/object_detection_on_video) Object detection (binary and multiclass classification) applied to image and video processing is one of the tasks that can be easily and efficiently implemented with DataRobot [Visual AI](https://docs.datarobot.com/en/docs/classic-ui/modeling/special-workflows/visual-ai/index.html), which allows you to train deep learning models intended for Computer Vision based-projects. You can also bring your own Computer Vision model and deploy it in DataRobot via the [Custom Model Workshop](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/custom-models/custom-model-workshop/index.html). This accelerator demonstrates how deep learning models trained and deployed with DataRobot can be used for object detection on a video stream. (Consider the example of detection when the person in front of the camera wears glasses.) The Elastic-Net Classifier (L2 / Binomial Deviance) along with Pretrained MobileNetV3-Small-Pruned Multi-Level Global Average Pooling Image Featurizer with no image augmentation are used in this accelerator. The dataset used contains images for two classes: persons with glasses and persons without glasses and is linked in the accelerator on GitHub. A sample of the dataset (100 images for each class) is used for this accelerator. The video stream is captured with OpenCV Computer Vision library. The frontend is implemented as a Streamlit application. --- # MLOps smart audit URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-deploy-mlops/smart-audit.html > Audit and visualize MLOps deployment configurations. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/advanced_ml_and_api_approaches/mlops_smart_audit) This accelerator outlines a workflow to create an application that provides an interactive dashboard for analyzing the MLOps configuration across multiple machine learning deployments. The application examines each deployment for enabled capabilities (e.g., data drift detection, accuracy monitoring, notifications, etc.) and produces a summarized, interactive view. This helps MLOps administrators assess deployment quality, identify gaps, prioritize improvements, and check compliance scores. The key feature of the accelerator are outlined below: - Deployment overview:Quickly see which MLOps functions are enabled or disabled across your deployments. - Quality and compliance assessment:Each deployment is assigned a quality score based on the percentage of enabled capabilities and a compliance score based on a set of mandatory functions and model risk levels. - Advanced filtering and search:Use sidebar filters to refine deployments by type, owner, capabilities, or score ranges. - LLM-Based insights (optional):When enabled, Azure OpenAI provides natural language summaries and recommendations. - Capability governance:View governance rules for capabilities categorized by importance (Critical, High, Moderate, Low) for both predictive and generative models. --- # Custom metrics for model selection URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/ai-custom-metrics.html > This AI Accelerator demonstrates how one can leverage DataRobot's Python client to extract predictions, compute custom metrics, and sort their DataRobot models accordingly. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/custom_leaderboard_metrics/custom_metrics.ipynb) When it comes to evaluating model performance, DataRobot provides many of the standard metrics [out-of-the box](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/opt-metric.html), either on the [Leaderboard](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/leaderboard-ref.html) or as part of a [model insight](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/index.html). However, depending on the industry, you may need to sort your DataRobot leaderboard by a specific metric not natively supported by DataRobot. This AI Accelerator demonstrates how one can leverage DataRobot's Python client to extract predictions, compute custom metrics, and sort their DataRobot models accordingly. The topics covered are as follows: - Setup: import libraries and connect to DataRobot - Build models with Autopilot - Retrieve predictions and actuals - Sort models by Brier Skill Score (BSS) - Sort models by Rate@Top1% - Sort models by return-on-investment (ROI) In addition, although sometimes difficult, assigning the ROI of utilizing machine learning can be vital for use case adoption and model implementation. Creating a payoff matrix uses a computed dollar figure rather than a machine learning metric. --- # t-SNE dimensionality reduction URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/dim-reduction.html > Review examples for taking a DataRobot project and exporting its model insights as both machine readable files and plots in various file formats. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/Dimensionality%20reduction%20in%20DataRobot%20with%20t-SNE/Dimensionality%20reduction%20in%20DataRobot%20with%20t-SNE.ipynb) This accelerator provides examples for taking a DataRobot project and exporting its model insights as both machine readable files and plots in various file formats using t-Distributed Stochastic Neighbor Embedding (t-SNE). t-SNE is a powerful technique for dimensionality reduction that can effectively visualize high-dimensional data in a lower-dimensional space. Dimensionality reduction can improve machine learning results by reducing computational complexity of the algorithms, preventing overfitting, and focusing on the most relevant features in the dataset. Note that this technique should only be used when the number of features is low. --- # Monitor generative AI metrics URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/genai-metrics.html > Monitor LLMs and generative AI solutions to measure alignment, return on investment, provide guardrails. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/Using%20Custom%20Metrics%20to%20effectively%20monitor%20Generative%20AI/Using%20Custom%20Metrics%20to%20effectively%20monitor%20Generative%20AI.ipynb) While it gets easier to build generative AI solutions with each passing day, it is becoming evident that it is critical to effectively monitor these solutions to measure alignment and ROI, and to provide guardrails. Monitoring generative AI solutions or [LLMOps](https://www.pluralsight.com/resources/blog/data/what-is-llmops) is a multi-faceted endeavor and requires more than simple out-of-the-box metrics. Each business is unique and the solutions they build require customized monitoring metrics. This accelerator illustrates how businesses can use DataRobot to effectively and holistically monitor generative AI solutions, using metrics that segment into themes for covering the monitoring requirements of a majority of businesses. --- # Model evaluation and metrics URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/index.html > Model evaluation and metrics accelerators that you can add to your experiment workflow. | Topic | Description | | --- | --- | | Custom metrics for model selection | Demonstrates how to leverage DataRobot's Python client to extract predictions, compute custom metrics, and sort DataRobot models accordingly. | | t-SNE dimensionality reduction | Learn how to use t-SNE for dimensionality reduction and visualization of high-dimensional data, with examples for exporting these insights as files and plots. | | Monitor generative AI metrics | Monitor LLMs and generative AI solutions to measure alignment, return on investment, and provide guardrails using custom metrics. | | Event log viewer | Change the output of the User Activity Monitor to drop or anonymize columns for privacy while maintaining reporting consistency. | | LLM observability | Enable LLMOps or Observability in your existing Generative AI Solutions without refactoring code, with examples for major LLMs. | | Partial dependence plots (PDP/ICE) | Create one-way and two-way partial dependence plots (PDP), and Individual Conditional Expectations (ICE) insights using DataRobot. | | LIME explanations for models | Apply Local Interpretable Model-agnostic Explanations (LIME) to models built and deployed with DataRobot. | | Steel defect detection | Train a highly accurate and robust machine learning model capable of detecting and classifying any-sized scratch present in steel plates. | | Export model insights | Review examples for exporting a variety of DataRobot model insights and performance metrics as both machine-readable files and plots in multiple formats. | --- # Event log viewer URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/log-viewer.html > Change the output of the User Activity Monitor to allow you to drop an entire column of output or change the contents of that column in a way to preserve the anonymity of the column but maintain consistency for reporting. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/Simple_Log_Lister/Simple_Log_Lister.ipynb) This accelerator provides you with a method to change the output of the User Activity Monitor to allow you to drop an entire column of output or change the contents of that column in a way to preserve the anonymity of the column but maintain consistency for reporting. For the full list of columns, please refer to [the DataRobot documentation](https://docs.datarobot.com/en/docs/api/reference/public-api/analytics.html#get-apiv2eventlogs). --- # LLM observability URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/observability.html > Enable LLMOps or Observability in your existing Generative AI Solutions without refactoring code. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/generative_ai/external_monitoring/README.md) This accelerator shows how you can quickly and seamlessly enable LLMOps or [Observability](https://www.datarobot.com/platform/generative-ai/) in your existing Generative AI Solutions without refactoring code. This accelerator includes examples of the industry leading LLMs to show how easy it is to start monitoring an LLM and the solution built on top of it with all the hallmark features available in DataRobot's MLOps platform. The following LLMs are showcased in the accelerator: - PaLM 2 by Google - GPT4 by OpenAI - Titan by AWS Bedrock - Claude by Anthropic --- # Partial dependence plots (PDP/ICE) URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/pdp-ice.html > Create one-way and two-way partial dependence plots (PDP), and Individual Conditional Expectations (ICE) insights using DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/PDP_ICE/PDP%20and%20ICE%20AV.ipynb) This accelerator presents an example workflow to create one-way and two-way partial dependence plots (PDP), and Individual Conditional Expectations (ICE) insights using DataRobot. This accelerator has two parts: 1. Score data against a deployment and join the predictions back with the full dataset. 2. Use the scored dataset to gain insights by generation PDP and ICE plots. --- # LIME explanations for models URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/run-lime.html > Apply Local Interpretable Model-agnostic Explanations (LIME) to models built and deployed with DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/LIME%20with%20DataRobot%20Models/LIME%20analysis%20with%20DataRobot.ipynb) This accelerator shows how you can apply Local Interpretable Model-agnostic Explanations (LIME) to models built and deployed with DataRobot. LIME serves as another method in your toolbox to explain model predictions, complementing the built-in DataRobot capabilities of XEMP and SHAP prediction explanations. The accelerator demonstrates how to: - Connect to DataRobot using the API. - Build and deploy a model in DataRobot. - Execute LIME analysis with the deployed model. --- # Steel defect detection URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/steel-plate.html > Train a highly accurate and robust machine learning model capable of detecting and classifying any-sized scratch present in steel plates. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/faster-rcnn-custom-model/FasterR-CNN_training.ipynb) Modern machine learning techniques are now capable of assisting manufacturers streamline their product development in numerous ways. In this notebook, you are going to focus on detecting and classifying product defects using state of the art computer vision systems. Utilizing machine learning brings immense value to manufacturers, transforming their production processes and giving them an overall competitive edge. By leveraging these advanced methods, manufacturers can streamline product development, enhance defect detection accuracy, optimize operational efficiency, reduce costs, and ultimately deliver higher-quality products to meet the ever-growing demands of the market. In this accelerator, you will leverage computer vision to tackle the task of identifying product defects in hot-rolled steel plates, which are used extensively in construction and agriculture due to their superior strength and high formability. By leveraging an object detection model powered by machine learning, we can achieve precise and efficient detection and classification of one of the most prevalent product defects that steel manufacturers encounter: scratches. In practical applications, the inspection of steel plates is performed visually by an in-factory human examiner, which is time consuming and potentially unreliable. The approach will stand out from traditional techniques that do not utilize machine learning, as it offers the ability to automate the detection process, enhance accuracy, and reduce human effort and error. - Download the data - Perform the necessary data preprocessing - Split the data into training and validation datasets - Create our model - Write custom training and validation loops - Create a visualizer to evaluate model performance and take a look at the model's predictions At the end, you will have successfully trained a highly accurate and robust machine learning model capable of detecting and classifying any sized scratch present in steel plates. --- # Export model insights URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/model-eval-metrics/viz-output.html > Review examples for exporting a variety of DataRobot model insights and performance metrics as both machine-readable files and plots in multiple formats. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/Viz%20Output/Viz%20Output.ipynb) This accelerator presents some examples for taking a DataRobot project and exporting its model insights as both machine-readable files and plots in various file formats. It will demonstrate how to use the Python API client to: - Securely connect to DataRobot. - Get data. - Start a DataRobot binary classification project. - Retrieve and evaluate model performance and insights. - Package up insights and output them in various file formats. --- # AML alert scoring URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/alert-scoring.html > Develop a machine learning model that utilizes historical data, including customer and transactional information, to identify alerts that resulted in the generation of a Suspicious Activity Report (SAR). [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/anti-money-laundering) In this accelerator, delve into the exciting world of machine learning applied to Anti-Money Laundering (AML) alert scoring. The primary goal is to develop a powerful predictive model that utilizes historical customer and transactional data, enabling you to identify suspicious activities and generate crucial Suspicious Activity Reports (SARs). To ensure a smooth and efficient machine learning process, rely on the DataRobot Workbench. This tool allows you to analyze, clean, and curate the data, ensuring its quality and suitability for modeling. By utilizing the DataRobot API, you can seamlessly create and manage experiments, exploring a wide range of machine learning algorithms tailored for the AML alert scoring task. The flexibility and ease-of-use of the API make it a valuable asset for data scientists throughout the process. With just a few lines of code, you can train multiple machine learning models simultaneously, saving valuable time and computational resources. The model insights offered through the API provide invaluable interpretability. Additionally, the DataRobot API allows us to compute predictions on new data before deploying the model into production. This pre-deployment testing phase enables us to evaluate the model's performance in real-world scenarios and make necessary adjustments to address any potential issues. Uncover the incredible potential of machine learning in AML alert scoring, where data-driven insights make a tangible difference in the fight against money laundering. --- # Cold start demand forecasting URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/cold-start.html > This accelerator provides a framework to compare several approaches for cold start modeling on series with limited or no history. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/Demand_forecasting2_cold_start/End_to_end_demand_forecasting_cold_start.ipynb) The cold start demand forecasting problem refers to the challenge of predicting future demand for a new product or service with little or no historical sales data available. This situation typically arises when a company introduces a new product or service to the market or a new product is launched in a store that is already being sold in other stores, and there is no past data available for training a machine learning model to predict future demand. In traditional demand forecasting, historical sales data is used to train a machine learning model that can predict future demand. However, in the case of a new product, there is no historical data available. This presents a significant challenge because accurate demand forecasting is critical for making informed decisions about inventory, pricing, and marketing strategies. This second accelerator of a three-part series on demand forecasting provides the building blocks for cold start modeling workflow on series with limited or no history. This accelerator provides a framework to compare several approaches for cold start modeling. The previous notebook aims to inspect and handle common data and modeling challenges, identifies common pitfalls in real-life time series data, and provides helper functions to scale experimentation with the tools mentioned above and more. The dataset consists of 50 series (46 SKUs across 22 stores) over a 2 year period with varying series history, typical of a business releasing and removing products over time. The test dataset contains 20 additional series with little or no history which were not present in the training dataset. --- # Demand forecasting with Databricks URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/dbx-forecast.html > How to use DataRobot with Databricks to develop, evaluate, and deploy a multi-series demand forecasting model. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/ecosystem_integration_templates/Databricks%20%26%20Datarobot%20-%20Large%20Scale%20Forecasting/Databricks%20%26%20Datarobot%20-%20Large%20Scale%20Forecasting.ipynb) This accelerator is developed for use with Databricks to help you leverage the power of DataRobot for time-series modeling within a Databricks ecosystem. Demand forecasting models are valuable to many businesses because they apply to high-value use cases such as improving inventory management, supply chain processes, and store staffing. However, building forecasting models can be challenging and time-consuming given the amount of experimentation typically required, from performing time series feature engineering to implementing diverse and complex time-series algorithms and evaluating results. The time series capabilities of DataRobot accelerate this process so you can rapidly build and test many modeling approaches and productionalize your models with model monitoring. This accelerator can be imported into Databricks notebooks to walk you through how to use DataRobot with Databricks to develop, evaluate, and deploy a multi-series demand forecasting model. The notebook utilizes the DataRobot API to access DataRobot capabilities while ingesting data from Databricks for model building and scoring. In this accelerator you will: - Connect to DataRobot in a Databricks Notebook - Import data from Databricks into the AI Catalog - Create a time series forecasting project and run Autopilot - Retrieve and evaluate model performances and insights - Make new predictions with a test dataset - Deploy a model with monitoring in DataRobot MLOps - Forecast predictions via the Prediction API --- # Time series demand forecasting URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/demand-flow.html > Perform large-scale demand forecasting using DataRobot's Python package. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/Demand_forecasting1_end_to_end/End_to_end_demand_forecasting.ipynb) Demand forecasting models have many common challenges: large quantities of SKUs or series to predict, partial history or irregular history for many SKUs, multiple locations with different local or regional demand patterns, and cold-start prediction requests from the business for new products. The list goes on. Time series in DataRobot, however, has a diverse range of functionality to help tackle these challenges. For example: - Automatic feature engineering and creation of lagged variables across multiple data types, as well as training dataset creation. - Diverse approaches for time series modeling with text data, learning from cross-series interactions and scaling to hundreds or thousands of series. - Feature generation from an uploaded calendar of events file specific to your business or use case. - Automatic backtesting controls for regular and irregular time-series. - Training dataset creation for irregular series via custom aggregations. - Segmented modeling, hierarchical clustering for multi-series models, multimodal modeling, and ensembling. - Periodicity and stationarity detection, and automatic feature list creation with various differencing strategies. - Cold start modeling on series with limited or no history. - Insights for all of the above. In this first installment of a three-part series on demand forecasting, this accelerator provides the building blocks for a time-series experimentation and production workflow. This notebook provides a framework to inspect and handle common data and modeling challenges, identifies common pitfalls in real-life time series data, and provides helper functions to scale experimentation with the tools mentioned above and more. The dataset consists of 50 series (46 SKUs across 22 stores) over a two year period with varying series history, typical of a business releasing and removing products over time. --- # Demand forecasting retraining URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/df-retrain.html > Implement retraining policies with DataRobot MLOps demand forecast deployments. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/Demand_forecasting3_retraining/End_to_end_demand_forecasting_retraining.ipynb) This accelerator demonstrates retraining policies with DataRobot MLOps demand forecast deployments. This accelerator is a another installment of a series on demand forecasting. The [first accelerator](https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/demand-flow.html) focuses on handling common data and modeling challenges, identifies common pitfalls in real-life time series data, and provides helper functions to scale experimentation. The [second accelerator](https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/cold-start.html) provides the building blocks for cold start modeling workflow on series with limited or no history. They can be used as a starting point to create a model deployment for the app. The [third accelerator](https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/ml-what-if.html) is a what-if app that allows you to adjust certain known in advance variable values to see how changes in those factors might affect the forecasted demand. --- # Financial planning analysis URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/fin-plan.html > This accelerator illustrates an end-to-end financial planning and analysis workflow in DataRobot. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/FP%26A/FP%26A.ipynb) This accelerator illustrates an end-to-end financial planning and analysis workflow in DataRobot. Time series forecasting in DataRobot has a huge suite of tools and approaches to handle highly complex multiseries problems. DataRobot is used for the model training, selection, deployment, and creation of forecasts. While this example will leverage a snapshot file as a data source, this workflow applies to any data source, e.g. Redshift, S3, Big Query, Synapse, etc. This notebook will demonstrate how to use the Python API client to: - Connect to DataRobot - Import and preparation of data for time series modeling - Create a time series forecasting project and run Autopilot - Retrieve and evaluate model performance and insights - Making forward looking forecasts - Evaluating forecasts vs. historical trends - Deploy a model --- # Flight delay prediction URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/flight-delays.html > Designed for DataRobot trial users, experience an end-to-end DataRobot workflow using a use case that predicts flight delays. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/Flight%20Delays%20-%20Starter%20Use%20Case%20for%20New%20DataRobot%20Users/Flight%20Delays%20-%20Starter%20Use%20Case%20for%20New%20DataRobot%20Users.ipynb) This accelerator aims to assist DataRobot trial users by providing a guided walkthrough of the trial experience. DataRobot suggests that you complete the Flight Delays sample use case in the graphical user interface first, and then return to this accelerator. In this notebook, you will: - Create a Use Case. - Import data from an S3 bucket (this differs from the UI walkthrough). - Perform a data wrangling operation to create the target feature with code (this also differs from the UI walkthrough). - Register the wrangled data set. - Explore the new data set. - Create an experiment and allow DataRobot automation to populate it with many modeling pipelines. - Explore model insights for the best performing model. - View the modeling pipeline for the best performing model. - Register a model in Registry. - Configure a deployment. - Create a deployment. - Make predictions using the deployment. - Review deployment metrics. --- # Fraud detection with Neo4j URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/fraud-detection.html > Build a fraud detection pipeline using Neo4j for storing and querying a knowledge graph. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/datarobot-neo4j-knowledge-graph-for-fraud-detection) This accelerator demonstrates how to build a fraud detection pipeline using Neo4j and DataRobot. Use Neo4j to store and query a knowledge graph of clients, loans, addresses, and more. Then, use DataRobot to build a predictive model with graph-based features. The accelerator contains multiple notebooks. The first notebook walks through installing a Neo4j 4.4.11 instance, loading a Neo4j database, and uploading a dump file with the CLI. The second notebook outlines how to extract graph data into training and holdout CSVs, upload training data to DataRobot, and build a classification model for scoring. --- # Time series and specific use cases URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/index.html > Accelerators for time series and other specific use cases that you can add to your experiment workflow. | Topic | Description | | --- | --- | | AML alert scoring | Develop a machine learning model that utilizes historical data, including customer and transactional information, to identify alerts that resulted in the generation of a Suspicious Activity Report (SAR). | | Cold start demand forecasting | This accelerator provides a framework to compare several approaches for cold start modeling on series with limited or no history. | | Demand forecasting with Databricks | How to use DataRobot with Databricks to develop, evaluate, and deploy a multi-series demand forecasting model. | | Time series demand forecasting | Perform large-scale demand forecasting using DataRobot's Python package. | | Demand forecasting retraining | Implement retraining policies with DataRobot MLOps demand forecast deployments. | | Financial planning analysis | This accelerator illustrates an end-to-end financial planning and analysis workflow in DataRobot. | | Flight delay prediction | Designed for DataRobot trial users, experience an end-to-end DataRobot workflow using a use case that predicts flight delays. | | Fraud detection with Neo4j | Build a fraud detection pipeline using Neo4j for storing and querying a knowledge graph. | | Multi-model analysis | Use Python functions to aggregate DataRobot model insights into visualizations. | | Netlift modeling | Leverage machine learning to find patterns around the types of people for whom marketing campaigns are most effective. | | What-if demand forecasting | Discover how to use a what-if app to adjust known-in-advance variables and explore how changes in factors like promotions, pricing, or seasonality can impact demand forecasts. | | No-show appointment prediction | Build a model that identifies patients most likely to miss appointments, with correlating reasons. | | Lumber price forecasting with Ready Signal | Use Ready Signal to add external control data, such as census and weather data, to improve time series predictions. | | Recommendation engine | Explore how to use historical user purchase data in order to create a recommendation model, which will attempt to guess which products out of a basket of items the customer will be likely to purchase at a given point in time. | | Panel data self-joins | Explore how to implement self-joins in panel data analysis. | | Technical price prediction | Leverage historical insurance claim data for modeling and analysis. | | Statistical tests with Airflow | Review an example workflow for carrying out statistical tests, notify stakeholders of any issues via Slack, and generate automated compliance documentation with the test results. | | Trading volume profile curve | Use a framework to build models that will allow you to predict how much of the next day trading volume will happen at each time interval. | | Hierarchical reconciliation | Learn how to reconcile independent time series forecasts with a hierarchical structure. | | Visual AI for geospatial data | Learn how to use Visual AI to represent geospatial data for enhanced analysis. | --- # Multi-model analysis URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/ml-analysis.html > Use Python functions to aggregate DataRobot model insights into visualizations. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/multi_model_analysis/Multi-Model%20Analysis.ipynb) DataRobot is designed to help you experiment with different modeling approaches, data preparation techniques, and problem framings. You can iterate fast with a tight feedback loop to quickly arrive at the best approach. Sometimes you may wish to break your use case into multiple models, likely across multiple DataRobot projects. Maybe you want to build a separate model for each country or one for different periods of the year. In this case, it helps to bring all of your model performances and insights into one chart. This accelerator shares several Python functions that can take the DataRobot insights—specifically model error, feature effects (partial dependence), and feature importance (SHAP or permutation-based) and bring them together into one chart, allowing you to understand all of your models in one place and more easily share your findings with stakeholders. --- # Netlift modeling URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/ml-uplift.html > Leverage machine learning to find patterns around the types of people for whom marketing campaigns are most effective. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/marketing_uplift_modeling/uplift_modeling.ipynb) Uplift modeling, also referred to as "netlift" modeling, is an approach used often in marketing to isolate the impact of a marketing campaign on specific prospective customers’ propensity to purchase something. The underlying example in this DataRobot AI Accelerator is exactly that, but more generally this approach could be used to isolate the impact of any “intervention” on the propensity of any positive response. The key challenge in uplift modeling is to isolate the effect of the campaign, because no individual person can be observed both receiving the campaign and not receiving the campaign. The accelerator addresses this key challenge, as well as other tips and tricks for uplift modeling. In many cases, the historical strategy for determining who received a campaign targeted those already likely to purchase the product (or generally, produce a favorable response). That approach would suggest a simple trend that receiving the campaign increases the likelihood to purchase, but many other features about the customers may be confounding the isolated impact of the campaign. In fact, it's possible that a campaign that targeted already high-probability buyers actually reduced their probability of purchase. These are the so-called "sleeping dogs'' in marketing lingo. From an ROI standpoint, increasing the probability to purchase on one group of prospects from 25% to 50% is just as valuable as increasing that probability on another group from 50% to 75% (assuming the groups are roughly the same size, with the same expected revenue values). So what you're really trying to ask from machine learning models is this: on which prospective customers will the campaign increase the probability of purchase by the greatest amount? This accelerator uses a generic dataset where the favorable outcome is binary: whether or not a product was purchased. The "treatment", or campaign, is simple: a single campaign type that was sent randomly to some prospective buyers, though it also discusses how these methods can be extrapolated to the common case where there was selection bias in the campaign. Leverage machine learning to find patterns around the types of people for whom the campaign is most effective, controlling for their baseline likelihood to purchase in the case that they don't see a campaign. Uplift use cases require some additional post-processing to extract and evaluate the "uplift score", and thus this use case is an ideal candidate for leveraging the DataRobot programmatic API, to seamlessly integrate powerful machine learning with one's typical coding pipeline. While working through the provided Jupyter Notebook, the following concepts and strategies will be reinforced: 1. Data formatting tricks to extract the most from your uplift models. 2. How to leverage DataRobot's API to integrate powerful machine learning into your code-first pipelines. 3. How to extract uplift scores from a single, binary classification model. 4. How to evaluate and understand those uplift scores, and their implied ROI. 5. Considerations for cases where your historical, training data exhibits selection bias, where the campaign was not randomly sent. --- # What-if demand forecasting URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/ml-what-if.html > Discover how to use a what-if app to adjust known-in-advance variables and explore how changes in factors like promotions, pricing, or seasonality can impact demand forecasts. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/Demand_forecasting4_what_if_app/README.md) This demand forecasting what-if app allows you to adjust certain known in advance variable values to see how changes in those factors might affect the forecasted demand. Some examples of factors that might be adjusted include marketing promotions, pricing, seasonality, or competitor activity. By using the app to explore different scenarios and adjust key inputs, you can make more accurate predictions about future demand and plan accordingly. This app is a third installment of a three-part series on demand forecasting. The [first accelerator](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/Demand_forecasting1_end_to_end/End_to_end_demand_forecasting.ipynb) focuses on handling common data and modeling challenges, identifies common pitfalls in real-life time series data, and provides helper functions to scale experimentation. The [second accelerator](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/Demand_forecasting2_cold_start/End_to_end_demand_forecasting_cold_start.ipynb) provides the building blocks for cold start modeling workflow on series with limited or no history. They can be used as a starting point to create a model deployment for the app. --- # No-show appointment prediction URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/no-show.html > Build a model that identifies patients most likely to miss appointments, with correlating reasons. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/healthcare_appointment_no_show_prediction/no_show.ipynb) Many people are guilty of having canceled a doctor’s appointment. However, although canceling an appointment does not seem too disastrous from the patient’s point of view, no-shows cost outpatient health centers a staggering 14% of anticipated daily revenue (JAOA). Missed appointments trickle into lower utilization rates for not only doctors and nurses but also the overhead costs required to run outpatient centers. In addition, patients missing their appointments risk facing poorer health outcomes as they are unable to access timely care. While outpatient centers employ solutions such as calling patients ahead of time, these high touch resources investments are often not prioritized for patients with the highest risk of no-shows. Low touch solutions such as automated texts are effective tools for mass reminders but do not offer necessary personalization for patients at the highest risk of no-shows. This accelerator shows how to identify clients who are likely to miss appointments ("no-shows") and take action to prevent that from happening. --- # Lumber price forecasting with Ready Signal URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/ready-signal.html > Use Ready Signal to add external control data, such as census and weather data, to improve time series predictions. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced_ml_and_api_approaches/data_enrichment_ready_signal_ts/DataRobot_RXA.ipynb) In this accelerator, you will explore how to bring external data from Ready Signal to help augment your time series forecasting accuracy. Ready Signal is an AI-powered data platform that provides access to over 500 normalized, aggregated, and automatically updated data sources for predictive modeling, experimentation, business intelligence, and other data enrichment needs. The data catalog includes micro/macro-economic indicators, labor statistics, demographics, weather, and more. Its AI recommendation engine and auto feature engineering capabilities make it easy to integrate with existing data pipelines and analytics tooling, accelerating and enhancing how relevant third-party data is leveraged. Here, DataRobot provides an example of predicting lumber price combined with the most relevant external data automatically identified by ReadySignal based on correlation with the target variable. The workflow can be applied to any time series forecasting project. --- # Recommendation engine URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/rec-engine.html > Explore how to use historical user purchase data in order to create a recommendation model, which will attempt to guess which products out of a basket of items the customer will be likely to purchase at a given point in time. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/Ecommerce_recommendation_engine/Recommendation%20Engine.ipynb) The accelerator provided in this notebook trains a model on historical customer purchases in order to make recommendations for future visits. The DataRobot features that will be utilized in this notebook are multi-Label modeling and feature discovery. Together the resulting model can provide rank ordered suggestions of content, product, or services that a specific customer might like. In the notebook, you will: - Analyze the datasets required - Create a multilabel dataset for training - Connect to DataRobot - Configure a feature discovery project - Generate features and models - Generate recommendations for new visits --- # Panel data self-joins URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/self-joins.html > Explore how to implement self-joins in panel data analysis. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/Self_join_technique_for_panel_data) In this accelerator, explore how to implement self-joins in panel data analysis. Regardless of your industry, if you work with panel data, this guide is tailored to help you accelerate feature engineering and extract valuable insights. Panel data, with multiple observations for consistent subjects over time, is ubiquitous in various domains. While panel data is often spread across multiple tables, it can also exist in a single dataset with multiple features suitable as panel dimensions. The self-join technique enables automated, time-aware feature engineering with just one dataset, generating hundreds of candidate features of lagged aggregations and statistics. Combining these features within panel dimensions can substantially improve predictive model performance. The accelerator focuses on predicting airline take-off delays of 30 minutes or more to illustrate the self-join technique. However, this framework applies broadly across verticals and can easily be adapted to your use case. Using a single dataset, join it four times across different features, engineer time-based features from each join, using the AI Catalog for data management. The accelerator covers data preparation with multiple joins and time horizons, how to mitigate target leakage with multiple feature lists as well as time gaps in time-aware joins. Panel data analysis unlocks valuable insights into subjects evolving over time, and is often overlooked when there is a singular dataset. --- # Technical price prediction URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/tech-prices.html > Leverage historical insurance claim data for modeling and analysis. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/insurance_pricing/Code%20first%20Incurred%20Claims%20-%20Insurance%20pricing.ipynb) This accelerator serves as a comprehensive guide to insurance pricing, leveraging historical claims data for modeling and analysis. The primary objective of this notebook is to enable insurance professionals and data scientists to predict insurance pricing accurately and efficiently with DataRobot platform. This accelerator does the following: - Set up the environment for insurance pricing modeling - Import the necessary libraries and emphasize data preparation for use with DataRobot - Visualize the distribution of claim amounts - Create two options for modeling workflows for the insurance pricing project: Pure Premium vs. Frequency and Severity - Explore different feature list and model customization Following the modeling phases, the accelerator transitions to result analysis and business considerations. It discusses testing the models and computing various business metrics and scenarios. The accelerator also covers how to convert from a technical price to a market premium with the inclusion of fixed expenses and variable costs. This part also includes computing loss ratios by various segments, which is crucial for assessing risk and profitability. Finally, the analysis phase includes a dislocation premium chart to visualize premium impact. After thorough analysis and fine-tuning, the accelerator explains how to deploy developed models into production. This is a critical step for implementing the insurance pricing models in real-world scenarios and utilizing them for decision-making. The final section of the accelerator is dedicated to advanced workflows. It introduces feature discovery using secondary claims databases. This advanced approach can significantly reduce the time investment needed from data scientists and engineers, making it a valuable addition to the insurance pricing modeling process. --- # Statistical tests with Airflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/test-airflow.html > Review an example workflow for carrying out statistical tests, notify stakeholders of any issues via Slack, and generate automated compliance documentation with the test results. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/shu/stat-tests/advanced_ml_and_api_approaches/stat_test_airflow/stat_test_airflow.ipynb) This notebook presents an example workflow for carrying out statistical tests, notifying stakeholders of any issues via Slack, and generating automated compliance documentation with the test results. It will demonstrate how to create a pipeline of statistical tests, at different stages of the model development cycle, and integrate with Apache Airflow, including: - Using exploratory statistical tests as part of model training. - Scoring a DataRobot model. - Running any arbitrary statistical tests. - Registering the test results to the model version. - Generating automated compliance documentation using customized templates. - Creating the pipeline with Apache Airflow. --- # Trading volume profile curve URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/ts-factory.html > Use a framework to build models that will allow you to predict how much of the next day trading volume will happen at each time interval. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/tree/main/use_cases_and_horizontal_approaches/trading_volume_profile_curve_model_factory) In securities trading, it’s often useful to have an idea of how trading volume for a particular instrument will be distributed over the market session. This is done by building a volume curve — essentially, a prediction of how much of the volume will fall within the different time intervals (“time slices”) in a trading day. Volume curves allow traders to better anticipate how to time and pace their orders and are used as inputs into algorithmic execution strategies such as VWAP (volume weighted average price) and IS (implementation shortfall). Historically, volume curves have been built by taking the average share of volume for a particular time slice over the last N trading days (for instance, the share of the daily volume in AAPL that traded between 10:35 and 10:40am on each of the last 20 trading days, on average), with manual adjustments to take account of scheduled events and anticipated differences. Machine learning allows you to do this in a structured, systematic way. The goal of this AI accelerator is to provide a framework to build models that will allow you to predict how much of the next day trading volume will happen at each time interval. The granularity can vary from minute by minute (or even lower) to hourly or daily. If you are working with high granularity, such as minute by minute intervals, having a single time series model to predict the next 1440 minutes (or 480, based on how long the market is open) becomes problematic. Instead, consider a time series model per interval (minute, half hour, hour, etc.) so that each model is only forecasting one step ahead. You can then bring together the predictions of all the models to create the full curve for the next day. Furthermore, while a model is built to predict each time interval, the model isn't restricted to data for that interval, but can leverage a wider window. While the motivation for this repository is a financial markets use case, it should be useful in other scenarios where predictions are required at a high resolution, such as predictive maintenance. ## Challenges - The number of models or deployments can explode, and you need to keep track of all of them. - Each model needs slightly different data. - Even if you are creating a model per minute, you want to use data from earlier and later on in the day. - You want to see a unified result (a single curve for the whole trading day). ## Approach - Train a model per interval, but leverage data outside of the interval by "widening" the time window on which it is trained. - Use a data frame to track all the projects, models and deployments corresponding to each interval. This will make it easy to stitch all the predictions together to build the next day(s) curve. --- # Hierarchical reconciliation URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/ts-recon.html > Learn how to reconcile independent time series forecasts with a hierarchical structure. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/time_series_hierarchical_reconciliation/time_series_hierarchical_reconciliation.ipynb) This AI Accelerator demonstrates how to reconcile (e.g., post-processing to sum appropriately) independent time series forecasts with a hierarchical structure. Reconciling, also known as making ["coherent"](https://otexts.com/fpp3/hierarchical.html) forecasts, is often a requirement when submitting hierarchical forecasts to stakeholders. This notebook leverages the increasingly popular [HierarchicalForecast](https://nixtlaverse.nixtla.io/hierarchicalforecast/index.html) python library to do the reconciliation on forecasts generated from DataRobot time series deployments. The steps demonstrated are as follows: 1. Installing hierarchicalforecast 2. Importing libraries 3. Loading the example dataset 4. Preparing training data for each hierarchy 5. Building models for each level 6. Deploying models for each level 7. Making forecasts 8. Preparing the forecasts 9. Reconcile forecasts 10. Comparing forecasts 11. Conclusion Note that steps 2-6 steps are purely for providing example time series deployments and forecasts from those deployments (in case you don't have any). If you already have a set of forecasts you want to reconcile, feel free to skip to step 7. --- # Visual AI for geospatial data URL: https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/time-series/viz-geo.html > Learn how to use Visual AI to represent geospatial data for enhanced analysis. [Access this AI accelerator on GitHub](https://github.com/datarobot-community/ai-accelerators/blob/main/use_cases_and_horizontal_approaches/VisualAI_for_geospatial/Visual%20AI%20for%20geospatial%20data.ipynb) This accelerator shows how you can use Visual AI on geospatial data. Instead of deriving numeric features from the georeferenced data, you look at the geospatial data as images. For example, if you have a map of population distribution, instead of extracting the population that corresponds to each row of the main table you can pass the region of the map that corresponds to that row. This provides more information than a raw count of the population would, as it also encodes the distribution within the region (is it uniform or does it concentrate in some areas? what is the shape?, etc.) The example used to illustrate the approach comes from work done with the Virtue Foundation. As part of the "Data Mapping Initiative", DataRobot has built models to identify suitable locations for new healthcare facilities. By looking at the location of existing hospitals and clinics as a function of several features (road networks, population, terrain, etc.) you find which other areas are suitable in terms of these features (similar to a propensity model). --- # AI consumable assets URL: https://docs.datarobot.com/en/docs/api/dev-learning/ai-assets.html > Structured exports and indexes so AI assistants can use DataRobot documentation reliably. Use the following resources to give AI assistants reliable access to DataRobot documentation. | Resource | Description | | --- | --- | | llms.txt | A markdown file that gives LLMs a structured map to find key information in the DataRobot documentation. Start here for a compact index and pointers into the docs. It complements llms-full.md by providing navigation and context instead of the full corpus alone. | | llms-full.md | A markdown file that contains the entirety of the DataRobot documentation in one place for search, retrieval, or grounding. This is a large file. | | docs-llm.zip | A compressed archive of the same full documentation export as llms-full.md, for easier download and offline use. | | Per-page Markdown (*.html.md) | For any published documentation page, you can download a Markdown version at the same path by appending .md after the .html in the URL. For example, this page: ai-assets.html.md. | | OpenAPI specification | Reference the OpenAPI specification for the DataRobot REST API, which helps automate the generation of a client for languages that DataRobot doesn't directly support. It also assists with the design, implementation, and testing integration with DataRobot's REST API using a variety of automated OpenAPI-compatible tools. Note that accessing the OpenAPI spec requires you to be logged in to the DataRobot application. | --- # Developer quickstart URL: https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html > Set up your machine for DataRobot development—Personal API keys, CLI (`dr`), Python SDK, REST (cURL), Agent Assist, authentication, verification, and an optional end-to-end modeling and deployment lab. Use this page to get started with the [DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html), [Python API client](https://pypi.org/project/datarobot/), [REST API](https://docs.datarobot.com/en/docs/api/reference/public-api/index.html), and [Agent Assist](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/installation.html). The following sections include ways to supply credentials, install these tools, and [a hands-on modeling lab](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html#hands-on-build-and-deploy-a-first-model) that trains, deploys, and scores predictions. | Tool | What it is | Best for | | --- | --- | --- | | Python API client and REST API | Ways to programmatically interact with DataRobot. | Scripting, notebooks, CI pipelines, custom pipelines | | DataRobot CLI (dr) | An open source tool for working with DataRobot from your terminal. | Local development, templates, dr run, dr task, agentic projects | | Agent Assist (dr assist) | An interactive AI assistant optimized for the development of AI agents. | Designing, coding, and deploying agents | ## Prerequisites You need the following on your machine, depending on the tools you want to use: **Python:** Python 3.7 or later for the datarobot package (Python 3.10+ is required if you also use Agent Assist or related agentic tooling) A DataRobot account pip **REST:** curl jq (optional, for readable JSON in examples) A DataRobot account **CLI:** DataRobot account: Access to a DataRobot instance (cloud or Self-Managed). If you don't have an account, sign up at DataRobot or contact your organization's DataRobot administrator. Git: For cloning templates (version 2.0+). Install Git from git-scm.com if not already installed. Verify installation: git --version Task: For running tasks. Install Task from taskfile.dev if not already installed. Verify installation: task --version Terminal: For CLI access. macOS/Linux: Use Terminal, iTerm2, or your preferred terminal emulator. Windows: Use PowerShell, Command Prompt, or Windows Terminal. **Agent Assist:** Operating system: macOS or Linux (Windows requires WSL or another supported environment) Python: 3.10 or higher Tool Version Description Installation dr-cli >= 0.2.50 The DataRobot CLI. dr-cli installation git >= 2.30.0 Version control. git installation uv >= 0.9.0 Python package manager. uv installation Pulumi >= 3.163.0 Infrastructure as Code. Pulumi installation Taskfile >= 3.43.3 Task runner. Taskfile installation Node.js >= 24 JavaScript runtime (for example, for template frontend). Node.js installation On macOS, several tools can be installed at once: ``` brew install datarobot-oss/taps/dr-cli uv pulumi/tap/pulumi go-task node git python ``` ## Configure your environment This section is the long-form reference for keys, regions, and credentials. You can work through it in parallel with [Install methods](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html#install-methods) if you are bringing up the CLI and Python packages first. You will use what you set here in [Verify your setup](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html#verify-your-setup). Configure API keys (your identity on the platform), the regional API endpoint (where your tenant lives), and how your tools read credentials ( `drconfig.yaml`, environment variables, or explicit code). ### Create a DataRobot API key For local scripts, notebooks, and general API access, create a key on the Personal API keys tab. In the DataRobot UI, open the user menu and choose API keys and tools (sometimes listed under account or developer settings). Step-by-step UI detail and screenshots are in [API keys and tools](https://docs.datarobot.com/en/docs/platform/acct-settings/api-key-mgmt.html#api-key-management). > [!TIP] Direct link (US SaaS) > If you sign in at `https://app.datarobot.com`, you can open [Developer tools](https://app.datarobot.com/account/developer-tools), select Personal API keys, then Create new key. For EU or JP, replace the hostname with the same domain you use in the browser (for example `app.eu.datarobot.com` or `app.jp.datarobot.com`). 1. From the DataRobot UI, click the user icon and selectAPI keys and tools. 2. Stay onPersonal API keysand clickCreate new key. 3. Name the key and confirm creation. The key is active immediately. The API keys and tools page can show more than one key type. Use the tab that matches your needs: | Tab | Use it for | | --- | --- | | Personal API keys | This developer quickstart—Python, cURL, local development, most automation. Inherits your user permissions. | | Application API keys | Custom applications (for example Streamlit or React apps) calling DataRobot on behalf of users. Scoped to a registered application. See Application API keys. | | Agent API keys | Deployed agents and agentic workflows making service-to-service calls. See Agent API keys if you build agents. | If you are running code from a terminal or IDE, start with a personal API key. Switch key types when you ship an app or a deployed agent that needs its own scoped credential. Each personal key lists: | Label | Element | Description | | --- | --- | --- | | (1) | Name | Editable label for the key. | | (2) | Key | Secret value used in the Authorization header. | | (3) | Date created | Creation date; unused keys may show "—". | | (4) | Last used | Last time the key was used. | ### Retrieve the API endpoint Every request uses a base URL that ends with `/api/v2`. Use the same hostname you use to open DataRobot in the browser, then append `/api/v2`. | Region | Example UI URL | API endpoint root | | --- | --- | --- | | AI Platform (US) | https://app.datarobot.com | https://app.datarobot.com/api/v2 | | AI Platform (EU) | https://app.eu.datarobot.com | https://app.eu.datarobot.com/api/v2 | | AI Platform (JP) | https://app.jp.datarobot.com | https://app.jp.datarobot.com/api/v2 | | Self-Managed AI Platform | Your organization's URL | https://{your-datarobot-host}/api/v2 | ### Configure API authentication Your code or shell needs an endpoint and a bearer token (personal API key). Common options include: **drconfig.yaml:** A `drconfig.yaml` file is the default for DataRobot's Python client. Typical location: `~/.config/datarobot/drconfig.yaml`. You can use other paths and pass them explicitly to the client. ``` endpoint: 'https://app.datarobot.com/api/v2' token: 'token-string' ``` Python (default path): ``` import datarobot as dr ``` Python (custom path): ``` import datarobot as dr dr.Client(config_path="") ``` cURL (read values into environment variables): ``` export DATAROBOT_ENDPOINT=$(grep 'endpoint:' ~/.config/datarobot/drconfig.yaml | cut -d "'" -f2) export DATAROBOT_API_TOKEN=$(grep 'token:' ~/.config/datarobot/drconfig.yaml | cut -d "'" -f2) ``` ``` curl --location -X GET "${DATAROBOT_ENDPOINT}/projects" --header "Authorization: Bearer ${DATAROBOT_API_TOKEN}" ``` **Environment variables:** Windows: ``` setx DATAROBOT_ENDPOINT "https://app.datarobot.com/api/v2" setx DATAROBOT_API_TOKEN "your_api_token" ``` Close and reopen the terminal. To persist through the UI, search for "Environment Variables", then add `DATAROBOT_ENDPOINT` and `DATAROBOT_API_TOKEN` under system variables. macOS and Linux: ``` export DATAROBOT_ENDPOINT="https://app.datarobot.com/api/v2" export DATAROBOT_API_TOKEN="your_api_token" ``` Add the same lines to `~/.zshrc`, `~/.bashrc`, or `~/.bash_profile` if you want them in every session. Python: ``` import datarobot as dr dr.Project.list() ``` cURL: ``` curl --location -X GET "${DATAROBOT_ENDPOINT}/projects" --header "Authorization: Bearer ${DATAROBOT_API_TOKEN}" ``` **Embed in your code:** Optional and convenient for experiments. Never commit secrets to Git. Python: ``` import datarobot as dr dr.Client(endpoint='https://app.datarobot.com/api/v2', token='token-string') ``` cURL: ``` curl --location --request GET 'https://app.datarobot.com/api/v2/projects/' \ --header 'Authorization: Bearer ' ``` > [!TIP] Optional: DataRobot CLI (dr auth login) > If you use the [DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html), run `dr auth login` to complete browser-based authentication and store settings under `~/.config/datarobot/drconfig.yaml` (see [Authentication management](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/auth.html)). The Python client can read that file, so you may not need a separate manual `drconfig.yaml` step. > [!TIP] Authenticate with your own identity provider (Inbound OAuth) > If your organization uses Inbound OAuth, you can authenticate with an access token from your own IdP (Okta, Microsoft Entra ID, Ping, Auth0) instead of a DataRobot API key. Exchange the IdP token for a DataRobot access token, then use it as the bearer token above. See [Authenticate with an external IdP token](https://docs.datarobot.com/en/docs/api/dev-learning/inbound-oauth-tokens.html). #### Credential resolution order DataRobot tools typically resolve credentials in this order: 1. DATAROBOT_API_TOKEN and DATAROBOT_ENDPOINT environment variables 2. A .env file in the current working directory (when the tool you use loads it) 3. ~/.config/datarobot/drconfig.yaml (for example after dr auth login ) 4. ~/.config/datarobot/agent_assist_config.yaml for some Agent Assist flows For `drconfig.yaml` layout and cURL examples, see [Configure API authentication](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html#configure-api-authentication). ## Install methods The following sections outline various methods for installing DataRobot's code-first tools. ### CLI installer macOS / Linux: Installs the DataRobot CLI. Then add the Python packages and Agent Assist yourself: ``` curl https://cli.datarobot.com/install | sh pip install datarobot datarobot-predict dr plugin install assist ``` Windows (PowerShell): From [CLI installation](https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html#installation): ``` irm https://cli.datarobot.com/winstall | iex ``` After install, run `pip install datarobot datarobot-predict` and `dr plugin install assist` in an environment where Python is available. If you have not set an endpoint and token yet, work through [Configure API authentication](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html#configure-api-authentication) (for example with `dr auth login` and [Authentication management](https://docs.datarobot.com/en/docs/agentic-ai/cli/commands/auth.html)), then [Verify your setup](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html#verify-your-setup). For Homebrew, pinned versions, or binaries, see [Getting started with DataRobot CLI](https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html). ### Setup via AI coding tool Paste the block below into Claude Code, Cursor, or another agentic coding tool with an empty or disposable directory. It links to install and auth topics on docs.datarobot.com so the agent can install prerequisites, configure credentials, and verify with a live API call. ``` You are helping me set up DataRobot for local development. Do all of the following: 1. Read the following docs and follow the install steps exactly: - https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html - https://docs.datarobot.com/en/docs/agentic-ai/cli/getting-started.html - https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/installation.html 2. Detect my OS (macOS / Linux / WSL). On macOS use Homebrew where the docs recommend it; on Linux use the documented installers. 3. Install: Python 3.10 or later, git, uv, dr-cli, Pulumi, go-task, and Node.js 24 (or the minimum versions described in those docs and the manual table below). 4. Install the Python SDK: `pip install datarobot datarobot-predict` 5. Prompt me for my DataRobot Personal API key. If I don't have one, open https://app.datarobot.com/account/developer-tools and tell me to use the "Personal API keys" tab (not Application or Agent keys). 6. Run `dr auth login` to persist credentials in `~/.config/datarobot/drconfig.yaml` and, if I want shell persistence, add `DATAROBOT_ENDPOINT` and `DATAROBOT_API_TOKEN` to my shell rc file. 7. Install Agent Assist as a CLI plugin: `dr plugin install assist` 8. Verify everything works by: - Running `dr --version`, `dr plugin list`, and `dr assist --help` - Executing this Python snippet and printing the first three project names: ```python import datarobot as dr dr.Client() for p in dr.Project.list()[:3]: print(p.project_name) ``` 9. Print a summary of what was installed and the config file locations. Do not run `dr assist` yet — only install and verify. ``` ### Manual install: Python packages for full agentic stack macOS ``` brew install datarobot-oss/taps/dr-cli uv pulumi/tap/pulumi go-task node git python pip install datarobot datarobot-predict dr plugin install assist ``` Linux / WSL — install each tool from its official installer (see the table), then: ``` pip install datarobot datarobot-predict dr plugin install assist ``` | Tool | Minimum version | Install | | --- | --- | --- | | dr-cli | 0.2.50 | datarobot-oss/cli installation | | Git | 2.30.0 | git-scm.com/downloads | | uv | 0.9.0 | Install uv | | Pulumi | 3.163.0 | Download and install | | Task | 3.43.3 | Task installation | | Node.js | 24 | Node.js download | | Python | 3.10 | python.org/downloads | Native Windows is not supported for Agent Assist; use [WSL](https://learn.microsoft.com/en-us/windows/wsl/install), a Linux VM, or a [DataRobot Codespace](https://docs.datarobot.com/en/docs/workbench/wb-notebook/codespaces/index.html). See [Prerequisites and installation](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/installation.html). ### Install packages for building blueprints `pip install datarobot datarobot-predict` (Optional) If you would like to build custom blueprints programmatically, install two additional packages: `graphviz` and `blueprint-workshop`. For Windows users: [Download the graphviz installer](https://www.graphviz.org/download/#windows) For Ubuntu users: `sudo apt-get install graphviz` For Mac users: `brew install graphviz` Once graphviz is installed, install the workshop: `pip install datarobot-bp-workshop` ## Verify your setup After keys and endpoint are configured, confirm that your session can reach the API. Use the subsections for the path you care about. ### Python SDK ``` import datarobot as dr dr.Client() # reads env vars or drconfig.yaml projects = dr.Project.list() print(f"Connected. You can see {len(projects)} project(s).") ``` ### cURL Requires `DATAROBOT_ENDPOINT` and `DATAROBOT_API_TOKEN` in the shell. ``` curl -s "${DATAROBOT_ENDPOINT}/projects/" \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" | head ``` ### CLI ``` dr --version dr plugin list # should include assist dr task --help ``` > [!TIP] CLI sanity check > With the CLI installed: `dr --version`. For Agent Assist as a plugin, see [Prerequisites and installation](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/installation.html) ( `dr plugin install assist`). ### Agent Assist Agent Assist clones a template into the current directory, so use a fresh folder: ``` mkdir my-first-agent && cd my-first-agent dr assist ``` On first run, Agent Assist checks dependencies, verifies auth, and starts an interactive session. Describe the agent you want in plain language. If you'd like to start working with Agent Assist now, go to [Workflows and prompting](https://docs.datarobot.com/en/docs/agentic-ai/agent-assist/workflows-and-prompting.html#start-agent-assist). ## Troubleshooting | Symptom | What to check | | --- | --- | | 401 Unauthorized | Personal API key value; endpoint region matches where you log in (US vs EU vs JP); key not expired or revoked. Regenerate a key from Developer tools (US SaaS) or API keys and tools. | | dr: command not found | dr-cli is not on your PATH. Install the CLI, reopen the terminal, or on macOS with Homebrew run brew link dr-cli. On Linux, add the install location from the release to PATH. | | dr plugin install assist fails | Check dr --version is at least 0.2.50; older CLIs may not expose the plugin index. See Prerequisites and installation. | | Agent Assist: directory not empty | Run dr assist only from an empty directory; the assistant clones a template and refuses to overwrite files. | | Windows and Agent Assist | Agent Assist targets macOS/Linux; on Windows use WSL2 or a codespace per installation. | ## Hands-on: Build and deploy a first model The section below is a compact modeling lab you can run locally: same steps many teams automate later in production. Examples use Python or cURL. You will predict miles per gallon from the classic Auto MPG dataset. > [!NOTE] Note > The Python snippets use client 3.x APIs ( `Dataset`, registered model deployment flow). Upgrade the package if anything fails to import. If you use the Self-Managed AI Platform, see the [Self-Managed AI Platform API resources](https://docs.datarobot.com/en/docs/api/reference/self-managed.html) page to confirm supported client versions for your installation. For more code-first samples, see [AI accelerators](https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/index.html). You will: 1. Upload a dataset and create a project. 2. Train with Autopilot and pick the recommended model. 3. Deploy that model to a serverless prediction environment. 4. Predict on a holdout CSV using the deployment. ### Upload a dataset Download `auto-mpg.csv` and `auto-mpg-test.csv` from [this zip archive](https://datarobot-doc-assets.s3.us-east-1.amazonaws.com/auto.zip). The Python tab below uses `config_path="./drconfig.yaml"` next to the script so the lab is self-contained. The same client also picks up the default `~/.config/datarobot/drconfig.yaml` (or your environment variables) as described in [Configure API authentication](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html#configure-api-authentication), if you prefer that layout. **Python:** ``` import datarobot as dr dr.Client(config_path="./drconfig.yaml") # Set to the location of your auto-mpg.csv and auto-mpg-test.csv data files # Example: dataset_file_path = '/Users/myuser/Downloads/auto-mpg.csv' training_dataset_file_path = './auto-mpg.csv' test_dataset_file_path = './auto-mpg-test.csv' print("--- Starting DataRobot Model Training Script ---") # Load dataset training_dataset = dr.Dataset.create_from_file(training_dataset_file_path) # Create a new project based on dataset project = dr.Project.create_from_dataset(training_dataset.id, project_name='Auto MPG DR-Client') ``` **cURL:** ``` DATAROBOT_API_TOKEN=${DATAROBOT_API_TOKEN} DATAROBOT_ENDPOINT=${DATAROBOT_ENDPOINT} DATASET_FILE_PATH="./auto-mpg.csv" location=$(curl -Lsi \ -X POST \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ -F 'projectName="Auto MPG"' \ -F "file=@${DATASET_FILE_PATH}" \ "${DATAROBOT_ENDPOINT}"/projects/ | grep -i 'Location: .*$' | \ cut -d " " -f2 | tr -d '\r') echo "Uploaded dataset. Checking status of project at: ${location}" while true; do project_id=$(curl -Ls \ -X GET \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" "${location}" \ | grep -Eo 'id":\s"\w+' | cut -d '"' -f3 | tr -d '\r') if [ "${project_id}" = "" ] then echo "Setting up project..." sleep 10 else echo "Project setup complete." echo "Project ID: ${project_id}" break fi done ``` ### Train models DataRobot Autopilot trains many candidate models and surfaces a recommended model for your target ( `mpg` here). See [model recommendation](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/model-rec-process.html) in the UI documentation for how that choice is made. > [!NOTE] Note > This code can open a browser window on the classic project experience. Use the NextGen UI menu and open Console if you prefer the newer navigation. > > [https://docs.datarobot.com/en/docs/images/access-nextgen-console.png](https://docs.datarobot.com/en/docs/images/access-nextgen-console.png) **Python:** ``` # Use training data to build models from datarobot import AUTOPILOT_MODE # Set the project's target and initiate Autopilot (runs in Quick mode unless a different mode is specified) project.analyze_and_model(target='mpg', worker_count=-1, mode=AUTOPILOT_MODE.QUICK) print("\nAutopilot is running. This may take some time...") project.wait_for_autopilot() print("Autopilot has completed!") # Open the project in a web browser to view progress print("Opening the project in your default web browser to view real-time events...") project.open_in_browser() # Get the recommended model (the best model for deployment) print("\nRetrieving the best model from the Leaderboard...") best_model = project.recommended_model() print(f"Best Model Found:") print(f" - Model Type: {best_model.model_type}") print(f" - Blueprint ID: {best_model.blueprint_id}") ``` **cURL:** ``` response=$(curl -Lsi \ -X PATCH \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ -H "Content-Type: application/json" \ --data '{"target": "mpg", "mode": "quick"}' \ "${DATAROBOT_ENDPOINT}/projects/${project_id}/aim" | grep 'location: .*$' \ | cut -d " " | tr -d '\r') echo "AI training initiated. Checking status of training at: ${response}" while true; do initial_project_status=$(curl -Ls \ -X GET \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" "${response}" \ | grep -Eo 'stage":\s"\w+' | cut -d '"' -f3 | tr -d '\r') if [ "${initial_project_status}" = "" ] then echo "Setting up AI training..." sleep 10 else echo "Training AI." echo "Grab a coffee or catch up on email." break fi done echo "Polling for Autopilot completion..." while true; do autopilot_done=$(curl -s \ -X GET \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ "${DATAROBOT_ENDPOINT}/projects/${project_id}/" \ | grep -Eo '"autopilotDone":\s*(true|false)' | cut -d ':' -f2 | tr -d ' ') if [ "${autopilot_done}" = "true" ]; then echo "Autopilot training complete. Model ready to deploy." break else echo "Autopilot training in progress... checking again in 60 seconds." sleep 60 fi done # Get the recommended model ID recommended_model_id=$(curl -s \ -X GET \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ "${DATAROBOT_ENDPOINT}/projects/${project_id}/recommendedModels/recommendedModel/" \ | grep -Eo 'modelId":\s"\w+' | cut -d '"' -f3 | tr -d '\r') echo "Recommended model ID: ${recommended_model_id}" ``` ### Deploy the model A deployment serves predictions from a trained model in a managed environment. See the [deployment overview](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-overview/nxt-overview.html) for concepts and options. **Python:** ``` # Deploy the model to a serverless prediction environment print("\nDeploying the model to a serverless prediction environment...") # Find or create a serverless prediction environment serverless_env = None for env in dr.PredictionEnvironment.list(): if env.platform == 'datarobotServerless': serverless_env = env break if serverless_env is None: print("Creating a new serverless prediction environment...") serverless_env = dr.PredictionEnvironment.create( name="Auto MPG Serverless Environment", platform='datarobotServerless' ) # First, register the model to create a registered model version print("Registering the model...") # Check if the registered model already exists registered_model_name = "Auto MPG Registered Model" existing_models = [m for m in dr.RegisteredModel.list() if m.name == registered_model_name] if existing_models: print(f"Using existing registered model: {registered_model_name}") registered_model = existing_models[0] # Create a new version of the existing model registered_model_version = dr.RegisteredModelVersion.create_for_leaderboard_item( best_model.id, name="Auto MPG Model", registered_model_id=registered_model.id ) else: print(f"Creating new registered model: {registered_model_name}") # Create a new registered model registered_model_version = dr.RegisteredModelVersion.create_for_leaderboard_item( best_model.id, name="Auto MPG Model", registered_model_name=registered_model_name ) # Retrieve the newly created registered model object by ID registered_model = dr.RegisteredModel.get(registered_model_version.registered_model_id) # Wait for the model build to complete print("Waiting for model build to complete...") while True: current_version = registered_model.get_version(registered_model_version.id) if current_version.build_status in ('READY', 'complete'): print("Model build completed successfully!") registered_model_version = current_version # Update our reference break elif current_version.build_status == 'FAILED': raise Exception("Model build failed. Please check the model registration.") else: print(f"Build status: {current_version.build_status}. Waiting...") import time time.sleep(30) # Wait 30 seconds before checking again # Deploy the model to the serverless environment using the registered model version deployment = dr.Deployment.create_from_registered_model_version( registered_model_version.id, label="Auto MPG Predictions", description="Deployed with DataRobot client for Auto MPG predictions", prediction_environment_id=serverless_env.id ) print(f"Model deployed successfully! Deployment ID: {deployment.id}") ``` **cURL:** ``` # Use the recommended model ID from training section echo "Using recommended model ID: ${recommended_model_id}" # Find or create a serverless prediction environment echo "Looking for serverless prediction environment..." serverless_env_id=$(curl -s -X GET \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ "${DATAROBOT_ENDPOINT}/predictionEnvironments/" \ | grep -Eo '"id":"[^"]*".*"platform":"datarobotServerless"' \ | grep -Eo '"id":"[^"]*"' | cut -d '"' -f4 | head -1) if [ -z "${serverless_env_id}" ]; then echo "Creating new serverless prediction environment..." serverless_env_response=$(curl -s -X POST \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ -H "Content-Type: application/json" \ --data '{"name":"Auto MPG Serverless Environment","platform":"datarobotServerless"}' \ "${DATAROBOT_ENDPOINT}/predictionEnvironments/") serverless_env_id=$(echo "$serverless_env_response" | grep -Eo '"id":"[^"]*"' | cut -d '"' -f4) echo "Created serverless environment ID: ${serverless_env_id}" else echo "Using existing serverless environment ID: ${serverless_env_id}" fi # Check if registered model already exists registered_model_name="Auto MPG Registered Model" existing_model_id=$(curl -s -X GET \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ "${DATAROBOT_ENDPOINT}/registeredModels/" \ | grep -Eo '"id":"[^"]*".*"'${registered_model_name}'"' \ | grep -Eo '"id":"[^"]*"' | cut -d '"' -f4 | head -1) if [ -n "${existing_model_id}" ]; then echo "Using existing registered model: ${registered_model_name}" # Create new version of existing model model_version_response=$(curl -s -X POST \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ -H "Content-Type: application/json" \ --data "{\"name\":\"Auto MPG Model\",\"registeredModelId\":\"${existing_model_id}\",\"leaderboardItemId\":\"${recommended_model_id}\"}" \ "${DATAROBOT_ENDPOINT}/registeredModels/${existing_model_id}/versions/") else echo "Creating new registered model: ${registered_model_name}" # Create new registered model model_response=$(curl -s -X POST \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ -H "Content-Type: application/json" \ --data "{\"name\":\"${registered_model_name}\"}" \ "${DATAROBOT_ENDPOINT}/registeredModels/") existing_model_id=$(echo "$model_response" | grep -Eo '"id":"[^"]*"' | cut -d '"' -f4) # Create first version model_version_response=$(curl -s -X POST \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ -H "Content-Type: application/json" \ --data "{\"name\":\"Auto MPG Model\",\"registeredModelId\":\"${existing_model_id}\",\"leaderboardItemId\":\"${recommended_model_id}\"}" \ "${DATAROBOT_ENDPOINT}/registeredModels/${existing_model_id}/versions/") fi model_version_id=$(echo "$model_version_response" | grep -Eo '"id":"[^"]*"' | cut -d '"' -f4) echo "Model version ID: ${model_version_id}" # Wait for model build to complete echo "Waiting for model build to complete..." while true; do build_status=$(curl -s -X GET \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ "${DATAROBOT_ENDPOINT}/registeredModels/${existing_model_id}/versions/${model_version_id}/" \ | grep -Eo '"buildStatus":"[^"]*"' | cut -d '"' -f4) if [ "${build_status}" = "READY" ] || [ "${build_status}" = "complete" ]; then echo "Model build completed successfully!" break elif [ "${build_status}" = "FAILED" ]; then echo "Model build failed. Please check the model registration." exit 1 else echo "Build status: ${build_status}. Waiting..." sleep 30 fi done # Deploy the model using the registered model version echo "Deploying the model to the serverless environment..." deployment_response=$(curl -s -X POST \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ -H "Content-Type: application/json" \ --data "{\"label\":\"Auto MPG Predictions\",\"description\":\"Deployed with cURL for Auto MPG predictions\",\"predictionEnvironmentId\":\"${serverless_env_id}\",\"registeredModelVersionId\":\"${model_version_id}\"}" \ "${DATAROBOT_ENDPOINT}/deployments/fromRegisteredModelVersion/") deployment_id=$(echo "$deployment_response" | grep -Eo '"id":"[^"]*"' | cut -d '"' -f4) echo "Model deployed successfully! Deployment ID: ${deployment_id}" # Get the prediction URL for the deployment echo "Retrieving prediction URL for deployment..." prediction_url=$(curl -s -X GET \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ "${DATAROBOT_ENDPOINT}/deployments/${deployment_id}/" \ | grep -Eo '"predictionUrl":"[^"]*"' | cut -d '"' -f4) echo "Prediction URL: ${prediction_url}" ``` ### Make predictions against the deployed model Use the Prediction API to score new rows. That path unlocks [model management](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/index.html) features such as drift and accuracy tracking. See [prediction methods](https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/index.html) for an overview. In the UI, open Deployments → your deployment → Predictions → Prediction API for a copy-paste snippet. **Python:** This example scores `auto-mpg-test.csv` using `datarobot-predict`. ``` # Make predictions on test data print("\nMaking predictions on test data...") # Read the test data directly import pandas as pd from datarobot_predict.deployment import predict test_data = pd.read_csv(test_dataset_file_path) # Use datarobot-predict for deployment predictions predictions, response_headers = predict(deployment, test_data) # Display the results print("\nPrediction Results:") print(predictions.head()) print(f"\nTotal predictions made: {len(predictions)}") ``` **cURL:** ``` # Use the prediction URL from deployment section TEST_DATASET_FILE_PATH="./auto-mpg-test.csv" # Make predictions by sending the CSV data directly predictions=$(curl -s -X POST \ -H "Authorization: Bearer ${DATAROBOT_API_TOKEN}" \ -H "Content-Type: text/csv; charset=UTF-8" \ --data-binary "@${TEST_DATASET_FILE_PATH}" \ "${prediction_url}") echo "Prediction Results:" echo "$predictions" | jq '.' prediction_count=$(echo "$predictions" | jq '.data | length') echo "Total predictions made: ${prediction_count}" ``` ## Next steps Keep exploring the [developer learning section](https://docs.datarobot.com/en/docs/index.html) for notebooks and task-based tutorials. Try [AI accelerators](https://docs.datarobot.com/en/docs/api/dev-learning/accelerators/index.html) for modular workflows, and use the [reference documentation](https://docs.datarobot.com/en/docs/api/reference/index.html) for the REST API and Python client. | If you want to... | Go to | | --- | --- | | Run the hands-on modeling lab | Hands-on: Build and deploy a first model | | Authenticate with your own IdP token (Inbound OAuth) | Authenticate with an external IdP token | | Use Agent Assist to build an agent | Agent workflows and prompting | | Configure LLM providers for agentic work | Agentic LLM providers | | Connect Cursor, Claude Code, or other MCP clients | Agentic MCP clients | | Learn dr commands | CLI quick reference | | Browse Agent Assist docs | Agent Assist | --- # Inbound OAuth URL: https://docs.datarobot.com/en/docs/api/dev-learning/inbound-oauth-tokens.html title: "Authenticate with an external IdP token (OAuth 2.0 token exchange)" description: Exchange an external identity provider (IdP) access token for a DataRobot access token using OAuth 2.0 token exchange (RFC 8693). The external OAuth developer workflow for inbound OAuth: bring your own IdP (Okta, Microsoft Entra ID, Ping, Auth0) for user authentication to the DataRobot API. With inbound OAuth, you authenticate to the DataRobot API using an access token from your own identity provider (IdP) instead of a DataRobot API key. You exchange your IdP access token for a short-lived DataRobot access token using OAuth 2.0 token exchange ( [RFC 8693](https://datatracker.ietf.org/doc/html/rfc8693)), then send that DataRobot token as a bearer token on API requests. This external OAuth workflow lets your users authenticate with credentials they already have. This page covers the developer workflow. For the administrator setup (registering a trusted IdP, provisioning, and mapping), see [Inbound OAuth](https://docs.datarobot.com/en/docs/platform/admin/inbound-oauth.html). ## Prerequisites - An administrator has configured inbound OAuth for your organization or application and given you the token-exchange endpoint. - You can obtain an OAuth 2.0 access token from your IdP (for example, Okta, Microsoft Entra ID, Ping, or Auth0). - Your IdP issues access tokens that include the dr.impersonation scope and carry the DataRobot audience ( aud ) your administrator provides. DataRobot validates both during the exchange; a missing scope or mismatched audience is rejected. See Step 1 . - You have the DataRobot API endpoint for your region (see Retrieve the API endpoint ). ## Workflow overview 1. Get an access token from your IdP. 2. Exchange it for a DataRobot access token. 3. Call the DataRobot API with the DataRobot token. ## Step 1: Get an IdP access token Obtain an OAuth 2.0 access token from your IdP using your organization's standard flow (for example, the authorization code flow). The token must be a JWT access token issued by the IdP your administrator registered. > [!NOTE] Important > The token must include the `dr.impersonation` scope. DataRobot validates this scope during the exchange and rejects tokens without it. If your IdP doesn't add the scope by default, request it when you obtain the token, or ask your administrator to configure the IdP to include it. Refer to your IdP's documentation for how to obtain a token. The following sections refer to that token as `IDP_ACCESS_TOKEN`. > [!NOTE] Important > The IdP access token must include the `dr.impersonation` scope, in the claim your administrator mapped (typically `scp` or `scope`). DataRobot validates this scope during the exchange and rejects the request with `invalid_scope` if the token carries scopes but not `dr.impersonation`. Configure your IdP's authorization server (or app and scope registration) to add `dr.impersonation` to the tokens it issues. ## Step 2: Exchange for a DataRobot access token Exchange the IdP token at the token-exchange endpoint. The endpoint is served at your DataRobot application root, not under `/api/v2`: | Configuration scope | Endpoint | | --- | --- | | Global | https://YOUR_DATAROBOT_HOST/oauth2/token | | Organization | https://YOUR_DATAROBOT_HOST/oauth2/token/ORGANIZATION_ID | > [!NOTE] Note > Use the DataRobot host you sign in with (for example, `https://app.datarobot.com`), not the `/api/v2` endpoint. Confirm the exact token-exchange URL for your environment with your administrator. Send a `POST` request with the [RFC 8693](https://datatracker.ietf.org/doc/html/rfc8693) token-exchange parameters, form-encoded: | Parameter | Value | | --- | --- | | grant_type | urn:ietf:params:oauth:grant-type:token-exchange | | subject_token | Your IDP_ACCESS_TOKEN. | | subject_token_type | urn:ietf:params:oauth:token-type:access_token | **cURL:** ``` export IDP_ACCESS_TOKEN="YOUR_IDP_ACCESS_TOKEN" curl --location -X POST \ "https://YOUR_DATAROBOT_HOST/oauth2/token/ORGANIZATION_ID" \ --header "Accept: application/json" \ --header "Content-Type: application/x-www-form-urlencoded" \ --data-urlencode "grant_type=urn:ietf:params:oauth:grant-type:token-exchange" \ --data-urlencode "subject_token=${IDP_ACCESS_TOKEN}" \ --data-urlencode "subject_token_type=urn:ietf:params:oauth:token-type:access_token" ``` **Python:** ``` import requests DATAROBOT_HOST = "https://YOUR_DATAROBOT_HOST" ORGANIZATION_ID = "ORGANIZATION_ID" # your organization's ID idp_access_token = "YOUR_IDP_ACCESS_TOKEN" response = requests.post( f"{DATAROBOT_HOST}/oauth2/token/{ORGANIZATION_ID}", headers={"Accept": "application/json"}, data={ "grant_type": "urn:ietf:params:oauth:grant-type:token-exchange", "subject_token": idp_access_token, "subject_token_type": "urn:ietf:params:oauth:token-type:access_token", }, ) response.raise_for_status() datarobot_access_token = response.json()["access_token"] ``` A successful exchange returns a DataRobot access token: ``` { "access_token": "", "issued_token_type": "urn:ietf:params:oauth:token-type:access_token", "token_type": "Bearer", "expires_in": 3600, "scope": "dr.impersonation" } ``` | Field | Description | | --- | --- | | access_token | The DataRobot access token to use on API requests. | | token_type | Always Bearer. | | expires_in | Lifetime of the token, in seconds. | | scope | The granted scope. Inbound OAuth tokens carry the dr.impersonation scope. | ## Step 3: Call the DataRobot API Send the DataRobot access token as a bearer token, the same as you would a DataRobot API key. Use the `/api/v2` endpoint for your region. **cURL:** ``` export DATAROBOT_ENDPOINT="https://YOUR_DATAROBOT_HOST/api/v2" export DATAROBOT_ACCESS_TOKEN="DATAROBOT_ACCESS_TOKEN_FROM_STEP_2" curl --location -X GET "${DATAROBOT_ENDPOINT}/projects/" \ --header "Authorization: Bearer ${DATAROBOT_ACCESS_TOKEN}" ``` **Python:** ``` import datarobot as dr client = dr.Client( endpoint="https://YOUR_DATAROBOT_HOST/api/v2", token=datarobot_access_token, # from Step 2 ) print(dr.Project.list()) ``` ## Token lifetime DataRobot access tokens from inbound OAuth are short-lived (typically one hour). Read `expires_in` from the exchange response rather than hard-coding a value. When a token expires, repeat [Step 2](https://docs.datarobot.com/en/docs/api/dev-learning/inbound-oauth-tokens.html#step-2-exchange-for-a-datarobot-access-token) to get a new one. Tokens can't be revoked individually. ## Error reference The token-exchange endpoint returns errors defined by [RFC 6749](https://datatracker.ietf.org/doc/html/rfc6749) and [RFC 8693](https://datatracker.ietf.org/doc/html/rfc8693): ``` { "error": "", "error_description": "" } ``` | Error code | HTTP status | Description | | --- | --- | --- | | invalid_request | 400 | The request is missing a parameter, has an unsupported parameter, or is otherwise malformed. | | invalid_client | 401 | Unknown client, missing client details, or invalid bearer token. | | invalid_grant | 400 | The provided subject token is invalid or expired. | | unauthorized_client | 400 | The client isn't authorized to use the token-exchange grant type. | | unsupported_grant_type | 400 | The server doesn't support the requested grant type. | | invalid_scope | 400 | The token's scopes are invalid or don't include the required dr.impersonation scope. | | invalid_target | 400 | The audience is invalid. | | access_denied | 403 | Token exchange isn't available or enabled for this configuration. | > [!TIP] Tip > An `invalid_scope` error (HTTP 400) with the message "The token does not contain the required scope: dr.impersonation" means your IdP token doesn't carry the `dr.impersonation` scope. Configure your IdP to include it, then confirm the [scope mapping](https://docs.datarobot.com/en/docs/platform/admin/inbound-oauth.html#scope-mapping) with your administrator. ## FAQ ### How do I authenticate to the DataRobot API using my Okta or Microsoft Entra ID token? Get an access token from Okta or Microsoft Entra ID, exchange it for a DataRobot access token ( [Step 2](https://docs.datarobot.com/en/docs/api/dev-learning/inbound-oauth-tokens.html#step-2-exchange-for-a-datarobot-access-token)), and call the API with the DataRobot token ( [Step 3](https://docs.datarobot.com/en/docs/api/dev-learning/inbound-oauth-tokens.html#step-3-call-the-datarobot-api)). Your administrator must first register the IdP through [Inbound OAuth](https://docs.datarobot.com/en/docs/platform/admin/inbound-oauth.html). ## Related - Inbound OAuth —administrator configuration for the trusted IdP, provisioning, and mapping. - Developer quickstart —API keys, endpoints, and credential setup. --- # Developer learning URL: https://docs.datarobot.com/en/docs/api/dev-learning/index.html > Review educational resources for getting started with DataRobot, including the developer quickstart, notebooks, and code-first tools. Review the learning resources outlined in the table below to get started with DataRobot's API and code-first tools. | Resource | Description | | --- | --- | | Developer quickstart | Install the Python client, configure API keys and endpoints, verify connectivity, then run an end-to-end modeling and deployment example. The same credentials power the CLI and Agent Assist when you expand your toolchain. | | AI consumable assets | Structured exports and indexes so AI assistants can reliably use DataRobot documentation. | | Python API client user guide | Review outlines and explanations of the methods that comprise the API client. To access previous version of the Python API client documentation, access ReadTheDocs. | | REST API code examples | Review code examples that outline usage of the DataRobot REST API. | | AI accelerators | Jump-start modeling with code-first workflows. | --- # Credentials URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html You can store credentials for use with databases and data connections. To interact with credentials API, use the [Credential](https://docs.datarobot.com/en/docs/api/reference/sdk/credentials.html#credential-api) class. ## List credentials To retrieve the list of all credentials accessible to you, use [Credential.list](https://docs.datarobot.com/en/docs/api/reference/sdk/credentials.html#datarobot.models.Credential.list). ``` import datarobot as dr credentials = dr.Credential.list() ``` Each credential object contains the `credential_id` string field which can be used in, for example, [Batch predictions](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#batch-predictions-s3-creds-usage). ## Basic credentials Use the code below to store generic username and password credentials: ``` >>> import datarobot as dr >>> cred = dr.Credential.create_basic( ... name='my_db_cred', ... user='', ... password='', ... ) >>> cred Credential('5e429d6ecf8a5f36c5693e0f', 'my_db_cred', 'basic'), # Store cred.credential_id >>> cred = dr.Credential.get(credential_id) >>> cred.credential_id '5e429d6ecf8a5f36c5693e0f' ``` Stored credentials can be used in [Batch predictions for JDBC intake or output](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob). ## S3 credentials You can store AWS credentials either using the following three parameters: - aws_access_key_id - aws_secret_access_key - aws_session_token or by using the ID of the saved shared secure configuration: - config_id ``` >>> import datarobot as dr >>> cred = dr.Credential.create_s3( ... name='my_s3_cred', ... aws_access_key_id='', ... aws_secret_access_key='', ... aws_session_token='', ... ) >>> cred Credential('5e429d6ecf8a5f36c5693e03', 'my_s3_cred', 's3'), # Using config_id >>> cred = dr.Credential.dr.Credential.create_s3( ... name='my_s3_cred_with_config_id', ... config_id='', ... ) >>> cred Credential('65ef55ef4cec97f0f733835c', 'my_s3_cred_with_config_id', 's3') # Store cred.credential_id >>> cred = dr.Credential.get(credential_id) >>> cred.credential_id '5e429d6ecf8a5f36c5693e03' ``` Stored credential can be used, for example, in [Batch predictions for S3 intake or output](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#batch-predictions-s3-creds-usage). ## OAuth credentials You can store OAuth credentials in the data store. ``` >>> import datarobot as dr >>> cred = dr.Credential.create_oauth( ... name='my_oauth_cred', ... token='', ... refresh_token='', ... ) >>> cred Credential('5e429d6ecf8a5f36c5693e0f', 'my_oauth_cred', 'oauth'), # Store cred.credential_id >>> cred = dr.Credential.get(credential_id) >>> cred.credential_id '5e429d6ecf8a5f36c5693e0f' ``` ## Snowflake key pair credentials You can store Snowflake key pair credentials in the store. It accepts either of the following parameters: - private_key - passphrase Or you can use the ID of the saved shared secure configuration. - config_id ``` >>> import datarobot as dr >>> cred = dr.Credential.create_snowflake_key_pair( ... name='my_snowflake_key_pair_cred', ... user='', ... private_key="""""", ... passphrase='', ... ) >>> cred Credential('65e9b55e4b0d925c678bb847', 'my_snowflake_key_pair_cred', 'snowflake_key_pair_user_account') >>> cred = dr.Credential.create_snowflake_key_pair( ... name='my_snowflake_key_pair_cred_with_config_id', ... config_id='', ... ) >>> cred Credential('65e9b9494b0d925c678bb84d', 'my_snowflake_key_pair_cred_with_config_id', 'snowflake_key_pair_user_account') ``` ## Databricks access token credentials You can store Databricks access token credentials in the data store. ``` >>> import datarobot as dr >>> cred = dr.Credential.create_databricks_access_token( ... name='my_databricks_access_token_cred', ... databricks_access_token='', ... ) >>> cred Credential('65e9bace4b0d925c678bb850', 'my_databricks_access_token_cred', 'databricks_access_token_account') ``` ## Databricks service principal credentials You can store Databricks service principal credentials in the store. It accepts either of the following parameters: - client_id - client_secret You can also use the ID of the saved shared secure configuration. - config_id ``` >>> import datarobot as dr >>> cred = dr.Credential.create_databricks_service_principal( ... name='my_databricks_service_principal_cred', ... client_id='', ... client_secret='', ... ) >>> cred Credential('65e9bb864b0d925c678bb853', 'my_databricks_service_principal_cred', 'databricks_service_principal_account') >>> cred = dr.Credential.create_databricks_service_principal( ... name='my_databricks_service_principal_cred_with_config_id', ... config_id='', ... ) >>> cred Credential('65e9bcc14b0d925c678bb85e', 'my_databricks_service_principal_cred_with_config_id', 'databricks_service_principal_account') ``` ## Azure Service Principal credentials You can store Azure Service Principal credentials using any of the three following parameters: - client_id - client_secret - azure_tenant_id You can also use the ID of the saved shared secure configuration. - config_id ``` >>> import datarobot as dr >>> cred = dr.Credential.create_azure_service_principal( ... name='my_azure_service_principal_cred', ... client_id='', ... client_secret='', ... azure_tenant_id='', ... ) >>> cred Credential('66c920fc4ef80072a8225e56', 'my_azure_service_principal_cred2', 'azure_service_principal') # Using config_id >>> cred = dr.Credential.dr.Credential.create_azure_service_principal( ... name='my_azure_service_principal_cred_with_config_id', ... config_id='', ... ) >>> cred Credential('66c921aa0ff7aea1ce225e2d', 'my_azure_service_principal_cred_with_config_id', 'azure_service_principal') # Store cred.credential_id >>> cred = dr.Credential.get(credential_id) >>> cred.credential_id '66c921aa0ff7aea1ce225e2d' ``` ## ADLS OAuth credentials You can store ADLS OAuth credentials using any of the three following parameters: - client_id - client_secret - oauth_scopes You can also use the ID of the saved shared secure configuration. - config_id ``` >>> import datarobot as dr >>> cred = dr.Credential.create_adls_oauth( ... name='my_adls_oauth_cred', ... client_id='', ... client_secret='', ... oauth_scopes=[''], ... ) >>> cred Credential('66c9227e3b268d3278225e41', 'my_adls_oauth_cred', 'adls_gen2_oauth') # Using config_id >>> cred = dr.Credential.dr.Credential.create_adls_oauth( ... name='my_adls_oauth_cred_with_config_id', ... config_id='', ... ) >>> cred Credential('66c922b3ae75806f1d126f06', 'my_adls_oauth_cred_with_config_id', 'adls_gen2_oauth') # Store cred.credential_id >>> cred = dr.Credential.get(credential_id) >>> cred.credential_id '66c922b3ae75806f1d126f06' ``` ## Credential data For methods that accept credential data instead of a username/password or credential ID: ``` { "credentialType": "basic", "user": "user123", "password": "pass123", } ``` ``` { "credentialType": "s3", "awsAccessKeyId": "key123", "awsSecretAccessKey": "secret123", } ``` ``` { "credentialType": "s3", "configId": "id123", } ``` ``` { "credentialType": "oauth", "oauthRefreshToken": "token123", "oauthClientId": "client123", "oauthClientSecret": "secret123", } ``` ``` { "credentialType": "snowflake_key_pair_user_account", "user": "user123", "privateKey": "privatekey123", "passphrase": "passphrase123", } ``` ``` { "credentialType": "snowflake_key_pair_user_account", "configId": "id123", } ``` ``` { "credentialType": "databricks_access_token_account", "databricksAccessToken": "token123", } ``` ``` { "credentialType": "databricks_service_principal_account", "clientId": "client123", "clientSecret": "secret123", } ``` ``` { "credentialType": "databricks_service_principal_account", "configId": "id123", } ``` ``` { "credentialType": "azure_service_principal", "clientId": "client123", "clientSecret": "secret123", "azureTenantId": "tenant123" } ``` ``` { "credentialType": "azure_service_principal", "configId": "id123", } ``` ``` { "credentialType": "adls_gen2_oauth", "clientId": "client123", "clientSecret": "secret123", "oauthScopes": ["scope123"] } ``` ``` { "credentialType": "adls_gen2_oauth", "configId": "id123", } ``` --- # Administration URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/index.html The administration section provides details for users and administrators about managing credentials and sharing permissions. --- # Sharing URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/sharing.html Once you have created entities in DataRobot, you may want to share them with collaborators. DataRobot provides an API for sharing the following entities: > Data sources and data stores ( seeDatabase connectivityfor more info on connecting to JDBC databases)DatasetsProjectsCalendar filesModel deployments (seeDeployment sharingfor more information on sharing deployments)Use Cases (Sharing for Use Cases is slightly different than what’s documented on this page. SeeUse Case sharingfor more information and examples.) ## Access levels Entities can be shared at varying access levels. For example, you can allow someone to create projects from a data source you have built without allowing them to delete it. Each entity type uses slightly different permission names intended to specifically convey what kind of actions are available. These roles fall into three categories. These generic role names can be used in the sharing API for any entity. For the complete set of actions granted by each role on a given entity, see the [UI documentation for roles and permissions](https://docs.datarobot.com/en/docs/get-started/acct-mgmt/data-sharing/roles-permissions.html). > OWNERUsed for all entities.Allows any action including deletion.READ_WRITEKnown asEDITORon data sources and data stores.Allows modifications to the state, such as renaming and creating data sources from a data store, butnotdeleting the entity.READ_ONLYKnown asCONSUMERon data sources and data stores.For data sources, enables creating projects and predictions; for data stores, only allows you to view them. When a user’s new role is specified as `None`, their access will be revoked. In addition to the role, some entities (data sources and data stores) allow separate control over whether a new user should be able to share that entity further. When granting access to a user, the `can_share` parameter determines whether that user can, in turn, share this entity with another user. When this parameter is set as false, the user in question has all the access to the entity granted by their role and can remove themselves if desired, but are unable to change the role of any other user. ## Examples Transfer access to the data source from [mailto:old_user@datarobot.com](mailto:old_user@datarobot.com) to [mailto:new_user@datarobot.com](mailto:new_user@datarobot.com). ``` import datarobot as dr new_access = dr.SharingAccess( "new_user@datarobot.com", dr.enums.SHARING_ROLE.OWNER, can_share=True, ) access_list = [dr.SharingAccess("old_user@datarobot.com", None), new_access] dr.DataSource.get('my-data-source-id').share(access_list) ``` To check access to a project: ``` import datarobot as dr project = dr.Project.create('mydata.csv', project_name='My Data') access_list = project.get_access_list() access_list[0].username ``` To transfer ownership of all projects owned by your account to [mailto:new_user@datarobot.com](mailto:new_user@datarobot.com) without sending notifications: ``` import datarobot as dr # Put path to YAML credentials below dr.Client(config_path= '.yaml') # Get all projects for your account and store the ids in a list projects = dr.Project.list() project_ids = [project.id for project in projects] # List of emails to share with share_targets = ['new_user@datarobot.com'] # Target role target_role = dr.enums.SHARING_ROLE.OWNER for pid in project_ids: project = dr.Project.get(project_id=pid) shares = [] for user in share_targets: shares.append(dr.SharingAccess(username=user, role=target_role)) project.share(shares, send_notification=False) ``` --- # Data wrangling URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/data/data_wrangling.html To clean, prepare, and wrangle your data into your desired shape, DataRobot provides reusable recipes for data preparation. Each recipe acts like a blueprint, taking one or more datasets or data sources as input and applying a series of operations to filter, modify, join or transform your data. You can then use the recipe to create a dataset ready for consumption. Recipes allow for quick iteration on data prep workflows, and enable re-use via its simple operations API. ## Recipe terminology Recipes use the following terminology: - Recipe : A reusable blueprint for how to create a new dataset by applying operations to transform one or more data inputs. - Recipe dialect : The dialect data wrangling operations should use when working with recipe inputs. For example, use Snowflake dialect when working with data assets from Snowflake. - Input : A dataset or data source providing data to a recipe. A recipe can have multiple inputs. A recipe's inputs must either be all datasets, or all tables from data sources pointing to the same data store. - Primary input : The input used as the base for the recipe. If no operations are applied by the recipe, a dataset identical to the primary input will be output by the recipe. A recipe will only have a single primary input. - Secondary input : An additional input to a recipe. A recipe can have multiple secondary inputs. Data from secondary inputs must be introduced into a recipe via join or other similar operation. - Recipe preview : A sample view of the recipe's data computed by applying the operations in a recipe on its inputs. The data featured in a recipe's preview is generally a sample of the recipe's fully transformed data. - Sampling : A setting through which the number of rows read from a recipe's primary input is modified when computing the recipe's preview. - Downsampling : A setting through which the number of rows written to the dataset published by a recipe is modified. - Operation : A way to modify how a recipe works with data from its inputs. - Wrangling operation : Transformation to apply to a recipe's data. Recipes can stack multiple wrangling operations on top of each other to transform data from its inputs. - Downsampling operation : Modification to the recipe's number of rows to write to a dataset when publishing. Recipes can optionally use a single downsampling operation. - Sampling operation : Modification to the number of rows to read from a recipe's primary data input. Recipes can optionally set a single sampling operation on their primary input. - Publishing : Action to create a new dataset containing the result of applying the recipe's operations on its inputs. Review the recommended workflow to create, iterate, and publish with recipes below. 1. Create a datarobot.Recipe to work on a datarobot.Dataset or data from a datarobot.DataSource . The recipe will belong to a datarobot.Usecase . 2. Modify the recipe by updating its metadata, settings, inputs, operations, or downsampling. 3. Verify the recipe's data by requesting a recipe preview. If you are unhappy with the result, go back to step 2. 4. Publish the recipe to create a new datarobot.Dataset constructed according to the transformations in the recipe. ## Create a recipe There are two ways to create a recipe. You can use either a dataset or a table from a JDBC data source. They become the primary input for the recipe. You will also need a [datarobot.Usecase](https://docs.datarobot.com/en/docs/api/reference/sdk/use-cases.html#datarobot.UseCase), as each recipe will belong to a use case. Choose the [DataWranglingDialect](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.enums.DataWranglingDialect) that best matches the source of the dataset or data source. ### Create a recipe from a dataset Use the [Recipe.from_dataset](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe.Recipe.from_dataset) method to create a recipe from an existing [datarobot.Dataset](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset): ``` >>> import datarobot as dr >>> from datarobot.enums import DataWranglingDialect, RecipeType >>> from datarobot.models.recipe_operation import RandomSamplingOperation >>> >>> # Get your use case and dataset >>> my_use_case = dr.UseCase.list(search_params={"search": "My Use Case"})[0] >>> dataset = dr.Dataset.get('5f43a1b2c9e77f0001e6f123') >>> >>> # Create a recipe from the dataset >>> recipe = dr.Recipe.from_dataset( ... use_case=my_use_case, ... dataset=dataset, ... dialect=DataWranglingDialect.SPARK, ... recipe_type=RecipeType.WRANGLING, ... sampling=RandomSamplingOperation(rows=500) ... ) ``` ### Create a recipe from a JDBC table Use the [Recipe.from_data_store](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe.Recipe.from_data_store) method to create a recipe directly from tables in a connected data source: ``` >>> import datarobot as dr >>> from datarobot.enums import DataWranglingDataSourceTypes, DataWranglingDialect, RecipeType >>> from datarobot.models.recipe import DataSourceInput >>> from datarobot.models.recipe_operation import LimitSamplingOperation >>> >>> # Configure your data source input >>> data_source_input = DataSourceInput( ... canonical_name='Sales_Data_Connection', # data connection name ... table='sales_transactions', ... schema='PUBLIC', ... sampling=LimitSamplingOperation(rows=1000) ... ) >>> >>> # Get your use case and data store >>> my_use_case = dr.UseCase.list(search_params={"search": "Sales Analysis"})[0] >>> data_store = dr.DataStore.get('2g33a1b2c9e88f0001e6f657') >>> >>> # Create recipe from data source >>> recipe = dr.Recipe.from_data_store( ... use_case=my_use_case, ... data_store=data_store, ... data_source_type=DataWranglingDataSourceTypes.JDBC, ... dialect=DataWranglingDialect.POSTGRES, ... data_source_inputs=[data_source_input], ... recipe_type=RecipeType.WRANGLING ... ) ``` ## Retrieve recipes You can retrieve a specific recipe by ID, or a list of all recipes, filtering the list as required. ``` >>> import datarobot as dr >>> >>> # Get a specific recipe by ID >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # List all recipes >>> all_recipes = dr.Recipe.list() >>> >>> # Filter recipes. Use any number of params to filter. >>> filtered_recipes = dr.Recipe.list( ... search="My Recipe Name", ... dialect=dr.enums.DataWranglingDialect.SPARK, ... status="draft", ... recipe_type=dr.enums.RecipeType.WRANGLING, ... order_by="-updatedAt", # Most recently updated first ... created_by_username="data_scientist_user" ... ) ``` ### Retrieve information about a recipe The recipe object contains basic information about the recipe that you can query, as shown below. ``` >>> import datarobot as dr >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> recipe.id u'690bbf77aa31530d8287ae5f' >>> recipe.name u"Customer Segmentation Dataset Recipe" ``` You can also retrieve the list of inputs and operations, as well as the settings for downsampling and general recipe settings. ``` >>> import datarobot as dr >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> # Access inputs, operations, downsampling and settings >>> inputs = recipe.inputs >>> primary_input = inputs[0] # First input is the primary input >>> secondary_inputs = inputs[1:] # All others in the list are secondary inputs >>> >>> operations = recipe.operations >>> downsampling_operation = recipe.downsampling >>> settings = recipe.settings ``` ## Update recipe metadata fields You can update the recipe's metadata fields (name, description, etc.) with [Recipe.update](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe.Recipe.update) as follows: ``` >>> import datarobot as dr >>> >>> # Retrieve an existing recipe >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Update metadata fields >>> recipe.update( ... name="Customer Segmentation Dataset Recipe", ... description="Recipe to create customer segmentation dataset." ... ) ``` ## Update recipe inputs You can update the list of inputs for a recipe with the [Recipe.update](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe.Recipe.update) method as shown below. By updating the list of inputs, you change the data fed into the recipe to transform. The first input in the list will be the recipe's primary input, with the rest being secondary inputs. > [!NOTE] Recipe input considerations > The [Recipe.update](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe.Recipe.update) method will replace all existing inputs. If adding inputs, always include the existing primary input to avoid breaking the recipe. > > Data from secondary inputs will not appear in the recipe preview unless somehow joined or combined with data from the primary input. > > Recipe inputs must either be all datasets, or all tables from data sources pointing to the same data store. ``` >>> import datarobot as dr >>> from datarobot.models.recipe import RecipeDatasetInput, JDBCTableDataSourceInput >>> >>> # Get the recipe and additional datasets >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> secondary_dataset = dr.Dataset.get('5f43a1b2c9e77f0001e6f456') >>> >>> # Add a secondary dataset input if the primary input is also a dataset >>> recipe.update( ... inputs=[ ... recipe.inputs[0], # Keep the original primary input ... RecipeDatasetInput.from_dataset( ... dataset=secondary_dataset, ... alias='customers_data' ... ) ... ] ... ) >>> >>> # You can also add data from a table in a data store >>> data_store = dr.DataStore.get('5e1b4f8f2a3c4d5e6f7g8h9i') >>> data_source = DataSource.create( ... data_source_type="jdbc", ... canonical_name="My Snowflake connection", ... params=dr.DataSourceParameters( ... data_store_id=data_store.id, ... schema="PUBLIC", ... table="stock_prices" ... ) ... ) >>> table = data_source.create_dataset() >>> # Add data from a table in a data store if the primary input is also a table from the same data store >>> recipe.update( ... inputs=[ ... recipe.inputs[0], # Primary input ... JDBCTableDataSourceInput( ... input_type=RecipeInputType.DATASOURCE, ... data_source_id=data_source.id, ... data_store_id=data_store.id, ... dataset_id=table.id, ... sampling=LimitSamplingOperation(rows=250), ... alias='my_table_alias' ... ) ... ] ... ) ``` ### Update primary input sampling You can choose to limit the number of rows to work with when iterating on your recipe operations. By specifying a [sampling operation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#sampling-operations) on the primary input of your recipe, you enable faster computation of the recipe preview. Sampling operations will not modify the number of rows when publishing to a dataset. You should only specify a sampling operation on the primary input. Since secondary inputs are always joined or combined with the primary input, the primary input is the only input that determines the number of rows to show in the recipe preview. ``` >>> from datarobot.models.recipe_operation import LimitSamplingOperation >>> >>> # Configure sampling for an input >>> my_dataset = dr.Dataset.get('5f43a1b2c9e77f0001e6f456') >>> dataset_input = RecipeDatasetInput.from_dataset( ... dataset=my_dataset, ... alias='sampled_data', ... sampling=LimitSamplingOperation(rows=100) ... ) >>> # Update recipe with sampled input >>> recipe.update(inputs=[dataset_input]) ``` ## Update recipe wrangling operations [Wrangling operations](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#wrangling-operations) are the building blocks of your recipe and define the transformations applied to your data. Operations are processed sequentially; the output of one operation becomes the input for the next operation, creating a transformation pipeline. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import * >>> from datarobot.enums import FilterOperationFunctions, AggregationFunctions >>> >>> # Get your recipe >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Create a series of operations >>> operations = [ ... # Filter rows where age > 18 ... FilterOperation( ... conditions=[ ... FilterCondition( ... column="age", ... function=FilterOperationFunctions.GREATER_THAN, ... function_arguments=[18] ... ) ... ], ... keep_rows=True ... ), ... # Then create new column with full name ... ComputeNewOperation( ... expression="CONCAT(first_name, " ", last_name)", ... new_feature_name="full_name" ... ), ... # Then group by department and calculate average salary ... AggregationOperation( ... aggregations=[ ... AggregateFeature( ... feature="salary", ... functions=[AggregationFunctions.AVERAGE] ... ) ... ], ... group_by_columns=["department"] ... ), ... ] >>> >>> # Update the recipe with new list of wrangling operations >>> recipe.update(operations=operations) ``` Below appears the list of available data wrangling operations with an example of their transformation. ### Lags operation The [LagsOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.LagsOperation) creates lagged versions of a column based on datetime ordering. The operation creates new columns for each specified lag. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import LagsOperation >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Create lags for 1 and 2 days for stock price analysis >>> lags_op = LagsOperation( ... column="stock_price", ... orders=[1, 2], ... datetime_partition_column="trade_date", ... multiseries_id_column="ticker_symbol" # For multiseries data (multiple stocks in this example) ... ) >>> >>> # Apply the operation to the recipe >>> recipe.update(operations=[lags_op]) ``` Primary input dataset: | ticker_symbol | trade_date | stock_price | | --- | --- | --- | | AAPL | 2024-01-01 | 150.00 | | AAPL | 2024-01-02 | 152.50 | | AAPL | 2024-01-03 | 149.75 | | AAPL | 2024-01-04 | 153.20 | | MSFT | 2024-01-01 | 380.00 | | MSFT | 2024-01-02 | 385.75 | | MSFT | 2024-01-03 | 382.30 | | MSFT | 2024-01-04 | 388.90 | Recipe preview: | ticker_symbol | trade_date | stock_price | stock_price (1st lag) | stock_price (2nd lag) | | --- | --- | --- | --- | --- | | AAPL | 2024-01-01 | 150.00 | | | | AAPL | 2024-01-02 | 152.50 | 150.00 | | | AAPL | 2024-01-03 | 149.75 | 152.50 | 150.00 | | AAPL | 2024-01-04 | 153.20 | 149.75 | 152.50 | | MSFT | 2024-01-01 | 380.00 | | | | MSFT | 2024-01-02 | 385.75 | 380.00 | | | MSFT | 2024-01-03 | 382.30 | 385.75 | 380.00 | | MSFT | 2024-01-04 | 388.90 | 382.30 | 385.75 | ### Window categorical statistics operation The [WindowCategoricalStatsOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.WindowCategoricalStatsOperation) calculates categorical statistics for a rolling window, creating new columns for each statistical method. This can be used to track trends in categorical data over time. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import WindowCategoricalStatsOperation >>> from datarobot.enums import CategoricalStatsMethods >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Compute most frequent purchase in last 3 purchases >>> window_cat_op = WindowCategoricalStatsOperation( ... column="product_category", ... window_size=3, # Last 3 purchases ... methods=[CategoricalStatsMethods.MOST_FREQUENT], ... datetime_partition_column="purchase_date", ... multiseries_id_column="customer_id" ... ) >>> >>> # Apply the operation to the recipe >>> recipe.update(operations=[window_cat_op]) ``` Primary input dataset: | customer_id | purchase_date | product_category | | --- | --- | --- | | CUST001 | 2024-01-01 | Electronics | | CUST001 | 2024-01-02 | Clothing | | CUST001 | 2024-01-03 | Electronics | | CUST001 | 2024-01-04 | Electronics | | CUST002 | 2024-01-01 | Books | | CUST002 | 2024-01-02 | Books | | CUST002 | 2024-01-03 | Electronics | | CUST002 | 2024-01-04 | Books | Recipe preview: | customer_id | purchase_date | product_category | product_category (3 rows most frequent) | | --- | --- | --- | --- | | CUST001 | 2024-01-01 | Electronics | Electronics | | CUST001 | 2024-01-02 | Clothing | Electronics | | CUST001 | 2024-01-03 | Electronics | Electronics | | CUST001 | 2024-01-04 | Electronics | Electronics | | CUST002 | 2024-01-01 | Books | Books | | CUST002 | 2024-01-02 | Books | Books | | CUST002 | 2024-01-03 | Electronics | Books | | CUST002 | 2024-01-04 | Books | Books | ### Window numerical statistics operation The [WindowNumericStatsOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.WindowNumericStatsOperation) calculates numeric statistics for a rolling window, creating new columns for each statistical method. This operation is useful for computing moving averages, maximums, minimums, and other statistics over time. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import WindowNumericStatsOperation >>> from datarobot.enums import NumericStatsMethods >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Track max and average of last 3 transactions >>> window_num_op = WindowNumericStatsOperation( ... column="sales_amount", ... window_size=3, # Last 3 transactions ... methods=[NumericStatsMethods.AVG, NumericStatsMethods.MAX], ... datetime_partition_column="transaction_date", ... multiseries_id_column="store_id" ... ) >>> >>> # Apply the operation to the recipe >>> recipe.update(operations=[window_num_op]) ``` Primary input dataset: | store_id | transaction_date | sales_amount | | --- | --- | --- | | STORE01 | 2024-01-01 | 100.00 | | STORE01 | 2024-01-02 | 150.00 | | STORE01 | 2024-01-03 | 120.00 | | STORE01 | 2024-01-04 | 200.00 | | STORE02 | 2024-01-01 | 80.00 | | STORE02 | 2024-01-02 | 90.00 | | STORE02 | 2024-01-03 | 110.00 | | STORE02 | 2024-01-04 | 95.00 | Recipe preview: | store_id | transaction_date | sales_amount | sales_amount (3 rows avg) | sales_amount (3 rows max) | | --- | --- | --- | --- | --- | | STORE01 | 2024-01-01 | 100.00 | 100.00 | 100.00 | | STORE01 | 2024-01-02 | 150.00 | 125.00 | 150.00 | | STORE01 | 2024-01-03 | 120.00 | 123.33 | 150.00 | | STORE01 | 2024-01-04 | 200.00 | 156.67 | 200.00 | | STORE02 | 2024-01-01 | 80.00 | 80.00 | 80.00 | | STORE02 | 2024-01-02 | 90.00 | 85.00 | 90.00 | | STORE02 | 2024-01-03 | 110.00 | 93.33 | 110.00 | | STORE02 | 2024-01-04 | 95.00 | 98.33 | 110.00 | ### Time series operation The [TimeSeriesOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.TimeSeriesOperation) generates a dataset ready for time series modeling by creating forecast points, distances, and various time-aware features. By defining a task plan, multiple time-series transformations like lags and rolling statistics are executed and added as features to the recipe data. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import TimeSeriesOperation, TaskPlanElement, Lags >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Define task plan for feature engineering >>> task_plan = [ ... TaskPlanElement( ... column="sales_amount", ... task_list=[Lags(orders=[1])] ... ) ... ] >>> >>> # Create time series operation >>> time_series_op = TimeSeriesOperation( ... target_column="sales_amount", ... datetime_partition_column="sale_date", ... forecast_distances=[1], # Predict 1 period ahead ... task_plan=task_plan ... ) >>> >>> # Apply the operation to the recipe >>> recipe.update(operations=[time_series_op]) ``` Primary input dataset: | store_id | sale_date | sales_amount | | --- | --- | --- | | STORE01 | 2024-01-01 | 1000 | | STORE01 | 2024-01-02 | 1200 | | STORE01 | 2024-01-03 | 1100 | | STORE01 | 2024-01-04 | 1300 | Recipe preview: | store_id (actual) | sale_date (actual) | sales_amount (actual) | Forecast Point | Forecast Distance | sales_amount (1st lag) | sales_amount (naive 1 row seasonal value) | | --- | --- | --- | --- | --- | --- | --- | | STORE01 | 2024-01-02 | 1200 | 2024-01-01 | 1 | 1000 | 1000 | | STORE01 | 2024-01-03 | 1100 | 2024-01-02 | 1 | 1200 | 1200 | | STORE01 | 2024-01-04 | 1300 | 2024-01-03 | 1 | 1100 | 1100 | ### Compute new operation The [ComputeNewOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.ComputeNewOperation) creates a new feature using a SQL expression, allowing you to derive calculated fields from existing columns. This operation can be useful for creating custom business logic, mathematical transformations, and feature combinations. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import ComputeNewOperation >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Create compute new operation to compute total cost, factoring in a discount % >>> compute_op = ComputeNewOperation( ... expression="ROUND(quantity * unit_price * (1 - discount), 2)", ... new_feature_name="total_cost" ... ) >>> >>> # Apply the operation to the recipe >>> recipe.update(operations=[compute_op]) ``` Primary input dataset: | order_id | quantity | unit_price | discount | | --- | --- | --- | --- | | ORD001 | 3 | 25.50 | 0.10 | | ORD002 | 1 | 15.00 | 0.00 | | ORD003 | 2 | 40.00 | 0.15 | | ORD004 | 5 | 12.25 | 0.05 | Recipe preview: | order_id | quantity | unit_price | discount | total_cost | | --- | --- | --- | --- | --- | | ORD001 | 3 | 25.50 | 0.10 | 68.85 | | ORD002 | 1 | 15.00 | 0.00 | 15.00 | | ORD003 | 2 | 40.00 | 0.15 | 68.00 | | ORD004 | 5 | 12.25 | 0.05 | 58.19 | ### Rename column operation The [RenameColumnsOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.RenameColumnsOperation) renames one or more columns. This operation is often useful for standardizing column names, making them more descriptive, or ensuring consistent column naming for specific downstream processes. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import RenameColumnsOperation >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Rename customer id, product name and quantity columns >>> rename_op = RenameColumnsOperation( ... column_mappings={ ... 'cust_id': 'customer_id', ... 'prod_name': 'product_name', ... 'qty': 'quantity' ... } ... ) >>> >>> # Apply the operation to the recipe >>> recipe.update(operations=[rename_op]) ``` Primary input dataset: | cust_id | prod_name | qty | price | | --- | --- | --- | --- | | C001 | Widget A | 3 | 25.99 | | C002 | Gadget B | 1 | 15.50 | | C001 | Tool C | 2 | 45.00 | | C003 | Widget A | 5 | 25.99 | Recipe preview: | customer_id | product_name | quantity | price | | --- | --- | --- | --- | | C001 | Widget A | 3 | 25.99 | | C002 | Gadget B | 1 | 15.50 | | C001 | Tool C | 2 | 45.00 | | C003 | Widget A | 5 | 25.99 | ### Filter operation The [FilterOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.FilterOperation) removes or keeps rows based on one or more filter conditions. Apply multiple conditions with AND/OR logic to create complex filtering rules. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import FilterOperation, FilterCondition >>> from datarobot.enums import FilterOperationFunctions >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Create filter conditions to keep customers over 18 with active status >>> conditions = [ ... FilterCondition( ... column="age", ... function=FilterOperationFunctions.GREATER_THAN_OR_EQUALS, ... function_arguments=[18] ... ), ... FilterCondition( ... column="status", ... function=FilterOperationFunctions.EQUALS, ... function_arguments=["active"] ... ) ... ] >>> >>> # Create filter operation >>> filter_op = FilterOperation( ... conditions=conditions, ... keep_rows=True, # Keep matching rows ... operator="and" # Both conditions must be true ... ) >>> >>> # Apply the operation to the recipe >>> recipe.update(operations=[filter_op]) ``` Primary input dataset: | customer_id | age | status | purchase_amount | | --- | --- | --- | --- | | C001 | 25 | active | 150.00 | | C002 | 17 | active | 75.00 | | C003 | 30 | inactive | 200.00 | | C004 | 22 | active | 95.00 | Recipe preview: | customer_id | age | status | purchase_amount | | --- | --- | --- | --- | | C001 | 25 | active | 150.00 | | C004 | 22 | active | 95.00 | ### Drop columns operation The [DropColumnsOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.DropColumnsOperation) removes one or more columns. This operation is useful for eliminating unnecessary fields, sensitive information, or columns that won't be used in downstream processes. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import DropColumnsOperation >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Create operation to drop 2 extra columns >>> drop_op = DropColumnsOperation( ... columns=['internal_notes', 'legacy_id'] ... ) >>> # Apply the operation to the recipe >>> recipe.update(operations=[drop_op]) ``` Primary input dataset: | customer_id | name | email | internal_notes | legacy_id | | --- | --- | --- | --- | --- | | C001 | John Doe | john@email.com | VIP customer | L001 | | C002 | Jane Doe | jane@email.com | New customer | L002 | | C003 | Bob Lee | bob@email.com | Frequent buyer | L003 | Recipe preview: | customer_id | name | email | | --- | --- | --- | | C001 | John Doe | john@email.com | | C002 | Jane Doe | jane@email.com | | C003 | Bob Lee | bob@email.com | ### Dedupe rows operation The [DedupeRowsOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.DedupeRowsOperation) removes duplicate rows, keeping only unique combinations of values. The operation references values across all columns. This operation helps clean data by eliminating redundant records. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import DedupeRowsOperation >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Create dedupe rows operation >>> dedupe_op = DedupeRowsOperation() >>> # Apply the operation to the recipe >>> recipe.update(operations=[dedupe_op]) ``` Primary input dataset: | customer_id | product | quantity | price | | --- | --- | --- | --- | | C001 | Widget A | 2 | 25.99 | | C002 | Gadget B | 1 | 15.50 | | C001 | Widget A | 2 | 25.99 | | C003 | Tool C | 3 | 45.00 | | C002 | Gadget B | 1 | 15.50 | Recipe preview: | customer_id | product | quantity | price | | --- | --- | --- | --- | | C001 | Widget A | 2 | 25.99 | | C002 | Gadget B | 1 | 15.50 | | C003 | Tool C | 3 | 45.00 | ### Find-and-replace operation The [FindAndReplaceOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.FindAndReplaceOperation) searches for specific strings or patterns in a column and replaces them with new values. The operation supports exact matches, partial matches, or regular expressions for flexible text manipulation. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import FindAndReplaceOperation >>> from datarobot.enums import FindAndReplaceMatchMode >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Replace instances of 'In Progress' (case insensitive) with 'Active' >>> replace_op = FindAndReplaceOperation( ... column="status", ... find="In Progress", ... replace_with="Active", ... match_mode=FindAndReplaceMatchMode.EXACT, ... is_case_sensitive=False ... ) >>> # Apply the operation to the recipe >>> recipe.update(operations=[replace_op]) ``` Primary input dataset: | order_id | status | customer_name | | --- | --- | --- | | ORD001 | In Progress | John Smith | | ORD002 | Completed | Jane Doe | | ORD003 | in progress | Bob Johnson | | ORD004 | Cancelled | Alice Brown | Recipe preview: | order_id | status | customer_name | | --- | --- | --- | | ORD001 | Active | John Smith | | ORD002 | Completed | Jane Doe | | ORD003 | Active | Bob Johnson | | ORD004 | Cancelled | Alice Brown | ### Aggregation operation The [AggregationOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.AggregationOperation) groups data by the specified columns and calculates summary features like sum, average and count. This operation is useful for creating analytical summaries and computing derived features. A new column will be created for each aggregation function applied for each feature chosen for aggregation. ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import AggregationOperation >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Group by customer id and product category >>> # Compute the sum of orders and customer's average order amount >>> agg_op = AggregationOperation( ... group_by_columns=['customer_id', 'product_category'], ... aggregations=[ ... AggregateFeature( ... feature="order_amount", ... functions=[AggregationFunctions.SUM, AggregationFunctions.AVERAGE] ... ) ... ] ... ) >>> # Apply the operation to the recipe >>> recipe.update(operations=[agg_op]) ``` Primary input dataset: | customer_id | product_category | order_id | order_amount | | --- | --- | --- | --- | | C001 | Electronics | ORD001 | 150.00 | | C001 | Electronics | ORD002 | 200.00 | | C001 | Clothing | ORD003 | 75.00 | | C002 | Electronics | ORD004 | 300.00 | | C002 | Electronics | ORD005 | 125.00 | Recipe preview: | customer_id | product_category | order_amount_sum | order_amount_avg | | --- | --- | --- | --- | | C001 | Electronics | 350.00 | 175.00 | | C001 | Clothing | 75.00 | 75.00 | | C002 | Electronics | 425.00 | 212.50 | ### Join operation The [JoinOperation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe_operation.JoinOperation) allows for joining an additional data input to the recipe's current data. This operation can enable you to enrich your primary dataset with additional information from secondary datasets. The join operation only supports one or more equality predicates as the join condition. > [!NOTE] Note > The additional data input is treated as the right side of the join. ``` >>> import datarobot as dr >>> from datarobot.models.recipe import RecipeDatasetInput >>> from datarobot.models.recipe_operation import JoinOperation >>> from datarobot.enums import JoinType >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Get the secondary dataset and add it as an input >>> dataset = dr.Dataset.get('5f43a1b2c9e77f0001e6f123') >>> recipe.update( ... inputs=[ ... recipe.inputs[0], # Keep the original primary input ... RecipeDatasetInput.from_dataset( ... dataset=dataset, ... alias='customers' ... ) ... ] ... ) >>> >>> # Join secondary dataset on customer id >>> # Right dataset in join will always be the new dataset >>> join_op = JoinOperation.join_dataset( ... dataset=dataset, ... join_type=JoinTypes.INNER, ... right_prefix='cust_', ... left_keys=['customer_id'], ... right_keys=['id'] ... ) >>> # Apply the operation to the recipe >>> recipe.update(operations=[join_op]) ``` Primary input dataset (orders) | order_id | customer_id | amount | | --- | --- | --- | | ORD001 | C001 | 150.00 | | ORD002 | C002 | 200.00 | | ORD003 | C001 | 75.00 | Secondary input dataset (customers) | id | name | city | | --- | --- | --- | | C001 | John Smith | New York | | C002 | Jane Doe | Los Angeles | | C003 | Bob Lee | Chicago | Recipe preview: | order_id | customer_id | amount | cust_id | cust_name | cust_city | | --- | --- | --- | --- | --- | --- | | ORD001 | C001 | 150.00 | C001 | John Smith | New York | | ORD002 | C002 | 200.00 | C002 | Jane Doe | Los Angeles | | ORD003 | C001 | 75.00 | C001 | John Smith | New York | ## Set recipe SQL transformation directly For advanced use cases, you can set the recipe's transformation using a SQL expression. This provides maximum flexibility for complex operations that may not be available through standard wrangling operations. Important: Setting SQL directly changes the recipe type to SQL and bypasses any existing wrangling operations. ``` >>> import datarobot as dr >>> >>> # Get your recipe >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Define your SQL transformation >>> sql_query = "MY SQL EXPRESSION HERE" >>> # Update the recipe with SQL >>> recipe.update(sql=sql_query) ``` ## Preview recipe data Before publishing your recipe, you can preview the transformed data to validate your transformations and ensure they produce the expected results with [Recipe.get_preview](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe.Recipe.get_preview). ``` >>> import datarobot as dr >>> >>> # Get your recipe >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Generate a preview of the transformed data >>> preview = recipe.get_preview() >>> # View preview data as a DataFrame >>> preview.df ``` ## Update recipe downsampling Downsampling modifies the size of the dataset published by the recipe, which can improve performance for large datasets and speed up development and testing. This is particularly useful when working with millions of rows where a representative sample is sufficient when publishing to a dataset. Downsampling does not affect the number of rows in the recipe preview. Set a recipe's downsampling with a [downsampling operation](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#downsampling-operations). ``` >>> import datarobot as dr >>> from datarobot.models.recipe_operation import RandomDownsamplingOperation >>> >>> # Get your recipe >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Configure random downsampling to 50,000 rows >>> downsampling = RandomDownsamplingOperation(max_rows=50_000) >>> # Apply downsampling to the recipe >>> recipe.update(downsampling=downsampling) >>> # Disable downsampling >>> recipe.update(downsampling=None) ``` ## Publish recipe to dataset Once your recipe is complete, you can publish it with [Recipe.publish_to_dataset](https://docs.datarobot.com/en/docs/api/reference/sdk/data-wrangling.html#datarobot.models.recipe.Recipe.publish_to_dataset) to create a dataset with your transformed data. ``` >>> import datarobot as dr >>> >>> # Get your recipe >>> recipe = dr.Recipe.get('690bbf77aa31530d8287ae5f') >>> >>> # Publish recipe to create a new dataset >>> dataset = recipe.publish_to_dataset( ... name="Customer Segmentation Data", ... do_snapshot=True ... ) >>> >>> # Publish and attach to an existing use case >>> use_case = dr.UseCase.get('5e1b4f8f2a3c4d5e6f7g8h9i') >>> dataset_with_use_case = recipe.publish_to_dataset( ... name="Advanced Customer Analytics", ... use_cases=use_case ... ) ``` --- # Database connectivity URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/data/database_connectivity.html [Databases](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/dataprep/add-data-usecase.html) are a widely used tool to carry valuable business data. To enable integration with a variety of enterprise databases, DataRobot provides a self-service JDBC product for database connectivity setup. Once configured, you can read data from production databases for model building and predictions. This allows you to quickly train and retrain models on that data, and avoids the unnecessary step of exporting data from your enterprise database to a CSV for ingest to DataRobot. With access to more diverse data, you can build more accurate models. ## Database connection terminology Database connection configuration uses the following terminology: - Data store : A configured connection to a database. It has a name, a specified driver, and a JDBC URL. You can register data stores with DataRobot for ease of re-use. A data store has one connector but can have many data sources. - Data source : A configured connection to the backing data store (the location of data within a given endpoint). A data source specifies, via a SQL query or a selected table and schema data, which data to extract from the data store to use for modeling or predictions. A data source has one data store and one connector but can have many datasets. - Data driver : The software that allows the application to interact with a database; each data store is associated with either a driver or a connector (created by the administrator). The driver configuration saves the storage location in the application of the JAR file and any additional dependency files associated with the driver. - Connector : Similarly to data drivers, a connector allows the application to interact with a database; each data store is associated with either a driver or a connector (created by the administrator). The connector configuration saves the storage location in the application of the JAR file and any additional dependency files associated with the connector. - Dataset : Data, a file or the content of a data source, at a particular point in time. A data source can produce multiple datasets; a dataset has exactly one data source. Review the workflow to set up projects or prediction datasets below. 1. An administrator sets up datarobot.DataDriver to access a particular database. For any particular driver, this setup is performed once for the entire system and the resulting driver is used by all users. 2. Users create a datarobot.DataStore which represents an interface to a particular database using that driver. 3. Users create a datarobot.DataSource representing a particular set of data to be extracted from the data store. 4. Users create projects and prediction datasets from a data source. Users can manage their data stores and data sources, while administrators can manage drivers by listing, retrieving, updating, and deleting existing instances of them. ## Create a driver To create a driver, administrators must specify the following: - class_name : The Java class name for the driver if the type is JDBC; otherwise None. - canonical_name : A user-friendly name or resulting driver to display in the API and the GUI. - files :A list of local files which contain the driver if the type is JDBC; otherwise omitted. - typ : The enum for the type of driver. Defaults to dr.enums.DataDriverTypes.JDBC and can also be dr.enums.DataDriverTypes.DR_DATABASE_V1 . - database_driver : The type of native database to use for non-JDBC. For example, dr.enums.DrDatabaseV1Types.BIGQUERY . ``` >>> import datarobot as dr >>> driver = dr.DataDriver.create( ... class_name='org.postgresql.Driver', ... canonical_name='PostgreSQL', ... files=['/tmp/postgresql-42.2.2.jar'] ... ) >>> driver DataDriver('PostgreSQL') ``` Use the code below to create a non-JDBC driver: ``` driver = dr.DataDriver.create(None, "BigQuery Native", typ=dr.enums.DataDriverTypes.DR_DATABASE_V1, database_driver=dr.enums.DrDatabaseV1Types.BIGQUERY) ``` To retrieve information about existing drivers, such as the driver ID for data store creation, you can use `dr.DataDriver.list()`. ## Create a data store After an administrator has created drivers, any user can use them to create a `DataStore`. A data store represents a JDBC database or a non-JDBC database. When creating them, you should specify the following: - type : The type must be either dr.enums.DataStoreTypes.DR_DATABASE_V1 or dr.enums.DataStoreTypes.JDBC . - canonical_name : A user-friendly name to display in the API and GUI for the data store. - driver_id : The ID of the driver to use to connect to the database. - jdbc_url : The full URL specifying the database connection settings such as the database type, server address, port, and database name if the type is JDBC. - fields : The fields used if the type is dr.enums.DataStoreTypes.DR_DATABASE_V1 . A list of dictionary entries, where each entry has an ID, a name, and a value field. > [!NOTE] Note > You can only create data stores with drivers when using the Python client. Drivers and connectors are not interchangeable for this method. To create a data store with a connector, instead use the [REST API](https://docs.datarobot.com/en/docs/api/reference/public-api/data_connectivity.html#create-a-data-store). ``` >>> import datarobot as dr >>> data_store = dr.DataStore.create( ... data_store_type='jdbc', ... canonical_name='Demo DB', ... driver_id='5a6af02eb15372000117c040', ... jdbc_url='jdbc:postgresql://my.db.address.org:5432/perftest' ... ) >>> data_store DataStore('Demo DB') >>> data_store.test(username='username', password='password') {'message': 'Connection successful'} ``` You can create a non-JDBC Data Store using fields instead: ``` >>> fields = [ ... { ... "id": "bq.project_id", ... "name": "Project Id", ... "value": "mldata-358421", ... } ... ] >>> data_store = dr.DataStore.create( ... data_store_type=dr.enums.DataStoreTypes.DR_DATABASE_V1, ... canonical_name='BigQuery Native Connection', ... driver_id=driver_id, ... fields=fields ... ) ``` ### View data stores To view data stores that already exist, use the code below. However, note that not all data stores show up by default with a call to `dr.DataStore.list()`. You must explicitly pass a `typ=dr.enums.DataStoreListTypes.ALL` argument, since the default is to only show JDBC connections. List all JDBC data stores (default): ``` data_stores = dr.DataStore.list() print(f"Found {len(data_stores)} DataStore(s):") for ds in data_stores: print(f" - {ds.canonical_name} (ID: {ds.id}, Type: {ds.data_store_type})") ``` List all data stores: ``` all_stores = dr.DataStore.list(typ=dr.enums.DataStoreListTypes.ALL) ``` ### Query a data store Once you have a [datarobot.DataStore](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore), you can run SQL directly against it. Use [preview_query()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore.preview_query) for queries that return results, and [execute_update()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore.execute_update) for update statements such as `UPDATE`, `INSERT`, and `DELETE`. > [!NOTE] Note > This functionality is only available for data stores that support structured data, that is, data stores you can run SQL queries against. Use [preview_query](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore.preview_query) to execute a SQL statement and return a preview of the results: ``` >>> import datarobot as dr >>> data_store = dr.DataStore.get('5a8ac90b09a57a0001be306e') >>> preview = data_store.preview_query( ... "SELECT * FROM my_catalog.my_schema.my_table WHERE name LIKE ?", ... credential_id='9963d544d5ce3se783r12190', ... max_rows=10, ... bind_parameters=['%Doe%'], ... ) >>> preview.columns ['id', 'name', 'email'] >>> preview.records [ {'id': 1, 'name': 'John Doe', 'email': 'john.doe@example.com'}, {'id': 2, 'name': 'Jane Doe', 'email': 'jane.doe@example.com'}, ] ``` The `bind_parameters` argument binds values, in order, to the `?` placeholders in the SQL statement. Use [execute_update](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore.execute_update) to run an update statement against the data store: ``` >>> data_store.execute_update( ... "UPDATE my_table SET name = ? WHERE id = ?", ... credential_id='9963d544d5ce3se783r12190', ... bind_parameters=['John', 1], ... ) 'OK' ``` [execute_update](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore.execute_update) returns the `'OK'` message from the server on success. To check the result programmatically, use [DataStore.is_execute_update_success()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore.is_execute_update_success): ``` >>> message = data_store.execute_update( ... "UPDATE my_table SET name = 'John Doe' WHERE id = 1" ... ) >>> dr.DataStore.is_execute_update_success(message) True ``` ## Query using a JDBC connection You can also run SQL against a JDBC URL directly, without creating a [datarobot.DataStore](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore) first. Supply the JDBC URL and credentials directly to [datarobot.JdbcPreview](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.JdbcPreview), and DataRobot connects to the database for the duration of the request. Use [JdbcPreview.preview()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.JdbcPreview.preview) for queries that return results, and [JdbcPreview.execute_update()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.JdbcPreview.execute_update) for update statements such as `UPDATE`, `INSERT`, and `DELETE`. Use [preview()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.JdbcPreview.preview) to execute a SQL statement against the JDBC URL and return a [JdbcPreviewData](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.JdbcPreviewData) preview of the results: ``` >>> import datarobot as dr >>> preview = dr.JdbcPreview.preview( ... jdbc_url='jdbc:postgresql://localhost:5432/mydb', ... sql='SELECT id, name, email FROM public.users WHERE age = ?', ... max_rows=5, ... parameters={'user': 'dbuser', 'password': 'secret'}, ... bind_parameters=[25], ... ) >>> preview.columns ['id', 'name', 'email'] >>> len(preview.records) 5 >>> preview.df['name'].head(5) 0 John Doe 1 Jane Doe 2 Alice Smith 3 Bob Johnson 4 Carol White Name: name, dtype: object ``` The `parameters` argument passes connection credentials and options (such as `user`, `password`, `ssl`, and `timeout`) as key-value pairs, instead of embedding them in the JDBC URL. Use [execute_update()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.JdbcPreview.execute_update) to run an update statement against the JDBC URL: ``` >>> dr.JdbcPreview.execute_update( ... jdbc_url='jdbc:postgresql://localhost:5432/mydb', ... sql='INSERT INTO my_table (id, name) VALUES (?, ?)', ... parameters={'user': 'dbuser', 'password': 'secret'}, ... bind_parameters=[1, 'John'], ... ) 'OK' ``` ## Query using QueryEngine [QueryEngine](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine) is a `datarobot` package add-on that wraps the querying methods above in familiar [SQLAlchemy](https://www.sqlalchemy.org/) syntax, so you can use named parameter binding ( `:name`) and SQLAlchemy constructs (such as `select()` and `insert()`) instead of positional `?` placeholders and raw SQL strings. Install the `datarobot` `query-engine` package add-on to use it: ``` pip install 'datarobot[query-engine]' ``` Create a [QueryEngine](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine) from either a JDBC connection ( [from_jdbc_connection](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine.from_jdbc_connection)) or an existing data store's ID ( [from_data_store](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine.from_data_store)): ``` >>> from datarobot.query_engine import QueryEngine >>> engine = QueryEngine.from_jdbc_connection( ... jdbc_url='jdbc:postgresql://localhost:5432/mydb', ... jdbc_params={'user': 'dbuser', 'password': 'secret'}, ... ) ``` ``` >>> engine = QueryEngine.from_data_store( ... data_store_id='5a8ac90b07a57a0001be501e', ... credential_id='9963d544d5ce3se783r12190', ... ) ``` Use [execute()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine.execute) to run a statement, passing named parameters: ``` >>> results = engine.execute( ... "SELECT * FROM users WHERE name = :name", ... params={"name": "John Doe"}, ... ) >>> results.all() [(1, "John Doe")] ``` You can also pass an existing SQLAlchemy construct instead of a raw SQL string, and bind parameters to it the same way: ``` >>> from sqlalchemy import select, bindparam, column, table >>> USER_TABLE = table("users", column("name"), column("status")) >>> results = engine.execute( ... select(USER_TABLE) ... .where(USER_TABLE.c.name == "John Doe") ... .where(USER_TABLE.c.status == bindparam("status")), ... params={"status": "active"}, ... ) >>> results.all() [("John Doe", "active")] ``` [execute()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine.execute) uses a best-efforts heuristic to guess whether a statement returns rows, based on its first keyword: statements starting with `SELECT`, `WITH`, `SHOW`, and similar query keywords are fetched as results, while statements like `INSERT`, `UPDATE`, and `DELETE` are assumed to be side-effecting and return nothing. > [!NOTE] Note > If [execute()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine.execute) should return rows for a statement but doesn't, [QueryEngine](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine) could have guessed incorrectly that the statement does not return results. Pass `mode` = [QueryMode.QUERY](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryMode) to force [execute()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine.execute) to fetch results: > > ``` > >>> from datarobot.query_engine import QueryMode > >>> results = engine.execute( > ... "UPDATE users SET status = 'active' WHERE name = 'John Doe' RETURNING id, name, status", > ... mode=QueryMode.QUERY, > ... ) > >>> results.all() > [(1, "John Doe", "active")] > ``` See the [QueryEngine](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.query_engine.engine.QueryEngine) API reference for additional examples, including bound parameter expansion for lists & tuples. ## Create a data source Once you have a data store, you can query datasets via the data source. When creating a data source, first create a [datarobot.DataSourceParameters](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataSourceParameters) object from a data store's ID and a query. Then, create the data source with the following: - type : The type must be either dr.enums.DataStoreTypes.DR_DATABASE_V1 or dr.enums.DataStoreTypes.JDBC . - canonical_name : A user-friendly name to display in the API and GUI. - params : The DataSourceParameters object. ``` >>> import datarobot as dr >>> params = dr.DataSourceParameters( ... data_store_id='5a8ac90b07a57a0001be501e', ... query='SELECT * FROM airlines10mb WHERE "Year" >= 1995;' ... ) >>> data_source = dr.DataSource.create( ... data_source_type='jdbc', ... canonical_name='airlines stats after 1995', ... params=params ... ) >>> data_source DataSource('airlines stats after 1995') ``` You can create a non-JDBC Data Store with fields. ``` >>> params = dr.DataSourceParameters( ... data_store_id=data_store_id, ... catalog=catalog, ... schema=schema, ... table=table, ... ) >>> data_source = dr.DataSource.create( ... data_source_type=dr.enums.DataStoreTypes.DR_DATABASE_V1, ... canonical_name='BigQuery Data Source', ... driver_id=driver_id, ... params=params ... ) ``` ## Create projects You can create new projects from a data source, demonstrated below. ``` >>> import datarobot as dr >>> project = dr.Project.create_from_data_source( ... data_source_id='5ae6eee9962d740dd7b86886', ... username='username', ... password='password' ... ) ``` As of v3.0 of the Python API client, you can alternatively pass in the `credential_id` of an existing [Dataset.Credential](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#datarobot.models.Credential) object. ``` >>> import datarobot as dr >>> project = dr.Project.create_from_data_source( ... data_source_id='5ae6eee9962d740dd7b86886', ... credential_id='9963d544d5ce3se783r12190' ... ) ``` Alternatively, pass in `credential_data`, which conforms to `CredentialDataSchema`. ``` >>> import datarobot as dr >>> s3_credential_data = {"credentialType": "s3", "awsAccessKeyId": "key123", "awsSecretAccessKey": "secret123"} >>> project = dr.Project.create_from_data_source( ... data_source_id='5ae6eee9962d740dd7b86886', ... credential_data=s3_credential_data ... ) ``` ## Create prediction datasets Given a data source, new prediction datasets can be created for any project. ``` >>> import datarobot as dr >>> project = dr.Project.get('5ae6f296962d740dd7b86887') >>> prediction_dataset = project.upload_dataset_from_data_source( ... data_source_id='5ae6eee9962d740dd7b86886', ... username='username', ... password='password' ... ) ``` --- # Dataset URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/data/dataset.html To create a project and begin modeling, you first need to [upload your data to DataRobot](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/dataprep/add-data-usecase.html) to prepare a dataset. ## Create a dataset There are several ways to create a dataset.[Dataset.upload](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.upload) takes either a path to a local file, a streamable file object via external URL, or a Pandas DataFrame. ``` >>> import datarobot as dr >>> # Upload a local file >>> dataset_one = dr.Dataset.upload("./data/examples.csv") >>> # Create a dataset with a URL >>> dataset_two = dr.Dataset.upload("https://raw.githubusercontent.com/curran/data/gh-pages/dbpedia/cities/data.csv") >>> # Create a dataset using a pandas DataFrame >>> dataset_three = dr.Dataset.upload(my_df) >>> # Create a dataset using a local file >>> with open("./data/examples.csv", "rb") as file_pointer: ... dataset_four = dr.Dataset.create_from_file(filelike=file_pointer) ``` [Dataset.create_from_file](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.create_from_file) can take either a path to a local file or any streamable file object. ``` >>> import datarobot as dr >>> dataset = dr.Dataset.create_from_file(file_path='data_dir/my_data.csv') >>> with open('data_dir/my_data.csv', 'rb') as f: ... other_dataset = dr.Dataset.create_from_file(filelike=f) ``` [Dataset.create_from_in_memory_data](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.create_from_in_memory_data) creates a dataset from either a `pandas.Dataframe` or a list of dictionaries representing rows of data. Dictionaries representing rows of data must contain the same keys. ``` >>> import pandas as pd >>> data_frame = pd.read_csv('data_dir/my_data.csv') >>> pandas_dataset = dr.Dataset.create_from_in_memory_data(data_frame=data_frame) >>> in_memory_data = [{'key1': 'value', 'key2': 'other_value', ...}, ... {'key1': 'new_value', 'key2': 'other_new_value', ...}, ...] >>> in_memory_dataset = dr.Dataset.create_from_in_memory_data(records=other_data) ``` [Dataset.create_from_url](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.create_from_url) takes CSV data from a URL. If you have set `DISABLE_CREATE_SNAPSHOT_DATASOURCE`, you must set `do_snapshot=False`. ``` >>> url_dataset = dr.Dataset.create_from_url('https://s3.amazonaws.com/my_data/my_dataset.csv', ... do_snapshot=False) ``` [Dataset.create_from_data_source](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.create_from_data_source) takes data from a data source. If you have set `DISABLE_CREATE_SNAPSHOT_DATASOURCE`, you must set `do_snapshot=False`. ``` >>> data_source_dataset = dr.Dataset.create_from_data_source(data_source.id, do_snapshot=False) ``` or ``` >>> data_source_dataset = data_source.create_dataset(do_snapshot=False) ``` ### Use datasets After creating a dataset, you can create [Projects](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/project.html#projects) from it and begin training models. You can also combine project creation and a dataset upload in one method using [Project.create](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project.create). However, using this method means the data is only accessible to the project which created it. ``` >>> project = dataset.create_project(project_name='New Project') >>> project.analyze_and_model('some target') Project(New Project) ``` ## Get information from a dataset The dataset object contains some basic information that you can query, as shown in the snippet below. ``` >>> dataset.id u'5e31cdac39782d0f65842518' >>> dataset.name u'my_data.csv' >>> dataset.categories ["TRAINING", "PREDICTION"] >>> dataset.created_at datetime.datetime(2020, 2, 7, 16, 51, 10, 311000, tzinfo=tzutc()) ``` The snippet below outlines several methods available to retrieve details from a dataset. ``` # Details >>> details = dataset.get_details() >>> details.last_modification_date datetime.datetime(2020, 2, 7, 16, 51, 10, 311000, tzinfo=tzutc()) >>> details.feature_count_by_type [FeatureTypeCount(count=1, feature_type=u'Text'), FeatureTypeCount(count=1, feature_type=u'Boolean'), FeatureTypeCount(count=16, feature_type=u'Numeric'), FeatureTypeCount(count=3, feature_type=u'Categorical')] >>> details.to_dataset().id == details.dataset_id True # Projects >>> dr.Project.create_from_dataset(dataset.id, project_name='Project One') Project(Project One) >>> dr.Project.create_from_dataset(dataset.id, project_name='Project Two') Project(Project Two) >>> dataset.get_projects() [ProjectLocation(url=u'https://app.datarobot.com/api/v2/projects/5e3c94aff86f2d10692497b5/', id=u'5e3c94aff86f2d10692497b5'), ProjectLocation(url=u'https://app.datarobot.com/api/v2/projects/5e3c94eb9525d010a9918ec1/', id=u'5e3c94eb9525d010a9918ec1')] >>> first_id = dataset.get_projects()[0].id >>> dr.Project.get(first_id).project_name 'Project One' # Features >>> all_features = dataset.get_all_features() >>> feature = next(dataset.iterate_all_features(offset=2, limit=1)) >>> feature.name == all_features[2].name True >>> print(feature.name, feature.feature_type, feature.dataset_id) (u'Partition', u'Numeric', u'5e31cdac39782d0f65842518') >>> feature.get_histogram().plot [{'count': 3522, 'target': None, 'label': u'0.0'}, {'count': 3521, 'target': None, 'label': u'1.0'}, ... ] # The raw data >>> with open('myfile.csv', 'wb') as f: ... dataset.get_file(filelike=f) ``` ## Retrieve datasets You can retrieve specific datasets, a list of all datasets, or an iterator that retrieves all or some datasets. ``` >>> dataset_id = '5e387c501a438646ed7bf0f2' >>> dataset = dr.Dataset.get(dataset_id) >>> dataset.id == dataset_id True # A blocking call that returns all datasets >>> dr.Dataset.list() [Dataset(name=u'Untitled Dataset', id=u'5e3c51e0f86f2d1087249728'), Dataset(name=u'my_data.csv', id=u'5e3c2028162e6a5fe9a0d678'), ...] # Avoid listing datasets that fail to properly upload >>> dr.Dataset.list(filter_failed=True) [Dataset(name=u'my_data.csv', id=u'5e3c2028162e6a5fe9a0d678'), Dataset(name=u'my_other_data.csv', id=u'3efc2428g62eaa5f39a6dg7a'), ...] # An iterator that lazily retrieves from the server page-by-page >>> from itertools import islice >>> iterator = dr.Dataset.iterate(offset=2) >>> for element in islice(iterator, 3): ... print(element) Dataset(name='some_data.csv', id='5e8df2f21a438656e7a23d12') Dataset(name='other_data.csv', id='5e8df2e31a438656e7a23d0b') Dataset(name='Untitled Dataset', id='5e6127681a438666cc73c2b0') ``` ## Manage datasets You can modify, delete, and restore datasets. Note that you need the dataset’s ID in order to restore it from deletion. If you do not keep track of the ID, you will be unable to restore a dataset. If your deleted dataset was used to create a project, that project can still access it, but you will not be able to create new projects using that dataset. ``` >>> dataset.modify(name='A Better Name') >>> dataset.name 'A Better Name' >>> new_project = dr.Project.create_from_dataset(dataset.id) >>> stored_id = dataset.id >>> dr.Dataset.delete(dataset.id) # new_project is still ok >>> dr.Project.create_from_dataset(stored_id) Traceback (most recent call last): ... datarobot.errors.ClientError: 410 client error: {u'message': u'Requested Dataset 5e31cdac39782d0f65842518 was previously deleted.'} >>> dr.Dataset.un_delete(stored_id) >>> dr.Project.create_from_dataset(stored_id, project_name='Successful') Project(Successful) ``` You can share a dataset as demonstrated in the following code snippet. ``` >>> from datarobot.enums import SHARING_ROLE >>> from datarobot.models.dataset import Dataset >>> from datarobot.models.sharing import SharingAccess >>> >>> new_access = SharingAccess( >>> "new_user@datarobot.com", >>> SHARING_ROLE.OWNER, >>> can_share=True, >>> ) >>> access_list = [ >>> SharingAccess("old_user@datarobot.com", SHARING_ROLE.OWNER, can_share=True), >>> new_access, >>> ] >>> >>> Dataset.get('my-dataset-id').share(access_list) ``` ## Manage dataset feature lists You can create, modify, and delete custom feature lists on a given dataset. Some feature lists are automatically created by DataRobot and cannot be modified or deleted. Note that you cannot restore a deleted feature list. ``` >>> dataset.get_featurelists() [DatasetFeaturelist(Raw Features), DatasetFeaturelist(universe), DatasetFeaturelist(Informative Features)] >>> dataset_features = [feature.name for feature in dataset.get_all_features()] >>> custom_featurelist = dataset.create_featurelist('Custom Features', dataset_features[:5]) >>> custom_featurelist DatasetFeaturelist(Custom Features) >>> dataset.get_featurelists() [DatasetFeaturelist(Raw Features), DatasetFeaturelist(universe), DatasetFeaturelist(Informative Features), DatasetFeaturelist(Custom Features)] >>> custom_featurelist.update('New Name') >>> custom_featurelist.name 'New Name' >>> custom_featurelist.delete() >>> dataset.get_featurelists() [DatasetFeaturelist(Raw Features), DatasetFeaturelist(universe), DatasetFeaturelist(Informative Features)] ``` ### Use credential data For methods that accept credential data instead of username and password or a credential ID, see the [Credential data](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#id1) section. --- # Feature discovery URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/data/feature_discovery.html [Feature Discovery](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/dataprep/perform-safer.html) allows you to generate features automatically from secondary datasets connected to a primary dataset (training data). You can create this type of connection using relationship configuration. ## Register a primary dataset to create a project To create a Feature Discovery project, upload the primary (training) dataset from a [project](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/project.html#projects). ``` import datarobot as dr primary_dataset = dr.Dataset.create_from_file(file_path='your-training_file.csv') project = dr.Project.create_from_dataset(primary_dataset.id, project_name='Lending Club') ``` ## Register secondary datasets Next, register all the secondary datasets which you want to connect with the primary dataset. You can register the dataset using [Dataset.create_from_file](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.create_from_file), which can take either a path to a local file or any streamable file object. ``` profile_dataset = dr.Dataset.create_from_file(file_path='your_profile_file.csv') transaction_dataset = dr.Dataset.create_from_file(file_path='your_transaction_file.csv') ``` ## Create dataset definitions and relationships Create the [DatasetDefinition](https://docs.datarobot.com/en/docs/api/reference/sdk/features.html#dataset-definition) and [Relationship](https://docs.datarobot.com/en/docs/api/reference/sdk/features.html#relationship) for the profile and transaction datasets created above using helper functions. ``` profile_catalog_id = profile_dataset.id profile_catalog_version_id = profile_dataset.version_id transac_catalog_id = transaction_dataset.id transac_catalog_version_id = transaction_dataset.version_id profile_dataset_definition = dr.DatasetDefinition( identifier='profile', catalog_id=profile_catalog_id, catalog_version_id=profile_catalog_version_id ) transaction_dataset_definition = dr.DatasetDefinition( identifier='transaction', catalog_id=transac_catalog_id, catalog_version_id=transac_catalog_version_id, primary_temporal_key='Date' ) profile_transaction_relationship = dr.Relationship( dataset1_identifier='profile', dataset2_identifier='transaction', dataset1_keys=['CustomerID'], dataset2_keys=['CustomerID'] ) primary_profile_relationship = dr.Relationship( dataset2_identifier='profile', dataset1_keys=['CustomerID'], dataset2_keys=['CustomerID'], feature_derivation_window_start=-14, feature_derivation_window_end=-1, feature_derivation_window_time_unit='DAY', prediction_point_rounding=1, prediction_point_rounding_time_unit='DAY' ) dataset_definitions = [profile_dataset_definition, transaction_dataset_definition] relationships = [primary_profile_relationship, profile_transaction_relationship] ``` ## Create a relationship configuration Create a relationship configuration using the dataset definitions and relationships created above. ``` # Create the relationships configuration to define connection between the datasets relationship_config = dr.RelationshipsConfiguration.create(dataset_definitions=dataset_definitions, relationships=relationships) ``` ## Create a Feature Discovery project Once you have configured relationships for your datasets, you can create a Feature Discovery project. ``` # Set the datetime partitionining column (`date` in this example) partitioning_spec = dr.DatetimePartitioningSpecification('date') # As of v3.0, use ``Project.set_datetime_partitioning`` instead of passing the spec to ``Project.analyze_and_model`` via ``partitioning_method``. project.set_datetime_partitioning(datetime_partition_spec=partitioning_spec) # Set the target for the project and start Feature discovery (if ``Project.set_datetime_partitioning`` was used there is no need to pass ``partitioning_method``) project.analyze_and_model(target='BadLoan', relationships_configuration_id=relationship_config.id, mode='manual', partitioning_method=partitioning_spec) Project(train.csv) ``` To start training a model, reference the ref: `modeling ` documentation. ## Create secondary dataset configuration for predictions Create configurations for your secondary datasets with [Secondary Dataset](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#secondary-dataset): ``` new_secondary_dataset_config = dr.SecondaryDatasetConfigurations.create( project_id=project.id, name='My config', secondary_datasets=secondary_datasets ) ``` For more details, reference the [Secondary Dataset](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#secondary-dataset) configuration documentation. ## Make predictions with a trained model To make predictions with a trained model, reference the [Predictions documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/index.html). ``` dataset_from_path = project.upload_dataset( './data_to_predict.csv', secondary_datasets_config_id=new_secondary_dataset_config.id ) predict_job_1 = model.request_predictions(dataset_from_path.id) ``` ### Common errors #### Dataset registration failed You may have a job not successfully complete when registering a dataset: ``` datasetdr.Dataset.create_from_file(file_path='file.csv') datarobot.errors.AsyncProcessUnsuccessfulError: The job did not complete successfully. ``` There are two possible solutions: - Check the internet connectivity. Sometimes network flakiness can cause upload errors. - Check the dataset file size. If a file is too large, you should consider uploading the dataset via a URL rather than uploading the file directly. #### Relationship configuration errors It's possible to submit invalid field data when configuring data relationships: ``` datarobot.errors.ClientError: 422 client error: {u'message': u'Invalid field data', u'errors': {u'datasetDefinitions': {u'1': {u'identifier': u'value cannot contain characters: $ - " . { } / \\'}, u'0': {u'identifier': u'value cannot contain characters: $ - " . { } / \\'}}}} ``` There are two possible solutions: - Check the identifier name passed in datasets_definitions and relationships. - Do not use the name of the dataset if you did not specify it when registering the dataset to the Data Registry. ``` datarobot.errors.ClientError: 422 client error: {u'message': u'Invalid field data', u'errors': {u'datasetDefinitions': {u'1': {u'primaryTemporalKey': u'date column doesnt exist'}, }}} ``` Solution: - Check if the name of the column passed as primaryTemporalKey is correct, as it is case-sensitive. ## Configure relationships A relationship’s configuration specifies additional datasets to be included to a project, how these datasets are related to each other, and the primary dataset. When a relationships configuration is specified for a project, Feature Discovery will create features automatically from these datasets. You can create a relationship configuration from uploaded AI Catalog items. After uploading all the secondary datasets in the AI Catalog: - Create the dataset’s definition to specify which datasets to be used as secondary datasets along with its details - Configure relationships among the above datasets. ``` relationship_config = dr.RelationshipsConfiguration.create(dataset_definitions=dataset_definitions, relationships=relationships) >>> relationship_config.id u'5506fcd38bd88f5953219da0' ``` ## Dataset definitions and relationships using helper functions Create the [DatasetDefinition](https://docs.datarobot.com/en/docs/api/reference/public-api/features.html#dataset-definition) and [Relationship](https://docs.datarobot.com/en/docs/api/reference/public-api/features.html#relationship) for the profile and transaction dataset using helper functions. ``` profile_catalog_id = '5ec4aec1f072bc028e3471ae' profile_catalog_version_id = '5ec4aec2f072bc028e3471b1' transac_catalog_id = '5ec4aec268f0f30289a03901' transac_catalog_version_id = '5ec4aec268f0f30289a03900' profile_dataset_definition = dr.DatasetDefinition( identifier='profile', catalog_id=profile_catalog_id, catalog_version_id=profile_catalog_version_id ) transaction_dataset_definition = dr.DatasetDefinition( identifier='transaction', catalog_id=transac_catalog_id, catalog_version_id=transac_catalog_version_id, primary_temporal_key='Date' ) profile_transaction_relationship = dr.Relationship( dataset1_identifier='profile', dataset2_identifier='transaction', dataset1_keys=['CustomerID'], dataset2_keys=['CustomerID'] ) primary_profile_relationship = dr.Relationship( dataset2_identifier='profile', dataset1_keys=['CustomerID'], dataset2_keys=['CustomerID'], feature_derivation_window_start=-14, feature_derivation_window_end=-1, feature_derivation_window_time_unit='DAY', prediction_point_rounding=1, prediction_point_rounding_time_unit='DAY' ) dataset_definitions = [profile_dataset_definition, transaction_dataset_definition] relationships = [primary_profile_relationship, profile_transaction_relationship] ``` ## Dataset definition and relationship using a dictionary Create the dataset definitions and relationships for the profile and transaction dataset using dict directly. ``` profile_catalog_id = profile_dataset.id profile_catalog_version_id = profile_dataset.version_id transac_catalog_id = transaction_dataset.id transac_catalog_version_id = transaction_dataset.version_id dataset_definitions = [ { 'identifier': 'transaction', 'catalogVersionId': transac_catalog_version_id, 'catalogId': transac_catalog_id, 'primaryTemporalKey': 'Date', 'snapshotPolicy': 'latest', }, { 'identifier': 'profile', 'catalogId': profile_catalog_id, 'catalogVersionId': profile_catalog_version_id, 'snapshotPolicy': 'latest', }, ] relationships = [ { 'dataset2Identifier': 'profile', 'dataset1Keys': ['CustomerID'], 'dataset2Keys': ['CustomerID'], 'featureDerivationWindowStart': -14, 'featureDerivationWindowEnd': -1, 'featureDerivationWindowTimeUnit': 'DAY', 'predictionPointRounding': 1, 'predictionPointRoundingTimeUnit': 'DAY', }, { 'dataset1Identifier': 'profile', 'dataset2Identifier': 'transaction', 'dataset1Keys': ['CustomerID'], 'dataset2Keys': ['CustomerID'], }, ] ``` ## Retrieving relationship configuration You can retrieve a specific relationship’s configuration using the ID of the relationship configuration. ``` relationship_config_id = '5506fcd38bd88f5953219da0' relationship_config = dr.RelationshipsConfiguration(id=relationship_config_id).get() >>> relationship_config.id == relationship_config_id True # Get all the datasets used in this relationship's configuration >> len(relationship_config.dataset_definitions) == 2 True >> relationship_config.dataset_definitions[0] { 'feature_list_id': '5ec4af93603f596525d382d3', 'snapshot_policy': 'latest', 'catalog_id': '5ec4aec268f0f30289a03900', 'catalog_version_id': '5ec4aec268f0f30289a03901', 'primary_temporal_key': 'Date', 'is_deleted': False, 'identifier': 'transaction', 'feature_lists': [ { 'name': 'Raw Features', 'description': 'System created featurelist', 'created_by': 'User1', 'creation_date': datetime.datetime(2020, 5, 20, 4, 18, 27, 150000, tzinfo=tzutc()), 'user_created': False, 'dataset_id': '5ec4aec268f0f30289a03900', 'id': '5ec4af93603f596525d382d1', 'features': [u'CustomerID', u'AccountID', u'Date', u'Amount', u'Description'] }, { 'name': 'universe', 'description': 'System created featurelist', 'created_by': 'User1', 'creation_date': datetime.datetime(2020, 5, 20, 4, 18, 27, 172000, tzinfo=tzutc()), 'user_created': False, 'dataset_id': '5ec4aec268f0f30289a03900', 'id': '5ec4af93603f596525d382d2', 'features': [u'CustomerID', u'AccountID', u'Date', u'Amount', u'Description'] }, { 'features': [u'CustomerID', u'AccountID', u'Date', u'Amount', u'Description'], 'description': 'System created featurelist', 'created_by': u'Garvit Bansal', 'creation_date': datetime.datetime(2020, 5, 20, 4, 18, 27, 179000, tzinfo=tzutc()), 'dataset_version_id': '5ec4aec268f0f30289a03901', 'user_created': False, 'dataset_id': '5ec4aec268f0f30289a03900', 'id': u'5ec4af93603f596525d382d3', 'name': 'Informative Features' } ] } # Get information regarding how the datasets are connected among themselves as well as theprimary dataset >> relationship_config.relationships [ { 'dataset2Identifier': 'profile', 'dataset1Keys': ['CustomerID'], 'dataset2Keys': ['CustomerID'], 'featureDerivationWindowStart': -14, 'featureDerivationWindowEnd': -1, 'featureDerivationWindowTimeUnit': 'DAY', 'predictionPointRounding': 1, 'predictionPointRoundingTimeUnit': 'DAY', }, { 'dataset1Identifier': 'profile', 'dataset2Identifier': 'transaction', 'dataset1Keys': ['CustomerID'], 'dataset2Keys': ['CustomerID'], }, ] ``` ## Update details of a relationship configuration Use the snippet below as an example of how to update the details of the existing relationship configuration. ``` relationship_config_id = '5506fcd38bd88f5953219da0' relationship_config = dr.RelationshipsConfiguration(id=relationship_config_id) # Remove obsolete dataset definitions and its relationships new_datasets_definiton = [ { 'identifier': 'user', 'catalogVersionId': '5c88a37770fc42a2fcc62759', 'catalogId': '5c88a37770fc42a2fcc62759', 'snapshotPolicy': 'latest', }, ] # Get information regarding how the datasets are connected among themselves as well as the primary dataset new_relationships = [ { 'dataset2Identifier': 'user', 'dataset1Keys': ['user_id', 'dept_id'], 'dataset2Keys': ['user_id', 'dept_id'], }, ] new_config = relationship_config.replace(new_datasets_definiton, new_relationships) >>> new_config.id == relationship_config_id True >>> new_config.datasets_definition [ { 'identifier': 'user', 'catalogVersionId': '5c88a37770fc42a2fcc62759', 'catalogId': '5c88a37770fc42a2fcc62759', 'snapshotPolicy': 'latest', }, ] >>> new_config.relationships [ { 'dataset2Identifier': 'user', 'dataset1Keys': ['user_id', 'dept_id'], 'dataset2Keys': ['user_id', 'dept_id'], }, ] ``` ## Delete relationships configuration You can delete a relationship configuration that is not used by any project. ``` relationship_config_id = '5506fcd38bd88f5953219da0' relationship_config = dr.RelationshipsConfiguration(id=relationship_config_id) result = relationship_config.get() >>> result.id == relationship_config_id True # Delete the relationships configuration >>> relationship_config.delete() >>> relationship_config.get() ClientError: Relationships Configuration 5506fcd38bd88f5953219da0 not found ``` ## Secondary dataset configuration Secondary dataset configuration allows you to use the different secondary datasets for a Feature Discovery project when making predictions. ## Secondary datasets using helper functions Create the [Secondary Dataset](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#secondary-dataset) using helper functions. ``` >>> profile_catalog_id = '5ec4aec1f072bc028e3471ae' >>> profile_catalog_version_id = '5ec4aec2f072bc028e3471b1' >>> transac_catalog_id = '5ec4aec268f0f30289a03901' >>> transac_catalog_version_id = '5ec4aec268f0f30289a03900' profile_secondary_dataset = dr.SecondaryDataset( identifier='profile', catalog_id=profile_catalog_id, catalog_version_id=profile_catalog_version_id, snapshot_policy='latest' ) transaction_secondary_dataset = dr.SecondaryDataset( identifier='transaction', catalog_id=transac_catalog_id, catalog_version_id=transac_catalog_version_id, snapshot_policy='latest' ) secondary_datasets = [profile_secondary_dataset, transaction_secondary_dataset] ``` ## Create secondary datasets with dict You can create secondary datasets using raw dict structure. ``` secondary_datasets = [ { 'snapshot_policy': u'latest', 'identifier': u'profile', 'catalog_version_id': u'5fd06b4af24c641b68e4d88f', 'catalog_id': u'5fd06b4af24c641b68e4d88e' }, { 'snapshot_policy': u'dynamic', 'identifier': u'transaction', 'catalog_version_id': u'5fd1e86c589238a4e635e98e', 'catalog_id': u'5fd1e86c589238a4e635e98d' } ] ``` ## Create a secondary dataset configuration Create a secondary dataset configuration for a Feature Discovery Project which uses two secondary datasets: `profile` and `transaction`. ``` import datarobot as dr project = dr.Project.get(project_id='54e639a18bd88f08078ca831') new_secondary_dataset_config = dr.SecondaryDatasetConfigurations.create( project_id=project.id, name='My config', secondary_datasets=secondary_datasets ) >>> new_secondary_dataset_config.id '5fd1e86c589238a4e635e93d' ``` ## Retrieve a secondary dataset configuration You can retrieve specific secondary dataset configurations using the configuration ID. ``` >>> config_id = '5fd1e86c589238a4e635e93d' secondary_dataset_config = dr.SecondaryDatasetConfigurations(id=config_id).get() >>> secondary_dataset_config.id == config_id True >>> secondary_dataset_config { 'created': datetime.datetime(2020, 12, 9, 6, 16, 22, tzinfo=tzutc()), 'creator_full_name': u'abc@datarobot.com', 'creator_user_id': u'asdf4af1gf4bdsd2fba1de0a', 'credential_ids': None, 'featurelist_id': None, 'id': u'5fd1e86c589238a4e635e93d', 'is_default': True, 'name': u'My config', 'project_id': u'5fd06afce2456ec1e9d20457', 'project_version': None, 'secondary_datasets': [ { 'snapshot_policy': u'latest', 'identifier': u'profile', 'catalog_version_id': u'5fd06b4af24c641b68e4d88f', 'catalog_id': u'5fd06b4af24c641b68e4d88e' }, { 'snapshot_policy': u'dynamic', 'identifier': u'transaction', 'catalog_version_id': u'5fd1e86c589238a4e635e98e', 'catalog_id': u'5fd1e86c589238a4e635e98d' } ] } ``` ## List all secondary dataset configurations You can list all secondary dataset configurations created in the project. ``` >>> secondary_dataset_configs = dr.SecondaryDatasetConfigurations.list(project.id) >>> secondary_dataset_configs[0] { 'created': datetime.datetime(2020, 12, 9, 6, 16, 22, tzinfo=tzutc()), 'creator_full_name': u'abc@datarobot.com', 'creator_user_id': u'asdf4af1gf4bdsd2fba1de0a', 'credential_ids': None, 'featurelist_id': None, 'id': u'5fd1e86c589238a4e635e93d', 'is_default': True, 'name': u'My config', 'project_id': u'5fd06afce2456ec1e9d20457', 'project_version': None, 'secondary_datasets': [ { 'snapshot_policy': u'latest', 'identifier': u'profile', 'catalog_version_id': u'5fd06b4af24c641b68e4d88f', 'catalog_id': u'5fd06b4af24c641b68e4d88e' }, { 'snapshot_policy': u'dynamic', 'identifier': u'transaction', 'catalog_version_id': u'5fd1e86c589238a4e635e98e', 'catalog_id': u'5fd1e86c589238a4e635e98d' } ] } ``` --- # Features URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/data/features-python.html Features represent the columns in your dataset that DataRobot uses for modeling. Each feature has properties such as type, statistics, and importance that help you understand your data and make informed modeling decisions. This page describes how to work with features in your projects. ## Retrieve features You can retrieve all features from a project or get a specific feature by name. ### Get all features To retrieve all features from a project, use `Project.get_features()`: ``` >>> import datarobot as dr >>> project = dr.Project.get('5e3c94aff86f2d10692497b5') >>> features = project.get_features() >>> len(features) 21 >>> features[0].name 'Partition' >>> features[0].feature_type 'Numeric' ``` You can also iterate through features using `Project.iterate_features()`: ``` >>> from itertools import islice >>> feature_iterator = project.iterate_features(offset=0, limit=10) >>> for feature in islice(feature_iterator, 5): ... print(feature.name, feature.feature_type) Partition Numeric CustomerID Categorical Age Numeric Income Numeric Education Categorical ``` ### Get a specific feature To retrieve a single feature by name, use `Feature.get()`: ``` >>> feature = dr.Feature.get(project_id=project.id, feature_name='Age') >>> feature.name 'Age' >>> feature.feature_type 'Numeric' >>> feature.project_id '5e3c94aff86f2d10692497b5' ``` ## Explore feature properties Each feature object contains detailed information about the feature's characteristics and distribution. ### Basic feature information ``` >>> feature = project.get_features()[0] >>> feature.name 'Age' >>> feature.feature_type 'Numeric' >>> feature.id 12345 >>> feature.project_id '5e3c94aff86f2d10692497b5' ``` ### Feature statistics For numeric features, you can access summary statistics from the EDA sample: ``` >>> numeric_feature = dr.Feature.get(project_id=project.id, feature_name='Income') >>> numeric_feature.min 25000.0 >>> numeric_feature.max 150000.0 >>> numeric_feature.mean 67500.0 >>> numeric_feature.median 65000.0 >>> numeric_feature.std_dev 18500.5 ``` For date features, summary statistics are expressed as ISO-8601 formatted date strings: ``` >>> date_feature = dr.Feature.get(project_id=project.id, feature_name='TransactionDate') >>> date_feature.min '2020-01-01T00:00:00Z' >>> date_feature.max '2023-12-31T23:59:59Z' ``` ### Feature data quality To check data quality metrics for features: ``` >>> feature = dr.Feature.get(project_id=project.id, feature_name='Email') >>> feature.unique_count 1250 >>> feature.na_count 5 >>> feature.low_information False >>> feature.importance 0.85 ``` The `importance` attribute provides a numeric measure of the strength of relationship between the feature and target, independent of any model. This value may be `None` for non-modeling features such as the partition columns. ### Target leakage detection To check if a feature has [target leakage](https://docs.datarobot.com/en/docs/reference/data-ref/data-quality-ref.html#target-leakage): ``` >>> feature = dr.Feature.get(project_id=project.id, feature_name='CustomerID') >>> feature.target_leakage 'FALSE' ``` Target leakage can return the following values: - FALSE : No target leakage detected. - MODERATE : Moderate risk of target leakage. - HIGH_RISK : High risk of target leakage. - SKIPPED_DETECTION : Target leakage detection was not run on this feature. ### Time series eligibility For time series projects, check if a feature can be used as the datetime partition column. ``` >>> date_feature = dr.Feature.get(project_id=project.id, feature_name='Date') >>> date_feature.time_series_eligible True >>> date_feature.time_series_eligibility_reason 'Suitable for use as datetime partition column' >>> date_feature.time_step 1 >>> date_feature.time_unit 'DAY' ``` ## Get feature histograms Histograms provide a visual representation of feature distributions. To retrieve histogram data for any feature: ``` >>> feature = dr.Feature.get(project_id=project.id, feature_name='Age') >>> histogram = feature.get_histogram() >>> histogram.plot [{'count': 150, 'target': None, 'label': '18-25'}, {'count': 320, 'target': None, 'label': '26-35'}, {'count': 450, 'target': None, 'label': '36-45'}, {'count': 280, 'target': None, 'label': '46-55'}, {'count': 100, 'target': None, 'label': '56+'}] ``` You can specify the maximum number of bins: ``` >>> histogram = feature.get_histogram(bin_limit=20) >>> len(histogram.plot) 20 ``` ## Work with feature lists Feature lists are collections of features used for modeling. You can retrieve feature lists from a project and examine which features they contain. ### Get project feature lists ``` >>> project = dr.Project.get('5e3c94aff86f2d10692497b5') >>> featurelists = project.get_featurelists() >>> featurelists [Featurelist('Raw Features'), Featurelist('Informative Features'), Featurelist('universe')] ``` ### Examine features in a feature list ``` >>> raw_features = project.get_featurelists()[0] >>> raw_features.features ['Partition', 'CustomerID', 'Age', 'Income', 'Education', 'Email'] >>> len(raw_features.features) 21 ``` ### Create a custom feature list You can create custom feature lists from a subset of available features: ``` >>> all_features = project.get_features() >>> selected_feature_names = [f.name for f in all_features if f.feature_type == 'Numeric'] >>> custom_featurelist = project.create_featurelist( ... name='Numeric Features Only', ... features=selected_feature_names ... ) >>> custom_featurelist Featurelist('Numeric Features Only') ``` ## Analyze categorical features For categorical features, you can access additional insights about the distribution of categories. ### Get key summary for categorical features For summarized categorical features, you can retrieve statistics for the top keys: ``` >>> categorical_feature = dr.Feature.get(project_id=project.id, feature_name='ProductCategory') >>> key_summary = categorical_feature.key_summary >>> key_summary[0] {'key': 'Electronics', 'summary': {'min': 0, 'max': 29815.0, 'stdDev': 6498.029, 'mean': 1490.75, 'median': 0.0, 'pctRows': 5.0}} ``` The key summary provides statistics for the top 50 keys, including: - `min`: Minimum value of the key. - `max`: Maximum value of the key. - `mean`: Mean value of the key. - `median`: Median value of the key. - `stdDev`: Standard deviation of the key. - `pctRows`: Percentage occurrence of key in the EDA sample. ## Analyze multicategorical features For multicategorical features, you can retrieve specialized insights about label relationships. ### Get a multicategorical histogram ``` >>> multicat_feature = dr.Feature.get(project_id=project.id, feature_name='Tags') >>> histogram = multicat_feature.get_multicategorical_histogram() >>> histogram MulticategoricalHistogram(...) ``` ### Get pairwise correlations Analyze correlations between labels in a multicategorical feature: ``` >>> correlations = multicat_feature.get_pairwise_correlations() >>> correlations PairwiseCorrelations(...) ``` ### Get pairwise joint probabilities ``` >>> joint_probs = multicat_feature.get_pairwise_joint_probabilities() >>> joint_probs PairwiseJointProbabilities(...) ``` ### Get pairwise conditional probabilities ``` >>> cond_probs = multicat_feature.get_pairwise_conditional_probabilities() >>> cond_probs PairwiseConditionalProbabilities(...) ``` ## Time series feature properties For time series projects, you can retrieve additional properties for features when used with multiseries or cross-series configurations. ### Get multiseries properties Retrieve time series properties for a potential multiseries datetime partition column: ``` >>> date_feature = dr.Feature.get(project_id=project.id, feature_name='Date') >>> properties = date_feature.get_multiseries_properties( ... multiseries_id_columns=['StoreID'] ... ) >>> properties {'time_series_eligible': True, 'time_unit': 'DAY', 'time_step': 1} ``` ### Get cross-series properties To retrieve cross-series properties for multiseries ID columns: ``` >>> multiseries_feature = dr.Feature.get(project_id=project.id, feature_name='StoreID') >>> properties = multiseries_feature.get_cross_series_properties( ... datetime_partition_column='Date', ... cross_series_group_by_columns=['Region'] ... ) >>> properties {'name': 'StoreID', 'eligibility': 'Eligible as cross-series group-by column', 'isEligible': True} ``` ## Filter and search features You can filter features by various criteria to find specific features of interest. ### Filter by feature type ``` >>> all_features = project.get_features() >>> numeric_features = [f for f in all_features if f.feature_type == 'Numeric'] >>> categorical_features = [f for f in all_features if f.feature_type == 'Categorical'] >>> text_features = [f for f in all_features if f.feature_type == 'Text'] ``` ### Find features with missing data ``` >>> features_with_missing = [f for f in project.get_features() ... if f.na_count is not None and f.na_count > 0] >>> for feature in features_with_missing: ... print(f"{feature.name}: {feature.na_count} missing values") Email: 5 missing values Phone: 12 missing values ``` ### Find low-information features ``` >>> low_info_features = [f for f in project.get_features() if f.low_information] >>> [f.name for f in low_info_features] ['ConstantColumn', 'SingleValueColumn'] ``` ### Find features by importance threshold ``` >>> important_features = [f for f in project.get_features() ... if f.importance is not None and f.importance > 0.5] >>> sorted_features = sorted(important_features, key=lambda x: x.importance, reverse=True) >>> for feature in sorted_features[:5]: ... print(f"{feature.name}: {feature.importance:.3f}") Income: 0.892 Age: 0.756 Education: 0.643 ``` ## Common workflows ### Analyze all features in a project ``` >>> project = dr.Project.get('5e3c94aff86f2d10692497b5') >>> features = project.get_features() >>> >>> print(f"Total features: {len(features)}") >>> print(f"Feature types: {set(f.feature_type for f in features)}") >>> >>> for feature in features: ... print(f"\n{feature.name} ({feature.feature_type}):") ... if feature.na_count is not None: ... print(f" Missing values: {feature.na_count}") ... if feature.importance is not None: ... print(f" Importance: {feature.importance:.3f}") ... if feature.target_leakage != 'SKIPPED_DETECTION': ... print(f" Target leakage: {feature.target_leakage}") ``` ### Export feature information to a DataFrame ``` >>> import pandas as pd >>> >>> features = project.get_features() >>> feature_data = [] >>> for feature in features: ... feature_data.append({ ... 'name': feature.name, ... 'type': feature.feature_type, ... 'importance': feature.importance, ... 'missing_count': feature.na_count, ... 'unique_count': feature.unique_count, ... 'low_information': feature.low_information, ... 'target_leakage': feature.target_leakage ... }) >>> >>> df = pd.DataFrame(feature_data) >>> df.to_csv('feature_summary.csv', index=False) ``` ### Compare features across projects ``` >>> project1 = dr.Project.get('5e3c94aff86f2d10692497b5') >>> project2 = dr.Project.get('5e3c94aff86f2d10692497b6') >>> >>> features1 = {f.name: f for f in project1.get_features()} >>> features2 = {f.name: f for f in project2.get_features()} >>> >>> common_features = set(features1.keys()) & set(features2.keys()) >>> print(f"Common features: {len(common_features)}") >>> >>> for feature_name in common_features: ... f1 = features1[feature_name] ... f2 = features2[feature_name] ... if f1.feature_type != f2.feature_type: ... print(f"{feature_name}: type mismatch ({f1.feature_type} vs {f2.feature_type})") ``` ## Considerations - Feature statistics ( min , max , mean , median , std_dev ) are calculated from the EDA sample data. - For non-numeric features or features created prior to summary statistics becoming available, their values will be None . - The importance attribute is independent of any model and measures the relationship strength between the feature and target. - In time series projects, Feature objects represent input features, while ModelingFeature objects represent features used for modeling after partitioning. - Feature histograms are based on the EDA sample and may not reflect the full dataset distribution. --- # File registry URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/data/file_registry.html To work with unstructured data, DataRobot provides a simple file system interface through which you upload and work with files. This file system interface mimics a traditional file system with a directory structure and supports common Unix file system operations. DataRobot's file system uses containers, referred to as catalog items, to store one or more files using a key-value storage approach where the file's path is the key and its contents the value. Uploaded files can be leveraged behind the scenes in other areas or workflows in the DataRobot platform, such as creating a [vector database](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/vector-databases.html#vector-databases) with files uploaded from a SharePoint site. Using DataRobot's [datarobot.fs.DataRobotFileSystem](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem), an [fsspec](https://filesystem-spec.readthedocs.io/en/latest/index.html) -compatible implementation, you can quickly stand up file-based workflows that leverage the same code patterns as other fsspec-backed file systems. ## File system terminology The DataRobot file system uses the following terminology: - Catalog item : The container that stores one or more files. Catalog items are a form of data assets in the DataRobot platform. Each catalog item has its own ID, permissions, and version history. - Catalog item directory : The top-level directory in the file system that maps to a catalog item. The directory name matches the catalog item's ID. All files in a catalog item live as paths inside this directory. - Path : The location of a file or directory in the file system. A path includes the catalog item ID and the internal path within the catalog item. Paths take the form dr:///path/to/file or /path/to/file . - Overwrite strategy : A setting that controls the behavior when an upload or write targets a path where a file already exists. See FilesOverwriteStrategy for the available options. - Signed URL : A temporary, time-limited URL that grants direct read access to a single file without further authentication. Useful for sharing files or handing them to external tools. ## Reminders The following should be kept in mind when working with the DataRobot file system: - The file system simulates a top-level directory structure by giving each catalog item its own directory named after its ID. Files inside the catalog item appear as paths inside that directory. - Permissions are attached to the catalog item containing the files. All files inside a catalog item inherit permissions from the catalog item with respect to utilizing the File System API documented here. Files may also have external access control lists (ACLs) permissions attached to them if the connector used to ingest the files supports it. See the documentation for ACL Hydration for more information. - Because the file system uses key-value pairs to store files inside containers, directory structures are simulated and may change based on their contents. This results in the following consequences: - A catalog item itself may be empty even though empty directories inside it are not supported. - Some file operations may cause name collisions when creating/moving/copying files in the file system. File collisions are handled according to the overwrite strategy specified when performing the operation. ## Set up the file system The examples in this guide build on each other. The setup below configures the [DataRobot client](https://docs.datarobot.com/en/docs/api/reference/sdk/client-setup.html#client-setup) and creates a [DataRobotFileSystem](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem) to use. Using Python 3.9+, install the `datarobot` `fs` package add-on: ``` pip install 'datarobot[fs]' ``` ``` import datarobot as dr from datarobot.fs import DataRobotFileSystem dr.Client(token="", endpoint="https://app.datarobot.com/api/v2") fs = DataRobotFileSystem() ``` ## Create a new catalog item A catalog item is the container that holds your files in the DataRobot file system. Every path you reference is rooted at a catalog item, so you'll need one to start your workflow. There are two ways to create a file catalog item: create a new empty catalog item, or clone an existing catalog item. Both approaches return the new catalog item's ID, which you'll reuse to build paths in the format `dr://{catalog_id}/...` for every subsequent operation. Use [create_catalog_item_dir](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.create_catalog_item_dir) to create a brand-new, empty catalog item. ``` # Create a brand-new, empty catalog item catalog_id = fs.create_catalog_item_dir() ``` Use [clone_catalog_item_dir](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.clone_catalog_item_dir) to create a copy of an existing catalog item. Pass `files_to_omit` to exclude specific files from the clone. The paths in `files_to_omit` are relative to the source catalog item's root. ``` # Clone an existing catalog item, copying every file into a new one source_catalog_id = "" clone_id = fs.clone_catalog_item_dir(source_catalog_id) # Or clone but omit specific files from the source partial_clone_id = fs.clone_catalog_item_dir( source_catalog_id, files_to_omit=["data/scores.csv", "notes/draft.txt"], ) ``` ## Add files Add files to the DataRobot file system by uploading files from your local machine, a public URL, or a [data source](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/database_connectivity.html#database-connectivity-overview). Alternatively, write content directly to a file path to create a new file. ### Write content directly to new files Write content directly to a file path to create a new file in that location. Use [open](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.open) in write mode for text or buffered binary writes, and [pipe_file](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.pipe_file) or [pipe](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.pipe) for a one-shot write of raw bytes. ``` # Write a text file in place with fs.open(f"dr://{catalog_id}/notes/readme.txt", mode="w") as f: f.write("This catalog item contains demo files for the file system guide.") # Write raw bytes in a single call. fs.pipe_file(f"dr://{catalog_id}/data/sample.csv", b"name,score\nCharlie,72\n") ``` By default, `open` uses [FilesOverwriteStrategy.REPLACE](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.enums.FilesOverwriteStrategy), so writing to a path that already contains a file will overwrite the existing file. Alternatively, specify a different `overwrite_strategy` to change this behavior. For example, use [FilesOverwriteStrategy.RENAME](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.enums.FilesOverwriteStrategy) to create a duplicate file suffixed with `(2)` instead. ``` from datarobot.enums import FilesOverwriteStrategy # Write to the existing path notes/readme.txt. # The new content will be placed in a new file /notes/readme (2).txt with fs.open( f"dr://{catalog_id}/notes/readme.txt", mode="w", overwrite_strategy=FilesOverwriteStrategy.RENAME ) as f: f.write("This content is written to a new file because RENAME was specified.") ``` ### Upload local files To copy a file from your local machine into the catalog item, use [put_file](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.put_file) or [ `put`]file-system#datarobot.fs.file_system.DataRobotFileSystem.put){ target=_blank } for multiple files or directories. The example below first creates a few small local files, then uploads them in two different ways. ``` import tempfile import fsspec # Use the local fsspec implementation to stage demo files in a temp directory. local_fs = fsspec.filesystem("local") local_dir = tempfile.mkdtemp() local_fs.makedirs(f"{local_dir}/notes", exist_ok=True) with local_fs.open(f"{local_dir}/scores.csv", "w") as f: f.write("name,score\nAlice,95\nBob,87\n") with local_fs.open(f"{local_dir}/notes/agenda.txt", "w") as f: f.write("Q3 planning agenda") with local_fs.open(f"{local_dir}/notes/actions.txt", "w") as f: f.write("1. Review roadmap\n2. Confirm budget\n") # Upload a single file fs.put_file(f"{local_dir}/scores.csv", f"dr://{catalog_id}/data/scores.csv") # Upload a directory recursively. Trailing slashes mark both paths as directories. fs.put(f"{local_dir}/notes/", f"dr://{catalog_id}/notes/", recursive=True) ``` ### Upload files from a URL Use [put_from_url](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.put_from_url) to ingest a file directly from any URL the DataRobot server can reach. The file is streamed server-side, so there is no need to download it locally first. ``` # Ingest from url and create file dr:///external/iris.csv fs.put_from_url( path=f"dr://{catalog_id}/external/", url="https://s3.amazonaws.com/datarobot_public_datasets/iris.csv", ) ``` By default, `put_from_url` blocks until the upload completes. To start the upload and return immediately, pass `wait_for_completion=False`, or use `upload_timeout` to control how long to wait when blocking. ### Upload files from a data source To bring files in from a connector-backed system (S3, SharePoint, Google Drive, Confluence, and others), use [put_from_data_source](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.put_from_data_source). This requires a [DataSource](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataSource) configured against an unstructured [DataStore](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore), plus a [Credential](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#datarobot.models.Credential) that can access it. The example below configures an S3 bucket [DataSource](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataSource) and copies a folder of documents into the catalog item. The same pattern can be applied for other source systems. See [put_from_data_source](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.put_from_data_source) for SharePoint and Google Drive variants. ``` credential = dr.Credential.create_s3( name="S3 Credential", aws_access_key_id="", aws_secret_access_key="", ) s3_connector = next(c for c in dr.Connector.list() if c.connector_type == "s3") s3_data_store = dr.DataStore.create( data_store_type=dr.enums.DataStoreTypes.DR_CONNECTOR_V1, canonical_name="My S3 Bucket", fields=[ {"id": "fs.defaultFS", "name": "Bucket Name", "value": "my-bucket-name"}, {"id": "fs.rootDirectory", "name": "Prefix", "value": "/"}, {"id": "fs.s3.awsRegion", "name": "S3 Bucket Region", "value": "us-east-1"}, ], connector_id=s3_connector.id, ) s3_data_source = dr.DataSource.create( data_source_type=dr.enums.DataStoreTypes.DR_CONNECTOR_V1, canonical_name="S3 Documents", params=dr.DataSourceParameters( data_store_id=s3_data_store.id, path="documents/", ), ) fs.put_from_data_source( path=f"dr://{catalog_id}/s3_documents/", data_source_id=s3_data_source.id, credential_id=credential.credential_id, ) ``` By default, `put_from_data_source` blocks until the upload completes. To start the upload and return immediately, pass `wait_for_completion=False` or use `upload_timeout` to control how long to wait when blocking. ## Browse and search files The file system supports the standard [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) discovery methods: - ls : Shallow listing of directory contents. - find : Recursively look through all files (optionally including directories). - walk : Generator that yields directory trees one level at a time (similar to Python's os.walk ). - glob : Match files or directories by pattern. - tree : Visualize the directory tree structure. ``` # List the immediate contents of the catalog item. Set detail=False for just the paths. fs.ls(f"dr://{catalog_id}/", detail=False) # Use detail=True (the default) to also retrieve size, type, and format. for item in fs.ls(f"dr://{catalog_id}/", detail=True): print(f"{item['name']:50s} type={item['type']:10s} size={item['size']}") # Recursively list every file. Pass withdirs=True to include directories. all_files = fs.find(f"dr://{catalog_id}/") # Walk the directory tree one level at a time, similar to os.walk(). for dirpath, dirnames, filenames in fs.walk(f"dr://{catalog_id}/"): print((dirpath, dirnames, filenames)) # Glob lets you match by pattern. Supports *, **, ?, and [abc] character classes. csv_files = fs.glob(f"dr://{catalog_id}/**/*.csv") # Visualize the catalog item layout. The recursion_limit controls how deep to walk. print(fs.tree(f"dr://{catalog_id}/", recursion_limit=3)) ``` > [!TIP] Tip > Patterns ending with `/` will only match directories. For example, `dr://{catalog_id}/*/` returns the top-level subdirectories of the catalog item. ## Manipulate files The DataRobot file system supports methods to copy, move, and delete files. All three methods accept single paths, lists of paths, and glob patterns, and support recursive operations on directories. ### Copy files Use [copy](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.copy) to duplicate files or directories to a new path. Pass `recursive=True` to copy a directory and all of its contents, and use glob patterns to copy multiple files at once. Pass `overwrite_strategy` to specify how to handle naming collisions at the destination. Copying between catalog items is also supported, provided the user has permissions to the source and destination catalog items. ``` from datarobot.enums import FilesOverwriteStrategy # Copy a single file. fs.copy( f"dr://{catalog_id}/data/scores.csv", f"dr://{catalog_id}/backups/scores_backup.csv", ) # Copy a directory recursively, skipping any files that already exist in the destination. # Both paths end with / to mark them as directories. fs.copy( f"dr://{catalog_id}/notes/", f"dr://{catalog_id}/archive/notes_snapshot/", recursive=True, overwrite_strategy=FilesOverwriteStrategy.SKIP ) # Use a glob pattern to copy all .txt files into a single folder. fs.copy( f"dr://{catalog_id}/**/*.txt", f"dr://{catalog_id}/all_text_files/", recursive=True, ) ``` ### Move and rename files Use [mv](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.mv) to move a file to a new path or rename it. Moving a file between catalog items is supported, provided the user has permissions to both the source and destination catalog items. ``` # Rename a file by moving it to a new path within the same catalog item. fs.mv(f"dr://{catalog_id}/backups/scores_backup.csv", f"dr://{catalog_id}/backups/scores_v1.csv") # Move a file into a different directory (note the trailing slash on the target). fs.mv(f"dr://{catalog_id}/backups/scores_v1.csv", f"dr://{catalog_id}/archive/") ``` ### Delete files Use [rm](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.rm) to delete files and directories. Pass `recursive=True` to delete a directory and all of its contents. Use glob patterns to delete multiple files at once. ``` # Delete a single file. fs.rm(f"dr://{catalog_id}/archive/scores_v1.csv") # Delete a directory recursively. fs.rm(f"dr://{catalog_id}/all_text_files/", recursive=True) # Delete every csv file under archive. fs.rm(f"dr://{catalog_id}/archive/**/*.csv", recursive=True) ``` > [!NOTE] Deleting a directory or deleting a catalog item > Deleting all files inside a directory automatically removes the directory because the DataRobot file system does not support empty directories. However, catalog items are different. Deleting all files inside a catalog item does not delete the catalog item. > > To delete the catalog item itself, call `fs.rm` on the catalog item root (for example: `fs.rm(f"dr://{catalog_id}/")`). This soft-deletes the catalog item. A soft-deleted catalog item is hidden but can be restored with `Files.un_delete()` if you change you want to restore it. ## Read files To read files, use: - open for streaming and standard file-like access. - cat or cat_file for one-shot reads. - sign to generate a temporary signed URL. - get to download files locally. ### Stream a file with open [open](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.open) returns a [DataRobotFile](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFile) that behaves like a standard Python file object. This is the most flexible way to read large files, supporting iteration line-by-line, seek to a position, or read fixed-size chunks. ``` # Iterate line by line. Never loads the full file into memory. with fs.open(f"dr://{catalog_id}/data/scores.csv", mode="r") as f: for line in f: print(line.rstrip()) # Read in binary mode with seeking. with fs.open(f"dr://{catalog_id}/data/scores.csv", mode="rb") as f: header = f.read(20) f.seek(0) full = f.read() ``` ### Read files with cat [cat](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.cat) returns the file contents in a single call. Pass a single path to get back bytes, or a glob/list of paths to get back a `{path: bytes}` dictionary. ``` # Read a single file as bytes. data = fs.cat(f"dr://{catalog_id}/data/scores.csv") # Read every CSV file in the catalog item at once. all_csvs = fs.cat(f"dr://{catalog_id}/**/*.csv", recursive=True) for path, content in all_csvs.items(): print(f"{path}: {len(content)} bytes") ``` ### Generate a signed URL A signed URL gives a third-party tool (a browser, a downstream service, a notebook user) temporary read access to a file without sharing your DataRobot API token. Use a signed URL to download a file from the DataRobot file system. ``` import requests url = fs.sign(f"dr://{catalog_id}/data/scores.csv", expiration=300) # Download file locally using signed url local_path = "scores.csv" with requests.get(url, stream=True) as r: r.raise_for_status() with open(local_path, "wb") as f: for chunk in r.iter_content(chunk_size=8192): if chunk: f.write(chunk) ``` ### Download files locally To download a file to your local machine, use [get_file](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.get_file) for a single file, or [get](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.get) for multiple files or an entire directory. ``` import tempfile local_dir = tempfile.mkdtemp() # Download a single file to a local path. fs.get_file(f"dr://{catalog_id}/data/scores.csv", f"{local_dir}/scores.csv") # Download a directory recursively. Trailing slashes mark both paths as directories. fs.get(f"dr://{catalog_id}/notes/", f"{local_dir}/notes/", recursive=True) # Use a glob pattern to download all .csv files into a single local directory. fs.get(f"dr://{catalog_id}/**/*.csv", f"{local_dir}/all_csvs/", recursive=True) ``` ## Inspect files and directories - Use info to get detailed metadata for a single file or directory. - Use exists to check if a file or directory exists. - Use isfile to determine if the path refers to a file. - Use isdir to determine if the path refers to a directory. - Use du to check disk usage for files and directories. ``` # Detailed metadata for a single file or directory. file_info = fs.info(f"dr://{catalog_id}/data/scores.csv") # Quick existence checks. print("exists?", fs.exists(f"dr://{catalog_id}/data/scores.csv")) print("isfile?", fs.isfile(f"dr://{catalog_id}/data/scores.csv")) print("isdir?", fs.isdir(f"dr://{catalog_id}/data/")) # Total disk usage for the entire catalog item. print(f"Total bytes: {fs.du(f'dr://{catalog_id}/', total=True):,}") ``` ## Dict-like access with get_mapper [get_mapper](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFileSystem.get_mapper) returns an instance of [DataRobotFSMap](https://docs.datarobot.com/en/docs/api/reference/sdk/file-system.html#datarobot.fs.file_system.DataRobotFSMap), a [MutableMapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping) rooted at the given path. This is useful when working with libraries that accept fsspec-style mappers (for example, Zarr, Xarray, and other array stores), or when you simply prefer dictionary semantics for reading and writing files. ``` mapper = fs.get_mapper(f"dr://{catalog_id}/data/") # List files in mapping print("Keys:", list(mapper)) # Read bytes from a file print("scores.csv first 30 bytes:", mapper["scores.csv"][:30]) # Write to create a new file mapper["generated.txt"] = b"This file was created via the mapper interface." # Membership checks and size. print("'scores.csv' in mapper?", "scores.csv" in mapper) print("Total files:", len(mapper)) # Delete file del mapper["generated.txt"] ``` --- # Data URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/data/index.html Data integrity and quality are cornerstones for creating highly accurate predictive models. These sections describe the tools and visualizations provided to ensure that your project doesn’t suffer the “garbage in, garbage out” outcome. | Resource | Description | | --- | --- | | Create and manage datasets | Ingest, transform, and store your data for experimentation. | | Build data connections | Integrate with a variety of enterprise databases. | | Recipes | Clean and wrangle data with reusable recipes for data preparation. | | Features | How to work with features and retrieve their statistics in your projects. | | Feature Discovery | Deploy, monitor, manage, and govern all your models in production, regardless of how they were created or when and where they were deployed. | | File registry | Ingest, organize, read, and write files using a simple file system connected to downstream workflows. | --- # Agentic memory URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html The agentic memory SDK persists chat-style history for agentic applications. It exposes three types: a [MemorySpace](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html#memory-space) that groups related conversations, a [Session](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html#sessions) that represents a single conversation, and an [Event](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html#events) that records a message within a session. > [!NOTE] Premium > DataRobot's Agentic AI capabilities are a premium feature; contact your DataRobot representative for enablement information. ## Memory terminology The agentic memory SDK uses the following terminology: - MemorySpace : A container for sessions and their events. A memory space provides an isolation boundary for chat-style sessions and the events recorded in them. - Session : A single chat conversation within a memory space. Each session tracks its participants, optional metadata, and the ordered stream of events. - Event : A single message within a session. Each event is one message on the conversation stack—for example, a user message, an agent response, or a tool result. Events are ordered by sequence_id within their session. - Lifecycle strategy : A server-side rule that governs how long a session and its events are retained. If you do not provide one when creating a session, the server attaches a default strategy. The recommended workflow is: 1. Create a datarobot.models.memory.MemorySpace to hold related conversations. 2. Create one or more datarobot.models.memory.Session objects in that memory space, one per conversation. 3. Append datarobot.models.memory.Event records to a session as the conversation progresses, and update events when they need to be revised. ## Memory space A memory space is the top-level container. It holds sessions and is scoped to the calling user and tenant. ### Create and modify a memory space Use [MemorySpace.create](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html#memory-space) to create a memory space, then call `update` on the returned object to change its description or associated LLM model name. Pass `None` to clear an optional field. ``` >>> from datarobot.models.memory import MemorySpace >>> >>> # Create a new memory space >>> memory_space = MemorySpace.create( ... description="Customer support assistant memory", ... llm_model_name="gpt-4o", ... ) >>> memory_space.id 'deadbeef-cafe-babe-feed-cafebabe0000' >>> >>> # Update the description; leave llm_model_name unchanged >>> memory_space.update(description="Customer support assistant -- production") >>> memory_space.description 'Customer support assistant -- production' >>> >>> # Clear the LLM model name by passing None >>> memory_space.update(llm_model_name=None) >>> memory_space.llm_model_name is None True ``` ### Retrieve memory spaces You can list memory spaces accessible to the current user, or fetch a specific one by ID. ``` >>> from datarobot.models.memory import MemorySpace >>> >>> # Returns one server-paginated page; use offset and limit to walk further pages >>> memory_spaces = MemorySpace.list() >>> next_page = MemorySpace.list(offset=len(memory_spaces), limit=20) >>> >>> # Fetch a specific memory space by ID >>> memory_space = MemorySpace.get("deadbeef-cafe-babe-feed-cafebabe0000") ``` ### Delete a memory space Deleting a memory space removes all its sessions and events. The operation is not reversible. ``` >>> memory_space.delete() ``` ## Sessions and events A session represents a single conversation. Events are appended to a session in order; each event has a server-assigned `sequence_id` that reflects its position within the session. ### Create a session and append events Create a [Session](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html#sessions) with [Session.create](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html#sessions), passing the parent `memory_space_id` and the `participants` list. Use [Session.post_event](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html#events) to append events as the conversation progresses. The `emitter` dictionary identifies the entity that produced the event -- typically `{"type": "user", "id": }` for a human turn and `{"type": "agent", "id": }` for an agent reply. > [!NOTE] One participant per session > Multi-participant sessions are not supported. Pass a single-element list to `participants`; the server rejects sessions with more than one participant. ``` >>> from datarobot.models.memory import Session >>> >>> participant_id = "ba5eba11deadbeefcafebabe" >>> agent_id = "agent-001" >>> >>> # Create a session in the memory space >>> session = Session.create( ... memory_space_id=memory_space.id, ... participants=[participant_id], ... description="Order status inquiry", ... metadata={"channel": "web", "priority": "high"}, ... ) >>> >>> # Append a user message >>> user_event = session.post_event( ... body={"content": "Can you check the status of my order?"}, ... emitter={"type": "user", "id": participant_id}, ... event_type="message", ... ) >>> user_event.sequence_id 0 >>> >>> # Append an agent reply >>> agent_event = session.post_event( ... body={"content": "Let me check that for you."}, ... emitter={"type": "agent", "id": agent_id}, ... event_type="message", ... ) >>> agent_event.sequence_id 1 ``` ### Update an event Use [Session.update_event](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html#events) to revise an event's body, type, or emitter. Identify the target event by its `sequence_id`. To guard against concurrent writes, pass the event's `created_at` timestamp; the server rejects the update if the event has been modified since that timestamp, and the caller must reload the event before retrying. ``` >>> # The agent's answer needs to be replaced after fetching real data >>> updated = session.update_event( ... sequence_id=agent_event.sequence_id, ... body={"content": "Your order has shipped and is scheduled to arrive within two business days."}, ... created_at=agent_event.created_at, # Optimistic concurrency check ... ) >>> updated.body {'content': 'Your order has shipped and is scheduled to arrive within two business days.'} ``` ### List events Read events back with [Session.events](https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/agentic-memory.html#events). Use `last_n` to fetch the most recent events, or `offset` and `limit` to page from the beginning. The two are mutually exclusive. Filter by `event_type` to retrieve only a specific kind of event. ``` >>> # Fetch the most recent 50 message events >>> recent = session.events(last_n=50, event_type="message") >>> >>> # Read the first 100 events from the start of the session >>> first_page = session.events(offset=0, limit=100) >>> >>> # Walk subsequent pages with offset >>> next_page = session.events(offset=len(first_page), limit=100) ``` ### Update or delete a session Update a session's description or metadata with `Session.update`. Pass `None` to clear a field. Delete a session and its events with `Session.delete`. ``` >>> session.update(metadata={"channel": "web", "priority": "low"}) >>> session.delete() ``` --- # Chats prompting URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/chats-prompting.html [Chats](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-prompting.html#datarobot.models.genai.chat.Chat) provide a way to interact with LLMs through prompts, maintaining conversation history and context. Chats allow you to have multi-turn conversations with your LLM applications, where each prompt can reference previous messages in the conversation. ## Create a chat Create a new chat from an LLM blueprint. When creating a chat, you should specify the following: - name : A user-friendly name for the chat. - llm_blueprint : The LLM blueprint ID or LLMBlueprint object to associate the chat with. ``` import datarobot as dr blueprint = dr.genai.LLMBlueprint.get(blueprint_id) chat = dr.genai.Chat.create( name="Customer Support Chat", llm_blueprint=blueprint_id ) chat ``` ## Submit prompts to a chat Send prompts and receive responses using [datarobot.ChatPrompt.submit()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-prompting.html#datarobot.models.genai.chat_prompt.ChatPrompt.submit): ``` chat = dr.genai.Chat.get(chat_id) prompt = dr.genai.ChatPrompt.create( chat=chat.id, text="What is your return policy?" ) prompt.result_text ``` Submit a follow-up prompt that maintains the conversation history: ``` followup = dr.genai.ChatPrompt.create( chat=chat.id, text="How long does it take to process a return?" ) print(f"Response: {followup.result_text}") ``` ## Retrieve chat history Get all prompts in a chat using [datarobot.ChatPrompt.list()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-prompting.html#datarobot.models.genai.chat_prompt.ChatPrompt.list): ``` chat = dr.genai.Chat.get(chat_id) prompts = dr.genai.ChatPrompt.list(chat=chat.id) for prompt in prompts: prompt.text prompt.result_text ``` Filter prompts by LLM blueprint: ``` blueprint = dr.genai.LLMBlueprint.get(blueprint_id) blueprint_prompts = dr.genai.ChatPrompt.list(llm_blueprint=blueprint.id) ``` Filter prompts by playground: ``` playground = dr.genai.Playground.get(playground_id) playground_prompts = dr.genai.ChatPrompt.list(playground=playground.id) ``` ## Manage chats List all chats: ``` all_chats = dr.genai.Chat.list() print(f"Found {len(all_chats)} chat(s):") for chat in all_chats: print(f" - {chat.name} (ID: {chat.id})") ``` Filter by LLM blueprint: ``` blueprint = dr.genai.LLMBlueprint.get(blueprint_id) blueprint_chats = dr.genai.Chat.list(llm_blueprint=blueprint.id) ``` Update the chat name: ``` chat = dr.genai.Chat.get(chat_id) chat.update(name="Updated Chat Name") ``` Delete a chat: ``` chat.delete() ``` ## Get chat information Retrieve chat details: ``` chat = dr.genai.Chat.get(chat_id) print(f"Name: {chat.name}") print(f"LLM Blueprint: {chat.llm_blueprint_id}") print(f"Is frozen: {chat.is_frozen}") print(f"Prompts count: {chat.prompts_count}") print(f"Warning: {chat.warning}") ``` --- # Custom model validation URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/custom-model-validation.html If you have a custom model deployment that serves as an LLM, you need to validate it before using it in LLM blueprints. Validation ensures the deployment can properly handle LLM requests. Use [datarobot.CustomModelLLMValidation](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.custom_model_llm_validation.CustomModelLLMValidation) to validate your deployments. ## Start LLM validation Validate a deployment as a custom model LLM. Specify the following: - deployment_id : The ID of the deployment to validate. - name : An optional name for the validation. - prompt_column_name : The name of the column the deployed model uses for prompt text input (required for non-chat API deployments). - target_column_name : The name of the column the deployed model uses for prediction output (required for non-chat API deployments). - chat_model_id : The model ID to specify when calling the chat completion API (for deployments that support the chat completion API). - wait_for_completion : If set to True, the code will wait for the validation job to complete before returning results. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id) use_case = dr.UseCase.get(use_case_id) validation = dr.genai.CustomModelLLMValidation.create( deployment_id=deployment.id, use_case=use_case.id, name="My Custom LLM", prompt_column_name="prompt", target_column_name="response", wait_for_completion=True ) validation ``` For deployments that support the chat completion API, use `chat_model_id`: ``` validation = dr.genai.CustomModelLLMValidation.create( deployment_id=deployment.id, name="Chat API LLM", chat_model_id="model-id-from-deployment", wait_for_completion=True ) ``` ## Check validation status If you don't wait for completion, poll for status using [datarobot.CustomModelLLMValidation.get()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.custom_model_llm_validation.CustomModelLLMValidation.get): ``` validation = dr.genai.CustomModelLLMValidation.create( deployment_id=deployment.id, name="My Custom LLM", prompt_column_name="prompt", target_column_name="response", wait_for_completion=False ) while validation.validation_status == "TESTING": import time time.sleep(5) validation = dr.genai.CustomModelLLMValidation.get(validation.id) if validation.validation_status == "PASSED": print("Validation passed!") print(f"Access data: {validation.deployment_access_data}") else: print(f"Validation failed: {validation.error_message}") ``` ## Use a validated LLM in a blueprint Once validation passes, use the validation ID in an LLM blueprint: ``` validation = dr.genai.CustomModelLLMValidation.get(validation_id) use_case = dr.UseCase.get(use_case_id) playground = dr.genai.Playground.create( name="Custom LLM Playground", use_case=use_case.id ) blueprint = dr.genai.LLMBlueprint.create( playground=playground.id, name="Custom Model Blueprint", llm="custom-model", llm_settings={ "system_prompt": "You are a helpful assistant.", "validation_id": validation.id } ) blueprint ``` ## Update validation settings Modify validation parameters using [datarobot.CustomModelLLMValidation.update()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.custom_model_llm_validation.CustomModelLLMValidation.update): ``` validation = dr.genai.CustomModelLLMValidation.get(validation_id) validation.update( name="Updated Custom LLM Name", prediction_timeout=60 ) ``` --- # Generative AI URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/index.html How to build, validate, and deploy generative AI applications using LLMs, vector databases, and moderation tools. This section covers creating LLM blueprints, managing chats and prompts, setting up vector databases for RAG (Retrieval-Augmented Generation), and implementing safety measures through moderation. | Topic | Description | | --- | --- | | LLM blueprints | Create and manage LLM blueprints that define how large language models are configured and used in your applications. | | Validate custom model LLMs | Validate deployments as custom model LLMs before using them in LLM blueprints. | | Vector databases | Set up and manage vector databases for RAG (Retrieval-Augmented Generation) workflows. | | Chats and prompting | Manage chat sessions and interact with LLMs through prompts, maintaining conversation history. | | End-to-end RAG application workflow | A complete example of building a RAG application from scratch. | | Agentic memory | Persist chat history for agentic applications using memory spaces, sessions, and events. | --- # LLM blueprints URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/llm-blueprints.html [LLM blueprints](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/build-llm-blueprints.html#working-with-llm-blueprints) define how large language models are configured and used in your generative AI applications. You can use pre-configured LLMs provided by DataRobot or create custom model LLMs from your own deployments. LLM blueprints allow you to configure system prompts, temperature settings, and other parameters that control how the LLM behaves in your applications. ## List available LLMs To see all available LLMs, use [datarobot.LLMDefinition.list()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.llm.LLMDefinition.list): ``` import datarobot as dr llms = dr.genai.LLMDefinition.list() for llm in llms: print(f"{llm.name}: {llm.description}") print(f" Vendor: {llm.vendor}") print(f" Context size: {llm.context_size}") print(f" Active: {llm.is_active}") ``` You can also filter LLMs by Use Case: ``` use_case = dr.UseCase.get(use_case_id) llms = dr.genai.LLMDefinition.list(use_case=use_case) ``` ## Create a playground Playgrounds are required to create LLM blueprints. A playground is a workspace where you can experiment with different LLM configurations. When creating a playground, you should specify the following: - name : A user-friendly name for the playground. - description : An optional description of the playground's purpose. - use_case : A Use Case to link the playground to. - playground_type : The type of playground, defaults to dr.enums.PlaygroundType.RAG . ``` import datarobot as dr playground = dr.genai.Playground.create( name="My GenAI Playground", use_case=use_case.id ) playground ``` You can also create a playground in a Use Case: ``` use_case = dr.UseCase.get(use_case_id) playground = dr.genai.Playground.create( name="Use Case Playground", use_case=use_case.id ) ``` ## Create an LLM blueprint Create a new LLM blueprint with custom settings. When creating an LLM blueprint, you should specify the following: - playground : The playground ID or Playground object to associate the blueprint with. - name : A user-friendly name for the LLM blueprint. - llm : The LLM definition ID or LLMDefinition object to use. - llm_settings : A dictionary containing LLM configuration settings such as system prompts, temperature, and max completion length. - description : An optional description of the blueprint. - prompt_type : The prompting strategy, defaults to dr.enums.PromptType.CHAT_HISTORY_AWARE . ``` import datarobot as dr llms = dr.genai.LLMDefinition.list() gpt4 = [llm for llm in llms if 'gpt-4' in llm.name.lower()][0] playground = dr.genai.Playground.create(name="Customer Support Playground") blueprint = dr.genai.LLMBlueprint.create( playground=playground.id, name="Customer Support Assistant", description="LLM for customer support queries", llm=gpt4.id, llm_settings={ "system_prompt": "You are a helpful customer support assistant. Be concise and professional.", "temperature": 0.7, "max_completion_length": 500 } ) blueprint ``` ## Retrieve and list LLM blueprints Get a specific blueprint or list all available blueprints using [datarobot.LLMBlueprint.get()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.llm_blueprint.LLMBlueprint.get) and [datarobot.LLMBlueprint.list()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.llm_blueprint.LLMBlueprint.list). To retrieve a specific blueprint: ``` blueprint = dr.genai.LLMBlueprint.get(blueprint_id) blueprint ``` To list all blueprints: ``` all_blueprints = dr.genai.LLMBlueprint.list() print(f"Found {len(all_blueprints)} LLM blueprint(s):") for bp in all_blueprints: print(f" - {bp.name} (ID: {bp.id})") ``` To filter blueprints by playground: ``` playground_blueprints = dr.genai.LLMBlueprint.list(playground=playground.id) ``` To filter by LLM type: ``` gpt_blueprints = dr.genai.LLMBlueprint.list(llms=[gpt4.id]) ``` ## Update an LLM blueprint Modify blueprint settings using [datarobot.LLMBlueprint.update()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.llm_blueprint.LLMBlueprint.update): ``` blueprint = dr.genai.LLMBlueprint.get(blueprint_id) blueprint.update( name="Updated Customer Support Assistant", llm_settings={ "system_prompt": "You are an expert customer support assistant.", "temperature": 0.5 } ) ``` Save the blueprint to lock settings: ``` blueprint.update(is_saved=True) ``` Star the blueprint for easy access: ``` blueprint.update(is_starred=True) ``` ## Create a blueprint from an existing blueprint Create a copy of an existing blueprint to experiment with variations using [datarobot.LLMBlueprint.create_from_llm_blueprint()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-llm-generation.html#datarobot.models.genai.llm_blueprint.LLMBlueprint.create_from_llm_blueprint): ``` original_blueprint = dr.genai.LLMBlueprint.get(blueprint_id) new_blueprint = dr.genai.LLMBlueprint.create_from_llm_blueprint( llm_blueprint=original_blueprint, name="Experimental Variant", description="Testing different temperature settings" ) new_blueprint ``` --- # Generative AI moderation URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/moderation.html > Use the DataRobot Python client to list moderation templates, create configurations, and filter unsafe generative AI prompts and responses. [Moderation configurations](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-moderation.html#datarobot.models.moderation.configuration.ModerationConfiguration) help ensure your generative AI applications produce safe and appropriate content by filtering prompts and responses. Moderation can block or warn on problematic content at different stages of the generation process. ## List moderation templates To retrieve all available moderation templates: ``` import datarobot as dr templates = dr.ModerationTemplate.list() for template in templates: print(f"Template: {template.name}") print(f" Description: {template.description}") ``` ## Create a moderation configuration Create a moderation configuration from a template. When creating a moderation configuration, you should specify the following: - template_id : The ID of the template to base this configuration on. - name : A user-friendly name for the configuration. - description : A description of the configuration. - stages : The stages of moderation where this guard is active (e.g., PROMPT, RESPONSE). - entity_id : The ID of the custom model version or playground this configuration applies to. - entity_type : The type of the associated entity (CUSTOM_MODEL_VERSION or PLAYGROUND). - intervention : The action to take if moderation fails (BLOCK or WARN). - llm_type : The backing LLM this guard uses. ``` templates = dr.ModerationTemplate.list() template = templates[0] custom_model_version = dr.CustomModelVersion.get(version_id) moderation_config = dr.ModerationConfiguration.create( template_id=template.id, name="Content Safety Guard", description="Filters inappropriate content", stages=[dr.ModerationGuardStage.PROMPT, dr.ModerationGuardStage.RESPONSE], entity_id=custom_model_version.id, entity_type=dr.ModerationGuardEntityType.CUSTOM_MODEL_VERSION, intervention=dr.ModerationIntervention.BLOCK, llm_type=dr.ModerationGuardLlmType.DATAROBOT_LLM ) moderation_config ``` You can also create moderation for playgrounds: ``` playground = dr.genai.Playground.get(playground_id) moderation_config = dr.ModerationConfiguration.create( template_id=template.id, name="Playground Safety Guard", description="Filters content in playground", stages=[dr.ModerationGuardStage.PROMPT, dr.ModerationGuardStage.RESPONSE], entity_id=playground.id, entity_type=dr.ModerationGuardEntityType.PLAYGROUND, intervention=dr.ModerationIntervention.WARN, llm_type=dr.ModerationGuardLlmType.DATAROBOT_LLM ) ``` ## List moderation configurations To retrieve configurations for an entity: ``` custom_model_version = dr.CustomModelVersion.get(version_id) configs = dr.ModerationConfiguration.list( entity_id=custom_model_version.id, entity_type=dr.ModerationGuardEntityType.CUSTOM_MODEL_VERSION ) for config in configs: print(f"Config: {config.name}") print(f" Stages: {config.stages}") print(f" Intervention: {config.intervention}") ``` ## Get a moderation configuration To retrieve a specific configuration: ``` config = dr.ModerationConfiguration.get(config_id) print(f"Name: {config.name}") print(f"Description: {config.description}") print(f"Stages: {config.stages}") ``` ## Update moderation configuration To update moderation settings: ``` config = dr.ModerationConfiguration.get(config_id) config.update( name="Updated Safety Guard", description="Enhanced content filtering", intervention=dr.ModerationIntervention.WARN ) ``` ## Get the overall moderation configuration Retrieve the overall moderation configuration for an entity: ``` custom_model_version = dr.CustomModelVersion.get(version_id) overall_config = dr.OverallModerationConfig.get( entity_id=custom_model_version.id, entity_type=dr.ModerationGuardEntityType.CUSTOM_MODEL_VERSION ) if overall_config: print(f"Moderation enabled: {overall_config.is_enabled}") print(f"Configurations: {len(overall_config.configurations)}") ``` ## List the overall moderation configurations To get all of the overall moderation configurations: ``` overall_configs = dr.OverallModerationConfig.list() for config in overall_configs: print(f"Entity: {config.entity_id}") print(f" Enabled: {config.is_enabled}") ``` ## Update the overall moderation configuration To modify the overall moderation settings: ``` overall_config = dr.OverallModerationConfig.get( entity_id=custom_model_version.id, entity_type=dr.ModerationGuardEntityType.CUSTOM_MODEL_VERSION ) overall_config.update(is_enabled=True) ``` --- # End-to-end RAG application workflow URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/rag-workflow.html > Build a production-style RAG application with the Python client—playgrounds, LLMs, vector databases, moderation, and testing in one workflow. This example demonstrates how to build a complete RAG (Retrieval-Augmented Generation) application, combining LLM blueprints, vector databases, moderation, and testing. This workflow shows how to integrate all the components of a production-ready generative AI application. ### Create a playground A playground is a workspace for experimenting with LLM configurations. All LLM blueprints must be associated with a playground. ``` import datarobot as dr playground = dr.genai.Playground.create( name="RAG Application Playground", use_case=use_case.id ) ``` ### Select an LLM Choose from available LLMs provided by DataRobot. You can filter by use case or select based on model capabilities. ``` llms = dr.genai.LLMDefinition.list() llm = [l for l in llms if 'gpt-4' in l.name.lower()][0] ``` ### Build a vector database Upload your documents as a dataset, then create a vector database with appropriate chunking parameters. The vector database stores document embeddings for RAG. ``` dataset = dr.Dataset.upload("company_docs.csv") supported = dr.genai.VectorDatabase.get_supported_embeddings(dataset_id=dataset.id) from datarobot.models.genai.vector_database import ChunkingParameters chunking_params = ChunkingParameters( embedding_model=supported.default_embedding_model, chunking_method="semantic", chunk_size=500, chunk_overlap_percentage=10 ) use_case = dr.UseCase.get(use_case_id) vector_db = dr.genai.VectorDatabase.create( dataset_id=dataset.id, use_case=use_case.id, name="Company Knowledge Base", chunking_parameters=chunking_params ) ``` ### Create an LLM blueprint Combine the LLM with the vector database to create a RAG-enabled blueprint. Configure system prompts and other LLM settings. ``` blueprint = dr.genai.LLMBlueprint.create( playground=playground.id, name="RAG Customer Support", description="RAG-enabled customer support assistant", llm=llm.id, llm_settings={ "system_prompt": "You are a customer support assistant. Use the provided context to answer questions accurately.", "temperature": 0.7 }, vector_database=vector_db.id, vector_database_settings={ "max_documents_retrieved_per_prompt": 3 } ) ``` ### Register custom model Register the blueprint as a custom model version so it can be deployed and used in production. ``` custom_model_version = blueprint.register_custom_model() ``` ### Add moderation Set up content filtering to ensure safe and appropriate responses. Moderation can block or warn on problematic content. ``` template = dr.ModerationTemplate.list()[0] moderation = dr.ModerationConfiguration.create( template_id=template.id, name="Content Safety", description="Filter inappropriate content", stages=[dr.ModerationGuardStage.PROMPT, dr.ModerationGuardStage.RESPONSE], entity_id=custom_model_version.id, entity_type=dr.ModerationGuardEntityType.CUSTOM_MODEL_VERSION, intervention=dr.ModerationIntervention.BLOCK ) ``` ### Interact via chats Create chat sessions to interact with your RAG application. Chats maintain conversation history for context-aware responses. ``` chat = dr.genai.Chat.create( name="Customer Support Session", llm_blueprint=blueprint.id ) prompt = dr.genai.ChatPrompt.create( chat=chat.id, text="What is your return policy?" ) prompt.text prompt.result_text ``` --- # AI robustness testing URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/robustness-testing.html > Configure generative AI robustness insights and tests with the Python client to measure quality, toxicity, relevance, and similar metrics on prompts and responses. [AI robustness tests](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-testing.html#datarobot.models.genai.insights_configuration.InsightsConfiguration) help validate that your generative AI models perform well under various conditions and meet quality standards. Robustness testing allows you to configure insights that measure different aspects of model performance, such as toxicity, relevance, and coherence. ## Configure insights for robustness testing Set up insights to test AI model robustness. When creating an insight configuration, you should specify the following: - custom_model_version_id : The ID of the custom model version to test. - insight_name : A user-friendly name for the insight. - insight_type : The type of insight (e.g., OOTB_METRIC ). - ootb_metric_name : The name of the DataRobot-provided metric (for the OOTB_METRIC type). - stage : The stage when the metric is calculated ( PROMPT or RESPONSE ). ``` import datarobot as dr custom_model_version = dr.CustomModelVersion.get(version_id) insight_config = dr.InsightsConfiguration.create( custom_model_version_id=custom_model_version.id, insight_name="Toxicity Check", insight_type=dr.InsightTypes.OOTB_METRIC, ootb_metric_name="Toxicity", stage=dr.InsightStage.RESPONSE ) insight_config ``` ## Set up test configurations To configure test parameters: ``` eval_dataset = dr.EvaluationDatasetConfiguration.create( name="Robustness Test Dataset", dataset_id=dataset.id ) cost_config = dr.CostConfiguration.create( name="Test Cost Config", cost_per_token=0.0001 ) insight_config.update( evaluation_dataset_configuration_id=eval_dataset.id, cost_configuration_id=cost_config.id ) ``` ## Run robustness tests To execute tests and review results: ``` insight_config = dr.InsightsConfiguration.get(insight_config_id) if insight_config.execution_status == "COMPLETED": results = insight_config.get_results() print(f"Test results: {results}") elif insight_config.execution_status == "ERROR": print(f"Test failed: {insight_config.error_message}") print(f"Resolution: {insight_config.error_resolution}") ``` ## Configure multiple insights To set up multiple tests for comprehensive validation: ``` insights = [ { "insight_name": "Toxicity Check", "ootb_metric_name": "Toxicity", "stage": dr.InsightStage.RESPONSE }, { "insight_name": "Relevance Check", "ootb_metric_name": "Relevance", "stage": dr.InsightStage.RESPONSE }, { "insight_name": "Coherence Check", "ootb_metric_name": "Coherence", "stage": dr.InsightStage.RESPONSE } ] for insight_config in insights: dr.InsightsConfiguration.create( custom_model_version_id=custom_model_version.id, **insight_config ) ``` ## Get an insight configuration To retrieve a specific insight configuration: ``` insight_config = dr.InsightsConfiguration.get(insight_config_id) print(f"Insight name: {insight_config.insight_name}") print(f"Insight type: {insight_config.insight_type}") print(f"Execution status: {insight_config.execution_status}") ``` ## List insight configurations Get all insights for a custom model version: ``` custom_model_version = dr.CustomModelVersion.get(version_id) insights = dr.InsightsConfiguration.list(custom_model_version_id=custom_model_version.id) for insight in insights: print(f"{insight.insight_name}: {insight.execution_status}") ``` --- # Vector databases URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/genai/vector-databases.html [Vector databases](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-vector-databases.html#datarobot.models.genai.vector_database.VectorDatabase) enable RAG (Retrieval-Augmented Generation) workflows by storing document embeddings and retrieving relevant context for LLM prompts. Vector databases allow you to create knowledge bases from your documents and use them to provide context-aware responses in your generative AI applications. ## Validate a deployment as a vector database Before using a deployment as a vector database, validate it using [datarobot.CustomModelVectorDatabaseValidation](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-vector-databases.html#datarobot.models.genai.vector_database.CustomModelVectorDatabaseValidation): ``` import datarobot as dr validation = dr.CustomModelVectorDatabaseValidation.create( deployment_id=deployment.id, name="My Vector Database", prompt_column_name="query", target_column_name="citations", wait_for_completion=True ) if validation.validation_status == "PASSED": print("Vector database validation passed!") ``` ## Get supported embeddings View available embedding models using [datarobot.VectorDatabase.get_supported_embeddings()](https://docs.datarobot.com/en/docs/api/reference/sdk/gen-vector-databases.html#datarobot.models.genai.vector_database.VectorDatabase.get_supported_embeddings): ``` supported = dr.genai.VectorDatabase.get_supported_embeddings() print(f"Default embedding model: {supported.default_embedding_model}") for model in supported.embedding_models: print(f" {model.name}: {model.description}") ``` You can also get recommended embeddings for a specific dataset: ``` dataset = dr.Dataset.get(dataset_id) supported = dr.genai.VectorDatabase.get_supported_embeddings(dataset_id=dataset.id) print(f"Default embedding: {supported.default_embedding_model}") ``` ## Get supported text chunking configurations To view available text chunking options: ``` chunking_configs = dr.genai.VectorDatabase.get_supported_text_chunkings() for config in chunking_configs.text_chunking_configs: print(f"Chunking config: {config}") ``` ## Create a vector database Create a vector database from a dataset containing your documents. When creating a vector database, you should specify the following: - dataset_id : The ID of the dataset used for creation. - chunking_parameters : Parameters defining how documents are split and embedded, including embedding model, chunking method, chunk size, and overlap percentage. - name : An optional user-friendly name for the vector database. - use_case : An optional Use Case to link the vector database to. ``` dataset = dr.Dataset.upload("documents.csv") supported = dr.genai.VectorDatabase.get_supported_embeddings(dataset_id=dataset.id) from datarobot.models.genai.vector_database import ChunkingParameters chunking_params = ChunkingParameters( embedding_model=supported.default_embedding_model, chunking_method="semantic", chunk_size=500, chunk_overlap_percentage=10 ) vector_db = dr.genai.VectorDatabase.create( dataset_id=dataset.id, name="Document Knowledge Base", chunking_parameters=chunking_params, use_case=use_case_id ) vector_db ``` ## Update a vector database Add more documents or update an existing vector database: ``` new_dataset = dr.Dataset.upload("updated_documents.csv") updated_vector_db = dr.genai.VectorDatabase.create( dataset_id=new_dataset.id, parent_vector_database_id=vector_db.id, update_llm_blueprints=True ) ``` ## Link vector database to LLM blueprint Associate a vector database with an LLM blueprint for RAG: ``` vector_db = dr.genai.VectorDatabase.get(vector_db_id) blueprint = dr.genai.LLMBlueprint.get(blueprint_id) blueprint.update( vector_database=vector_db.id, vector_database_settings={ "max_documents_retrieved_per_prompt": 3 } ) ``` ## List and manage vector databases List all vector databases: ``` all_dbs = dr.genai.VectorDatabase.list() print(f"Found {len(all_dbs)} vector database(s):") for db in all_dbs: print(f" - {db.name} (ID: {db.id}, Status: {db.execution_status})") ``` Filter vector databases by Use Case: ``` use_case = dr.UseCase.get(use_case_id) use_case_dbs = dr.genai.VectorDatabase.list(use_case=use_case) ``` Get vector database details: ``` vector_db = dr.genai.VectorDatabase.get(vector_db_id) print(f"Name: {vector_db.name}") print(f"Size: {vector_db.size} bytes") print(f"Status: {vector_db.execution_status}") print(f"Chunks count: {vector_db.chunks_count}") ``` Delete a vector database: ``` vector_db.delete() ``` ## Export a vector database as a dataset To export a vector database as a dataset: ``` vector_db = dr.genai.VectorDatabase.get(vector_db_id) export_job = vector_db.submit_export_dataset_job() exported_dataset_id = export_job.dataset_id ``` --- # Python API client user guide URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/index.html > Review outlines and explanations of the methods that comprise the API client. The Python API client user guide outlines and explanations of the methods that comprise the API client. To access previous version of the Python API client documentation, access [ReadTheDocs](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest-release/). | Resource | Description | | --- | --- | | Administration | Manage DataRobot Self-Managed AI Platform installations. | | Data | Ingest, transform, and store your data for experimentation. | | MLOps | Deploy, monitor, manage, and govern all your models in production, regardless of how they were created or when and where they were deployed. | | Modeling | Understand the elements of the basic modeling workflow as well as methods for building additional models in a project. | | Generative AI | Build, validate, and deploy generative AI applications using LLMs, vector databases, and moderation tools. | | Predictions | How to get predictions with new data from a model. | | Use Cases | Group assets that solve a specific business problem. | --- # Agent2Agent protocol URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/a2a.html > Enable interoperability between AI agents in DataRobot. The [Agent2Agent (A2A)](https://google.github.io/A2A/) protocol enables interoperability between AI agents. By registering an external agent in the DataRobot model registry and attaching an agent card, you make the agent available for discovery. This page outlines the end-to-end workflow, summarized below: 1. Register an external model in the Model Registry. 2. Create an external prediction environment. 3. Deploy the registered model version. 4. Upload an agent card to the deployment. After completing these steps, the agent is available for A2A interactions within DataRobot. ## Prerequisites ``` import datarobot as dr # Authenticate with the DataRobot API dr.Client(token="YOUR_API_TOKEN", endpoint="https://app.datarobot.com/api/v2") ``` ## Register an external model Use [RegisteredModelVersion.create_for_external](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.model_registry.RegisteredModelVersion.create_for_external) to create a registered model version for an external agent. The `target` parameter uses the [ExternalTarget](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.model_registry.registered_model_version.ExternalTarget) type dict. DataRobot provides a dedicated target type for agents — set `type` to `"AgenticWorkflow"`. ``` import datarobot as dr registered_model_version = dr.RegisteredModelVersion.create_for_external( name="My A2A Agent v1", target={"name": "target", "type": "AgenticWorkflow"}, registered_model_name="My A2A Agent", ) registered_model_version.id >>> '66a1b2c3d4e5f6a7b8c9d0e1' registered_model_version.registered_model_id >>> '66a1b2c3d4e5f6a7b8c9d0e2' ``` If you already have a registered model and want to add a new version, pass `registered_model_id` instead of `registered_model_name`: ``` new_version = dr.RegisteredModelVersion.create_for_external( name="My A2A Agent v2", target={"name": "target", "type": "AgenticWorkflow"}, registered_model_id=registered_model_version.registered_model_id, ) ``` Refer to the [Model Registry documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/model_registry.html#model-registry) for additional details on managing registered models. ## Create an external prediction environment External agentic workflow models must be deployed to an external prediction environment. Deploying to a regular DataRobot prediction server is not supported and will result in a `422` error. Use [PredictionEnvironment.create](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.PredictionEnvironment.create) to create an external prediction environment. Choose a `platform` value that matches where your agent is hosted (e.g.`"aws"`, `"gcp"`, `"azure"`, or `"other"`). ``` import datarobot as dr from datarobot.enums import PredictionEnvironmentPlatform prediction_environment = dr.PredictionEnvironment.create( name="A2A Agent Environment", platform=PredictionEnvironmentPlatform.OTHER, description="External prediction environment for A2A agents", ) prediction_environment.id >>> '66a0b1c2d3e4f5a6b7c8d9e0' ``` If you already have a suitable external prediction environment, you can reuse it: ``` prediction_environments = dr.PredictionEnvironment.list() prediction_environment = prediction_environments[0] ``` ## Deploy the registered model Use [Deployment.create_from_registered_model_version](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.create_from_registered_model_version) to create a deployment from the registered model version. Pass `prediction_environment_id` (not `default_prediction_server_id`) — this is required for agentic workflow targets. ``` import datarobot as dr deployment = dr.Deployment.create_from_registered_model_version( model_package_id=registered_model_version.id, label="My A2A Agent Deployment", description="External A2A agent deployment", prediction_environment_id=prediction_environment.id, ) deployment.id >>> '66b2c3d4e5f6a7b8c9d0e1f2' ``` Refer to the [Deployments documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/deployment.html#deployments) for more information on deployment management. ## Upload an agent card Use [Deployment.upload_agent_card](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.upload_agent_card) to attach an A2A agent card to the deployment. The agent card is a dictionary describing the agent's capabilities and metadata, following the [A2A Agent Card specification](https://google.github.io/A2A/#/documentation?id=agent-card). ``` agent_card = { "name": "My A2A Agent", "description": "An agent that performs data analysis tasks.", "version": "1.0.0", "url": "https://my-agent.example.com/a2a", "capabilities": { "streaming": True, "pushNotifications": False, }, "skills": [ { "id": "data-analysis", "name": "Data Analysis", "description": "Analyzes datasets and provides statistical summaries.", } ], } uploaded_card = deployment.upload_agent_card(agent_card) ``` After uploading, the agent is available in the DataRobot registry as an `A2A` agent. ### Retrieve an agent card Use [Deployment.get_agent_card](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_agent_card) to fetch the agent card from an existing deployment: ``` agent_card = deployment.get_agent_card() agent_card["name"] >>> 'My A2A Agent' ``` ### Delete an agent card Use [Deployment.delete_agent_card](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.delete_agent_card) to remove the agent card from a deployment. This operation is idempotent — it returns successfully even if no agent card exists. ``` deployment.delete_agent_card() ``` ## Discover A2A agent deployments Use the `is_a2a_agent` filter on [DeploymentListFilters](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.DeploymentListFilters) to list only deployments that have an A2A agent card: ``` import datarobot as dr from datarobot.models.deployment import DeploymentListFilters filters = DeploymentListFilters(is_a2a_agent=True) a2a_deployments = dr.Deployment.list(filters=filters) a2a_deployments >>> [Deployment('My A2A Agent Deployment'), Deployment('Another A2A Agent')] ``` You can then retrieve the agent card from any discovered deployment to learn about its capabilities before initiating an A2A interaction: ``` for dep in a2a_deployments: card = dep.get_agent_card() print(f"{card['name']} (v{card['version']}): {card['description']}") ``` ## Full example The following example demonstrates the complete flow of registering an external A2A agent: ``` import datarobot as dr from datarobot.enums import PredictionEnvironmentPlatform from datarobot.models.deployment import DeploymentListFilters # 1. Connect to DataRobot dr.Client(token="YOUR_API_TOKEN", endpoint="https://app.datarobot.com/api/v2") # 2. Register an external model registered_model_version = dr.RegisteredModelVersion.create_for_external( name="Research Assistant Agent v1", target={"name": "target", "type": "AgenticWorkflow"}, registered_model_name="Research Assistant Agent", ) # 3. Create an external prediction environment prediction_environment = dr.PredictionEnvironment.create( name="A2A Agent Environment", platform=PredictionEnvironmentPlatform.OTHER, description="External prediction environment for A2A agents", ) # 4. Deploy the registered model deployment = dr.Deployment.create_from_registered_model_version( model_package_id=registered_model_version.id, label="Research Assistant Agent", description="An A2A agent that assists with research tasks.", prediction_environment_id=prediction_environment.id, ) # 5. Upload the agent card agent_card = { "name": "Research Assistant", "description": "Assists with literature review, summarization, and citation management.", "version": "1.0.0", "url": "https://research-agent.example.com/a2a", "capabilities": { "streaming": True, "pushNotifications": False, }, "skills": [ { "id": "literature-review", "name": "Literature Review", "description": "Searches and summarizes academic papers on a given topic.", }, { "id": "citation-management", "name": "Citation Management", "description": "Formats and organizes citations in various styles.", }, ], } deployment.upload_agent_card(agent_card) # 6. Verify the agent is available for A2A protocol filters = DeploymentListFilters(is_a2a_agent=True) a2a_deployments = dr.Deployment.list(filters=filters) for dep in a2a_deployments: card = dep.get_agent_card() print(f"{card['name']} (v{card['version']}): {card['description']}") ``` --- # Batch monitoring jobs URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/batch-monitoring.html > Manage batch monitoring job definitions and jobs for batch-enabled deployments with the Python API client. [Batch monitoring jobs](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-predictions/nxt-prediction-jobs.html) let you monitor predictions data over time. This page describes how to manage batch monitoring jobs and job definitions with the Python API client. ## Prerequisites - Data and output configuration for running a monitoring job (e.g., intake and output settings compatible with batch prediction jobs ). ## Batch monitoring job definitions A batch monitoring job definition is a saved configuration for a batch monitoring job. Definitions can be run once on demand or on a schedule. The Python client exposes these via [BatchMonitoringJobDefinition](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/batch-monitoring.html#datarobot.models.batch_monitoring.BatchMonitoringJobDefinition). ### List job definitions List all batch monitoring job definitions. You can also filter them by deployment or name. ``` import datarobot as dr # List all definitions definitions = dr.BatchMonitoringJobDefinition.list() # Filter by deployment definitions = dr.BatchMonitoringJobDefinition.list(deployment_id='5c939e08962d741e34f609f0') # Search by name definitions = dr.BatchMonitoringJobDefinition.list(search_name='Daily Monitoring') ``` Refer to [BatchMonitoringJobDefinition.list](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/batch-monitoring.html#datarobot.models.batch_monitoring.BatchMonitoringJobDefinition.list) for parameters such as `offset` and `limit`. ### Get a job definition Retrieve a single job definition via its ID: ``` import datarobot as dr definition = dr.BatchMonitoringJobDefinition.get(job_definition_id='550e8400-e29b-41d4-a716-446655440000') print(definition.name, definition.deployment_id) ``` ### Create a job definition Create a batch monitoring job definition with the desired intake, output, and deployment. The payload matches the [batch monitoring job create](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-batch-monitoring.html) API (e.g., `deploymentId`, `intakeSettings`, `outputSettings`). You can set `enabled` and `schedule` to run it on a schedule. ``` import datarobot as dr definition = dr.BatchMonitoringJobDefinition.create( deployment_id='5c939e08962d741e34f609f0', name='Weekly batch monitoring', intake_settings={'type': 'localFile', 'url': 'file:///data/predictions.csv'}, output_settings={'type': 'localFile', 'path': '/out/monitoring'}, enabled=False ) ``` Refer to [BatchMonitoringJobDefinition.create](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/batch-monitoring.html#datarobot.models.batch_monitoring.BatchMonitoringJobDefinition.create) and the [Batch Monitoring API](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-monitoring/nxt-batch-monitoring.html) for all supported options (e.g., `monitoringBatchPrefix`, `monitoringColumns`, `chunkSize`). ### Update a job definition Update an existing definition (e.g., name, schedule, or enabled state): ``` import datarobot as dr definition = dr.BatchMonitoringJobDefinition.get(job_definition_id='550e8400-e29b-41d4-a716-446655440000') definition.update(name='Daily batch monitoring', enabled=True) ``` See [BatchMonitoringJobDefinition.update](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/batch-monitoring.html#datarobot.models.batch_monitoring.BatchMonitoringJobDefinition.update) for updatable fields. ### Run a job definition To execute a definition once without changing its schedule: ``` import datarobot as dr definition = dr.BatchMonitoringJobDefinition.get(job_definition_id='550e8400-e29b-41d4-a716-446655440000') job = definition.run_once() print(job.id, job.status) ``` ### Run a job definition on a schedule Start running the definition on its configured schedule (if any): ``` import datarobot as dr definition = dr.BatchMonitoringJobDefinition.get(job_definition_id='550e8400-e29b-41d4-a716-446655440000') definition.run_on_schedule() ``` ### Delete a job definition Remove a batch monitoring job definition. The definition cannot have jobs currently running. ``` import datarobot as dr definition = dr.BatchMonitoringJobDefinition.get(job_definition_id='550e8400-e29b-41d4-a716-446655440000') definition.delete() ``` ## Batch monitoring jobs A batch monitoring job is a single run (either from a single submission or from a definition). Use [BatchMonitoringJob](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/batch-monitoring.html#datarobot.models.batch_monitoring.BatchMonitoringJob) to retrieve the job status and details. ### Get a batch monitoring job Retrieve a job by ID to check status or results (for example, after running a definition with `run_once()`): ``` import datarobot as dr job = dr.BatchMonitoringJob.get(job_id='660e8400-e29b-41d4-a716-446655440001') print(job.status, job.deployment_id) ``` --- # Challenger models URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/challengers.html > Create and manage challenger models to compare against the deployed champion with the Python API client. [Challenger models](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-mitigation/nxt-challengers.html) are alternative models that you can compare against your currently deployed model (the champion model). This allows you to test new models in production, compare their performance, and make data-driven decisions about whether to replace the champion with a better-performing challenger. This page describes how to create, manage, and work with challenger models. ## Best practices ### When to use challengers - A/B testing: Test new models against the current champion in production. - Model validation: Validate that a new model performs well before replacing the champion. - Performance comparison: Compare multiple model candidates simultaneously. - Risk mitigation: Test models with different characteristics (e.g., different algorithms, feature sets). ### Challenger management 1. Naming convention: Use descriptive names that indicate the model type or purpose (e.g., "XGBoost_v2_Challenger"). 2. Limit active challenger models: Too many challengers can impact prediction performance; typically 2-3 challengers is sufficient. 3. Monitor performance: Regularly review challenger performance metrics before deciding to promote one to champion. 4. Clean up: Remove challengers that are no longer being evaluated to keep your deployment clean. 5. Document: Keep track of why each challenger was created and what makes it different from the champion. ### Prerequisites Before creating a challenger model, ensure you have: - A registered model version (model package) ready to use. - An appropriate prediction environment configured. - Challenger models enabled for the deployment. - An understanding of what you want to test or compare. ## Create a challenger model To create a challenger model, you need a deployment, a model package (registered model version), and a prediction environment. The challenger will use the specified model package and prediction environment to make predictions alongside the champion model. ### Basic challenger model creation ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') # Get a model package (registered model version) to use as challenger project = dr.Project.get('6527eb38b9e5dead5fc12491') model = project.get_models()[0] registered_model_version = dr.RegisteredModelVersion.create_for_leaderboard_item( model_id=model.id, name="Challenger Model Version", registered_model_name='My Registered Model' ) # Get a prediction environment prediction_environments = dr.PredictionEnvironment.list() prediction_environment = prediction_environments[0] # Create the challenger challenger = dr.Challenger.create( deployment_id=deployment.id, model_package_id=registered_model_version.id, prediction_environment_id=prediction_environment.id, name='Elastic-Net Classifier Challenger' ) ``` ### Create a challenger that waits for completion By default, challenger creation is asynchronous. You can specify a maximum wait time for the creation to complete: ``` # Wait up to 600 seconds for creation to complete challenger = dr.Challenger.create( deployment_id=deployment.id, model_package_id=registered_model_version.id, prediction_environment_id=prediction_environment.id, name='Random Forest Challenger', max_wait=600 ) ``` ## List challengers You can retrieve all challengers associated with a deployment. ### List all challengers for a deployment ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') challengers = dr.Challenger.list(deployment_id=deployment.id) ``` You can also use the `list_challengers()` method: ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') challengers = deployment.list_challengers() for challenger in challengers: print(f"{challenger.name}: {challenger.id}") ``` ## Get a specific challenger To retrieve a single challenger by its ID: ``` challenger = dr.Challenger.get( deployment_id='5c939e08962d741e34f609f0', challenger_id='5c939e08962d741e34f609f1' ) ``` ### Access challenger properties ``` challenger = dr.Challenger.get( deployment_id=deployment.id, challenger_id='5c939e08962d741e34f609f1' ) ``` ## Update a challenger model You can update a challenger model's name and prediction environment. ### Update a challenger model name ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') challenger = deployment.list_challengers()[0] challenger.update(name='Updated Challenger Name') ``` ### Update the prediction environment ``` # Get a different prediction environment prediction_environments = dr.PredictionEnvironment.list() new_environment = prediction_environments[1] challenger.update(prediction_environment_id=new_environment.id) ``` To update both the name and prediction environment: ``` challenger.update( name='Final Challenger Name', prediction_environment_id=new_environment.id ) ``` ## Delete a challenger To remove a challenger from a deployment: ``` challenger = dr.Challenger.get( deployment_id=deployment.id, challenger_id='5c939e08962d741e34f609f1' ) challenger.delete() # Verify deletion challengers = deployment.list_challengers() challenger_ids = [c.id for c in challengers] ``` ## Manage challenger settings You can enable or disable challenger models for a deployment and configure challenger-related settings. ### Get challenger settings ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') settings = deployment.get_challenger_models_settings() ``` ### Update challenger settings To enable or disable challenger models for a deployment: ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update_challenger_models_settings(challenger_models_enabled=True) ``` To disable challenger models: ``` deployment.update_challenger_models_settings(challenger_models_enabled=False) ``` ## Work with challenger predictions Challengers make predictions alongside the champion model, allowing you to compare their performance. ### Understanding challenger predictions When challengers are enabled, predictions made to the deployment will also be scored by the challenger models. This allows you to: - Compare prediction outputs between champion and challengers. - Monitor challenger performance metrics. - Make informed decisions about model replacement. ### Score challenger models You can trigger challenger scoring for existing prediction requests: ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') # Score challengers for predictions with a specific timestamp deployment.score_challenger_predictions(timestamp='2024-01-15T10:00:00Z') ``` ## Common workflows ### Create multiple challengers from different models ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') project = dr.Deployment.get(deployment_id=deployment.id).model['project_id'] project = dr.Project.get(project) # Get top 3 models from the project models = project.get_models()[:3] prediction_environment = dr.PredictionEnvironment.list()[0] challengers = [] for i, model in enumerate(models): registered_model_version = dr.RegisteredModelVersion.create_for_leaderboard_item( model_id=model.id, name=f"Challenger {i+1}", registered_model_name=f'Challenger Model {i+1}' ) challenger = dr.Challenger.create( deployment_id=deployment.id, model_package_id=registered_model_version.id, prediction_environment_id=prediction_environment.id, name=f'{model.model_type} Challenger' ) challengers.append(challenger) print(f"Created {len(challengers)} challengers") ``` ### Compare challenger information ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') challengers = deployment.list_challengers() print("Challenger Comparison:") print(f"{'Name':<40} {'Model Type':<30} {'Model Package ID':<20}") print("-" * 90) for challenger in challengers: model_type = challenger.model.get('type', 'Unknown') model_package_id = challenger.model_package.get('id', 'Unknown') print(f"{challenger.name:<40} {model_type:<30} {model_package_id:<20}") ``` ### Clean up old challenger models ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') challengers = deployment.list_challengers() # Delete challengers older than a certain date or based on criteria # Example: delete challengers with specific naming pattern for challenger in challengers: if 'Old' in challenger.name: print(f"Deleting challenger: {challenger.name}") challenger.delete() ``` ### Replace a challenger with a new model ``` deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') # Get existing challenger old_challenger = deployment.list_challengers()[0] # Create new model version # Get a different model new_model = project.get_models()[1] new_registered_model_version = dr.RegisteredModelVersion.create_for_leaderboard_item( model_id=new_model.id, name="Updated Challenger Version", registered_model_name='Challenger Model' ) # Delete old challenger old_challenger.delete() # Create new challenger with same name but new model new_challenger = dr.Challenger.create( deployment_id=deployment.id, model_package_id=new_registered_model_version.id, prediction_environment_id=old_challenger.prediction_environment['id'], name=old_challenger.name ) ``` ## Considerations - Challenger creation is an asynchronous process. The max_wait parameter controls how long to wait for creation to complete. - A deployment can have multiple challenger models active simultaneously. - Challengers use the same prediction requests as the champion, allowing for direct comparison. - The champion model (the currently deployed model) cannot be deleted while challengers exist that reference it. - Challenger models must use compatible prediction environments with the deployment. - Model packages used for challengers must have the same target type and compatible settings as the champion model. --- # Custom metrics URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_metrics.html > Define and submit custom metrics for deployments with the Python API client. Custom metrics are used to compute and monitor business or performance metrics. This feature allows you to implement your organization’s specialized metrics, expanding on the insights provided by DataRobot’s built-in service health, data drift, and accuracy metrics. ## Manage custom metrics The following sections outline how to manage custom metrics for deployments. ### Create custom metric To create a custom metric, use `CustomMetric.create`, as shown in the following example. Provide information for all of the required custom metric fields: ``` from datarobot.models.deployment import CustomMetric from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality custom_metric = CustomMetric.create( deployment_id="5c939e08962d741e34f609f0", name="My custom metric", units="x", is_model_specific=True, aggregation_type=CustomMetricAggregationType.AVERAGE, directionality=CustomMetricDirectionality.HIGHER_IS_BETTER, ) ``` To set the baseline value during metric creation, use the following example. ``` from datarobot.models.deployment import CustomMetric from datarobot.enums import ( CustomMetricAggregationType, CustomMetricDirectionality, CustomMetricBucketTimeStep, ) custom_metric = CustomMetric.create( deployment_id="5c939e08962d741e34f609f0", name="My custom metric 2", units="y", baseline_value=12, is_model_specific=True, aggregation_type=CustomMetricAggregationType.AVERAGE, directionality=CustomMetricDirectionality.HIGHER_IS_BETTER, time_step=CustomMetricBucketTimeStep.HOUR, ) ``` Define the names of the columns that are used when submitting values from a dataset. ``` from datarobot.models.deployment import CustomMetric from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality custom_metric = CustomMetric.create( deployment_id="5c939e08962d741e34f609f0", name="My custom metric 3", units="z", baseline_value=1000, is_model_specific=False, aggregation_type=CustomMetricAggregationType.SUM, directionality=CustomMetricDirectionality.LOWER_IS_BETTER, timestamp_column_name="My Timestamp column", timestamp_format="%d/%m/%y", value_column_name="My Value column", sample_count_column_name="My Sample Count column", ) ``` For batches, see the column configuration below. ``` from datarobot.models.deployment import CustomMetric from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality custom_metric = CustomMetric.create( deployment_id="5c939e08962d741e34f609f0", name="My custom metric 4", units="z", baseline_value=1000, is_model_specific=False, aggregation_type=CustomMetricAggregationType.SUM, directionality=CustomMetricDirectionality.LOWER_IS_BETTER, batch_column_name="My Batch column", ) ``` ### List custom metrics To list all custom metrics available for a given deployment, use `CustomMetric.list`. ``` from datarobot.models.deployment import CustomMetric custom_metrics = CustomMetric.list(deployment_id="5c939e08962d741e34f609f0") custom_metrics >>> [CustomMetric('66015bdda7ba87e66baa09ee' | 'My custom metric 2'), CustomMetric('66015bdc5f850c5df3aa09f0' | 'My custom metric')] ``` ### Retrieve custom metrics To get a custom metric by unique identifier, use `CustomMetric.get`. ``` from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") custom_metric >>> CustomMetric('66015bdc5f850c5df3aa09f0' | 'My custom metric') ``` ### Update custom metrics To retrieve a custom metric by its unique identifier and update it, use `CustomMetric.get()` and then `update()`. ``` from datarobot.models.deployment import CustomMetric from datarobot.enums import CustomMetricAggregationType, CustomMetricDirectionality custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") custom_metric.update( name="Updated custom metric", units="foo", baseline_value=-12, aggregation_type=CustomMetricAggregationType.SUM, directionality=CustomMetricDirectionality.LOWER_IS_BETTER, ) ``` ### Reset the custom metric baseline To reset the current metric baseline, use `unset_baseline()`, as shown in the example below. ``` from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") custom_metric.baseline_values >>> [{'value': -12.0}] custom_metric.unset_baseline() custom_metric.baseline_values >>> [] ``` ### Delete custom metrics To delete a custom metric by unique identifier, use `CustomMetric.delete`, as in the following example: ``` from datarobot.models.deployment import CustomMetric CustomMetric.delete(deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") ``` ## Submit custom metric values The following sections outline how to submit custom metric values from various sources. ### Submit values from JSON To submit aggregated custom metric values from JSON, use the `submit_values` method. Submit data in the form of a list of dictionaries. ``` from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") data = [{'value': 12, 'sample_size': 3, 'timestamp': '2024-03-15T18:00:00'}, {'value': 11, 'sample_size': 5, 'timestamp': '2024-03-15T17:00:00'}, {'value': 14, 'sample_size': 3, 'timestamp': '2024-03-15T16:00:00'}] custom_metric.submit_values(data=data) # data witch association IDs data = [{'value': 15, 'sample_size': 2, 'timestamp': '2024-03-15T21:00:00', 'association_id': '65f44d04dbe192b552e752aa'}, {'value': 13, 'sample_size': 6, 'timestamp': '2024-03-15T20:00:00', 'association_id': '65f44d04dbe192b552e753bb'}, {'value': 17, 'sample_size': 2, 'timestamp': '2024-03-15T19:00:00', 'association_id': '65f44d04dbe192b552e754cc'}] custom_metric.submit_values(data=data) ``` To submit data in the form of a pandas DataFrame: ``` from datetime import datetime import pandas as pd from datarobot.models.deployment import CustomMetric df = pd.DataFrame( data={ "timestamp": [ datetime(year=2024, month=3, day=10), datetime(year=2024, month=3, day=11), datetime(year=2024, month=3, day=12), datetime(year=2024, month=3, day=13), datetime(year=2024, month=3, day=14), datetime(year=2024, month=3, day=15), ], "value": [28, 34, 29, 1, 2, 13], "sample_size": [1, 2, 3, 4, 1, 2], } ) custom_metric.submit_values(data=df) ``` For deployment-specific metrics, do not provide model information. For model specific metrics set `model_package_id` or `model_id`. ``` custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25") custom_metric.submit_values(data=data, model_id="6444482e5583f6ee2e572265") ``` Use a dry run to test uploads without saving metric data in DataRobot. This option is disabled by default. ``` custom_metric.submit_values(data=data, dry_run=True) ``` To send data for a given segment, it must be specified as follows. Note that more than one segment can be specified. ``` segments = [{"name": "custom_seg", "value": "baz"}] custom_metric.submit_values(data=data, segments=segments) ``` Batch mode requires specifying batch IDs. Batches always specify a model by `model_package_id` or `model_id`. ``` from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f600e1", custom_metric_id="65f17bdcd2d66683cdfc2224") data = [{'value': 12, 'sample_size': 3, 'batch': '65f44c93fedc5de16b673aaa'}, {'value': 11, 'sample_size': 5, 'batch': '65f44c93fedc5de16b673bbb'}, {'value': 14, 'sample_size': 3, 'batch': '65f44c93fedc5de16b673ccc'}] custom_metric.submit_values(data=data, model_package_id="6421df32525c58cc6f991f25") ``` ### Submit a single value To report a single metric value at the current moment, use the `submit_single_value` method. View the example below, which uses deployment-specific metrics. ``` from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") custom_metric.submit_single_value(value=16) ``` For model-specific metrics, set `model_package_id` or `model_id`. ``` from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") custom_metric.submit_single_value(value=16, model_package_id="6421df32525c58cc6f991f25") custom_metric.submit_single_value(value=16, model_id="6444482e5583f6ee2e572265") ``` Dry run and segments work analogously to report aggregated metric values. ``` custom_metric.submit_single_value(value=16, dry_run=True) segments = [{"name": "custom_seg", "value": "boo"}] custom_metric.submit_single_value(value=16, segments=segments) ``` The sent value timestamp indicates the time the request was sent; the number of sample values is always `1`. This method does not support batch submissions. ### Submit values from a dataset To report aggregated custom metrics values from a dataset in the Data Registry, use the `submit_values_from_catalog` method. ``` from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") # for deployment specific metrics custom_metric.submit_values_from_catalog(dataset_id="61093144cabd630828bca321") # for model specific metrics set model_package_id or model_id custom_metric.submit_values_from_catalog( dataset_id="61093144cabd630828bca321", model_package_id="6421df32525c58cc6f991f25" ) ``` For segmented analysis, define the name of the column in the dataset and the segment to which it corresponds. ``` segments = [{"name": "custom_seg", "column": "column_with_segment_values"}] custom_metric.submit_values_from_catalog( dataset_id="61093144cabd630828bca321", model_package_id="6421df32525c58cc6f991f25", segments=segments ) ``` For batches, specify the batch IDs in the dataset, or send the entire dataset for a single batch ID. ``` custom_metric.submit_values_from_catalog( dataset_id="61093144cabd630828bca432", model_package_id="6421df32525c58cc6f991f25", batch_id="65f7f71198c2f234b4cb2f7d" ) ``` The names of the columns in the dataset should correspond to the names of the columns that were defined in the custom metric. In addition, the format of the timestamps should also be the same as defined in the metric. If the sample size is not specified, it is treated as a 1 sample by default. The following example shows the shape of a dataset saved in the AI catalog. | timestamp | sample_size | value | | --- | --- | --- | | 12/12/22 | 1 | 22 | | 13/12/22 | 2 | 23 | | 14/12/22 | 3 | 24 | | 15/12/22 | 4 | 25 | The following table shows a sample dataset for batches. | batch | sample_size | value | | --- | --- | --- | | 6572db2c9f9d4ad3b9de33d0 | 1 | 22 | | 6572db2c9f9d4ad3b9de33d0 | 2 | 23 | | 6572db319f9d4ad3b9de33d9 | 3 | 24 | | 6572db319f9d4ad3b9de33d9 | 4 | 25 | ## Retrieve custom metric values over time The following sections outline how to retrieve custom metric values. ### Retrieve values over a time period To retrieve values of a custom metric over a time period, use `get_values_over_time`. ``` from datetime import datetime, timedelta from datarobot.enums import BUCKET_SIZE from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") now = datetime.now() # specify the time window and bucket size by which results are grouped, the default bucket is 7 days values_over_time = custom_metric.get_values_over_time( start=now - timedelta(days=2), end=now, bucket_size=BUCKET_SIZE.P1D) values_over_time >>> CustomMetricValuesOverTime('2024-03-21 20:15:00+00:00'- '2024-03-23 20:15:00+00:00')" values_over_time.bucket_values >>>{datetime.datetime(2024, 3, 22, 10, 0, tzinfo=tzutc()): 1.0, >>> datetime.datetime(2024, 3, 22, 11, 0, tzinfo=tzutc()): 123.0}} values_over_time.bucket_sample_sizes >>>{datetime.datetime(2024, 3, 22, 10, 0, tzinfo=tzutc()): 1, >>> datetime.datetime(2024, 3, 22, 11, 0, tzinfo=tzutc()): 1}} values_over_time.get_buckets_as_dataframe() >>> start end value sample_size >>> 0 2024-03-21 00:00:00+00:00 2024-03-22 00:00:00+00:00 1.0 1 >>> 1 2024-03-22 00:00:00+00:00 2024-03-23 00:00:00+00:00 123.0 1 ``` For model-specific metrics, set `model_package_id` or `model_id`. ``` values_over_time = custom_metric.get_values_over_time( start=now - timedelta(days=1), end=now, model_package_id="6421df32525c58cc6f991f25") values_over_time = custom_metric.get_values_over_time( start=now - timedelta(days=1), end=now, model_id="6444482e5583f6ee2e572265") ``` To retrieve values for a specific segment, specify the segment name and its value: ``` values_over_time = custom_metric.get_values_over_time( start=now - timedelta(days=1), end=now, segment_attribute="custom_seg", segment_value="val_1") ``` ### Retrieve a summary over a time period To retrieve a summary of a custom metric over a time period, use the `get_summary` method. ``` from datetime import datetime, timedelta from datarobot.enums import BUCKET_SIZE from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113") now = datetime.now() # specify the time window summary = custom_metric.get_summary(start=now - timedelta(days=7), end=now) print(summary) >> "CustomMetricSummary(2024-03-15 15:52:13.392178+00:00 - 2024-03-22 15:52:13.392168+00:00: {'id': '65fd9b1c0c1a840bc6751ce0', 'name': 'My custom metric', 'value': 215.0, 'sample_count': 13, 'baseline_value': 12.0, 'percent_change': 24.02})" ``` For model-specific metrics, set `model_package_id` or `model_id`. ``` summary = custom_metric.get_summary( start=now - timedelta(days=7), end=now, model_package_id="6421df32525c58cc6f991f25") summary = custom_metric.get_summary( start=now - timedelta(days=7), end=now, model_id="6444482e5583f6ee2e572265") ``` To retrieve a summary for a specific segment, specify the segment name and its value. ``` summary = custom_metric.get_summary( start=now - timedelta(days=7), end=now, segment_attribute="custom_seg", segment_value="val_1") ``` ### Retrieve values over a batch To retrieve values of a custom metric over a batch, use `get_values_over_batch`. ``` from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113" ) # All batch metrics, all model-specific values_over_batch = custom_metric.get_values_over_batch(model_package_id='6421df32525c58cc6f991f25') values_over_batch.bucket_values >>> {'6572db2c9f9d4ad3b9de33d0': 35.0, '6572db2c9f9d4ad3b9de44e1': 105.0} values_over_batch.bucket_sample_sizes >>> {'6572db2c9f9d4ad3b9de33d0': 6, '6572db2c9f9d4ad3b9de44e1': 8} values_over_batch.get_buckets_as_dataframe() >>> batch_id batch_name value sample_size >>> 0 6572db2c9f9d4ad3b9de33d0 Batch 1 - 03/26/2024 13:04:46 35.0 6 >>> 1 6572db2c9f9d4ad3b9de44e1 Batch 2 - 03/26/2024 13:06:04 105.0 8 ``` For specific batches, set `batch_ids`. ``` values_over_batch = custom_metric.get_values_over_batch( model_package_id='6421df32525c58cc6f991f25', batch_ids=["65f44c93fedc5de16b673aaa", "65f44c93fedc5de16b673bbb"]) ``` To retrieve values for a specific segment, specify the segment name and its value. ``` values_over_batch = custom_metric.get_values_over_batch( model_package_id='6421df32525c58cc6f991f25', segment_attribute="custom_seg", segment_value="val_1") ``` ### Retrieve a summary over batch To retrieve a summary of a custom metric over batch, use `get_summary`: ``` from datarobot.models.deployment import CustomMetric custom_metric = CustomMetric.get( deployment_id="5c939e08962d741e34f609f0", custom_metric_id="65f17bdcd2d66683cdfc1113" ) # All batch metrics, all model-specific batch_summary = custom_metric.get_batch_summary(model_package_id='6421df32525c58cc6f991f25') print(batch_summary) >> CustomMetricBatchSummary({'id': '6605396413434b3a7b74342c', 'name': 'batch metric', 'value': 41.25, 'sample_count': 28, 'baseline_value': 123.0, 'percent_change': -66.46}) ``` For specific batches, set `batch_ids`. ``` batch_summary = custom_metric.get_batch_summary( model_package_id='6421df32525c58cc6f991f25', batch_ids=["65f44c93fedc5de16b673aaa", "65f44c93fedc5de16b673bbb"]) ``` To retrieve values for a specific segment, specify the segment name and its value. ``` batch_summary = custom_metric.get_batch_summary( model_package_id='6421df32525c58cc6f991f25', segment_attribute="custom_seg", segment_value="val_1") ``` ## Hosted custom metrics Hosted custom metrics allow you to implement up to 5 of your organization’s specialized metrics in a deployment, uploading the custom metric code to DataRobot and hosting the metric calculation on custom jobs infrastructure. After creation, hosted custom metrics can be reused for other deployments. DataRobot provides a variety of templates for common metrics. These metrics can be used as-is, or as a starting point for user-provided metrics. The following sections outline how to create a hosted custom metric using the Python API client. ### Prerequisites Import the necessary objects to create a custom metric and initialize the DataRobot client. ``` import datarobot as dr from datarobot.enums import HostedCustomMetricsTemplateMetricTypeQueryParams from datarobot.models.deployment.custom_metrics import HostedCustomMetricTemplate, HostedCustomMetric, \ HostedCustomMetricBlueprint, CustomMetric, MetricTimestampSpoofing, ValueField, SampleCountField, BatchField from datarobot.models.registry import JobRun from datarobot.models.registry.job import Job from datarobot import Deployment from datarobot.models.runtime_parameters import RuntimeParameterValue from datarobot.models.types import Schedule dr.Client(token="", endpoint="") gen_ai_deployment_1 = Deployment.get('') ``` ### List hosted custom metric templates Before creating a hosted custom metric from a template, retrieve the LLM metric template to use as the basis of the new metric. To do this, specify the `metric_type` and, because the deployments are LLM models handling Japanese text, search for the specific metric by name, limiting the search to 1 result. Store the result in `templates` for the following steps. ``` templates = HostedCustomMetricTemplate.list( search="[JP] Character Count", metric_type=HostedCustomMetricsTemplateMetricTypeQueryParams.LLM, limit=1, offset=0, ) ``` ### Create a hosted custom metric After locating the custom metric template, create the hosted custom metric from that template. This method is a shortcut, combining two steps to create the new custom metric from the retrieved template: 1. Create a custom job for a hosted custom metric from the template previously retrieved (stored in templates ). 2. Connect the hosted custom metric job to the deployment previously defined (stored in gen_ai_deployment_1 ). Specify both the job name and custom metric name in addition to the template and deployment IDs, as you are creating two objects. Additionally, define the job schedule and the runtime parameter overrides for the deployment. ``` hosted_custom_metric = HostedCustomMetric.create_from_template( template_id=templates[0].id, deployment_id=gen_ai_deployment_1.id, job_name="Hosted Custom Metric Character Count", custom_metric_name="Character Count", job_description="Hosted Custom Metric", custom_metric_description="LLM Character Count", baseline_value=10, timestamp=MetricTimestampSpoofing( column_name="timestamp", time_format="%Y-%m-%d %H:%M:%S", ), value = ValueField(column_name="value"), sample_count=SampleCountField(column_name='Sample Count'), batch=BatchField(column_name='Batch'), schedule=Schedule( day_of_week=[0], hour=['*'], minute=['*'], day_of_month=[12], month=[1], ), parameter_overrides=[RuntimeParameterValue(field_name='DRY_RUN', value="0", type="string")] ) ``` Once we have created the hosted custom metric, initiate the manual run. ``` job_run = JobRun.create( job_id=hosted_custom_metric.custom_job_id runtime_parameter_values=[ RuntimeParameterValue(field_name='DRY_RUN', value="1", type="string"), RuntimeParameterValue(field_name='DEPLOYMENT_ID', value=gen_ai_deployment_1.id, type="deployment"), RuntimeParameterValue(field_name='CUSTOM_METRIC_ID', value=hosted_custom_metric.id, type="customMetric"), ] ) print(job_run.status) ``` ### Manually create hosted custom metrics You can alternatively create hosted custom metrics in a manual sequenced process. This is useful if you want to edit the custom metric blueprint before attaching the custom job to the deployment. When you attach the job to the deployment, most settings are copied from the blueprint (unless you provide an override). To create the hosted custom metric manually, first create a custom job from the template (stored in `templates`). ``` job = Job.create_from_custom_metric_gallery_template( template_id=templates[0].id, name="Job created from template", description="Job created from template" ) ``` Next, retrieve the default blueprint provided by the template, and edit it. ``` blueprint = HostedCustomMetricBlueprint.get(job.id) print(f"Original directionality: {blueprint.directionality}") ``` Then, update the parameters of the custom metric in the blueprint. ``` updated_blueprint = blueprint.update( directionality='lowerIsBetter', units='characters', type='gauge', time_step='hour', is_model_specific=False ) print(f"Updated directionality: {updated_blueprint.directionality}") ``` Now, create the hosted custom metric. As in the shortcut method, you can provide the job schedule, runtime parameter overrides, and custom metric parameters specific to this deployment. ``` another_hosted_custom_metric = HostedCustomMetric.create_from_custom_job( custom_job_id=job.id, deployment_id=gen_ai_deployment_1.id, name="Custom metric created in 2 steps", ) ``` After creating and configuring the metric, verify that the changes to the blueprint are reflected in the custom metric. ``` another_custom_metric = CustomMetric.get(custom_metric_id=another_hosted_custom_metric.id, deployment_id=gen_ai_deployment_1.id) print(f"Directionality of another custom metric: {another_custom_metric.directionality}") ``` Finally, create a manual job run for the custom metric job. ``` job_run = JobRun.create( job_id=job.id, runtime_parameter_values=[ RuntimeParameterValue(field_name='DRY_RUN', value="1", type="string"), RuntimeParameterValue(field_name='DEPLOYMENT_ID', value=gen_ai_deployment_1.id, type="deployment"), RuntimeParameterValue(field_name='CUSTOM_METRIC_ID', value=another_hosted_custom_metric.id, type="customMetric"), ] ) print(job_run.status) ``` ### List hosted custom metrics To list all hosted custom metrics associated with a custom job, use the following code: ``` hosted_custom_metrics = HostedCustomMetric.list(deployment_id=hosted_custom_metric.custom_job_id) for metric in hosted_custom_metrics: print(metric.name) ``` ### Delete hosted custom metrics In addition, you can delete the hosted custom metric, which removes it from deployment while keeping the job, allowing you to create the metric for another deployment. ``` hosted_custom_metric.delete() another_hosted_custom_metric.delete() ``` If necessary, you can delete the entire custom job. If there are any custom metrics associated with that job, they are also deleted. ``` job.delete() ``` --- # Custom models URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html > Manage custom inference models and execution environments with the Python API client. Custom models provide the ability to run arbitrary modeling code in a user-defined environment. ## Manage execution environments The execution environment defines the runtime environment for custom models. Execution environment version is a revision of execution environment with an actual runtime definition. Refer to [DataRobot User Models repository](https://github.com/datarobot/datarobot-user-models) for sample environments. ### Create an execution environment To create an execution environment: ``` import datarobot as dr execution_environment = dr.ExecutionEnvironment.create( name="Python3 PyTorch Environment", description="This environment contains Python3 pytorch library.", ) execution_environment.id >>> '5b6b2315ca36c0108fc5d41b' ``` #### Create an execution environment version from Docker Context or Docker URI You can create an Execution Environment Version using either a Docker image URI, a Docker context, or both. If you provide both, the environment version is built from the image URI, while the context is uploaded for informational purposes and can be later downloaded. There are two ways to create an execution environment version: synchronously and asynchronously. The synchronous method blocks program execution until the execution environment version is created or creation fails. ``` import datarobot as dr # Use the execution_environment created previously environment_version = dr.ExecutionEnvironmentVersion.create( execution_environment.id, docker_context_path="datarobot-user-models/public_dropin_environments/python3_pytorch", max_wait=3600, # 1 hour timeout ) environment_version.id >>> '5eb538959bc057003b487b2d' environment_version.build_status >>> 'success' ``` The asynchronous method does not block execution, but the execution environment version will not be ready for use until the creation process is finished. In this case, you must manually call [refresh()](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.ExecutionEnvironmentVersion.refresh) for the execution environment version and check if its `build_status` is “success”. To create an execution environment version without blocking a program, set `max_wait` to `None`: ``` import datarobot as dr # Use the execution environment created earlier # create environment version from docker context environment_version = dr.ExecutionEnvironmentVersion.create( execution_environment.id, docker_context_path="datarobot-user-models/public_dropin_environments/python3_pytorch", max_wait=None, # Set None to not block execution on this method ) environment_version.id >>> '5eb538959bc057003b487b2d' environment_version.build_status >>> 'processing' # After some time environment_version.refresh() environment_version.build_status >>> 'success' # now create anoter environment version from docker image URI environment_version = dr.ExecutionEnvironmentVersion.create( execution_environment.id, docker_image_uri="test_org/test_repo:test_tag", max_wait=None, # set None to not block execution on this method ) environment_version.id >>> '5eb538959bc057003b4943d2' environment_version.build_status >>> 'success' environment_version.docker_image_uri 'test_org/test_repo:test_tag' ``` If your environment requires additional metadata to be supplied for models using it, you can create an environment with additional metadata keys. Custom model versions that use this environment must specify values for these keys before they can be used to run tests or make deployments. The values will be baked in as environment variables with `field_name` as the environment variable name. ``` import datarobot as dr from datarobot.models.execution_environment import RequiredMetadataKey execution_environment = dr.ExecutionEnvironment.create( name="Python3 PyTorch Environment", description="This environment contains Python3 pytorch library.", required_metadata_keys=[ RequiredMetadataKey(field_name="MY_VAR", display_name="A value needed by hte environment") ], ) model_version = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=execution_environment.id, folder_path=custom_model_folder, required_metadata={"MY_VAR": "a value"} ) ``` ### List execution environments To list execution environments available to the user: ``` import datarobot as dr execution_environments = dr.ExecutionEnvironment.list() execution_environments >>> [ExecutionEnvironment('[DataRobot] Python 3 PyTorch Drop-In'), ExecutionEnvironment('[DataRobot] Java Drop-In')] environment_versions = dr.ExecutionEnvironmentVersion.list(execution_environment.id) environment_versions >>> [ExecutionEnvironmentVersion('v1')] ``` Refer to [ExecutionEnvironment](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.ExecutionEnvironment) for properties of the execution environment object and [ExecutionEnvironmentVersion](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.ExecutionEnvironmentVersion) for properties of the execution environment object version. You can also filter the execution environments that are returned by passing a string as a `search_for` parameter. Only the execution environments that contain the passed string in `name` or `description` are returned. ``` import datarobot as dr execution_environments = dr.ExecutionEnvironment.list(search_for='java') execution_environments >>> [ExecutionEnvironment('[DataRobot] Java Drop-In')] ``` Execution environment versions can be filtered by build status. ``` import datarobot as dr environment_versions = dr.ExecutionEnvironmentVersion.list( execution_environment.id, dr.EXECUTION_ENVIRONMENT_VERSION_BUILD_STATUS.PROCESSING ) environment_versions >>> [ExecutionEnvironmentVersion('v1')] ``` ### Retrieve an execution environment To retrieve an execution environment and an execution environment version by identifier (rather than list all available environments): ``` import datarobot as dr execution_environment = dr.ExecutionEnvironment.get(execution_environment_id='5506fcd38bd88f5953219da0') execution_environment >>> ExecutionEnvironment('[DataRobot] Python 3 PyTorch Drop-In') environment_version = dr.ExecutionEnvironmentVersion.get( execution_environment_id=execution_environment.id, version_id='5eb538959bc057003b487b2d') environment_version >>> ExecutionEnvironmentVersion('v1') ``` ### Update an execution environment To update name or description of the execution environment: ``` import datarobot as dr execution_environment = dr.ExecutionEnvironment.get(execution_environment_id='5506fcd38bd88f5953219da0') execution_environment.update(name='new name', description='new description') ``` ### Delete Execution Environment To delete the execution environment and execution environment version: ``` import datarobot as dr execution_environment = dr.ExecutionEnvironment.get(execution_environment_id='5506fcd38bd88f5953219da0') execution_environment.delete() ``` ### Get execution environment build logs To get an execution environment version build log: ``` import datarobot as dr environment_version = dr.ExecutionEnvironmentVersion.get( execution_environment_id='5506fcd38bd88f5953219da0', version_id='5eb538959bc057003b487b2d') log, error = environment_version.get_build_log() ``` ## Manage custom models A custom model is user-defined modeling code that supports making predictions against it. Custom models support regression, binary classification, multiclass, multilabel, anomaly detection, and unstructured target types. To upload actual modeling code, you must create a custom model version for a custom model. See [Custom Model Version documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html#custom-model-versions) for more information. ### Create a custom model #### Regression model To create a regression custom model: ``` import datarobot as dr custom_model = dr.CustomInferenceModel.create( name='Python 3 PyTorch Custom Model', target_type=dr.TARGET_TYPE.REGRESSION, target_name='MEDV', description='This is a Python3-based custom model. It has a simple PyTorch model built on boston housing', language='python' ) custom_model.id >>> '5b6b2315ca36c0108fc5d41b' ``` #### Binary classification model When creating a binary classification custom model, `positive_class_label` and `negative_class_label` must be set: ``` import datarobot as dr custom_model = dr.CustomInferenceModel.create( name='Python 3 PyTorch Custom Model', target_type=dr.TARGET_TYPE.BINARY, target_name='readmitted', positive_class_label='False', negative_class_label='True', description='This is a Python3-based custom model. It has a simple PyTorch model built on 10k_diabetes dataset', language='Python 3' ) custom_model.id >>> '5b6b2315ca36c0108fc5d41b' ``` #### Multiclass model When creating a multiclass classification custom model, you must provide `class_labels`: ``` import datarobot as dr custom_model = dr.CustomInferenceModel.create( name='Python 3 PyTorch Custom Model', target_type=dr.TARGET_TYPE.MULTICLASS, target_name='readmitted', class_labels=['hot dog', 'burrito', 'hoagie', 'reuben'], description='This is a Python3-based custom model. It has a simple PyTorch model built on sandwich dataset', language='Python 3' ) custom_model.id >>> '5b6b2315ca36c0108fc5d41b' ``` Multiclass labels can also be provided as a file in cases where there are many class labels. The file should have each class label separated by a new line. ``` import datarobot as dr custom_model = dr.CustomInferenceModel.create( name='Python 3 PyTorch Custom Model', target_type=dr.TARGET_TYPE.MULTICLASS, target_name='readmitted', class_labels_file='/path/to/classlabels.txt', description='This is a Python3-based custom model. It has a simple PyTorch model built on sandwich dataset', language='Python 3' ) custom_model.id >>> '5b6b2315ca36c0108fc5d41b' ``` #### Multilabel model When creating a multilabel classification custom model, you must provide at least two labels in `class_labels`: ``` import datarobot as dr custom_model = dr.CustomInferenceModel.create( name='Multilabel Custom Model', target_type=dr.TARGET_TYPE.MULTILABEL, target_name='genre', class_labels=['action', 'comedy', 'sci-fi'], language='python' ) ``` Multilabel custom models support inference only; you cannot create multilabel custom training tasks. #### Unstructured model For unstructured models, the `target_name` parameter is optional and ignored if provided. To create an unstructured custom model: ``` import datarobot as dr custom_model = dr.CustomInferenceModel.create( name='Python 3 Unstructured Custom Model', target_type=dr.TARGET_TYPE.UNSTRUCTURED, description='This is a Python3-based unstructured model', language='python' ) custom_model.id >>> '5b6b2315ca36c0108fc5d41b' ``` #### Anomaly detection model For anomaly detection models, the `target_name` parameter is also optional and is ignored if provided. To create an anomaly detection custom model: ``` import datarobot as dr custom_model = dr.CustomInferenceModel.create( name='Python 3 Unstructured Custom Model', target_type=dr.TARGET_TYPE.ANOMALY, description='This is a Python3-based anomaly detection model', language='python' ) custom_model.id >>> '5b6b2315ca36c0108fc5d41b' ``` #### k8s resources Custom model k8s resources are optional and unless specifically provided, the configured defaults are used. To create a custom model with specific k8s resources: ``` import datarobot as dr custom_model = dr.CustomInferenceModel.create( name='Python 3 PyTorch Custom Model', target_type=dr.TARGET_TYPE.BINARY, target_name='readmitted', positive_class_label='False', negative_class_label='True', description='This is a Python3-based custom model. It has a simple PyTorch model built on 10k_diabetes dataset', language='Python 3', maximum_memory=512*1024*1024, ) ``` ### Assign training data to custom models To create a custom model that enables training data assignment on the model version level, provide the `is_training_data_for_versions_permanently_enabled=True` parameter. For more information, refer to the [Custom model version creation with training data](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html#custom-inference-model-version-training-data) documentation. ``` import datarobot as dr custom_model = dr.CustomInferenceModel.create( name='Python 3 PyTorch Custom Model', target_type=dr.TARGET_TYPE.REGRESSION, target_name='MEDV', description='This is a Python3-based custom model. It has a simple PyTorch model built on boston housing', language='python', is_training_data_for_versions_permanently_enabled=True ) custom_model.id >>> '5b6b2315ca36c0108fc5d41b' ``` ### List custom models To list the custom models available to you: ``` import datarobot as dr dr.CustomInferenceModel.list() >>> [CustomInferenceModel('my model 2'), CustomInferenceModel('my model 1')] # use these parameters to filter results: dr.CustomInferenceModel.list( is_deployed=True, # set to return only deployed models order_by='-updated', # set to define order of returned results search_for='model 1', # return only models containing 'model 1' in name or description ) >>> CustomInferenceModel('my model 1') ``` Refer to [list()](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.CustomInferenceModel.list) for detailed parameter descriptions. ### Retrieve a custom model To retrieve a specific custom model: ``` import datarobot as dr dr.CustomInferenceModel.get('5ebe95044024035cc6a65602') >>> CustomInferenceModel('my model 1') ``` ### Update custom model To update custom model properties: ``` import datarobot as dr custom_model = dr.CustomInferenceModel.get('5ebe95044024035cc6a65602') custom_model.update( name='new name', description='new description', ) ``` Please, refer to [update()](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.CustomInferenceModel.update) for the full list of properties that can be updated. ### Download latest revision of a custom model To download content of the latest Custom Model Version of `CustomInferenceModel` as a ZIP archive: ``` import datarobot as dr path_to_download = '/home/user/Documents/myModel.zip' custom_model = dr.CustomInferenceModel.get('5ebe96b84024035cc6a6560b') custom_model.download_latest_version(path_to_download) ``` ### Assign training data to a custom model This example assigns training data on the model level. To assign training data on the model version level, see the [Custom model version creation with training data](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html#custom-inference-model-version-training-data) documentation. To assign training data to custom inference model: ``` import datarobot as dr path_to_dataset = '/home/user/Documents/trainingDataset.csv' dataset = dr.Dataset.create_from_file(file_path=path_to_dataset) custom_model = dr.CustomInferenceModel.get('5ebe96b84024035cc6a6560b') custom_model.assign_training_data(dataset.id) ``` To assign training data without blocking a program, set `max_wait` to `None`: ``` import datarobot as dr path_to_dataset = '/home/user/Documents/trainingDataset.csv' dataset = dr.Dataset.create_from_file(file_path=path_to_dataset) custom_model = dr.CustomInferenceModel.get('5ebe96b84024035cc6a6560b') cmv = dr.CustomModelVersion.create_from_previous(custom_model_id = custom_model.id, training_dataset_id = dataset.id) cmv.refresh() while cmv.training_data.assignment_in_progress: time.sleep(60) cmv.refresh() ``` Note: training data must be assigned to retrieve feature impact from a custom model version. See the [Custom Model Version documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html#custom-model-version-feature-impact). ## Manage versions Modeling code for custom models can be uploaded by creating a custom model version. When creating a Custom Model Version, the version must be associated with a base execution environment. If the base environment supports additional model dependencies (R or Python environments) and the custom model version contains a valid `requirements.txt` file, the model version will run in an environment based on the base environment with the additional dependencies installed. ### Create a custom model version You can upload custom model content by creating a clean custom model version: ``` import os import datarobot as dr custom_model_folder = "datarobot-user-models/model_templates/python3_pytorch" # Add files from the folder to the custom model model_version = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=execution_environment.id, folder_path=custom_model_folder, ) custom_model.id >>> '5b6b2315ca36c0108fc5d41b' # Alternatively, add a list of files to the custom model model_version_2 = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=execution_environment.id, files=[(os.path.join(custom_model_folder, 'custom.py'), 'custom.py')], ) # You can also set k8s resources to the custom model model_version_3 = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=execution_environment.id, files=[(os.path.join(custom_model_folder, 'custom.py'), 'custom.py')], network_egress_policy=dr.NETWORK_EGRESS_POLICY.PUBLIC, maximum_memory=512*1024*1024, replicas=1, ) ``` To create a new custom model version from a previous one that modifies some files, use the following code. ``` import os import datarobot as dr custom_model_folder = "datarobot-user-models/model_templates/python3_pytorch" file_to_delete = model_version_2.items[0].id model_version_3 = dr.CustomModelVersion.create_from_previous( custom_model_id=custom_model.id, base_environment_id=execution_environment.id, files=[(os.path.join(custom_model_folder, 'custom.py'), 'custom.py')], files_to_delete=[file_to_delete], ) ``` Reference [CustomModelFileItem](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.models.custom_model_version.CustomModelFileItem) for more information about custom model file properties. You can specify a custom environment version when creating a custom model version. By default a version of the same environment does not change between consecutive model versions. However, this behavior can be overridden: ``` import os import datarobot as dr custom_model_folder = "datarobot-user-models/model_templates/python3_pytorch" # Create a clean version and specify an explicit environment version. model_version = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=execution_environment.id, base_environment_version_id="642209acc5638929a9b8dc3d", folder_path=custom_model_folder, ) # Create a version from a previous one, specifying an explicit environment version. model_version_2 = dr.CustomModelVersion.create_from_previous( custom_model_id=custom_model.id, base_environment_id=execution_environment.id, base_environment_version_id="660186775d016eabb290aee9", ) ``` To create a new custom model version from a previous one, with just new k8s resource values: ``` import os import datarobot as dr custom_model_folder = "datarobot-user-models/model_templates/python3_pytorch" file_to_delete = model_version_2.items[0].id model_version_3 = dr.CustomModelVersion.create_from_previous( custom_model_id=custom_model.id, base_environment_id=execution_environment.id, maximum_memory=1024*1024*1024, ) ``` ### Create a custom model version with training data Model version creation allows you to provide training (and holdout) data information. Every custom model has to be explicitly switched to allow training data assignment for model versions. Note that the training data assignment differs for structured and unstructured models, and should be handled differently. #### Enable training data assignment for custom model versions By default, custom model training data is assigned on the model level; for more information, see the [Custom model training data assignment](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html#custom-inference-model-assign-data) documentation. When training data is assigned to a model, the same training data is used for every model version. This method of training data assignment is deprecated and scheduled for removal; however, to avoid introducing issues for existing models, you must individually convert existing models to perform training data assignment by model version. Note that this change is permanent and cannot be undone. Because the conversion process is irreversible, it is highly recommended that you do not convert critical models to the new training data assignment method. Instead, you should duplicate the existing model and test the new method. Use the code below to permanently enable a training data assignment on the model version level for the specified model. ``` import datarobot as dr dr.Client(token=my_token, endpoint=endpoint) custom_model = dr.CustomInferenceModel.get(custom_model_id) custom_model.update(is_training_data_for_versions_permanently_enabled=True) custom_model.is_training_data_for_versions_permanently_enabled # True ``` #### Assign training data for structured models Training data assignment is performed asynchronously, so you can create a version in a blocking or non-blocking way (shown in the examples below). Create a structured model version with blocking (default `max_wait=600`) and wait for the training data assignment result. If the training data assignment fails: - A datarobot.errors.TrainingDataAssignmentError exception is raised. The exception contains the custom model ID, the custom model version ID, and the failure message. - A new custom model version is still created and can be fetched for further processing, but it’s not possible to create a model package from it or deploy it. ``` import datarobot as dr from datarobot.errors import TrainingDataAssignmentError dr.Client(token=my_token, endpoint=endpoint) try: version = dr.CustomModelVersion.create_from_previous( custom_model_id="6444482e5583f6ee2e572265", base_environment_id="642209acc563893014a41e24", training_dataset_id="6421f2149a4f9b1bec6ad6dd", ) except TrainingDataAssignmentError as e: print(e) ``` To fetch the model version in the case of an assignment error: ``` import datarobot as dr from datarobot.errors import TrainingDataAssignmentError dr.Client(token=my_token, endpoint=endpoint) try: version = dr.CustomModelVersion.create_from_previous( custom_model_id="6444482e5583f6ee2e572265", base_environment_id="642209acc563893014a41e24", training_dataset_id="6421f2149a4f9b1bec6ad6dd", ) except TrainingDataAssignmentError as e: version = CustomModelVersion.get( custom_model_id="6444482e5583f6ee2e572265", custom_model_version_id=e.custom_model_version_id, ) print(version.training_data.dataset_id) print(version.training_data.dataset_version_id) print(version.training_data.dataset_name) print(version.training_data.assignment_error) ``` Below is another example of fetching the model version in the case of an assignment error. ``` import datarobot as dr from datarobot.errors import TrainingDataAssignmentError dr.Client(token=my_token, endpoint=endpoint) custom_model = dr.CustomInferenceModel.get("6444482e5583f6ee2e572265") try: version = dr.CustomModelVersion.create_from_previous( custom_model_id="6444482e5583f6ee2e572265", base_environment_id="642209acc563893014a41e24", training_dataset_id="6421f2149a4f9b1bec6ad6dd", ) except TrainingDataAssignmentError as e: pass custom_model.refresh() version = custom_model.latest_version print(version.training_data.dataset_id) print(version.training_data.dataset_version_id) print(version.training_data.dataset_name) print(version.training_data.assignment_error) ``` Create a structured model version with a non-blocking (set `max_wat=None`) training data assignment. In this case, it is the user’s responsibility to poll for `version.training_data.assignment_in_progress`. Once the assignment is finished, check for errors if `version.training_data.assignment_in_progress==False`. If `version.training_data.assignment_error` is None, then there is no error. ``` import datarobot as dr dr.Client(token=my_token, endpoint=endpoint) version = dr.CustomModelVersion.create_from_previous( custom_model_id="6444482e5583f6ee2e572265", base_environment_id="642209acc563893014a41e24", training_dataset_id="6421f2149a4f9b1bec6ad6dd", max_wait=None, ) while version.training_data.assignment_in_progress: time.sleep(10) version.refresh() if version.training_data.assignment_error: print(version.training_data.assignment_error["message"]) ``` #### Assign training data for unstructured models For unstructured models, you can provide the parameters `training_dataset_id` and `holdout_dataset_id`. The training data assignment is performed synchronously and the `max_wait` parameter is ignored. The example below shows how to create an unstructured model version with training and holdout data. ``` import datarobot as dr dr.Client(token=my_token, endpoint=endpoint) version = dr.CustomModelVersion.create_from_previous( custom_model_id="6444482e5583f6ee2e572265", base_environment_id="642209acc563893014a41e24", training_dataset_id="6421f2149a4f9b1bec6ad6dd", holdout_dataset_id="6421f2149a4f9b1bec6ad6ef", ) if version.training_data.assignment_error: print(version.training_data.assignment_error["message"]) ``` #### Remove training data By default, training and holdout data are copied to a new model version from the previous model version. If you don’t want to keep training and holdout data for the new version, set `keep_training_holdout_data` to False. ``` import datarobot as dr dr.Client(token=my_token, endpoint=endpoint) version = dr.CustomModelVersion.create_from_previous( custom_model_id="6444482e5583f6ee2e572265", base_environment_id="642209acc563893014a41e24", keep_training_holdout_data=False, ) ``` ### List custom model versions To list custom model versions available to you: ``` import datarobot as dr dr.CustomModelVersion.list(custom_model.id) >>> [CustomModelVersion('v2.0'), CustomModelVersion('v1.0')] ``` ### Retrieve a custom model version To retrieve a specific custom model version, run the code below. ``` import datarobot as dr dr.CustomModelVersion.get(custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b') >>> CustomModelVersion('v2.0') ``` ### Update a version description To update a custom model version description: ``` import datarobot as dr custom_model_version = dr.CustomModelVersion.get( custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b', ) custom_model_version.update(description='new description') custom_model_version.description >>> 'new description' ``` ### Download a version To download the contents of a custom model version as a ZIP archive: ``` import datarobot as dr path_to_download = '/home/user/Documents/myModel.zip' custom_model_version = dr.CustomModelVersion.get( custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b', ) custom_model_version.download(path_to_download) ``` ### Start custom model inference legacy conversion Custom model versions may include SAS files, with a main program entry point. In order to use a model, a conversion must be run. The conversion can later be fetched and examined by reading the conversion print-outs. By default, a conversion is initiated in a non-blocking mode. If a `max_wait` parameter is provided, than the call is blocked until the conversion is completed. The results can than be read by fetching the conversion entity. ``` import datarobot as dr # Read a custom model version custom_model_version = dr.CustomModelVersion.get(model_id, model_version_id) # Find the main program item ID main_program_item_id = None for item in cm_ver.items: if item.file_name.lower().endswith('.sas'): main_program_item_id = item.id # Execute the conversion if async: # This is a non-blocking call conversion_id = dr.models.CustomModelVersionConversion.run_conversion( custom_model_version.custom_model_id, custom_model_version.id, main_program_item_id, ) else: # This call is blocked until a completion or a timeout conversion_id = dr.models.CustomModelVersionConversion.run_conversion( custom_model_version.custom_model_id, custom_model_version.id, main_program_item_id, max_wait=60, ) ``` #### Monitor model conversion If a custom model version conversion was initiated in a non-blocking mode, it is possible to monitor the progress as follows: ``` import datarobot as dr while True: conversion = dr.models.CustomModelVersionConversion.get( custom_model_id, custom_model_version_id, conversion_id, ) if conversion.conversion_in_progress: logging.info('Conversion is in progress...') time.sleep(1) else: if conversion.conversion_succeeded: logging.info('Conversion succeeded') else: logging.error(f'Conversion failed!\n{conversion.log_message}') break ``` #### Stop conversion It is possible to stop a custom model version conversion that is in progress. The call is non-blocking and you may keep monitoring the conversion progress (see above) until is it completed. ``` import datarobot as dr dr.models.CustomModelVersionConversion.stop_conversion( custom_model_id, custom_model_version_id, conversion_id, ) ``` ### Calculate Feature Impact To trigger the calculation of a custom model version’s Feature Impact, training data must be assigned to a custom model. (For more information about custom model training data, reference the [custom model documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html#custom-inference-model-assign-data).) If training data is assigned, run the following code to trigger the calculation of its Feature Impact: ``` import datarobot as dr version = dr.CustomModelVersion.get(custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b') version.calculate_feature_impact() ``` To trigger Feature Impact calculation without blocking a program, set `max_wait` to `None`: ``` import datarobot as dr version = dr.CustomModelVersion.get(custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b') version.calculate_feature_impact(max_wait=None) ``` ### Retrieve custom model image Feature Impact To retrieve a custom model image’s Feature Impact, it must be calculated beforehand. Reference the [Custom model version Feature Impact documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html#custom-model-version-calculate-feature-impact) for more information. To get Feature Impact: ``` import datarobot as dr version = dr.CustomModelVersion.get(custom_model.id, custom_model_version_id='5ebe96b84024035cc6a6560b') version.get_feature_impact() >>> [{'featureName': 'B', 'impactNormalized': 1.0, 'impactUnnormalized': 1.1085356209402688, 'redundantWith': 'B'}...] ``` ### Prepare a custom model version for use If your custom model version has dependencies, a dependency build must be completed before the model can be used. The dependency build installs your model’s dependencies into the base environment associated with the model version. ### Start a dependency build To start a custom model version dependency build: ``` import datarobot as dr build_info = dr.CustomModelVersionDependencyBuild.start_build( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, max_wait=3600, # 1 hour timeout ) build_info.build_status >>> 'success' ``` To start a custom model version dependency build without blocking a program until the test finishes, set `max_wait` to `None`: ``` import datarobot as dr build_info = dr.CustomModelVersionDependencyBuild.start_build( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, max_wait=None, ) build_info.build_status >>> 'submitted' # after some time build_info.refresh() build_info.build_status >>> 'success' ``` In case the build fails, or you are just curious, do the following to retrieve the build log once complete: ``` print(build_info.get_log()) ``` To cancel a custom model version dependency build, simply run: ``` build_info.cancel() ``` ## Manage custom model tests A custom model test represents testing performed on custom models. ### Create a custom model test To create a custom model test: ``` import datarobot as dr path_to_dataset = '/home/user/Documents/testDataset.csv' dataset = dr.Dataset.create_from_file(file_path=path_to_dataset) custom_model_test = dr.CustomModelTest.create( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, dataset_id=dataset.id, max_wait=3600, # 1 hour timeout ) custom_model_test.overall_status >>> 'succeeded' ``` or, with k8s resources: ``` import datarobot as dr path_to_dataset = '/home/user/Documents/testDataset.csv' dataset = dr.Dataset.create_from_file(file_path=path_to_dataset) custom_model_test = dr.CustomModelTest.create( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, dataset_id=dataset.id, max_wait=3600, # 1 hour timeout maximum_memory=1024*1024*1024, ) custom_model_test.overall_status >>> 'succeeded' ``` To start a custom model test without blocking a program until the test finishes, set `max_wait` to `None`: ``` import datarobot as dr path_to_dataset = '/home/user/Documents/testDataset.csv' dataset = dr.Dataset.create_from_file(file_path=path_to_dataset) custom_model_test = dr.CustomModelTest.create( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, dataset_id=dataset.id, max_wait=None, ) custom_model_test.overall_status >>> 'in_progress' # after some time custom_model_test.refresh() custom_model_test.overall_status >>> 'succeeded' ``` Running a custom model test uses the custom model version’s base image with its dependencies installed as an execution environment. To start a custom model test using an execution environment as-is, without the model’s dependencies installed, supply an environment ID and (optionally) an environment version ID: ``` import datarobot as dr path_to_dataset = '/home/user/Documents/testDataset.csv' dataset = dr.Dataset.create_from_file(file_path=path_to_dataset) custom_model_test = dr.CustomModelTest.create( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, dataset_id=dataset.id, max_wait=3600, # 1 hour timeout ) custom_model_test.overall_status >>> 'succeeded' ``` In case a test fails, do the following to examine details of the failure: ``` for name, test in custom_model_test.detailed_status.items(): print('Test: {}'.format(name)) print('Status: {}'.format(test['status'])) print('Message: {}'.format(test['message'])) print(custom_model_test.get_log()) ``` To cancel a custom model test: ``` custom_model_test.cancel() ``` To start a custom model test for an unstructured custom model, dataset details should not be provided: ``` import datarobot as dr custom_model_test = dr.CustomModelTest.create( custom_model_id=custom_model.id, custom_model_version_id=model_version.id, ) ``` ### List custom model tests To list the custom model tests available to the user: ``` import datarobot as dr dr.CustomModelTest.list(custom_model_id=custom_model.id) >>> [CustomModelTest('5ec262604024031bed5aaa16')] ``` ### Retrieve a custom model test To retrieve a specific custom model test: ``` import datarobot as dr dr.CustomModelTest.get(custom_model_test_id='5ec262604024031bed5aaa16') >>> CustomModelTest('5ec262604024031bed5aaa16') ``` --- # Data exports URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/data_exports.html > Export prediction, actuals, training, and data quality data from deployments with the Python API client. Use deployment data export to retrieve data sent for predictions along with the associated predictions. ## Prediction data export The following sections outline how to manage prediction data exports. ### Create a prediction data export To create a prediction data export, use `PredictionDataExport.create`, defining the time window to include in the export using the `start` and `end` parameters: ``` from datetime import datetime, timedelta from datarobot.models.deployment import PredictionDataExport now=datetime.now() prediction_data_export = PredictionDataExport.create( deployment_id='5c939e08962d741e34f609f0', start=now - timedelta(days=7), end=now) ``` Specify the model ID for export. Otherwise, the champion model ID is used by default: ``` from datetime import datetime, timedelta from datarobot.models.deployment import PredictionDataExport now=datetime.now() prediction_data_export = PredictionDataExport.create( deployment_id='5c939e08962d741e34f609f0', model_id='6444482e5583f6ee2e572265', start=now - timedelta(days=7), end=now ) ``` For deployments in batch mode, provide batch IDs to export prediction data for those batches: ``` from datetime import datetime, timedelta from datarobot.models.deployment import PredictionDataExport now=datetime.now() prediction_data_export = PredictionDataExport.create( deployment_id='5c939e08962d741e34f609f0', model_id='6444482e5583f6ee2e572265', start=now - timedelta(days=7), end=now, batch_ids=['6572db2c9f9d4ad3b9de33d0', '6572db2c9f9d4ad3b9de33d0'] ) ``` The `start` and `end` of the export can be defined as a datetime or string type. ### List prediction data exports To list prediction data exports, use `PredictionDataExport.list`: ``` from datarobot.models.deployment import PredictionDataExport prediction_data_exports = PredictionDataExport.list(deployment_id='5c939e08962d741e34f609f0', limit=0) prediction_data_exports >>> [PredictionDataExport('65fbe59aaa3f847bd5acc75b'), PredictionDataExport('65fbe59aaa3f847bd5acc75c'), PredictionDataExport('65fbe59aaa3f847bd5acc75a')] ``` To list all prediction data exports, set the limit to `0`. Adjust additional parameters to filter the data as needed: ``` from datarobot.enums import ExportStatus from datarobot.models.deployment import PredictionDataExport prediction_data_exports = PredictionDataExport.list(deployment_id='5c939e08962d741e34f609f0', limit=100, offset=100) # Use additional filters prediction_data_exports = PredictionDataExport.list( deployment_id='5c939e08962d741e34f609f0', model_id="6444482e5583f6ee2e572265", batch=False, status=ExportStatus.FAILED ) ``` ### Retrieve a prediction data export To get a prediction data export by identifier, use `PredictionDataExport.get`: ``` from datarobot.models.deployment import PredictionDataExport prediction_data_export = PredictionDataExport.get( deployment_id='5c939e08962d741e34f609f0', export_id='65fbe59aaa3f847bd5acc75b' ) prediction_data_exports >>> PredictionDataExport('65fbe59aaa3f847bd5acc75b') ``` ### Fetch prediction export datasets To return data from a prediction export as `dr.Dataset`, use the `fetch_data` method. This method can return a list of datasets; however, usually it returns one dataset. There are cases, like time series, when more than one element is returned. The obtained dataset (or datasets) can be transformed into, for example, a pandas DataFrame. ``` from datarobot.models.deployment import PredictionDataExport prediction_data_export = PredictionDataExport.get( deployment_id='5c939e08962d741e34f609f0', export_id='65fbe59aaa3f847bd5acc75b' ) prediction_datasets = prediction_data_export.fetch_data() prediction_datasets >>> [Dataset(name='Deployment prediction data', id='65f240b0e37a9f1a104bf450')] prediction_dataset = prediction_datasets[0] df = prediction_dataset.get_as_dataframe() df.head(2) >>> DR_RESERVED_PREDICTION_TIMESTAMP ... upstream_x_datarobot_version 0 2024-03-13 23:00:38.998000+00:00 ... predictionapi/X/X 1 2024-03-13 23:00:38.998000+00:00 ... predictionapi/X/X ``` ## Actuals data export The following examples outline how to manage actuals data exports. ### Create actuals data export To create an actuals data export, use `ActualsDataExport.create`, defining the time window to include in the export using the `start` and `end` parameters: ``` from datetime import datetime, timedelta from datarobot.models.deployment import ActualsDataExport now=datetime.now() actuals_data_export = ActualsDataExport.create( deployment_id='5c939e08962d741e34f609f0', start=now - timedelta(days=7), end=now ) ``` Specify the model ID for export. Otherwise, the champion model ID is used by default: ``` from datetime import datetime, timedelta from datarobot.models.deployment import ActualsDataExport now=datetime.now() actuals_data_export = ActualsDataExport.create( deployment_id='5c939e08962d741e34f609f0', model_id="6444482e5583f6ee2e572265", start=now - timedelta(days=7), end=now, ) ``` To export only actuals that are matched to predictions, set `only_matched_predictions` to `True`; by default all available actuals are exported. The `start` and `end` of the export can be defined as a `datetime` or `string` type. ``` from datetime import datetime, timedelta from datarobot.models.deployment import ActualsDataExport now=datetime.now() actuals_data_export = ActualsDataExport.create( deployment_id='5c939e08962d741e34f609f0', only_matched_predictions=True, start=now - timedelta(days=7), end=now, ) ``` ### List actuals data exports To list actuals data exports, use `ActualsDataExport.list`: ``` from datarobot.models.deployment import ActualsDataExport actuals_data_exports = ActualsDataExport.list(deployment_id='5c939e08962d741e34f609f0', limit=0) actuals_data_exports >>> [ActualsDataExport('660456a332d0081029ee5031'), ActualsDataExport('660456a332d0081029ee5032'), ActualsDataExport('660456a332d0081029ee5033')] ``` To list all actuals data exports, set the limit to `0`. Adjust additional parameters to filter the data as needed: ``` from datarobot.enums import ExportStatus from datarobot.models.deployment import ActualsDataExport # use additional filters actuals_data_exports = ActualsDataExport.list( deployment_id='5c939e08962d741e34f609f0', offset=500, limit=50, status=ExportStatus.SUCCEEDED ) ``` ### Retrieve actuals data export To get actuals data export by identifier, use `ActualsDataExport.get`, as in the following example: ``` from datarobot.models.deployment import ActualsDataExport actuals_data_export = ActualsDataExport.get( deployment_id='5c939e08962d741e34f609f0', export_id='660456a332d0081029ee4031' ) actuals_data_export >>> ActualsDataExport('660456a332d0081029ee4031') ``` ### Fetch actuals export datasets To return data from actuals export as `dr.Dataset`, use the `fetch_data` method: ``` from datarobot.models.deployment import ActualsDataExport actuals_data_export = ActualsDataExport.get( deployment_id='5c939e08962d741e34f609f0', export_id='660456a332d0081029ee4031' ) actuals_datasets = actuals_data_export.fetch_data() actuals_datasets >>> [Dataset(name='Deployment prediction data', id='65f240b0e37a9f1a104bf450')] actuals_dataset = actuals_datasets[0] df = actuals_dataset.get_as_dataframe() df.head(2) >>> association_id timestamp actuals predictions 0 1 2024-03-20 15:00:00+00:00 21.0 18.125388 1 10 2024-03-20 15:00:00+00:00 12.0 22.805252 ``` This method may return a list of datasets; however, it usually returns one dataset. The obtained dataset (or datasets) can be transformed into, for example, a pandas DataFrame. ## Training data export The following examples outline how to manage training data exports. ### Create training data export To create a training data export, use `TrainingDataExport.create` and define the deployment ID: ``` from datarobot.models.deployment import TrainingDataExport dataset_id = TrainingDataExport.create(deployment_id='5c939e08962d741e34f609f0') ``` Specify the model ID for export. Otherwise, the champion model ID is used by default: ``` from datarobot.models.deployment import TrainingDataExport dataset_id = TrainingDataExport.create( deployment_id='5c939e08962d741e34f609f0', model_id='6444482e5583f6ee2e572265') dataset_id >>> 65fb0c25019ca3333bbb4c10 ``` This method returns the ID of the dataset that contains the training data. This dataset is saved in the AI Catalog. ### List training data exports To list training data exports, use `TrainingDataExport.list`: ``` from datarobot.models.deployment import TrainingDataExport training_data_exports = TrainingDataExport.list(deployment_id='5c939e08962d741e34f609f0') training_data_exports >>> [TrainingDataExport('6565fbf2356124f1daa3acc522')] ``` ### Retrieve a training data export To get training data export by identifier, use `TrainingDataExport.get`. ``` from datarobot.models.deployment import ActualsDataExport training_data_export = TrainingDataExport.get( deployment_id='5c939e08962d741e34f609f0', export_id='65fbf2356124f1daa3acc522' ) training_data_export >>> TrainingDataExport('6565fbf2356124f1daa3acc522') ``` ### Fetch a training export dataset To return data from the training export as `dr.Dataset`, use `fetch_data`. This method returns a single training dataset. The obtained dataset can be transformed into, for example, a pandas DataFrame. ``` from datarobot.models.deployment import TrainingDataExport training_data_export = TrainingDataExport.get( deployment_id='5c939e08962d741e34f609f0', export_id='660456a332d0081029ee4031' ) training_dataset = training_data_export.fetch_data() training_dataset >>> [Dataset(name='training-data-10k_diabetes.csv', id='65fb0c25019ca3333bbb4c10')] df = training_dataset.get_as_dataframe() df.head(2) >>> acetohexamide time_in_hospital ... number_outpatient payer_code 0 No 1 ... 0 YY 1 No 2 ... 0 XX ``` ## Data quality export The data-quality exports provide feedback on LLM deployments. It is intended to be used in conjunction with custom-metrics for prompt monitoring. ### Data quality export list To list data quality exports, use `DataQualityExport.list`: The `start` and `end` of the export can be defined as a `datetime` or `string` type. There are many options for filtering and ordering the data. ``` from datetime import datetime, timedelta from datarobot.models.deployment import DataQualityExport now=datetime.now() data_quality_exports = DataQualityExport.list( deployment_id='66903c40f18e6ec90fd7c8c7', start=now - timedelta(days=1), end=now, ) data_quality_exports >>> [DataQualityExport(6447ca39c6a04df6b5b0ed19c6101e3c), ... DataQualityExport(0ff46fd3636545a9ac3e15ee1dbd8638)] data_quality_deports[0].metrics >>> [{'id': '669688f90a23524131e2d301', 'name': 'metric 3', 'value': None}, {'id': '669688e633ae1ffce40eb2f8', 'name': 'metric 2', 'value': 45.0}, {'id': '669688d282c9384ab8068a6c', 'name': 'metric 1', 'value': 178.0}] ``` --- # Deployments URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/deployment.html > Deploy, manage, and monitor models with the Python API client. This page outlines how you can deploy, manage, and monitor models with the Python API client. ## Manage deployments The following commands can be used to manage deployments. ### Create a deployment A new deployment can be created from: - A DataRobot model - use create_from_registered_model_version() . Refer to the Model Registry documentation to reference how to create a registered model version. When creating a new deployment, you must provide a DataRobot `registered_model_version_id` (also known as `model_package_id`) and a `label`. Optionally, provide a `description` to document the purpose of the deployment. The default prediction server is used when making predictions against the deployment and is required for creating a deployment on DataRobot in managed SaaS environments. For Self-Managed users, you cannot provide a default prediction server. Instead use a pre-configured prediction server. Refer to [datarobot.PredictionServer.list](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.PredictionServer.list) for more information on retrieving available prediction servers. ``` import datarobot as dr project = dr.Project.get('6527eb38b9e5dead5fc12491') model = project.get_models()[0] prediction_server = dr.PredictionServer.list()[0] registered_model_version = dr.RegisteredModelVersion.create_for_leaderboard_item( model_id=model.id, name="Name of the version(aka model package)", registered_model_name='Name of the registered model unique across the org ' ) deployment = dr.Deployment.create_from_registered_model_version( registered_model_version.id, label='New Deployment', description='A new deployment', default_prediction_server_id=prediction_server.id) >>> Deployment('New Deployment') ``` ### List deployments To list deployments a user can view: ``` import datarobot as dr deployments = dr.Deployment.list() deployments >>> [Deployment('New Deployment'), Deployment('Previous Deployment')] ``` Refer to [Deployment](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment) to learn about the properties of the deployment object. You can also filter the deployments that are returned by passing an instance of the [DeploymentListFilters](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.DeploymentListFilters) class to the `filters` keyword argument. ``` import datarobot as dr filters = dr.models.deployment.DeploymentListFilters( role='OWNER', accuracy_health=dr.enums.DEPLOYMENT_ACCURACY_HEALTH_STATUS.FAILING ) deployments = dr.Deployment.list(filters=filters) deployments >>> [Deployment('Deployment Owned by Me w/ Failing Accuracy 1'), Deployment('Deployment Owned by Me w/ Failing Accuracy 2')] ``` ### Retrieve a deployment It is possible to retrieve a single deployment with its identifier, rather than list all deployments: ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.id >>> '5c939e08962d741e34f609f0' deployment.label >>> 'New Deployment' ``` Refer to [Deployment](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment) to learn about the properties of the deployment object. ### Update a deployment To update a deployment’s label and description: ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update(label='new label') ``` ### Delete a deployment To mark a deployment as deleted: ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.delete() ``` ### Activate or deactivate a deployment To activate a deployment: ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.activate() deployment.status >>> 'active' ``` To deactivate a deployment: ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.deactivate() deployment.status >>> 'inactive' ``` ### Make batch predictions with a deployment DataRobot provides a small utility function to make batch predictions using a deployment: [Deployment.predict_batch](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.predict_batch). ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') # To note: `source` can be a file path, a file, or a pandas DataFrame prediction_results_as_dataframe = deployment.predict_batch( source="./my_local_file.csv", ) ``` ## Model replacement A deployment’s model can be replaced effortlessly with zero interruption of predictions. Model replacement is an asynchronous process, which means some preparatory work may be performed after the initial request is completed. Predictions made against this deployment will start using the new model as soon as the request is completed. There will be no interruption for predictions throughout the process. The [replace_model()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.replace_model) function won’t return until the asynchronous process is fully finished. Alongside the identifier of the new model, a `reason` is also required. The reason is stored in model history of the deployment for documentation purposes. An enum, `MODEL_REPLACEMENT_REASON`, is provided for convenience. All possible values are documented below: - MODEL_REPLACEMENT_REASON.ACCURACY - MODEL_REPLACEMENT_REASON.DATA_DRIFT - MODEL_REPLACEMENT_REASON.ERRORS - MODEL_REPLACEMENT_REASON.SCHEDULED_REFRESH - MODEL_REPLACEMENT_REASON.SCORING_SPEED - MODEL_REPLACEMENT_REASON.OTHER Below is an example of model replacement: ``` import datarobot as dr from datarobot.enums import MODEL_REPLACEMENT_REASON project = dr.Project.get('5cc899abc191a20104ff446a') model = project.get_models()[0] deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.model['id'], deployment.model['type'] >>> ('5c0a979859b00004ba52e431', 'Decision Tree Classifier (Gini)') deployment.replace_model('5c0a969859b00004ba52e41b', MODEL_REPLACEMENT_REASON.ACCURACY) deployment.model['id'], deployment.model['type'] >>> ('5c0a969859b00004ba52e41b', 'Support Vector Classifier (Linear Kernel)') ``` ### Validation Before initiating the model replacement request, it is usually a good idea to use the [validate_replacement_model()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.validate_replacement_model) function to validate if the new model can be used as a replacement. The [validate_replacement_model()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.validate_replacement_model) function returns the validation status, a message and a checks dictionary. If the status is ‘passing’ or ‘warning’, use [replace_model()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.replace_model) to perform model the replacement. If status is ‘failing’, refer to the `checks` dict for more details on why the new model cannot be used as a replacement. ``` import datarobot as dr project = dr.Project.get('5cc899abc191a20104ff446a') model = project.get_models()[0] deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') status, message, checks = deployment.validate_replacement_model(new_model_id=model.id) status >>> 'passing' # `checks` can be inspected for detail, showing two examples here: checks['target'] >>> {'status': 'passing', 'message': 'Target is compatible.'} checks['permission'] >>> {'status': 'passing', 'message': 'User has permission to replace model.'} ``` ## Monitoring Deployment monitoring can be categorized into several area of concerns: - Service stats over time - Accuracy over time With a [Deployment](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment) object, `get` functions are provided so that you can query monitoring data. Alternatively, it is also possible to retrieve monitoring data directly using a deployment ID: ``` from datarobot.models import Deployment, ServiceStats deployment_id = '5c939e08962d741e34f609f0' # Call `get` functions on a `Deployment` object deployment = Deployment.get(deployment_id) service_stats = deployment.get_service_stats() # Directly fetch without a `Deployment` object service_stats = ServiceStats.get(deployment_id) ``` When querying monitoring data, you can optionally provide a start and end time (accepted as either a `datetime` object or a `string`). Note that only top of the hour datetimes are accepted. For example: `2019-08-01T00:00:00Z`. By default, the end time of the query will be the next top of the hour, the start time will be 7 days before the end time. In the over time variants, an optional `bucket_size` can be provided to specify the resolution of time buckets. For example, if the start time is `2019-08-01T00:00:00Z`, the end time is `2019-08-02T00:00:00Z`, and the `bucket_size` is `T1H`, then 24 time buckets are generated, each providing data calculated over one hour. Use [construct_duration_string()](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.helpers.partitioning_methods.construct_duration_string) to help construct a bucket size string. > NOTE¶The minimum bucket size is one hour. ### Service stats Service stats are metrics tracking deployment utilization and how well deployments respond to prediction requests. Use `SERVICE_STAT_METRIC.ALL` to retrieve a list of supported metrics. [ServiceStats](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.ServiceStats) retrieves values for all service stats metrics.[ServiceStatsOverTime](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.ServiceStatsOverTime) can be used to fetch how one single metric changes over time. ``` from datetime import datetime from datarobot.enums import SERVICE_STAT_METRIC from datarobot.helpers.partitioning_methods import construct_duration_string from datarobot.models import Deployment deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') service_stats = deployment.get_service_stats( start_time=datetime(2019, 8, 1, hour=15), end_time=datetime(2019, 8, 8, hour=15) ) service_stats[SERVICE_STAT_METRIC.TOTAL_PREDICTIONS] >>> 12597 total_predictions = deployment.get_service_stats_over_time( start_time=datetime(2019, 8, 1, hour=15), end_time=datetime(2019, 8, 8, hour=15), bucket_size=construct_duration_string(days=1), metric=SERVICE_STAT_METRIC.TOTAL_PREDICTIONS ) total_predictions.bucket_values >>> OrderedDict([(datetime.datetime(2019, 8, 1, 15, 0, tzinfo=tzutc()), 1610), (datetime.datetime(2019, 8, 2, 15, 0, tzinfo=tzutc()), 2249), (datetime.datetime(2019, 8, 3, 15, 0, tzinfo=tzutc()), 254), (datetime.datetime(2019, 8, 4, 15, 0, tzinfo=tzutc()), 943), (datetime.datetime(2019, 8, 5, 15, 0, tzinfo=tzutc()), 1967), (datetime.datetime(2019, 8, 6, 15, 0, tzinfo=tzutc()), 2810), (datetime.datetime(2019, 8, 7, 15, 0, tzinfo=tzutc()), 2775)]) ``` ### Data drift Data drift measures how much the distribution of target or a feature has changed comparing to the training data. Deployment’s target drift and feature drift can be retrieved separately using [datarobot.models.deployment.TargetDrift](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.TargetDrift) and [datarobot.models.deployment.FeatureDrift](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.FeatureDrift). Use `DATA_DRIFT_METRIC.ALL` to retrieve a list of supported metrics. ``` from datetime import datetime from datarobot.enums import DATA_DRIFT_METRIC from datarobot.models import Deployment, FeatureDrift deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') target_drift = deployment.get_target_drift( start_time=datetime(2019, 8, 1, hour=15), end_time=datetime(2019, 8, 8, hour=15) ) target_drift.drift_score >>> 0.00408514 feature_drift_data = FeatureDrift.list( deployment_id='5c939e08962d741e34f609f0', start_time=datetime(2019, 8, 1, hour=15), end_time=datetime(2019, 8, 8, hour=15), metric=DATA_DRIFT_METRIC.HELLINGER ) feature_drift = feature_drift_data[0] feature_drift.name >>> 'age' feature_drift.drift_score >>> 4.16981594 ``` #### Predictions over time Predictions over time gives insight on how deployment’s prediction response has changed over time. Different data can be retrieved in each bucket, depending on deployment’s target type: - row_count : The number of rows in the bucket, available for all target types. - mean_predicted_value : The average of the predicted values for all rows in the bucket. Available for regression target type. - mean_probabilities : The mean of the predicted probability for each class. Available for binary or multiclass classification target types. - class_distribution : The count and percent of the predicted class labels. Available for binary or multiclass classification target types. - percentiles : The 10th and 90th percentile of a predicted value or positive class probability. Available for regression and binary target types. ``` from datetime import datetime from datarobot.enums import BUCKET_SIZE from datarobot.models import Deployment # Deployment with regression target type deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') predictions_over_time = deployment.get_predictions_over_time( start_time=datetime(2023, 4, 1), end_time=datetime(2023, 4, 30), bucket_size=BUCKET_SIZE.P1D, ) predicted = [bucket['mean_predicted_value'] for bucket in predictions_over_time.buckets] predicted >>> [0.3772, 0.6642, ...., 0.7937] # Deployment with binary target type deployment = Deployment.get(deployment_id='62fff28a0f5fee488587ce92') predictions_over_time = deployment.get_predictions_over_time( start_time=datetime(2023, 4, 1), end_time=datetime(2023, 4, 22), bucket_size=BUCKET_SIZE.P7D, ) predicted = [ {item['class_name']: item['value'] for item in bucket['mean_probabilities']}.get('True') for bucket in predictions_over_time.buckets ] predicted >>> [0.3955, 0.4274, None] ``` ### Accuracy A collection of metrics are provided to measure the accuracy of a deployment’s predictions. For deployed classification models, use `ACCURACY_METRIC.ALL_CLASSIFICATION` for all supported metrics; for deployed regression models, use `ACCURACY_METRIC.ALL_REGRESSION` instead. [Accuracy](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.Accuracy) and [AccuracyOverTime](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.AccuracyOverTime) are provided to retrieve all default accuracy metrics and measure how one single metric changes over time. ``` from datetime import datetime from datarobot.enums import ACCURACY_METRIC from datarobot.helpers.partitioning_methods import construct_duration_string from datarobot.models import Deployment deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') accuracy = deployment.get_accuracy( start_time=datetime(2019, 8, 1, hour=15), end_time=datetime(2019, 8, 1, 15, 0) ) accuracy[ACCURACY_METRIC.RMSE] >>> 943.225 rmse = deployment.get_accuracy_over_time( start_time=datetime(2019, 8, 1), end_time=datetime(2019, 8, 3), bucket_size=construct_duration_string(days=1), metric=ACCURACY_METRIC.RMSE ) rmse.bucket_values >>> OrderedDict([(datetime.datetime(2019, 8, 1, 15, 0, tzinfo=tzutc()), 1777.190657), (datetime.datetime(2019, 8, 2, 15, 0, tzinfo=tzutc()), 1613.140772)]) ``` It is also possible to retrieve how multiple metrics changed over the same period of time, enabling easier side-by-side comparison across different metrics. ``` from datarobot.enums import ACCURACY_METRIC from datarobot.models import Deployment accuracy_over_time = AccuracyOverTime.get_as_dataframe( ram_app.id, [ACCURACY_METRIC.RMSE, ACCURACY_METRIC.GAMMA_DEVIANCE, ACCURACY_METRIC.MAD]) ``` #### Predictions vs. actuals over time Predictions vs. actuals over time can be used to analyze how deployment’s predictions compare against actuals. Different data can be retrieved in each bucket, depending on deployment’s target type: - row_count_total : The number of rows with or without actuals in the bucket. Available for all target types. - row_count_with_actual : The number of rows with actuals in the bucket. Available for all target types. - mean_predicted_value : The mean of the predicted value for all rows matched with an actual in the bucket. Available for the regression target type. - mean_actual_value : The mean of the actual value for all rows in the bucket. Available for the regression target type. - predicted_class_distribution : The count and percent of predicted class labels. Available for binary and multiclass classification target types. - actual_class_distribution : The count and percent of actual class labels. Available for binary or multiclass classification target types. ``` from datetime import datetime from datarobot.enums import BUCKET_SIZE from datarobot.models import Deployment # Deployment with the regression target type deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') predictions_over_time = deployment.get_predictions_vs_actuals_over_time( start_time=datetime(2023, 4, 1), end_time=datetime(2023, 4, 30), bucket_size=BUCKET_SIZE.P1D, ) predicted = [bucket['mean_actual_value'] for bucket in predictions_over_time.buckets] predicted >>> [0.2806, 0.9170, ...., 0.0314] # Deployment with the binary target type deployment = Deployment.get(deployment_id='62fff28a0f5fee488587ce92') predictions_over_time = deployment.get_predictions_vs_actuals_over_time( start_time=datetime(2023, 4, 1), end_time=datetime(2023, 4, 22), bucket_size=BUCKET_SIZE.P7D, ) predicted = [ {item['class_name']: item['value'] for item in bucket['mean_predicted_value']}.get('True') for bucket in predictions_over_time.buckets ] predicted >>> [0.5822, 0.6305, None] ``` ### Delete data Monitoring data accumulated on a deployment can be deleted using [delete_monitoring_data()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.delete_monitoring_data). A start and end timestamp could be provided to limit data deletion to certain time period. #### WARNING Monitoring data is not recoverable once deleted. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.delete_monitoring_data(model_id=deployment.model['id']) ``` ### List deployment prediction data exports Prediction data exports for a deployment can be retrieved using [list_prediction_data_exports()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.list_prediction_data_exports). ``` from datarobot.enums import ExportStatus from datarobot.models import Deployment deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') prediction_data_exports = deployment.list_prediction_data_exports(limit=0) prediction_data_exports >>> [PredictionDataExport('65fbe59aaa3f847bd5acc75b'), PredictionDataExport('65fbe59aaa3f847bd5acc75c'), PredictionDataExport('65fbe59aaa3f847bd5acc75a')] ``` To list all prediction data exports, set the limit to 0. Adjust additional parameters to filter the data as needed: ``` from datarobot.enums import ExportStatus from datarobot.models import Deployment deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') prediction_data_exports = deployment.list_prediction_data_exports( model_id="6444482e5583f6ee2e572265", batch=False, status=ExportStatus.SUCCEEDED, limit=100, offset=50, ) ``` ### List deployment actuals data exports Actuals data exports for a deployment can be retrieved using [list_actuals_data_exports()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.list_actuals_data_exports). ``` from datarobot.enums import ExportStatus from datarobot.models import Deployment deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') actuals_data_exports = deployment.list_actuals_data_exports(limit=0) actuals_data_exports >>> [ActualsDataExport('660456a332d0081029ee5031'), ActualsDataExport('660456a332d0081029ee5032'), ActualsDataExport('660456a332d0081029ee5033')] ``` To list all actuals data exports, set the limit to `0`. Adjust additional parameters to filter the data as needed: ``` from datarobot.enums import ExportStatus from datarobot.models import Deployment deployment = Deployment.get(deployment_id='5c939e08962d741e34f609f0') actuals_data_exports = deployment.list_actuals_data_exports( deployment_id='5c939e08962d741e34f609f0', offset=500, limit=50, status=ExportStatus.SUCCEEDED ) ``` ### List deployment training data exports To retrieve successful training data exports for a deployment, use [list_training_data_exports()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.list_training_data_exports). ``` from datarobot.models.deployment import TrainingDataExport training_data_exports = TrainingDataExport.list(deployment_id='5c939e08962d741e34f609f0') training_data_exports >>> [TrainingDataExport('6565fbf2356124f1daa3acc522')] ``` ### List deployment data quality exports To retrieve successful data quality exports for a deployment, use [list_data_quality_exports()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.list_data_quality_exports). ``` from datarobot.models import Deployment deployment = Deployment.get('66903c40f18e6ec90fd7c8c7') data_quality_exports = deployment.list_data_quality_exports(start='2024-07-01', end='2024-08-01') data_quality_exports >>> [DataQualityExport(6447ca39c6a04df6b5b0ed19c6101e3c), ... DataQualityExport(0ff46fd3636545a9ac3e15ee1dbd8638)] ``` There are many filtering and sorting options available. ### Segment analysis Segment analysis is a deployment utility that filters service stats, data drift, and accuracy statistics into unique segment attributes and values. Use [get_segment_attributes()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_segment_attributes) to retrieve segment analysis data. Use [get_segment_values()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_segment_values) to retrieve segment value data. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') segment_attributes_service_health = deployment.get_segment_attributes(DEPLOYMENT_MONITORING_TYPE.SERVICE_HEALTH) >>>['DataRobot-Consumer', 'DataRobot-Host-IP', 'DataRobot-Remote-IP'] segment_attributes_data_drift = deployment.get_segment_attributes(DEPLOYMENT_MONITORING_TYPE.DATA_DRIFT) >>>['DataRobot-Consumer', 'attribute_1', 'attribute_2'] segment_values = deployment.get_segment_values(segmentAttribute=ReservedSegmentAttributes.CONSUMER) >>>['DataRobot-Consumer', 'datarobotuser@email.com'] ``` ## Challengers Challenger models can be used to compare the currently deployed model (the “champion” model) to another model. The following functions can be used to manage deployment’s challenger models: - List: list_challengers() or list() . - Create: create() . - Get: get() . - Update: update() . - Delete: delete() . ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') challenger = deployment.list_challengers()[-1] challenger.update(name='New Challenger Name') challenger.name >>> 'New Challenger Name' ``` ### Settings Use [get_challenger_models_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_challenger_models_settings) and [update_challenger_models_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_challenger_models_settings) to retrieve and update challenger model settings. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update_challenger_models_settings(challenger_models_enabled=True) settings = deployment.get_challenger_models_settings() settings >>> {'enabled': True} ``` Use [get_challenger_replay_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_challenger_replay_settings) and [update_challenger_replay_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_challenger_replay_settings) to retrieve and update challenger replay settings. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update_challenger_replay_settings(enabled=True) settings = deployment.get_challenger_replay_settings() settings['enabled'] >>> True ``` ## Settings Review the sections below to learn how to manage a deployment’s settings. ### Drift tracking settings Drift tracking is used to help analyze and monitor the performance of a model after it is deployed. When the model of a deployment is replaced drift tracking status will not be altered. Use [get_drift_tracking_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_drift_tracking_settings) to retrieve the current tracking status for target drift and feature drift. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') settings = deployment.get_drift_tracking_settings() settings >>> {'target_drift': {'enabled': True}, 'feature_drift': {'enabled': True}} ``` Use [update_drift_tracking_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_drift_tracking_settings) to update target drift and feature drift tracking status. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update_drift_tracking_settings(target_drift_enabled=True, feature_drift_enabled=True) ``` ### Association ID settings Association ID is used to identify predictions, so that when actuals are acquired, accuracy can be calculated. Use [get_association_id_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_association_id_settings) to retrieve current association ID settings. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') settings = deployment.get_association_id_settings() settings >>> {'column_names': ['application_id'], 'required_in_prediction_requests': True} ``` Use [update_association_id_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_association_id_settings) to update association ID settings. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update_association_id_settings(column_names=['application_id'], required_in_prediction_requests=True) ``` ### Predictions by forecast date Forecast date setting for the deployment. Use [get_predictions_by_forecast_date_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_predictions_by_forecast_date_settings) to retrieve current predictions by forecast date settings. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') settings = deployment.get_predictions_by_forecast_date_settings() settings >>> {'enabled': False, 'column_name': 'date (actual)', 'datetime_format': '%Y-%m-%d'} ``` Use [update_predictions_by_forecast_date_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_predictions_by_forecast_date_settings) to update predictions by forecast date settings. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update_predictions_by_forecast_date_settings( enable_predictions_by_forecast_date=True, forecast_date_column_name='date (actual)', forecast_date_format='%Y-%m-%d') ``` ### Health settings Health settings APIs can be used to customize definitions for deployment health status. Use [get_health_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_health_settings) to retrieve current health settings, and [get_default_health_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_default_health_settings) to retrieve default health settings. To perform updates, use [update_health_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_health_settings). ``` import datarobot as dr # Get current data drift threshold deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') settings = deployment.get_health_settings() settings['data_drift']['drift_threshold'] >>> 0.15 # Update accuracy health metric settings['accuracy']['metric'] = 'AUC' settings = deployment.update_health_settings(accuracy=settings['accuracy']) settings['accuracy']['metric'] >>> 'AUC' # Set accuracy health metric to default default_settings = deployment.get_default_health_settings() settings = deployment.update_health_settings(accuracy=default_settings['accuracy']) settings['accuracy']['metric'] >>> 'LogLoss' ``` ### Segment analysis settings Segment analysis is a deployment utility that filters data drift and accuracy statistics into unique segment attributes and values. Use [get_segment_analysis_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_segment_analysis_settings) to retrieve current segment analysis settings. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') settings = deployment.get_segment_analysis_settings() settings >>> {'enabled': False, 'attributes': []} ``` Use [update_segment_analysis_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_segment_analysis_settings) to update segment analysis settings. Any categorical column can be a segment attribute. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update_segment_analysis_settings( segment_analysis_enabled=True, segment_analysis_attributes=["country_code", "is_customer"]) ``` ### Predictions data collection settings Predictions data collection configures whether prediction requests and results should be saved to predictions data storage. Use [get_predictions_data_collection_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_predictions_data_collection_settings) to retrieve current settings of predictions data collection. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') settings = deployment.get_predictions_data_collection_settings() settings >>> {'enabled': True} ``` Use [update_predictions_data_collection_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_predictions_data_collection_settings) to update predictions data collection settings. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update_predictions_data_collection_settings(enabled=True) ``` ### Prediction warning settings Prediction warning is used to enable Humble AI for a deployment which determines if a model is misbehaving when a prediction goes outside of the calculated boundaries. Use [get_prediction_warning_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_prediction_warning_settings) to retrieve the current prediction warning settings. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') settings = deployment.get_prediction_warning_settings() settings >>> { {'enabled': True}, 'custom_boundaries': {'upper': 1337, 'lower': 0} } ``` Use [update_prediction_warning_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_prediction_warning_settings) to update current prediction warning settings. ``` import datarobot as dr # Set custom boundaries deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') deployment.update_prediction_warning_settings( prediction_warning_enabled=True, use_default_boundaries=False, lower_boundary=1337, upper_boundary=2000, ) # Reset boundaries deployment.update_prediction_warning_settings( prediction_warning_enabled=True, use_default_boundaries=True, ) ``` ### Secondary dataset configuration settings The secondary dataset configuration for a deployed Feature Discovery model can be replaced and retrieved. Secondary dataset configuration is used to specify which secondary datasets to use during prediction for a given deployment. Use [update_secondary_dataset_config()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_secondary_dataset_config) to update the secondary dataset configuration. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') config = deployment.update_secondary_dataset_config(secondary_dataset_config_id='5f48cb94408673683eca0fab') config >>> '5f48cb94408673683eca0fab' ``` Use [get_secondary_dataset_config()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_secondary_dataset_config) to get the secondary dataset config. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') config = deployment.get_secondary_dataset_config() config >>> '5f48cb94408673683eca0fab' ``` ### Share deployments You can grant or revoke other users’ access to a deployment. #### Access levels For deployments, there are 3 access levels: `OWNER` - Allows all actions on a deployment. `USER` - Can see the deployment in the DataRobot UI and see the prediction statistics of the deployment, but cannot edit or delete the deployment. `CONSUMER` - Can only make predictions on the deployment. Cannot see the deployment in the DataRobot UI or retrieve prediction statistics for the deployment in the API. #### Sharing Use [list_shared_roles()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.list_shared_roles) to get a list of users, groups, and organizations that currently have a role on the project. Each role will be returned as a [datarobot.models.deployment.DeploymentSharedRole](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.DeploymentSharedRole). ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') roles = deployment.list_shared_roles() [role.to_dict() for role in roles] >>> [{'role': 'OWNER', 'id': '5c939e08962d741e34f609f0', 'share_recipient_type': 'user', 'name': 'user@datarobot.com'}, {'role': 'USER', 'id': '5c939e08962d741e34f609f1', 'share_recipient_type': 'group', 'name': 'Example Group'}, {'role': 'CONSUMER', 'id': '5c939e08962d741e34f609f2', 'share_recipient_type': 'organization', 'name': 'Example Org'}] ``` Use [update_shared_roles()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_shared_roles) to grant and revoke roles on the deployment. This function takes a list of [datarobot.models.deployment.DeploymentGrantSharedRoleWithId](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.DeploymentGrantSharedRoleWithId) and [datarobot.models.deployment.DeploymentGrantSharedRoleWithUsername](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.DeploymentGrantSharedRoleWithUsername) objects and updates roles accordingly. ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') roles = deployment.list_shared_roles() [role.to_dict() for role in roles] >>> [{'role': 'OWNER', 'id': '5c939e08962d741e34f609f0', 'share_recipient_type': 'user', 'name': 'user@datarobot.com'}] new_role = DeploymentGrantSharedRoleWithUsername(username='user_2@datarobot.com', role='OWNER') response = deployment.update_shared_roles([new_role]) response.status_code >>> 204 roles = deployment.list_shared_roles() [role.to_dict() for role in roles] >>> [{'role': 'OWNER', 'id': '5c939e08962d741e34f609f0', 'share_recipient_type': 'user', 'name': 'user@datarobot.com'}, {'role': 'OWNER', 'id': '5c939e08962d741e34f609f1', 'share_recipient_type': 'user', 'name': 'user_2@datarobot.com'}] revoke_role = DeploymentGrantSharedRoleWithUsername(username='user_2@datarobot.com', role='NO_ROLE') response = deployment.update_shared_roles([revoke_role]) response.status_code >>> 204 roles = deployment.list_shared_roles() [role.to_dict() for role in roles] >>> [{'role': 'OWNER', 'id': '5c939e08962d741e34f609f0', 'share_recipient_type': 'user', 'name': 'user@datarobot.com'}] ``` --- # MLOps URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/index.html > Deploy, monitor, manage, and govern all your models in production with the Python API client. DataRobot MLOps provides a central hub to deploy, monitor, manage, and govern all your models in production. | Topic | Description | | --- | --- | | Deployments | Deploy, manage, and monitor models; batch predictions; model replacement; monitoring data. | | Batch monitoring | Manage batch monitoring job definitions and jobs for batch-enabled deployments. | | Challenger models | Create and manage challenger models to compare against the deployed champion. | | Model Registry | Create and manage registered models and versions. | | Custom models | Manage custom inference models and execution environments. | | Custom metrics | Define and submit custom metrics for deployments. | | Data exports | Export prediction, actuals, training, and data quality data. | | Jobs | Run and manage custom jobs. | | Key values | Manage key-value configuration for the platform. | --- # Jobs URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/jobs.html > Run and manage custom jobs for automation with the Python API client. You can create custom jobs to implement automation (for example, custom tests and custom metrics) for models and deployments. Each job serves as an automated workload, and the exit code determines if it passed or failed. You can run the custom jobs you create for one or more models or deployments. The automated workloads defined through custom jobs can make prediction requests, fetch inputs, and store outputs using DataRobot’s Public API. ## Manage jobs The sections below outline how to manage custom jobs. ### Create job To create a job, use `dr.registry.Job.create`: ``` import os import datarobot as dr # Add files content using `file_data` argument job = dr.registry.Job.create( "my-job", environment_id="65c4f3ed001d3e27a382608f", file_data={"run.sh": "echo 'hello world'"}, ) # Add files from the folder job_folder = "my-folder/files" job_2 = dr.registry.Job.create( "my-job", environment_id="65c4f3ed001d3e27a382608f", folder_path=job_folder, ) # Add files as a list of individual file paths job_3 = dr.registry.Job.create( "my-job", environment_id="65c4f3ed001d3e27a382608f", files=[(os.path.join(job_folder, 'run.sh'), 'run.sh')], ) # If the files should be added to the root of the job filesystem with # the same names as on the local file system, the above can be simplified to the following: job_4 = dr.registry.Job.create( "my-job", environment_id="65c4f3ed001d3e27a382608f", files=[os.path.join(job_folder, 'run.sh')], ) # Alternatively, a job can be created without the files, # and the files can be added later using the `update` method job_5 = dr.registry.Job.create("my-job") ``` ### Create hosted custom metric job from a template To create a hosted custom metric job from a gallery template, use `dr.registry.Job.create_from_custom_metric_gallery_template`: ``` import datarobot as dr templates = dr.models.deployment.custom_metrics.HostedCustomMetricTemplate.list() template_id = templates[0].id job = dr.registry.Job.create_from_custom_metric_gallery_template( template_id = template_id, name = "my-job", ) ``` ### List jobs To list all jobs available to the current user, use `dr.registry.Job.list`: ``` import datarobot as dr jobs = dr.registry.Job.list() jobs >>> [Job('my-job')] ``` ### Retrieve jobs To get a job by unique identifier, use `dr.registry.Job.get`: ``` import datarobot as dr job = dr.registry.Job.get("65f4453e6ea907cb0405ff7f") job >>> Job('my-job') ``` ### Update jobs To get a job by unique identifier and update it, use `dr.registry.Job.get()` and then `update()`: ``` import datarobot as dr job = dr.registry.Job.get("65f4453e6ea907cb0405ff7f") job.update( environment_id="65c4f3ed001d3e27a382608f", description="My Job", folder_path=job_folder, file_data={"README.md": "My README file"}, ) ``` ### Delete jobs To get a job by unique identifier and delete it, use `dr.registry.Job.get()` and then `delete()`: ``` import datarobot as dr job = dr.registry.Job.get("65f4453e6ea907cb0405ff7f") job.delete() ``` ## Manage job runs Use the following commands to manage job runs. ### Create job runs To create a job run, use `dr.registry.JobRun.create`: ``` import datarobot as dr import time job_id = "65f4453e6ea907cb0405ff7f" # Block until job run is finished job_run = dr.registry.JobRun.create(job_id) # or run job without blocking the thread, and check the job run status manually job_run = dr.registry.JobRun.create(job_id, max_wait=None) while job_run.status == dr.registry.JobRunStatus.RUNNING: time.sleep(1) job_run.refresh() ``` ### List job runs To list all job runs, use `dr.registry.JobRun.list`: ``` import datarobot as dr job_id = "65f4453e6ea907cb0405ff7f" job_runs = dr.registry.JobRun.list(job_id) job_runs >>> [JobRun('65f856957d897d46b0e54b37'), JobRun('65f8567f7d897d46b0e54b32'), JobRun('65f856617d897d46b0e54b2d')] ``` ### Retrieve job runs To get a job run with an identifier, use `dr.registry.JobRun.get`,: ``` import datarobot as dr job_id = "65f4453e6ea907cb0405ff7f" job_run = dr.registry.JobRun.get(job_id, "65f856957d897d46b0e54b37") job_run >>> JobRun('65f856957d897d46b0e54b37') ``` ### Update job runs To get a job run by unique identifier and update it, use `dr.registry.JobRun.get()` and then `update()`: ``` import datarobot as dr job_id = "65f4453e6ea907cb0405ff7f" job_run = dr.registry.JobRun.get(job_id, "65f856957d897d46b0e54b37") job_run.update(description="The description of this job run") ``` ### Cancel a job run To get a running job run by identifier and cancel it, use `dr.registry.JobRun.get()` and then `cancel()`: ``` import datarobot as dr job_id = "65f4453e6ea907cb0405ff7f" job_run = dr.registry.JobRun.get(job_id, "65f856957d897d46b0e54b37") job_run.cancel() ``` ### Retrieve job run logs To get job run logs, use `dr.registry.JobRun.get_logs`: ``` import datarobot as dr job_id = "65f4453e6ea907cb0405ff7f" job_run = dr.registry.JobRun.get(job_id, "65f856957d897d46b0e54b37") job_run.get_logs() >>> 2024-03-18T16:06:46.044946476Z Some log output ``` ### Delete job run logs To delete job run logs, use `dr.registry.JobRun.delete_logs`: ``` import datarobot as dr job_id = "65f4453e6ea907cb0405ff7f" job_run = dr.registry.JobRun.get(job_id, "65f856957d897d46b0e54b37") job_run.delete_logs() ``` --- # Key values URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/key_values.html > Manage key-value configuration for models, deployments, and other entities with the Python API client. Key values associated with a DataRobot model, deployment, job, or other DataRobot entities are key-value pairs containing information about the related entity. Each key-value pair has the following: - Name : The unique and descriptive name of the key (for the model package or version). - Value type : The data type of the value associated with the key. The possible types are string, numeric, boolean, URL, image, dataset, pickle, binary, JSON, or YAML. - Category : The type of model information provided by the key value. The possible types are training parameter, metric, tag, artifact, and runtime parameter. - Value : The stored data or file. You can include string, numeric, boolean, image, and dataset key values in custom compliance documentation templates. In addition, with key values for registered models, when you generate compliance documentation for a model package and reference a supported key value in the template, DataRobot inserts the matching values from the associated model package. ## Manage key values Use the following commands to manage key values. ### Create a key value To create a key value, use `dr.KeyValue.create`: ``` import datarobot as dr registered_model_id = "65ccb597732422fa2297199e" key_value = dr.KeyValue.create( registered_model_id, dr.KeyValueEntityType.REGISTERED_MODEL, "my-kv-name", dr.KeyValueCategory.TAG, dr.KeyValueType.STRING, "tag-name", ) key_value.id >>> '65f32822be17d11dec9ebdfb' ``` ### List key values To list all key values available to the current user, use `dr.KeyValue.list`: ``` import datarobot as dr registered_model_id = "65ccb597732422fa2297199e" key_values = dr.KeyValue.list(registered_model_id, dr.KeyValueEntityType.REGISTERED_MODEL) key_values >>> [KeyValue('my-kv-name')] ``` ### Retrieve a key value To get a key value by unique identifier, use `dr.KeyValue.get`: ``` import datarobot as dr key_value = dr.KeyValue.get("65f32822be17d11dec9ebdfb") key_value >>> KeyValue('my-kv-name') ``` ### Find key values by name To find a key value by name, use `dr.KeyValue.find`. Provide the entity ID, entity type, and key value name: ``` import datarobot as dr key_value = dr.KeyValue.find("65f32822be17d11dec9ebdfb", dr.KeyValueEntityType.REGISTERED_MODEL, "my-kv-name") key_value >>> KeyValue('my-kv-name') ``` ### Update key values To get a key value by unique identifier and update it, use `dr.KeyValue.get()` and then `update()`: ``` import datarobot as dr key_value = dr.KeyValue.get("65f32822be17d11dec9ebdfb") key_value.update(value=4.7) key_value.update(value_type=dr.KeyValueType.STRING, value="abc") key_value.update(name="new-kv-name") ``` ### Get key value data To get the value from a key value, use `dr.KeyValue.get_value()`. Provide the key value ID: ``` import datarobot as dr key_value = dr.KeyValue.get("65f32822be17d11dec9ebdfb") key_value.update(value=4.7) key_value.get_value() >>> 4.7 key_value.update(value_type=dr.KeyValueType.STRING, value="abc") key_value.get_value() >>> "abc" key_value.update(value_type=dr.KeyValueType.BOOLEAN, value=True) key_value.get_value() >>> True ``` ### Delete key values To get a key value by unique identifier and delete it, use `dr.KeyValue.get()` and then `delete()`: ``` import datarobot as dr key_value = dr.KeyValue.get("65f32822be17d11dec9ebdfb") key_value.delete() ``` --- # Model Registry URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/model_registry.html > Create and manage registered models and versions with the Python API client. Registered models are generic containers that group multiple versions of models which can be deployed, used as a challenger model, or replace a deployed model. Each registered model can have multiple versions. Each version can be created from a DataRobot model, custom model, or external model. Registered models can have versions of different types (Leaderboard, custom, or external) simultaneously as long as they have same target properties and time series settings, where applicable. ## Create a registered model To create a registered model or add a version to an existing model: ``` LEADERBOARD_MODEL_ID = "650c2372c538ffa4480567d1" # Passing registered_model_name creates new version first_version = dr.RegisteredModelVersion.create_for_leaderboard_item( model_id=LEADERBOARD_MODEL_ID, name="Name of the version(aka model package)", registered_model_name='DEMO 3: Name of the registered model unique across the org ' ) # Add custom model as a version # passing registered_model_id adds version to existing registered model CUSTOM_MODEL_VERSION_ID = "619679c86c1abbc2bd628ed1" second_version_from_custom = dr.RegisteredModelVersion.create_for_custom_model_version( custom_model_version_id=CUSTOM_MODEL_VERSION_ID, registered_model_id=first_version.registered_model_id, name="Another Name of the version(aka model package)", ) # Add external model as a version second_version_from_external = dr.RegisteredModelVersion.create_for_external( name='Another name', target={'name': 'Target', 'type': 'Regression'}, registered_model_id=first_version.registered_model_id, ) ``` ## List and filter registered models Use the following command to list registered models. You can filter the registered models that are returned by passing an instance of the [RegisteredModelListFilters](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.model_registry.RegisteredModelListFilters) class to the `filters` keyword argument. You can also filter the registered model versions that are returned by passing an instance of the [RegisteredModelVersionsListFilters](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.model_registry.RegisteredModelVersionsListFilters) class to the `filters` keyword argument. ``` demo_registered_models = dr.RegisteredModel.list(search="DEMO") registered_model_filters = dr.models.model_registry.RegisteredModelListFilters( created_at_start=datetime.datetime(2020, 1, 1), created_at_end=datetime.datetime(2024, 1, 2), modified_at_start=datetime.datetime(2020, 1, 1), modified_at_end=datetime.datetime(2024, 1, 2), target_name='readmitted', target_type='Binary', created_by='john.doe@example.com', compatible_with_model_package_id='650a9f57d3f427ce1cc64747', prediction_threshold=0.5, imported=False, for_challenger=False, ) registered_models = dr.RegisteredModel.list(filters=registered_model_filters, search="10k") registered_model = registered_models[0] versions = registered_model.list_versions() # Similarly to registered models, versions also support fine-grain filtering and search filters = dr.models.model_registry.RegisteredModelVersionsListFilters( target_name='readmitted', ) versions_with_search = registered_model.list_versions(search="Elastic", filters=filters) ``` ## Manage registered models Use the following command to archive registered models. Archiving registered models archives all the versions of the registered model. ``` REGISTERED_MODEL_ID = "651bd2317aed25ed7d4bca7f" dr.RegisteredModel.archive(REGISTERED_MODEL_ID) ``` Use the following command to update registered models. ``` REGISTERED_MODEL_ID = "651bd2317aed25ed7d4bca7f" dr.RegisteredModel.update(REGISTERED_MODEL_ID, name="New name") ``` To share registered models with other users or groups, or retrieve existing roles on the deployment: ``` registered_model = dr.RegisteredModel.get('645b62d5373ed49b485d73e9') # EXISTING ROLES roles = registered_model.get_shared_roles() role = dr.SharingRole( share_recipient_type="user", id='5ca19879a950d002c61ea3e7', role="USER", ) registered_model.share([role]) ``` ## List deployments associated with a registered model Use the following command to list deployments associated with registered model. The deployment is considered associated if one of the versions of the registered model is either a champion or a challenger model. ``` model_with_deployments = dr.RegisteredModel.get('65035d911e9ff5b07f00f2ea') # we can list deployments associated with this registered model. Method is searchable and paginated. model_associated_deployments = model_with_deployments.list_associated_deployments() # we can also list deployments associated with specific version of the registered model version = model_with_deployments.list_versions()[1] version.list_associated_deployments() ``` --- # Blueprints URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/blueprint.html > Learn how to work with DataRobot blueprints, which define computation paths for datasets before producing predictions. Blueprints are a set of computation paths that a dataset passes through before producing predictions from data. A blueprint can be trained on a dataset to generate a model. To modify blueprints using Python, reference the [Blueprint Workshop documentation](https://docs.datarobot.com/en/docs/api/reference/bp-workshop/index.html). The following code block summarizes the interactions available for blueprints. ``` # Get the set of blueprints recommended by datarobot import datarobot as dr my_projects = dr.Project.list() project = my_projects[0] menu = project.get_blueprints() first_blueprint = menu[0] project.train(first_blueprint) ``` ## List blueprints When you upload a file to a project and set a target, you receive a set of recommended blueprints that are appropriate for the task at hand. Use `get_blueprints` to get the list of blueprints recommended for a project: ``` project = dr.Project.get('5506fcd38bd88f5953219da0') menu = project.get_blueprints() blueprint = menu[0] ``` ## Get a blueprint If you already have a `blueprint_id` from a model you can retrieve the blueprint directly. ``` project_id = '5506fcd38bd88f5953219da0' project = dr.Project.get(project_id) models = project.get_models() model = models[0] blueprint = Blueprint.get(project_id, model.blueprint_id) ``` ## Get a blueprint chart You can retrieve charts for all blueprints that are either from a blueprint menu or are already used in a model. You can also get a blueprint’s representation in Graphviz DOT format to render it into the format you need. ``` project_id = '5506fcd38bd88f5953219da0' blueprint_id = '4321fcd38bd88f595321554223' bp_chart = BlueprintChart.get(project_id, blueprint_id) print(bp_chart.to_graphviz()) ``` ## Get blueprint documentation You can retrieve documentation for tasks used in a blueprint. The documentation contains information about the task, its parameters, and links and references to additional sources. All documents are instances of the [BlueprintTaskDocument](https://docs.datarobot.com/en/docs/api/reference/sdk/blueprints.html#datarobot.models.BlueprintTaskDocument) class. ``` project_id = '5506fcd38bd88f5953219da0' blueprint_id = '4321fcd38bd88f595321554223' bp = Blueprint.get(project_id, blueprint_id) docs = bp.get_documents() print(docs[0].task) >>> Average Blend print(docs[0].links[0]['url']) >>> https://en.wikipedia.org/wiki/Ensemble_learning ``` ## Blueprint attributes The `Blueprint` class holds the data required to use the blueprint for modeling. This includes the `blueprint_id` and `project_id`. There are also two attributes that help distinguish blueprints: `model_type` and `processes`. ``` print(blueprint.id) >>> u'8956e1aeecffa0fa6db2b84640fb3848' print(blueprint.project_id) >>> u5506fcd38bd88f5953219da0' print(blueprint.model_type) >>> Logistic Regression print(blueprint.processes) >>> [u'One-Hot Encoding', u'Missing Values Imputed', u'Standardize', u'Logistic Regression'] ``` ## Build a model from a blueprint You can also use a blueprint to train a model. The model is trained on the associated project’s dataset by default. Note that [Project.train](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.train) is used for non-datetime partitioned projects.[Project.train_datetime](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.train_datetime) should be used for datetime partitioned projects. ``` model_job_id = project.train(blueprint) # For datetime partitioned projects model_job = project.train_datetime(blueprint.id) ``` Both [Project.train](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.train) and [Project.train_datetime](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.train_datetime) will put a new modeling job into the queue. However, note that `Project.train` returns the ID of the created [ModelJob](https://docs.datarobot.com/en/docs/api/reference/sdk/jobs.html), while `Project.train_datetime` returns the `ModelJob` object itself. You can pass a ModelJob ID to [wait_for_async_model_creation](https://docs.datarobot.com/en/docs/api/reference/sdk/jobs.html#wait-for-async-model-creation-label) function, which polls the async model creation status and returns the newly created model when it’s finished. --- # Modeling URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/index.html > Information to help you easily navigate the process of building, understanding, and analyzing models. The modeling section provides information to help you easily navigate the process of building, understanding, and analyzing models. | Resource | Description | | --- | --- | | Model insights | Information to help you easily navigate the process of building, understanding, and analyzing models. | | Specialized workflows | Alternative workflows for a variety of specialized data types. | | Projects | Create, configure, and manage DataRobot projects for modeling. | | Models | Train, retrieve, and analyze DataRobot models. | | Blueprints | Work with DataRobot blueprints which define computation paths for datasets before producing predictions. | | Jobs | Manage and monitor jobs in DataRobot projects, including model creation jobs and queue management. | | Model recommendation | Learn how to retrieve and work with DataRobot model recommendations for deployment. | | DataRobot Prime | Learn how to use DataRobot Prime to download executable code that approximates models. | --- # Automated documentation URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/insights/automated_documentation.html > Learn how to generate and manage automated documentation for DataRobot models and projects. DataRobot can generate automated documentation about various entities within the platform, such as specific models or projects. These reports can be downloaded and shared to help with regulatory compliance as well as to provide a general understanding of the AI lifecycle. ## Check available document types Automated documentation is available behind different feature flags set up according to your POC settings or subscription plan.`MODEL_COMPLIANCE` documentation is a premium add-on DataRobot product, while `AUTOPILOT_SUMMARY` report is available behind an optional feature flag for Self-Service and other platforms. ``` import datarobot as dr # Connect to your DataRobot platform with your token dr.Client(token=my_token, endpoint=endpoint) options = dr.AutomatedDocument.list_available_document_types() ``` In response, you get a `data` dictionary with a list of document types that are available for generation with your account. ## Generate automated documents Now that you know which documents you can generate, create one with `AutomatedDocument .generate` method. Note that for `AUTOPILOT_SUMMARY` report, you need to assign a project ID to the `entity_id` parameter, while `MODEL_COMPLIANCE` expects an ID of a model with the `entity_id` parameter. ``` import datarobot as dr dr.Client(token=my_token, endpoint=endpoint) doc_type = "AUTOPILOT_SUMMARY" entity_id = "5e8b6a34d2426053ab9a39ed" # This is an ID of a project file_format="docx" doc = dr.AutomatedDocument(document_type=doc_type, entity_id=entity_id, output_format=file_format) doc.generate() ``` You can specify other attributes. For example, `filepath` presets the file location and name to use when downloading the document. See the API reference for more details. ## Download automated documents If you followed the steps above to generate an automated document, you can use the `AutomatedDocument.download` method right away to get the document. ``` doc.filepath = "Users/jeremy/DR_project_docs/autopilot_report_staff_2021.docx" doc.download() ``` You can set a desired `filepath` (that includes the future file’s name) before you download a document. Otherwise, it will be automatically downloaded to the directory from which you launched your script. Please note that to download the document, you need its ID. When you generate a document with the Python client, the ID is set automatically without your interference. However, if the document has already been generated from the application interface (or REST API) and you want to download it using the Python client, you need to provide the ID of the document you want to download: ``` import datarobot as dr dr.Client(token=my_token, endpoint=endpoint) doc_id = "604f81f0f3d6397d250c35bc" path = "Users/jeremy/DR_project_docs/xgb_model_doc_staff_project_2021.docx" doc = dr.AutomatedDocument(id=doc_id, filepath=path) doc.download() ``` ## List previously generated automated documents You can retrieve information about previously generated documents available for your account. The information includes document ID and type, ID of the entity it was generated for, time of creation, and other information. Documents are sorted by creation time – `created_at` key – from most recent to oldest. ``` import datarobot as dr dr.Client(token=my_token, endpoint=endpoint) docs = dr.AutomatedDocument.list_generated_documents() ``` This returns list of `AutomatedDocument` objects. You can request a list of specific documents. For example, get a list of all `MODEL_COMPLIANCE` documents: ``` model_docs = dr.AutomatedDocument.list_generated_documents(document_types=["MODEL_COMPLIANCE"]) ``` Or get a list of documents created for specific entities: ``` otv_project_reports = dr.AutomatedDocument.list_generated_documents( entity_ids=["604f81f0f3d6397d250c35bc", "5ed60de32f18d97d250c3db5"] ) ``` For more information about all query options, see `AutomatedDocument.list_generated_documents` in the API reference. ## Delete automated documents To delete a document from the DataRobot application, use the `AutomatedDocument.delete` method. ``` import datarobot as dr dr.Client(token=my_token, endpoint=endpoint) doc = dr.AutomatedDocument(id="604f81f0f3d6397d250c35bc") doc.delete() ``` All locally saved automated documents will remain intact. --- # External testset URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/insights/external_testset.html > Learn how to test models with external datasets to evaluate performance and compute metric scores. Compute metric scores and insights on an external test dataset to help validate a model's performance and its generalization capabilities before deployment. Evaluating the model against data it hasn't seen during training ("external") provides insights into how well and consistently the model will perform in real-world scenarios. Note that this testing is not available for time series models. ## Request external scores and insights To compute scores and insights on a dataset: - Upload a prediction dataset that contains the target column ( PredictionDataset.contains_target_values == True ). - The dataset must have the same structure as the original project data. ``` import datarobot as dr # Upload dataset project = dr.Project(project_id) dataset = project.upload_dataset('./test_set.csv') dataset.contains_target_values >>>True # request external test to compute metric scores and insights on dataset # select model using project.get_models() external_test_job = model.request_external_test(dataset.id) # once job is complete, scores and insights are ready for retrieving external_test_job.wait_for_completion() ``` ## Retrieve external metric scores and insights After completion of the external test job, metric scores and insights for external test sets will be ready. > [!NOTE] Note > Some notes: > > Check > PredictionDataset.data_quality_warnings > for dataset warnings. > Insights are not available if the dataset is fewer than 10 rows. > The ROC curve cannot be calculated if the dataset has only one class in the target column. ## Retrieve external metric scores ``` import datarobot as dr # retrieving list of external metric scores on multiple datasets metric_scores_list = dr.ExternalScores.list(project_id, model_id) # retrieving external metric scores on one dataset metric_scores = dr.ExternalScores.get(project_id, model_id, dataset_id) ``` ## Retrieve an external lift chart ``` import datarobot as dr # retrieving list of lift charts on multiple datasets lift_list = dr.ExternalLiftChart.list(project_id, model_id) # retrieving one lift chart for dataset lift = dr.ExternalLiftChart.get(project_id, model_id, dataset_id) ``` ## Retrieve an external multiclass lift chart Available for multiclass classification models only. ``` import datarobot as dr # retrieving list of lift charts on multiple datasets lift_list = ExternalMulticlassLiftChart.list(project_id, model_id) # retrieving one lift chart for dataset and a target class lift = ExternalMulticlassLiftChart.get(project_id, model_id, dataset_id, target_class) ``` ## Retrieve an external ROC curve Available for binary classification models only. ``` import datarobot as dr # retrieving list of roc curves on multiple datasets roc_list = ExternalRocCurve.list(project_id, model_id) # retrieving one ROC curve for dataset roc = ExternalRocCurve.get(project_id, model_id, dataset_id) ``` ## Retrieve a multiclass confusion matrix Available for multiclass classification models only. ``` import datarobot as dr # retrieving list of confusion charts on multiple datasets confusion_list = ExternalConfusionChart.list(project_id, model_id) # retrieving one confusion chart for dataset confusion = ExternalConfusionChart.get(project_id, model_id, dataset_id) ``` ## Retrieve a residuals chart Available for regression models only. ``` import datarobot as dr # retrieving list of residuals charts on multiple datasets residuals_list = ExternalResidualsChart.list(project_id, model_id) # retrieving one residuals chart for dataset residuals = ExternalResidualsChart.get(project_id, model_id, dataset_id) ``` --- # Model insights URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/insights/index.html > Information to help you easily navigate the process of building, understanding, and analyzing models. The Modeling section provides information to help you easily navigate the process of building, understanding, and analyzing models. | Resource | Description | | --- | --- | | Prediction explanations | Retrieve prediction explanations for DataRobot models. | | SHAP insights | Use SHAP (SHapley Additive exPlanations) insights for explaining model predictions. | | Model performance insights | Compute and retrieve ROC curves, lift charts, and residuals for DataRobot models. | | Automated documentation | Generate and manage automated documentation for DataRobot models and projects. | | External testset | Test models with external datasets to evaluate performance and compute metric scores. | | Rating table | Download and upload rating tables for Generalized Additive Models. | --- # Model performance insights URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/insights/model_performance_insights.html > Learn how to compute and retrieve ROC curves, lift charts, and residuals for DataRobot models. DataRobot provides several insights to help you understand and evaluate model performance: - ROC Curve : Receiver Operating Characteristic curve showing the trade-off between true positive rate and false positive rate at various classification thresholds. Available for binary classification models. - Lift Chart : Depicts how well a model segments the target population and how well the model performs for different ranges of values of the target variable. Available for classification and regression models. - Residuals : Displays the distribution of prediction errors for regression models. These insights can be computed for different data partitions (validation, cross-validation, holdout) and can be filtered by data slices. The following example code assumes that you have a trained model object called `model`. ## ROC Curve The following example shows how to compute and retrieve ROC curves for a binary classification model. ``` from datarobot.insights.roc_curve import RocCurve model_id = model.id # or model_id = 'YOUR_MODEL_ID' # Compute a new ROC curve and wait for it to complete roc = RocCurve.create(entity_id=model_id) # default source is 'validation' # Access ROC curve properties print(roc.auc) >>> 0.85 print(roc.kolmogorov_smirnov_metric) >>> 0.42 print(roc.roc_points[:2]) >>> [{'accuracy': 0.539375, 'f1_score': 0.0, 'false_negative_score': 737, ...}, ...] # Compute ROC curve on a different partition, and return immediately with job reference job = RocCurve.compute(entity_id=model_id, source='holdout') # Wait for the job to complete roc_holdout = job.get_result_when_complete() print(roc_holdout.auc) >>> 0.83 # Get a pre-existing ROC curve (if already computed) existing_roc = RocCurve.get(entity_id=model_id, source='validation') # List all available ROC curves for a model roc_list = RocCurve.list(entity_id=model_id) print(roc_list) >>> [, ...] print([(r.source, r.auc) for r in roc_list]) >>> [('validation', 0.85), ('holdout', 0.83)] ``` ### ROC curve with data slices You can compute ROC curves for specific data slices: ``` from datarobot.insights.roc_curve import RocCurve # Get a pre-existing ROC curve for a specific data slice roc_sliced = RocCurve.get(entity_id=model_id, source='validation', data_slice_id='slice_id_here') # Or compute a new one job = RocCurve.compute(entity_id=model_id, source='validation', data_slice_id='slice_id_here') roc_sliced = job.get_result_when_complete() ``` ## Lift chart Lift charts depict how well a model segments the target population and how well the model performs for different ranges of values of the target variable. ``` from datarobot.insights.lift_chart import LiftChart model_id = model.id # or model_id = 'YOUR_MODEL_ID' # Compute a new lift chart and wait for it to complete lift = LiftChart.create(entity_id=model_id) # default source is 'validation' # Access lift chart bins print(lift.bins[:3]) >>> [{'actual': 0.4, 'predicted': 0.227, 'bin_weight': 5.0}, ...] # Compute lift chart on a different partition, and return immediately with job reference job = LiftChart.compute(entity_id=model_id, source='holdout') # Wait for the job to complete lift_holdout = job.get_result_when_complete() # Get a pre-existing lift chart (if already computed) existing_lift = LiftChart.get(entity_id=model_id, source='validation') # List all available lift charts for a model lift_list = LiftChart.list(entity_id=model_id) print(lift_list) >>> [, ...] print([l.source for l in lift_list]) >>> ['validation', 'holdout', 'crossValidation'] # Get lift chart for a specific data slice lift_sliced = LiftChart.get(entity_id=model_id, source='validation', data_slice_id='slice_id_here') ``` ## Residuals The residuals chart shows the distribution of prediction errors for regression models. ``` from datarobot.insights.residuals import Residuals model_id = model.id # or model_id = 'YOUR_MODEL_ID' # Compute a new residuals chart and wait for it to complete residuals = Residuals.create(entity_id=model_id) # default source is 'validation' # Access residuals properties print(residuals.coefficient_of_determination) >>> 0.85 print(residuals.residual_mean) >>> 0.023 print(residuals.standard_deviation) >>> 1.42 # Access histogram data print(residuals.histogram[:3]) >>> [{'interval_start': -33.37, 'interval_end': -32.52, 'occurrences': 1}, ...] # Access raw chart data (actual, predicted, residual, row number) print(residuals.chart_data[:3]) >>> [[45.2, 43.8, -1.4, 0], [52.1, 51.9, -0.2, 1], ...] # Compute residuals on a different partition, and return immediately with job reference job = Residuals.compute(entity_id=model_id, source='holdout') # Wait for the job to complete residuals_holdout = job.get_result_when_complete() # Get a pre-existing residuals chart (if already computed) existing_residuals = Residuals.get(entity_id=model_id, source='validation') # List all available residuals charts for a model residuals_list = Residuals.list(entity_id=model_id) print(residuals_list) >>> [, ...] print([r.source for r in residuals_list]) >>> ['validation', 'holdout'] # Get residuals for a specific data slice residuals_sliced = Residuals.get(entity_id=model_id, source='validation', data_slice_id='slice_id_here') ``` --- # Prediction explanations URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/insights/prediction_explanations.html > Learn how to compute and retrieve prediction explanations for DataRobot models. To compute prediction explanations you need to have [feature impact](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/model.html#feature-impact-label) computed for a model, and predictions for an uploaded dataset computed with a selected model. Computing prediction explanations is a resource-intensive task, but you can configure it with maximum explanations per row and prediction value thresholds to speed up the process. > [!NOTE] Consideration for reading data > Note that when DataRobot is reading data, the `dr_enums.DEFAULT_TIMEOUT.READ` duration is 60 seconds by default. If the data requires more time to be read, consider adjusting that duration manually by setting it to 600 seconds: > > ``` > import datarobot.enums as dr_enums > dr_enums.DEFAULT_TIMEOUT.READ = 600 > ``` ## Quick reference ``` import datarobot as dr # Get project my_projects = dr.Project.list() project = my_projects[0] # Get model models = project.get_models() model = models[0] # Compute feature impact feature_impacts = model.get_or_request_feature_impact() # Upload dataset dataset = project.upload_dataset('./data_to_predict.csv') # Compute predictions predict_job = model.request_predictions(dataset.id) predict_job.wait_for_completion() # Initialize prediction explanations pei_job = dr.PredictionExplanationsInitialization.create(project.id, model.id) pei_job.wait_for_completion() # Compute prediction explanations with default parameters pe_job = dr.PredictionExplanations.create(project.id, model.id, dataset.id) pe = pe_job.get_result_when_complete() # Iterate through predictions with prediction explanations for row in pe.get_rows(): print(row.prediction) print(row.prediction_explanations) # download to a CSV file pe.download_to_csv('prediction_explanations.csv') ``` ## List prediction explanations You can use the `PredictionExplanations.list()` method to return a list of prediction explanations computed for a project’s models: ``` import datarobot as dr prediction_explanations = dr.PredictionExplanations.list('58591727100d2b57196701b3') print(prediction_explanations) >>> [PredictionExplanations(id=585967e7100d2b6afc93b13b, project_id=58591727100d2b57196701b3, model_id=585932c5100d2b7c298b8acf), PredictionExplanations(id=58596bc2100d2b639329eae4, project_id=58591727100d2b57196701b3, model_id=585932c5100d2b7c298b8ac5), PredictionExplanations(id=58763db4100d2b66759cc187, project_id=58591727100d2b57196701b3, model_id=585932c5100d2b7c298b8ac5), ...] pe = prediction_explanations[0] pe.project_id >>> u'58591727100d2b57196701b3' pe.model_id >>> u'585932c5100d2b7c298b8acf' ``` You can pass following parameters to filter the result: - model_id – str, used to filter returned prediction explanations by model_id. - limit – int, limit for number of items returned, default: no limit. - offset – int, number of items to skip, default: 0. List prediction explanations example: ``` project_id = '58591727100d2b57196701b3' model_id = '585932c5100d2b7c298b8acf' dr.PredictionExplanations.list(project_id, model_id=model_id, limit=20, offset=100) ``` ## Initialize prediction explanations In order to compute prediction explanations you have to initialize it for a particular model. ``` dr.PredictionExplanationsInitialization.create(project_id, model_id) ``` ## Compute prediction explanations on new data If all prerequisites are in place, you can compute prediction explanations in the following way: ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' dataset_id = '5506fcd98bd88a8142b725c8' pe_job = dr.PredictionExplanations.create(project_id, model_id, dataset_id, max_explanations=2, threshold_low=0.2, threshold_high=0.8) pe = pe_job.get_result_when_complete() ``` Where: - max_explanations are the maximum number of prediction explanations to compute for each row. - threshold_low and threshold_high are thresholds for the value of the prediction of the row. Prediction explanations will be computed for a row if the row’s prediction value is higher than threshold_high or lower than threshold_low . If no thresholds are specified, prediction explanations will be computed for all rows. ## Compute prediction explanations on training data To compute prediction explanations on training data, use the code snippet below. The prerequisites are generally the same as computing on newly uploaded data: - Feature Impact calculations must have completed. - Prediction explanations must be initialized. - Predictions on training data must first be computed for the model. The `dataset_id` parameter is the ID of the feature list that was used to train the model. ``` import datarobot as dr project_id = '67771742b4d4cf44277b1ff0' model_id = '677859cfeaea57c1bc9a150a' model = dr.Model.get(project_id, model_id) dataset_id = model.featurelist_id # Request feature impact if not done yet. model.request_feature_impact() # Request training predictions for the model if not done yet. # The subset 'all' includes training, validation, and holdout data. model.request_training_predictions(dr.enums.DATA_SUBSET.ALL) # Initialize explanations. dr.PredictionExplanationsInitialization.create(project_id, model_id) # Request and download explanations for the full training data. pe_job = dr.PredictionExplanations.create_on_training_data(project_id, model_id, dataset_id) result = pe_job.get_result_when_done() df = result.get_all_as_dataframe() ``` ## Get previously generated predictions If you don’t have a `PredictJob`, there are two more ways to retrieve predictions from the `Predictions` interface: 1. Get all prediction rows as a pandas.DataFrame object: ``` import datarobot as dr preds = dr.Predictions.get("5b61bd68ca36c04aed8aab7f", prediction_id="5b6b163eca36c0108fc5d411") df = preds.get_all_as_dataframe() df_with_serializer = preds.get_all_as_dataframe(serializer='csv') ``` 1. Download all prediction rows to a file as a CSV: ``` import datarobot as dr preds = dr.Predictions.get("5b61bd68ca36c04aed8aab7f", prediction_id="5b6b163eca36c0108fc5d411") preds.download_to_csv('predictions.csv') preds.download_to_csv('predictions_with_serializer.csv', serializer='csv') ``` ## Retrieve prediction explanations You have three options for retrieving prediction explanations. > [!NOTE] Note > `PredictionExplanations.get_all_as_dataframe()` and `PredictionExplanations.download_to_csv()` reformat prediction explanations to match the schema of the CSV file downloaded from the UI (RowId, Prediction, Explanation 1- N Strength, Explanation 1- N Feature, Explanation 1- N Value). Get prediction explanations rows one by one as [PredictionExplanationsRow](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/insights/prediction_explanations.html#datarobot.models.prediction_explanations.PredictionExplanationsRow) objects: ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' prediction_explanations_id = '5506fcd98bd88f1641a720a3' pe = dr.PredictionExplanations.get(project_id, prediction_explanations_id) for row in pe.get_rows(): print(row.prediction_explanations) ``` Get all rows as `pandas.DataFrame`: ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' prediction_explanations_id = '5506fcd98bd88f1641a720a3' pe = dr.PredictionExplanations.get(project_id, prediction_explanations_id) prediction_explanations_df = pe.get_all_as_dataframe() ``` Download all rows to a file as CSV document: ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' prediction_explanations_id = '5506fcd98bd88f1641a720a3' pe = dr.PredictionExplanations.get(project_id, prediction_explanations_id) pe.download_to_csv('prediction_explanations.csv') ``` ## Adjusted predictions in prediction explanations In some projects such as insurance projects, the prediction adjusted by exposure is more useful compared with raw prediction. For example, the raw prediction (e.g. claim counts) is divided by exposure (e.g. time) in the project with exposure column. The adjusted prediction provides insights with regard to the predicted claim counts per unit of time. To include that information, set `exclude_adjusted_predictions` to False in correspondent method calls. ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' prediction_explanations_id = '5506fcd98bd88f1641a720a3' pe = dr.PredictionExplanations.get(project_id, prediction_explanations_id) pe.download_to_csv('prediction_explanations.csv', exclude_adjusted_predictions=False) prediction_explanations_df = pe.get_all_as_dataframe(exclude_adjusted_predictions=False) ``` ## Multiclass/clustering prediction explanation modes When calculating prediction explanations for the multiclass or clustering model you need to specify which classes should be explained in each row. By default we only explain the predicted class but it can be set with the mode parameter of [PredictionExplanations.create](https://docs.datarobot.com/en/docs/api/reference/sdk/insights.html#datarobot.PredictionExplanations.create) ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' dataset_id = '5506fcd98bd88a8142b725c8' # Explain predicted and second-best class results in each row pe_job = dr.PredictionExplanations.create(project_id, model_id, dataset_id, mode=dr.models.TopPredictionsMode(2)) pe = pe_job.get_result_when_complete() # Explain results for classes "setosa" and "versicolor" in each row pe_job = dr.PredictionExplanations.create(project_id, model_id, dataset_id, mode=dr.models.ClassListMode(["setosa", "versicolor"])) pe = pe_job.get_result_when_complete() ``` ## SHAP based prediction explanations There are two types of SHAP prediction explanations available, universal SHAP explanations and model-specific SHAP explanations. All models support universal SHAP explanations, which use the permutation based explainer algorithm. Selected models support SHAP explanations such as the tree-based explainer or kernel explainer. Universal SHAP explanations can be computed and retrieved very simply and do not require any prerequisites. They can be computed for any available partition, and can be restricted to specific data slices. ``` import datarobot as dr from datarobot.insights import ShapMatrix project_id = '5ea6d3354cfad121cf33a5e1' model_id = '5ea6d38b4cfad121cf33a60d' project = dr.Project.get(project_id) model = dr.Model.get(project=project_id, model_id=model_id) # Additional parameters can be passed to specify the partition, # data slice, and other parametrers. shap_insight = ShapMatrix.create(model_id) # Get all computed SHAP matrices all_shap_insights = ShapMatrix.list(model_id) # Retrieve the SHAP matrix as a numpy array matrix = shap_insight.matrix # Retrieve the SHAP matrix columns columns = shap_insight.columns # Retrieve the SHAP base value for additivity checks base_value = shap_insight.base_value ``` You can request model-specific SHAP based prediction explanations using previously uploaded scoring dataset for supported models. Unlike for XEMP prediction explanations you do not need to have [feature impact](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/model.html#feature-impact-label) computed for a model, and predictions for an uploaded dataset. See [datarobot.models.ShapMatrix.create()](https://docs.datarobot.com/en/docs/api/reference/sdk/insights.html#datarobot.models.ShapMatrix.create) reference for a description of the types of parameters that can be passed in. ``` import datarobot as dr project_id = '5ea6d3354cfad121cf33a5e1' model_id = '5ea6d38b4cfad121cf33a60d' project = dr.Project.get(project_id) model = dr.Model.get(project=project_id, model_id=model_id) # check if model supports SHAP model_capabilities = model.get_supported_capabilities() print(model_capabilities.get('supportsShap')) >>> True # upload dataset to generate prediction explanations dataset_from_path = project.upload_dataset('./data_to_predict.csv') shap_matrix_job = ShapMatrix.create(project_id=project_id, model_id=model_id, dataset_id=dataset_from_path.id) shap_matrix_job >>> Job(shapMatrix, status=inprogress) # wait for job to finish shap_matrix = shap_matrix_job.get_result_when_complete() shap_matrix >>> ShapMatrix(id='5ea84b624cfad1361c53f65d', project_id='5ea6d3354cfad121cf33a5e1', model_id='5ea6d38b4cfad121cf33a60d', dataset_id='5ea84b464cfad1361c53f655') # retrieve SHAP matrix as pandas.DataFrame df = shap_matrix.get_as_dataframe() # list as available SHAP matrices for a project shap_matrices = dr.ShapMatrix.list(project_id) shap_matrices >>> [ShapMatrix(id='5ea84b624cfad1361c53f65d', project_id='5ea6d3354cfad121cf33a5e1', model_id='5ea6d38b4cfad121cf33a60d', dataset_id='5ea84b464cfad1361c53f655')] shap_matrix = shap_matrices[0] # retrieve SHAP matrix as pandas.DataFrame df = shap_matrix.get_as_dataframe() ``` --- # Rating table URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/insights/rating_table.html > Learn how to download and upload rating tables for Generalized Additive Models. A rating table is an exportable csv representation of a Generalized Additive Model. They contain information about the features and coefficients used to make predictions. Users can influence predictions by downloading and editing values in a rating table, then re-uploading the table and using it to create a new model. See the page about interpreting Generalized Additive Models’ output in the DataRobot user guide for more details on how to interpret and edit rating tables. ## Download a rating table You can retrieve a rating table from the list of rating tables in a project: ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' project = dr.Project.get(project_id) rating_tables = project.get_rating_tables() rating_table = rating_tables[0] ``` Or you can retrieve a rating table from a specific model. The model must already exist: ``` import datarobot as dr from datarobot.models import RatingTableModel, RatingTable project_id = '5506fcd38bd88f5953219da0' project = dr.Project.get(project_id) # Get model from list of models with a rating table rating_table_models = project.get_rating_table_models() rating_table_model = rating_table_models[0] # Or retrieve model by id. The model must have a rating table. model_id = '5506fcd98bd88f1641a720a3' rating_table_model = dr.RatingTableModel.get(project=project_id, model_id=model_id) # Then retrieve the rating table from the model rating_table_id = rating_table_model.rating_table_id rating_table = dr.RatingTable.get(projcet_id, rating_table_id) ``` Then you can download the contents of the rating table: ``` rating_table.download('./my_rating_table.csv') ``` ## Upload a rating table After you’ve retrieved the rating table CSV and made the necessary edits, you can re-upload the CSV so you can create a new model from it: ``` job = dr.RatingTable.create(project_id, model_id, './my_rating_table.csv') new_rating_table = job.get_result_when_complete() job = new_rating_table.create_model() model = job.get_result_when_complete() ``` --- # SHAP insights URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/insights/shap_insights.html > Learn how to compute and use SHAP (SHapley Additive exPlanations) insights for explaining model predictions. SHAP is an open-source method for explaining the predictions from machine learning models. (You can find more information about SHAP at its repository on GitHub: [https://github.com/slundberg/shap](https://github.com/slundberg/shap)) DataRobot supports SHAP computations for all regression and binary classification blueprints. You can compute three different insights: - "SHAP matrix": Raw SHAP values for each feature column and each row. - "SHAP impact": Overall importance for each feature column across all rows, based on aggregated SHAP matrix values. - "SHAP preview": SHAP values for the most important features in each row, presented with the values of the features in that row. The following example code assumes that you have a trained model object called `model`. ``` import datarobot as dr from datarobot.insights.shap_matrix import ShapMatrix from datarobot.insights.shap_impact import ShapImpact from datarobot.insights.shap_preview import ShapPreview model_id = model.id # or model_id = 'YOUR_MODEL_ID' # request SHAP Matrix, and wait for it to complete result = ShapMatrix.create(entity_id=model_id) # default source is 'validation' # view the properties of the SHAP Matrix print(result.columns) >>> ['AUCGUART', 'Color', 'Make', ... print(result.matrix) >>> [[ 1.22604372e-02 1.98424454e-01 2.23308013e-01 ...] ... ] # request SHAP Matrix on a different partition, and return immediately with job reference job = ShapMatrix.compute(entity_id=model_id, source='holdout') # wait for the job to complete result = job.get_result_when_complete() print(result.columns) >>> ['AUCGUART', 'Color', 'Make', ... print(result.matrix) >>> [[-0.11443075 -0.01130723 0.22330801 ... ] ... ] # request SHAP Impact; only works for training currently job = ShapImpact.compute(entity_id=model_id, source='training') result = job.get_result_when_complete() # Impacts are listed as [feature_name, normalized_impact, unnormalized_impact] print(result.shap_impacts) >>> [['AUCGUART', 0.07989059458051094, 0.022147886593333888], ...] # list all matrices computed for this model, including each partition matrix_list = ShapMatrix.list(entity_id=model_id) print(matrix_list) >>> [, ...] print([(matrix_obj, matrix_obj.source) for matrix_obj in matrix_list]) >>> [(, 'validation'), ... ] # upload a file to the AI Catalog dataset = dr.Dataset.upload("./path/to/dataset.csv") # request explanations for that file in the "preview" format job = ShapPreview.compute(entity_id=model_id, source='externalTestSet', external_dataset_id=dataset.id) result = job.get_result_when_complete() print(result.previews[0]) >>> {'row_index': 0, >>> 'prediction_value': 0.3024851286385187, >>> 'preview_values': [{'feature_rank': 1, >>> 'feature_name': 'BYRNO', >>> 'feature_value': '21973', >>> 'shap_value': 0.22025144078391848, >>> 'has_text_explanations': False, >>> 'text_explanations': []}, >>> ... } ``` # SHAP insights for custom models You can compute SHAP insights for custom models, not just native DataRobot models. To do this, first complete the following setup: 1. Create a custom model version with an execution environment and a training dataset; note the version ID. 2. Register the custom model version as a registered model. 3. Initialize the registered model for insights, using the AutomatedDocument.initialize_model_compliance method. At this point, the model is ready for SHAP insights computation. Once these steps are completed for a given registered model version, they do not have to be repeated. As an example, the code snippet below outlines the preparation steps and then requests a ShapMatrix computation on an external dataset via the AI Catalog. It assumes that you have a Scoring Code file, `model.jar`, for the custom model, which you will run using the Java drop-in execution environment, as well as a training dataset called `training.csv`. ``` import datarobot as dr from datarobot.insights.shap_matrix import ShapMatrix # 1: create a custom model version with an execution environment and a training dataset, and note the version id model_args = { "target_type": dr.TARGET_TYPE.REGRESSION, "target_name": "time_in_hospital", "language": "java", } training_dataset = dr.Dataset.create_from_file(file_path="path/to/training.csv") execution_environment = dr.ExecutionEnvironment.list(search_for="java")[0] custom_model = dr.CustomInferenceModel.create( name="model.jar", **model_args, ) custom_model_version = dr.CustomModelVersion.create_clean( custom_model_id=custom_model.id, base_environment_id=execution_environment.id, training_dataset_id=dataset.id, files=[("path/to/model.jar", "model.jar")], ) custom_model_version_id = custom_model_version.id # 2. register the custom model version as a registered model registered_model = dr.RegisteredModelVersion.create_for_custom_model_version( custom_model_version_id=custom_model_version.id, name=model_name, registered_model_name=model_name ) # 3. initialize the registered model for insights autodocs = dr.AutomatedDocument( entity_id=registered_model.id, document_type="MODEL_COMPLIANCE", ) autodocs.initialize_model_compliance() assert autodocs.is_model_compliance_initialized[0] # Add the scoring dataset to the AI catalog scoring_dataset = dr.Dataset.create_from_file(file_path="path/to/scoring_dataset.csv") # Request the ShapMatrix computation, and retrieve results when it finishes job = ShapMatrix.compute( entity_id=custom_model_version_id, source='externalTestSet', external_dataset_id=scoring_dataset.id, entity_type="customModel", ) result = job.get_result_when_complete() print(result.columns) >>> ['AUCGUART', 'Color', 'Make', ... print(result.matrix) >>> [[ 1.22604372e-02 1.98424454e-01 2.23308013e-01 ...] ... ] ``` --- # Jobs URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/job.html > Learn how to manage and monitor jobs in DataRobot projects, including model creation jobs and queue management. The [Job](https://docs.datarobot.com/en/docs/api/reference/sdk/jobs.html#jobs-api) class is a generic representation of jobs running in a project's queue. Many tasks for modeling, such as creating a new model or computing Feature Impact for a model, use a job to track the worker usage and progress of the associated task. ## Check the contents of the queue To see what jobs running or waiting in the queue for a project, use the `Project.get_all_jobs` method. ``` from datarobot.enums import QUEUE_STATUS jobs_list = project.get_all_jobs() # gives all jobs queued or inprogress jobs_by_type = {} for job in jobs_list: if job.job_type not in jobs_by_type: jobs_by_type[job.job_type] = [0, 0] if job.status == QUEUE_STATUS.QUEUE: jobs_by_type[job.job_type][0] += 1 else: jobs_by_type[job.job_type][1] += 1 for type in jobs_by_type: (num_queued, num_inprogress) = jobs_by_type[type] print('{} jobs: {} queued, {} inprogress'.format(type, num_queued, num_inprogress)) ``` ## Cancel a job If a job is taking too long to run or no longer necessary, it can be cancelled from the `Job` object. ``` from datarobot.enums import QUEUE_STATUS project.pause_autopilot() bad_jobs = project.get_all_jobs(status=QUEUE_STATUS.QUEUE) for job in bad_jobs: job.cancel() project.unpause_autopilot() ``` ## Retrieve results from a job You can retrieve the results of a job once it is complete. Note that the type of the returned object varies depending on the `job_type`. All return types are documented in `Job.get_result`. ``` from datarobot.enums import JOB_TYPE time_to_wait = 60 * 60 # how long to wait for the job to finish (in seconds) - i.e. an hour assert my_job.job_type == JOB_TYPE.MODEL my_model = my_job.get_result_when_complete(max_wait=time_to_wait) ``` ### Model jobs Model creation is an asynchronous process. This means that when explicitly invoking new model creation (with `project.train` or `model.train` for example), all you get is the ID of the process responsible for model creation. With this ID, you can get info about the model that is being created—or the model itself, once the creation process is finished—by using the `ModelJob` class. ## Get an existing model job To retrieve existing model jobs, use the `ModelJob.get` method. For this, you need the ID of the project from which the model was built and the ID of the model job. The model job is useful if you want to know the parameters for a model’s creation (automatically chosen by the API backend) before the actual model was created. If the model is already created, `ModelJob.get` will raise the `PendingJobFinished` exception. ``` import time import datarobot as dr blueprint_id = '5506fcd38bd88f5953219da0' model_job_id = project.train(blueprint_id) model_job = dr.ModelJob.get(project_id=project.id, model_job_id=model_job_id) model_job.sample_pct >>> 64.0 # wait for model to be created (in a very inefficient way) time.sleep(10 * 60) model_job = dr.ModelJob.get(project_id=project.id, model_job_id=model_job_id) >>> datarobot.errors.PendingJobFinished # get the job attached to the model model_job.model >>> Model('5d518cd3962d741512605e2b') ``` ## Get a created model After a model is created, you can use `ModelJob.get_model` to get the newly-created model. ``` import datarobot as dr model = dr.ModelJob.get_model(project_id=project.id, model_job_id=model_job_id) ``` ## Async model creation If you want to get the created model after getting the model job ID, you can use the [wait_for_async_model_creation](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#wait-for-async-model-creation-api-label) function. It will poll for the status of the model creation process until it’s finished, and then return the newly-created model. Note the differences below between datetime partitioned projects and non-datetime partitioned projects. ``` from datarobot.models.modeljob import wait_for_async_model_creation # Used during training based on blueprint model_job_id = project.train(blueprint, sample_pct=33) new_model = wait_for_async_model_creation( project_id=project.id, model_job_id=model_job_id, ) # Used during training based on existing model model_job_id = existing_model.train(sample_pct=33) new_model = wait_for_async_model_creation( project_id=existing_model.project_id, model_job_id=model_job_id, ) # For datetime-partitioned projects, use project.train_datetime. Note that train_datetime returns a model job instead # of just an ID. model_job = project.train_datetime(blueprint) new_model = wait_for_async_model_creation( project_id=project.id, model_job_id=model_job.id ) ``` --- # Models URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/model.html > Learn how to train, retrieve, and analyze DataRobot models. When a blueprint has been trained on a specific dataset at a specified sample size, the result is a model. Models can be inspected to analyze their accuracy. ## Start training a model To start training a model, use the [Project.train](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.train) method with a blueprint object: ``` import datarobot as dr project = dr.Project.get('5506fcd38bd88f5953219da0') blueprints = project.get_blueprints() model_job_id = project.train(blueprints[0].id) ``` For a datetime partitioned project (see the specialized workflows section), use [Project.train_datetime](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.train_datetime): ``` import datarobot as dr project = dr.Project.get('5506fcd38bd88f5953219da0') blueprints = project.get_blueprints() model_job_id = project.train_datetime(blueprints[0].id) ``` ## List finished models You can use the [Project.get_models](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.get_models) method to return a list of the project models that have finished training: ``` import datarobot as dr project = dr.Project.get('5506fcd38bd88f5953219da0') models = project.get_models() print(models[:5]) >>> [Model(Decision Tree Classifier (Gini)), Model(Auto-tuned K-Nearest Neighbors Classifier (Minkowski Distance)), Model(Gradient Boosted Trees Classifier (R)), Model(Gradient Boosted Trees Classifier), Model(Logistic Regression)] model = models[0] project.id >>> u'5506fcd38bd88f5953219da0' model.id >>> u'5506fcd98bd88f1641a720a3' ``` You can pass following parameters to change the result: - search_params - A dict. Used to filter returned projects. Currently, you can query models by name , sample_pct , and is_starred . - order_by — A str or list. If passed, returned models are ordered by this attribute(s). You can sort by the metric and sample_pct attributes. If the `sort` attribute is preceded by a hyphen, models will be sorted in descending order, otherwise, in ascending order. Multiple `sort` attributes can be included as a comma-delimited string or in a list, e.g., `order_by='sample_pct,-metric'` or `order_by=['sample_pct', '-metric']`. Using `metric` to sort will result in models being sorted according to their validation score by how well they did according to the project metric. - with_metric – A str. If not set as None , the returned models will only have scores for this metric. Otherwise, all the metrics are returned. Review an example of listing models below. ``` import datarobot as dr dr.Project('5506fcd38bd88f5953219da0').get_models(order_by=['sample_pct', '-metric']) # Getting models that contain "Ridge" in name # and with sample_pct more than 64 dr.Project('5506fcd38bd88f5953219da0').get_models( search_params={ 'sample_pct__gt': 64, 'name': "Ridge" }) # Getting models marked as starred dr.Project('5506fcd38bd88f5953219da0').get_models( search_params={ 'is_starred': True }) ``` ## Retrieve a known model If you know the `model_id` and `project_id` values of a model, you can retrieve it directly: ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' model = dr.Model.get(project=project_id, model_id=model_id) ``` You can also use an instance of `Project` as the parameter for [Model.get](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get). ``` model = dr.Model.get(project=project, model_id=model_id) ``` ## Retrieve the highest scoring model for a given metric You can retrieve the highest scoring model for a project based on a metric of your choice. If you decide not to pass a metric to this method or if you pass the default project metric (the value of the `metric` attribute of your project instance), the result of [Project.recommended_model](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.recommended_model) is returned. ``` import datarobot as dr project = dr.Project.get('5506fcd38bd88f5953219da0') top_model_r_squared = project.get_top_model(metric="R Squared") ``` ## Train a model on a different sample size One of the key insights into a model and the data behind it is how its performance varies with more training data. In Autopilot, DataRobot runs at several sample sizes by default, but you can also create a job that will run at a specific sample size, or specify a feature list that should be used for training the new model. The [Model.train](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.train) method of a `Model` instance will put a new modeling job into the queue and return the ID of the created [ModelJob](https://docs.datarobot.com/en/docs/api/reference/sdk/jobs.html). You can pass the model job ID to the [wait_for_async_model_creation](https://docs.datarobot.com/en/docs/api/reference/sdk/jobs.html#wait-for-async-model-creation-label) function, which polls the async model creation status and returns the newly-created model when it’s finished. ``` import datarobot as dr model_job_id = model.train(sample_pct=33) # Retrain a model on a custom featurelist using cross validation. # Note that you can specify a custom value for `sample_pct`. model_job_id = model.train( sample_pct=55, featurelist_id=custom_featurelist.id, scoring_type=dr.SCORING_TYPE.cross_validation, ) ``` ## Cross-validating a model By default, models are evaluated on the first validation partition. To start cross-validation, use [Model.cross_validate](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.cross_validate): ``` import datarobot as dr model_job_id = model.cross_validate() ``` For a :doc:datetime partitioned project , backtesting is the only cross-validation method supported. To run backtesting for a datetime model, use the [DatetimeModel.score_backtests](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.score_backtests) method: ``` import datarobot as dr # `model` here must be an instance of `dr.DatetimeModel`. model_job_id = model.score_backtests() ``` ## Find the features used Because each project can have many associated feature lists, it is important to know which features a model requires in order to run. This helps ensure that the necessary features are provided when generating predictions. ``` feature_names = model.get_features_used() print(feature_names) >>> ['MonthlyIncome', 'VisitsLast8Weeks', 'Age'] ``` ## Feature Impact Feature Impact measures how much worse a model’s error score would be if DataRobot made predictions after randomly shuffling a particular column (a technique sometimes called `Permutation Importance`). The following example code snippet shows how a feature list with just the features with the highest feature impact could be created. ``` import datarobot as dr max_num_features = 10 time_to_wait_for_impact = 4 * 60 # seconds feature_impacts = model.get_or_request_feature_impact(time_to_wait_for_impact) feature_impacts.sort(key=lambda x: x['impactNormalized'], reverse=True) final_names = [f['featureName'] for f in feature_impacts[:max_num_features]] project.create_featurelist('highest_impact', final_names) ``` For datetime-aware models, Feature Impact can be calculated for any backtest and holdout. ``` import datarobot as dr datetime_model = dr.Model.get(project=project_id, model_id=model_id) feature_impacts = datetime_model.get_or_request_feature_impact(backtest=1, with_metadata=True) ``` ## Feature Effects Feature Effects helps to understand how changing a single feature affects the target while holding all other features constant. Feature Effects provides partial dependence plot and prediction vs accuracy plot data. ``` import datarobot as dr feature_effects = model.get_or_request_feature_effect(source='validation') ``` For multiclass models use `request_feature_effects_multiclass` and `get_feature_effects_multiclass` or `get_or_request_feature_effects_multiclass` methods. ``` import datarobot as dr feature_effects = model.get_feature_effect(source='validation') ``` ## Predict new data After creating models, you can use them to generate predictions on new data. See the [predictions documentation](https://docs.datarobot.com/en/docs/workbench/nxt-console/nxt-predictions/index.html#predictions) for further information on how to request predictions from a model. ## Model IDs vs. blueprint IDs Each model has both a `model_id` and a `blueprint_id`. A model is the result of training a blueprint on a dataset at a specified sample percentage. The `blueprint_id` is used to keep track of which blueprint was used to train the model, while the `model_id` is used to locate the trained model in the system. ## Model parameters Some models can have parameters that provide data needed to reproduce their predictions. For additional usage information see [Coefficients](https://docs.datarobot.com/en/docs/workbench/experiments/experiment-insights/ml-coefficients.html#coefficients). ``` import datarobot as dr model = dr.Model.get(project=project, model_id=model_id) mp = model.get_parameters() print(mp.derived_features) >>> [{ 'coefficient': -0.015, 'originalFeature': u'A1Cresult', 'derivedFeature': u'A1Cresult->7', 'type': u'CAT', 'transformations': [{'name': u'One-hot', 'value': u"'>7'"}] }] ``` ## Create a blender model You can blend multiple models; in many cases, the resulting blender model is more accurate than the parent models. To do so, you need to select parent models and a blender method from `datarobot.enums.BLENDER_METHOD`. If this is a time series project, only methods in `datarobot.enums.TS_BLENDER_METHOD` are allowed. Be aware that the tradeoff for better prediction accuracy is bigger resource consumption and slower predictions. ``` import datarobot as dr pr = dr.Project.get(pid) models = pr.get_models() parent_models = [model.id for model in models[:2]] pr.blend(parent_models, dr.enums.BLENDER_METHOD.AVERAGE) ``` ## Lift chart retrieval You can use the `Model` methods `get_lift_chart` and `get_all_lift_charts` to retrieve lift chart data. The first will get it from specific source (validation data, cross validation, or unlocked Holdout) and the second will list all available data. For multiclass models, you can get a list of per-class lift charts using the `Model` method `get_multiclass_lift_chart`. ## ROC curve retrieval Same as with the lift chart, you can use `Model` methods `get_roc_curve` and `get_all_roc_curves` to retrieve ROC curve data. The first gets the curve from a specific source (validation data, cross validation, or unlocked Holdout); the second lists all available data. More information about working with ROC curves can be found in [ROC curve](https://docs.datarobot.com/en/docs/workbench/experiments/experiment-insights/ml-roc-curve.html). To get the best F1 threshold and all threshold data for validation (or holdout) data: ``` import datarobot as dr import pandas as pd p = dr.Project.get("68dc7024820ddddeaa9b5b72") ms = p.get_models() m = ms[1] roc = m.get_roc_curve("validation") # Best threshold (maximal F1 score; same as preselected in the ROC curve tab): roc.get_best_f1_threshold() # All thresholds and metrics: pd.DataFrame(roc.roc_points) ``` ## Residuals chart retrieval Just as with the lift and ROC charts, you can use `Model` methods `get_residuals_chart` and `get_all_residuals_charts` to retrieve residuals chart data. The first will get it from a specific source (validation data, cross-validation data, or unlocked Holdout). The second retrieves all available data. ## Word cloud If your dataset contains text columns, DataRobot can create text processing models that will contain word cloud insight data. An example of such a model is any “Auto-Tuned Word N-Gram Text Modeler” model. You can use the `Model.get_word_cloud` method to retrieve those insights — it provides up to the 200 most important ngrams in the model and coefficients corresponding to their influence. ## Scoring Code A subset of models support code generation. For each of those models, you can download a JAR file with Scoring Code to make predictions locally using `model.download_scoring_code`. For details on how to do so, see [Scoring Code](https://docs.datarobot.com/en/docs/predictions/port-pred/scoring-code/index.html). Optionally, you can download source code in Java to see what calculations those models do internally. Be aware that the source code JAR isn’t compiled so it cannot be used for making predictions. ## Get a model blueprint chart For any model, you can retrieve its blueprint chart. You can also get its representation in graphviz DOT format to render it into the format you need. ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' model = dr.Model.get(project=project_id, model_id=model_id) bp_chart = model.get_model_blueprint_chart() print(bp_chart.to_graphviz()) ``` ## Get a model missing values report For the majority of models, you can retrieve their missing values reports on training data per each numeric and categorical feature. Model needs to have at least one of the supported tasks in the blueprint in order to have a missing values report (blenders are not supported). Report is gathered for Numerical Imputation tasks and Categorical converters like Ordinal Encoding, One-Hot Encoding, etc. Missing values report is available to users with access to full blueprint docs. A report is collected for those features which are considered eligible by a given blueprint task. For instance, a categorical feature with a lot of unique values may not be considered as eligible in the One-Hot encoding task. Please refer to [Missing report attributes description](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#missing-values-report-api) for report interpretation. ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' model = dr.Model.get(project=project_id, model_id=model_id) missing_reports_per_feature = model.get_missing_report_info() for report_per_feature in missing_reports_per_feature: print(report_per_feature) ``` Consider the following example of a Decision Tree Classifier (Gini) blueprint chart representation. A summary of the results is outlined below. ``` print(blueprint_chart.to_graphviz()) >>> digraph "Blueprint Chart" { graph [rankdir=LR] 0 [label="Data"] -2 [label="Numeric Variables"] 2 [label="Missing Values Imputed"] 3 [label="Decision Tree Classifier (Gini)"] 4 [label="Prediction"] -1 [label="Categorical Variables"] 1 [label="Ordinal encoding of categorical variables"] 0 -> -2 -2 -> 2 2 -> 3 3 -> 4 0 -> -1 -1 -> 1 1 -> 3 } ``` And a missing report: ``` print(report_per_feature1) >>> {'feature': 'Veh Year', 'type': 'Numeric', 'missing_count': 150, 'missing_percentage': 50.00, 'tasks': [ {'id': u'2', 'name': u'Missing Values Imputed', 'descriptions': [u'Imputed value: 2006'] } ] } print(report_per_feature2) >>> {'feature': 'Model', 'type': 'Categorical', 'missing_count': 100, 'missing_percentage': 33.33, 'tasks': [ {'id': u'1', 'name': u'Ordinal encoding of categorical variables', 'descriptions': [u'Imputed value: -2'] } ] } ``` The numeric feature “Veh Year” has 150 missing values and, respectively, 50% in training data. It was transformed by the “Missing Values Imputed” task with imputed value 2006. Task has ID 2, and its output goes into Decision Tree Classifier (Gini), which can be inferred from the chart. The “Model” categorical feature was transformed by “Ordinal encoding of categorical variables” task with imputed value -2. ## Get a blueprint's documentation You can retrieve documentation on tasks used to build a model. It will contain information about the task, its parameters and (when available) links and references to additional sources. All documents are instances of `BlueprintTaskDocument` class. ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' model = dr.Model.get(project=project_id, model_id=model_id) docs = model.get_model_blueprint_documents() print(docs[0].task) >>> Average Blend print(docs[0].links[0]['url']) >>> https://en.wikipedia.org/wiki/Ensemble_learning ``` ## Request training predictions You can request a model’s predictions for a particular subset of its training data. See [datarobot.models.Model.request_training_predictions()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_training_predictions) reference for all the valid subsets. See [training predictions reference](https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/predict_job.html#training-predictions) for more details. ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' model = dr.Model.get(project=project_id, model_id=model_id) training_predictions_job = model.request_training_predictions(dr.enums.DATA_SUBSET.HOLDOUT) training_predictions = training_predictions_job.get_result_when_complete() for row in training_predictions.iterate_rows(): print(row.row_id, row.prediction) ``` ## Advanced tuning You can perform advanced tuning on a model — generate a new model by taking an existing model and rerunning it with modified tuning parameters. The `AdvancedTuningSession` class exists to track the creation of an advanced tuning model on the client. It enables browsing and setting advanced tuning parameters one at a time, and using human-readable parameter names rather than requiring opaque parameter IDs in all cases. No information is sent to the server until the `run()` method is called on the AdvancedTuningSession. See [datarobot.models.Model.get_advanced_tuning_parameters()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_advanced_tuning_parameters) reference for a description of the types of parameters that can be passed in. As of v2.17 of the Python client, all models other than blenders, open source, and user-created models support Advanced Tuning. The use of Advanced Tuning via the API for non-Eureqa models is in beta, but is enabled by default for all users. ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' model = dr.Model.get(project=project_id, model_id=model_id) tune = model.start_advanced_tuning_session() # Get available task names, # and available parameter names for a task name that exists on this model tune.get_task_names() tune.get_parameter_names('Eureqa Generalized Additive Model Classifier (3000 Generations)') tune.set_parameter( task_name='Eureqa Generalized Additive Model Classifier (3000 Generations)', parameter_name='EUREQA_building_block__sine', value=1) job = tune.run() ``` For specific grid search options, pass the grid search arguments when starting the session. ``` import datarobot as dr from datarobot.models.advanced_tuning import GridSearchArguments from datarobot.enums import GridSearchSearchType, GridSearchAlgorithm project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' model = dr.Model.get(project=project_id, model_id=model_id) # Add grid search arguments grid_args = GridSearchArguments( search_type=GridSearchSearchType.SMART, algorithm=GridSearchAlgorithm.PATTERN_SEARCH, batch_size=7, max_iterations=50, random_state=7, wall_clock_time_limit=1000, ) tune = model.start_advanced_tuning_session(grid_search_arguments=grid_args) # Get available task names, # and available parameter names for a task name that exists on this model tune.get_task_names() tune.get_parameter_names('Generalized Additive Model') params_to_tune = { 'colsample_bytree': 0.5, 'learning_rate': [0.08, 0.07], 'max_depth': 6, 'min_child_weight': 2, 'n_estimators':320, } for param, value in params_to_tune.items(): tune.set_parameter( task_name='Generalized Additive Model', parameter_name=param, value=value) job = tune.run() job.get_result_when_complete() ``` ## SHAP Feature Impact SHAP is an open-source method for explaining the predictions from machine learning models. You can find more information about SHAP at its repository on [GitHub](https://github.com/slundberg/shap). DataRobot supports SHAP computations for all regression and binary classification blueprints. You can compute SHAP feature impact, which reports the overall importance for each feature column across all rows, based on aggregated SHAP matrix values. The following example code assumes that you have a trained model object called `model`. ``` import datarobot as dr from datarobot.insights.shap_impact import ShapImpact project_id = '5ec3d6884cfad17cd8c0ed62' model_id = model.id # or model_id = 'YOUR_MODEL_ID' # request SHAP Impact; only works for training currently job = ShapImpact.compute(entity_id=model_id, source='training') result = job.get_result_when_complete() # Impacts are listed as [feature_name, normalized_impact, unnormalized_impact] print(result.shap_impacts) >>> [['AUCGUART', 0.07989059458051094, 0.022147886593333888], ...] # Retrieve a SHAP Impact record that was previously calculated shap_impact = ShapImpact.get(entity_id=model_id, source='training') ``` ## Number of iterations trained Early-stopping models will train a subset of max estimators/iterations that are defined in advanced tuning. This method allows the user to retrieve the actual number of estimators that were trained by an early-stopping tree-based model (currently the only model type supported). The method returns the projectId, modelId, and a list of dictionaries containing the number of iterations trained for each model stage. In the case of single-stage models, this dictionary will contain only one entry. ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' model = dr.Model.get(project=project_id, model_id=model_id) num_iterations = model.get_num_iterations_trained() print(num_iterations) >>> {"projectId": "5506fcd38bd88f5953219da0", "modelId": "5506fcd98bd88f1641a720a3", "data" [{"stage": "FREQ", "numIterations":250}, {"stage":"SEV", "numIterations":50}]} ``` --- # Model recommendation URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/model_recommendation.html > Learn how to retrieve and work with DataRobot model recommendations for deployment. During Autopilot, DataRobot recommends a model for deployment based on its accuracy and complexity. When running Autopilot in Full or Comprehensive mode, DataRobot uses the following deployment preparation process: 1. First, DataRobot calculates Feature Impact for the selected model and uses it to generate a reduced feature list. 2. Next, DataRobot retrains the selected model on the reduced feature list. If the new model performs better than the original model, DataRobot uses the new model for the next stage. Otherwise, the original model is used. 3. DataRobot then retrains the selected model at an up-to-holdout sample size (typically 80%). As long as the sample is under the frozen threshold (1.5GB), the stage is not frozen. 4. Finally, DataRobot retrains the selected model as a frozen run (hyperparameters are not changed from the up-to-holdout run) using a 100% sample size and selects it as Recommended for Deployment . > [!NOTE] Note > The higher sample size DataRobot uses in Step 3 is either: > > Up to holdout > if the training sample size > does not > exceed the maximum Autopilot size threshold: sample size is the training set plus the validation set (for TVH) or 5-folds (for CV). In this case, DataRobot compares retrained and original models on the holdout score. > Up to validation > if the training sample size > does > exceed the maximum Autopilot size threshold: sample size is the training set (for TVH) or 4-folds (for CV). In this case, DataRobot compares retrained and original models on the validation score. DataRobot gives one model the Recommended for Deployment* badge. This is the most accurate individual, non-blender model on the Leaderboard. After completing the steps described above, it will receive the Prepared for Deployment badge. ## Retrieve all recommendations The following code will return all models recommended for the project. ``` import datarobot as dr recommendations = dr.ModelRecommendation.get_all(project_id) ``` ## Retrieve a default recommendation If you are unsure about the tradeoffs between the various types of recommendations, DataRobot can make this choice for you. The following route will return the “Recommended for Deployment” model to use for predictions for the project. ``` import datarobot as dr recommendation = dr.ModelRecommendation.get(project_id) ``` ## Retrieve a specific recommendation If you know which recommendation you want to use, you can select a specific recommendation using the following code. ``` import datarobot as dr recommendation_type = dr.enums.RECOMMENDED_MODEL_TYPE.RECOMMENDED_FOR_DEPLOYMENT recommendations = dr.ModelRecommendation.get(project_id, recommendation_type) ``` ## Get recommended model You can use method `get_model()` of a recommendation object to retrieve a recommended model. ``` import datarobot as dr recommendation = dr.ModelRecommendation.get(project_id) recommended_model = recommendation.get_model() ``` --- # DataRobot Prime URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/prime.html > Learn how to use DataRobot Prime to download executable code that approximates models. > [!NOTE] Availability information > The ability to create new DataRobot Prime models has been removed. This does not affect existing Prime models or deployments. To export Python code in the future, use the Python code export function in any RuleFit model. DataRobot Prime allows the download of executable code approximating models. For more information about this feature, see the documentation within the DataRobot webapp. Contact your Account Executive or CFDS for information on enabling DataRobot Prime, if needed. ## Approximate a model Given a Model you wish to approximate, `Model.request_approximation` will start a job creating several `Ruleset` objects approximating the parent model. Each of those rulesets will identify how many rules were used to approximate the model, as well as the validation score the approximation achieved. ``` rulesets_job = model.request_approximation() rulesets = rulesets_job.get_result_when_complete() for ruleset in rulesets: info = (ruleset.id, ruleset.rule_count, ruleset.score) print('id: {}, rule_count: {}, score: {}'.format(*info)) ``` ## Prime models vs. models Given a ruleset, you can create a model based on that ruleset. We consider such models to be Prime models. The `PrimeModel` class inherits from the `Model` class, so anything a Model can do, as PrimeModel can do as well. The `PrimeModel` objects available within a `Project` can be listed by `project.get_prime_models`, or a particular one can be retrieve via `PrimeModel.get`. If a ruleset has not yet had a model built for it, `ruleset.request_model` can be used to start a job to make a PrimeModel using a particular ruleset. ``` rulesets = parent_model.get_rulesets() selected_ruleset = sorted(rulesets, key=lambda x: x.score)[-1] if selected_ruleset.model_id: prime_model = PrimeModel.get(selected_ruleset.project_id, selected_ruleset.model_id) else: prime_job = selected_ruleset.request_model() prime_model = prime_job.get_result_when_complete() ``` The `PrimeModel` class has two additional attributes and one additional method. The attributes are `ruleset`, which is the Ruleset used in the PrimeModel, and `parent_model_id` which is the id of the model it approximates. Finally, the new method defined is `request_download_validation` which is used to prepare code download for the model and is discussed later on in this document. ## Retrieving Code from a PrimeModel Given a PrimeModel, you can download the code used to approximate the parent model, and view and execute it locally. The first step is to validate the PrimeModel, which runs some basic validation of the generated code, as well as preparing it for download. We use the `PrimeFile` object to represent code that is ready to download.`PrimeFiles` can be prepared by the `request_download_validation` method on `PrimeModel` objects, and listed from a project with the `get_prime_files` method. Once you have a `PrimeFile` you can check the `is_valid` attribute to verify the code passed basic validation, and then download it to a local file with `download`. ``` validation_job = prime_model.request_download_validation(enums.PRIME_LANGUAGE.PYTHON) prime_file = validation_job.get_result_when_complete() if not prime_file.is_valid: raise ValueError('File was not valid') prime_file.download('/home/myuser/drCode/primeModelCode.py') ``` --- # Projects URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/project.html > Learn how to create, configure, and manage DataRobot projects for modeling. All of the modeling within DataRobot happens within a project. Each project has one dataset that is used as the source from which to train models. ## Create a project You can create a project from previously-created [Datasets](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/dataset.html#datasets) or directly from a data source. ``` import datarobot as dr dataset = Dataset.create_from_file(file_path='/home/user/data/last_week_data.csv') project = dr.Project.create_from_dataset(dataset.id, project_name='New Project') ``` The following command creates a new project directly from a data source. You must specify a path to data file, file object URL (starting with `http://`, `https://`, `file://`, or `s3://`), raw file contents, or a `pandas.DataFrame` object when creating a new project. Path to file can be either a path to a local file or a publicly accessible URL. ``` import datarobot as dr project = dr.Project.create('/home/user/data/last_week_data.csv', project_name='New Project') ``` You can use the following commands to view the project ID and name: ``` project.id >>> u'5506fcd38bd88f5953219da0' project.project_name >>> u'New Project' ``` ## Select modeling parameters The final information needed to begin modeling includes the target feature, queue mode, metric for comparing models, and optional parameters such as weights, offset, exposure, and downsampling. ### Target The target must be the name of one of the columns of data uploaded to the project. ### Metric The optimization metric used to compare models is an important factor in building accurate models. If a metric is not specified, the default metric recommended by DataRobot will be used. You can use the following code to view a list of valid metrics for a specified target: ``` target_name = 'ItemsPurchased' project.get_metrics(target_name) >>> {'available_metrics': [ 'Gini Norm', 'Weighted Gini Norm', 'Weighted R Squared', 'Weighted RMSLE', 'Weighted MAPE', 'Weighted Gamma Deviance', 'Gamma Deviance', 'RMSE', 'Weighted MAD', 'Tweedie Deviance', 'MAD', 'RMSLE', 'Weighted Tweedie Deviance', 'Weighted RMSE', 'MAPE', 'Weighted Poisson Deviance', 'R Squared', 'Poisson Deviance'], 'feature_name': 'SalePrice'} ``` ### Partitioning method DataRobot projects always have a `Holdout` set used for final model validation. You can use two different approaches for testing prior to the Holdout set: - Split the remaining data into training and validation sets. - Cross-validation, in which the remaining data is split into a number of folds (partitions); each fold serves as a validation set, with models trained on the other folds and evaluated on that fold. There are several other options you can control. To specify a partition method, create an instance of one of the [Partition Classes](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#partitions-api), and pass it as the `partitioning_method` argument in your call to `project.analyze_and_model` or `project.start`. As of v3.0 of the Python client, you can alternately use `project.set_partitioning_method`. See [here](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/datetime_partition.html#set-up-datetime) for more information on using datetime partitioning. Several partitioning methods include parameters for `validation_pct` and `holdout_pct`, specifying desired percentages for the validation and holdout sets. Note that there may be constraints that prevent the actual percentages used from exactly (or some cases, even closely) matching the requested percentages. ### Queue mode You can use the API to set the DataRobot modeling process to run Autopilot in manual, quick, or comprehensive mode. Autopilot mode means that the modeling process will proceed completely automatically, including running recommended models, running at different sample sizes, and blending. Manual mode means that DataRobot will populate a list of recommended models, but will not insert any of them into the queue. This mode lets you specify which models to execute before starting the modeling process. Quick mode means that a smaller set of blueprints is used, so Autopilot finishes faster. ### Weights DataRobot also supports using a `weight` parameter, which are often used to help compensate for rare events in data. You can specify a column name in the project dataset to be used as a `weight` column. ### Offsets Starting with Python client v2.6, DataRobot also supports using an offset parameter. Offsets are commonly used in insurance modeling to include effects that are outside of the training data due to regulatory compliance or constraints. You can specify the names of several columns in the project dataset to be used as the offset columns. ### Exposure Starting with version v2.6, DataRobot also supports using an exposure parameter. Exposure is often used to model insurance premiums where strict proportionality of premiums to duration is required. You can specify the name of the column in the project dataset to be used as an exposure column. ## Start modeling Once you have selected modeling parameters, you can use the following code structure to specify parameters and start the modeling process. ``` import datarobot as dr project.analyze_and_model(target='ItemsPurchased', metric='Tweedie Deviance', mode=dr.AUTOPILOT_MODE.FULL_AUTO) ``` You can also pass additional parameters to `project.analyze_and_model` to change parts of the modeling process. Some of those parameters include: - worker_count - int, sets number of workers used for modeling. - partitioning_method - PartitioningMethod object. - positive_class - str, float, or int; Specifies a level of the target column that should be treated as the positive class for binary classification. May only be specified for binary classification targets. - advanced_options - AdvancedOptions object; Used to set advanced options of modeling process. Can alternatively call set_options on a project instance which will be used automatically if nothing is passed here. - target_type - str; Overrides the automatically selected target_type . An example usage would be setting the target_type=TARGET_TYPE.MULTICLASS when you want to perform a multiclass classification task on a numeric column that has a low cardinality. You can run different Autopilot modes with the `mode` parameter.`AUTOPILOT_MODE.FULL_AUTO` is the default, which will trigger modeling with no further actions necessary. Other accepted modes include `AUTOPILOT_MODE.MANUAL` for manual mode (choose your own models to run rather than use the DataRobot autopilot), `AUTOPILOT_MODE.QUICK` (run on a more limited set of models to get insights more quickly), and `AUTOPILOT_MODE.COMPREHENSIVE` (used to invest more time to find the most accurate model to serve your use case). For a full reference of available parameters, see [Project.analyze_and_model](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.analyze_and_model). ### Clone a project Once a project has been successfully created, you may clone it using the following code structure: ``` new_project = project.clone_project(new_project_name='This is my new project') new_project.project_name >> 'This is my new project' new_project.id != project.id >> True ``` The `new_project_name` attribute is optional. If it is omitted, the default new project name will be ‘Copy of ’. ## Interact with a project The following commands can be used to manage DataRobot projects. ### List projects Returns a list of projects associated with current API user. ``` import datarobot as dr dr.Project.list() >>> [Project(Project One), Project(Two)] dr.Project.list(search_params={'project_name': 'One'}) >>> [Project(One)] ``` You can pass following parameter to change the result: - search_params – dict; Used to filter returned projects. You can only query projects by project_name . ### Get an existing project Rather than querying the full list of projects every time you need to interact with a project, you can retrieve its `ID` value and use that to reference the project. ``` import datarobot as dr project = dr.Project.get(project_id='5506fcd38bd88f5953219da0') project.id >>> '5506fcd38bd88f5953219da0' project.project_name >>> 'Churn Projection' ``` ### Get feature association statistics for an existing project You can retrieve either feature association or correlation statistics and metadata on informative features for a given project. ``` import datarobot as dr project = dr.Project.get(project_id='5506fcd38bd88f5953219da0') association_data = project.get_associations(assoc_type='association', metric='mutualInfo') association_data.keys() >>> ['strengths', 'features'] ``` ### Get whether your featurelists have association statistics Get whether an association matrix job has been run on each of your feature lists. ``` import datarobot as dr project = dr.Project.get(project_id='5506fcd38bd88f5953219da0') featurelists = project.get_association_featurelists() featurelists['featurelists'][0] >>> {"featurelistId": "54e510ef8bd88f5aeb02a3ed", "hasFam": True, "title": "Informative Features"} ``` ### Create association statistics for a featurelist Generate the feature association statistics for all features in a feature list. ``` import datarobot as dr from datarobot.models.feature_association_matrix import FeatureAssociationMatrix project = dr.Project.get(project_id='5506fcd38bd88f5953219da0') featurelist = project.get_featurelist_by_name("Raw Features") status = FeatureAssociationMatrix.create(project.id, featurelist.id) # two ways to wait for completion # option 1 status.wait_for_completion() fam = FeatureAssociationMatrix.get(project_id=project.id, featurelist_id=featurelist.id) # or option 2 # fam = status.get_result_when_complete() ``` ### Get a project's feature list by name Get a feature list by name. ``` import datarobot as dr project = dr.Project.get(project_id='5506fcd38bd88f5953219da0') featurelist = project.get_featurelist_by_name("Raw Features") featurelist >>> Featurelist(Raw Features) # Trying to get feature list that does not exist featurelist = project.get_featurelist_by_name("Flying Circus") featurelist is None >>> True ``` ### Create project feature lists Using a project’s `create_featurelist()` method, you can create feature lists in multiple ways: ``` import datarobot as dr project = dr.Project.get(project_id='5506fcd38bd88f5953219da0') featurelist_one = project.create_featurelist( name="Testing featurelist creation", features=["age", "weight", "number_diagnoses"], ) featurelist_one >>> Featurelist(Testing featurelist creation) featurelist_one.features >>> ['age', 'weight', 'number_diagnoses'] # Create a feature list using another feature list as a starting point (`starting_featurelist`) # To Note: this example passes the `featurelist` object but you can also pass the # id (`starting_featurelist_id`) or the name (`starting_featurelist_name`) featurelist_two = project.create_featurelist( starting_featurelist=featurelist_one, features_to_exclude=["number_diagnoses"], # Please see docs for use of `features_to_include` ) featurelist_two # Note below we have an auto-generated name because we did not pass `name` >>> Featurelist(Testing featurelist creation - 2022-07-12) >>> # Note below we have a new feature list which has `"number_diagnoses"` excluded featurelist_two.features >>> ['age', 'weight'] ``` ### Get values for a pair of features in an existing project Get a sample of the exact values used in the feature association matrix plotting. ``` import datarobot as dr project = dr.Project.get(project_id='5506fcd38bd88f5953219da0') feature_values = project.get_association_matrix_details(feature1='foo', feature2='bar') feature_values.keys() >>> ['features', 'types', 'values'] ``` ### Update a project You can update various attributes of a project. To update the name of the project: ``` project.rename(new_name) ``` To update the number of workers used by your project (this will fail if you request more workers than you have available; the special value `-1` will request your maximum number): ``` project.set_worker_count(num_workers) ``` To unlock the Holdout set, allowing holdout scores to be shown and models to be trained on more data: ``` project.unlock_holdout() ``` To add or change the project description: ``` project.set_project_description(project_description) ``` To add or change the project’s `advanced_options`: ``` # Using kwargs project.set_options(blend_best_models=False) # Using an ``AdvancedOptions`` instance project.set_options(AdvancedOptions(blend_best_models=False)) ``` ### Delete a project Use the following command to delete a project: ``` project.delete() ``` ### Wait for Autopilot to finish Once the modeling Autopilot is started, in some cases you will want to wait for Autopilot to finish: ``` project.wait_for_autopilot() ``` ### Play/Pause Autopilot If your project is running in Autopilot, it will continually use available workers, subject to the number of workers allocated to the project and the total number of simultaneous workers allowed according to the user permissions. To pause a project running in Autopilot: ``` project.pause_autopilot() ``` To resume running a paused project: ``` project.unpause_autopilot() ``` ### Start Autopilot on another feature list You can start Autopilot on an existing feature list. ``` import datarobot as dr featurelist = project.create_featurelist('test', ['feature 1', 'feature 2']) project.start_autopilot(featurelist.id) >>> True # Starting autopilot that is already running on the provided featurelist project.start_autopilot(featurelist.id) >>> dr.errors.AppPlatformError ``` > [!NOTE] Note > This method should be used on a project where the target has already been set. > An error will be raised if autopilot is currently running on or has already finished running on the provided feature list. ### Start preparing a specific model for deployment You can start preparing a specific model for deployment. The model will then go through the various recommendation stages including retraining on a reduced feature list and retraining the model on a higher sample size (recent data for datetime partitioned). ``` # prepare a specific model for deployment and wait for the process to complete project.start_prepare_model_for_deployment(model_id=model.id) project.wait_for_autopilot(check_interval=5, timeout=600) # get the prepared model prepared_for_deployment_model = dr.models.ModelRecommendation.get( project.id, recommendation_type=RECOMMENDED_MODEL_TYPE.PREPARED_FOR_DEPLOYMENT ) prepared_for_deployment_model_id = prepared_for_deployment_model.model_id ``` > [!NOTE] Note > This method should be used on a project where the target has already been set. > An error will be raised if autopilot is currently running on the project or another model in the project is being prepared for deployment. ### Using credential data For methods that accept credential data instead of user/password or credential ID, please see [Credential Data documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#id1). --- # Work with binary data URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/binary_data.html > Learn how to work with binary data files in DataRobot projects. ## Prepare data for training Working with binary files using the DataRobot API requires prior dataset preparation in one of the supported formats. See [“Prepare the dataset”](https://docs.datarobot.com/en/docs/modeling/special-workflows/visual-ai/vai-model.html#prepare-the-dataset) for more detail. When the dataset is ready, you can start a project following one of the methods described in working with [Datasets](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/dataset.html#datasets) and [Projects](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/project.html#projects). ## Prepare data for predictions For project creation and a lot of the prediction options, DataRobot allows you to upload archives with binary files (e.g. images files). Whenever possible it is recommended to use this option. However, in a few cases the API routes only allow you to upload your dataset in the JSON or CSV format. In these cases, you can add the binary files as base64 strings to your dataset. ## Process images ### Installation To enable support for processing images, install the datarobot library with the `images` option: ``` pip install datarobot[images] ``` This will install all needed dependencies for image processing. ### Process images When working with image files, helper functions may first transform your images before encoding their binary data as base64 strings. Specifically, helper functions will perform these steps: - Retrieve binary data from the file in the specified location (local path or URL). - Resize images to the image size used by DataRobot and save them in a different format - Convert binary data to base64-encoded strings. Working with images locally and located on external servers differs only in the steps related to binary file retrieval. The following steps for transformation and conversion to base64-encoded strings are the same. This examples uses data stored in a folder structure: ``` /home/user/data/predictions ├── images ├  ├──animal01.jpg ├  ├──animal02.jpg ├  ├──animal03.png ├── data.csv ``` As an input for processing, DataRobot needs a collection of image locations. Helper functions will process the images and return base64-encoded strings in the same order. The first example uses the contents of data.csv as an input. This file holds data needed for model predictions and also the image storage locations (in the “image_path” column). Contents of data.csv: ``` weight_in_grams,age_in_months,image_path 5000,34,/home/user/data/predictions/images/animal01.jpg 4300,56,/home/user/data/predictions/images/animal02.jpg 4200,22,/home/user/data/predictions/images/animal03.png ``` This code snippet will read each image from the “image_path” column and store the base64-string with image data in the “image_base64” column. ``` import os import pandas as pd from datarobot.helpers.binary_data_utils import get_encoded_image_contents_from_paths dataset_dir = '/home/user/data/predictions' file_in = os.path.join(dataset_dir, 'data.csv') file_out = os.path.join(dataset_dir, 'out.csv') df = pd.read_csv(file_in) df['image_base64'] = get_encoded_image_contents_from_paths(df['image_path']) df.to_csv(file_out, index=False) ``` The same helper function will work with other iterables: ``` import os from datarobot.helpers.binary_data_utils import get_encoded_image_contents_from_paths images_dir = '/home/user/data/predictions/images' images_absolute_paths = [ os.path.join(images_dir, file) for file in ['animal01.jpg', 'animal02.jpg', 'animal03.png'] ] images_base64 = get_encoded_image_contents_from_paths(images_absolute_paths) ``` Above examples used absolute paths. When working with relative paths, by default the helper function will resolve them relative to the script location. To override this behavior, use `base_path` parameter to specify the base path for relative paths. ``` from datarobot.helpers.binary_data_utils import get_encoded_image_contents_from_paths images_dir = '/home/user/data/predictions/images' images_relative_paths = ['animal01.jpg', 'animal02.jpg', 'animal03.png'] images_base64 = get_encoded_image_contents_from_paths( images_relative_paths, base_path=images_dir ) ``` There is also one helper function to work with remote data. This function retrieves binary content from specified URLs, transforms the images, and returns base64-encoded strings (in the same way as it does for images loaded from local paths). Example: ``` import os from datarobot.helpers.binary_data_utils import get_encoded_image_contents_from_urls image_urls = [ 'https:///animal01.jpg', 'https:///animal02.jpg', 'https:///animal03.png' ] images_base64 = get_encoded_image_contents_from_urls(image_urls) ``` Examples of helper functions up to this points have used default settings. If needed, the following functions allow for further customization by passing explicit parameters related to error handling, image transformations, and request header customization. ### Custom image transformations By default helper functions will apply transformations, which have proven good results. The default values align with the preprocessing used for images uploaded in archives for training. Therefore, using default values should be the first choice when preparing datasets with images for predictions. However, you can also specify custom image transformation settings to override default transformations before converting data into base64 strings. To override the default behavior, create an instance of the `ImageOptions` class and pass it as an additional parameter to the helper function. Note that there is no guarantee that images converted by DataRobot during archive dataset upload match images converted by you on a pixel level, even if the default `ImageOptions` are used. However, if you use `ImageOptions`, you most likely will not be able to visually identify any differences. Examples: ``` import os from datarobot.helpers.image_utils import ImageOptions from datarobot.helpers.binary_data_utils import get_encoded_image_contents_from_paths images_dir = '/home/user/data/predictions/images' images_absolute_paths = [ os.path.join(images_dir, file) for file in ['animal01.jpg', 'animal02.jpg', 'animal03.png'] ] # Override the default behavior for image quality and subsampling, but the images # will still be resized because that's the default behavior. Note: the `keep_quality` # parameter for JPEG files by default preserves the quality of the original images, # so this behavior must be disabled to manually override the quality setting with an # explicit value. image_options = ImageOptions(keep_quality=False, image_quality=80, image_subsampling=0) images_base64 = get_encoded_image_contents_from_paths( paths=images_absolute_paths, image_options=image_options ) # overwrite default behavior for image resizing, this will keep image aspect # ratio and will resize all images using specified size: width=300 and height=300. # Note: if image had different aspect ratio originally it will generate image # thumbnail, not larger than the original, that will fit in requested image size image_options = ImageOptions(image_size=(300, 300)) images_base64 = get_encoded_image_contents_from_paths( paths=images_absolute_paths, image_options=image_options ) # Override the default behavior for image resizing, This will force the image # to be resized to size: width=300 and height=300. When the image originally # had a different aspect ratio - than resizing it using `force_size` parameter # will alter its aspect ratio modifying the image (e.g. stretching) image_options = ImageOptions(image_size=(300, 300), force_size=True) images_base64 = get_encoded_image_contents_from_paths( paths=images_absolute_paths, image_options=image_options ) # overwrite default behavior and retain original image sizes image_options = ImageOptions(should_resize=False) images_base64 = get_encoded_image_contents_from_paths( paths=images_absolute_paths, image_options=image_options ) ``` ### Custom request headers If needed, you can specify custom request headers for downloading binary data. Example: ``` import os from datarobot.helpers.binary_data_utils import get_encoded_image_contents_from_urls token = 'Nl69vmABaEuchUsj88N0eOoH2kfUbhCCByhoFDf4whJyJINTf7NOhhPrNQKqVVJJ' custom_headers = { 'User-Agent': 'My User Agent', 'Authorization': 'Bearer {}'.format(token) } image_urls = [ 'https:///animal01.jpg', 'https:///animal02.jpg', 'https:///animal03.png', ] images_base64 = get_encoded_image_contents_from_urls(image_urls, custom_headers) ``` ### Handling errors When processing multiple images, any error during processing will, by default, stop operations (i.e., the helper function will raise `datarobot.errors.ContentRetrievalTerminatedError` and terminate further processing). In the case of an error during content retrieval (“connectivity issue”, “file not found” etc), you can override this behavior by passing `continue_on_error=True` to the helper function. When specified, processing will continue. In rows where the error was raised, the value``None`` value will be returned instead of a base64-encoded string. This applies only to errors during content retrieval, other errors will always terminate execution. Example: ``` import os from datarobot.helpers.binary_data_utils import get_encoded_image_contents_from_paths images_dir = '/home/user/data/predictions/images' images_absolute_paths = [ os.path.join(images_dir, file) for file in ['animal01.jpg', 'missing.jpg', 'animal03.png'] ] # This execution will print None for missing files and base64 strings for exising files images_base64 = get_encoded_image_contents_from_paths(images_absolute_paths, continue_on_error=True) for value in images_base64: print(value) # This execution will raise error during processing of missing file terminating operation images_base64 = get_encoded_image_contents_from_paths(images_absolute_paths) ``` ## Process other binary files Other binary files can be processed by dedicated functions. These functions work similarly to the functions used for images, although they do not provide functionality for any transformations. Processing follows two steps instead of three: > Retrieve binary data from the file in the specified location (local path or URL).Convert binary data to base64-encoded strings. To process documents into base64-encoded strings use these functions: > To retrieve files from local paths:get_encoded_file_contents_from_paths- tTo retrieve files from locations specified as URLs:get_encoded_file_contents_from_urls- Examples: ``` import os from datarobot.helpers.binary_data_utils import get_encoded_file_contents_from_urls document_urls = [ 'https:///document01.pdf', 'https:///missing.pdf', 'https:///document03.pdf', ] # this call will return base64 strings for existing documents and None for missing files documents_base64 = get_encoded_file_contents_from_urls(document_urls, continue_on_error=True) for value in documents_base64: print(value) # This execution will raise error during processing of missing file terminating operation documents_base64 = get_encoded_file_contents_from_urls(document_urls) ``` --- # Composable ML URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/custom_task.html > Learn how to use custom tasks and composable ML in DataRobot. Composable ML consists of two major components: [the DataRobot Blueprint Workshop](https://blueprint-workshop.datarobot.com/) and custom tasks, detailed below. Custom tasks provide users the ability to train models with arbitrary code in an environment defined by the user. For details on using environments, see: [Manage execution environments](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html#custom-models-execution-environments). ## Manage custom tasks Before you can upload code for a custom task, you need to create the entity that holds all the metadata. ``` import datarobot as dr from datarobot.enums import CUSTOM_TASK_TARGET_TYPE transform = dr.CustomTask.create( name="a convenient display name", # required target_type=CUSTOM_TASK_TARGET_TYPE.TRANSFORM, # required language="python", description="a longer description of the task" ) binary = dr.CustomTask.create( name="this or that", target_type=CUSTOM_TASK_TARGET_TYPE.BINARY, ) ``` A task, by itself is an empty metadata container. Before using your tasks, you need create a [CustomTaskVersion](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/custom_task.html#custom-task-versions) associated with it. A task that is ready for use will have a `latest_version` field populated with this task. ``` binary.latest_version >>> None execution_environment = dr.ExecutionEnvironment.create( name="Python3 PyTorch Environment", description="This environment contains Python3 pytorch library.", ) custom_task_folder = "datarobot-user-tasks/task_templates/python3_pytorch" task_version = dr.CustomTaskVersion.create_clean( custom_task_id=binary.id, base_environment_id=execution_environment.id, folder_path=custom_task_folder, ) binary.refresh() # In order to see the change, you need to GET it from DataRobot binary.latest_version >>> CustomTaskVersion('v1.0') ``` If you create a new version, that will be returned as the `latest_version`. You can download the latest version as a zip file. ``` binary.latest_version >>> CustomTaskVersion('v1.0') custom_task_folder = "/home/my-user-name/tasks/my-updated-task/" task_version = dr.CustomTaskVersion.create_clean( custom_task_id=binary.id, base_environment_id=execution_environment.id, folder_path=custom_task_folder, ) binary.refresh() binary.latest_version >>> CustomTaskVersion('v2.0') binary.download_latest_version("/home/my-user-name/downloads/my-task-files.zip") ``` You can `get`, `list`, `copy`, exactly as you would expect.`copy` makes a complete copy of the task: new copies of the metadata, new copies of the versions, new copies of uploaded files for the new versions. ``` all_tasks = CustomTask.list() assert {el.id for el in all_tasks} == {binary.id, transform.id} new_binary = CustomTask.copy(binary.id) assert new_binary.latest_version.id != binary.latest_version.id original_binary = CustomTask.get(binary.id) assert len(CustomTask.list()) == 3 ``` You can `update` the metadata of a task. When you do this, the object is also updated to the latest data. ``` assert binary.description == new_binary.description binary.update(description="totally new description") assert binary.description != new_binary.description assert original_binary.description != binary.description # hasn't refreshed from the server yet original_binary.refresh() assert original_binary.description == binary.description ``` And finally, you can `delete` only if the task is not in use by any of the following: - Trained models - Deployments - Blueprints in the AI catalog Once you have deleted the objects that use the task, you will be able to delete the task itself. ## Manage custom task versions Code for Custom Tasks can be uploaded by creating a Custom Task Version. When creating a Custom Task Version, the version must be associated with a base execution environment. If the base environment supports additional task dependencies (R or Python environments) and the Custom Task Version contains a valid requirements.txt file, the task version will run in an environment based on the base environment with the additional dependencies installed. ### Create custom task version Upload actual custom task content by creating a clean Custom Task Version: ``` import os from datarobot.enums import CustomTaskOutboundNetworkPolicy custom_task_id = binary.id custom_task_folder = "datarobot-user-tasks/task_templates/python3_pytorch" # add files from the folder to the custom task task_version = dr.CustomTaskVersion.create_clean( custom_task_id=custom_task_id, base_environment_id=execution_environment.id, folder_path=custom_task_folder, outbound_network_policy=CustomTaskOutboundNetworkPolicy.PUBLIC, ) ``` To create a new Custom Task Version from a previous one, with just some files added or removed, do the following: ``` import os import datarobot as dr new_files_folder = "datarobot-user-tasks/task_templates/my_files_to_add_to_pytorch_task" file_to_delete = task_version.items[0].id task_version_2 = dr.CustomTaskVersion.create_from_previous( custom_task_id=custom_task_id, base_environment_id=execution_environment.id, folder_path=new_files_folder, ) ``` Please refer to [CustomTaskFileItem](https://docs.datarobot.com/en/docs/api/reference/sdk/blueprints.html#datarobot.models.custom_task_version.CustomTaskFileItem) for description of custom task file properties. ### List custom task versions Use the following command to list Custom Task Versions available to the user: ``` import datarobot as dr dr.CustomTaskVersion.list(custom_task_id) >>> [CustomTaskVersion('v2.0'), CustomTaskVersion('v1.0')] ``` ### Retrieve custom task version To retrieve a specific Custom Task Version, run: ``` import datarobot as dr dr.CustomTaskVersion.get(custom_task_id, custom_task_version_id='5ebe96b84024035cc6a6560b') >>> CustomTaskVersion('v2.0') ``` ### Update custom task version To update Custom Task Version description execute the following: ``` import datarobot as dr custom_task_version = dr.CustomTaskVersion.get( custom_task_id, custom_task_version_id='5ebe96b84024035cc6a6560b', ) custom_task_version.update(description='new description') custom_task_version.description >>> 'new description' ``` ### Download custom task version Download content of the Custom Task Version as a ZIP archive: ``` import datarobot as dr path_to_download = '/home/user/Documents/myTask.zip' custom_task_version = dr.CustomTaskVersion.get( custom_task_id, custom_task_version_id='5ebe96b84024035cc6a6560b', ) custom_task_version.download(path_to_download) ``` ## Prepare a custom task version for use If your custom task version has dependencies, a dependency build must be completed before the task can be used. The dependency build installs your task’s dependencies into the base environment associated with the task version. see: [Prepare a custom model version for use](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/custom_model.html#custom-models-dependencies) --- # Datetime partitioned projects URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/datetime_partition.html > Learn how to set up and work with datetime partitioned projects in DataRobot. If your dataset is modeling events taking place over time, datetime partitioning may be appropriate. Datetime partitioning ensures that when partitioning the dataset for training and validation, rows are ordered according to the value of the date partition feature. ## Set up datetime partitioned projects After creating a project and before setting the target, create a [DatetimePartitioningSpecification](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datetime-part-spec) to define how the project should be partitioned. By passing the specification into `DatetimePartitioning.generate`, the full partitioning can be previewed before finalizing the partitioning. After verifying that the partitioning is correct for the project dataset, pass the specification into `Project.analyze_and_model` via the `partitioning_method` argument. Alternatively, as of v3.0, by using `Project.set_datetime_partitioning()`, the partitioning (and individual options of the partitioning specification) can be updated (with repeated method calls) up until calling `Project.analyze_and_model`. Once modeling begins, the project can be used as normal. The following code block shows the basic workflow for creating datetime partitioned projects. ``` import datarobot as dr project = dr.Project.create('some_data.csv') spec = dr.DatetimePartitioningSpecification('my_date_column') # can customize the spec as needed partitioning_preview = dr.DatetimePartitioning.generate(project.id, spec) # the preview generated is based on the project's data print(partitioning_preview.to_dataframe()) # hmm ... I want more backtests spec.number_of_backtests = 5 partitioning_preview = dr.DatetimePartitioning.generate(project.id, spec) print(partitioning_preview.to_dataframe()) # looks good project.analyze_and_model('target_column') # As of v3.0, ``Project.set_datetime_partitioning()`` and ``Project.list_datetime_partition_spec()`` # are available as an alternative: # view settings project.list_datetime_partition_spec() # maybe I want to also disable holdout before starting modeling project.set_datetime_partitioning(disable_holdout=True) # view settings project.list_datetime_partition_spec() # all of the settings look good # don't need to pass the spec into ``analyze_and_model`` because it's already been set project.analyze_and_model('target_column') # I can retrieve the partitioning settings after the target has been set too partitioning = dr.DatetimePartitioning.get(project.id) ``` ### Configure backtests Backtests are configurable using one of two methods: Method 1: > index (int): The index from zero of this backtest.gap_duration (str): A duration string such as those returned by thepartitioning_methods.construct_duration_stringhelper method. This represents the gap between training and validation scoring data for this backtest.validation_start_date (datetime.datetime): Represents the start date of the validation scoring data for this backtest.validation_duration (str): A duration string such as those returned by thepartitioning_methods.construct_duration_stringhelper method. This represents the desired duration of the validation scoring data for this backtest.importdatarobotasdrfromdatetimeimportdatetimepartitioning_spec=dr.DatetimePartitioningSpecification(backtests=[# modify the first backtest using option 1dr.BacktestSpecification(index=0,gap_duration=dr.partitioning_methods.construct_duration_string(),validation_start_date=datetime(year=2010,month=1,day=1),validation_duration=dr.partitioning_methods.construct_duration_string(years=1),)],# other partitioning settings...) Method 2 (New in version v2.20): > validation_start_date (datetime.datetime): Represents the start date of the validation scoring data for this backtest.validation_end_date (datetime.datetime): Represents the end date of the validation scoring data for this backtest.primary_training_start_date (datetime.datetime): Represents the desired start date of the training partition for this backtest.primary_training_end_date (datetime.datetime): Represents the desired end date of the training partition for this backtest.importdatarobotasdrfromdatetimeimportdatetimepartitioning_spec=dr.DatetimePartitioningSpecification(backtests=[# modify the first backtest using option 2dr.BacktestSpecification(index=0,primary_training_start_date=datetime(year=2005,month=1,day=1),primary_training_end_date=datetime(year=2010,month=1,day=1),validation_start_date=datetime(year=2010,month=1,day=1),validation_end_date=datetime(year=2011,month=1,day=1),)],# other partitioning settings...) Note that Method 2 allows you to directly configure the start and end dates of each partition, including the training partition. The gap partition is calculated as the time between `primary_training_end_date` and `validation_start_date`. Using the same date for both `primary_training_end_date` and `validation_start_date` will result in no gap being created. After configuring backtests, you can set `use_project_settings` to `True` in calls to [Model.train_datetime](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.train_datetime). This will create models that are trained and validated using your custom backtest training partition start and end dates. ## Model with datetime partitioned projects While `Model` objects can still be used to interact with the project, [DatetimeModel](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datetime-mod) objects, which are only retrievable from datetime partitioned projects, provide more information including which date ranges and how many rows are used in training and scoring the model as well as scores and statuses for individual backtests. The autopilot workflow is the same as for other projects, but to manually train a model, `Project.train_datetime` and `Model.train_datetime` should be used in the place of `Project.train` and `Model.train`. To create frozen models, `Model.request_frozen_datetime_model` should be used in place of `DatetimeModel.request_frozen_datetime_model`. Unlike other projects, to trigger computation of scores for all backtests use `DatetimeModel.score_backtests` instead of using the `scoring_type` argument in the `train` methods. ## Accuracy over time plots For datetime partitioned model you can retrieve the Accuracy over Time plot. To do so use [DatetimeModel.get_accuracy_over_time_plot](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_accuracy_over_time_plot). You can also retrieve the detailed metadata using [DatetimeModel.get_accuracy_over_time_plots_metadata](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_accuracy_over_time_plots_metadata), and the preview plot using [DatetimeModel.get_accuracy_over_time_plot_preview](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_accuracy_over_time_plot_preview). ## Dates, datetimes, and durations When specifying a date or datetime for datetime partitioning, the client expects to receive and will return a `datetime`. Timezones may be specified, and will be assumed to be UTC if left unspecified. All dates returned from DataRobot are in UTC with a timezone specified. Datetimes may include a time, or specify only a date; however, they may have a non-zero time component only if the partition column included a time component in its date format. If the partition column included only dates like “24/03/2015”, then the time component of any datetimes, if present, must be zero. When date ranges are specified with a start and an end date, the end date is exclusive, so only dates earlier than the end date are included, but the start date is inclusive, so dates equal to or later than the start date are included. If the start and end date are the same, then no dates are included in the range. Durations are specified using a subset of ISO8601. Durations will be of the form `PnYnMnDTnHnMnS` where each “n” may be replaced with an integer value. Within the duration string, > nY represents the number of yearsthe nM following the “P” represents the number of monthsnD represents the number of daysnH represents the number of hoursthe nM following the “T” represents the number of minutesnS represents the number of seconds and “P” is used to indicate that the string represents a period and “T” indicates the beginning of the time component of the string. Any section with a value of 0 may be excluded. As with datetimes, if the partition column did not include a time component in its date format, the time component of any duration must be either unspecified or consist only of zeros. Example Durations: > “P3Y6M” (three years, six months)“P1Y0M0DT0H0M0S” (one year)“P1Y5DT10H” (one year, 5 days, 10 hours) [datarobot.helpers.partitioning_methods.construct_duration_string](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#dur-string-helper) is a helper method that can be used to construct appropriate duration strings. --- # Specialized workflows URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/index.html > Alternative workflows for a variety of specialized data types. The following sections describe alternative workflows for a variety of specialized data types. | Resource | Description | | --- | --- | | Time series projects | Learn how to set up and work with time series projects in DataRobot. | | Work with binary data | Learn how to work with binary data files in DataRobot projects. | | Datetime partitioned projects | Learn how to set up and work with datetime partitioned projects in DataRobot. | | Segmented modeling projects | Learn how to create and work with segmented modeling projects in DataRobot. | | Monotonic Constraints | Learn how to use monotonic constraints in DataRobot models. | | Composable ML | Learn how to use custom tasks and composable ML in DataRobot. | | Unsupervised Projects (Anomaly Detection) | Learn how to create and work with unsupervised anomaly detection projects in DataRobot. | | Unsupervised Projects (Clustering) | Learn how to create and work with unsupervised clustering projects in DataRobot. | | Visual AI projects | Learn how to create and work with Visual AI projects using image data in DataRobot. | --- # Monotonic constraints URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/monotonic_constraints.html > Learn how to use monotonic constraints in DataRobot models. Training with monotonic constraints allows users to force models to learn monotonic relationships with respect to some features and the target. This helps users create accurate models that comply with regulations (e.g. insurance, banking). Currently, only certain blueprints (e.g. xgboost) support this feature, and it is only supported for regression and binary classification projects. Typically working with monotonic constraints follows the following two workflows: Workflow one - Running a project with default monotonic constraints - set the target and specify default constraint lists for the project - when running autopilot or manually training models without overriding constraint settings, all blueprints that support monotonic constraints will use the specified default constraint featurelists Workflow two - Running a model with specific monotonic constraints - create featurelists for monotonic constraints - train a blueprint that supports monotonic constraints while specifying monotonic constraint featurelists - the specified constraints will be used, regardless of the defaults on the blueprint ## Create feature lists When specifying monotonic constraints, users must pass a reference to a featurelist containing only the features to be constrained, one for features that should monotonically increase with the target and another for those that should monotonically decrease with the target. ``` import datarobot as dr project = dr.Project.get(project_id) features_mono_up = ['feature_0', 'feature_1'] # features that have monotonically increasing relationship with target features_mono_down = ['feature_2', 'feature_3'] # features that have monotonically decreasing relationship with target flist_mono_up = project.create_featurelist(name='mono_up', features=features_mono_up) flist_mono_down = project.create_featurelist(name='mono_down', features=features_mono_down) ``` ## Specify default monotonic constraints for a project Users can specify default monotonic constraints for the project, to ensure that autopilot models use the desired settings, and optionally to ensure that only blueprints supporting monotonic constraints appear in the project. Regardless of the defaults specified via advanced options selection, the user can override them when manually training a particular model. ``` import datarobot as dr from datarobot.enums import AUTOPILOT_MODE project = dr.Project.get(project_id) # As of v3.0, ``Project.set_options`` may be used as an alternative to passing `advanced_options`` into ``Project.analyze_and_model``. project.set_options( monotonic_increasing_featurelist_id=flist_mono_up.id, monotonic_decreasing_featurelist_id=flist_mono_down.id, only_include_monotonic_blueprints=True ) project.analyze_and_model(target='target', mode=AUTOPILOT_MODE.FULL_AUTO) ``` If `Project.set_options` is not used, alternatively, an advanced options instance may be passed directly to `project.analyze_and_model`: ``` project.analyze_and_model( target='target', mode=AUTOPILOT_MODE.FULL_AUTO, advanced_options=AdvancedOptions(monotonic_increasing_featurelist_id=flist_mono_up.id, monotonic_decreasing_featurelist_id=flist_mono_down.id, only_include_monotonic_blueprints=True) ) ``` ## Retrieve models and blueprints using monotonic constraints When retrieving models, users can inspect to see which supports monotonic constraints, and which actually enforces them. Some models will not support monotonic constraints at all, and some may support constraints but not have any constrained features specified. ``` import datarobot as dr project = dr.Project.get(project_id) models = project.get_models() # retrieve models that support monotonic constraints models_support_mono = [model for model in models if model.supports_monotonic_constraints] # retrieve models that support and enforce monotonic constraints models_enforce_mono = [model for model in models if (model.monotonic_increasing_featurelist_id or model.monotonic_decreasing_featurelist_id)] ``` When retrieving blueprints, users can check if they support monotonic constraints and see what default constraint lists are associated with them. The monotonic featurelist ids associated with a blueprint will be used every time it is trained, unless the user specifically overrides them at model submission time. ``` import datarobot as dr project = dr.Project.get(project_id) blueprints = project.get_blueprints() # retrieve blueprints that support monotonic constraints blueprints_support_mono = [blueprint for blueprint in blueprints if blueprint.supports_monotonic_constraints] # retrieve blueprints that support and enforce monotonic constraints blueprints_enforce_mono = [blueprint for blueprint in blueprints if (blueprint.monotonic_increasing_featurelist_id or blueprint.monotonic_decreasing_featurelist_id)] ``` ## Train a model with specific monotonic constraints Even after specifying default settings for the project, users can override them to train a new model with different constraints, if desired. ``` import datarobot as dr features_mono_up = ['feature_2', 'feature_3'] # features that have monotonically increasing relationship with target features_mono_down = ['feature_0', 'feature_1'] # features that have monotonically decreasing relationship with target project = dr.Project.get(project_id) flist_mono_up = project.create_featurelist(name='mono_up', features=features_mono_up) flist_mono_down = project.create_featurelist(name='mono_down', features=features_mono_down) model_job_id = project.train( blueprint, sample_pct=55, featurelist_id=featurelist.id, monotonic_increasing_featurelist_id=flist_mono_up.id, monotonic_decreasing_featurelist_id=flist_mono_down.id ) ``` --- # Segmented modeling URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/segmented_modeling.html > Learn how to create and work with segmented modeling projects in DataRobot. Many [time series](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/time_series.html#time-series) multiseries projects introduce complex forecasting use cases that require using different models for subsets of series (i.e., sales of groceries and clothing can be very different). Within the segmented modeling framework, DataRobot runs multiple time series projects (one per segment / group of series), selects the best models for each segment, and then combines those models to make predictions. ## Segment A segment is a group of series in a multiseries project. For example, given `store` and `country` columns in dataset, you can use the former as the series identifier and the latter as the segment identifier. For the best results, group series with similar patterns into segments (instead of random selection). ## Segmentation task A segmentation task is an entity that defines how input dataset is partitioned. Currently only user-defined segmentation is supported. That is, the dataset must have a separate column that is used to identify segment (and the user must select it). All records within a series must have the same segment identifier. ## Combined model A combined model in a segmented modeling project can be thought of as a meta-model made of references to the best model within each segment. While being quite different from a standard DataRobot model in its creation, its use is very much the same after the model is complete (for example, deploying or making predictions). The following examples illustrate how to set up, run, and manage a segmented modeling project using the Python public API client. For details please refer to [Segmented Modeling API Reference](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#segmented-modeling-api). ### Start a segmentation project with a user-defined segment ID `Time series` modeling must be enabled for your account to run segmented modeling projects. Use the standard method to create a DataRobot project: ``` from datarobot import DatetimePartitioningSpecification from datarobot import enums from datarobot import Project from datarobot import SegmentationTask project_name = "Segmentation Demo with Segmentation ID" project_dataset = "multiseries_segmentation.csv" project = Project.create(project_dataset, project_name=project_name) datetime_partition_column = "timestamp" multiseries_id_column = "series_id" user_defined_segment_id_column = "物类segment_id" target = "target" ``` Create a simple datetime specification for a time series project: ``` spec = DatetimePartitioningSpecification( use_time_series=True, datetime_partition_column=datetime_partition_column, multiseries_id_columns=[multiseries_id_column], ) ``` Create a segmentation task for the project: ``` segmentation_task_results = SegmentationTask.create( project_id=project.id, target=target, use_time_series=True, datetime_partition_column=datetime_partition_column, multiseries_id_columns=[multiseries_id_column], user_defined_segment_id_columns=[user_defined_segment_id_column], ) segmentation_task = segmentation_task_results["completedJobs"][0] ``` Start a segmented project by passing the `segmentation_task_id` argument: ``` project.analyze_and_model( target=target, partitioning_method=spec, mode=enums.AUTOPILOT_MODE.QUICK, worker_count=-1, segmentation_task_id=segmentation_task.id, ) ``` ### Work with combined models Retrieve Combined Models: ``` from datarobot import Project, CombinedModel project_id = "60ff165dde5f3ceacda0f2d6" # Get an existing segmentation project project = Project.get(segmented_project_id) # Retrieve list of all combined models in the project combined_models = project.get_combined_models() # Or just an active (current) combined model current_combined_model = project.get_active_combined_model() ``` Get information about segments in the Combined Model: ``` segments_info = current_combined_model.get_segments_info() # Alternatively this information can be retrieved as a Pandas DataFrame segments_df = current_combined_model.get_segments_as_dataframe() # Or even in CSV format current_combined_model.get_segments_as_csv("combined_model_segments.csv") ``` Ensure Autopilot has completed for all segments: ``` segments_info = current_combined_model.get_segments_info() assert all(segment.autopilot_done for segment in segments_info) ``` Optionally, view a list of all models associated with individual segments: ``` segments_and_child_models = project.get_segments_models(current_combined_model.id) ``` Set a new champion for a segment in the Combined Model, specifying the `project_id` of the segmented project and the `model_id` from that project: ``` segment_project_id = "60ff165dde5f3ceacdaabcde" new_champion_id = "60ff165dde5f3ceacdaa12f7" CombinedModel.set_segment_champion(project_id=segment_project_id, model_id=new_champion_id) ``` If active Combined Model has already been deployed - changing champions is not allowed. In this case, create a copy of Combined Model, make it active, and set champion for it (deployed model remains unchanged): ``` new_combined_model = CombinedModel.set_segment_champion(project_id=segment_project_id, model_id=new_champion_id, clone=True) ``` Run predictions on the Combined Model: ``` prediction_dataset = "multiseries_predictions.csv" # Upload dataset dataset = project.upload_dataset( sourcedata=prediction_dataset, ) # Request predictions predictions_job = current_combined_model.request_predictions( dataset_id=dataset.id, ) predictions = predictions_job.get_result_when_complete() ``` --- # Time series projects URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/time_series.html > Learn how to set up and work with time series projects in DataRobot. Time series projects, like OTV projects, use [datetime partitioning](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/datetime_partition.html#set-up-datetime), and all the workflow changes that apply to other datetime partitioned projects also apply to them. Unlike other projects, time series projects produce different types of models which forecast multiple future predictions instead of an individual prediction for each row. DataRobot uses a general time series framework to configure how time series features are created and what future values the models will output. This framework consists of a Forecast Point (defining a time a prediction is being made), a Feature Derivation Window (a rolling window used to create features), and a Forecast Window (a rolling window of future values to predict). These components are described in more detail below. Time series projects will automatically transform the dataset provided in order to apply this framework. During the transformation, DataRobot uses the Feature Derivation Window to derive time series features (such as lags and rolling statistics), and uses the Forecast Window to provide examples of forecasting different distances in the future (such as time shifts). After project creation, a new dataset and a new feature list are generated and used to train the models. This process is reapplied automatically at prediction time as well in order to generate future predictions based on the original data features. The `time_unit` and `time_step` used to define the Feature Derivation and Forecast Windows are taken from the datetime partition column, and can be retrieved for a given column in the input data by looking at the corresponding attributes on the [datarobot.models.Feature](https://docs.datarobot.com/en/docs/api/reference/sdk/features.html#datarobot.models.Feature) object. If `windows_basis_unit` is set to `ROW`, then Feature Derivation and Forecast Windows will be defined using number of the rows. ## Set up time series projects To set up a time series project, follow the standard [datetime partitioning](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/datetime_partition.html#set-up-datetime) workflow and use the six new time series specific parameters on the [datarobot.DatetimePartitioningSpecification](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.DatetimePartitioningSpecification) Parameter | Description use_time_series | bool, set this to True to enable time series for the project. default_to_known_in_advance | bool, set this to True to default to treating all features as known in advance, or a priori, features. Otherwise, they will not be handled as known in advance features. Individual features can be set to a value different than the default by using the featureSettings parameter. See [the prediction documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/time_series.html#time-series-predict) for more information. default_to_do_not_derive | bool, set this to True to default to excluding all features from feature derivation. Otherwise, they will not be excluded and will be included in the feature derivation process. Individual features can be set to a value different than the default by using the featureSettings parameter. feature_derivation_window_start | int, specifies how many units of the `windows_basis_unit` from the forecast point into the past is the start of the feature derivation window. feature_derivation_window_end | int, specifies how many units of the `windows_basis_unit` from the forecast point into the past is the end of the feature derivation window. forecast_window_start | int, specifies how many units of the `windows_basis_unit` from the forecast point into the future is the start of the forecast window. forecast_window_end | int, specifies how many units of the `windows_basis_unit` from the forecast point into the future is the end of the forecast window windows_basis_unit | string, set this to `ROW` to define feature derivation and forecast windows in terms of the rows, rather than time units. If omitted, will default to the detected time unit (one of the `datarobot.enums.TIME_UNITS`). feature_settings | list of FeatureSettings specifying per feature settings, can be left unspecified. ### Feature derivation window The Feature Derivation window represents the rolling window that is used to derive time series features and lags, relative to the Forecast Point. It is defined in terms of `feature_derivation_window_start` and `feature_derivation_window_end` which are integer values representing datetime offsets in terms of the `time_unit` (e.g. hours or days). The Feature Derivation Window start and end must be less than or equal to zero, indicating they are positioned before the forecast point. Additionally, the window must be specified as an integer multiple of the `time_step` which defines the expected difference in time units between rows in the data. The window is closed, meaning the edges are considered to be inside the window. ### Forecast window The Forecast Window represents the rolling window of future values to predict, relative to the Forecast Point. It is defined in terms of the `forecast_window_start` and `forecast_window_end`, which are positive integer values indicating datetime offsets in terms of the `time_unit` (e.g. hours or days). The Forecast Window start and end must be positive integers, indicating they are positioned after the forecast point. Additionally, the window must be specified as an integer multiple of the `time_step` which defines the expected difference in time units between rows in the data. The window is closed, meaning the edges are considered to be inside the window. ### Multiseries projects Certain time series problems represent multiple separate series of data, e.g. “I have five different stores that all have different customer bases. I want to predict how many units of a particular item will sell, and account for the different behavior of each store”. When setting up the project, a column specifying series ids must be identified, so that each row from the same series has the same value in the multiseries id column. Using a multiseries id column changes which partition columns are eligible for time series, as each series is required to be unique and regular, instead of the entire partition column being required to have those properties. In order to use a multiseries id column for partitioning, a detection job must first be run to analyze the relationship between the partition and multiseries id columns. If needed, it will be automatically triggered by calling [datarobot.models.Feature.get_multiseries_properties()](https://docs.datarobot.com/en/docs/api/reference/sdk/features.html#datarobot.models.Feature.get_multiseries_properties) on the desired partition column. The previously computed multiseries properties for a particular partition column can then be accessed via that method. The computation will also be automatically triggered when calling [datarobot.DatetimePartitioning.generate()](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.DatetimePartitioning.generate) or [datarobot.models.Project.analyze_and_model()](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.analyze_and_model) with a multiseries id column specified. Note that currently only one multiseries id column is supported, but all interfaces accept lists of id columns to ensure multiple id columns will be able to be supported in the future. In order to create a multiseries project: > Set up a datetime partitioning specification with the desired partition column and multiseries id columns.(Optionally) Usedatarobot.models.Feature.get_multiseries_properties()to confirm the inferred time step and time unit of the partition column when used with the specified multiseries id column.(Optionally) Specify the multiseries id column in order to preview the full datetime partitioning settings usingdatarobot.DatetimePartitioning.generate().Specify the multiseries id column when sending the target and partitioning settings viadatarobot.models.Project.analyze_and_model().project=dr.Project.create('path/to/multiseries.csv',project_name='my multiseries project')partitioning_spec=dr.DatetimePartitioningSpecification('timestamp',use_time_series=True,multiseries_id_columns=['multiseries_id'])# manually confirm time step and time unit are as expecteddatetime_feature=dr.Feature.get(project.id,'timestamp')multiseries_props=datetime_feature.get_multiseries_properties(['multiseries_id'])print(multiseries_props)# manually check out the partitioning settings like feature derivation window and backtests# to make sure they make sense before moving onfull_part=dr.DatetimePartitioning.generate(project.id,partitioning_spec)print(full_part.feature_derivation_window_start,full_part.feature_derivation_window_end)print(full_part.to_dataframe())# As of v3.0, can use ``Project.set_datetime_partitioning`` instead of passing the spec into ``Project.analyze_and_model`` via ``partitioning_method``.# The spec options can be passed individually:project.set_datetime_partitioning(use_time_series=True,datetime_partition_column='date',multiseries_id_columns=['series_id'])# Or the whole spec object can be passed:project.set_datetime_partitioning(datetime_partitioning_spec=datetime_spec)# finalize the project and start the autopilotproject.analyze_and_model('target',partitioning_method=partitioning_spec) You can also access optimized partitioning in the API where the target over time is inspected to ensure that the default backtests cover regions of interest and adjust backtests avoid common problems with missing target values or partitions with single values (e.g. zero-inflated datasets). In this case you need to pass the target column when generating the partitioning specification (either by calling `DatetimePartitioning.generate` or `Project.set_datetime_partitioning`) and then pass the full partitioning specification when starting autopilot (if `Project.set_datetime_partitioning` is not used). ``` project = dr.Project.create('path/to/multiseries.csv', project_name='my multiseries project') partitioning_spec = dr.DatetimePartitioningSpecification( 'timestamp', use_time_series=True, multiseries_id_columns=['multiseries_id'] ) # Pass the target column to generate optimized partitions full_part = dr.DatetimePartitioning.generate(project.id, partitioning_spec, 'target') # Or, as of v3.0, call ``Project.set_datetime_partitioning`` after specifying the project target # to generate optimized partitions. project.target = 'target' project.set_datetime_partitioning(datetime_partition_spec=partitioning_spec) # finalize the project and start the autopilot, passing in the full partitioning spec # (if ``Project.set_datetime_partitioning`` was used there is no need to pass ``partitioning_method``) project.analyze_and_model('target', partitioning_method=full_part.to_specification()) ``` ### Feature settings [datarobot.FeatureSettings](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.FeatureSettings) constructor receives `feature_name` and settings. For now settings `known_in_advance` and `do_not_derive` are supported. ``` # I have 10 features, 8 of them are known in advance and two are not # Also, I do not want to derive new features from previous_day_sales not_known_in_advance_features = ['previous_day_sales', 'amount_in_stock'] do_not_derive_features = ['previous_day_sales'] feature_settings = [dr.FeatureSettings(feat_name, known_in_advance=False) for feat_name in not_known_in_advance_features] feature_settings += [dr.FeatureSettings(feat_name, do_not_derive=True) for feat_name in do_not_derive_features] spec = dr.DatetimePartitioningSpecification( # ... default_to_known_in_advance=True, feature_settings=feature_settings ) ``` ## Model data and time series features In time series projects, a new set of modeling features is created after setting the partitioning options. If a featurelist is specified with the partitioning options, it will be used to select which features should be used to derived modeling features; if a featurelist is not specified, the default featurelist will be used. These features are automatically derived from those in the project’s dataset and are the features used for modeling - note that the Project methods `get_featurelists` and `get_modeling_featurelists` will return different data in time series projects. Modeling featurelists are the ones that can be used for modeling and will be accepted by the backend, while regular featurelists will continue to exist but cannot be used. Modeling features are only accessible once the target and partitioning options have been set. In projects that don’t use time series modeling, once the target has been set, modeling and regular features and featurelists will behave the same. ### Restore discarded features [datarobot.models.restore_discarded_features.DiscardedFeaturesInfo](https://docs.datarobot.com/en/docs/api/reference/sdk/features.html#datarobot.models.restore_discarded_features.DiscardedFeaturesInfo) can be used to get and restore features that have been removed by the time series feature generation and reduction functionality. ``` project = Project(project_id) discarded_feature_info = project.get_discarded_features() restored_features_info = project.restore_discarded_features(discarded_features_info.features) ``` ## Make predictions Prediction datasets are uploaded [as normal](https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/predict_job.html#predictions). However, when uploading a prediction dataset, a new parameter `forecast_point` can be specified. The forecast point of a prediction dataset identifies the point in time relative which predictions should be generated, and if one is not specified when uploading a dataset, the server will choose the most recent possible forecast point. The forecast window specified when setting the partitioning options for the project determines how far into the future from the forecast point predictions should be calculated. To simplify the predictions process, starting in version v2.20 a forecast point or prediction start and end dates can be specified when requesting predictions, instead of being specified at dataset upload. Upon uploading a dataset, DataRobot will calculate the range of dates available for use as a forecast point or for batch predictions. To that end, [Predictions](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Predictions) objects now also contain the following new fields: > forecast_point: The default point relative to which predictions will be generatedpredictions_start_date: The start date for bulk historical predictions.predictions_end_date: The end date for bulk historical predictions. Similar settings are provided as part of the [batch prediction API](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#batch-prediction-api) and the [real-time prediction API](https://docs.datarobot.com/en/docs/predictions/api/dr-predapi.html#making-predictions-with-time-series) to make predictions using deployed time series models. datarobot.models.BatchPredictionJob.score When setting up a time series project, input features could be identified as known-in-advance features. These features are not used to generate lags, and are expected to be known for the rows in the forecast window at predict time (e.g. “how much money will have been spent on marketing”, “is this a holiday”). Enough rows of historical data must be provided to cover the span of the effective Feature Derivation Window (which may be longer than the project’s Feature Derivation Window depending on the differencing settings chosen). The effective Feature Derivation Window of any model can be checked via the `effective_feature_derivation_window_start` and `effective_feature_derivation_window_end` attributes of a [DatetimeModel](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel). When uploading datasets to a time series project, the dataset might look something like the following, where “Time” is the datetime partition column, “Target” is the target column, and “Temp.” is an input feature. If the dataset was uploaded with a forecast point of “2017-01-08” and the effective feature derivation window start and end for the model are -5 and -3 and the forecast window start and end were set to 1 and 3, then rows 1 through 3 are historical data, row 6 is the forecast point, and rows 7 though 9 are forecast rows that will have predictions when predictions are computed. ``` Row, Time, Target, Temp. 1, 2017-01-03, 16443, 72 2, 2017-01-04, 3013, 72 3, 2017-01-05, 1643, 68 4, 2017-01-06, , 5, 2017-01-07, , 6, 2017-01-08, , 7, 2017-01-09, , 8, 2017-01-10, , 9, 2017-01-11, , ``` On the other hand, if the project instead used “Holiday” as an a priori input feature, the uploaded dataset might look like the following: ``` Row, Time, Target, Holiday 1, 2017-01-03, 16443, TRUE 2, 2017-01-04, 3013, FALSE 3, 2017-01-05, 1643, FALSE 4, 2017-01-06, , FALSE 5, 2017-01-07, , FALSE 6, 2017-01-08, , FALSE 7, 2017-01-09, , TRUE 8, 2017-01-10, , FALSE 9, 2017-01-11, , FALSE ``` ## Calendars You can upload a [calendar file](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.CalendarFile) containing a list of events relevant to your dataset. When provided, DataRobot automatically derives and creates time series features based on the calendar events (e.g., time until the next event, labeling the most recent event). The calendar file: - Should span the entire training data date range, as well as all future dates in which model will be forecasting. - Must be in csv or xlsx format with a header row. - Must have one date column which has values in the date-only format YYY-MM-DD (i.e., no hour, month, or second). - Can optionally include a second column that provides the event name or type. - Can optionally include a series ID column which specifies which series an event is applicable to. This column name must match the name of the column set as the series ID. Multiseries ID columns are used to add an ability to specify different sets of events for different series, e.g. holidays for different regions.Values of the series ID may be absent for specific events. This means that the event is valid for all series in project dataset (e.g. New Year’s Day is a holiday in all series in the example below).If a multiseries ID column is not provided, all listed events will be applicable to all series in the project dataset. - Cannot be updated in an active project. You must specify all future calendar events at project start. To update the calendar file, you will have to train a new project. An example of a valid calendar file: ``` Date, Name 2019-01-01, New Year's Day 2019-02-14, Valentine's Day 2019-04-01, April Fools 2019-05-05, Cinco de Mayo 2019-07-04, July 4th ``` An example of a valid multiseries calendar file: ``` Date, Name, Country 2019-01-01, New Year's Day, 2019-05-27, Memorial Day, USA 2019-07-04, July 4th, USA 2019-11-28, Thanksgiving, USA 2019-02-04, Constitution Day, Mexico 2019-03-18, Benito Juárez's birth, Mexico 2019-12-25, Christmas Day, ``` Once created, a calendar can be used with a time series project by specifying the `calendar_id` field in the [datarobot.DatetimePartitioningSpecification](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.DatetimePartitioningSpecification) object for the project: ``` import datarobot as dr # create the project project = dr.Project.create('input_data.csv') # create the calendar calendar = dr.CalendarFile.create('calendar_file.csv') # specify the calendar_id in the partitioning specification datetime_spec = dr.DatetimePartitioningSpecification( use_time_series=True, datetime_partition_column='date' calendar_id=calendar.id ) # As of v3.0, can use ``Project.set_datetime_partitioning`` instead of passing the spec into ``Project.analyze_and_model`` via ``partitioning_method``. # The spec options can be passed individually: project.set_datetime_partitioning(use_time_series=True, datetime_partition_column='date', calendar_id=calendar.id) # Or the whole spec object can be passed: project.set_datetime_partitioning(datetime_partitioning_spec=datetime_spec) # start the project, specifying the partitioning method (if ``Project.set_datetime_partitioning`` was used there is no need to pass ``partitioning_method``) project.analyze_and_model( target='project target', partitioning_method=datetime_spec ) ``` As of version v2.23 it is possible to ask DataRobot to generate a calendar file for you using [CalendarFile.create_calendar_from_country_code](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.CalendarFile.create_calendar_from_country_code). This method allows you to provide a country code specifying which country’s holidays to use in generating the calendar, along with a start and end date indicating the bounds of the calendar. Allowed country codes can be retrieved using [CalendarFile.get_allowed_country_codes](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.CalendarFile.get_allowed_country_codes). See the following code block for example usage: ``` import datarobot as dr from datetime import datetime # create the project project = dr.Project.create('input_data.csv') # retrieve the allowed country codes and use the first one country_code = dr.CalendarFile.get_allowed_country_codes()[0]['code'] calendar = dr.CalendarFile.create_calendar_from_country_code( country_code, datetime(2018, 1, 1), datetime(2018, 7, 4) ) # specify the calendar_id in the partitioning specification datetime_spec = dr.DatetimePartitioningSpecification( use_time_series=True, datetime_partition_column='date' calendar_id=calendar.id ) # As of v3.0, can use ``Project.set_datetime_partitioning`` instead of passing the spec into ``Project.analyze_and_model`` via ``partitioning_method``. # The spec options can be passed individually: project.set_datetime_partitioning(use_time_series=True, datetime_partition_column='date', calendar_id=calendar.id) # Or the whole spec object can be passed: project.set_datetime_partitioning(datetime_partitioning_spec=datetime_spec) # Start the project, specifying the partitioning method (if ``Project.set_datetime_partitioning`` was used there is no need to pass ``partitioning_method``) project.analyze_and_model( target='project target', partitioning_method=datetime_spec ) ``` ## Datetime trend plots As a version v2.25, it is possible to retrieve Datetime Trend Plots for time series models to estimate the accuracy of the model. This includes Accuracy over Time and Forecast vs Actual for supervised projects, and Anomaly over Time for unsupervised projects. You can retrieve respective plots using following methods: - DatetimeModel.get_accuracy_over_time_plot - DatetimeModel.get_forecast_vs_actual_plot - DatetimeModel.get_anomaly_over_time_plot By default, the plots would be automatically computed when accessed via retrieval methods. You can compute Datetime Trend Plots separately using a common method [DatetimeModel.compute_datetime_trend_plots](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.compute_datetime_trend_plots). In addition, you can retrieve the respective detailed metadata for each plot type: - DatetimeModel.get_accuracy_over_time_plots_metadata - DatetimeModel.get_forecast_vs_actual_plots_metadata - DatetimeModel.get_anomaly_over_time_plots_metadata And the preview plots: - DatetimeModel.get_accuracy_over_time_plot_preview - DatetimeModel.get_forecast_vs_actual_plot_preview - DatetimeModel.get_anomaly_over_time_plot_preview ## Prediction intervals For each model, prediction intervals estimate the range of values DataRobot expects actual values of the target to fall within. They are similar to a confidence interval of a prediction, but are based on the residual errors measured during the backtesting for the selected model. Note that because calculation depends on the backtesting values, prediction intervals are not available for predictions on models that have not had all backtests completed. To that end, note that creating a prediction with prediction intervals through the API will automatically complete all backtests if they were not already completed. For start-end retrained models, the parent model will be used for backtesting. Additionally, prediction intervals are not available when the number of points per forecast distance is less than 10, due to insufficient data. In a prediction request, users can specify a prediction interval’s size, which specifies the desired probability of actual values falling within the interval range. Larger values are less precise, but more conservative. For example, specifying a size of 80 will result in a lower bound of 10% and an upper bound of 90%. More generally, for a specific `prediction_intervals_size`, the upper and lower bounds will be calculated as follows: - prediction_interval_upper_bound = 50% + ( prediction_intervals_size / 2) - prediction_interval_lower_bound = 50% - ( prediction_intervals_size / 2) Prediction intervals can be calculated for a [DatetimeModel](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel) using the [DatetimeModel.calculate_prediction_intervals](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.calculate_prediction_intervals) method. Users can also retrieve which intervals have already been calculated for the model using the [DatetimeModel.get_calculated_prediction_intervals](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_calculated_prediction_intervals) method. To view prediction intervals data for a prediction, the prediction needs to have been created using the [DatetimeModel.request_predictions](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.request_predictions) method and specifying `include_prediction_intervals = True`. The size for the prediction interval can be specified with the `prediction_intervals_size` parameter for the same function, and will default to 80 if left unspecified. Specifying either of these fields will result in prediction interval bounds being included in the retrieved prediction data for that request (see the [Predictions](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Predictions) class for retrieval methods). Note that if the specified interval size has not already been calculated, this request will automatically calculate the specified size. Prediction intervals are also supported for time series model deployments, and should be specified in deployment settings if desired. Use [Deployment.get_prediction_intervals_settings](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_prediction_intervals_settings) to retrieve current prediction intervals settings for a deployment, and [Deployment.update_prediction_intervals_settings](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_prediction_intervals_settings) to update prediction intervals settings for a deployment. ## Partial history predictions As of version v2.24 it is possible to ask DataRobot to allow to make predictions with incomplete historical data multiseries regression projects. To make predictions in regular project user has to provide enough data for the feature derivation. By setting the datetime partitioning attribute `allow_partial_history_time_series_predictions` to true ( [datarobot.DatetimePartitioningSpecification](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.DatetimePartitioningSpecification) object), the project would be created that allow to make such predictions. The number of models are significantly smaller compared to regular multiseries model, but they are designed to make predictions on unseen series with reasonable accuracy. ## External baseline predictions As of version v2.26 it is possible to ask DataRobot to scale accuracy metric by external predictions. Users can upload data into a Dataset (see [Dataset documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/dataset.html#datasets)) and compare the external time series predictions with DataRobot models’ accuracy performance. To use the external predictions dataset in the autopilot, the dataset must be validated first (see [Project.validate_external_time_series_baseline](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.validate_external_time_series_baseline)). Once the dataset is validated, it can be used with a time series project by specifying `external_time_series_baseline_dataset_id` field in [AdvancedOptions](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.helpers.AdvancedOptions) and passes the advanced options to the project. See the following code block for example usage: ``` import datarobot as dr from datarobot.helpers import AdvancedOptions from datarobot.models import Dataset # create the project project = dr.Project.create('input_data.csv') # prepare datetime partitioning for external baseline validation datetime_spec = dr.DatetimePartitioningSpecification( use_time_series=True, datetime_partition_column='date', multiseries_id_columns=['series_id'], ) datetime_partitioning = dr.DatetimePartitioning.generate( project_id=project.id, spec=datetime_spec, target='target', ) # create external baseline prediction dataset from local file external_baseline_dataset = Dataset.create_from_file(file_path='external_predictions.csv') # validate the external baseline prediction dataset validation_info = project.validate_external_time_series_baseline( catalog_version_id=external_baseline_dataset.version_id, target='target', datetime_partitioning=datetime_partitioning, ) print( 'External baseline predictions passes validation check:', validation_info.is_external_baseline_dataset_valid ) # As of v3.0, can use ``Project.set_datetime_partitioning`` instead of passing the spec into ``Project.analyze_and_model`` via ``partitioning_method``. # The spec options can be passed individually: project.set_datetime_partitioning(use_time_series=True, datetime_partition_column='date', multiseries_id_columns=['series_id']) # Or the whole spec object can be passed: project.set_datetime_partitioning(datetime_partitioning_spec=datetime_spec) # As of v3.0, add the validated dataset version id into advanced options project.set_options( external_time_series_baseline_dataset_id=external_baseline_dataset.version_id ) # start the project, specifying the partitioning method (if ``Project.set_datetime_partitioning`` and ``Project.set_options`` were not used) project.analyze_and_model( target='target', partitioning_method=datetime_spec advanced_options=AdvancedOptions(external_time_series_baseline_dataset_id) ) ``` ## Time series data prep As of version v2.27 it is possible to prepare a dataset for time series modeling in the AI catalog using the API client. Users can upload unprepped modeling data into a Dataset (see [Dataset documentation](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/dataset.html#datasets)) and the prep the dataset for time series modeling by aggregating data to a regular time step and filling gaps via a generated Spark SQL query in the AI catalog. Once the dataset is uploaded, the time series data prep query generator can be created using [DataEngineQueryGenerator.create](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.DataEngineQueryGenerator.create). As of version v3.1 convenience methods have been added to streamline the process of applying time series data prep for predictions. See the following code block for example usage: ``` import datarobot as dr from datarobot.models.data_engine_query_generator import ( QueryGeneratorDataset, QueryGeneratorSettings, ) from datetime import datetime # upload the dataset to the AI Catalog dataset = dr.Dataset.create_from_file('input_data.csv') # create a time series data prep query generator query_generator_dataset = QueryGeneratorDataset( alias='input_data_csv', dataset_id=dataset.id, dataset_version_id=dataset.version_id, ) query_generator_settings = QueryGeneratorSettings( datetime_partition_column="date", time_unit="DAY", time_step=1, default_numeric_aggregation_method="sum", default_categorical_aggregation_method="mostFrequent", target="y", multiseries_id_columns=["id"], default_text_aggregation_method="concat", start_from_series_min_datetime=True, end_to_series_max_datetime=True, ) query_generator = dr.DataEngineQueryGenerator.create( generator_type='TimeSeries', datasets = [query_generator_dataset], generator_settings=query_generator_settings, ) # prep the training dataset training_dataset = query_generator.create_dataset() # create a project project = dr.Project.create_from_dataset(training_dataset.id, project_name='prepped_dataset') # set up datetime partitioning, target, and train model(s) partitioning_spec = dr.DatetimePartitioningSpecification( datetime_partition_column='date', use_time_series=True ) project.analyze_and_model(target='y', mode='manual', partitioning_method=partitioning_spec) blueprints = project.get_blueprints() model_job = project.train_datetime(blueprints[0].id) model = model_job.get_result_when_complete() # query generator can be retrieved from the project if necessary # query_generator = dr.DataEngineQueryGenerator.get(project.query_generator_id) # prep and upload a prediction dataset to the project prediction_dataset = query_generator.prepare_prediction_dataset( 'prediction_data.csv', project.id ) # make predictions within the project # Either forecast point or predictions start/end dates must be specified model.request_predictions(prediction_dataset.id, forecast_point=datetime(2023, 1, 1)) # query generator can be retrieved from a deployed model via project if necessary # deployment = dr.Deployment.get(deployment_id) # project = dr.Project.get(deployment.model['project_id']) # query_generator = dr.DataEngineQueryGenerator.get(project.query_generator_id) # Deploy the model prediction_servers = dr.PredictionServer.list() deployment = dr.Deployment.create_from_learning_model( model.id, 'prepped_deployment', default_prediction_server_id=prediction_servers[0].id ) # Make batch predictions from batch prediction job, supports localFile or dataset for intake # and all types for output timeseries_settings = {'type': 'forecast', 'forecast_point': datetime(2023, 1, 1)} intake_settings = {'type': 'localFile', 'file': 'prediction_data.csv'} output_settings = {'type': 'localFile', 'path': 'predictions_out.csv'} batch_predictions_job = dr.BatchPredictionJob.apply_time_series_data_prep_and_score( deployment, intake_settings, timeseries_settings, output_settings=output_settings ) ``` --- # Unsupervised projects (anomaly detection) URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/unsupervised_anomaly.html > Learn how to create and work with unsupervised anomaly detection projects in DataRobot. When the data is not labelled and the problem can be interpreted either as anomaly detection or time series anomaly detection, projects in unsupervised mode become useful. ## Create unsupervised projects In order to create an unsupervised project set `unsupervised_mode` to `True` when setting the target. ``` import datarobot as dr project = Project.create('dataset.csv', project_name='unsupervised') project.analyze_and_model(unsupervised_mode=True) ``` ## Create time series unsupervised projects To create a time series unsupervised project pass `unsupervised_mode=True` to datetime partitioning creation and to project aim. The forecast window will be automatically set to nowcasting, i.e. forecast distance zero (FW = 0, 0). ``` import datarobot as dr project = Project.create('dataset.csv', project_name='unsupervised') spec = DatetimePartitioningSpecification('date', use_time_series=True, unsupervised_mode=True, feature_derivation_window_start=-4, feature_derivation_window_end=0) # this step is optional - preview the default partitioning which will be applied partitioning_preview = DatetimePartitioning.generate(project.id, spec) full_spec = partitioning_preview.to_specification() # As of v3.0, can use ``Project.set_datetime_partitioning`` and ``Project.list_datetime_partitioning_spec`` instead project.set_datetime_partitioning(datetime_partition_spec=spec) project.list_datetime_partitioning_spec() # If ``Project.set_datetime_partitioning`` was used there is no need to pass ``partitioning_method`` in ``Project.analyze_and_model`` project.analyze_and_model(unsupervised_mode=True, partitioning_method=full_spec) ``` ## Unsupervised project metrics In unsupervised projects, metrics are not used for the model optimization. Instead, they are used for the purpose of model ranking. There are two available unsupervised metrics – Synthetic AUC and synthetic LogLoss – both of which are calculated on artificially-labelled validation samples. ## Estimate accuracy of unsupervised anomaly detection datetime partitioned models For datetime partitioned unsupervised model you can retrieve the Anomaly over Time plot. To do so use [DatetimeModel.get_anomaly_over_time_plot](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_anomaly_over_time_plot). You can also retrieve the detailed metadata using [DatetimeModel.get_anomaly_over_time_plots_metadata](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_anomaly_over_time_plots_metadata), and the preview plot using [DatetimeModel.get_anomaly_over_time_plot_preview](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_anomaly_over_time_plot_preview). ## Explain unsupervised time series anomaly detection model predictions Within a timeseries unsupervised project for models supporting calculation of Shapley values, Anomaly Assessment insight can be computed to explain anomalies. Example 1: computation, retrieval and deletion of the anomaly assessment insight. ``` import datarobot as dr # Initialize Anomaly Assessment for the backtest 0, training subset and series "series1" model = dr.DatetimeModel.get(project_id, model_id) anomaly_assessment_record = model.initialize_anomaly_assessment(0, "training", "series1") # Get available Anomaly Assessment for the project and model all_records = model.get_anomaly_assessment_records() # Get most recent anomaly assessment explanations all_records[0].get_latest_explanations() # Get anomaly assessment explanations in the range all_records[0].get_explanations(start_date="2020-01-01", points_count=500) # Get anomaly assessment predictions preview all_records[0].get_predictions_preview() # Delete record all_records[0].delete() ``` Example 2: Find explanations for the anomalous regions (regions with maximum anomaly score >=0.6) for the multiseries project. Leave only explanations for the rows with anomaly score >= 0.5. ``` def collect_explanations(model, backtest, source, series_ids): for series in series_ids: try: model.initialize_anomaly_assessment(backtest, source, series) except ClientError: # when insight was already computed pass records_for_series = model.get_anomaly_assessment_records(source=source, backtest=backtest, with_data_only=True, limit=0) result = {} for record in records_for_series: preview = record.get_predictions_preview() anomalous_regions = preview.find_anomalous_regions(max_prediction_threshold=0.6) if anomalous_regions: result[record.series_id] = record.get_explanations_data_in_regions(anomalous_regions, prediction_threshold=0.5) return result import datarobot as dr model = dr.DatetimeModel.get(project_id, model_id) collect_explanations(model, 0, "validation", series_ids) ``` ## Assess unsupervised anomaly detection models on external test sets In unsupervised projects, if there is some labelled data, it may be used to assess anomaly detection models by checking computed classification metrics such as AUC and LogLoss, etc. and insights such as ROC and Lift. Such data is uploaded as a prediction dataset with a specified actual value column name, and, if it is a time series project, a prediction date range. The actual value column can contain only zeros and ones or True/False, and it should not have been seen during training time. ## Request external scores and insights (time series) There are two ways to specify an actual value column and compute scores and insights: 1. Upload a prediction dataset, specifyingpredictions_start_date,predictions_end_date, andactual_value_column, and request predictions on that dataset using a specific model. importdatarobotasdr# Upload datasetproject=dr.Project(project_id)dataset=project.upload_dataset('./data_to_predict.csv',predictions_start_date=datetime(2000,1,1),predictions_end_date=datetime(2015,1,1),actual_value_column='actuals')# run prediction job which also will calculate requested scores and insights.predict_job=model.request_predictions(dataset.id)# prediction output will have column with actualsresult=pred_job.get_result_when_complete() 2. Upload a prediction dataset without specifying any options, and request predictions for a specific model withpredictions_start_date,predictions_end_date, andactual_value_columnspecified. Note, these settings cannot be changed for the dataset after making predictions. ``` import datarobot as dr # Upload dataset project = dr.Project(project_id) dataset = project.upload_dataset('./data_to_predict.csv') # Check which columns are candidates for actual value columns dataset.detected_actual_value_columns [{'missing_count': 25, 'name': 'label_column'}] # run prediction job which also will calculate requested scores and insights. predict_job = model.request_predictions( dataset.id, predictions_start_date=datetime(2000, 1, 1), predictions_end_date=datetime(2015, 1, 1), actual_value_column='label_column' ) result = pred_job.get_result_when_complete() ``` ## Request external scores and insights for AutoML models To compute scores and insights on an external dataset for unsupervised AutoML models (Non Time series) Upload a prediction dataset that contains label column(s), request compute external test on one of `PredictionDataset.detected_actual_value_columns`. ``` import datarobot as dr # Upload dataset project = dr.Project(project_id) dataset = project.upload_dataset('./test_set.csv') dataset.detected_actual_value_columns >>>['label_column_1', 'label_column_2'] # request external test to compute metric scores and insights on dataset external_test_job = model.request_external_test(dataset.id, actual_value_column='label_column_1') # once job is complete, scores and insights are ready for retrieving external_test_job.wait_for_completion() ``` ## Retrieve external scores and insights Upon completion of prediction, external scores and insights can be retrieved to assess model performance. For unsupervised projects Lift Chart and ROC Curve are computed. If the dataset is too small insights will not be computed. If the actual value column contained only one class, the ROC Curve will not be computed. Information about the dataset can be retrieved using `PredictionDataset.get`. ``` import datarobot as dr # Check which columns are candidates for actual value columns scores_list = ExternalScores.list(project_id) scores = ExternalScores.get(project_id, dataset_id=dataset_id, model_id=model_id) lift_list = ExternalLiftChart.list(project_id, model_id) roc = ExternalRocCurve.get(project_id, model, dataset_id) # check dataset warnings, need to be called after predictions are computed. dataset = PredictionDataset.get(project_id, dataset_id) dataset.data_quality_warnings {'single_class_actual_value_column': True, 'insufficient_rows_for_evaluating_models': False, 'has_kia_missing_values_in_forecast_window': False} ``` --- # Unsupervised Projects (Clustering) URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/unsupervised_clustering.html > Learn how to create and work with unsupervised clustering projects in DataRobot. Use clustering when data is not labelled and the problem can be interpreted as grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar to each other than to those in other groups (clusters). It is a common task in data exploration when finding groups and similarities is needed. ## Create unsupervised projects To create an unsupervised project, set `unsupervised_mode` to `True` when setting the target. To specify clustering, set `unsupervised_type` to `CLUSTERING`. When setting the modeling mode is required, clustering supports either `AUTOPILOT_MODE.COMPREHENSIVE` for DataRobot-run Autopilot or `AUTOPILOT_MODE.MANUAL` for user control of which models/parameters to use. Example: ``` from datarobot import Project from datarobot.enums import UnsupervisedTypeEnum from datarobot.enums import AUTOPILOT_MODE project = Project.create("dataset.csv", project_name="unsupervised clustering") project.analyze_and_model( unsupervised_mode=True, mode=AUTOPILOT_MODE.COMPREHENSIVE, unsupervised_type=UnsupervisedTypeEnum.CLUSTERING, ) ``` You can optionally specify list of explicit cluster numbers. To do this, pass a list of integer values to optional `autopilot_cluster_list` parameter using the `analyze_and_model()` method. ``` project.analyze_and_model( unsupervised_mode=True, mode=AUTOPILOT_MODE.COMPREHENSIVE, unsupervised_type=UnsupervisedTypeEnum.CLUSTERING, autopilot_cluster_list=[7, 9, 11, 15, 19], ) ``` You can also do both in one step using the `Project.start()` method. This method by default will use `AUTOPILOT_MODE.COMPREHENSIVE` mode. ``` from datarobot import Project from datarobot.enums import UnsupervisedTypeEnum project = Project.start( "dataset.csv", unsupervised_mode=True, project_name="unsupervised clustering project", unsupervised_type=UnsupervisedTypeEnum.CLUSTERING, ) ``` ## Unsupervised clustering project metric Unsupervised clustering projects use the `Silhouette Score` metric for model ranking (instead of using it for model optimization). It measures the average similarity of objects within a cluster and their distance to the other objects in the other clusters. ## Retrieve information about clusters In a trained model, you can retrieve information about clusters in along with standard model information. To do this, when training completes, retrieve a model and view basic clustering information: > n_clusters: number of clusters for modelis_n_clusters_dynamically_determined: how clustering model picks number of clusters Here is a code snippet to retrieve information about the number of clusters for model: ``` from datarobot import ClusteringModel model = ClusteringModel.get(project_id, model_id) print("{} clusters found".format(model.n_clusters)) ``` You can retrieve more details about clusters and their data using cluster insights. ## Work with cluster insights You can compute insights to gain deep insights into clusters and their characteristics. This process will perform calculations and return detailed information about each feature and its importance, as well as a detailed per-cluster breakdown. To compute and retrieve cluster insights, use the `ClusteringModel` and its `compute_insights` method. The method starts the cluster insights compute job, waits for its completion for the number of seconds specified in the optional parameter `max_wait` (default: 600), and returns results when insights are ready. If clusters are already computed, access them using the `insights` property of the `ClusteringModel` method. ``` from datarobot import ClusteringModel model = ClusteringModel.get(project_id, model_id) insights = model.compute_insights() ``` This call, with the specified `wait_time`, will run and wait for specified time: ``` from datarobot import ClusteringModel model = ClusteringModel.get(project_id, model_id) insights = model.compute_insights(max_wait=60) ``` If computation fails to finish before `max_wait` expires, the method will raise an `AsyncTimeoutError`. You can retrieve cluster insights after jobs computation finishes. To retrieve cluster insights already computed: ``` from datarobot import ClusteringModel model = ClusteringModel.get(project_id, model_id) for insight in model.insights: print(insight) ``` To see the full depth of information available, add `print(insight.insights)`. ``` from datarobot import ClusteringModel model = ClusteringModel.get(project_id, model_id) for insight in model.insights: print(insight) print(nsight.insights) ``` ## Work with clusters By default, DataRobot names clusters “Cluster 1”, “Cluster 2”, … , “Cluster N” . You can retrieve these names and alter them according to preference. When retrieving clusters before computing insights, clusters will contain only names. After insight computation completes, each cluster will also hold information about the percentage of data that is represented by the Cluster. For example: ``` from datarobot import ClusteringModel model = ClusteringModel.get(project_id, model_id) # helper function def print_summary(name, percent): if not percent: percent = "?" print("'{}' holds {} % of data".format(name, percent)) for cluster in model.clusters: print_summary(cluster.name, cluster.percent) model.compute_insights() for cluster in model.clusters: print_summary(cluster.name, cluster.percent) ``` For a model with three clusters, the code snippet will output: ``` 'Cluster 1' holds ? % of data 'Cluster 2' holds ? % of data 'Cluster 3' holds ? % of data -- Cluster insights computation finished -- 'Cluster 1' holds 27.1704180064 % of data 'Cluster 2' holds 36.9131832797 % of data 'Cluster 3' holds 35.9163987138 % of data ``` Use the following methods of `ClusteringModel` class to alter cluster names: - `update_cluster_names` - changes multiple cluster names using mapping in dictionary - `update_cluster_name` - changes one cluster name After update, each method will return a list of clusters with changed names. For example: ``` from datarobot import ClusteringModel model = ClusteringModel.get(project_id, model_id) # update multiple cluster_name_mappings = [ ("Cluster 1", "AAA"), ("Cluster 2", "BBB"), ("Cluster 3", "CCC") ] clusters = model.update_cluster_names(cluster_name_mappings) # update single clusters = model.update_cluster_name("CCC", "DDD") ``` ## Clustering classes reference ### ClusteringModel ### class datarobot.models.model.ClusteringModel ClusteringModel extends [Model](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model) class. It provides properties and methods specific to clustering projects. #### compute_insights(max_wait=600) Compute and retrieve cluster insights for model. This method awaits completion of job computing cluster insights and returns results after it is finished. If computation takes longer than specified `max_wait` exception will be raised. - Parameters: - project_id ( str ) – Project to start creation in. - model_id ( str ) – Project’s model to start creation in. - max_wait ( int ) – Maximum number of seconds to wait before giving up - Return type: List of ClusterInsight - Raises: - ClientError – Server rejected creation due to client error. Most likely cause is bad project_id or model_id . - AsyncFailureError – If any of the responses from the server are unexpected - AsyncProcessUnsuccessfulError – If the cluster insights computation has failed or was cancelled. - AsyncTimeoutError – If the cluster insights computation did not resolve in time #### property insights : List[ClusterInsight] Return actual list of cluster insights if already computed. - Return type: List of ClusterInsight #### property clusters : List[Cluster] Return actual list of Clusters. - Return type: List of Cluster #### update_cluster_names(cluster_name_mappings) Change many cluster names at once based on list of name mappings. - Parameters: cluster_name_mappings ( List of tuples ) – Cluster names mapping consisting of current cluster name and old cluster name. Example: ``` cluster_name_mappings = [ ("current cluster name 1", "new cluster name 1"), ("current cluster name 2", "new cluster name 2")] ``` #### update_cluster_name(current_name, new_name) Change cluster name from current_name to new_name. - Parameters: - current_name ( str ) – Current cluster name. - new_name ( str ) – New cluster name. - Return type: List of Cluster - Raises: datarobot.errors.ClientError – Server rejected update of cluster names. ### Cluster ### class datarobot.models.model.Cluster Representation of a single cluster. - Variables: - name ( str ) – Current cluster name - percent ( float ) – Percent of data contained in the cluster. This value is reported after cluster insights are computed for the model. #### classmethod list(project_id, model_id) Retrieve a list of clusters in the model. - Parameters: - project_id ( str ) – ID of the project that the model is part of. - model_id ( str ) – ID of the model. - Return type: List of clusters #### classmethod update_multiple_names(project_id, model_id, cluster_name_mappings) Update many clusters at once based on list of name mappings. - Parameters: - project_id ( str ) – ID of the project that the model is part of. - model_id ( str ) – ID of the model. - cluster_name_mappings(Listoftuples) – Cluster name mappings, consisting of current and previous names for each cluster. Example: cluster_name_mappings=[("current cluster name 1","new cluster name 1"),("current cluster name 2","new cluster name 2")] * Return type: List of clusters * Raises: * datarobot.errors.ClientError – Server rejected update of cluster names. * ValueError – Invalid cluster name mapping provided. #### classmethod update_name(project_id, model_id, current_name, new_name) Change cluster name from current_name to new_name - Parameters: - project_id ( str ) – ID of the project that the model is part of. - model_id ( str ) – ID of the model. - current_name ( str ) – Current cluster name - new_name ( str ) – New cluster name - Return type: List of Cluster ### ClusterInsight ### class datarobot.models.model.ClusterInsight Holds data on all insights related to feature as well as breakdown per cluster. - Parameters: - feature_name ( str ) – Name of a feature from the dataset. - feature_type ( str ) – Type of feature. - insights ( List[ClusterInsight] ) – List provides information regarding the importance of a specific feature in relation to each cluster. Results help understand how the model is grouping data and what each cluster represents. - feature_impact ( float ) – Impact of a feature ranging from 0 to 1. #### classmethod compute(project_id, model_id, max_wait=600) Starts creation of cluster insights for the model and if successful, returns computed ClusterInsights. This method allows calculation to continue for a specified time and if not complete, cancels the request. - Parameters: - project_id ( str ) – ID of the project to begin creation of cluster insights for. - model_id ( str ) – ID of the project model to begin creation of cluster insights for. - max_wait ( int ) – Maximum number of seconds to wait canceling the request. - Return type: List[ClusterInsight] - Raises: - ClientError – Server rejected creation due to client error. Most likely cause is bad project_id or model_id . - AsyncFailureError – Indicates whether any of the responses from the server are unexpected. - AsyncProcessUnsuccessfulError – Indicates whether the cluster insights computation failed or was cancelled. - AsyncTimeoutError – Indicates whether the cluster insights computation did not resolve within the specified time limit (max_wait). --- # Visual AI projects URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/visualai.html > Learn how to create and work with Visual AI projects using image data in DataRobot. With Visual AI, DataRobot allows you to use image data for modeling. You can create projects with one or multiple image features and also mix them with other DataRobot-supported feature types. You can find more information about [Visual AI](https://docs.datarobot.com/en/docs/modeling/special-workflows/visual-ai/index.html) in the Platform documentation. ## Create a Visual AI project DataRobot offers you different ways to prepare your dataset and to start a Visual AI project. The various ways to do this are covered in detail in the documentation, [Preparing the dataset](https://docs.datarobot.com/en/docs/modeling/special-workflows/visual-ai/vai-model.html#prepare-the-dataset). For the examples given here the images are partitioned into named directories. In the following, images are partitioned into named directories, which serve as labels for the project. For example, to predict on images of cat and dog breeds, labels could be abyssinian, american_bulldog, etc. ``` /home/user/data/imagedataset ├── abyssinian │   ├── abyssinian01.jpg │   ├── abyssinian02.jpg │   ├── … ├── american_bulldog │ ├── american_bulldog01.jpg │ ├── american_bulldog02.jpg │ ├── … ``` You then compress the directory containing the named directories into a ZIP file, creating the dataset used for the project. ``` from datarobot.models import Project, Dataset dataset = Dataset.create_from_file(file_path='/home/user/data/imagedataset.zip') project = Project.create_from_dataset(dataset.id, project_name='My Image Project') ``` ### Target Since this example uses named directories the target name must be `class`, which will contain the name of each directory in the ZIP file. ### Other parameters Setting modeling parameters, such as partitioning method, queue mode, etc, functions in the same way as starting a non-image project. ## Start modeling Once you have set modeling parameters, use the following code snippet to specify parameters and start the modeling process. ``` from datarobot import AUTOPILOT_MODE project.analyze_and_model(target='class', mode=AUTOPILOT_MODE.QUICK) ``` You can also pass optional parameters to `project.analyze_and_model` to change aspects of the modeling process. Some of those parameters include: - worker_count – int, sets the number of workers used for modeling. - partitioning_method – PartitioningMethod object. For a full reference of available parameters, see [Project.analyze_and_model](https://docs.datarobot.com/en/docs/api/reference/sdk/projects.html#datarobot.models.Project.analyze_and_model). You can use the `mode` parameter to set the Autopilot mode.`AUTOPILOT_MODE.FULL_AUTO`, is the default, triggers modeling with no further actions necessary. Other accepted modes include `AUTOPILOT_MODE.MANUAL` for manual mode (choose your own models to run rather than running the full Autopilot) and `AUTOPILOT_MODE.QUICK` to run on a more limited set of models and get insights more quickly (“quick run”). ## Interact with a Visual AI project The following code snippets may be used to access Visual AI images and insights. ### List sample images Sample images allow you to see a subset of images, chosen by DataRobot, in the dataset. The returned `SampleImage` objects have an associated `target_value` that will allow you to categorize the images (abyssinian, american_bulldog, etc). Until you set the target and EDA2 has finished, the `target_value` will be `None`. ``` import io import PIL.Image from datarobot.models.visualai import SampleImage column_name = "image" number_of_images_to_show = 5 for sample in SampleImage.list(project.id, column_name)[:number_of_images_to_show]: # Display the image in the GUI bio = io.BytesIO(sample.image.image_bytes) img = PIL.Image.open(bio) img.show() ``` The results would be images such as: ### List duplicate images Duplicate images, images with different names but are determined by DataRobot to be the same, may exist in a dataset. If this happens, the code returns one of the images and the number of times it occurs in the dataset. ``` from datarobot.models.visualai import DuplicateImage column_name = "image" for duplicate in DuplicateImage.list(project.id, column_name): # To show an image see the previous sample image example print(f"Image id = {duplicate.image.id} has {duplicate.count} duplicates") ``` ### Activation maps Activation maps are overlaid on the images to show which image areas are driving model prediction decisions. Detailed explanations are available in DataRobot Platform documentation, [Model insights](https://docs.datarobot.com/en/docs/modeling/special-workflows/visual-ai/vai-insights.html). #### Compute activation maps To begin, you must first compute activation maps. The following snippet is an example of starting the computation for a Keras model in a Visual AI project. The `compute` method returns a URL that can be used to determine when the computation completes. ``` from datarobot.models.visualai import ImageActivationMap keras_model = project.get_models(search_params={'name': 'Keras'})[0] status_url = ImageActivationMap.compute(project.id, keras_model.id) print(status_url) ``` #### List activation maps After activation maps are computed, you can download them from the DataRobot server. The following snippet is an example of how to get the activation maps and how to plot them. ``` import PIL.Image from datarobot.models.visualai import ImageActivationMap column_name = "image" max_activation_maps = 5 keras_model = project.get_models(search_params={'name': 'Keras'})[0] for activation_map in ImageActivationMap.list(project.id, keras_model.id, column_name)[:max_activation_maps]: bio = io.BytesIO(activation_map.overlay_image.image_bytes) img = PIL.Image.open(bio) img.show() ``` ### Image embeddings Image embeddings allow you to get an impression on how similar two images look to a featurizer network. The embeddings project images from their high-dimensional feature space onto a 2D plane. The closer the images appear in this plane, the more similar they look to the featurizer. Detailed explanations are available in the DataRobot Platform documentation, [Model insights](https://docs.datarobot.com/en/docs/modeling/special-workflows/visual-ai/vai-insights.html). #### Compute image embeddings You must compute image embeddings before retrieving. The following snippet is an example of starting the computation for a Keras model in our Visual AI project. The `compute` method returns a URL that can be used to determine when the computation is complete. ``` from datarobot.models.visualai import ImageEmbedding keras_model = project.get_models(search_params={'name': 'Keras'})[0] status_url = ImageEmbedding.compute(project.id, keras_model.id) print(status_url) ``` #### List image embeddings After image embeddings are computed, you can download them from the DataRobot server. The following snippet is an example of how to get the embeddings for a model and plot them. ``` from matplotlib.offsetbox import OffsetImage, AnnotationBbox import matplotlib.pyplot as plt import numpy as np import PIL.Image from datarobot.models.visualai import ImageEmbedding column_name = "image" keras_model = project.get_models(search_params={'name': 'Keras'})[0] zoom = 0.15 fig, ax = plt.subplots(figsize=(15,10)) for image_embedding in ImageEmbedding.list(project.id, keras_model.id, column_name): image_bytes = image_embedding.image.image_bytes x_position = image_embedding.position_x y_position = image_embedding.position_y image = PIL.Image.open(io.BytesIO(image_bytes)) offset_image = OffsetImage(np.array(image), zoom=zoom) annotation_box = AnnotationBbox(offset_image, (x_position, y_position), xycoords='data', frameon=False) ax.add_artist(annotation_box) ax.update_datalim([(x_position, y_position)]) ax.autoscale() ax.grid(True) fig.show() ``` ### Image augmentation Image Augmentation is a processing step in the DataRobot blueprint that creates new images for training by randomly transforming existing images, thereby increasing the size of (i.e., “augmenting”) the training data. Detailed explanations are available in the DataRobot Platform documentation, [Creating augmented models](https://docs.datarobot.com/en/docs/modeling/special-workflows/visual-ai/tti-augment/ttia-introduction.html). #### Create image augmentation list To create image augmentation samples, you need to provide an image augmentation list. This list holds all information required to compute image augmentation samples. The following snippet shows how to create an image augmentation list. It is then used to compute image augmentation samples. ``` from datarobot.models.visualai import ImageAugmentationList blur_param = {"name": "maximum_filter_size", "currentValue": 10} blur = {"name": "blur", "params": [blur_param]} flip = {"name": "horizontal_flip", "params": []} image_augmentation_list = ImageAugmentationList.create( name="my blur and flip augmentation list", project_id=project.id, feature_name="image", transformation_probability=0.5, number_of_new_images=5, transformations=[blur, flip], ) print(image_augmentation_list) ``` #### List image augmentation lists You can retrieve all available augmentation lists for a project by project_id. ``` from datarobot.models.visualai import ImageAugmentationList image_augmentation_lists = ImageAugmentationList.list( project_id=project.id ) print(image_augmentation_lists) ``` #### Compute and retrieve image augmentation samples You must compute image augmentation samples before retrieving. To compute image augmentation sample, you will need an image augmentation list. This list holds all parameters and transformation information needed to compute samples. You can either create a new one or retrieve an existing one. The following snippet is an example of computing and retrieving image augmentation samples. It uses the previous snippet that creates an image augmentation list, but instead uses it to compute and retrieve image augmentation samples using the `compute_samples` method. ``` from datarobot.models.visualai import ImageAugmentationList, ImageAugmentationSample image_augmentation_list = ImageAugmentationList.get('') for sample in image_augmentation_list.compute_samples(): # Display the image in popup widows bio = io.BytesIO(sample.image.image_bytes) img = PIL.Image.open(bio) img.show() ``` #### List image augmentation samples If image augmentation samples were already computed instead of recomputing them we can retrieve the last sample that was computed for image augmentation list from DataRobot server. The following snippet is an example of how to get the image augmentation samples. ``` import io import PIL.Image from datarobot.models.visualai import ImageAugmentationList image_augmentation_list = ImageAugmentationList.get('') for sample in image_augmentation_list.retrieve_samples(): # Display the image in popup widows bio = io.BytesIO(sample.image.image_bytes) img = PIL.Image.open(bio) img.show() ``` #### Configure augmentations to use during training In order to automatically augment a dataset during training the DataRobot server will look for an augmentation list associated with the project that has the key `initial_list` set to `True`. An augmentation list like this can be created with the following code snippet. If it is created for the project before autopilot is started. It will be used to automatically augment the images in the training dataset. ``` from datarobot.models.visualai import ImageAugmentationList blur_param = {"name": "maximum_filter_size", "currentValue": 10} blur = {"name": "blur", "params": [blur_param]} flip = {"name": "horizontal_flip", "params": []} transforms_to_apply = ImageAugmentationList.create(name="blur and scale", project_id=project.id, feature_name='image', transformation_probability=0.5, number_of_new_images=5, transformations=[blur, flip], initial_list=True) ``` #### Determine available transformations for augmentations The Augmentation List in the example above supports horizontal flip and blur transformations, but DataRobot supports several other transformations. To retrieve the list of supported transformations use the `ImageAugmentationOptions` object as the example below shows. ``` from datarobot.models.visualai import ImageAugmentationOptions options = ImageAugmentationOptions.get(project.id) ``` #### Convert images to base64-encoded strings for predictions If your training dataset contained images, images in the prediction dataset need to be converted to a base64-encoded strings so it can be fully contained in the prediction request (for example, in a CSV file or JSON). For more detail, see: [Work with binary data](https://docs.datarobot.com/en/docs/api/dev-learning/python/modeling/spec/binary_data.html#binary-data) ### License For the examples here we used the [The Oxford-IIIT Pet Dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/) licensed under [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/) --- # Batch predictions URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/batch_predictions.html The batch prediction API provides a way to score large datasets using flexible options for intake and output on the Prediction Servers you have already deployed. The main features are: - Flexible options for intake and output. - Stream local files and start scoring while still uploading and simultaneously downloading the results. - Score large datasets from and to S3. - Connect to your database using JDBC with bidirectional streaming of scoring data and results. - Intake and output options can be mixed and do not need to match. So scoring from a JDBC source to an S3 target is also an option. - Protection against overloading your prediction servers with the option to control the concurrency level for scoring. - Prediction explanations can be included (with the option to add thresholds). - Passthrough columns are supported to correlate scored data with source data. - You can include prediction warnings in the output. To interact with batch predictions, see the [BatchPredictionJob](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#batch-prediction-api) class. ## Make batch predictions with a deployment DataRobot provides a utility function to make batch predictions using a deployment: [Deployment.predict_batch](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.predict_batch). ``` import datarobot as dr deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0') # To note: `source` can be a file path, a file or a pandas DataFrame prediction_results_as_dataframe = deployment.predict_batch( source="./my_local_file.csv", ) ``` ## Scoring local CSV files DataRobot provides a utility function for scoring to and from local CSV files: [BatchPredictionJob.score_to_file](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score_to_file). The first parameter can be either: - A path to a CSV dataset - A file-like object - A Pandas DataFrame For larger datasets, you should avoid using a DataFrame, as it loads the entire dataset into memory. The other options do not. ``` import datarobot as dr deployment_id = '5dc5b1015e6e762a6241f9aa' dr.BatchPredictionJob.score_to_file( deployment_id, './data_to_predict.csv', './predicted.csv', ) ``` The input file is streamed to DataRobot’s API and scoring starts immediately. As soon as results start coming in, they start to be downloaded. The entire call is blocked until the file has been scored. ## Scoring from and to S3 DataRobot provides a small utility function for scoring to and from CSV files hosted on S3: [BatchPredictionJob.score_s3](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score_s3). This requires that the intake and output buckets share the same credentials (see [Credentials](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#credentials-api-doc) and [Credential.create_s3](https://docs.datarobot.com/en/docs/api/reference/sdk/credentials.html#datarobot.models.Credential.create_s3)) or that their access policy is set to public: Note that the S3 output functionality has a limit of 100 GB. ``` import datarobot as dr deployment_id = '5dc5b1015e6e762a6241f9aa' cred = dr.Credential.get('5a8ac9ab07a57a0001be501f') job = dr.BatchPredictionJob.score_s3( deployment=deployment_id, source_url='s3://mybucket/data_to_predict.csv', destination_url='s3://mybucket/predicted.csv', credential=cred, ) ``` ## Scoring from and to Azure Cloud Storage DataRobot provides the same support for Azure through the utility function [BatchPredictionJob.score_azure](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score_azure). This requires that you add an Azure connection string to the DataRobot credentials store. (see [Credentials](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#credentials-api-doc) and [Credential.create_azure](https://docs.datarobot.com/en/docs/api/reference/sdk/credentials.html#datarobot.models.Credential.create_azure)) ``` import datarobot as dr deployment_id = '5dc5b1015e6e762a6241f9aa' cred = dr.Credential.get('5a8ac9ab07a57a0001be501f') job = dr.BatchPredictionJob.score_azure( deployment=deployment_id, source_url='https://mybucket.blob.core.windows.net/bucket/data_to_predict.csv', destination_url='https://mybucket.blob.core.windows.net/results/predicted.csv', credential=cred, ) ``` ## Scoring from and to Google Cloud Platform DataRobot provides the same support for GCP through the utility function [BatchPredictionJob.score_gcp](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score_gcp). It requires you to add a GCP connection string to the DataRobot credentials store. (See [Credentials](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#credentials-api-doc) and [Credential.create_gcp](https://docs.datarobot.com/en/docs/api/reference/sdk/credentials.html#datarobot.models.Credential.create_gcp).) ``` import datarobot as dr deployment_id = '5dc5b1015e6e762a6241f9aa' cred = dr.Credential.get('5a8ac9ab07a57a0001be501f') job = dr.BatchPredictionJob.score_gcp( deployment=deployment_id, source_url='gs:/bucket/data_to_predict.csv', destination_url='gs://results/predicted.csv', credential=cred, ) ``` ## Manually configure a batch prediction job If you can’t use any of the utilities above, you are also free to manually configure your job. This requires configuring an intake and output option using the `intake_settings` and `output_settings` parameters on [BatchPredictionJob.score](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score). Credentials may be created with [Credentials API](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#credentials-api-doc). ``` import datarobot as dr deployment_id = '5dc5b1015e6e762a6241f9aa' dr.BatchPredictionJob.score( deployment_id, intake_settings={ 'type': 's3', 'url': 's3://public-bucket/data_to_predict.csv', 'credential_id': '5a8ac9ab07a57a0001be501f', }, output_settings={ 'type': 'localFile', 'path': './predicted.csv', }, ) ``` ### Supported intake types The `type` field selects the intake adapter. Supported values are `localFile`, `s3`, `azure`, `gcp`, `dataset`, `jdbc`, `snowflake`, `synapse`, `bigquery`, and `datasphere`. Intake and output types can be mixed (for example, JDBC intake to S3 output). The following sections describe configuration parameters for each intake type: #### Local file intake Set `type` to `localFile` and pass scoring data with `file` —a file-like object, a string path to a CSV file, or a `pandas.DataFrame`: ``` intake_settings={ 'type': 'localFile', 'file': './data_to_predict.csv', } ``` #### S3 CSV intake S3 CSV intake requires you to pass an S3 URL to the CSV file to be scored in the `url` parameter. Optional parameters: - credential_id : str, ID of stored AWS credentials (see Credential API ). - endpoint_url : str, a non-default S3 endpoint URL. Omit to use the default AWS endpoint. ``` intake_settings={ 'type': 's3', 'url': 's3://public-bucket/data_to_predict.csv', } ``` If the bucket is not publicly accessible, you can supply AWS credentials using the following parameters: - aws_access_key_id - aws_secret_access_key - aws_session_token Save it to the [Credential API](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#s3-creds-usage): ``` import datarobot as dr # get to make sure it exists credential_id = '5a8ac9ab07a57a0001be501f' cred = dr.Credential.get(credential_id) intake_settings={ 'type': 's3', 'url': 's3://private-bucket/data_to_predict.csv', 'credential_id': cred.credential_id, } ``` #### JDBC intake JDBC intake requires you to create a [DataStore](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/database_connectivity.html#database-connectivity-overview) and [Credential](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#basic-creds-usage) for your database. Identify the data source with either a SQL `query` or a combination of `table`, `schema`, and/or `catalog`: - data_store_id : str, the ID of the external data store connected to the JDBC data source. - query : str (optional if table , schema , and/or catalog is specified), a SELECT statement for the data to predict. - table : str (optional if query is specified), the database table name. - schema : str (optional if query is specified), the database schema name. - catalog : str (optional if query is specified), the database catalog name (new in v2.22). - fetch_size : Optional[int], row batch size when reading from the database. Adjust to balance throughput and memory usage. - credential_id : Optional[str], ID of credentials with read access (see Credential ). ``` # get to make sure it exists datastore_id = '5a8ac9ab07a57a0001be5010' data_store = dr.DataStore.get(datastore_id) credential_id = '5a8ac9ab07a57a0001be501f' cred = dr.Credential.get(credential_id) intake_settings = { 'type': 'jdbc', 'table': 'table_name', 'schema': 'public', # optional, if supported by database 'catalog': 'master', # optional, if supported by database 'data_store_id': data_store.id, 'credential_id': cred.credential_id, } ``` #### Azure blob storage intake Azure intake uses the same parameters as S3 intake, with an Azure blob storage URL in `url`. See [BatchPredictionJob.score_azure](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score_azure) for a utility method, or configure manually: ``` intake_settings={ 'type': 'azure', 'url': 'https://storage_account.blob.core.windows.net/container/data_to_predict.csv', 'credential_id': '5a8ac9ab07a57a0001be501f', } ``` #### Google Cloud Storage intake GCP intake uses the same parameters as S3 intake, with a GCS URL in `url`. See [BatchPredictionJob.score_gcp](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score_gcp) for a utility method, or configure manually: ``` intake_settings={ 'type': 'gcp', 'url': 'gs://bucket/data_to_predict.csv', 'credential_id': '5a8ac9ab07a57a0001be501f', } ``` #### Snowflake and Synapse intake For Snowflake and Synapse data sources, set `type` to `snowflake` or `synapse` and use the same parameters as [JDBC intake](https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/batch_predictions.html#batch-predictions-jdbc-intake). #### BigQuery intake BigQuery intake requires you to create a GCS [Credential](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#basic-creds-usage) for your database: ``` # get to make sure it exists credential_id = '5a8ac9ab07a57a0001be501f' cred = dr.Credential.get(credential_id) intake_settings = { 'type': 'bigquery', 'dataset': 'dataset_name', 'table': 'table_or_view_name', 'bucket': 'bucket_in_gcs', 'credential_id': cred.credential_id, } ``` #### AI Catalog intake Set `type` to `dataset` and pass the scoring data as a `dr.Dataset` object in the `dataset` parameter. You must set both `type` and `dataset`; passing a dataset ID string alone is not supported. Create a [Dataset](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/dataset.html#datasets) and retrieve it with `dr.Dataset.get()`: ``` # get to make sure it exists dataset_id = '5a8ac9ab07a57a0001be501f' dataset = dr.Dataset.get(dataset_id) intake_settings={ 'type': 'dataset', 'dataset': dataset } ``` Or, if you want a `version_id` other than the latest, supply your own. ``` # get to make sure it exists dataset_id = '5a8ac9ab07a57a0001be501f' dataset = dr.Dataset.get(dataset_id) intake_settings={ 'type': 'dataset', 'dataset': dataset, 'dataset_version_id': 'another_version_id' } ``` #### Datasphere intake Datasphere intake requires you to create a [DataStore](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/database_connectivity.html#database-connectivity-overview) and [Credential](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#basic-creds-usage) for your database. Set `type` to `datasphere` and pass the following parameters: | Parameter | Type | Required | Description | | --- | --- | --- | --- | | data_store_id | str | Yes | ID of the external data store connected to the Datasphere data source. | | table | str | Yes | Name of the database table. | | schema | str | Yes | Name of the database schema. | | credential_id | str | Yes | ID of credentials with read access to the Datasphere data source. | ``` # get to make sure it exists datastore_id = '5a8ac9ab07a57a0001be5011' data_store = dr.DataStore.get(datastore_id) credential_id = '5a8ac9ab07a57a0001be501f' cred = dr.Credential.get(credential_id) intake_settings = { 'type': 'datasphere', 'table': 'table_name', 'schema': 'DATASPHERE_SPACE_NAME', 'data_store_id': data_store.id, 'credential_id': cred.credential_id, } ``` #### DSS (training data) intake For scoring subsets of training data with a leaderboard model, use the `dss` intake type. You must also set `timeseries_settings` with `type` set to `training`. - project_id : str, the project to fetch training data from. Access to the project is required. - partition : str, subset of training data to score, one of datarobot.enums.TrainingDataSubsets . ``` intake_settings = { 'type': 'dss', 'project_id': '5a8ac9ab07a57a0001be5010', 'partition': 'holdout', } ``` ### Supported output types The `type` field selects the output adapter. Supported values are `localFile`, `s3`, `azure`, `gcp`, `jdbc`, `snowflake`, `synapse`, `bigquery`, and `datasphere`. The sections below describe configuration parameters for each output type. #### Local file output Set `type` to `localFile`. The optional `path` parameter controls how results are retrieved: | Parameter | Type | Required | Description | | --- | --- | --- | --- | | path | str | No | Path to save scored data as CSV. | If `path` is not specified, [BatchPredictionJob.score](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score) returns after the upload completes and you must download results yourself with [BatchPredictionJob.download](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.download). If `path` is specified, the call blocks until the job finishes. When no other jobs are processing on the targeted prediction instance, uploading, scoring, and downloading run in parallel without waiting for the full job to complete. Otherwise, the call still blocks but begins downloading scored data as soon as it is generated. Specifying `path` is the fastest way to get predictions. ``` output_settings={ 'type': 'localFile', 'path': './predicted.csv', } ``` Alternatively, omit `path` and download results after the job completes. If the job is not finished scoring, the call to [BatchPredictionJob.download](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.download) streams data scored so far and blocks until more is available. You can poll for job completion using [BatchPredictionJob.get_status](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.get_status) or use [BatchPredictionJob.wait_for_completion](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.wait_for_completion) to wait. ``` import datarobot as dr deployment_id = '5dc5b1015e6e762a6241f9aa' job = dr.BatchPredictionJob.score( deployment_id, intake_settings={ 'type': 'localFile', 'file': './data_to_predict.csv', }, output_settings={ 'type': 'localFile', }, ) job.wait_for_completion() with open('./predicted.csv', 'wb') as f: job.download(f) ``` #### S3 CSV output S3 CSV output requires you to pass an S3 URL to the CSV file where the scored data should be saved in the `url` parameter. Optional parameters: - credential_id : str, ID of stored AWS credentials (see Credential API ). - endpoint_url : str, a non-default S3 endpoint URL. Omit to use the default AWS endpoint. ``` output_settings={ 'type': 's3', 'url': 's3://public-bucket/predicted.csv', } ``` Most likely, the bucket is not publicly accessible for writes, but you can supply AWS credentials using these parameters: - aws_access_key_id - aws_secret_access_key - aws_session_token Save it to the [Credential API](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#s3-creds-usage). Here is an example: ``` # get to make sure it exists credential_id = '5a8ac9ab07a57a0001be501f' cred = dr.Credential.get(credential_id) output_settings={ 'type': 's3', 'url': 's3://private-bucket/predicted.csv', 'credential_id': cred.credential_id, } ``` #### JDBC output Just as for the input, JDBC output requires you to create a [DataStore](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/database_connectivity.html#database-connectivity-overview) and [Credential](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#basic-creds-usage) for your database. You must also specify `statement_type`, which should be one of `datarobot.enums.AVAILABLE_STATEMENT_TYPES`: - data_store_id : str, the ID of the external data store connected to the JDBC data source. - table : str, the database table name. - schema : Optional[str], the database schema name. - catalog : Optional[str], the database catalog name (new in v2.22). - statement_type : str, the type of insertion statement to create. - update_columns : list[string] (optional), column names to update when statement_type is an update or upsert variant. - where_columns : list[string] (optional), column names used in the WHERE clause when statement_type is insert or update. - credential_id : str, ID of credentials with write access (see Credential ). ``` # get to make sure it exists datastore_id = '5a8ac9ab07a57a0001be5010' data_store = dr.DataStore.get(datastore_id) credential_id = '5a8ac9ab07a57a0001be501f' cred = dr.Credential.get(credential_id) output_settings = { 'type': 'jdbc', 'table': 'table_name', 'schema': 'public', # optional, if supported by database 'catalog': 'master', # optional, if supported by database 'statement_type': 'insert', 'data_store_id': data_store.id, 'credential_id': cred.credential_id, } ``` #### BigQuery output Just as for the input, BigQuery requires you to create a GCS [Credential](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#basic-creds-usage) to access BigQuery: ``` # get to make sure it exists credential_id = '5a8ac9ab07a57a0001be501f' cred = dr.Credential.get(credential_id) output_settings = { 'type': 'bigquery', 'dataset': 'dataset_name', 'table': 'table_name', 'bucket': 'bucket_in_gcs', 'credential_id': cred.credential_id, } ``` #### Datasphere output Datasphere output requires you to create a [DataStore](https://docs.datarobot.com/en/docs/api/dev-learning/python/data/database_connectivity.html#database-connectivity-overview) and [Credential](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#basic-creds-usage) for your database. Set `type` to `datasphere` and pass the following parameters: | Parameter | Type | Required | Description | | --- | --- | --- | --- | | data_store_id | str | Yes | ID of the external data store connected to the Datasphere data source. | | table | str | Yes | Name of the database table. | | schema | str | Yes | Name of the database schema. | | credential_id | str | Yes | ID of credentials with write access to the Datasphere data source. | ``` # get to make sure it exists datastore_id = '5a8ac9ab07a57a0001be5010' data_store = dr.DataStore.get(datastore_id) credential_id = '5a8ac9ab07a57a0001be501f' cred = dr.Credential.get(credential_id) output_settings = { 'type': 'datasphere', 'table': 'table_name', 'schema': 'DATASPHERE_SPACE_NAME', 'data_store_id': data_store.id, 'credential_id': cred.credential_id, } ``` #### Azure blob storage output Azure output uses the same parameters as S3 output, with an Azure blob storage URL in `url`. ``` output_settings={ 'type': 'azure', 'url': 'https://storage_account.blob.core.windows.net/container/predicted.csv', 'credential_id': '5a8ac9ab07a57a0001be501f', } ``` #### Google Cloud Storage output GCP output uses the same parameters as S3 output, with a GCS URL in `url`. ``` output_settings={ 'type': 'gcp', 'url': 'gs://bucket/predicted.csv', 'credential_id': '5a8ac9ab07a57a0001be501f', } ``` #### Snowflake and Synapse output For Snowflake and Synapse destinations, set `type` to `snowflake` or `synapse` and use the same parameters as [JDBC output](https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/batch_predictions.html#batch-predictions-jdbc-output). ### CSV settings `csv_settings` is an optional dict passed to [BatchPredictionJob.score](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score) that configures CSV parsing for intake and output files: - delimiter : str (optional, default , ), field delimiter. Use the string tab for TSV. Must be a one-character string or tab . - quotechar : str (optional, default " ), character used to quote fields containing the delimiter. - encoding : str (optional, default utf-8 ), file encoding (for example, shift_jis , latin_1 , or mskanji ). ### Time series settings `timeseries_settings` configures time series scoring for [BatchPredictionJob.score](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score) and [BatchPredictionJob.score_with_leaderboard_model](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score_with_leaderboard_model): - type : str, one of forecast (default), historical , or training . forecast makes predictions using forecast_point or rows without a target. historical calculates predictions for all forecast points and distances within the predictions_start_date / predictions_end_date range. training scores subsets of training data and must be used with the dss intake type. - forecast_point : Optional[datetime.datetime], forecast point for the dataset. Inferred from the dataset if omitted. Used when type is forecast . - predictions_start_date : Optional[datetime.datetime], start date for historical predictions. Inferred from the dataset if omitted. Used when type is historical . - predictions_end_date : Optional[datetime.datetime], end date for historical predictions. Inferred from the dataset if omitted. Used when type is historical . - relax_known_in_advance_features_check : bool (default False ). If True , missing values in known in advance features are allowed in the forecast window. For [BatchPredictionJob.apply_time_series_data_prep_and_score](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.apply_time_series_data_prep_and_score), `forecast_point` is required when `type` is `forecast`. ### Scoring parameters The following optional parameters are available on [BatchPredictionJob.score](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score) and related methods: | Parameter | Description | | --- | --- | | num_concurrent | Number of concurrent chunks to score simultaneously. Defaults to the deployment's available cores. Lower this to reserve resources for real-time scoring. | | chunk_size | Chunk size strategy or fixed size in bytes. Named strategies: auto (fixed or dynamic based on flipper), fixed (1MB for explanations, 5MB otherwise), dynamic. Or pass an integer for a fixed byte size. | | passthrough_columns | List of scoring columns to include in the output. Useful for correlating predictions with source data. | | passthrough_columns_set | Set to all to pass through every scoring column. Takes precedence over passthrough_columns. | | max_explanations | Number of features for which to compute prediction explanations. | | max_ngram_explanations | Number of ngram text explanations to compute, or all. Defaults to no ngram explanations. | | threshold_high | Only compute explanations for predictions above this threshold. Can be combined with threshold_low. | | threshold_low | Only compute explanations for predictions below this threshold. Can be combined with threshold_high. | | explanations_mode | Mode for multiclass and clustering prediction explanations. Defaults to explaining only the predicted class (equivalent to TopPredictionsMode(1)). | | prediction_warning_enabled | Include prediction warnings in the output. Supported for regression models only. | | include_prediction_status | Include the prediction_status column in the output. Defaults to False. | | skip_drift_tracking | Skip drift tracking for predictions from this job. Useful for non-production workloads. Defaults to False. | | abort_on_error | Abort the job when too many rows fail scoring. Set to False to score every row. Defaults to True. | | column_names_remapping | Dict mapping output column names to new names. Map a column to None to discard it. Defaults to {}. | | include_probabilities | Return probability columns in the output. Defaults to True. | | include_probabilities_classes | Subset of class probability columns to return. Defaults to all classes. | | prediction_threshold | Classification threshold between 0.0 and 1.0. Observations above the threshold are classified as the positive class. | | download_timeout | Seconds to wait for a local file download to become available. Set to -1 to wait indefinitely. If the timeout is reached, the job is aborted and RuntimeError is raised. | | download_read_timeout | Seconds to wait for the server to respond between download chunks. | | upload_read_timeout | Seconds to wait for the server to respond after a local file upload. | #### prediction_instance `prediction_instance` overrides the prediction server connection. Defaults to the deployment or system configuration: - hostName : str, prediction instance hostname. - sslEnabled : bool (optional, default True ). Set to False to disable SSL. - datarobotKey : Optional[str], organization-level DataRobot key for Managed AI Cloud prediction instances. - apiKey : Optional[str], API key for prediction requests. Defaults to the job creator's API key. ## Copy a previously submitted job To submit a job using parameters from a job that was previously submitted, use [BatchPredictionJob.score_from_existing](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score_from_existing). The first parameter is the job ID of another job. ``` import datarobot as dr previously_submitted_job_id = '5dc5b1015e6e762a6241f9aa' dr.BatchPredictionJob.score_from_existing( previously_submitted_job_id, ) ``` ## Scoring an in-memory Pandas DataFrame When working with DataFrames, DataRobot provides a method for scoring the data without first writing it to a CSV file and subsequently reading the data back from a CSV file: `BatchPredictionJob.score_pandas `. This method also joins the computed predictions into the existing DataFrame. The first parameter is the deployment ID and the second is the DataFrame to score. ``` import datarobot as dr import pandas as pd deployment_id = '5dc5b1015e6e762a6241f9aa' df = pd.read_csv('testdata/titanic_predict.csv') job, df = dr.BatchPredictionJob.score_pandas(deployment_id, df) ``` The method returns a copy of the job status and the updated DataFrame with the predictions added. So your DataFrame will now contain the following extra columns: - Survived_1_PREDICTION - Survived_0_PREDICTION - Survived_PREDICTION - THRESHOLD - POSITIVE_CLASS - prediction_status ``` print(df) PassengerId Pclass Name ... Survived_PREDICTION THRESHOLD POSITIVE_CLASS 0 892 3 Kelly, Mr. James ... 0 0.5 1 1 893 3 Wilkes, Mrs. James (Ellen Needs) ... 1 0.5 1 2 894 2 Myles, Mr. Thomas Francis ... 0 0.5 1 3 895 3 Wirz, Mr. Albert ... 0 0.5 1 4 896 3 Hirvonen, Mrs. Alexander (Helga E Lindqvist) ... 1 0.5 1 .. ... ... ... ... ... ... ... 413 1305 3 Spector, Mr. Woolf ... 0 0.5 1 414 1306 1 Oliva y Ocana, Dona. Fermina ... 0 0.5 1 415 1307 3 Saether, Mr. Simon Sivertsen ... 0 0.5 1 416 1308 3 Ware, Mr. Frederick ... 0 0.5 1 417 1309 3 Peter, Master. Michael J ... 1 0.5 1 [418 rows x 16 columns] ``` If you don’t want all of them or if you’re not happy with the names of the added columns, they can be modified using column remapping: ``` import datarobot as dr import pandas as pd deployment_id = '5dc5b1015e6e762a6241f9aa' df = pd.read_csv('testdata/titanic_predict.csv') job, df = dr.BatchPredictionJob.score_pandas( deployment_id, df, column_names_remapping={ 'Survived_1_PREDICTION': None, # discard column 'Survived_0_PREDICTION': None, # discard column 'Survived_PREDICTION': 'predicted', # rename column 'THRESHOLD': None, # discard column 'POSITIVE_CLASS': None, # discard column }, ) ``` Any column mapped to `None` will be discarded. Any column mapped to a string will be renamed. Any column not mentioned will be kept in the output untouched. Your DataFrame now contains the following extra columns: - predicted - prediction_status Refer to the documentation for [BatchPredictionJob.score](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score) to see the full range of available options. ## Batch prediction job definitions To submit a working Batch Prediction job, you must supply a variety of elements to the [datarobot.models.BatchPredictionJob.score()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJob.score) request payload depending on what type of prediction is required. Additionally, you must consider the type of intake and output adapters used for a given job. Every time a new batch prediction is created, the same amount of information must be stored somewhere outside of DataRobot and resubmitted every time. #### NOTE The `name` parameter must be unique across your organization. If you attempt to create multiple definitions with the same name, the request will fail. If you wish to free up a name, you must first [datarobot.models.BatchPredictionJobDefinition.delete()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.delete) the existing definition before creating this one. Alternatively, you can just [datarobot.models.BatchPredictionJobDefinition.update()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.update) the existing definition with a new name. For example, a request could look like: ``` import datarobot as dr deployment_id = "5dc5b1015e6e762a6241f9aa" job = dr.BatchPredictionJob.score( deployment_id, intake_settings={ "type": "s3", "url": "s3://bucket/container/file.csv", "credential_id": "5dc5b1015e6e762a6241f9bb" }, output_settings={ "type": "s3", "url": "s3://bucket/container/output.csv", "credential_id": "5dc5b1015e6e762a6241f9bb" }, ) job.wait_for_completion() with open("./predicted.csv", "wb") as f: job.download(f) ``` ## Job definitions If your use case requires the same (or similar) type(s) of predictions to be made multiple times, you can choose to create a Job Definition of the batch prediction job and store it for future use. The method for creating job definitions is [datarobot.models.BatchPredictionJobDefinition.create()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.create), which includes the `enabled`, `name`, and `schedule` parameters. ``` >>> import datarobot as dr >>> job_spec = { ... "num_concurrent": 4, ... "deployment_id": "5dc5b1015e6e762a6241f9aa", ... "intake_settings": { ... "url": "s3://foobar/123", ... "type": "s3", ... "format": "csv", ... "credential_id": "5dc5b1015e6e762a6241f9bb" ... }, ... "output_settings": { ... "url": "s3://foobar/123", ... "type": "s3", ... "format": "csv", ... "credential_id": "5dc5b1015e6e762a6241f9bb" ... }, ...} >>> definition = BatchPredictionJobDefinition.create( ... enabled=False, ... batch_prediction_job=job_spec, ... name="some_definition_name", ... schedule=None ... ) >>> definition BatchPredictionJobDefinition(foobar) ``` ## Execute a job definition ### Manual job execution To submit a stored job definition for scoring, you can either do so on a scheduled basis, described below, or manually submit the definition ID using [datarobot.models.BatchPredictionJobDefinition.run_once()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.run_once): ``` >>> import datarobot as dr >>> definition = dr.BatchPredictionJobDefinition.get("5dc5b1015e6e762a6241f9aa") >>> job = definition.run_once() >>> job.wait_for_completion() ``` ### Scheduled job execution A scheduled batch prediction job works just like a regular batch prediction job, but instead DataRobot handles the execution of the job. In order to schedule the execution of a batch prediction job, a definition must first be created using [datarobot.models.BatchPredictionJobDefinition.create()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.create), or updated using [datarobot.models.BatchPredictionJobDefinition.update()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.update). In this case, `enabled` is set to `True` and a `schedule` payload is provided. Alternatively, use a shorthand version with [datarobot.models.BatchPredictionJobDefinition.run_on_schedule()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.run_on_schedule): ``` >>> import datarobot as dr >>> schedule = { ... "day_of_week": [ ... 1 ... ], ... "month": [ ... "*" ... ], ... "hour": [ ... 16 ... ], ... "minute": [ ... 0 ... ], ... "day_of_month": [ ... 1 ... ] ...} >>> definition = dr.BatchPredictionJob.get("5dc5b1015e6e762a6241f9aa") >>> job = definition.run_on_schedule(schedule) ``` If the created job was not enabled previously, this method will also enable it. ## The schedule payload The `schedule` payload defines at what intervals the job should run, which can be combined in various ways to construct complex scheduling terms if needed. In all of the elements in the objects, you can supply either an asterisk `["*"]` denoting “every” time denomination or an array of integers (e.g.`[1, 2, 3]`) to define a specific interval. #### The schedule payload elements | Key | Possible values | Example | Description | | --- | --- | --- | --- | | minute | ["*"] or [0 ... 59] | [15, 30, 45] | The job will run at these minute values for every hour of the day. | | hour | ["*"] or [0 ... 23] | [12,23] | The hour(s) of the day that the job will run. | | month | ["*"] or [1 ... 12] | ["jan"] | Strings, either 3-letter abbreviations or the full name of the month, can be used interchangeably (e.g., “jan” or “october”).Months that are not compatible with day_of_month are ignored, for example {"day_of_month": [31], "month":["feb"]}. | | day_of_week | ["*"] or [0 ... 6] where (Sunday=0) | ["sun"] | The day(s) of the week that the job will run. Strings, either 3-letter abbreviations or the full name of the day, can be used interchangeably (e.g., “sunday”, “Sunday”, “sun”, or “Sun”, all map to [0]).NOTE: This field is additive with day_of_month, meaning the job will run both on the date specified by day_of_month and the day defined in this field. | | day_of_month | ["*"] or [1 ... 31] | [1, 25] | The date(s) of the month that the job will run. Allowed values are either [1 ... 31] or ["*"] for all days of the month.NOTE: This field is additive with day_of_week, meaning the job will run both on the date(s) defined in this field and the day specifiedby day_of_week (for example, dates 1st, 2nd, 3rd, plus every Tuesday). If day_of_month is set to ["*"] and day_of_week is defined,the scheduler will trigger on every day of the month that matches day_of_week (for example, Tuesday the 2nd, 9th, 16th, 23rd, 30th).Invalid dates such as February 31st are ignored. | ### Disable a scheduled job Job definitions are only be executed by the scheduler if `enabled` is set to `True`. If you have a job definition that was previously running as a scheduled job, but should now be stopped, simply [datarobot.models.BatchPredictionJobDefinition.delete()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.delete) to remove it completely, or [datarobot.models.BatchPredictionJobDefinition.update()](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.update) it with `enabled=False` if you want to keep the definition, but stop the scheduled job from executing at intervals. If a job is currently running, this will finish execution regardless. ``` >>> import datarobot as dr >>> definition = dr.BatchPredictionJobDefinition.get("5dc5b1015e6e762a6241f9aa") >>> definition.delete() ``` --- # Predictions URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/index.html The following sections describe the components to making predictions in DataRobot: - Generate predictions : Initiate a prediction job with the Model.request_predictions() method. This method can use either a training dataset or predictions dataset for scoring. - Batch predictions : Score large sets of data with batch predictions. You can define jobs and their schedule. - Prediction API : Use DataRobot’s Prediction API . to make predictions on both a dedicated and/or a standalone prediction server. - Scoring Code : Qualifying models allow you to export Scoring Code and use DataRobot-generated models outside of the platform --- # Predict job URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/predictions/predict_job.html Making predictions is an asynchronous process. This means that when starting predictions with [Model.request_predictions()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_predictions) you will receive a `PredictJob` object in return for tracking the process responsible for fulfilling your request. You can use this object to get information about the predictions generation process before it has finished and be rerouted to the predictions themselves when the process is finished. To do so, use the [PredictJob](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#predict-job-api) class. ## Start making predictions Before actually requesting predictions, you should upload the dataset you wish to predict via `Project.upload_dataset`. Previously uploaded datasets can be viewed using `Project.get_datasets`. When uploading the dataset you can provide the path to a local file, a file object, raw file content, a `pandas.DataFrame` object, or the URL to a publicly available dataset. To start predicting on new data using a finished model, use [Model.request_predictions()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_predictions). It creates a new prediction generation process and returns a `PredictJob` object tracking this process. With it, you can monitor an existing `PredictJob` and retrieve generated predictions when the corresponding `PredictJob` is finished. ``` import datarobot as dr project_id = '5506fcd38bd88f5953219da0' model_id = '5506fcd98bd88f1641a720a3' project = dr.Project.get(project_id) model = dr.Model.get( project=project_id, model_id=model_id, ) # As of v3.0, in addition to passing a ``dataset_id``, you can pass in a ``dataset``, ``file``, ``file_path`` or # ``dataframe`` to `Model.request_predictions`. predict_job = model.request_predictions(file_path='./data_to_predict.csv') # Alternative version uploading the dataset from a local path and passing it by its id dataset_from_path = project.upload_dataset('./data_to_predict.csv') predict_job = model.request_predictions(dataset_id=dataset_from_path.id) # Alternative version: upload the dataset as a file object and pass it by using its dataset id with open('./data_to_predict.csv') as data_to_predict: dataset_from_file = project.upload_dataset(data_to_predict) predict_job = model.request_predictions(dataset_id=dataset_from_file.id) # OR predict_job = model.request_predictions(dataset_id=dataset_from_file.id) ``` ## Listing Predictions Use `Predictions.list()` to return a list of predictions generated on a project: ``` import datarobot as dr predictions = dr.Predictions.list('58591727100d2b57196701b3') print(predictions) >>>[Predictions(prediction_id='5b6b163eca36c0108fc5d411', project_id='5b61bd68ca36c04aed8aab7f', model_id='5b61bd7aca36c05744846630', dataset_id='5b6b1632ca36c03b5875e6a0'), Predictions(prediction_id='5b6b2315ca36c0108fc5d41b', project_id='5b61bd68ca36c04aed8aab7f', model_id='5b61bd7aca36c0574484662e', dataset_id='5b6b1632ca36c03b5875e6a0'), Predictions(prediction_id='5b6b23b7ca36c0108fc5d422', project_id='5b61bd68ca36c04aed8aab7f', model_id='5b61bd7aca36c0574484662e', dataset_id='55b6b1632ca36c03b5875e6a0') ] ``` You can pass following parameters to filter the result: - model_id : A string used to filter returned predictions by model_id . - dataset_id : A string used to filter returned predictions by dataset_id . ## Get an existing PredictJob Use `PredictJob.get` method to retrieve an existing job. This will give you a `PredictJob` matching the latest status of the job if it has not completed. If predictions have finished building, `PredictJob.get` will raise a `PendingJobFinished` exception. ``` import time import datarobot as dr predict_job = dr.PredictJob.get( project_id=project_id, predict_job_id=predict_job_id, ) predict_job.status >>> 'queue' # wait for generation of predictions (in a very inefficient way) time.sleep(10 * 60) predict_job = dr.PredictJob.get( project_id=project_id, predict_job_id=predict_job_id, ) >>> dr.errors.PendingJobFinished # now the predictions are finished predictions = dr.PredictJob.get_predictions( project_id=project.id, predict_job_id=predict_job_id, ) ``` ## Get generated predictions After predictions are generated, use `PredictJob.get_predictions` to get newly-generated predictions. If predictions have not yet been finished, it will raise a `JobNotFinished` exception. ``` import datarobot as dr predictions = dr.PredictJob.get_predictions( project_id=project.id, predict_job_id=predict_job_id, ) ``` ## Retrieve results If you just want to get generated predictions from a `PredictJob`, use `PredictJob.get_result_when_complete`. This function polls the status of the predictions generation process until it has finished, and then will return predictions. ``` dataset = project.get_datasets()[0] predict_job = model.request_predictions(dataset.id) predictions = predict_job.get_result_when_complete() ``` ## Get previously generated predictions If you don’t have a `PredictJob`, there are two more ways to retrieve predictions from the `Predictions` interface: 1. Get all prediction rows as a pandas.DataFrame object: ``` import datarobot as dr preds = dr.Predictions.get("5b61bd68ca36c04aed8aab7f", prediction_id="5b6b163eca36c0108fc5d411") df = preds.get_all_as_dataframe() df_with_serializer = preds.get_all_as_dataframe(serializer='csv') ``` 1. Download all prediction rows to a file as a CSV: ``` import datarobot as dr preds = dr.Predictions.get("5b61bd68ca36c04aed8aab7f", prediction_id="5b6b163eca36c0108fc5d411") preds.download_to_csv('predictions.csv') preds.download_to_csv('predictions_with_serializer.csv', serializer='csv') ``` ### Training predictions The training predictions interface allows you to compute and retrieve out-of-sample predictions for a model using the original project dataset. The predictions can be computed for all the rows, or restricted to validation or holdout data. As the predictions generated will be out-of-sample, they can be expected to have different results than if the project dataset were re-uploaded as a prediction dataset. ## Quick reference Training predictions generation is an asynchronous process. This means that when starting predictions with [datarobot.models.Model.request_training_predictions()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_training_predictions) you will receive back a [datarobot.models.TrainingPredictionsJob](https://docs.datarobot.com/en/docs/api/reference/sdk/jobs.html#datarobot.models.TrainingPredictionsJob) for tracking the process responsible for fulfilling your request. Actual predictions may be obtained with the help of a [datarobot.models.training_predictions.TrainingPredictions](https://docs.datarobot.com/en/docs/api/reference/sdk/training_predictions.html#datarobot.models.training_predictions.TrainingPredictions) object returned as the result of the training predictions job. There are three ways to retrieve training predictions: 1. Iterate prediction rows one by one as named tuples: ```python import datarobot as dr # Calculate new training predictions on all dataset training_predictions_job = model.request_training_predictions(dr.enums.DATA_SUBSET.ALL) training_predictions = training_predictions_job.get_result_when_complete() # Fetch rows from API and print them for prediction in training_predictions.iterate_rows(batch_size=250): print(prediction.row_id, prediction.prediction) ``` 1. Get all prediction rows as a pandas.DataFrame object: ``` import datarobot as dr # Calculate new training predictions on holdout partition of dataset training_predictions_job = model.request_training_predictions(dr.enums.DATA_SUBSET.HOLDOUT) training_predictions = training_predictions_job.get_result_when_complete() # Fetch training predictions as data frame dataframe = training_predictions.get_all_as_dataframe() ``` 1. Download all prediction rows to a file as a CSV document: ``` import datarobot as dr # Calculate new training predictions on all dataset training_predictions_job = model.request_training_predictions(dr.enums.DATA_SUBSET.ALL) training_predictions = training_predictions_job.get_result_when_complete() # Fetch training predictions and save them to file training_predictions.download_to_csv('my-training-predictions.csv') ``` --- # Chunking Service static dataset example URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/incremental-learning/chunking_service_v2_script_static.html Requirements: DataRobot Python SDK and the `SAMPLE_DATA_TO_START_PROJECT` feature flag enabled. Pipeline helpers use module-level `DR_API_TOKEN` and `DR_ENDPOINT` (set in the credentials cell). ## Authentication ``` import os DR_API_TOKEN = os.environ.get("DR_API_TOKEN", "") DR_ENDPOINT = os.environ.get("DR_ENDPOINT", "") if not DR_API_TOKEN or not DR_ENDPOINT: raise ValueError("Set DR_API_TOKEN and DR_ENDPOINT in the environment or edit this cell.") ``` ## Import libraries ``` from typing import Any, Optional import datarobot as dr from datarobot import UseCase from datarobot.enums import ChunkingPartitionMethod, ChunkingStrategy from datarobot.models.chunking_service_v2 import ChunkDefinition, DatasetDefinition from datarobot.models.project import Project _ = dr.Client(token=DR_API_TOKEN, endpoint=DR_ENDPOINT) ``` ## Configure pipeline functions ``` def add_project_to_use_case(use_case_id: str, project_id: str) -> None: project = dr.Project.get(project_id) use_case = UseCase.get(use_case_id=use_case_id) use_case.add(entity=project) def run_ssp_ai_catalog( dataset_id: str, target_column: str, target_class: Optional[str] = None, dataset_version_id: Optional[str] = None, use_case_id: Optional[str] = None, datetime_partition_column: Optional[str] = None, chunking_partition_method: Optional[ChunkingPartitionMethod] = ChunkingPartitionMethod.RANDOM, ) -> Project: """Create dataset + chunk definitions and a project with sample-to-start + incremental learning.""" dataset_definition = DatasetDefinition.create(dataset_id, dataset_version_id) DatasetDefinition.analyze(dataset_definition.id) dataset_definition = DatasetDefinition.get(dataset_definition.id) partition_args: dict[str, Any] = {} if chunking_partition_method == ChunkingPartitionMethod.RANDOM: partition_args["target_column"] = None partition_args["target_class"] = None partition_args["datetime_partition_column"] = None elif chunking_partition_method == ChunkingPartitionMethod.STRATIFIED: partition_args["target_column"] = target_column partition_args["target_class"] = target_class partition_args["datetime_partition_column"] = None else: partition_args["target_column"] = None partition_args["target_class"] = None partition_args["datetime_partition_column"] = datetime_partition_column chunk_definition = ChunkDefinition.create( dataset_definition.id, partition_method=chunking_partition_method, chunking_strategy_type=ChunkingStrategy.ROWS, **partition_args, ) ChunkDefinition.analyze(dataset_definition.id, chunk_definition.id) chunk_definition = ChunkDefinition.get(dataset_definition.id, chunk_definition.id) partition_label = (chunking_partition_method or ChunkingPartitionMethod.RANDOM).value.capitalize() project: Project = dr.Project.create_from_dataset( dataset_id, project_name=f"Sample to Start Project {partition_label} - Target: {target_column}", use_sample_from_dataset=True, max_wait=6000, ) project_partitioning_method = None if datetime_partition_column is not None: print(f"Setting up datetime partitioning for project {project.id}") spec = dr.DatetimePartitioningSpecification( datetime_partition_column=datetime_partition_column, use_time_series=False, ) full_part = dr.DatetimePartitioning.generate_optimized(project.id, spec, target_column) project_partitioning_method = dr.helpers.partitioning_methods.DatetimePartitioningId( full_part.datetime_partitioning_id, project.id ) print(f"Datetime partitioning set for project {project.id}") if use_case_id is not None: add_project_to_use_case(use_case_id=use_case_id, project_id=project.id) advanced_options = dr.helpers.AdvancedOptions( incremental_learning_only_mode=True, incremental_learning_on_best_model=True, chunk_definition_id=chunk_definition.id, incremental_learning_early_stopping_rounds=0, ) project.analyze_and_model( target=target_column, mode=dr.enums.AUTOPILOT_MODE.QUICK, partitioning_method=project_partitioning_method, advanced_options=advanced_options, worker_count=-1, max_wait=6000, ) return project ``` ## Configure and run Edit variables, then run.STRATIFIED needs `TARGET_CLASS`; DATE needs `DATETIME_PARTITION_COLUMN`. ``` # --- edit these --- DATASET_ID = "your-dataset-id" TARGET_COLUMN = "your_target" DATASET_VERSION_ID = None USE_CASE_ID = None TARGET_CLASS = None DATETIME_PARTITION_COLUMN = None CHUNKING_PARTITION_METHOD = "RANDOM" # RANDOM | STRATIFIED | DATE method = ChunkingPartitionMethod[CHUNKING_PARTITION_METHOD.upper()] if method == ChunkingPartitionMethod.DATE and not DATETIME_PARTITION_COLUMN: raise ValueError("DATETIME_PARTITION_COLUMN required for DATE") if method == ChunkingPartitionMethod.STRATIFIED and not TARGET_CLASS: raise ValueError("TARGET_CLASS required for STRATIFIED") project = run_ssp_ai_catalog( DATASET_ID, TARGET_COLUMN, target_class=TARGET_CLASS, dataset_version_id=DATASET_VERSION_ID, use_case_id=USE_CASE_ID, datetime_partition_column=DATETIME_PARTITION_COLUMN, chunking_partition_method=method, ) print(f"Project ID: {project.id}") ``` --- # Chunking Service dynamic dataset example URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/incremental-learning/dynamic_dataset_incremental_learning_v2.html Requirements: DataRobot Python SDK, credentials, named credential for the dataset, and the `SAMPLE_DATA_TO_START_PROJECT` feature flag enabled. Sample helpers use module-level `DR_API_TOKEN` and `DR_ENDPOINT` (set in the credentials cell). ## Authentication ``` import os DR_API_TOKEN = os.environ.get("DR_API_TOKEN", "") DR_ENDPOINT = os.environ.get("DR_ENDPOINT", "") if not DR_API_TOKEN or not DR_ENDPOINT: raise ValueError("Set DR_API_TOKEN and DR_ENDPOINT in the environment or edit this cell.") ``` ## Import libraries ``` import math import time from typing import Dict, List, Optional import datarobot as dr import pandas as pd from datarobot import UseCase from datarobot.enums import ChunkingPartitionMethod, ChunkingStrategy from datarobot.models.chunking_service_v2 import ChunkDefinition, DatasetDefinition from datarobot.models.project import Project from datarobot.utils.waiters import wait_for_async_resolution SAMPLE_SIZE = 500 * 1024**2 # bytes (500 MB) _ = dr.Client(token=DR_API_TOKEN, endpoint=DR_ENDPOINT) ``` ## Configure pipeline functions ``` def get_credential_id(credential_name: str) -> str: credentials = [cr for cr in dr.Credential.list() if cr.name == credential_name] if not credentials: raise ValueError(f"No credential found with name: {credential_name}") print(f"Credential found: {credential_name}") return str(credentials[0].credential_id) def add_project_to_use_case(use_case_id: str, project_id: str) -> None: project = dr.Project.get(project_id) use_case = UseCase.get(use_case_id=use_case_id) use_case.add(entity=project) def create_dataset_definition( dataset_id: str, dataset_version_id: Optional[str], name: str, credential_name: str, ) -> DatasetDefinition: credential_id = get_credential_id(credential_name) dataset_definition = DatasetDefinition.create( dataset_id, dataset_version_id, name=name, credentials_id=credential_id ) print(f"Created dataset definition: {dataset_definition.id}") DatasetDefinition.analyze(dataset_definition.id) dataset_definition = DatasetDefinition.get(dataset_definition.id) dataset_info = dataset_definition.dataset_info if dataset_info is not None: print( f"Dataset info - Total rows: {dataset_info.total_rows}, " f"Estimated size per row: {dataset_info.estimated_size_per_row}" ) return dataset_definition def create_sample_row_based(dataset_definition: DatasetDefinition) -> str: client = dr.Client(token=DR_API_TOKEN, endpoint=DR_ENDPOINT) dataset_info = dataset_definition.dataset_info if dataset_info is None: raise ValueError("Dataset definition has no dataset_info; ensure it has been analyzed") estimated_size_per_row = dataset_info.estimated_size_per_row dataset_props = dataset_definition.dataset_props sample_size_rows = math.ceil(SAMPLE_SIZE / estimated_size_per_row) print(f"Row count for {SAMPLE_SIZE / (1024**2):.1f}MB: {sample_size_rows}") sample_rows_payload = { "samplingStrategy": { "directive": "efficient-rowbased-sample", "arguments": {"size": sample_size_rows, "samplingMethod": "rows"}, } } response = client.post( f"{DR_ENDPOINT}/datasets/{dataset_props.dataset_id}/samples", json=sample_rows_payload, ) sample_response = response.json() row_based_sample_version_id = sample_response["catalogVersionId"] wait_for_async_resolution(client, response.headers["Location"]) response = client.get( f"{DR_ENDPOINT}/datasets/{dataset_props.dataset_id}/versions/?category=SAMPLE" ) samples = response.json() if samples["totalCount"] > 0: sample_df = pd.DataFrame(samples["data"]) mask = sample_df["categories"].apply(lambda x: "SAMPLE" in x) sample_df = sample_df[mask] assert row_based_sample_version_id in sample_df["versionId"].tolist(), ( f"Row based sample version {row_based_sample_version_id} not found in samples" ) else: raise ValueError(f"No sample version found for dataset {dataset_props.dataset_id}") return str(row_based_sample_version_id) def create_sample_percentage_based(dataset_definition: DatasetDefinition) -> str: client = dr.Client(token=DR_API_TOKEN, endpoint=DR_ENDPOINT) dataset_info = dataset_definition.dataset_info if dataset_info is None: raise ValueError("Dataset definition has no dataset_info; ensure it has been analyzed") source_size = dataset_info.source_size dataset_props = dataset_definition.dataset_props sample_size_percentage = (SAMPLE_SIZE / source_size) * 100 print(f"Percentage of dataset size for {SAMPLE_SIZE / (1024**2):.1f}MB: {sample_size_percentage:.2f}%") sample_percentage_payload = { "samplingStrategy": { "directive": "efficient-rowbased-sample", "arguments": {"size": sample_size_percentage, "samplingMethod": "percent"}, } } response = client.post( f"{DR_ENDPOINT}/datasets/{dataset_props.dataset_id}/samples", json=sample_percentage_payload, ) sample_response = response.json() percentage_sample_version_id = sample_response["catalogVersionId"] wait_for_async_resolution(client, response.headers["Location"]) response = client.get( f"{DR_ENDPOINT}/datasets/{dataset_props.dataset_id}/versions/?category=SAMPLE" ) samples = response.json() if samples["totalCount"] > 0: sample_df = pd.DataFrame(samples["data"]) mask = sample_df["categories"].apply(lambda x: "SAMPLE" in x) sample_df = sample_df[mask] assert percentage_sample_version_id in sample_df["versionId"].tolist(), ( f"Percentage sample version {percentage_sample_version_id} not found in samples" ) else: raise ValueError(f"No sample version found for dataset {dataset_props.dataset_id}") return str(percentage_sample_version_id) def create_project( dataset_id: str, target_column: str, chunk_definition: ChunkDefinition, use_case_id: Optional[str] = None, datetime_partition_column: Optional[str] = None, ) -> Project: project: Project = dr.Project.create_from_dataset( dataset_id, project_name="Dynamic Dataset Project with Incremental Learning OTV", use_sample_from_dataset=True, max_wait=6000, ) project_partitioning_method = None if datetime_partition_column is not None: print(f"Setting up datetime partitioning for project {project.id}") spec = dr.DatetimePartitioningSpecification( datetime_partition_column=datetime_partition_column, use_time_series=False, ) full_part = dr.DatetimePartitioning.generate_optimized(project.id, spec, target_column) project_partitioning_method = dr.helpers.partitioning_methods.DatetimePartitioningId( full_part.datetime_partitioning_id, project.id ) print(f"Datetime partitioning set for project {project.id}") else: print(f"No datetime partitioning; proceeding with defaults for project {project.id}") if use_case_id: add_project_to_use_case(use_case_id=use_case_id, project_id=project.id) advanced_options = dr.helpers.AdvancedOptions( incremental_learning_only_mode=True, incremental_learning_on_best_model=True, chunk_definition_id=chunk_definition.id, incremental_learning_early_stopping_rounds=0, ) project.analyze_and_model( target=target_column, mode=dr.enums.AUTOPILOT_MODE.QUICK, partitioning_method=project_partitioning_method, advanced_options=advanced_options, worker_count=-1, max_wait=6000, ) return project def create_chunk_definition( dataset_definition: DatasetDefinition, sorting_columns: List[str], target_column: Optional[str] = None, datetime_partition_column: Optional[str] = None, target_class: Optional[str] = None, chunking_partition_method: Optional[ChunkingPartitionMethod] = ChunkingPartitionMethod.RANDOM, ) -> ChunkDefinition: partition_args_dict: Dict[str, Optional[str]] if chunking_partition_method == ChunkingPartitionMethod.RANDOM: partition_args_dict = { "target_column": None, "target_class": None, "datetime_partition_column": None, } elif chunking_partition_method == ChunkingPartitionMethod.STRATIFIED: partition_args_dict = { "target_column": target_column, "target_class": target_class, "datetime_partition_column": None, } else: partition_args_dict = { "target_column": None, "target_class": None, "datetime_partition_column": datetime_partition_column, } chunk_definition = ChunkDefinition.create( dataset_definition.id, partition_method=chunking_partition_method, chunking_strategy_type=ChunkingStrategy.ROWS, target_column=partition_args_dict["target_column"], target_class=partition_args_dict["target_class"], datetime_partition_column=partition_args_dict["datetime_partition_column"], order_by_columns=sorting_columns, ) print(f"Created chunk definition: {chunk_definition.id}") ChunkDefinition.analyze(dataset_definition.id, chunk_definition.id) chunk_definition = ChunkDefinition.get(dataset_definition.id, chunk_definition.id) stats = chunk_definition.chunk_definition_stats if stats is not None: print(f"Chunk definition stats - Number of chunks: {stats.total_number_of_chunks}") return chunk_definition def run_dynamic_dataset_pipeline( dataset_id: str, target_column: str, sorting_columns: List[str], credential_name: str, target_class: Optional[str] = None, chunking_partition_method: Optional[ChunkingPartitionMethod] = ChunkingPartitionMethod.RANDOM, dataset_version_id: Optional[str] = None, use_case_id: Optional[str] = None, datetime_partition_column: Optional[str] = None, ) -> Project: start_time = time.time() print("Starting dynamic dataset pipeline...") dataset_definition = create_dataset_definition( dataset_id, dataset_version_id, name="Dynamic Dataset", credential_name=credential_name ) print("Creating samples...") _ = create_sample_row_based(dataset_definition) print("Creating chunk definition...") chunk_definition = create_chunk_definition( dataset_definition, sorting_columns, target_column, datetime_partition_column, target_class=target_class, chunking_partition_method=chunking_partition_method, ) print("Creating and starting project...") project = create_project( dataset_id, target_column, chunk_definition, use_case_id, datetime_partition_column ) print(f"Time taken: {time.time() - start_time:.1f}s — Project ID: {project.id}") return project ``` ## Configure and run Edit variables, then run.STRATIFIED needs `TARGET_CLASS`; DATE needs `DATETIME_PARTITION_COLUMN`. ``` # --- edit these --- DATASET_ID = "your-dataset-id" TARGET_COLUMN = "your_target" SORTING_COLUMNS = ["TIME", "SERIES"] # order_by columns for chunking CREDENTIAL_NAME = "your-credential-name" # must exist in DataRobot DATASET_VERSION_ID = None USE_CASE_ID = None TARGET_CLASS = None DATETIME_PARTITION_COLUMN = None CHUNKING_PARTITION_METHOD = "RANDOM" # RANDOM | STRATIFIED | DATE method = ChunkingPartitionMethod[CHUNKING_PARTITION_METHOD.upper()] if method == ChunkingPartitionMethod.DATE and not DATETIME_PARTITION_COLUMN: raise ValueError("DATETIME_PARTITION_COLUMN required for DATE") if method == ChunkingPartitionMethod.STRATIFIED and not TARGET_CLASS: raise ValueError("TARGET_CLASS required for STRATIFIED") project = run_dynamic_dataset_pipeline( dataset_id=DATASET_ID, target_column=TARGET_COLUMN, sorting_columns=SORTING_COLUMNS, credential_name=CREDENTIAL_NAME, target_class=TARGET_CLASS, chunking_partition_method=method, dataset_version_id=DATASET_VERSION_ID, use_case_id=USE_CASE_ID, datetime_partition_column=DATETIME_PARTITION_COLUMN, ) print(f"Project ID: {project.id}") ``` ## Optional: percentage-based sample After `create_dataset_definition`, you can call `create_sample_percentage_based(dataset_definition)` instead of row-based inside a custom flow. --- # Incremental learning code examples URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/incremental-learning/index.html > Review comprehensive workflows, notebooks, and tutorials that help you find complete examples of incremental learning. The API user guide includes overviews and workflows for DataRobot's Python client that outline complete examples of incremental learning. | Topic | Describes... | | --- | --- | | Chunking Service static dataset example | An incremental learning sample workflow with a static dataset. | | Chunking Service dynamic dataset example | An incremental learning sample workflow with a dynamic dataset. | --- # Python code examples URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/index.html > Review comprehensive workflows, notebooks, and tutorials that help you find complete examples of common data science and machine learning workflows. The API user guide includes overviews and workflows for DataRobot's Python client that outline complete examples of common data science and machine learning workflows. Be sure to review the [Developer quickstart](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html) before using the notebooks below. | Topic | Describes... | | --- | --- | | Modeling and insights examples | Code examples that focus on the model building process and the insights you can generate for models. | | Prediction code examples | Code examples that outline various prediction methods. | | Feature selection examples | Notebooks that outline Feature Importance Rank Ensembling (FIRE) and advanced feature selection with Python. | | Pulumi code examples | How to perform common DataRobot tasks by using Pulumi. | --- # Generate advanced model insights URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/adv-insights.html This notebook explores model insights available for DataRobot's Python client. You can download this notebook using the icon in the top right of the page. Download the dataset used in this notebook [here](https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/10k_diabetes.csv). ## Setup ### Import libraries Import the libraries in the following snippet. Some of these will assist with the presentation of model insights. ``` %matplotlib inline import datarobot as dr from datarobot.enums import AUTOPILOT_MODE from datarobot.errors import ClientError import matplotlib.pyplot as plt import matplotlib.ticker as mtick import numpy as np import pandas as pd ``` ### Configure DataRobot API authentication Read more about different options for [connecting to DataRobot API from the client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` # If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call # dr.Client(config_path='path-to-drconfig.yaml') ``` ### Import data ``` data_path = "10k_diabetes.csv" df = pd.read_csv(data_path) df.head() ``` | | race | gender | age | weight | admission_type_id | discharge_disposition_id | admission_source_id | time_in_hospital | payer_code | medical_specialty | ... | glipizide_metformin | glimepiride_pioglitazone | metformin_rosiglitazone | metformin_pioglitazone | change | diabetesMed | readmitted | diag_1_desc | diag_2_desc | diag_3_desc | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0 | Caucasian | Female | [50-60) | ? | Elective | Discharged to home | Physician Referral | 1 | CP | Surgery-Neuro | ... | No | No | No | No | No | No | False | Spinal stenosis in cervical region | Spinal stenosis in cervical region | Effusion of joint, site unspecified | | 1 | Caucasian | Female | [20-30) | [50-75) | Urgent | Discharged to home | Physician Referral | 2 | UN | ? | ... | No | No | No | No | No | No | False | First-degree perineal laceration, unspecified ... | Diabetes mellitus of mother, complicating preg... | Sideroblastic anemia | | 2 | Caucasian | Male | [80-90) | ? | Not Available | Discharged/transferred to home with home healt... | NaN | 7 | MC | Family/GeneralPractice | ... | No | No | No | No | No | Yes | True | Pneumococcal pneumonia [Streptococcus pneumoni... | Congestive heart failure, unspecified | Hyperosmolality and/or hypernatremia | | 3 | AfricanAmerican | Female | [50-60) | ? | Emergency | Discharged to home | Transfer from another health care facility | 4 | UN | ? | ... | No | No | No | No | No | Yes | False | Cellulitis and abscess of face | Streptococcus infection in conditions classifi... | Diabetes mellitus without mention of complicat... | | 4 | AfricanAmerican | Female | [50-60) | ? | Emergency | Discharged to home | Emergency Room | 5 | ? | Psychiatry | ... | No | No | No | No | Ch | Yes | False | Bipolar I disorder, single manic episode, unsp... | Diabetes mellitus without mention of complicat... | Depressive type psychosis | 5 rows × 51 columns ## Modeling ### Create a project Create a new project using the `10K_diabetes.csv` dataset, containing the target `readmitted` (framed as a binary classification problem). ``` project = dr.Project.create(data_path, project_name="10K Diabetes Adv Modeling") print("Project ID: {}".format(project.id)) ``` ``` Project ID: 635c2f3ba5c95929466f3cb7 ``` ### Start Autopilot ``` project.analyze_and_model( target="readmitted", worker_count=-1, ) ``` ``` Project(10K Diabetes Adv Modeling) ``` ``` project.wait_for_autopilot() ``` ``` In progress: 14, queued: 0 (waited: 0s) In progress: 14, queued: 0 (waited: 1s) In progress: 14, queued: 0 (waited: 1s) In progress: 14, queued: 0 (waited: 2s) In progress: 14, queued: 0 (waited: 3s) In progress: 14, queued: 0 (waited: 5s) In progress: 11, queued: 0 (waited: 9s) In progress: 10, queued: 0 (waited: 16s) In progress: 6, queued: 0 (waited: 29s) In progress: 1, queued: 0 (waited: 49s) In progress: 7, queued: 0 (waited: 70s) In progress: 1, queued: 0 (waited: 90s) In progress: 16, queued: 0 (waited: 111s) In progress: 10, queued: 0 (waited: 131s) In progress: 6, queued: 0 (waited: 151s) In progress: 2, queued: 0 (waited: 172s) In progress: 0, queued: 0 (waited: 192s) In progress: 5, queued: 0 (waited: 213s) In progress: 1, queued: 0 (waited: 233s) In progress: 4, queued: 0 (waited: 253s) In progress: 1, queued: 0 (waited: 274s) In progress: 1, queued: 0 (waited: 294s) In progress: 0, queued: 0 (waited: 315s) In progress: 0, queued: 0 (waited: 335s) ``` ### Get the top-performing model ``` model = project.get_top_model() ``` ## Model insights The following sections outline the various model insights DataRobot has to offer. Before proceeding, set color constants to replicate the visual style of DataRobot. ``` dr_dark_blue = "#08233F" dr_blue = "#1F77B4" dr_orange = "#FF7F0E" dr_red = "#BE3C28" ``` ### Feature Impact [Feature Impact](https://docs.datarobot.com/en/docs/modeling/analyze-models/understand/feature-impact.html) measures how important a feature is in the context of a model. It measures how much the accuracy of a model would decrease if that feature was removed. Feature Impact is available for all model types and works by altering input data and observing the effect on a model’s score. It is an on-demand feature, meaning that you must initiate a calculation to see the results. Once DataRobot computes the feature impact for a model, that information is saved with the project. ``` feature_impacts = model.get_or_request_feature_impact() ``` ``` # Formats the ticks from a float into a percent percent_tick_fmt = mtick.PercentFormatter(xmax=1.0) impact_df = pd.DataFrame(feature_impacts) impact_df.sort_values(by="impactNormalized", ascending=True, inplace=True) # Positive values are blue, negative are red bar_colors = impact_df.impactNormalized.apply(lambda x: dr_red if x < 0 else dr_blue) ax = impact_df.plot.barh( x="featureName", y="impactNormalized", legend=False, color=bar_colors, figsize=(10, 14) ) ax.xaxis.set_major_formatter(percent_tick_fmt) ax.xaxis.set_tick_params(labeltop=True) ax.xaxis.grid(True, alpha=0.2) ax.set_facecolor(dr_dark_blue) plt.ylabel("") plt.xlabel("Effect") plt.xlim((None, 1)) # Allow for negative impact plt.title("Feature Impact", y=1.04) ``` ``` Text(0.5, 1.04, 'Feature Impact') ``` ### Histogram The [histogram](https://docs.datarobot.com/en/docs/data/analyze-data/histogram.html#histogram-chart) chart "buckets" numeric feature values into equal-sized ranges to show frequency distribution of the variable—the target observation (Y-axis) plotted against the frequency of the value (X-axis). The height of each bar represents the number of rows with values in that range. The helper function below, `matplotlib_pair_histogram`, is used to draw histograms paired with the project's target feature ( `readamitted` in this case). The function includes an orange line in every histogram bin that indicates the average target feature value for rows in that bin. ``` def matplotlib_pair_histogram(labels, counts, target_avgs, bin_count, ax1, feature): # Rotate categorical labels if feature.feature_type in ["Categorical", "Text"]: ax1.tick_params(axis="x", rotation=45) ax1.set_ylabel(feature.name, color=dr_blue) ax1.bar(labels, counts, color=dr_blue) # Instantiate a second axes that shares the same x-axis ax2 = ax1.twinx() ax2.set_ylabel(target_feature_name, color=dr_orange) ax2.plot(labels, target_avgs, marker="o", lw=1, color=dr_orange) ax1.set_facecolor(dr_dark_blue) title = "Histogram for {} ({} bins)".format(feature.name, bin_count) ax1.set_title(title) ``` The next function, `draw_feature_histogram`, gets the histogram data and draws the histogram using the previous helper function. Before using the function, you can retrieve downsampled histogram data using the snippet below: ``` feature = dr.Feature.get(project.id, "num_lab_procedures") feature.get_histogram(bin_limit=6).plot ``` ``` [{'label': '1.0', 'count': 755, 'target': 0.36026490066225164}, {'label': '14.5', 'count': 895, 'target': 0.3240223463687151}, {'label': '28.0', 'count': 1875, 'target': 0.3744}, {'label': '41.5', 'count': 2159, 'target': 0.38490041685965726}, {'label': '55.0', 'count': 1603, 'target': 0.45414847161572053}, {'label': '68.5', 'count': 557, 'target': 0.5080789946140036}] ``` For best accuracy, DataRobot recommends using divisors of 60 for `bin_limit`. Any value less than or equal to 60 can be used. The `target` values are project target input average values for a given bin. ``` def draw_feature_histogram(feature_name, bin_count): feature = dr.Feature.get(project.id, feature_name) # Retrieve downsampled histogram data from server # based on desired bin count data = feature.get_histogram(bin_count).plot labels = [row["label"] for row in data] counts = [row["count"] for row in data] target_averages = [row["target"] for row in data] f, axarr = plt.subplots() f.set_size_inches((10, 4)) matplotlib_pair_histogram(labels, counts, target_averages, bin_count, axarr, feature) ``` Lastly, specify the feature name, target, and desired bin count to create the feature histograms. You can view an example below: ``` feature_name = "num_lab_procedures" target_feature_name = "readmitted" draw_feature_histogram("num_lab_procedures", 12) ``` Categorical and other feature types are supported as well: ``` feature_name = "medical_specialty" draw_feature_histogram("medical_specialty", 10) ``` ### Lift Chart A [lift chart](https://docs.datarobot.com/en/docs/modeling/analyze-models/evaluate/lift-chart.html#lift-chart) shows you how close model predictions are to the actual values of the target in the training data. The lift chart data includes the average predicted value and the average actual values of the target, sorted by the prediction values in ascending order and split into up to 60 bins. ``` lc = model.get_lift_chart("validation") lc ``` ``` LiftChart(validation) ``` ``` bins_df = pd.DataFrame(lc.bins) bins_df.head() ``` | | actual | predicted | bin_weight | | --- | --- | --- | --- | | 0 | 0.000000 | 0.076155 | 27.0 | | 1 | 0.148148 | 0.117283 | 27.0 | | 2 | 0.076923 | 0.146873 | 26.0 | | 3 | 0.148148 | 0.168664 | 27.0 | | 4 | 0.111111 | 0.182873 | 27.0 | The following snippet defines functions for rebinning and plotting. ``` def rebin_df(raw_df, number_of_bins): cols = ["bin", "actual_mean", "predicted_mean", "bin_weight"] new_df = pd.DataFrame(columns=cols) current_prediction_total = 0 current_actual_total = 0 current_row_total = 0 x_index = 1 bin_size = 60 / number_of_bins for rowId, data in raw_df.iterrows(): current_prediction_total += data["predicted"] * data["bin_weight"] current_actual_total += data["actual"] * data["bin_weight"] current_row_total += data["bin_weight"] if (rowId + 1) % bin_size == 0: x_index += 1 bin_properties = { "bin": ((round(rowId + 1) / 60) * number_of_bins), "actual_mean": current_actual_total / current_row_total, "predicted_mean": current_prediction_total / current_row_total, "bin_weight": current_row_total, } new_df = new_df.append(bin_properties, ignore_index=True) current_prediction_total = 0 current_actual_total = 0 current_row_total = 0 return new_df def matplotlib_lift(bins_df, bin_count, ax): grouped = rebin_df(bins_df, bin_count) ax.plot(range(1, len(grouped) + 1), grouped["predicted_mean"], marker="+", lw=1, color=dr_blue) ax.plot(range(1, len(grouped) + 1), grouped["actual_mean"], marker="*", lw=1, color=dr_orange) ax.set_xlim([0, len(grouped) + 1]) ax.set_facecolor(dr_dark_blue) ax.legend(loc="best") ax.set_title("Lift chart {} bins".format(bin_count)) ax.set_xlabel("Sorted Prediction") ax.set_ylabel("Value") return grouped ``` Note that while this method works for any bin count less then 60, the most reliable result can be achieved when the number of bins is a divisor of 60. Additionally, this visualization method does not work for a bin count greater than 60 because DataRobot does not provide enough information for a larger resolution. ``` bin_counts = [10, 12, 15, 20, 30, 60] f, axarr = plt.subplots(len(bin_counts)) f.set_size_inches((8, 4 * len(bin_counts))) rebinned_dfs = [] for i in range(len(bin_counts)): rebinned_dfs.append(matplotlib_lift(bins_df, bin_counts[i], axarr[i])) plt.tight_layout() ``` ``` No handles with labels found to put in legend. No handles with labels found to put in legend. No handles with labels found to put in legend. No handles with labels found to put in legend. No handles with labels found to put in legend. No handles with labels found to put in legend. ``` ### Rebinned Data You can retrieve raw re-binned data for use in third-party tools or for additional evaluation ``` for rebinned in rebinned_dfs: print("Number of bins: {}".format(len(rebinned.index))) print(rebinned) ``` ``` Number of bins: 10 bin actual_mean predicted_mean bin_weight 0 1.0 0.13750 0.159916 160.0 1 2.0 0.17500 0.233332 160.0 2 3.0 0.27500 0.276564 160.0 3 4.0 0.28750 0.317841 160.0 4 5.0 0.41250 0.355449 160.0 5 6.0 0.33750 0.394435 160.0 6 7.0 0.49375 0.436481 160.0 7 8.0 0.54375 0.490176 160.0 8 9.0 0.62500 0.559797 160.0 9 10.0 0.68125 0.697142 160.0 Number of bins: 12 bin actual_mean predicted_mean bin_weight 0 1.0 0.134328 0.151886 134.0 1 2.0 0.180451 0.220872 133.0 2 3.0 0.210526 0.259316 133.0 3 4.0 0.313433 0.294237 134.0 4 5.0 0.293233 0.327699 133.0 5 6.0 0.413534 0.358398 133.0 6 7.0 0.353383 0.390993 133.0 7 8.0 0.440299 0.425269 134.0 8 9.0 0.556391 0.465567 133.0 9 10.0 0.556391 0.515761 133.0 10 11.0 0.609023 0.583067 133.0 11 12.0 0.701493 0.712181 134.0 Number of bins: 15 bin actual_mean predicted_mean bin_weight 0 1.0 0.084112 0.142650 107.0 1 2.0 0.177570 0.206029 107.0 2 3.0 0.207547 0.241613 106.0 3 4.0 0.271028 0.269917 107.0 4 5.0 0.308411 0.297614 107.0 5 6.0 0.264151 0.324330 106.0 6 7.0 0.420561 0.349149 107.0 7 8.0 0.367925 0.374717 106.0 8 9.0 0.336449 0.400959 107.0 9 10.0 0.485981 0.428771 107.0 10 11.0 0.518868 0.460771 106.0 11 12.0 0.551402 0.500419 107.0 12 13.0 0.603774 0.543591 106.0 13 14.0 0.635514 0.610431 107.0 14 15.0 0.719626 0.730594 107.0 Number of bins: 20 bin actual_mean predicted_mean bin_weight 0 1.0 0.0500 0.132253 80.0 1 2.0 0.2250 0.187579 80.0 2 3.0 0.1750 0.221244 80.0 3 4.0 0.1750 0.245419 80.0 4 5.0 0.2500 0.266226 80.0 5 6.0 0.3000 0.286902 80.0 6 7.0 0.3375 0.308215 80.0 7 8.0 0.2375 0.327466 80.0 8 9.0 0.4250 0.346325 80.0 9 10.0 0.4000 0.364573 80.0 10 11.0 0.3625 0.384512 80.0 11 12.0 0.3125 0.404358 80.0 12 13.0 0.4875 0.425218 80.0 13 14.0 0.5000 0.447743 80.0 14 15.0 0.5875 0.474525 80.0 15 16.0 0.5000 0.505826 80.0 16 17.0 0.6250 0.536862 80.0 17 18.0 0.6250 0.582731 80.0 18 19.0 0.6250 0.640753 80.0 19 20.0 0.7375 0.753532 80.0 Number of bins: 30 bin actual_mean predicted_mean bin_weight 0 1.0 0.037037 0.117812 54.0 1 2.0 0.132075 0.167957 53.0 2 3.0 0.245283 0.194772 53.0 3 4.0 0.111111 0.217077 54.0 4 5.0 0.264151 0.234340 53.0 5 6.0 0.150943 0.248885 53.0 6 7.0 0.259259 0.262677 54.0 7 8.0 0.283019 0.277293 53.0 8 9.0 0.283019 0.289984 53.0 9 10.0 0.333333 0.305103 54.0 10 11.0 0.226415 0.317688 53.0 11 12.0 0.301887 0.330972 53.0 12 13.0 0.415094 0.343545 53.0 13 14.0 0.425926 0.354649 54.0 14 15.0 0.396226 0.368169 53.0 15 16.0 0.339623 0.381265 53.0 16 17.0 0.314815 0.394318 54.0 17 18.0 0.358491 0.407725 53.0 18 19.0 0.452830 0.422268 53.0 19 20.0 0.518519 0.435153 54.0 20 21.0 0.509434 0.452046 53.0 21 22.0 0.528302 0.469495 53.0 22 23.0 0.641509 0.489711 53.0 23 24.0 0.462963 0.510929 54.0 24 25.0 0.641509 0.530756 53.0 25 26.0 0.566038 0.556426 53.0 26 27.0 0.666667 0.591609 54.0 27 28.0 0.603774 0.629608 53.0 28 29.0 0.698113 0.676879 53.0 29 30.0 0.740741 0.783314 54.0 Number of bins: 60 bin actual_mean predicted_mean bin_weight 0 1.0 0.037037 0.097886 27.0 1 2.0 0.037037 0.137739 27.0 2 3.0 0.076923 0.162243 26.0 3 4.0 0.185185 0.173459 27.0 4 5.0 0.333333 0.188488 27.0 5 6.0 0.153846 0.201298 26.0 6 7.0 0.148148 0.213213 27.0 7 8.0 0.074074 0.220940 27.0 8 9.0 0.307692 0.229899 26.0 9 10.0 0.222222 0.238617 27.0 10 11.0 0.111111 0.245402 27.0 11 12.0 0.192308 0.252501 26.0 12 13.0 0.259259 0.258865 27.0 13 14.0 0.259259 0.266489 27.0 14 15.0 0.230769 0.273597 26.0 15 16.0 0.333333 0.280852 27.0 16 17.0 0.333333 0.286678 27.0 17 18.0 0.230769 0.293418 26.0 18 19.0 0.259259 0.301547 27.0 19 20.0 0.407407 0.308660 27.0 20 21.0 0.346154 0.314679 26.0 21 22.0 0.111111 0.320585 27.0 22 23.0 0.307692 0.327277 26.0 23 24.0 0.296296 0.334530 27.0 24 25.0 0.407407 0.340926 27.0 25 26.0 0.423077 0.346264 26.0 26 27.0 0.444444 0.351782 27.0 27 28.0 0.407407 0.357515 27.0 28 29.0 0.461538 0.364479 26.0 29 30.0 0.333333 0.371723 27.0 30 31.0 0.407407 0.378530 27.0 31 32.0 0.269231 0.384105 26.0 32 33.0 0.407407 0.390886 27.0 33 34.0 0.222222 0.397751 27.0 34 35.0 0.461538 0.403918 26.0 35 36.0 0.259259 0.411391 27.0 36 37.0 0.481481 0.419135 27.0 37 38.0 0.423077 0.425521 26.0 38 39.0 0.555556 0.431010 27.0 39 40.0 0.481481 0.439296 27.0 40 41.0 0.538462 0.448068 26.0 41 42.0 0.481481 0.455876 27.0 42 43.0 0.576923 0.464854 26.0 43 44.0 0.481481 0.473965 27.0 44 45.0 0.703704 0.484397 27.0 45 46.0 0.576923 0.495230 26.0 46 47.0 0.444444 0.505163 27.0 47 48.0 0.481481 0.516694 27.0 48 49.0 0.615385 0.526190 26.0 49 50.0 0.666667 0.535152 27.0 50 51.0 0.592593 0.548849 27.0 51 52.0 0.538462 0.564293 26.0 52 53.0 0.555556 0.581138 27.0 53 54.0 0.777778 0.602079 27.0 54 55.0 0.576923 0.619633 26.0 55 56.0 0.629630 0.639213 27.0 56 57.0 0.666667 0.662629 27.0 57 58.0 0.730769 0.691678 26.0 58 59.0 0.666667 0.740971 27.0 59 60.0 0.814815 0.825658 27.0 ``` ### ROC Curve The receiver operating characteristic curve, or [ROC curve](https://docs.datarobot.com/en/docs/modeling/analyze-models/evaluate/roc-curve-tab/roc-curve.html#roc-curve), is a graphical plot that illustrates the performance of a binary classifier system as its discrimination threshold is varied. The curve is created by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. ``` roc = model.get_roc_curve("validation") roc ``` ``` RocCurve(validation) ``` ``` df = pd.DataFrame(roc.roc_points) df.head() ``` | | accuracy | f1_score | false_negative_score | true_negative_score | true_positive_score | false_positive_score | true_negative_rate | false_positive_rate | true_positive_rate | matthews_correlation_coefficient | positive_predictive_value | negative_predictive_value | threshold | fraction_predicted_as_positive | fraction_predicted_as_negative | lift_positive | lift_negative | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0 | 0.603125 | 0.000000 | 635 | 965 | 0 | 0 | 1.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000 | 0.603125 | 1.000000 | 0.000000 | 1.000000 | 0.000000 | 1.000000 | | 1 | 0.603750 | 0.003145 | 634 | 965 | 1 | 0 | 1.000000 | 0.000000 | 0.001575 | 0.030829 | 1.000 | 0.603502 | 0.878030 | 0.000625 | 0.999375 | 2.519685 | 1.000625 | | 2 | 0.605625 | 0.012520 | 631 | 965 | 4 | 0 | 1.000000 | 0.000000 | 0.006299 | 0.061715 | 1.000 | 0.604637 | 0.849079 | 0.002500 | 0.997500 | 2.519685 | 1.002506 | | 3 | 0.606875 | 0.021773 | 628 | 964 | 7 | 1 | 0.998964 | 0.001036 | 0.011024 | 0.069276 | 0.875 | 0.605528 | 0.788308 | 0.005000 | 0.995000 | 2.204724 | 1.003984 | | 4 | 0.608125 | 0.036866 | 623 | 961 | 12 | 4 | 0.995855 | 0.004145 | 0.018898 | 0.072540 | 0.750 | 0.606692 | 0.764327 | 0.010000 | 0.990000 | 1.889764 | 1.005914 | #### Threshold operations You can get the recommended threshold value with a maximal F1 score using the `RocCurve.get_best_f1_threshold` method. That is the same threshold that is preselected in the DataRobot application when you open the ROC curve tab. ``` threshold = roc.get_best_f1_threshold() threshold ``` ``` 0.3204921984454553 ``` To estimate metrics for different threshold values, pass it to the `RocCurve.estimate_threshold` method. This produces the same results as updating the threshold in the ROC Curve tab. ``` metrics = roc.estimate_threshold(threshold) metrics ``` ``` {'accuracy': 0.606875, 'f1_score': 0.6231276213301378, 'false_negative_score': 115, 'true_negative_score': 451, 'true_positive_score': 520, 'false_positive_score': 514, 'true_negative_rate': 0.46735751295336786, 'false_positive_rate': 0.5326424870466321, 'true_positive_rate': 0.8188976377952756, 'matthews_correlation_coefficient': 0.2929107725430276, 'positive_predictive_value': 0.5029013539651838, 'negative_predictive_value': 0.7968197879858657, 'threshold': 0.3204921984454553, 'fraction_predicted_as_positive': 0.64625, 'fraction_predicted_as_negative': 0.35375, 'lift_positive': 1.2671530178650299, 'lift_negative': 1.3211519800801919} ``` Use the following snippet to plot the ROC curve. ``` roc_df = df dr_roc_green = "#03c75f" white = "#ffffff" dr_purple = "#65147D" dr_dense_green = "#018f4f" threshold = roc.get_best_f1_threshold() fig = plt.figure(figsize=(8, 8)) axes = fig.add_subplot(1, 1, 1, facecolor=dr_dark_blue) plt.scatter(roc_df.false_positive_rate, roc_df.true_positive_rate, color=dr_roc_green) plt.plot(roc_df.false_positive_rate, roc_df.true_positive_rate, color=dr_roc_green) plt.plot([0, 1], [0, 1], color=white, alpha=0.25) plt.title("ROC curve") plt.xlabel("False Positive Rate") plt.xlim([0, 1]) plt.ylabel("True Positive Rate") plt.ylim([0, 1]) ``` ``` (0.0, 1.0) ``` ### Confusion matrix Using keys from the retrieved metrics, you can build a confusion matrix for the selected threshold. ``` roc_df = pd.DataFrame( { "Predicted Negative": [ metrics["true_negative_score"], metrics["false_negative_score"], metrics["true_negative_score"] + metrics["false_negative_score"], ], "Predicted Positive": [ metrics["false_positive_score"], metrics["true_positive_score"], metrics["true_positive_score"] + metrics["false_positive_score"], ], "Total": [ len(roc.negative_class_predictions), len(roc.positive_class_predictions), len(roc.negative_class_predictions) + len(roc.positive_class_predictions), ], } ) roc_df.index = pd.MultiIndex.from_tuples([("Actual", "-"), ("Actual", "+"), ("Total", "")]) roc_df.columns = pd.MultiIndex.from_tuples([("Predicted", "-"), ("Predicted", "+"), ("Total", "")]) roc_df.style.set_properties(**{"text-align": "right"}) roc_df ``` | | | Predicted | Total | | --- | --- | --- | --- | | Actual | - | 511 | 454 | | + | 144 | 491 | 638 | | Total | | 655 | 945 | ### Prediction distribution plot You can use various methods to plot prediction distribution. The method used depends on what packages you have installed. Three different visualizations are outlined below. #### Seaborn ``` import seaborn as sns sns.set_style("whitegrid", {"axes.grid": False}) fig = plt.figure(figsize=(8, 8)) axes = fig.add_subplot(1, 1, 1, facecolor=dr_dark_blue) shared_params = {"shade": True, "clip": (0, 1), "bw": 0.2} sns.kdeplot(np.array(roc.negative_class_predictions), color=dr_purple, **shared_params) sns.kdeplot(np.array(roc.positive_class_predictions), color=dr_dense_green, **shared_params) plt.title("Prediction Distribution") plt.xlabel("Probability of Event") plt.xlim([0, 1]) plt.ylabel("Probability Density") ``` ``` Text(0,0.5,'Probability Density') ``` #### SciPy ``` from scipy.stats import gaussian_kde fig = plt.figure(figsize=(8, 8)) axes = fig.add_subplot(1, 1, 1, facecolor=dr_dark_blue) xs = np.linspace(0, 1, 100) density_neg = gaussian_kde(roc.negative_class_predictions, bw_method=0.2) plt.plot(xs, density_neg(xs), color=dr_purple) plt.fill_between(xs, 0, density_neg(xs), color=dr_purple, alpha=0.3) density_pos = gaussian_kde(roc.positive_class_predictions, bw_method=0.2) plt.plot(xs, density_pos(xs), color=dr_dense_green) plt.fill_between(xs, 0, density_pos(xs), color=dr_dense_green, alpha=0.3) plt.title("Prediction Distribution") plt.xlabel("Probability of Event") plt.xlim([0, 1]) plt.ylabel("Probability Density") ``` ``` Text(0,0.5,'Probability Density') ``` #### Scikit-learn The scikit-learn method is most consistent with how DataRobot displays this plot in the application. This is because scikit-learn supports additional kernel options and you can configure the same kernel used in the application (an epanichkov kernel with size 0.05). The other examples above use a gaussian kernel, so they may slightly differ from the plot in the DataRobot application. ``` from sklearn.neighbors import KernelDensity fig = plt.figure(figsize=(8, 8)) axes = fig.add_subplot(1, 1, 1, facecolor=dr_dark_blue) xs = np.linspace(0, 1, 100) X_neg = np.asarray(roc.negative_class_predictions)[:, np.newaxis] density_neg = KernelDensity(bandwidth=0.05, kernel="epanechnikov").fit(X_neg) plt.plot(xs, np.exp(density_neg.score_samples(xs[:, np.newaxis])), color=dr_purple) plt.fill_between( xs, 0, np.exp(density_neg.score_samples(xs[:, np.newaxis])), color=dr_purple, alpha=0.3 ) X_pos = np.asarray(roc.positive_class_predictions)[:, np.newaxis] density_pos = KernelDensity(bandwidth=0.05, kernel="epanechnikov").fit(X_pos) plt.plot(xs, np.exp(density_pos.score_samples(xs[:, np.newaxis])), color=dr_dense_green) plt.fill_between( xs, 0, np.exp(density_pos.score_samples(xs[:, np.newaxis])), color=dr_dense_green, alpha=0.3 ) plt.title("Prediction Distribution") plt.xlabel("Probability of Event") plt.xlim([0, 1]) plt.ylabel("Probability Density") ``` ``` Text(0,0.5,'Probability Density') ``` ### Word Cloud Text variables often contain words that are highly indicative of the response. The [Word Cloud](https://docs.datarobot.com/en/docs/modeling/analyze-models/understand/word-cloud.html) insight displays the most relevant words and short phrases in word cloud format. This example shows you how to obtain word cloud data and visualize it in similar to how the insight is displayed in the DataRobot application. The example uses `colour` and `wordcloud` packages. First, create a color palette similar to DataRobot's style. ``` from colour import Color import wordcloud ``` ``` colors = [Color("#2458EB")] colors.extend(list(Color("#2458EB").range_to(Color("#31E7FE"), 81))[1:]) colors.extend(list(Color("#31E7FE").range_to(Color("#8da0a2"), 21))[1:]) colors.extend(list(Color("#a18f8c").range_to(Color("#ffad9e"), 21))[1:]) colors.extend(list(Color("#ffad9e").range_to(Color("#d80909"), 81))[1:]) webcolors = [c.get_web() for c in colors] ``` The variable `webcolors` now contains 201 ([-1, 1] interval with step 0.01) colors that will be used in the word cloud. Next, configure the palette. ``` from matplotlib.colors import LinearSegmentedColormap dr_cmap = LinearSegmentedColormap.from_list("DataRobot", webcolors, N=len(colors)) x = np.arange(-1, 1.01, 0.01) y = np.arange(0, 40, 1) X = np.meshgrid(x, y)[0] plt.xticks( [0, 20, 40, 60, 80, 100, 120, 140, 160, 180, 200], ["-1", "-0.8", "-0.6", "-0.4", "-0.2", "0", "0.2", "0.4", "0.6", "0.8", "1"], ) plt.yticks([], []) im = plt.imshow(X, interpolation="nearest", origin="lower", cmap=dr_cmap) ``` Now you can pick a model that provides a word cloud in the DataRobot. Any "Auto-Tuned Word N-Gram Text Modeler" model will work. ``` models = project.get_models() ``` ``` model_with_word_cloud = None for model in models: try: model.get_word_cloud() model_with_word_cloud = model break except ClientError as e: if e.json["message"] and "No word cloud data" in e.json["message"]: pass else: raise model_with_word_cloud ``` ``` Model(u'Auto-Tuned Word N-Gram Text Modeler using token occurrences - diag_1_desc') ``` ``` wc = model_with_word_cloud.get_word_cloud(exclude_stop_words=True) ``` ``` def word_cloud_plot(wc, font_path=None): # Stopwords usually dominate any word cloud, so we will filter them out dict_freq = { wc_word["ngram"]: wc_word["frequency"] for wc_word in wc.ngrams if not wc_word["is_stopword"] } dict_coef = {wc_word["ngram"]: wc_word["coefficient"] for wc_word in wc.ngrams} def color_func(*args, **kwargs): word = args[0] palette_index = int(round(dict_coef[word] * 100)) + 100 r, g, b = colors[palette_index].get_rgb() return "rgb({:.0f}, {:.0f}, {:.0f})".format(int(r * 255), int(g * 255), int(b * 255)) wc_image = wordcloud.WordCloud( stopwords=set(), width=1024, height=1024, relative_scaling=0.5, prefer_horizontal=1, color_func=color_func, background_color=(0, 10, 29), font_path=font_path, ).fit_words(dict_freq) plt.imshow(wc_image, interpolation="bilinear") plt.axis("off") ``` ``` word_cloud_plot(wc) ``` You can use the word cloud to get information about the most frequent and most important (highest absolute coefficient value) ngrams in your text. ``` wc.most_frequent(5) ``` ``` [{'coefficient': 0.6229774184805059, 'count': 534, 'frequency': 0.21876280213027446, 'is_stopword': False, 'ngram': u'failure'}, {'coefficient': 0.5680375262833832, 'count': 524, 'frequency': 0.21466612044244163, 'is_stopword': False, 'ngram': u'atherosclerosis'}, {'coefficient': 0.37932405511744804, 'count': 505, 'frequency': 0.2068824252355592, 'is_stopword': False, 'ngram': u'infarction'}, {'coefficient': 0.4689734305695615, 'count': 453, 'frequency': 0.18557968045882836, 'is_stopword': False, 'ngram': u'heart'}, {'coefficient': 0.7444542252245913, 'count': 452, 'frequency': 0.18517001229004507, 'is_stopword': False, 'ngram': u'heart failure'}] ``` ``` wc.most_important(5) ``` ``` [{'coefficient': -0.875917913896919, 'count': 38, 'frequency': 0.015567390413764851, 'is_stopword': False, 'ngram': u'obesity unspecified'}, {'coefficient': -0.8655105382141891, 'count': 38, 'frequency': 0.015567390413764851, 'is_stopword': False, 'ngram': u'obesity'}, {'coefficient': 0.8329465952065771, 'count': 9, 'frequency': 0.0036870135190495697, 'is_stopword': False, 'ngram': u'nephroptosis'}, {'coefficient': 0.7444542252245913, 'count': 452, 'frequency': 0.18517001229004507, 'is_stopword': False, 'ngram': u'heart failure'}, {'coefficient': 0.7029270716899754, 'count': 76, 'frequency': 0.031134780827529702, 'is_stopword': False, 'ngram': u'disorders'}] ``` #### Non-ASCII texts The word cloud has full Unicode support, but if you want to visualize it using the code from this notebook you should use the `font_path` parameter that leads to font supporting symbols used in your text. For example, for Japanese text in the model below you should use one of the [CJK fonts](https://en.wikipedia.org/wiki/List_of_CJK_fonts). If you do not have a compatible font, you can download an open-source font [like this one](https://github.com/googlei18n/noto-cjk/raw/master/NotoSansCJKjp-Regular.otf) from [Google's Noto project](https://www.google.com/get/noto/). For this section, download the Japanese-translation version of the "10k_diabetes.csv" dataset [here](https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/jp_10k.csv). ``` jp_project = dr.Project.create("jp_10k.csv", project_name="Japanese 10K") print("Project ID: {}".format(project.id)) ``` ``` Project ID: 5c0008e06523cd0233c49fe4 ``` ``` jp_project.set_target("readmitted_再入院", mode=AUTOPILOT_MODE.QUICK) jp_project.wait_for_autopilot() ``` ``` In progress: 2, queued: 12 (waited: 0s) In progress: 2, queued: 12 (waited: 1s) In progress: 2, queued: 12 (waited: 1s) In progress: 2, queued: 12 (waited: 2s) In progress: 2, queued: 12 (waited: 4s) In progress: 2, queued: 12 (waited: 6s) In progress: 2, queued: 11 (waited: 9s) In progress: 1, queued: 11 (waited: 16s) In progress: 2, queued: 9 (waited: 30s) In progress: 2, queued: 7 (waited: 50s) In progress: 2, queued: 5 (waited: 70s) In progress: 2, queued: 3 (waited: 91s) In progress: 2, queued: 1 (waited: 111s) In progress: 1, queued: 0 (waited: 132s) In progress: 2, queued: 5 (waited: 152s) In progress: 2, queued: 3 (waited: 172s) In progress: 2, queued: 2 (waited: 193s) In progress: 2, queued: 1 (waited: 213s) In progress: 1, queued: 0 (waited: 234s) In progress: 2, queued: 14 (waited: 254s) In progress: 2, queued: 14 (waited: 274s) In progress: 2, queued: 12 (waited: 295s) In progress: 1, queued: 12 (waited: 316s) In progress: 2, queued: 10 (waited: 336s) In progress: 2, queued: 9 (waited: 356s) In progress: 2, queued: 7 (waited: 377s) In progress: 2, queued: 6 (waited: 397s) In progress: 2, queued: 4 (waited: 418s) In progress: 2, queued: 3 (waited: 438s) In progress: 2, queued: 1 (waited: 459s) In progress: 1, queued: 0 (waited: 479s) In progress: 1, queued: 0 (waited: 499s) In progress: 0, queued: 0 (waited: 520s) In progress: 2, queued: 3 (waited: 540s) In progress: 2, queued: 1 (waited: 560s) In progress: 1, queued: 0 (waited: 581s) In progress: 1, queued: 0 (waited: 601s) In progress: 2, queued: 2 (waited: 621s) In progress: 2, queued: 0 (waited: 642s) In progress: 0, queued: 0 (waited: 662s) In progress: 1, queued: 0 (waited: 682s) In progress: 0, queued: 0 (waited: 703s) In progress: 0, queued: 0 (waited: 723s) ``` ``` jp_models = jp_project.get_models() jp_model_with_word_cloud = None for model in jp_models: try: model.get_word_cloud() jp_model_with_word_cloud = model break except ClientError as e: if e.json["message"] and "No word cloud data" in e.json["message"]: pass else: raise jp_model_with_word_cloud ``` ``` Model(u'Auto-Tuned Word N-Gram Text Modeler using token occurrences and tfidf - diag_1_desc_\u8a3a\u65ad1\u8aac\u660e') ``` ``` jp_wc = jp_model_with_word_cloud.get_word_cloud(exclude_stop_words=True) ``` ``` word_cloud_plot(jp_wc, font_path="NotoSansCJKjp-Regular.otf") ``` ### Cumulative gains and lift ROC curve data also contains information necessary for creating cumulative gains and lift charts. Use the fields `fraction_predicted_as_positive` and `fraction_predicted_as_negative` to get X axis and set: - Use true_positive_rate / true_negative_rate as the Y axis for cumulative gains - Use lift_positive / lift_negative as the Y axis for lift. You can use the code for visualizations below, along with baseline/random model (in gray) and ideal (in orange). ``` fig, ((ax_gains_pos, ax_gains_neg), (ax_lift_pos, ax_lift_neg)) = plt.subplots( nrows=2, ncols=2, figsize=(8, 8) ) total_rows = ( df.true_positive_score[0] + df.false_negative_score[0] + df.true_negative_score[0] + df.false_positive_score[0] ) fraction_of_positives = float(df.true_positive_score[0] + df.false_negative_score[0]) / total_rows fraction_of_negatives = 1 - fraction_of_positives # Cumulative gains (positive class) ax_gains_pos.set_facecolor(dr_dark_blue) ax_gains_pos.scatter(df.fraction_predicted_as_positive, df.true_positive_rate, color=dr_roc_green) ax_gains_pos.plot(df.fraction_predicted_as_positive, df.true_positive_rate, color=dr_roc_green) ax_gains_pos.plot([0, 1], [0, 1], color=white, alpha=0.25) ax_gains_pos.plot([0, fraction_of_positives, 1], [0, 1, 1], color=dr_orange) ax_gains_pos.set_title("Cumulative gains (positive class)") ax_gains_pos.set_xlabel("Fraction predicted as positive") ax_gains_pos.set_xlim([0, 1]) ax_gains_pos.set_ylabel("True Positive Rate (Sensitivity)") # Cumulative gains (negative class) ax_gains_neg.set_facecolor(dr_dark_blue) ax_gains_neg.scatter(df.fraction_predicted_as_negative, df.true_negative_rate, color=dr_roc_green) ax_gains_neg.plot(df.fraction_predicted_as_negative, df.true_negative_rate, color=dr_roc_green) ax_gains_neg.plot([0, 1], [0, 1], color=white, alpha=0.25) ax_gains_neg.plot([0, fraction_of_negatives, 1], [0, 1, 1], color=dr_orange) ax_gains_neg.set_title("Cumulative gains (negative class)") ax_gains_neg.set_xlabel("Fraction predicted as negative") ax_gains_neg.set_xlim([0, 1]) ax_gains_neg.set_ylabel("True Negative Rate (Specificity)") # Lift (positive class) ax_lift_pos.set_facecolor(dr_dark_blue) ax_lift_pos.scatter(df.fraction_predicted_as_positive, df.lift_positive, color=dr_roc_green) ax_lift_pos.plot(df.fraction_predicted_as_positive, df.lift_positive, color=dr_roc_green) ax_lift_pos.plot([0, 1], [1, 1], color=white, alpha=0.25) ax_lift_pos.set_title("Lift (positive class)") ax_lift_pos.set_xlabel("Fraction predicted as positive") ax_lift_pos.set_xlim([0, 1]) ax_lift_pos.set_ylabel("Lift") ideal_lift_pos_x = np.arange(0.01, 1.01, 0.01) ideal_lift_pos_y = np.minimum(1 / fraction_of_positives, 1 / ideal_lift_pos_x) ax_lift_pos.plot(ideal_lift_pos_x, ideal_lift_pos_y, color=dr_orange) # Lift (negative class) ax_lift_neg.set_facecolor(dr_dark_blue) ax_lift_neg.scatter(df.fraction_predicted_as_negative, df.lift_negative, color=dr_roc_green) ax_lift_neg.plot(df.fraction_predicted_as_negative, df.lift_negative, color=dr_roc_green) ax_lift_neg.plot([0, 1], [1, 1], color=white, alpha=0.25) # ax_lift_neg.plot([0, fraction_of_positives, 1], [0, 1, 1], color=dr_orange) ax_lift_neg.set_title("Lift (negative class)") ax_lift_neg.set_xlabel("Fraction predicted as negative") ax_lift_neg.set_xlim([0, 1]) ax_lift_neg.set_ylabel("Lift") ideal_lift_neg_x = np.arange(0.01, 1.01, 0.01) ideal_lift_neg_y = np.minimum(1 / fraction_of_negatives, 1 / ideal_lift_neg_x) ax_lift_neg.plot(ideal_lift_neg_x, ideal_lift_neg_y, color=dr_orange) # Adjust spacing for notebook plt.tight_layout() ``` --- # Configure datetime partitioning URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/datetime-v3.html This notebook outlines how to use [datetime partitioning](https://docs.datarobot.com/en/docs/api/reference/public-api/datetime_partitioning.html) with version 3.0 of DataRobot's Python client. When dividing your data for model training and validation, DataRobot randomly chooses a set of rows from the training dataset to assign among different cross-validation folds. This process verifies that you have not overfit your model to the training set and that the model can perform well on new data. However, when your data has an intrinsic time-based component, you must be cautious about [target leakage](https://docs.datarobot.com/en/docs/glossary/index.html#target-leakage). Although DataRobot offers datetime partitioning to guard against target leakage, you should always use your domain expertise to evaluate features prior to modeling. The project in this notebook simulates a project with a time-based component that uses out-of-time validation (OTV) modeling). Note that this is not the same as time series modeling, even though the way DataRobot defines backtests for time series is very similar. ### Requirements - Python version 3.7+. - DataRobot API version 3.0+. - A Pandas dataframe (df) with an indicated target feature. Find reference documentation for DataRobot's Python client [here](https://datarobot-public-api-client.readthedocs-hosted.com). ### Import libraries ``` from datetime import datetime import datarobot as dr ``` ### Connect to DataRobot Read more about different options for [connecting to DataRobot from the client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` # If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call # dr.Client(config_path='path-to-drconfig.yaml') ``` ### Configure a project with a datetime partition When configuring a datetime partition, you must specify the durations in strings using the format for the `dr.helpers.partitioning_methods.construct_duration_string()` [method](https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.27.2/autodoc/api_reference.html#dur-string-helper). ``` spec = dr.DatetimePartitioningSpecification( datetime_partition_column="Date", holdout_start_date=datetime(2017, 1, 2), holdout_duration="P1Y0M0DT0H0M0S", number_of_backtests=2, use_time_series=False, ) # Generate a preview based on your project's data partitioning_preview = dr.DatetimePartitioning.generate(project.id, spec) ``` As of v3.0, `Project.set_datetime_partitioning()` and `Project.list_datetime_partition_spec()` are available as an alternative: ``` # View partitioning settings project.list_datetime_partition_spec() # Uncomment to disable holdout before you begin modeling # project.set_datetime_partitioning(disable_holdout=True) ``` ### Create backtest specifications DataRobot provides further control to specify the validation start date as well as the duration. You can view an example in the following cells. The method below is applicable to both time series and out-of-time validation projects. The snippet provided uses `use_time_series = False` in the `dr.DatetimePartitioningSpecification()` [method](https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.27.2/reference/modeling/spec/datetime_partition.html#setting-up-a-datetime-partitioned-project) to initiate an OTV project. The methods used in the snippet below change the backtest specification for the first and second backtests. DataRobot recommends taking advantage of automated partitioning by setting `use_time_series=True` after you specify the number of backtests. ``` # Set duration of the validation backtests duration_1y = "P1Y0M0DT0H0M0S" duration_0s = "P0Y0M0DT0H0M0S" # Note that the dates are not project-specific; they are example dates spec.backtests = [ dr.BacktestSpecification( 0, gap_duration="P0Y0M0DT0H0M0S", validation_start_date=datetime(2016, 1, 2), validation_duration=duration_1y, ), dr.BacktestSpecification( 1, gap_duration="P0Y0M0DT0H0M0S", validation_start_date=datetime(2015, 1, 2), validation_duration=duration_0s, ), ] # Uncomment if you want more backtests # spec.number_of_backtests = 5 # Use the lines below to initiate the project project = dr.Project.create(sourcedata=df, project_name="Project Name") project.analyze_and_model("target_column", partitioning_method=spec) ``` Once backtests are configured for your project, you can proceed to modeling. See the use case for [predicting CO₂ levels](https://docs.datarobot.com/en/docs/api/guide/common-case/python2/otv-nb.html) as an example. --- # Advanced Feature Selection using Feature Importance Rank Ensembling (FIRE) URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/feat-select/Feature-Importance-Rank-Ensembling.html This notebook shows the benefits of [FIRE](https://www.datarobot.com/blog/using-feature-importance-rank-ensembling-fire-for-advanced-feature-selection/), advanced feature selection that uses median rank aggregation of feature impacts across several models created during a run of Autopilot. ### Requirements Python version >= 3.7.3 DataRobot API version >= 2.22.1. For additional information, reference the [Python package documentation](https://datarobot-public-api-client.readthedocs-hosted.com). This code example uses the MADLEON dataset from [this paper](https://archive.ics.uci.edu/ml/datasets/Madelon). It can also be found [here](https://s3.amazonaws.com/datarobot_public_datasets/madelon_combined_80.csv). ### Import libraries and connect to DataRobot Read more about different options for [connecting to DataRobot from the client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` import datarobot as dr import numpy as np import pandas as pd ``` ``` # If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call # dr.Client(config_path='path-to-drconfig.yaml') ``` ### FIRE feature selection function ``` def feature_importance_rank_ensembling( project, n_models=5, metric=None, by_partition="validation", feature_list_name=None, ratio=0.95, model_search_params=None, use_ranks=True, ): """ Function that implements the logic of Feature Selection using Feature Importance Rank Ensembling and restarts DR autopilot Parameters: ----------- project: DR project object, n_models: int, get top N best models on the leaderboard to compute feature impact on. Default 5 metric: str, DR metric to check performance against. Default None. If Default, it will use DR project defined metric by_partition: str, whether to use 'validation' or 'crossValidation' partition to get the best model on. Default 'validation' feature_list_name: str, name of the feature list to start iterating from. Default None ratio: float, ratio of total feature impact that new feature list will contain. Default 0.95 model_search_params: dict, dictonary of parameters to search the best model. See official DR python api docs. Default None use_ranks: Boolean, True to use median rank aggregation or False to use total impact unnormalized. Default True Returns: ----------- dr.Model object """ models = get_best_models( project, metric=metric, by_partition=by_partition, start_featurelist_name=feature_list_name, model_search_params=model_search_params, ) models = models.values[:n_models] all_impact = pd.DataFrame() print("Request Feature Impact calculations") # First, kick off all Feature Impact requests and let DataRobot handle parallelizing for model in models: try: model.request_feature_impact() except: pass for model in models: # Allow time for DataRobot to compute Feature Impact feature_impact = pd.DataFrame( model.get_or_request_feature_impact(max_wait=60 * 15) ) # 15min # Track model name and ID for auditing purposes feature_impact["model_type"] = model.model_type feature_impact["model_id"] = model.id # By sorting and re-indexing, the new index becomes our 'ranking' feature_impact = feature_impact.sort_values( by="impactUnnormalized", ascending=False ).reset_index(drop=True) feature_impact["rank"] = feature_impact.index.values # Add to the master list of all models' feature ranks all_impact = pd.concat([all_impact, feature_impact], ignore_index=True) # You need to get a threshold number of features to select. # The threshold is based on the cumulative sum of impact all_impact_agg = ( all_impact.groupby("featureName")[["impactNormalized", "impactUnnormalized"]] .sum() .sort_values("impactUnnormalized", ascending=False) .reset_index() ) # Calculate cumulative Feature Impact and take the first features that possess of total impact all_impact_agg["impactCumulative"] = all_impact_agg["impactUnnormalized"].cumsum() total_impact = all_impact_agg["impactCumulative"].max() * ratio tmp_fl = list( set( all_impact_agg[all_impact_agg.impactCumulative <= total_impact][ "featureName" ].values.tolist() ) ) # The number of features to use n_feats = len(tmp_fl) if use_ranks: # Get the top features based on median rank top_ranked_feats = list( all_impact.groupby("featureName") .median() .sort_values("rank") .head(n_feats) .index.values ) else: # Otherwise, get features based just on the total unnormalized feature impact top_ranked_feats = list(all_impact_agg.featureName.values[:n_feats]) # Create a new feature list featurelist = project.create_modeling_featurelist( f"Reduced FL by Median Rank, top{n_feats}", top_ranked_feats ) featurelist_id = featurelist.id # Start Autopilot print("Starting AutoPilot on a reduced feature list") project.start_autopilot( featurelist_id=featurelist_id, prepare_model_for_deployment=True, blend_best_models=False, ) project.wait_for_autopilot() print("... AutoPilot is completed.") # Return the previous best model return models[0] ``` ### Get the best-performing models Avoid using models trained on higher than 3rd stage of Autopilot sample size (80%, 100%). Blender and Frozen models are ignored, so DataRobot selects models trained on 64% percent of the data. ``` def get_best_models( project, metric=None, by_partition="validation", start_featurelist_name=None, model_search_params=None, ): """ Gets pd.Series of DR model objects sorted by performance. Excludes blenders, frozend and on DR Reduced FL Parameters: ----------- project: DR project object metric: str, metric to use for sorting models on lb, if None, default project metric will be used. Default None by_partiton: boolean, whether to use 'validation' or 'crossValidation' partitioning. Default 'validation' start_featurelist_name: str, initial featurelist name to get models on. Default None model_search_params: dict to pass model search params. Default None Returns: ----------- pd.Series of dr.Model objects, not blender, not frozen and not on DR Reduced Feature List """ # A list of metrics that get better as their value increases desc_metric_list = [ "AUC", "Area Under PR Curve", "Gini Norm", "Kolmogorov-Smirnov", "Max MCC", "Rate@Top5%", "Rate@Top10%", "Rate@TopTenth%", "R Squared", "FVE Gamma", "FVE Poisson", "FVE Tweedie", "Accuracy", "Balanced Accuracy", "FVE Multinomial", "FVE Binomial", ] if not metric: metric = project.metric if "Weighted" in metric: desc_metric_list = ["Weighted " + metric for metric in desc_metric_list] asc_flag = False if metric in desc_metric_list else True if project.is_datetime_partitioned: assert by_partition in [ "validation", "backtesting", "holdout", ], "Please specify correct partitioning, in datetime partitioned projects supported options are: 'validation', 'backtesting', 'holdout' " models_df = pd.DataFrame( [ [ model.metrics[metric]["validation"], model.metrics[metric]["backtesting"], model.model_category, model.is_frozen, model.featurelist_name, model, ] for model in project.get_datetime_models() ], columns=[ "validation", "backtesting", "category", "is_frozen", "featurelist_name", "model", ], ).sort_values([by_partition], ascending=asc_flag, na_position="last") else: assert by_partition in [ "validation", "crossValidation", "holdout", ], "Please specify correct partitioning, supported options are: 'validation', 'crossValidation', 'holdout' " models_df = pd.DataFrame( [ [ model.metrics[metric]["crossValidation"], model.metrics[metric]["validation"], model.model_category, model.is_frozen, model.featurelist_name, model, ] for model in project.get_models( with_metric=metric, search_params=model_search_params ) ], columns=[ "crossValidation", "validation", "category", "is_frozen", "featurelist_name", "model", ], ).sort_values([by_partition], ascending=asc_flag, na_position="last") if start_featurelist_name: return models_df.loc[ ( (models_df.category == "model") & (models_df.is_frozen == False) & (models_df.featurelist_name == start_featurelist_name) ), "model", ] else: return models_df.loc[ ( (models_df.category == "model") & (models_df.is_frozen == False) & (models_df.featurelist_name.str.contains("DR Reduced Features M") == False) ), "model", ] ``` ### Primary FIRE function This function automatically executes the FIRE feature selection algorithm on the top N models. Once the reduced feature list is created, DataRobot re-runs Autopilot and waits until it completes. DataRobot then automatically sorts the models based on the project metric, computes Feature Impact, and iterates over again. If the new feature list produces a model that ranks lower based on a metric, it will expend one "life". The algorithm will stop performing feature selection when no lives are available (you start with 3). ``` def main_feature_selection( project_id, start_featurelist_name=None, lifes=2, top_n_models=5, partition="validation", main_scoring_metric=None, initial_impact_reduction_ratio=0.95, best_model_search_params=None, use_ranks=True, ): """ Main function. Meant to get the optimal shortest feature list by repeating the feature selection process until stop criteria is met. Currently supports Binary, Regression, Multiclass, Datetime partitioned (OTV), and AutoTS DataRobot projects. Example usage: >> import datarobot as dr >> dr.Client(config_path='PATH_TO_DR_CONFIG/drconfig.yaml') TIP: set best_model_search_params = {'sample_pct__lte': 65} to avoid using models trained on a higher sample size than the third stage of Autopilot, which is typically ~64% of the data. >> main_feature_reduction('INSERT_PROJECT_ID', start_featurelist_name=None, lifes=3, top_n_models=5, partition='validation', main_scoring_metric=None, initial_impact_reduction_ratio=0.95, best_model_search_params=None, use_ranks=True) Parameters: ----------- project_id: str, id of DR project, start_featurelist_name: str, name of feature list to start iterating from. Default None lifes: int, stopping criteria, if no best model produced after lifes iterations, stop feature reduction. Default 3 top_n_models: int, only for 'Rank Aggregation method', get top N best models on the leaderboard. Default 5 partition: str, whether to use 'validation','crossValidation' or 'backtesting' partition to get the best model on. Default 'validation' main_scoring_metric: str, DR metric to check performance against, If None DR project metric will be used initial_impact_reduction_ratio: float, ratio of total feature impact that new feature list will contain. Default 0.95 best_model_search_params: dict, dictonary of parameters to search the best model. See official DR python api docs. Default None use_ranks: Boolean, True to use median rank aggregation or False to use total impact unnormalized. Default True Returns: ---------- dr.Model object of the best model on the leaderboard """ project = dr.Project.get(project_id) ratio = initial_impact_reduction_ratio assert ratio < 1, "Please specify initial_impact_reduction_ratio < 1" model_search_params = best_model_search_params runs = 0 # Main function loop while lifes > 0: if runs > 0: start_featurelist_name = None try: best_model = feature_importance_rank_ensembling( project, n_models=top_n_models, metric=main_scoring_metric, by_partition=partition, feature_list_name=start_featurelist_name, ratio=ratio, model_search_params=best_model_search_params, use_ranks=use_ranks, ) except dr.errors.ClientError as e: # decay the ratio ratio *= ratio print(e, f"\nWill try again with a ratio decay ... New ratio={ratio:.3f}") continue ############################## ### GET THE NEW BEST MODEL ### ############################## new_best_model = get_best_models( project, metric=main_scoring_metric, by_partition=partition, model_search_params=model_search_params, ).values[0] ################################# ##### PROCESS STOP CRITERIA ##### ################################# if best_model.id == new_best_model.id: # If no better model is produced with a recent run, expend 1 life lifes -= 1 # If no lives left -> stop if lifes <= 0: print( "New model performs worse. No lives left.\nAUTOMATIC FEATURE SELECTION PROCESS HAS BEEN STOPPED" ) return new_best_model # Decay the ratio ratio *= ratio print( f"New model performs worse. One life is burnt.\nRepeat again with decaying the cumulative impact ratio. New ratio={ratio:.3f}" ) runs += 1 print("Run ", runs, " completed") return new_best_model ``` ### Create a project and initiate Autopilot ``` project = dr.Project.create('https://s3.amazonaws.com/datarobot_public_datasets/madelon_combined_80.csv') project.set_target(target='y', project_name = 'FIRE' mode=dr.AUTOPILOT_MODE.QUICK, worker_count=-1, ) # Wait for Autopilot to finish. You can set verbosity to 0 if you do not wish to see progress updates project.wait_for_autopilot(verbosity=1) print(project.id) ``` ### Feature selection When Autopilot completes, perform feature selection. Then, start Autopilot again using a feature list based on the median rank aggregation of Feature Impact across the top 5 models trained on the "Informative Features" feature list. ``` # Adjust the function's parameters for your purposes best_model = main_feature_selection( project.id, partition="crossValidation", best_model_search_params={"sample_pct__lte": 65} ) ``` ### Report the most accurate model ``` print( f"The best model has {project.metric} score = {best_model.metrics[project.metric]['crossValidation']} on the cross-validation partition \ on the list of {len(best_model.get_features_used())} features" ) ``` ``` The best model has LogLoss score = 0.264978 on the cross-validation partition on the list of 13 features ``` --- # Feature selection notebooks URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/feat-select/index.html > Review notebooks that outline feature selection. DataRobot offers end-to-end code examples via Jupyter notebooks that help you find complete examples of common data science and machine learning workflows. Review the notebooks that outline feature selection below. | Topic | Describes... | | --- | --- | | Feature Importance Rank Ensembling | Learn about the benefits of Feature Importance Rank Ensembling (FIRE)—a method of advanced feature selection that uses a median rank aggregation of feature impacts across several models created during a run of Autopilot. | | Advanced feature selection with Python | Use Python to select features by creating aggregated Feature Impact. | --- # Advanced feature selection with Python URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/feat-select/python-select.html This notebooks shows how you can use DataRobot's Python client to accomplish feature selection by creating aggregated Feature Impact using models created during Autopilot. For more information about the allowed feature transformations, reference the [Python client documentation](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest/autodoc/api_reference.html#datarobot.models.Project.create_type_transform_feature). ## Requirements - Python version 3.7.3. - DataRobot API version 2.14.0. - A DataRobot Project object. - A DataRobot Model object. Small adjustments may be needed depending on the Python version and DataRobot API version you are using. ## Import libraries ``` import datarobot as dr import matplotlib.pyplot as plt import pandas as pd import seaborn as sns sns.set_style("ticks") sns.set_context("poster") ``` ### Connect to DataRobot Read more about different options for [connecting to DataRobot from the client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` # If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call # dr.Client(config_path='path-to-drconfig.yaml') ``` ## Select models For this workflow, select the top five performing models from the project. ``` project = dr.Project.get(project_id="") models = project.get_models() models = models[:5] print(models) ``` ## Create a dataframe Create a dataframe of features' relative rank for the top five models. ``` all_impact = pd.DataFrame() for model in models[0:5]: # This can take about one minute for each model feature_impact = model.get_or_request_feature_impact(max_wait=600) # Ready to be converted to dataframe df = pd.DataFrame(feature_impact) # Track model names and IDs for auditing purposes df["model_type"] = model.model_type df["model_id"] = model.id # By sorting and re-indexing, the new index becomes the 'ranking' df = df.sort_values(by="impactUnnormalized", ascending=False) df = df.reset_index(drop=True) df["rank"] = df.index.values # Add to the master list of all models' feature ranks all_impact = pd.concat([all_impact, df], ignore_index=True) ``` ``` all_impact.head() ``` | | featureName | impactNormalized | impactUnnormalized | redundantWith | model_type | model_id | rank | | --- | --- | --- | --- | --- | --- | --- | --- | | 0 | number_inpatient | 1.000000 | 0.031445 | None | eXtreme Gradient Boosted Trees Classifier with... | 5e620be2d7c7a80c003d16a2 | 0 | | 1 | discharge_disposition_id | 0.950723 | 0.029896 | None | eXtreme Gradient Boosted Trees Classifier with... | 5e620be2d7c7a80c003d16a2 | 1 | | 2 | medical_specialty | 0.828289 | 0.026046 | None | eXtreme Gradient Boosted Trees Classifier with... | 5e620be2d7c7a80c003d16a2 | 2 | | 3 | number_diagnoses | 0.609419 | 0.019163 | None | eXtreme Gradient Boosted Trees Classifier with... | 5e620be2d7c7a80c003d16a2 | 3 | | 4 | num_lab_procedures | 0.543238 | 0.017082 | None | eXtreme Gradient Boosted Trees Classifier with... | 5e620be2d7c7a80c003d16a2 | 4 | ## View rankings and distribution You can find the N features with the highest median ranking and visualize the distributions: ``` from matplotlib.axes._axes import _log as matplotlib_axes_logger matplotlib_axes_logger.setLevel("ERROR") n_feats = 20 top_feats = list( all_impact.groupby("featureName").median().sort_values("rank").head(n_feats).index.values ) top_feat_impact = all_impact.query("featureName in @top_feats").copy() fig, ax = plt.subplots(figsize=(20, 25)) sns.boxenplot(y="featureName", x="rank", data=top_feat_impact, order=top_feats, ax=ax, orient="h") plt.title("Features with highest Feature Impact rating") _ = ax.set_ylabel("Feature Name") _ = ax.set_xlabel("Rank") ``` ## Create a new feature list After analysis, you can create a new feature list with the top features and rerun Autopilot. Note that a feature list can also be created for a dataset and becomes usable across all projects that use that dataset in the future. ``` # Create new featurelist and run autopilot featurelist = project.create_featurelist("consensus-top-features", list(top_feats)) featurelist_id = featurelist.id project.start_autopilot(featurelist_id=featurelist_id) project.wait_for_autopilot() ``` --- # Modeling code examples URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/index.html > Review comprehensive workflows, notebooks, and tutorials that help you find complete examples of common data science and machine learning workflows for modeling. The API user guide includes overviews and workflows for DataRobot's Python client that outline complete examples of common data science and machine learning workflows. Be sure to review the [Developer quickstart](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html) before using the notebooks below. | Topic | Describes... | | --- | --- | | Modeling workflow overview | How to use DataRobot's Python client to train and experiment with models. | | Generate advanced model insights | Model insights available for DataRobot's Python client. | | Build a model factory | A system or a set of procedures that automatically generate predictive models with little to no human intervention. | | Configure datetime partitioning | How to use datetime partitioning to guard a project against time-based target leakage. | | Migrate models | How to transfer models from one DataRobot cluster to another as an .mlpkg file. | | Feature selection examples | Notebooks that outline Feature Importance Rank Ensembling (FIRE) and advanced feature selection with Python. | --- # Migrate a model to a new cluster URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/migrate-nb.html This notebook demonstrates how to migrate a model from one DataRobot cluster to another of the same version. ## Setup Note that the model you choose to migrate must by using Python 3 or a newer version. Additionally, `This notebook will not work using https://app.datarobot.com`. Reference documentation for this workflow's topics below: - Download -modelPackageFile-) - Upload ### Supported paths The following migration paths are currently known to work by default: - DataRobot v8.x -> 8.x - v8.x -> v9.alpha Note that there will be an extra required process when migrating from v7.3.2+ to v8.0.10+ (shown later in this notebook). ### Prerequisites - This notebook must be able to write to the model directory, located in the same directory where this notebook is run from. For the best results, run this notebook from the local file system. - Ensure that the model you choose to migrate must be a deployed model. - Provide API keys for both the source and destination clusters. - The Source and Destination users must have the " Enable Experimental API access " feature flag enabled to follow this workflow. - The notebook must have connectivity to the Source and Destination clusters - DataRobot versions on the clusters must be consistent with the Supported Paths above. - For models on clusters of DataRobot v7.x, you must have SSH access to the App Node of the cluster. - The Source and Destination DataRobot clusters must have the following in the config.yaml: ``` app_configuration: drenv_override: WHITELIST_EXPERIMENTAL_API: true EXPERIMENTAL_API_ACCESS: true ``` ### Install libraries If you are using VS Code, you can install the required DataRobot packages with the cell below. Otherwise, use the cell that follows it. ``` !{sys.executable} -m pip install --upgrade pip # !{sys.executable} -m pip install "datarobot>=2.28,<2.29" !{sys.executable} -m pip install datarobot !{sys.executable} -m pip install datetime requests --upgrade ``` ``` !pip3 install --upgrade pip # !pip3 install "datarobot>=2.28,<2.29" !pip3 install datarobot !pip3 install datetime requests --upgrade ``` ### Import libraries ``` import datetime from datetime import date import json import os import sys import time from timeit import default_timer import urllib.parse from IPython.display import display, HTML import datarobot as dr import requests print("Started: %s" % (str(datetime.datetime.now()))) ``` ### Configure source settings ``` # Provide the URL protocol, address (IP or FQDN), and path # Example: source_host = "http://1.2.3.4" source_host = "https://source.datarobot.example.com" # Do not use https://app.datarobot.com, because users do not have access to the "Enable Experimental API access" permission # Provide an API key from a user with permission from this cluster source_apikey = "" # Provide the source project ID deployment_id = "" #### Local save file path #### # Saves to the model directory by default, using the deployment_id as the file name model_path = "model/%s.mlpkg" % deployment_id #### Destination Settings #### # Example: destination_host = "http://4.3.2.1" destination_host = "https://destination.datarobot.example.com" # Provide an API key from the nodes referenced above from a user with the permissions referenced above destination_apikey = "" print("DataRobot client version: %s" % dr.__version__) print("Source url: %s | deployment_id: %s" % (source_host, deployment_id)) print("Output path: %s" % (model_path)) print("Destinastion url: %s" % (destination_host)) ``` ``` # Code block to ensure that the model directory exists os.makedirs(os.path.dirname(model_path), exist_ok=True) ``` ## Download the deployed model package The following cell downloads the generated data that represents the deployed model given by the source URL and the deployment_id. It is then saved to `models/{deployment_id}.mlpkg`. ``` # Build the headers and provide the token headers = {} headers["Authorization"] = "Bearer {}".format(source_apikey) # Optional - helps DataRobot track usage of this sample headers["User-Agent"] = "AIA-E2E-MIGRATION-19" # Create a new session session = requests.Session() session.headers.update(headers) print("Downloading the mlpkg file from: %s" % source_host) # Download Code # Makes request to generate an .mlpkg for download on the target server # Returns a URL in the location attribute in response header or None def _request_model_package_download(session, host, deployment_id): apiEndpoint = urllib.parse.urljoin( host, "/api/v2/deployments/%s/modelPackageFileBuilds/" % deployment_id ) print("using download apiEndpoint: %s" % apiEndpoint) ssl_verify = True if (urllib.parse.urlparse(host)).scheme == "https" else False try: r = session.post(apiEndpoint, verify=ssl_verify) r.raise_for_status() return r.headers.get("Location") except requests.exceptions.HTTPError as err: print("Error: %s" % err) return None # Downloads an .mlpkg file to the local system from the target server # Returns the binary data to be downloaded or None def get_model_package(session, host, deployment_id): location = _request_model_package_download(session, host, deployment_id) print("using location: %s" % location) ssl_verify = True if (urllib.parse.urlparse(host)).scheme == "https" else False attempts = 0 wait_length = 30 r = None while attempts <= 10: try: r = session.get(location, verify=ssl_verify) r.raise_for_status() print(r.json()) print("sleeping %s seconds" % wait_length) time.sleep(wait_length) attempts += 1 except ValueError: print("looks like no json, time to download") return r except: attempts += 1 print("exception, sleeping for 60 seconds") time.sleep(60) print( "Number of check attempts exceeded. please check the target instance to see if the package is still being assembled or not" ) return None start = default_timer() output = get_model_package(session, source_host, deployment_id) # if output is None: # print("download failed") print("Saving data to: %s" % model_path) with open(model_path, "wb") as f: f.write(output.content) print( "%s took %s seconds to download %s megs" % ( model_path, default_timer() - start, str(round(os.path.getsize(model_path) / (1024 * 1024), 2)), ) ) ``` ## Upload the model to the Model Registry The following cell uploads the .mlpkg file produced earlier to the `destination_host` provided above. ``` headers = {} headers["Authorization"] = "Bearer {}".format(destination_apikey) # Optional - helps DataRobot track usage of this sample headers["User-Agent"] = "AIA-E2E-MIGRATION-19" session = requests.Session() session.headers.update(headers) model_name = "" # Upload code # Makes a request to upload the .mlpkg file to the target server # Returns a URL in the location attribute of the response header or None def _request_package_upload(session, host, fileLocation): apiEndpoint = urllib.parse.urljoin(host, "/api/v2/modelPackages/fromFile/") print("using upload apiEndpoint: %s" % apiEndpoint) ssl_verify = True if (urllib.parse.urlparse(host)).scheme == "https" else False f = {"file": open(fileLocation, "rb")} try: r = session.post(apiEndpoint, files=f, verify=ssl_verify) r.raise_for_status() return r.headers.get("Location") except requests.exceptions.HTTPError as err: print("ERROR: %s" % err) return None # Uploads the .mlpkg file to the target server # Returns the ID of the new model package or None def upload_model_package(session, host, fileLocation): location = _request_package_upload(session, host, fileLocation) print("Location: %s" % location) ssl_verify = True if (urllib.parse.urlparse(host)).scheme == "https" else False attempts = 0 wait_length = 25 while attempts < 10: try: r = session.get(location, verify=ssl_verify) r.raise_for_status() data = r.json() # Check if you get a status or if it's redirected to the package object if data.get("status") is not None: print(data) else: print("Model Package Uploaded") return data.get("id"), data.get("importance"), data.get("name") attempts += 1 print("sleeping %s seconds" % wait_length) time.sleep(wait_length) except: attempts += 1 print("exception, sleeping 60") time.sleep(60) print( "ERROR: Number of check attempts exceeded. please check the target instance to see if there are errors" ) return None # Upload the .mlpkg start = default_timer() print("Uploading file: %s to: %s" % (model_path, destination_host)) destination_model_id, destination_model_importance, destination_model_name = upload_model_package( session, destination_host, model_path ) if destination_model_id is None: print("upload failed") else: link = urllib.parse.urljoin( destination_host, "/model-registry/model-packages/%s" % destination_model_id ) print("Upload took %s seconds" % (default_timer() - start)) ``` ## Find the dedicated prediction engine ID The next step is to find the prediction server used in the cluster. ``` dpeEndpoint = "%s/api/v2/predictionServers/" % (destination_host) prediction_environment_id = None prediction_environment_url = None ssl_verify = True if (urllib.parse.urlparse(destination_host)).scheme == "https" else False print("finding dpe with: %s" % dpeEndpoint) try: r = session.get(dpeEndpoint, verify=ssl_verify) r.raise_for_status() data = json.loads(r.text) prediction_environment_id = data["data"][data["count"] - 1]["id"] prediction_environment_url = data["data"][data["count"] - 1]["url"] except requests.exceptions.HTTPError as err: print("Error: %s" % err) raise Exception("Error: %s" % err) ## Debug # print("data: %s" % data ) print("Found DPE id: %s | url: %s" % (prediction_environment_id, prediction_environment_url)) ``` ## Create a new deployment from the target model package ``` # Returns Deployment ID or None def deploy_model(session, pid, mid, imp): apiEndpoint = "%s/api/v2/deployments/fromModelPackage/" % destination_host print("deploy from: %s" % apiEndpoint) ssl_verify = True if (urllib.parse.urlparse(destination_host)).scheme == "https" else False body_payload = { "label": "%s" % (destination_model_name), "description": "Cloned from: %s" % (urllib.parse.urlparse(source_host).netloc), "modelPackageId": mid, "importance": imp, } print("deployment settings: %s" % body_payload) try: r = session.post( apiEndpoint, data=json.dumps(body_payload), headers={"Content-Type": "application/json", "Accept": "application/json"}, verify=ssl_verify, ) r.raise_for_status() return r.text except requests.exceptions.HTTPError as err: print("ERROR: %s" % err) print(r.text) print(r.headers) return None start = default_timer() if destination_model_importance is None: destination_model_importance = "LOW" output = deploy_model( session, prediction_environment_id, destination_model_id, destination_model_importance ) print("Deplyment of: %s took: %s seconds" % (output, default_timer() - start)) ``` ## DataRobot v7.x extra steps As mentioned in the prerequisite section, there is an additional required process to finalize the migration. Upon executing the block below, you will have the commands required after SSHing in to the `destination_host`. ``` # # Debug command # destination_model_id = "foo" app_node = (urllib.parse.urlparse(destination_host)).netloc print("# Copy the commands below and paste them to the") print("# ssh command prompt on: %s" % app_node) print("") print("sudo su - datarobot") print( 'docker exec -it app /entrypoint /bin/bash -c "python3 support/upgrade_model_packages.py --save %s"' % destination_model_id ) ``` Upon successfu completion, you should see output like this: ``` Total seconds: 1.097603 | Avg 1.097603 seconds to process a package Successfully updated 1 packages ``` ## Copyright 2023 DataRobot Inc. All Rights Reserved. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, express or implied --- # Build a model factory URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/model-factory.html A model factory is a system or set of procedures that automatically generate predictive models with little to no human intervention. Model factories can have multiple layers of complexity, called modules. One module may train models while others can deploy or retrain models. In this example of a model factory, you set up projects and start them in a parallel loop. This allows you to start all projects simultaneously, without unexpected errors. Consider a scenario where you have 20,000 SKUs and you need to do sales forecasting for each one of them. Or, you may have multiple types of customers and you are trying to predict which types will churn. - Can one model handle the high dimensionality that comes with these problems? - Is a single model family able to address the scope of these problems? - Is one preprocessing method sufficient? In this example, use DataRobot to build a single project with the readmitted dataset to predict the probability that a hospital patient may be readmitted after discharge. Then, you will build multiple projects with the `admission id` feature as the target and find the best model for unique value for `admission id`. Lastly, you will prepare the selected models for deployment. ### Import Libraries ``` from time import sleep from dask import compute, delayed # For parallelization import datarobot as dr # Requires version >2.19 import matplotlib.pyplot as plt import pandas as pd import seaborn as sns sns.set(style="whitegrid") ``` ### Import data Download the sample dataset [here](https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/10k-diabetes.csv). ``` data_path = "https://docs.datarobot.com/en/docs/api/guide/python/10k-diabetes.csv" df = pd.read_csv(data_path) ``` ``` # Display the data df.head() ``` | | race | gender | age | weight | admission_type_id | discharge_disposition_id | admission_source_id | time_in_hospital | payer_code | medical_specialty | ... | glipizide_metformin | glimepiride_pioglitazone | metformin_rosiglitazone | metformin_pioglitazone | change | diabetesMed | readmitted | diag_1_desc | diag_2_desc | diag_3_desc | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 0 | Caucasian | Female | [50-60) | ? | Elective | Discharged to home | Physician Referral | 1 | CP | Surgery-Neuro | ... | No | No | No | No | No | No | False | Spinal stenosis in cervical region | Spinal stenosis in cervical region | Effusion of joint, site unspecified | | 1 | Caucasian | Female | [20-30) | [50-75) | Urgent | Discharged to home | Physician Referral | 2 | UN | ? | ... | No | No | No | No | No | No | False | First-degree perineal laceration, unspecified ... | Diabetes mellitus of mother, complicating preg... | Sideroblastic anemia | | 2 | Caucasian | Male | [80-90) | ? | Not Available | Discharged/transferred to home with home healt... | NaN | 7 | MC | Family/GeneralPractice | ... | No | No | No | No | No | Yes | True | Pneumococcal pneumonia [Streptococcus pneumoni... | Congestive heart failure, unspecified | Hyperosmolality and/or hypernatremia | | 3 | AfricanAmerican | Female | [50-60) | ? | Emergency | Discharged to home | Transfer from another health care facility | 4 | UN | ? | ... | No | No | No | No | No | Yes | False | Cellulitis and abscess of face | Streptococcus infection in conditions classifi... | Diabetes mellitus without mention of complicat... | | 4 | AfricanAmerican | Female | [50-60) | ? | Emergency | Discharged to home | Emergency Room | 5 | ? | Psychiatry | ... | No | No | No | No | Ch | Yes | False | Bipolar I disorder, single manic episode, unsp... | Diabetes mellitus without mention of complicat... | Depressive type psychosis | 5 rows × 51 columns ### Connect to DataRobot DataRobot recommends providing a configuration file containing your credentials (endpoint and API Key) to connect to DataRobot. ``` # If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call # dr.Client(config_path='path-to-drconfig.yaml') ``` ### Create a project Create a Datarobot project and initiate Autopilot using data from all patients in the dataset. ``` original_proj = dr.Project.start( df, # Pandas dataframe with data project_name="Readmissions", # Name of the project target="readmitted", # Target of the project metric="LogLoss", # Optimization metric (Default is LogLoss) worker_count=-1, ) # Amount of workers to use (-1 means every worker available) original_proj.wait_for_autopilot( verbosity=1 ) # Wait for Autopilot to finish. You can set verbosity to 0 if you do not wish to see progress updates ``` ### Get the best-performing model from the project ``` # Choose the most accurate model best_model = original_proj.get_models()[0] print(best_model) # Print the most accurate model's name best_model.metrics["LogLoss"]["crossValidation"] # Print the crossValidation score ``` ### Model insight functions Use the functions below to plot the ROC curve and Feature Impact for a model. ``` def plot_roc_curve(datarobot_model): """This function plots a roc curve. Input: datarobot_model: """ roc = datarobot_model.get_roc_curve("crossValidation") roc_df = pd.DataFrame(roc.roc_points) auc_score = datarobot_model.metrics["AUC"]["crossValidation"] plt.plot( roc_df["false_positive_rate"], roc_df["true_positive_rate"], "b", label="AUC = %0.2f" % auc_score, ) plt.legend(loc="lower right") plt.plot([0, 1], [0, 1], "r--") plt.xlim([0, 1]) plt.ylim([0, 1]) plt.ylabel("True Positive Rate") plt.xlabel("False Positive Rate") plt.show() def plot_feature_impact(datarobot_model, title=None): """This function plots feature impact Input: datarobot_model: title : --> title of graph """ # Get feature impact feature_impacts = datarobot_model.get_or_request_feature_impact() # Sort feature impact based on normalised impact feature_impacts.sort(key=lambda x: x["impactNormalized"], reverse=True) fi_df = pd.DataFrame(feature_impacts) # Save feature impact in pandas dataframe fig, ax = plt.subplots(figsize=(14, 5)) b = sns.barplot(x="featureName", y="impactNormalized", data=fi_df[0:5], color="b") b.axes.set_title("Feature Impact" if not title else title, fontsize=20) def wait_for_autopilot(proj, wait=120): total_wait = 0 while proj.get_status()["autopilot_done"] == False: sleep(wait) total_wait += wait total_jobs = len(proj.get_all_jobs()) print( "Autopilot still running! {} jobs running and in queue. Total wait time {}s".format( total_jobs, total_wait ) ) ``` ### Visualize the ROC Curve ``` plot_roc_curve(best_model) ``` ### Plot Feature Impact ``` plot_feature_impact(best_model) ``` ### Build a better model Use the `admission_type` feature as a splitting point to create multiple projects. ``` fig, ax = plt.subplots(figsize=(12, 5)) c = sns.countplot(x="admission_type_id", data=df) ``` ## Create a mini model factory Often when DataRobot needs to set up Automated Feature Discovery (AFD), it may take a while to perform Exploratory Data Analysis (EDA). You can save time when running multiple projects by initiating all of them in parallel. Use Python's dask module to do so. ``` def run_dr_factory(segment_num): try: temp_project = dr.Project.start( df.loc[df["admission_type_id"] == segment_num], project_name="Readmission_%s" % segment_num, target="readmitted", metric="LogLoss", worker_count=10, ) return temp_project except: # Catching the case when dataset has fewer than 20 rows. return f"There was an error in segment {segment_num}." ``` ``` delayed_dr_projects = [] # Create one project for each customer type for value in df["admission_type_id"].unique(): temp = delayed(run_dr_factory)(value) delayed_dr_projects.append(temp) projects = compute(delayed_dr_projects)[0] # Filter to the projects that did not throw errors projects_filtered = [project for project in projects if not isinstance(project, str)] ``` ### Get the best-performing model for each admission type Even though accuracy changes may be insignificant for this dataset, in applicable cases a model factory can produce measurable value. This concept becomes increasingly important with a higher cardinality in your data. For example, consider if your business owns a variety of products, and you build a model factory to produce a model for each product. DataRobot saves you large amounts of time by having handling the evaluation of accuracy for separate set of models built for each product. ``` best_models = {} # To save models for key, project in enumerate(projects_filtered): best_models[key] = projects_filtered[key].get_models()[0] print("--------------------------------") print("Best model for admission type id: %s" % project) print(best_models[key]) print(best_models[key].metrics["LogLoss"]["crossValidation"]) print("--------------------------------") ``` ### Generate Feature Impact Observe the differences in Feature Impact outlined below, which could lead to actionable insights. ``` for key, project in enumerate(projects_filtered): plot_feature_impact( best_models[key], title="Feature Impact for admission type id: %s" % project ) ``` ### Deploy the most accurate models After identifying the best-performing models, you can deploy them and use DataRobot's REST API to make HTTP requests with the deployment ID and return predictions. Once deployed, access monitoring capabilities such as: - Service health - Prediction accuracy - Model retraining ``` prediction_server = dr.PredictionServer.list()[0] for key in best_models: temp_deployment = dr.Deployment.create_from_learning_model( best_models[key].id, label="Readmissions_admission_type: %s" % key, description="Test deployment", default_prediction_server_id=prediction_server.id, ) ``` --- # Python modeling workflow overview URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/modeling-code/python-modeling.html This code example outlines how to use [DataRobot's Python API client](https://datarobot-public-api-client.readthedocs-hosted.com/) to train and experiment with models. It also offers ideas for integrating DataRobot with other products via the API. Specifically, you will: - Create a project and run Autopilot. - Experiment with feature lists, modeling algorithms, and hyperparameters. - Choose the best model. - Perform an in-depth evaluation of the selected model. - Deploy a model into production in a few lines of code. ## Data used for this example This walkthrough uses a synthetic dataset that illustrates a credit card company’s anti-money laundering (AML) compliance program, with the intent of detecting the following money-laundering scenarios: - A customer spends on the card, but overpays their credit card bill and seeks a cash refund for the difference. - A customer receives credits from a merchant without offsetting transactions, and either spends the money or requests a cash refund from the bank. A rule-based engine is in place to produce an alert when it detects potentially suspicious activity consistent with the scenarios above. The engine triggers an alert whenever a customer requests a refund of any amount. Small refund requests are included because they could be a money launderer’s way of testing the refund mechanism or trying to establish refund requests as a normal pattern for their account. The target feature is `SAR`, suspicious activity reports. It indicates whether or not the alert resulted in an SAR after manual review by investigators, which means that this project is a binary classification problem. The unit of analysis is an individual alert, so the model will be built on the alert level. Each alert will get a score ranging from 0 to 1, indicating the probability of being an alert leading to an SAR. The data consists of a mixture of numeric, categorical, and text data. ## Setup ### Import Libraries ``` import datarobot as dr from datarobot_bp_workshop import Visualize, Workshop import matplotlib.pyplot as plt import pandas as pd %matplotlib inline import time import warnings import graphviz import plotly.express as px import seaborn as sns warnings.filterwarnings("ignore") w = Workshop() # wider .head()s pd.options.display.width = 0 pd.options.display.max_columns = 200 pd.options.display.max_rows = 2000 sns.set_theme(style="darkgrid") ``` ### Connect to DataRobot Read more about different options for [connecting to DataRobot from the client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` # If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call # dr.Client(config_path='path-to-drconfig.yaml') ``` ### Select a training dataset ``` # To read from a local file, uncomment and use: # df = pd.read_csv('./data/DR_Demo_AML_Alert.csv') # To read from an s3 bucket: df = pd.read_csv("https://s3.amazonaws.com/datarobot_public_datasets/DR_Demo_AML_Alert.csv") df.head() ``` ``` # To view target distribution: df_target_summary = ( pd.DataFrame(df["SAR"].value_counts()) .reset_index() .rename(columns={"index": "SAR", "SAR": "Count"}) ) ax = sns.barplot(x="SAR", y="Count", data=df_target_summary) for index, row in df_target_summary.iterrows(): ax.text(row.SAR, row.Count, round(row.Count, 2), color="black", ha="center") plt.show() ``` ## Create a project and train models with Autopilot ``` # Create a project by uploading data. This will take a few minutes. project = dr.Project.create( sourcedata=df, project_name="DR_Demo_API_alert_AML_{}".format(pd.datetime.now().strftime("%Y-%m-%d %H:%M")), ) # Set the project's target and initiate Autopilot in Quick mode. # Wait for Autopilot to finish. You can set verbosity to 0 if you do not wish to see progress updates. project.analyze_and_model(target="SAR", worker_count=-1) # Open the project's Leaderboard to monitor the progress in UI. project.open_in_browser() ``` ### Retrieve and review results from the Leaderboard ``` def get_top_of_leaderboard(project, verbose=True): # A helper method to assemble a dataframe with Leaderboard results and print a summary: leaderboard = [] for m in project.get_models(): leaderboard.append( [ m.blueprint_id, m.featurelist.id, m.id, m.model_type, m.sample_pct, m.metrics["AUC"]["validation"], m.metrics["AUC"]["crossValidation"], ] ) leaderboard_df = pd.DataFrame( columns=[ "bp_id", "featurelist", "model_id", "model", "pct", "validation", "cross_validation", ], data=leaderboard, ) if verbose == True: # Print a Leaderboard summary: print("Unique blueprints tested: " + str(len(leaderboard_df["bp_id"].unique()))) print("Feature lists tested: " + str(len(leaderboard_df["featurelist"].unique()))) print("Models trained: " + str(len(leaderboard_df))) print("Blueprints in the project repository: " + str(len(project.get_blueprints()))) # Print the essential information for top models, sorted by accuracy from validation data: print("\n\nTop models on the Leaderboard:") leaderboard_top = ( leaderboard_df[leaderboard_df["pct"] == 64] .sort_values(by="cross_validation", ascending=False) .head() .reset_index(drop=True) ) display(leaderboard_top.drop(columns=["bp_id", "featurelist"], inplace=False)) # Show blueprints of top models: for index, m in leaderboard_top.iterrows(): Visualize.show_dr_blueprint(dr.Blueprint.get(project.id, m["bp_id"])) return leaderboard_top leaderboard_top = get_top_of_leaderboard(project) ``` ``` Unique blueprints tested: 15 Feature lists tested: 4 Models trained: 28 Blueprints in the project repository: 81 Top models on the Leaderboard: ``` | | model_id | model | pct | validation | cross_validation | | --- | --- | --- | --- | --- | --- | | 0 | 61ae672ab9e0c15c325b05b2 | RandomForest Classifier (Gini) | 64.0 | 0.94577 | 0.945790 | | 1 | 61ae73aa2a2a4649c04e2c73 | RandomForest Classifier (Gini) | 64.0 | 0.94598 | 0.945712 | | 2 | 61ae69df4e24ec75154c24ee | AVG Blender | 64.0 | 0.94573 | 0.945318 | | 3 | 61ae65a640d62771a45b0594 | eXtreme Gradient Boosted Trees Classifier with... | 64.0 | 0.94675 | 0.945166 | | 4 | 61ae65a540d62771a45b0592 | RandomForest Classifier (Gini) | 64.0 | 0.94542 | 0.944690 | ## Experiment to get better results When you run a project using Autopilot, DataRobot first creates blueprints based on the characteristics of your data and puts them in the Repository. Then, it chooses a subset from these to train; when training completes, these are the blueprints you’ll find on the Leaderboard. After the Leaderboard is populated, it can be useful to train some of those blueprints that DataRobot skipped. For example, you can try a more complex Keras blueprint like Keras Residual AutoInt Classifier using Training Schedule (3 Attention Layers with 2 Heads, 2 Layers: 100, 100 Units). In some cases, you may want to directly access the trained model through R and retrain it with a different feature list or tune its hyperparameters. ### Find blueprints trained for the project from the Repository ``` blueprints = project.get_blueprints() # After retrieving the blueprints, you can search for a specific blueprint # In the example below, search for all models that have "Gradient" in their name models_to_run = [] for blueprint in blueprints: if "Gradient" in blueprint.model_type: models_to_run.append(blueprint) ``` ``` models_to_run ``` ### Define and train a custom blueprint If you wish to instead create a custom blueprint rather than finding one from the Repository, use the following snippet. You can read more about composing custom blueprints via code by visiting the [blueprint workshop](https://blueprint-workshop.datarobot.com/) in DataRobot. ``` pdm3 = w.Tasks.PDM3(w.TaskInputs.CAT) pdm3.set_task_parameters(cm=50000, sc=10) ndc = w.Tasks.NDC(w.TaskInputs.NUM) rdt5 = w.Tasks.RDT5(ndc) ptm3 = w.Tasks.PTM3(w.TaskInputs.TXT) ptm3.set_task_parameters(d2=0.2, mxf=20000, d1=5, n="l2", id=True) kerasc = w.Tasks.KERASC(rdt5, pdm3, ptm3) kerasc.set_task_parameters( always_use_test_set=1, epochs=4, hidden_batch_norm=1, hidden_units="list(64)", hidden_use_bias=0, learning_rate=0.03, use_training_schedule=1, ) # Check task documentation: # kerasc.documentation() kerasc_blueprint = w.BlueprintGraph(kerasc, name="A Custom Keras BP (1 Layer: 64 Units)").save() kerasc_blueprint.show() kerasc_blueprint.train(project_id=project.id, sample_pct=64) ``` ``` Training requested! Blueprint Id: 4f5c40cbacfa89b3e37dc2f6d5c169a2 ``` ``` Name: 'A Custom Keras BP (1 Layer: 64 Units)' Input Data: Categorical | Numeric | Text Tasks: One-Hot Encoding | Numeric Data Cleansing | Smooth Ridit Transform | Matrix of word-grams occurrences | Keras Neural Network Classifier ``` ### Train a model with a different feature list ``` # Select a model from the Leaderboard: model = dr.Model.get(project=project.id, model_id=leaderboard_top.iloc[0]["model_id"]) # Retrieve Feature Impact: feature_impact = model.get_or_request_feature_impact() # Create a feature list using the top 25 features based on feature impact: feature_list = [f["featureName"] for f in feature_impact[:25]] new_list = project.create_featurelist("new_feat_list", feature_list) # Retrain models using the new feature list: model.retrain(featurelist_id=new_list.id) ``` ### Tune hyperparameters for a model ``` tune = model.start_advanced_tuning_session() # Get available task names, # and available parameter names for a task name that exists on this model tasks = tune.get_task_names() tune.get_parameter_names(tasks[2]) # Adjust this section as required as it may differ depending on task/parameter names as well as acceptable values tune.set_parameter(task_name=tasks[1], parameter_name="n_estimators", value=200) job = tune.run() ``` ### Select the best model ``` # View the top models on the Leaderboard leaderboard_top = get_top_of_leaderboard(project) ``` ``` Unique blueprints tested: 15 Feature lists tested: 4 Models trained: 28 Blueprints in the project repository: 81 Top models on the Leaderboard: ``` | | model_id | model | pct | validation | cross_validation | | --- | --- | --- | --- | --- | --- | | 0 | 61ae672ab9e0c15c325b05b2 | RandomForest Classifier (Gini) | 64.0 | 0.94577 | 0.945790 | | 1 | 61ae73aa2a2a4649c04e2c73 | RandomForest Classifier (Gini) | 64.0 | 0.94598 | 0.945712 | | 2 | 61ae69df4e24ec75154c24ee | AVG Blender | 64.0 | 0.94573 | 0.945318 | | 3 | 61ae65a640d62771a45b0594 | eXtreme Gradient Boosted Trees Classifier with... | 64.0 | 0.94675 | 0.945166 | | 4 | 61ae65a540d62771a45b0592 | RandomForest Classifier (Gini) | 64.0 | 0.94542 | 0.944690 | ``` # Select the model based on accuracy (AUC) top_model = dr.Model.get(project=project.id, model_id=leaderboard_top.iloc[0]["model_id"]) ``` ## In-depth model evaluation ### Retrieve and plot the ROC curve ``` roc = top_model.get_roc_curve("validation") df_roc = pd.DataFrame(roc.roc_points) dr_dark_blue = "#08233F" dr_roc_green = "#03c75f" white = "#ffffff" fig = plt.figure(figsize=(8, 8)) axes = fig.add_subplot(1, 1, 1, facecolor=dr_dark_blue) plt.scatter(df_roc.false_positive_rate, df_roc.true_positive_rate, color=dr_roc_green) plt.plot(df_roc.false_positive_rate, df_roc.true_positive_rate, color=dr_roc_green) plt.plot([0, 1], [0, 1], color=white, alpha=0.25) plt.title("ROC curve") plt.xlabel("False Positive Rate (Fallout)") plt.xlim([0, 1]) plt.ylabel("True Positive Rate (Sensitivity)") plt.ylim([0, 1]) plt.show() ``` ### Retrieve and plot Feature Impact ``` max_num_features = 15 # Retrieve Feature Impact feature_impacts = top_model.get_or_request_feature_impact() # Plot permutation-based Feature Impact feature_impacts.sort(key=lambda x: x["impactNormalized"], reverse=True) FeatureImpactDF = pd.DataFrame( [ {"Impact Normalized": f["impactNormalized"], "Feature Name": f["featureName"]} for f in feature_impacts[:max_num_features] ] ) FeatureImpactDF["X axis"] = FeatureImpactDF.index g = sns.lmplot(x="Impact Normalized", y="X axis", data=FeatureImpactDF, fit_reg=False) sns.barplot(y=FeatureImpactDF["Feature Name"], x=FeatureImpactDF["Impact Normalized"]) ``` ``` ``` ### Retrieve and plot Feature Effects ``` feature_effects = top_model.get_or_request_feature_effect(source="validation") max_features = 5 for f in feature_effects.feature_effects[:max_features]: plt.figure(figsize=(9, 6)) d = pd.DataFrame(f["partial_dependence"]["data"]) if f["feature_type"] == "numeric": d = d[d["label"] != "nan"] d["label"] = pd.to_numeric(d["label"]) sns.lineplot(x="label", y="dependence", data=d).set_title( f["feature_name"] + ": importance=" + str(round(f["feature_impact_score"], 2)) ) else: sns.scatterplot(x="label", y="dependence", data=d).set_title( f["feature_name"] + ": importance=" + str(round(f["feature_impact_score"], 2)) ) ``` ### Score data before deployment ``` # Use training data to test how the model makes predictions test_data = df.head(50) dataset_from_file = project.upload_dataset(test_data) predict_job_1 = top_model.request_predictions(dataset_from_file.id) predictions = predict_job_1.get_result_when_complete() display(predictions.head()) ``` | | row_id | prediction | positive_probability | prediction_threshold | class_0.0 | class_1.0 | | --- | --- | --- | --- | --- | --- | --- | | 0 | 0 | 0.0 | 0.000192 | 0.5 | 0.999808 | 0.000192 | | 1 | 1 | 0.0 | 0.214922 | 0.5 | 0.785078 | 0.214922 | | 2 | 2 | 0.0 | 0.256123 | 0.5 | 0.743877 | 0.256123 | | 3 | 3 | 0.0 | 0.000051 | 0.5 | 0.999949 | 0.000051 | | 4 | 4 | 0.0 | 0.215951 | 0.5 | 0.784049 | 0.215951 | ### Compute Prediction Explanations ``` # Prepare prediction explanations pe_job = dr.PredictionExplanationsInitialization.create(project.id, top_model.id) pe_job.wait_for_completion() ``` ``` # Compute prediction explanations with default parameters pe_job2 = dr.PredictionExplanations.create( project.id, top_model.id, dataset_from_file.id, max_explanations=3, threshold_low=0.1, threshold_high=0.5, ) pe = pe_job2.get_result_when_complete() display(pe.get_all_as_dataframe().head()) ``` ## Deploy a model After identifying the best-performing models, you can deploy them and use DataRobot's REST API to make HTTP requests and return predictions. You can also configure batch jobs to write back into your environment of choice. Once deployed, access monitoring capabilities such as: - Service health - Prediction accuracy - Model retraining ``` # Copy and paste the model ID from previous steps or from the UI: model_id = top_model.id prediction_server_id = dr.PredictionServer.list()[0].id deployment = dr.Deployment.create_from_learning_model( model_id, label="New Deployment", description="A new deployment", default_prediction_server_id=prediction_server_id, ) deployment ``` --- # Make batch predictions with Azure Blob storage URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/prediction-examples/azure-pred.html > Use the DataRobot Python Client package to set up a batch prediction job that reads an input file for scoring from Azure Blob storage and then writes the results back to Azure. The DataRobot Batch Prediction API allows you to take in large datasets and score them against deployed models running on a prediction server. The API also provides flexible options for the intake and output of these files. In this tutorial, you will learn how to use the DataRobot Python Client package (which calls the Batch Prediction API) to set up a batch prediction job. The job reads an input file for scoring from Azure Blob storage and then writes the results back to Azure. This approach also works for Azure Data Lake Storage Gen2 accounts because the underlying storage is the same. ## Requirements In order to use the code provided in this tutorial, make sure you have the following: - Python 2.7 or 3.4+ - The DataRobot Python package (2.21.0+) (pypi) (conda) - A DataRobot deployment - An Azure storage account - An Azure storage container - A scoring dataset in the storage container to use with your DataRobot deployment ## Create stored credentials Running batch prediction jobs requires the appropriate credentials to read and write to Azure Blob storage. You must provide the name of the Azure storage account and an access key. 1. To retrieve these credentials, select theAccess keysmenu in the Azure portal. 2. ClickShow keysto retrieve an access key. You can use either of the keys shown (key1 or key2). 3. Use the following code to create a new credential object within DataRobot that can be used in the batch prediction job to connect to your Azure storage account. AZURE_STORAGE_ACCOUNT="YOUR AZURE STORAGE ACCOUNT NAME"AZURE_STORAGE_ACCESS_KEY="AZURE STORAGE ACCOUNT ACCESS KEY"DR_CREDENTIAL_NAME="Azure_{}".format(AZURE_STORAGE_ACCOUNT)# Create Azure-specific credentials# You can also copy the connection string, which is found below the access key in Azure.credential=dr.Credential.create_azure(name=DR_CREDENTIAL_NAME,azure_connection_string="DefaultEndpointsProtocol=https;AccountName={};AccountKey={};".format(AZURE_STORAGE_ACCOUNT,AZURE_STORAGE_ACCESS_KEY))# Use this code to look up the ID of the credential object created.credential_id=Noneforcredindr.Credential.list():ifcred.name==DR_CREDENTIAL_NAME:credential_id=cred.credential_idbreakprint(credential_id) ## Run the prediction job With a credential object created, you can now configure the batch prediction job as shown in the code sample below: - Setintake_settingsandoutput_settingsto theazuretype. - Forintake_settingsandoutput_settings, seturlto the files in Blob storage that you want to read and write to (the output file does not need to exist already). - Provide the ID of the credential object that was created above. The code sample creates and runs the batch prediction job. Once finished, it provides the status of the job. This code also demonstrates how to configure the job to return both Prediction Explanations and passthrough columns for the scoring data. > [!NOTE] Note > You can find the deployment ID in the sample code output of the [Deployments > Predictions > Prediction API](https://docs.datarobot.com/en/docs/classic-ui/predictions/realtime/code-py.html) tab (with Interface set to "API Client"). ``` DEPLOYMENT_ID = 'YOUR DEPLOYMENT ID' AZURE_STORAGE_ACCOUNT = "YOUR AZURE STORAGE ACCOUNT NAME" AZURE_STORAGE_CONTAINER = "YOUR AZURE STORAGE ACCOUNT CONTAINER" AZURE_INPUT_SCORING_FILE = "YOUR INPUT SCORING FILE NAME" AZURE_OUTPUT_RESULTS_FILE = "YOUR OUTPUT RESULTS FILE NAME" # Set up our batch prediction job # Input: Azure Blob Storage # Output: Azure Blob Storage job = dr.BatchPredictionJob.score( deployment=DEPLOYMENT_ID, intake_settings={ 'type': 'azure', 'url': "https://{}.blob.core.windows.net/{}/{}".format(AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_CONTAINER,AZURE_INPUT_SCORING_FILE), "credential_id": credential_id }, output_settings={ 'type': 'azure', 'url': "https://{}.blob.core.windows.net/{}/{}".format(AZURE_STORAGE_ACCOUNT, AZURE_STORAGE_CONTAINER,AZURE_OUTPUT_RESULTS_FILE), "credential_id": credential_id }, # If explanations are required, uncomment the line below max_explanations=5, # If passthrough columns are required, use this line passthrough_columns=['column1','column2'] ) job.wait_for_completion() job.get_status() ``` When the job completes successfully, you should see the output file in your Azure Blob storage container. ## Documentation - Prediction API overview - DataRobot Batch Prediction API --- # Using the batch prediction API URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/prediction-examples/batch-pred-api.html This notebook demonstrates how to use DataRobot's batch prediction API to score large datasets with a deployed DataRobot model. The batch prediction API provides flexible intake and output options when scoring large datasets using prediction servers. The API is exposed through the DataRobot Public API and can be consumed using a REST-enabled client or Public API bindings for DataRobot's Python client. Some features of the batch prediction API include: - Intake and output configuration. - Support for streaming local files. - The ability to initiate scoring while still uploading data and simultaneously downloading the results. - Scoring large datasets from and to Amazon S3, Azure Blob, and Google Cloud Storage. - Connecting to external data sources using JDBC with bidirectional streaming of scoring data and results. - A mix of intake and output options; for example, the ability to score from a local file and return results to an S3 target. - Protection against prediction server overload with concurrency and size control level options. - Prediction Explanations (with an option to add thresholds). - Support for passthrough columns to correlate scored data with source data. - Prediction warnings in the output. ## Requirements - Python version 3.7.3 - DataRobot API version 2.26.0 - A deployed DataRobot model object Small adjustments may be required depending on the versions of Python and the DataRobot API you are using. You can also access [full documentation of the Python package](https://docs.datarobot.com/en/docs/predictions/batch/batch-prediction-api/index.html). ## Connect to DataRobot To inititate scoring jobs through the batch prediction API, you need to connect to DataRobot through the `datarobot.Client` command. DataRobot recommends providing a configuration file containing your credentials (endpoint and API Key) to connect to DataRobot. For more information about authentication, reference the [API Quickstart guide]( [https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html). ``` import datarobot as dr # If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call # dr.Client(config_path='path-to-drconfig.yaml') ``` ## Set the deployment ID Before proceeding, provide the deployed model's deployment ID (retrieved from the deployment's [Overview tab](https://docs.datarobot.com/en/docs/mlops/monitor/dep-overview.html)). ``` deployment_id = "YOUR_DEPLOYMENT_ID" ``` ## Determine input and output options DataRobot's batch prediction API allows you to score data from and to multiple sources. You can take advantage of the credentials and data sources you have already established previously through the UI for easy scoring. Credentials are usernames and passwords, while data sources are any databases with which you have previously established a connection (e.g., Snowflake). View the example code below outlining how to query credentials and data sources. You can reference the full list of DataRobot's supported [input](https://docs.datarobot.com/en/docs/predictions/batch/batch-prediction-api/intake-options.html) and [output options](https://docs.datarobot.com/en/docs/predictions/batch/batch-prediction-api/output-options.html). The snippet below shows how you can query all credentials tied to a DataRobot account. ``` dr.Credential.list() ``` ``` [Credential('5e6696ff820e737a5bd78430', 'adam', 'basic'), Credential('5ed55704397e667bb0caf1c8', 'DATAROBOT', 'basic'), Credential('5ed557e8ae4c4f7ccd1f0fda', 'ta_admin', 'basic'), Credential('5ed55e08397e667c2bcaf137', 'SourceCredentials_PredicitonJob_5ed55e07397e667c2bcaf134', 'basic'), Credential('5ed55e08397e667c2bcaf139', 'TargetCredentials_PredicitonJob_5ed55e07397e667c2bcaf134', 'basic'), Credential('5ed6ba3c397e6611f9caf27d', 'SourceCredentials_PredicitonJob_5ed6ba3c397e6611f9caf27a', 'basic'), Credential('5ed6ba3d397e6611f9caf27f', 'TargetCredentials_PredicitonJob_5ed6ba3c397e6611f9caf27a', 'basic')] ``` The output above returns multiple sets of credentials. The alphanumeric string included in each item of the list is the credentials ID. You can use that ID to access credentials through the API. The snippet below shows how you can query all data sources tied to a DataRobot account. The second line lists each datastore with an alphanumeric string; that is the datastore ID. ``` dr.DataStore.list() print(dr.DataStore.list()[0].id) ``` ``` 5e6696ff820e737a5bd78430 ``` ## Batch prediction scoring examples The snippets below demonstrate how to score data with the Batch Prediction API. Edit the `intake_settings` and `output_settings` to suit your needs. You can mix and match until you get the outcome you prefer. ### Score from CSV to CSV ``` # Scoring without Prediction Explanations if False: dr.BatchPredictionJob.score( deployment_id, intake_settings={ "type": "localFile", "file": "inputfile.csv", # Provide the filepath, Pandas dataframe, or file-like object here }, output_settings={"type": "localFile", "path": "outputfile.csv"}, ) # Scoring with Prediction Explanations if False: dr.BatchPredictionJob.score( deployment_id, intake_settings={ "type": "localFile", "file": "inputfile.csv", # Provide the filepath, Pandas dataframe, or file-like object here }, output_settings={"type": "localFile", "path": "outputfile.csv"}, max_explanations=3, # Compute Prediction Explanations for the amount of features indicated here ) ``` ### Score from S3 to S3 ``` if False: dr.BatchPredictionJob.score( deployment_id, intake_settings={ "type": "s3", "url": "s3://theos-test-bucket/lending_club_scoring.csv", # Provide the URL of your datastore here "credential_id": "YOUR_CREDENTIAL_ID_FROM_ABOVE", # Provide your credentials here }, output_settings={ "type": "s3", "url": "s3://theos-test-bucket/lending_club_scored2.csv", "credential_id": "YOUR_CREDENTIAL_ID_FROM_ABOVE", }, ) ``` ### Score from JDBC to JDBC ``` if False: dr.BatchPredictionJob.score( deployment_id, intake_settings={ "type": "jdbc", "table": "table_name", "schema": "public", "dataStoreId": data_store.id, # Provide the ID of your datastore here "credentialId": cred.credential_id, # Provide your credentials here }, output_settings={ "type": "jdbc", "table": "table_name", "schema": "public", "statementType": "insert", "dataStoreId": data_store.id, "credentialId": cred.credential_id, }, ) ``` --- # ESG score predictions with Python URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/prediction-examples/esg-score.html This notebook walks through Python code from an example application that uses DataRobot to predict the Environmental, Societal, and Corporate (ESG) scores for stocks. After completing this lab, you will be able to: - Use the DataRobot Python client to build a model from a training dataset and deploy the model. - Use the DataRobot predictions REST API to calculate predicted values. ### Goals This example workflow creates a model to predict a company's ESG score. [ESG](https://en.wikipedia.org/wiki/Environmental,_social_and_corporate_governance) is a rating of a corporation's Environmental, Societal, and Governance scores. A lower ESG score means a particular company (and its stock) is less exposed to risk in that area. For example, a company that deals with oil extraction might have a very high environmental impact rating, which will in turn increase their overall ESG score. Calculating ESG scores is an extensive process that involves in-depth analysis of a company's publicly available information, as well as data from news sources. As not every company will have their ESG score calculated, you will use DataRobot's ML technology to score a large number of companies across several different stock exchanges that don't have existing ESG scores. This example is part of a larger project — a full demo application called "Harv the Finance Finder." This lab only covers the portions of the application specific to DataRobot AutoML. For an example of how these predictions could be included in an application, [see the full application source](https://github.com/datarobot-community/harv-the-finance-finder) in GitHub. ## Setup ### Prerequisites In order to complete this workflow, you'll need: - A DataRobot account - Basic knowledge of Python - Familiarity with data science concepts and terminology - Python 3 and the DataRobot Python client installed. ## Explore the training data Start by downloading the sample data file: [stock_quotes_esg_train.csv](https://s3.amazonaws.com/datarobot_public/dru/esg/stock_quotes_esg_train.csv) Review the CSV file and note the features: - symbo is the stock ticker symbol — MSFT for Microsoft, V for Visa, GM for General Motors, and so on, as seen in companyName . - open , close , high , low , week52Low , and week52High indicate how the stock price has moved, either today or in the last year. All numbers are in USD. - marketCap tells us the total valuation of the company. - sector is the primary sector that the company operates in, for example, Electronic Technology, Health Services, Transportation, etc. - esg_category is the target feature you'll be training the model on. The companies are lumped into four ESG categories: 1 being the lowest ESG risk (best) and 4 being the highest (worst). ### Data source The application uses data from the [IEX API](http://iexcloud.io/). The stock dataset was created by merging stock data from various industry sectors into a single dataset. The data was collected on 25 May 2020. ESG scores are provided by various agencies, but not in a publicly accessible API. For this showcase application, DataRobot generated fake data to train the model, based on sustainability ratings available in [Yahoo Finance](https://finance.yahoo.com/quote/LULU/sustainability?p=LULU). The script used in the project is available on [GitHub](https://github.com/datarobot-community/harv-the-finance-finder/blob/master/scripts/generate_esg_data.py). ### Connect to DataRobot To read more about the options for connecting to DataRobot from the Python client, [review the API Quickstart guide](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` # If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call # dr.Client(config_path='path-to-drconfig.yaml') ``` ### Configure the Python client The most important package to import is the DataRobot Python Client(opens in a new tab) package, which provides the API to connect your client application to DataRobot. ``` import datarobot as dr ``` Then instantiate the DataRobot client. ``` dr.Client() ``` ## Upload data Upload the project with `dr.Project.create`, passing in the path to the dataset you downloaded above. Specify a project name of your choice. ``` import os, urllib.request DATASET_PATH = "./stock_quotes_esg_train.csv" DATASET_URL = "https://s3.amazonaws.com/datarobot_public/dru/esg/stock_quotes_esg_train.csv" if not os.path.exists(DATASET_PATH): print(f"Downloading dataset from {DATASET_URL} ...") urllib.request.urlretrieve(DATASET_URL, DATASET_PATH) print("Download complete.") project = dr.Project.create( sourcedata=DATASET_PATH, project_name="your project name" ) ``` ## Modeling ### Define the target As a best practice, this application would typically use `esg_category`, a numerical property, as the target feature. However, for learning purposes only, use [multiclass classification](https://docs.datarobot.com/en/docs/modeling/analyze-models/evaluate/multiclass.html#background) to predict ESG scores to be one of four categories, represented by integer values 1 to 4 rather than a numeric value. Do this by transforming the `esg_category` numeric feature to a categorical feature called `esg_category_categorical` using `project.create_type_transform_feature`. ``` # Transform esg_category into a categorical variable type # Note: not best practice, included for learning purposes only project.create_type_transform_feature( "esg_category_categorical", # new feature name "esg_category", # parent name dr.enums.VARIABLE_TYPE_TRANSFORM.CATEGORICAL_INT, ) ``` ### Start Autopilot To start the process of training models on this data, call `project.set_target()`, passing in the target name ( `esg_category_categorical`), which you created in previous steps. You can also pass the mode option, telling DataRobot to do a quick modeling run, building a limited set of models. This can be a long-running process. Call `project.wait_for_autopilot(`), which will print informative output and block the script until the modeling job is finished. ``` # This kicks off modeling using Quick Autopilot mode project.set_target(target="esg_category_categorical", mode=dr.enums.AUTOPILOT_MODE.QUICK) # Time for a cup of tea or a walk - this might take ~15 minutes project.wait_for_autopilot() ``` ### Get the recommended model After Autopilot has finished, you can get a list of all models it has created in your project, ranked by their accuracy. You can get DataRobot's recommendation by calling `dr.ModelRecommendation.get(project.id)`, and get our model from that using `get_model()`. ``` recommendation = dr.ModelRecommendation.get(project.id) recommended_model = recommendation.get_model() print(f"Recommended model is {recommended_model}") ``` ### Deploy the model Now that you have the recommended model, you can deploy it to a production environment to make predictions with new data. Models are not deployed to the same server used to train models; they are deployed to one or more prediction servers. The code below automatically retrieves the first available prediction server (required for DataRobot Cloud and on-premises accounts) and uses it when creating the deployment. After determining the prediction server, use `dr.Deployment.create_from_learning_model` to deploy the model. ``` # Get the prediction server ID. # Required for DataRobot Cloud and on-premises accounts; optional for trial accounts. prediction_servers = dr.PredictionServer.list() prediction_server_id = prediction_servers[0].id if prediction_servers else None deployment = dr.Deployment.create_from_learning_model( model_id=recommended_model.id, label="Financial ESG model", description="Model for scoring financial quote data", default_prediction_server_id=prediction_server_id, ) print(f"Deployment created: {deployment}, deployment id: {deployment.id}") ``` ## Calculate ESG scores for a dataset ### Download prediction data After your model has been deployed you can start making predictions using [DataRobot'sREST API](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi.html). You can use any language to call the API. This notebook uses Python. Start by downloading the dataset we want to calculate ESG scores for: [stock_quotes_all.csv](https://s3.amazonaws.com/datarobot_public/dru/esg/stock_quotes_all.csv). Be sure to save the file in the same location where you saved the training data previously. ### Configure application code In the real world, predictions are likely to happen in a separate application from creating and deploying a model. In this notebook, DataRobot assumes you are working in an interactive environment such as a Python shell or notebook, but this notebook walks through the process you'd go through for adding the code to an application. Make sure the `DATAROBOT_API_TOKEN` and `DATAROBOT_ENDPOINT` are set in your environment as described in the first step of this lab. Then set up your application in Python, which is similar to what you did when setting up for the model building steps: ``` import csv import json import os import sys import urllib.request import datarobot as dr import requests DR_API_KEY = os.environ["DATAROBOT_API_TOKEN"] dr.Client() ``` You might have taken note of the deployment ID when you deployed your model earlier. If you have that ID available, you can use it to find the prediction server for the deployed model. But that value may not be readily accessible in a real application, so use the following code to find the correct deployment using the label you set when deploying the model. Refer back to the earlier steps where you deployed the model to find the label specified and use it here to find the deployment and its associated prediction server. ``` for d in dr.Deployment.list(): if d.label == "Financial ESG model": deployment = d prediction_server_url = deployment.default_prediction_server["url"] ``` ### Make a prediction request Now you have everything you need to make your request. DataRobot's prediction API doesn't come with an SDK so you need to "handcraft" your API requests, using Python's requests. You are sending the following header values: - Content-Type in this case is text/plain as you're sending a CSV file. Alternatively, the API also accepts application/json for JSON payloads. - Authorization takes the same API key we used with the modeling API in the DataRobot Python SDK. - datarobot-key is the key specifically for the prediction server. Note that this value is not used for trial or pay-as-you-go DataRobot accounts. To predict, send a POST request to the prediction server with the data from the file you downloaded above as the payload. Prediction API responds in JSON format, and your predictions will be in the data field. ``` PRED_DATASET_PATH = "./stock_quotes_all.csv" PRED_DATASET_URL = "https://s3.amazonaws.com/datarobot_public/dru/esg/stock_quotes_all.csv" if not os.path.exists(PRED_DATASET_PATH): print(f"Downloading prediction dataset from {PRED_DATASET_URL} ...") urllib.request.urlretrieve(PRED_DATASET_URL, PRED_DATASET_PATH) print("Download complete.") headers = { "Content-Type": "text/plain; charset=UTF-8", "Authorization": f"Bearer {DR_API_KEY}", # comment out line below if using a trial or pay-as-you-go account. "datarobot-key": deployment.default_prediction_server["datarobot-key"], } url = f"{prediction_server_url}/predApi/v1.0/deployments/{deployment.id}/predictions?passthroughColumns=symbol" data = open(PRED_DATASET_PATH, "rb").read() predictions_response = requests.post(url, data=data, headers=headers) predictions = predictions_response.json()["data"] ``` ### Parse and save the prediction response The prediction data payload is a JSON array of objects for all predicted fields, in the same order that you sent it. The actual predicted value is in the prediction field. In this example, you're creating a new list of symbol/category pairs, and filling them by iterating through the returned predictions. ``` # Transform predictions into a CSV file of the format:# symbol, esg_category # where symbol is a value passed through from the prediction request esg_categories = [["symbol", "esg_category"]] for prediction in predictions: symbol = prediction["passthroughValues"]["symbol"] value = int(prediction["prediction"]) esg_entry = [symbol, value] esg_categories.append(esg_entry) # Write the data as a CSV file with open("stocks_esg_scores.csv", mode="w") as out_csv: csv_writer = csv.writer(out_csv) csv_writer.writerows(esg_categories) ``` Review the contents of the output file — `stocks_esg_scores.csv` — to confirm that the `esg_category` column contains ESG categories (i.e. integers 1 through 4). ## Recap In this lab, you walked through Python application code to: - Connect a client application to DataRobot. - Upload a training dataset to DataRobot AutoML. - Identify a target value. - Run Autopilot to generate a set of models . - Deploy the recommended model to a prediction server. - Request predictions from DataRobot for a dataset. --- # Make batch predictions with Google Cloud storage URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/prediction-examples/gcs-pred.html > Learn how to make and write predictions to Google Cloud Storage. The DataRobot Batch Prediction API allows you to take in large datasets and score them against deployed models running on a prediction server. The API also provides flexible options for the intake and output of these files. In this tutorial, you will learn how to use the DataRobot Python Client package (which calls the Batch Prediction API) to set up a batch prediction job. The job reads an input file for scoring from Google Cloud Storage (GCS) and then writes the results back to GCS. ## Requirements In order to use the code provided in this tutorial, make sure you have the following: - Python 2.7 or 3.4+ - The DataRobot Python package (version 2.21.0+) - A DataRobot deployment - A GCS bucket - A service account with access to the GCS bucket (detailed below) - A scoring dataset that lives in the GCS bucket to use with your DataRobot deployment ## Configure a GCP service account Running batch prediction jobs requires the appropriate credentials to read and write to GCS. You must create a service account within the Google Cloud Platform that has access to the GCS bucket, then download a key for the account to use in the batch prediction job. 1. To retrieve these credentials, log into the Google Cloud Platform console and selectIAM & Admin > Service Accountsfrom the sidebar. 2. ClickCreate Service Account. Provide a name and description for the account, then clickCreate > Done. 3. On theService Accountpage, find the account that you just created, navigate to theDetailspage, and clickKeys. 4. Go to theAdd Keymenu and clickCreate new key. Select JSON for the key type and clickCreateto generate a key and download a JSON file with the information required for the batch prediction job. 5. Return to your GCS bucket and navigate to thePermissionstab. ClickAdd, enter the email address for the service account user you created, and give the account the “Storage Admin” role. ClickSaveto confirm the changes. This grants your GCP service account access to the GCS bucket. ## Create stored credentials After downloading the JSON key, use the following code to create a new credential object within DataRobot. The credentials will be used in the batch prediction job to connect to the GCS bucket. Open the JSON key file and copy its contents into the key variable. The DataRobot Python client reads the JSON data as a dictionary and parses it accordingly. ``` # Set name for GCP credential in DataRobot DR_CREDENTIAL_NAME = "YOUR GCP DATAROBOT CREDENTIAL NAME" # Create a GCP-specific Credential # NOTE: This cannot be done from the UI # This can be generated and downloaded ready to drop in from within GCP # 1. Go to IAM & Admin -> Service Accounts # 2. Search for the Service Account you want to use (or create a new one) # 3. Go to Keys # 4. Click Add Key -> Create Key # 5. Selection JSON key type # 6. copy the contents of the json file into the gcp_key section of the credential code below key = { "type": "service_account", "project_id": "**********", "private_key_id": "***************", "private_key": "-----BEGIN PRIVATE KEY-----\n********\n-----END PRIVATE KEY-----\n", "client_email": "********", "client_id": "********", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/*********" } credential = dr.Credential.create_gcp( name=DR_CREDENTIAL_NAME, gcp_key=key ) # Use this code to look up the ID of the credential object created. credential_id = None for cred in dr.Credential.list(): if cred.name == DR_CREDENTIAL_NAME: credential_id = cred.credential_id break print(credential_id) ``` ## Run the prediction job With a credential object created, you can now configure the batch prediction job. Set the `intake_settings` and `output_settings` to the `gcp` type. Provide both attributes with the URL to the files in GCS that you want to read and write to (the output file does not need to exist already). Additionally, provide the ID of the credential object that was created above. The code below creates and runs the batch prediction job. Once finished, it provides the status of the job. This code also demonstrates how to configure the job to return both Prediction Explanations and passthrough columns for the scoring data. > [!NOTE] Note > You can find the deployment ID in the sample code output of the [Deployments > Predictions > Prediction API](https://docs.datarobot.com/en/docs/classic-ui/predictions/realtime/code-py.html) tab (with Interface set to "API Client"). ``` DEPLOYMENT_ID = 'YOUR DEPLOYMENT ID' # Set GCP Info GCP_BUCKET_NAME = "YOUR GCS BUCKET NAME" GCP_INPUT_SCORING_FILE = "YOUR INPUT SCORING FILE NAME" GCP_OUTPUT_RESULTS_FILE = "YOUR OUTPUT RESULTS FILE NAME" # Set up the batch prediction job # Input: Google Cloud Storage # Output: Google Cloud Storage job = dr.BatchPredictionJob.score( deployment=DEPLOYMENT_ID, intake_settings={ 'type': 'gcp', 'url': "gs://{}/{}".format(GCP_BUCKET_NAME,GCP_INPUT_SCORING_FILE), "credential_id": credential_id }, output_settings={ 'type': 'gcp', 'url': "gs://{}/{}".format(GCP_BUCKET_NAME,GCP_OUTPUT_RESULTS_FILE), "credential_id": credential_id }, # If explanations are required, uncomment the line below max_explanations=5, # If passthrough columns are required, use this line passthrough_columns=['column1','column2'] ) job.wait_for_completion() job.get_status() ``` When the job completes successfully, you will see the output file in the GCS bucket. ## Documentation - Prediction API overview - DataRobot Batch Prediction API --- # Prediction code examples URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/prediction-examples/index.html > Review comprehensive workflows, notebooks, and tutorials that help you find complete examples of prediction workflows. The API user guide includes overviews and workflows for DataRobot's Python client that outline complete examples of prediction workflows and tasks. Be sure to review the [Developer quickstart](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html) before using the notebooks below. | Topic | Describes... | | --- | --- | | Make batch predictions with Azure Blob storage | How to generate SHAP-based Prediction Explanations with a use case that determines what drives home value in Iowa. | | Using the Batch Prediction API | DataRobot's batch prediction API to score large datasets with a deployed DataRobot model. | | Make batch predictions with Google Cloud Storage | How to read input data from and write predictions back to Google Cloud Storage. | | Make Visual AI predictions via the API | Scripting code for making batch predictions for a Visual AI model via the API. | | ESG score predictions with Python | How to use Python code from an example application that uses DataRobot to predict the Environmental, Societal, and Corporate (ESG) scores for stocks. | | Create and schedule JDBC prediction jobs | How to use DataRobot's Python client to schedule prediction jobs and write them to a JDBC database. | --- # Schedule predictions with a JDBC database URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/prediction-examples/jdbc-nb.html Making predictions on a daily or monthly basis is a manual, time-consuming, and cumbersome process. Batch predictions are commonly used when you have to score new records over a certain frame of time (weeks, months, etc.). For example, you can use batch predictions to score new leads on a monthly basis to predict who will churn, or to predict on a daily basis which products someone is likely to purchase. This notebook outlines how to use DataRobot's Python client to schedule batch prediction jobs and write them to a JDBC database. Specifically, you will: 1. Retrieve existing data stores and credential information. 2. Configure prediction job specifications. 3. Set up a prediction job schedule. 4. Run a test prediction job and enable an automated schedule for scoring. Before proceeding, note that this workflow requires a [deployed DataRobot model](https://docs.datarobot.com/en/docs/mlops/deployment/deploy-methods/index.html) object to use for scoring and an established [data connection](https://docs.datarobot.com/en/docs/data/connect-data/data-conn.html) to read data and host prediction writeback. For more information about the Python client, reference the [documentation](https://datarobot-public-api-client.readthedocs-hosted.com). ### Import libraries ``` import datarobot as dr import pandas as pd ``` ### Connect to DataRobot Read more about different options for [connecting to DataRobot from the client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` # If the config file is not in the default location described in the API Quickstart guide, '~/.config/datarobot/drconfig.yaml', then you will need to call # dr.Client(config_path='path-to-drconfig.yaml') ``` ### List data stores To enable integration with a variety of enterprise databases, DataRobot provides a “self-service” JDBC product for database connectivity setup. Once configured, you can read data from production databases for model building and predictions. This allows you to quickly train and retrain models on that data while avoiding the unnecessary step of exporting data from your enterprise database to a CSV for ingest to DataRobot. It allows access to more diverse data, which results in more accurate models. Use the cell below to query all data sources tied to a DataRobot account. The second line lists each datastore with an alphanumeric string; that is the datastore ID. ``` for d in dr.DataStore.list(): print(d.id, d.canonical_name, d.params) ``` ### Retrieve credentials list You can reference the [DataRobot documentation](https://docs.datarobot.com/en/docs/data/connect-data/stored-creds.html#credentials-management) for more information about managing credentials. ``` dr.Credential.list() ``` The output above returns multiple sets of credentials. The alphanumeric string included in each item of the list is the credentials ID. You can use that ID to access credentials through the API. ### Specify the deployment and data connection Use the snippet below to indicate the deployment you want to use (by binding the deployment ID, retrieved from the deployment's [Overview tab](https://docs.datarobot.com/en/docs/mlops/monitor/dep-overview.html)) and the data store to which you want to write predictions (by providing the data store ID and the corresponding credentials ID). ``` deployment_id = "620219bb18f7f84dec6cec59" datastore_id = "614ca745c7fab1f23da7a632" data_store = dr.DataStore.get(datastore_id) credential_id = "63865454a351b56ce3cb78b3" cred = dr.Credential.get(credential_id) ``` ### Configure intake settings Use the snippet below to configure the intake settings for JDBC scoring. For more information, reference the [batch predictions documentation in the Python client](https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.25.0/entities/batch_predictions.html) and the [intake options documentation](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html). intake_settings = { 'type': 'jdbc', 'table': 'LENDING_CLUB_10K', 'schema': 'TRAINING', # optional, if supported by database 'catalog': 'DEMO', # optional, if supported by database 'data_store_id': data_store.id, 'credential_id': cred.credential_id, } print(intake_settings) ### Configure output settings Use the snippet below to configure the output settings for JDBC scoring. For more information, reference the [output options documentation](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-options.html#statement-types). ``` output_settings = { "type": "jdbc", "table": "LENDING_CLUB_10K_AA_Temp", "schema": "SCORING", # optional, if supported by database "catalog": "SANDBOX", # optional, if supported by database schema "statement_type": "insert", "create_table_if_not_exists": True, "data_store_id": data_store.id, "credential_id": cred.credential_id, } print(output_settings) # Uncomment and use the following lines for local file export: # output_settings={ # 'type': 'localFile', # 'path': './predicted.csv', # } # print(output_settings) ``` Use the code below to retrieve the name of the deployment. ``` deployment = dr.Deployment.get(deployment_id) deployment.label ``` ### Create a schedule Next, set up a [schedule](https://datarobot-public-api-client.readthedocs-hosted.com/en/v2.25.0/entities/batch_prediction_job_definitions.html?highlight=schedule) for making predictions. The snippet below creates a schedule that makes predictions on the first day of every month at 7:59 AM. ``` schedule = { "minute": [59], "hour": [7], "month": ["*"], "dayOfWeek": ["*"], "dayOfMonth": [1], } schedule ``` ### Configure a prediction job ``` job = { "deployment_id": deployment_id, "num_concurrent": 4, "intake_settings": intake_settings, "output_settings": output_settings, "passthroughColumnsSet": "all", } ``` ### Create a prediction job After configuring a prediction job, use the `BatchPredictionJobDefinition.create` method to create a prediction job definition based on the job and scheduled you configured above. ``` definition = dr.BatchPredictionJobDefinition.create( enabled=True, batch_prediction_job=job, name="Monthly Prediction Job JDBC", schedule=schedule ) definition ``` Lastly, the snippet below initiates the prediction job on the schedule. ``` job_run_automatically = definition.run_on_schedule(schedule) ``` --- # Make Visual AI predictions via the API URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/prediction-examples/vai-pred.html > Learn how to make predictions on Visual AI projects with API calls. This tutorial outlines how to make predictions on Visual AI projects with API calls. To complete this tutorial, you must have trained and deployed a visual AI model. ## Takeaways This tutorial shows how to: - Configure scripting code for making batch predictions via the API - Make an API call to get batch predictions for a visual AI model - Format images to a base64 format ## Predictions workflow 1. Prepare your data for Visual AI. Before making predictions, convert the images you want to score tobase64 format(the standard format for handling images in API calls). Note that when the model returns prediction results, images will return in base64 format. To convert data, use DataRobot's Python package, described in the guidePreparing binary data for predictions. 2. After training and deploying a Visual AI model, navigate to the deployment and access thePredictions > Prediction APItab. This tab provides the scripting code used to make predictions via the API. 3. To configure the scripting code, selectBatchas the prediction type andAPI Clientas the interface type. 4. Copy the code and save it as a Python script (e.g.,datarobot-predict.py). You can edit the script to incorporate additional steps. For example, add thepassthrough_columns_setargument toBatchPredictionJobif you would like to include columns from the input file (e.g.,image_id) to the output file. 5. Using the scripting code fromstep twoand a base64-converted image file (InputDataConverted.csv), make an API call to get predictions from the deployed model: python datarobot-predict.py InputDataConverted.csv Predictions.csv 6. Access the output file (Predictions.csv) to view prediction results. ## Learn More **Additional tools** Reference the [DataRobot Community GitHub](https://github.com/datarobot-community/visual-ai-data-prep/blob/master/visualai_data_prep.py) pages for data prep tools, including a script to help with the base64 conversion process. Log in to GitHub before clicking this link. For more information on the scripting code used in this tutorial, refer to the [Python Package documentation](https://datarobot-public-api-client.readthedocs-hosted.com/). **Multiclass example** For the multiclass classification problem's prediction results in step 6, note that the prediction output file includes the probability of the image falling under each class, the class name with the highest probability, and all the other optional columns requested from the Python scoring script (such as `prediction_status` and `image_id`). ## Documentation - Visual AI overview - Making predictions with Visual AI - Prediction API overview --- # Bolt-on governance with Pulumi URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/pulumi-examples/deploy-gov.html When DataRobot refers to "Bolt on Governance", it means wrapping a large language model endpoint in a DataRobot deployment, allowing you to engage with it through a DataRobot API token. In exchange for creating a deployment around the LLM, you get all of the benefits of DataRobot MLOps such as text drift monitoring, request history, and usage statistics. This notebook outlines how to use pulumi to create a deployment endpoint that interfaces with a large language model. ## Initialize the environment As a preliminary step, initialize our environment and make sure the LLM credentials work. ``` import os import datarobot as dr from openai import AzureOpenAI os.environ["PULUMI_CONFIG_PASSPHRASE"] = "default" assert ( "DATAROBOT_API_TOKEN" in os.environ ), "Please set the DATAROBOT_API_TOKEN environment variable" assert "DATAROBOT_ENDPOINT" in os.environ, "Please set the DATAROBOT_ENDPOINT environment variable" assert "OPENAI_API_BASE" in os.environ, "Please set the OPENAI_API_BASE environment variable" assert "OPENAI_API_KEY" in os.environ, "Please set the OPENAI_API_KEY environment variable" assert "OPENAI_API_VERSION" in os.environ, "Please set the OPENAI_API_VERSION environment variable" dr_client = dr.Client() ``` ``` def test_azure_openai_credentials(): """Test the provided OpenAI credentials.""" model_name = os.getenv("OPENAI_API_DEPLOYMENT_ID") try: client = AzureOpenAI( api_key=os.getenv("OPENAI_API_KEY"), azure_endpoint=os.getenv("OPENAI_API_BASE"), api_version=os.getenv("OPENAI_API_VERSION"), ) client.chat.completions.create( messages=[{"role": "user", "content": "hello"}], model=model_name, # type: ignore[arg-type] ) except Exception as e: raise ValueError( f"Unable to run a successful test completion against model '{model_name}' " "with provided Azure OpenAI credentials. Please validate your credentials." ) from e test_azure_openai_credentials() ``` ## Set up a project Configure the functions below to create and build or destroy the Pulumi stack. ``` from pulumi import automation as auto def stack_up(project_name: str, stack_name: str, program: callable) -> auto.Stack: # create (or select if one already exists) a stack that uses our inline program stack = auto.create_or_select_stack( stack_name=stack_name, project_name=project_name, program=program ) stack.refresh(on_output=print) stack.up(on_output=print) return stack def destroy_project(stack: auto.Stack): """Destroy pulumi project""" stack_name = stack.name stack.destroy(on_output=print) stack.workspace.remove_stack(stack_name) print(f"stack {stack_name} in project removed") ``` ## Declarative LLM deployment Deploying a bolt-on governance model isn't complicated, but it is more involved than creating a custom deployment for a standard classification model for three reasons. First, you want to set runtime parameters around the deployment specifying the LLM endpoint and other metadata. Second, you want to set up and apply a credential for model metadata that should be hidden, such as the API Token. The hidden credential you create will actually end up as one of our runtime parameters. Finally, you want to use a special environment called a Serverless Prediction Environment that works well for sending API calls through a deployment. You need to set up one of these specifically for this model. Once you set up your credentials and runtime parameters, put your source code onto DataRobot, register the model and then initialize the deployment. ``` import pulumi import pulumi_datarobot as datarobot def setup_runtime_parameters( credential: datarobot.ApiTokenCredential, ) -> list[datarobot.CustomModelRuntimeParameterValueArgs]: """Setup runtime parameters for bolt on goverance deployment. Each runtime parameter is a tuple trio with the key, type, and value. Args: credential (datarobot.ApiTokenCredential): The DataRobot credential representing the LLM api token """ return [ datarobot.CustomModelRuntimeParameterValueArgs( key=key, type=type_, value=value, # type: ignore[arg-type] ) for key, type_, value in [ ("OPENAI_API_KEY", "credential", credential.id), ("OPENAI_API_BASE", "string", os.getenv("OPENAI_API_BASE")), ("OPENAI_API_VERSION", "string", os.getenv("OPENAI_API_VERSION")), ( "OPENAI_API_DEPLOYMENT_ID", "string", os.getenv("OPENAI_API_DEPLOYMENT_ID"), ), ] ] def make_bolt_on_governance_deployment(): """ Deploy a trained model onto DataRobot's prediction environment. Upload source code to create a custom model version. Then create a registered model and deploy it to a prediction environment. """ # ID for Python 3.11 Moderations Environment python_environment_id = "65f9b27eab986d30d4c64268" custom_model_name = "App Template Minis - OpenAI LLM" registered_model_name = "App Template Minis - OpenAI Registered Model" deployment_name = "App Template Minis - Bolt on Goverance Deployment" prediction_environment = datarobot.PredictionEnvironment( resource_name="App Template Minis - Serverless Environment", platform=dr.enums.PredictionEnvironmentPlatform.DATAROBOT_SERVERLESS, ) llm_credential = datarobot.ApiTokenCredential( resource_name="App Template Minis - OpenAI LLM Credentials", api_token=os.getenv("OPENAI_API_KEY"), ) runtime_parameters = setup_runtime_parameters(llm_credential) deployment_files = [ ("./model_package/requirements.txt", "requirements.txt"), ("./model_package/custom.py", "custom.py"), ("./model_package/model-metadata.yaml", "model-metadata.yaml"), ] custom_model = datarobot.CustomModel( resource_name=custom_model_name, runtime_parameter_values=runtime_parameters, files=deployment_files, base_environment_id=python_environment_id, target_type=dr.enums.TARGET_TYPE.TEXT_GENERATION, target_name="content", language="python", replicas=2, ) registered_model = datarobot.RegisteredModel( resource_name=registered_model_name, custom_model_version_id=custom_model.version_id, ) deployment = datarobot.Deployment( resource_name=deployment_name, label=deployment_name, registered_model_version_id=registered_model.version_id, prediction_environment_id=prediction_environment.id, ) pulumi.export("serverless_environment_id", prediction_environment.id) pulumi.export("custom_model_id", custom_model.id) pulumi.export("registered_model_id", registered_model.id) pulumi.export("deployment_id", deployment.id) ``` ## Run the stack Running the stack takes the files that are in the `model_package` directory, puts them onto DataRobot as a custom model, registers that model, and deploys the result. ``` project_name = "AppTemplateMinis-BoltOnGovernance" stack_name = "MarshallsExtraSpecialLargeLanguageModel" stack = stack_up(project_name, stack_name, program=make_bolt_on_governance_deployment) ``` ### Interact with outputs Now that you have a bolt-on goverance deployment, you can interact with it directly through the OpenAI SDK. The only difference is that you pass the DataRobot API Token instead of your LLM credentials. ``` from pprint import pprint from openai import OpenAI deployment_id = stack.outputs().get("deployment_id").value deployment_chat_base_url = dr_client.endpoint + f"/deployments/{deployment_id}/" client = OpenAI(api_key=dr_client.token, base_url=deployment_chat_base_url) messages = [ {"role": "user", "content": "Why are ducks called ducks?"}, ] response = client.chat.completions.create(messages=messages, model="gpt-4o") pprint(response.choices[0].message.content) ``` ## Clear your work Use the following cell to shut down the stack, thereby deleting any assets created in DataRobot. ``` destroy_project(stack) ``` ### How does scoring code work? The following cell contains code used to upload so that DataRobot knows how to interact with our model. The bolt-on goverance model only requires you to define hooks for `load_model` and `chat1`, but you can add [others too](https://docs.datarobot.com/en/docs/mlops/deployment/custom-models/custom-model-assembly/custom-model-components.html). ``` from IPython.display import Code Code(filename="./model_package/custom.py", language="python") ``` --- # Deploy a custom model with Pulumi URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/pulumi-examples/deploy_custom_inference_model.html This notebook outlines how to use Pulumi to deploy a scikit learn classifier in two easy stages. ## Initialize the environment ``` import os import datarobot as dr os.environ["PULUMI_CONFIG_PASSPHRASE"] = "default" assert ( "DATAROBOT_API_TOKEN" in os.environ ), "Please set the DATAROBOT_API_TOKEN environment variable" assert "DATAROBOT_ENDPOINT" in os.environ, "Please set the DATAROBOT_ENDPOINT environment variable" dr.Client() ``` ## Set up a project Configure the functions below to create and build or destroy the Pulumi stack. ``` from pulumi import automation as auto def stack_up(project_name: str, stack_name: str, program: callable) -> auto.Stack: # create (or select if one already exists) a stack that uses our inline program stack = auto.create_or_select_stack( stack_name=stack_name, project_name=project_name, program=program ) stack.refresh(on_output=print) stack.up(on_output=print) return stack def destroy_project(stack: auto.Stack): """Destroy pulumi project""" stack_name = stack.name stack.destroy(on_output=print) stack.workspace.remove_stack(stack_name) print(f"stack {stack_name} in project removed") ``` ## Declarative custom model deployment To deploy a custom model, you have to put your source code onto DataRobot, register the model, and then initialize the deployment. The `make_custom_deployment` function below shows the declarative way to do this. ``` import pulumi import pulumi_datarobot as datarobot def make_custom_inference_deployment(): """ Deploy a trained model onto DataRobot's prediction environment. Upload source code to create a custom model version. Then create a registered model and deploy it to a prediction environment. """ # ID for Python 3.9 Scikit learn drop in environment base_environment_id = "5e8c889607389fe0f466c72d" # ID for the default prediction server default_prediction_server_id = "5dd7fa2274a35f003102f60d" custom_model_name = "App Template Minis - Readmitted Custom Model" registered_model_name = "App Template Minis - Readmitted Registered Model" deployment_name = "App Template Minis - Readmitted Deployed Model" deployment_files = [ ("./model_package/requirements.txt", "requirements.txt"), ("./model_package/custom.py", "custom.py"), ("./model_package/model.pkl", "model.pkl"), ] custom_model = datarobot.CustomModel( resource_name=custom_model_name, files=deployment_files, base_environment_id=base_environment_id, language="python", target_type="Binary", target_name="readmitted", ) registered_model = datarobot.RegisteredModel( resource_name=registered_model_name, custom_model_version_id=custom_model.version_id, ) deployment = datarobot.Deployment( resource_name=deployment_name, label=deployment_name, registered_model_version_id=registered_model.version_id, prediction_environment_id=default_prediction_server_id, ) pulumi.export("custom_model_id", custom_model.id) pulumi.export("registered_model_id", registered_model.id) pulumi.export("deployment_id", deployment.id) ``` ## Run the stack Running the stack takes the files that are in the `model_package` directory, puts them onto DataRobot as a custom model, registers that model, and deploys the result. ``` project_name = "AppTemplateMinis-CustomInferenceModels" stack_name = "MarshallsCustomReadmissionsPredictor" stack = stack_up(project_name, stack_name, program=make_custom_inference_deployment) ``` ### Interact with outputs ``` from datarobot_predict.deployment import predict import pandas as pd df = pd.read_csv("https://s3.amazonaws.com/datarobot_public_datasets/10k_diabetes.csv").tail(100) deployment_id = stack.outputs().get("deployment_id").value deployment = dr.Deployment.get(deployment_id) predict(deployment, data_frame=df).dataframe.head(10).iloc[:, :2] ``` ## Clear your work Use the following cell to shut down the stack, thereby deleting any assets created in DataRobot. ``` destroy_project(stack) ``` ### How does scoring code work? The following cell contains code used to upload so that DataRobot knows how to interact with our model. Deploying a custom inference model with minimal transformation only requires two hooks to be defined, but you could add [others too](https://docs.datarobot.com/en/docs/mlops/deployment/custom-models/custom-model-assembly/custom-model-components.html). Since the model is a standard scikit-learn binary classifier, DataRobot is smart enough to figure out how to interact with it without you defining any hooks. Since most model artifacts require some custom scoring logic though, you can make a `custom.py` file anyway. ``` from IPython.display import Code Code(filename="./model_package/custom.py", language="python") ``` ### What did I deploy? If you're curious how you got the fitted model in the first place, `fit_custom_model.py` shows the dataset and model fitting code. The following cell displays the code used to train and pickle the model. It's not important for running the template. ``` from IPython.display import Code Code(filename="./fit_custom_model.py", language="python") ``` --- # Deploy a custom application with Pulumi URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/pulumi-examples/deploy_dr_app.html This notebook outlines how to use Pulumi to deploy a custom application in two easy steps. ## Initialize the environment ``` import os import datarobot as dr os.environ["PULUMI_CONFIG_PASSPHRASE"] = "default" assert ( "DATAROBOT_API_TOKEN" in os.environ ), "Please set the DATAROBOT_API_TOKEN environment variable" assert "DATAROBOT_ENDPOINT" in os.environ, "Please set the DATAROBOT_ENDPOINT environment variable" dr.Client() ``` ## Set up a project Configure the functions below to create and build or destroy the Pulumi stack. ``` from pulumi import automation as auto def stack_up(project_name: str, stack_name: str, program: callable) -> auto.Stack: # create (or select if one already exists) a stack that uses our inline program stack = auto.create_or_select_stack( stack_name=stack_name, project_name=project_name, program=program ) stack.refresh(on_output=print) stack.up(on_output=print) return stack def destroy_project(stack: auto.Stack): """Destroy pulumi project""" stack_name = stack.name stack.destroy(on_output=print) stack.workspace.remove_stack(stack_name) print(f"stack {stack_name} in project removed") ``` ## 2. Declarative App Deployment To deploy a custom application, you have to put your source code onto DataRobot and initialize the deployment. The `make_custom_application` function below shows the declarative way to do this. ``` import pulumi import pulumi_datarobot as datarobot def make_custom_application(): """Make a custom app on DataRobot. Upload source code to create source. Then initialize application. """ file_mapping = [ ("frontend/app.py", "app.py"), ("frontend/requirements.txt", "requirements.txt"), ("frontend/start-app.sh", "start-app.sh"), ] app_source = datarobot.ApplicationSource( resource_name="App Template Minis - Custom App Source", files=file_mapping, base_environment_id="6542cd582a9d3d51bf4ac71e", # Python 3.9 streamlit environment ) app = datarobot.CustomApplication( resource_name="App Template Minis - Custom App", source_version_id=app_source.version_id, ) pulumi.export("Application Source Id", app_source.id) pulumi.export("Application Id", app.id) pulumi.export("Application Url", app.application_url) ``` ## Run the stack Running the stack takes the files that are in the `frontend` directory, puts them onto DataRobot, and initializes the application. ``` project_name = "AppTemplateMinis-CustomApplications" stack_name = "MarshallsCustomApplicationDeployer" stack = stack_up(project_name, stack_name, program=make_custom_application) ``` ### Interact with outputs ``` import webbrowser outputs = stack.outputs() app_url = outputs.get("Application Url").value webbrowser.open(app_url) ``` ## Clear your work Use the following cell to shut down the stack, thereby deleting any assets created in DataRobot. ``` destroy_project(stack) ``` --- # Pulumi code examples URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-code-examples/pulumi-examples/index.html > Review comprehensive workflows, notebooks, and tutorials that help you find complete examples of how to execute Pulumi tasks with DataRobot. The API user guide includes overviews and workflows for DataRobot's Python client that outline complete examples of common data science and machine learning workflows. Be sure to review the [Developer quickstart](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html) before using the notebooks below. | Topic | Describes... | | --- | --- | | Bolt-on governance with Pulumi | How to use Pulumi to create a deployment endpoint that interfaces with a large language model. | | Deploy a custom application with Pulumi | How to use Pulumi to deploy a custom application. | | Deploy a custom model with Pulumi | How to use Pulumi to deploy a scikit learn classifier. | --- # Troubleshooting the Python client URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/py-help.html > Review cases that can cause issues with using the Python client and known fixes. This page outlines cases that can cause issues with using the Python client and provides known fixes. ### InsecurePlatformWarning Python versions earlier than 2.7.9 might report an [InsecurePlatformWarning](https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning) in your output. To prevent this warning without updating your Python version, you should install the [pyOpenSSL](https://urllib3.readthedocs.org/en/latest/security.html#pyopenssl) package: `pip install pyopenssl ndg-httpsclient pyasn1` ### Unable to retrieve multiclass metrics The Python client does not compute the confusion matrix metrics for multiclass projects with more than 100 target classes. That is, the metrics object typically obtained using `get_confusion_chart().class_metrics` (as shown in [the API documentation](https://datarobot-public-api-client.readthedocs-hosted.com/en/latest-release/autodoc/api_reference.html?highlight=get_confusion_chart#datarobot.models.BlenderModel.get_confusion_chart)) is empty in such cases. In order to retrieve these metrics, DataRobot recommends using [this code snippet](https://gist.github.com/Templarrr/e40059c00b7d65f1f2c04f85ebb44c17). ### AttributeError: 'EntryPoint' object has no attribute 'resolve' Some earlier versions of [setuptools](https://setuptools.pypa.io/en/latest/) cause an error when importing DataRobot. ``` >>> import datarobot as dr ... File "/home/clark/.local/lib/python2.7/site-packages/trafaret/__init__.py", line 1550, in load_contrib trafaret_class = entrypoint.resolve() AttributeError: 'EntryPoint' object has no attribute 'resolve' ``` The recommended fix is upgrading setuptools to the latest version. `pip install --upgrade setuptools` If you are unable to upgrade, pin [trafaret](https://pypi.python.org/pypi/trafaret/) to version <=7.4 to correct this issue. ### Connection errors `configuration.rst` describes how to configure the DataRobot client with the `max_retries` parameter to fine tune behaviors like the number of attempts to retry failed connections. ### ConnectTimeout If you have a slow connection to your DataRobot installation, you may see a traceback like: ``` ConnectTimeout: HTTPSConnectionPool(host='my-datarobot.com', port=443): Max retries exceeded with url: /api/v2/projects/ (Caused by ConnectTimeoutError(, 'Connection to my-datarobot.com timed out. (connect timeout=6.05)')) ``` ### project.open_leaderboard_browser Calling `project.open_leaderboard_browser` may be blocked if you run it with a text-mode browser or on a server that doesn't have the ability to open a browser. --- # Use Cases URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/use_cases/index.html The Use Cases section provides details on how to utilize and manage DataRobot Use Cases in your Python code. --- # Use cases URL: https://docs.datarobot.com/en/docs/api/dev-learning/python/use_cases/use_cases.html Use Cases are folder-like containers in DataRobot Workbench that allow you to group all assets related to solving a specific business problem inside of a single, manageable entity. These assets include datasets, models, experiments, No-Code AI Apps, and notebooks. You can share entire Use Cases or the individual assets they contain. The primary benefit of a Use Case is that it enables experiment-based, iterative workflows. By housing all key insights in a single location, data scientists have improved navigation of assets and a cleaner interface for experiment creation and model training, review, and evaluation. Specifically, Use Cases allow you to: - Organize your work — group all related datasets, experiments, notebooks, etc. by the problem they solve. - Find assets easily. Use Cases eliminate the need to search through hundreds of unrelated projects or scrape emails for hyperlinks to specific assets. - Share collections of assets. You can share entire Use Cases, containing all the assets your team needs to participate. - Manage access. Add or remove members to a Use Case to control their access. - Monitor changes. Receive notifications when a team member adds, removes, or modifies any asset in a Use Case. Currently, Use Cases in the Python client support interactions with binary classification and regression projects, applications, and datasets. Development is ongoing, so see the release notes for a full list of supported capabilities. For a more in-depth look at Use Cases and the DataRobot Workbench, [refer to the Workbench documentation.](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/index.html) # Add to a Use Case Currently, only project, dataset, and application instances can be added to a Use Case via the Python client. The process of adding a dataset is shown in the example below: ``` import datarobot as dr dr.Client(token="", endpoint="https://app.datarobot.com/api/v2") risk_use_case = dr.UseCase.create( name="Financial Risk Experimentation Environment", description="For running experiments on modeling financial risks to our business.", ) new_dataset = dr.Dataset.create_from_file( file_path="/foo/bar/risk_data.csv", ) risk_use_case.add(entity=new_dataset) risk_use_case.list_datasets() >>> [Dataset(name='risk_data.csv', id='646e8bb507b108ce7b474b27')] ``` You can add an application to a Use Case in a similar way. The primary difference is that you cannot create applications with the Python client. Instead, retrieve an application using its ID or pull it from a retrieved list of applications and then add it to a Use Case: ``` import datarobot as dr dr.Client(token="", endpoint="https://app.datarobot.com/api/v2") risk_use_case = dr.UseCase.create( name="Financial Risk Experimentation Environment", description="For running experiments on modeling financial risks to our business.", ) existing_application = dr.Application.list()[0] risk_use_case.add(entity=existing_application) risk_use_case.list_applications() >>> [Application(name='Financial Risk Detection')] ``` Alternatively, the [UseCaseReferenceEntity](https://docs.datarobot.com/en/docs/api/reference/sdk/use-cases.html#datarobot.models.use_cases.use_case.UseCaseReferenceEntity) returned from [UseCase.add](https://docs.datarobot.com/en/docs/api/reference/sdk/use-cases.html#datarobot.UseCase.add) can be used to share an entity between Use Cases: ``` import datarobot as dr dr.Client(token="", endpoint="https://app.datarobot.com/api/v2") risk_use_case_1 = dr.UseCase.create( name="Financial Risk Experimentation Environment", description="For running experiments on modeling financial risks to our business.", ) risk_use_case_2 = dr.UseCase.create( name="Financial Risk Experimentation Environment 2", description="For running experiments on modeling financial risks to our business.", ) new_dataset = dr.Dataset.create_from_file( file_path="/foo/bar/risk_data.csv", ) dataset_entity = risk_use_case_1.add(entity=new_dataset) risk_use_case_2.add(entity=dataset_entity) risk_use_case_2.list_datasets() >>> [Dataset(name='risk_data.csv', id='646e8bb507b108ce7b474b27')] ``` To add a project to a Use Case, it must meet the following conditions: - It must be binary classification or regression project - The associated dataset must be linked to the same Use Case - Modeling must be in progress (via UI, the analyze_and_model method, or any other methods that initiate modeling) ``` import datarobot as dr dr.Client(token="", endpoint="https://app.datarobot.com/api/v2") risk_use_case = dr.UseCase.create( name="Financial Risk Experimentation Environment", description="For running experiments on modeling financial risks to our business.", ) new_dataset = dr.Dataset.create_from_file( file_path="/foo/bar/risk_data.csv", use_case=risk_use_case ) risk_use_case.add(entity=new_dataset) new_project = dr.Project.create_from_dataset( dataset_id=new_dataset.dataset_id, project_name="Risk Assessment v1", use_case=risk_use_case ) new_project.analyze_and_model(target="credit_risk") risk_use_case.add(entity=new_project) risk_use_case.list_projects() >>> [Project(Risk Assessment v1)] risk_use_case.list_datasets() >>> [Dataset(name='risk_data.csv', id='646e8bb507b108ce7b474b27')] ``` # Configuration There are three primary ways of adding new projects or datasets to Use Cases once they’ve been generated. 1. The easiest method is to directly pass a Use Case to one of the project or dataset creation methods. Passing the use case directly allows for you to finely control what is added to a Use Case in your code. For example, the following code example creates a new Use Case, then creates a new project that is automatically added to the Use Case. ``` import datarobot as dr dr.Client(token="", endpoint="https://app.datarobot.com/api/v2") risk_use_case = dr.UseCase.create( name="Financial Risk Experimentation Environment", description="For running experiments on modeling financial risks to our business.", ) new_project = dr.Project.create( sourcedata="/foo/bar/risk_data.csv", project_name="Risk Assessment v1", use_case=risk_use_case ) risk_use_case.list_projects() >>> [Project(Risk Assessment v1)] ``` 1. You can also use a context manager to perform a series of actions that automatically result in projects or datasets being added to a Use Case without having to manually pass the Use Case yourself. This can be extremely useful if you have a series of calls you want to make that all should be added to a Use Case. For example: ``` import datarobot as dr dr.Client(token="", endpoint="https://app.datarobot.com/api/v2") risk_use_case = dr.UseCase.create( name="Financial Risk Experimentation Environment", description="For running experiments on modeling financial risks to our business.", ) with risk_use_case: new_dataset = dr.Dataset.create_from_file( file_path="/foo/bar/risk_data.csv", ) risk_use_case.list_datasets() >>> [Dataset(name='risk_data.csv', id='646e8bb507b108ce7b474b27')] ``` 1. You can also set a global Use Case to automatically add all project and dataset instances that are created by your code. This is useful if all of the work you are doing should be contained in a single Use Case, but risks accidentally adding projects and datasets that should not be included in your Use Case. Setting a global default Use Case requires knowing the ID of your Use Case ahead of time. For example: ``` import datarobot as dr dr.Client(token="", endpoint="https://app.datarobot.com/api/v2", default_use_case="639ce542862e9b1b1bfa8f1b") new_dataset = dr.Dataset.create_from_file(file_path="/foo/bar/risk_data.csv") risk_use_case = dr.UseCase.get(id="639ce542862e9b1b1bfa8f1b") risk_use_case.list_datasets() >>> [Dataset(name='risk_data.csv', id='646e8bb507b108ce7b474b27')] ``` # Sharing ## Overview Instances of [datarobot.models.sharing.SharingRole](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.sharing.SharingRole) can be created to define a new role grant (or revocation). The [UseCase.share()](https://docs.datarobot.com/en/docs/api/reference/sdk/use-cases.html#datarobot.UseCase.share) instance method takes a list of [SharingRole](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.sharing.SharingRole) as its only argument. Calling this method will apply the list of [SharingRoles](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.sharing.SharingRole) to the given [UseCase](https://docs.datarobot.com/en/docs/api/reference/sdk/use-cases.html#datarobot.UseCase). Use Cases support `SHARING_ROLE.OWNER`, `SHARING_ROLE.EDITOR`, `SHARING_ROLE.CONSUMER` and `SHARING_ROLE.NO_ROLE` as possible `role` designations (see `datarobot.enums.SHARING_ROLE`). Currently, the only supported `SHARING_RECIPIENT_TYPE` is `USER`. ## Examples Suppose you had a list of user IDs you wanted to share this Use Case with. You could use a loop to generate a list of SharingRole objects for them, and bulk share this Use Case. ``` >>> from datarobot.models.use_cases.use_case import UseCase >>> from datarobot.models.sharing import SharingRole >>> from datarobot.enums import SHARING_ROLE, SHARING_RECIPIENT_TYPE >>> >>> user_ids = ["60912e09fd1f04e832a575c1", "639ce542862e9b1b1bfa8f1b", "63e185e7cd3a5f8e190c6393"] >>> sharing_roles = [] >>> for user_id in user_ids: ... new_sharing_role = SharingRole( ... role=SHARING_ROLE.CONSUMER, ... share_recipient_type=SHARING_RECIPIENT_TYPE.USER, ... id=user_id, ... can_share=True, ... ) ... sharing_roles.append(new_sharing_role) >>> use_case = UseCase.get(use_case_id="5f33f1fd9071ae13568237b2") >>> use_case.share(roles=sharing_roles) ``` Similarly, a [SharingRole](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.sharing.SharingRole) instance can be used to remove a user’s access if the `role` is set to `SHARING_ROLE.NO_ROLE`, like in this example: ``` >>> from datarobot.models.use_cases.use_case import UseCase >>> from datarobot.models.sharing import SharingRole >>> from datarobot.enums import SHARING_ROLE, SHARING_RECIPIENT_TYPE >>> >>> user_to_remove = "foo.bar@datarobot.com" ... remove_sharing_role = SharingRole( ... role=SHARING_ROLE.NO_ROLE, ... share_recipient_type=SHARING_RECIPIENT_TYPE.USER, ... username=user_to_remove, ... can_share=False, ... ) >>> use_case = UseCase.get(use_case_id="5f33f1fd9071ae13568237b2") >>> use_case.share(roles=[remove_sharing_role]) ``` # Looking beyond a Use Case Use Cases are a powerful tool for organizing your work, and can help if you need to focus only on those resources relevant to a specific business problem. However, occasionally you may want to look outside of a Use Case at other available DataRobot resources. The following code snippet demonstrates how to retrieve all Projects that your user has access to: ``` import datarobot as dr from datarobot.client import client_configuration with client_configuration(default_use_case=[]): all_projects = dr.Project.list() ``` --- # REST API code examples URL: https://docs.datarobot.com/en/docs/api/dev-learning/restapi/index.html > Review comprehensive workflows, notebooks, and tutorials that help you find complete examples of common data science and machine learning workflows. The API user guide includes overviews and workflows for DataRobot's REST API that outline complete examples of common data science and machine learning workflows. Be sure to review the [Developer quickstart](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html) before using the notebooks below. | Topic | Describes... | | --- | --- | | Create a multiseries project | How to initiate a DataRobot project for a multiseries time series problem using the DataRobot REST API. | | Create a clustering project | How to create a clustering project and initiate Autopilot in Manual mode via DataRobot's REST API. | | Fetch metadata from prediction jobs | How to retrieve metadata from prediction jobs with DataRobot's REST API. | --- # Create a multiseries project URL: https://docs.datarobot.com/en/docs/api/dev-learning/restapi/multi-rest.html This notebook outlines how to create a DataRobot project and begin modeling for a multiseries time series project with DataRobot's REST API. ### Requirements - DataRobot recommends Python version 3.7 or later. However, this workflow is compatible with earlier versions. - DataRobot API version 2.28.0 Small adjustments may be required depending on the Python version and DataRobot API version you are using. This notebook does not include a dataset and references several unknown columns; however, you can extrapolate to use your own series identifier. You can also reference [documentation for the DataRobot REST API](https://docs.datarobot.com/en/docs/api/reference/public-api/index.html). ### Import libraries ``` import datetime import json import time from pandas.io.json import json_normalize import requests import yaml ``` ### Set credentials ``` FILE_CREDENTIALS = "path-to-drconfig.yaml" parsed_file = yaml.load(open(FILE_CREDENTIALS), Loader=yaml.FullLoader) DR_ENDPOINT = parsed_file["endpoint"] API_TOKEN = parsed_file["token"] AUTH_HEADERS = {"Authorization": "token %s" % API_TOKEN} ``` ### Define functions The functions below handle responses, including asynchronous calls. ``` def wait_for_async_resolution(status_url): status = False while status == False: resp = requests.get(status_url, headers=AUTH_HEADERS) r = json.loads(resp.content) try: statusjob = r["status"].upper() except: statusjob = "" if resp.status_code == 200 and statusjob != "RUNNING" and statusjob != "INITIALIZED": status = True print("Finished: " + str(datetime.datetime.now())) return resp print("Waiting: " + str(datetime.datetime.now())) time.sleep(10) # Delays for 10 seconds. def wait_for_result(response): assert response.status_code in (200, 201, 202), response.content if response.status_code == 200: data = response.json() elif response.status_code == 201: status_url = response.headers["Location"] resp = requests.get(status_url, headers=AUTH_HEADERS) assert resp.status_code == 200, resp.content data = resp.json() elif response.status_code == 202: status_url = response.headers["Location"] resp = wait_for_async_resolution(status_url) data = resp.json() return data ``` ### Create the project Endpoint: `POST /api/v2/projects/` ``` FILE_DATASET = "/Volumes/GoogleDrive/My Drive/Datasets/Store Sales/STORE_SALES-TRAIN-2022-04-25.csv" ``` ``` payload = { # 'projectName': 'TestRESTTimeSeries_1', "file": ("Test_REST_TimeSeries_12", open(FILE_DATASET, "r")) } response = requests.post( "%s/projects/" % (DR_ENDPOINT), headers=AUTH_HEADERS, files=payload, timeout=180 ) response ``` ``` ``` ``` # Wait for async task to complete print("Uploading dataset and creating Project...") projectCreation_response = wait_for_result(response) project_id = projectCreation_response["id"] print("\nProject ID: " + project_id) ``` ``` Uploading dataset and creating Project... Waiting: 2022-07-29 17:55:32.507696 Waiting: 2022-07-29 17:55:43.092965 Waiting: 2022-07-29 17:55:53.670669 Waiting: 2022-07-29 17:56:04.252294 Waiting: 2022-07-29 17:56:14.841809 Finished: 2022-07-29 17:56:25.650896 Project ID: 62e402f1ce8ba47b224fcea3 ``` ### Update the project Endpoint: `PATCH /api/v2/projects/(projectId)/` ``` payload = {"workerCount": 16} response = requests.patch( "%s/projects/%s/" % (DR_ENDPOINT, project_id), headers=AUTH_HEADERS, json=payload, timeout=180 ) response ``` ``` ``` ### Run a detection job For a multiseries project, you must run a detection job to analyze the relationship between the partition and multiseries ID columns. Endpoint: `POST /api/v2/projects/(projectId)/multiseriesProperties/` ``` payload = {"datetimePartitionColumn": "Date", "multiseriesIdColumns": ["Store"]} response = requests.post( "%s/projects/%s/multiseriesProperties/" % (DR_ENDPOINT, project_id), headers=AUTH_HEADERS, json=payload, timeout=180, ) response ``` ``` ``` ``` print("Analyzing multiseries partitions...") multiseries_response = wait_for_result(response) ``` ``` Analyzing multiseries partitions... Waiting: 2022-07-29 17:56:27.571064 Waiting: 2022-07-29 17:56:38.156104 Finished: 2022-07-29 17:56:48.932686 ``` ### Initiate modeling Endpoint: `PATCH /api/v2/projects/(projectId)/aim/` ``` payload = { "target": "Sales", "mode": "quick", "datetimePartitionColumn": "Date", "featureDerivationWindowStart": -25, "featureDerivationWindowEnd": 0, "forecastWindowStart": 1, "forecastWindowEnd": 12, "numberOfBacktests": 2, "useTimeSeries": True, "cvMethod": "datetime", "multiseriesIdColumns": ["Store"], "blendBestModels": False, } response = requests.patch( "%s/projects/%s/aim/" % (DR_ENDPOINT, project_id), headers=AUTH_HEADERS, json=payload, timeout=180, ) response ``` ``` ``` ``` print("Waiting for tasks previous to training to complete...") autopilot_response = wait_for_result(response) ``` ``` Waiting for tasks previous to training to complete... Waiting: 2022-07-29 17:56:51.024036 Waiting: 2022-07-29 17:57:01.746376 Waiting: 2022-07-29 17:57:12.329879 Waiting: 2022-07-29 17:57:22.904449 Waiting: 2022-07-29 17:57:33.679282 Waiting: 2022-07-29 17:57:44.262096 Waiting: 2022-07-29 17:57:54.845494 Waiting: 2022-07-29 17:58:05.427372 Waiting: 2022-07-29 17:58:15.995107 Waiting: 2022-07-29 17:58:26.605621 Waiting: 2022-07-29 17:58:37.188681 Waiting: 2022-07-29 17:58:47.762809 Waiting: 2022-07-29 17:58:58.348806 Waiting: 2022-07-29 17:59:08.925445 Waiting: 2022-07-29 17:59:19.505174 Waiting: 2022-07-29 17:59:30.093026 Waiting: 2022-07-29 17:59:40.670835 Waiting: 2022-07-29 17:59:51.239278 Waiting: 2022-07-29 18:00:01.818356 Waiting: 2022-07-29 18:00:12.395658 Waiting: 2022-07-29 18:00:22.993393 Waiting: 2022-07-29 18:00:33.576738 Waiting: 2022-07-29 18:00:44.166028 Waiting: 2022-07-29 18:00:54.768693 Waiting: 2022-07-29 18:01:05.372862 Waiting: 2022-07-29 18:01:15.981022 Waiting: 2022-07-29 18:01:26.571205 Waiting: 2022-07-29 18:01:37.160074 Waiting: 2022-07-29 18:01:47.741388 Waiting: 2022-07-29 18:01:58.326862 Waiting: 2022-07-29 18:02:08.912622 Finished: 2022-07-29 18:02:19.739789 ``` --- # Fetch metadata from prediction jobs URL: https://docs.datarobot.com/en/docs/api/dev-learning/restapi/pred-metadata.html This notebook outlines how to retrieve metadata from prediction jobs with DataRobot's REST API. In the DataRobot UI, you can see prediction jobs on the Deployments page; this list includes all batch prediction jobs made from REST API code, through DataRobot's Python API client, or from job definitions. Using DataRobot's REST API, you can get more details on each of those predictions; however, you need to use Python to complete this task. ## Setup ### Import libraries ``` import getpass import os import datarobot as dr import pandas as pd import requests print(os.getcwd()) token = getpass.getpass() # Use your own token dr.Client(token=token, endpoint="https://app.datarobot.com/api/v2") ``` ### Connect to DataRobot Read more about different options for [connecting to DataRobot from the client](https://docs.datarobot.com/en/docs/api/api-quickstart/api-qs.html). ``` API_ENDPOINT = "https://app.datarobot.com/api/v2/batchPredictions" # Enter your API key here API_KEY = token session = requests.Session() session.headers = { "Authorization": "Bearer {}".format(API_KEY), } session.close() ``` ### Fetch metadata Use the snippet below to get metadata from your prediction jobs. The following cell displays an example of what the retrieved data looks like. ``` resp = session.get(API_ENDPOINT) print(resp.status_code) df = pd.json_normalize(resp.json()["data"]) df.head() ``` ### Fetch data points ``` log1 = pd.DataFrame(df.iloc[1,]) with pd.option_context( "display.max_rows", 1000, "display.max_columns", 1000 ): # more options can be specified also display(log1) ``` By analyzing the data points above, you can identify numerous insights: - Status details is missing columns - The Source field is using the UI prediction method (i.e., not using job definitions or the Python API client for batch predictions) - DatasetID and DeploymentID are provided, which you can use for further analysis or configuration ### Use metadata for troubleshooting You can use the metadata to inform other users about any issues or failures with predictions job and provide additional useful information to help resolve the issue. DataRobot recommends providing the following to troubleshoot: - URL to the dataset used for prediction - URL to the deployment - URL to the project - URL to the dataset used for training The prediction dataset can be fetched from the prediction job metadata in the cells above. It also provides the dataset ID. Run the following snippet to retrieve a URL that will direct you to the dataset. ``` datasetid = "5ebc89d21b7b850de6ab9a36" dataset = dr.Dataset.get(datasetid) print(dataset) print("https://app.datarobot.com/ai-catalog/" + datasetid) ``` Provide the deployment ID and then run the following snippet to retrieve the URL for the deployment. ``` deploymentid = "6290a642f2d99680864daad8" deployment = dr.Deployment.get(deploymentid) print(deployment) print("https://app.datarobot.com/deployments/" + deploymentid) ``` Use the following cell to get the project ID using the deployment ID. ``` # The deployment ID deploymentid = "6290a642f2d99680864daad8" # Define the API endpoint API_ENDPOINT = "https://app.datarobot.com/api/v2/deployments/" # Provide your API key here API_KEY = token session = requests.Session() session.headers = { "Authorization": "Bearer {}".format(API_KEY), } session.close() ``` Provide the project ID and then run the following snippet to retrieve the URL for the project. ``` projectid = "62908fa8929e0d7ef66e388e" project = dr.Project.get(projectid) print(project) print("https://app.datarobot.com/projects/" + projectid) ``` Lastly, by pulling the URL for the dataset used for training and using the project ID above, you can get the training dataset ID. ``` # The deployment ID projectid = "62908fa8929e0d7ef66e388e" # Define the API endpoint API_ENDPOINT = "https://app.datarobot.com/api/v2/projects/" # Provide your API key here API_KEY = token session = requests.Session() session.headers = { "Authorization": "Bearer {}".format(API_KEY), } resp = session.get(API_ENDPOINT + "?projectId=" + projectid) df = pd.json_normalize(resp.json()) df.T session.close() ``` ``` datasetid = "629086ace265bd23ab9c1de7" print("https://app.datarobot.com/ai-catalog/" + datasetid) ``` Click the link from the output above to access the dataset. --- # Create a clustering project URL: https://docs.datarobot.com/en/docs/api/dev-learning/restapi/rest-cluster.html This notebook outlines how to create a clustering project and initiate Autopilot in Manual mode via DataRobot's REST API. Manual mode allows you to select and train specific blueprints for modeling. If you run a clustering project in comprehensive Autopilot mode, some blueprints may take a long time to complete. For example, HDBSCAN is inherently a slow model to train. Because of these time constraints, this notebook only runs one blueprint (K-Means) and tests several clusters. ### Requirements - DataRobot recommends Python version 3.7 or later. - DataRobot API version 2.28.0 ### Import libraries ``` import datetime import json import time from pandas.io.json import json_normalize import requests import yaml ``` ### Set credentials ``` FILE_CREDENTIALS = ( "/Volumes/GoogleDrive/My Drive/rodrigo.miranda/mlops-admin/rodrigo.miranda_drconfig.yaml" ) parsed_file = yaml.load(open(FILE_CREDENTIALS), Loader=yaml.FullLoader) DR_ENDPOINT = parsed_file["endpoint"] API_TOKEN = parsed_file["token"] AUTH_HEADERS = {"Authorization": "token %s" % API_TOKEN} ``` ### Define functions The functions below handle responses, including asynchronous calls. ``` def wait_for_async_resolution(status_url): status = False while status == False: resp = requests.get(status_url, headers=AUTH_HEADERS) r = json.loads(resp.content) try: statusjob = r["status"].upper() except: statusjob = "" if resp.status_code == 200 and statusjob != "RUNNING" and statusjob != "INITIALIZED": status = True print("Finished: " + str(datetime.datetime.now())) return resp print("Waiting: " + str(datetime.datetime.now())) time.sleep(10) # Delays for 10 seconds. def wait_for_result(response): assert response.status_code in (200, 201, 202), response.content if response.status_code == 200: data = response.json() elif response.status_code == 201: status_url = response.headers["Location"] resp = requests.get(status_url, headers=AUTH_HEADERS) assert resp.status_code == 200, resp.content data = resp.json() elif response.status_code == 202: status_url = response.headers["Location"] resp = wait_for_async_resolution(status_url) data = resp.json() return data ``` ### Create a project Endpoint: `POST /api/v2/projects/` ``` FILE_DATASET = ( "/Volumes/GoogleDrive/My Drive/Datasets/Customer Invoices/clustering_customer_invoices.csv" ) ``` ``` payload = {"file": ("Clustering - Customer Invoices 02", open(FILE_DATASET, "r"))} response = requests.post( "%s/projects/" % (DR_ENDPOINT), headers=AUTH_HEADERS, files=payload, timeout=60 ) response ``` ``` ``` ``` # Wait for async task to complete print("Uploading dataset and creating Project...") projectCreation_response = wait_for_result(response) project_id = projectCreation_response["id"] print("\nProject ID: " + project_id) ``` ``` Uploading dataset and creating Project... Waiting: 2022-08-09 14:54:27.008806 Waiting: 2022-08-09 14:54:37.578847 Waiting: 2022-08-09 14:54:48.139574 Waiting: 2022-08-09 14:54:58.846699 Waiting: 2022-08-09 14:55:09.401604 Waiting: 2022-08-09 14:55:19.981831 Waiting: 2022-08-09 14:55:30.551482 Finished: 2022-08-09 14:55:41.361760 Project ID: 62f25900543b1c01e5bdaf59 ``` ### Initiate Autopilot This snippet begins modeling in Manual mode. Endpoint: `PATCH /api/v2/projects/(projectId)/aim/` ``` payload = {"unsupervisedMode": True, "unsupervisedType": "clustering", "mode": "manual"} response = requests.patch( "%s/projects/%s/aim/" % (DR_ENDPOINT, project_id), headers=AUTH_HEADERS, json=payload, timeout=60, ) response ``` ``` ``` ``` print("Creating project in Manual mode...") project_response = wait_for_result(response) ``` ``` Creating project in Manual mode... Waiting: 2022-08-09 14:55:43.398565 Waiting: 2022-08-09 14:55:53.961746 Waiting: 2022-08-09 14:56:04.548413 Waiting: 2022-08-09 14:56:15.131273 Waiting: 2022-08-09 14:56:25.715646 Waiting: 2022-08-09 14:56:36.270683 Waiting: 2022-08-09 14:56:46.828606 Waiting: 2022-08-09 14:56:57.391746 Waiting: 2022-08-09 14:57:08.098635 Waiting: 2022-08-09 14:57:18.650575 Finished: 2022-08-09 14:57:29.453144 ``` ### Retrieve blueprints Endpoint: `GET /api/v2/projects/(projectId)/blueprints/` ``` response = requests.get( "%s/projects/%s/blueprints/" % (DR_ENDPOINT, project_id), headers=AUTH_HEADERS ) response ``` ``` ``` ``` r = json.loads(response.content) r ``` ``` print("Available blueprints:\n") for bp in r: print(bp["modelType"]) print(bp["id"] + "\n") ``` ``` Available blueprints: Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) 500ce93b06e38c4df2800f62ade6650d Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) DBSCAN Hybrid Model 59478b8603dd3e270bd4277a9c1456d7 Gaussian Mixture Model 68a6aa4312a27d1fc55580f7fb1121bc K-Means Clustering 9c08a327281f53fa366bb52c817499d2 ``` ### Build models Endpoint: `POST /api/v2/projects/(projectId)/models/` Next, train 3 K-Means models simultaneously using a different number of clusters. ``` payload = {"blueprintId": "9c08a327281f53fa366bb52c817499d2", "nClusters": 3} response1 = requests.post( "%s/projects/%s/models/" % (DR_ENDPOINT, project_id), headers=AUTH_HEADERS, json=payload, timeout=60, ) response1 ``` ``` ``` ``` payload = {"blueprintId": "9c08a327281f53fa366bb52c817499d2", "nClusters": 5} response2 = requests.post( "%s/projects/%s/models/" % (DR_ENDPOINT, project_id), headers=AUTH_HEADERS, json=payload, timeout=60, ) response2 ``` ``` ``` ``` payload = {"blueprintId": "9c08a327281f53fa366bb52c817499d2", "nClusters": 10} response3 = requests.post( "%s/projects/%s/models/" % (DR_ENDPOINT, project_id), headers=AUTH_HEADERS, json=payload, timeout=60, ) response3 ``` ``` ``` ``` print("Waiting for models training to finish...") ``` ``` Waiting for models training to finish... Finished: 2022-08-09 15:06:54.415661 Finished: 2022-08-09 15:06:55.300248 Finished: 2022-08-09 15:06:56.155223 ``` --- # Developer documentation URL: https://docs.datarobot.com/en/docs/api/index.html > Use the REST and Python APIs and a variety of code-first tools to develop models. DataRobot supports REST, Python, and R APIs as a programmatic alternative to the UI for creating and managing DataRobot projects. It allows you to automate processes and iterate more quickly, and lets you use DataRobot with scripted control. The API provides an intuitive modeling and prediction interface. You can use the API with DataRobot—supported clients in either R or Python, or with your own custom code. The clients are supported in Windows, UNIX, and OS X environments. Additionally, you can generate predictions with the prediction and batch prediction APIs, and build DataRobot blueprints in the blueprint workshop. - Developer learning¶ Developer quickstart, tutorials, and notebooks for code-first work on DataRobot. - API reference¶ Access documentation for DataRobot APIs and packages. - DataRobot CLI¶ Access documentation for using and developing the DataRobot CLI. - Agent Assist¶ Access documentation for using the interactive agent-building assistant. - Code-first tools¶ Access documentation for the code-first tools provided by DataRobot. --- # Time series URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/batch-pred-ts.html > Outlines how to set up batch predictions for time series models. Includes settings details and code examples. Batch predictions for time series models work without any additional configuration. However, in most cases you need to either modify the default configuration or prepare the prediction dataset. ## Time series batch prediction settings The default configuration can be overridden using the `timeseriesSettings` job configuration property: | Parameter | Example | Description | | --- | --- | --- | | type | forecast | Must be either forecast (default) or historical. | | forecastPoint | 2019-02-04T00:00:00Z | (Optional) By default, DataRobot infers the forecast point from the dataset. To configure, type must be set to forecast. | | predictionsStartDate | 2019-01-04T00:00:00Z | (Optional) By default, DataRobot infers the start date from the dataset. To configure, type must be set to historical. | | predictionsEndDate | 2019-02-04T00:00:00Z | (Optional) By default, DataRobot infers the end date from the dataset. To configure, type must be set to historical. | | relaxKnownInAdvanceFeaturesCheck | false | (Optional) If activated, missing values in the known in advance features are allowed in the forecast window at prediction time. If omitted or false, missing values are not allowed. Default: false. | Here is a complete example job: ``` { "deploymentId": "5f22ba7ade0f435ba7217bcf", "intakeSettings": {"type": "localFile"}, "outputSettings": {"type": "localFile"}, "timeseriesSettings": { "type": "historical", "predictionsStartDate": "2020-01-01", "predictionsEndDate": "2020-03-31" } } ``` An example using the Python API client: ``` import datarobot as dr dr.Client( endpoint="https://app.datarobot.com/api/v2", token="...", ) deployment_id = "..." input_file = "to_predict.csv" output_file = "predicted.csv" job = dr.BatchPredictionJob.score_to_file( deployment_id, input_file, output_file, timeseries_settings={ "type": "historical", "predictions_start_date": "2020-01-01", "predictions_end_date": "2020-03-31", }, ) print("started scoring...", job) job.wait_for_completion() ``` ## Prediction type When using `forecast` mode, DataRobot makes predictions using `forecastPoint` or rows in the dataset without a target. In `historical` mode, DataRobot enables bulk predictions, which calculates predictions for all possible forecast points and forecast distances within `predictionsStartDate` and `predictionsEndDate` range. ## Requirements for the scoring dataset To ensure the Batch Prediction API can process your time series dataset, you must configure the following: - Sort prediction rows by their timestamps, with the earliest row first. - There is no limit on the number of series DataRobot supports. The only limit is the job timeout as mentioned in Limits . ### Single series forecast dataset example The following is an example forecast dataset for a single series: | date | y | | --- | --- | | 2020-01-01 | 9342.85 | | 2020-01-02 | 4951.33 | | 24 more historical rows | | | 2020-01-27 | 4180.92 | | 2020-01-28 | 5943.11 | | 2020-01-29 | | | 2020-01-30 | | | 2020-01-31 | | | 2020-02-01 | | | 2020-02-02 | | | 2020-02-03 | | | 2020-02-04 | | ### Multiseries forecast dataset example If scoring multiple series, the data must be ordered by series and timestamp: | date | series | y | | --- | --- | --- | | 2020-01-01 | A | 9342.85 | | 2020-01-02 | A | 4951.33 | | 24 more historical rows | | | | 2020-01-27 | A | 4180.92 | | 2020-01-28 | A | 5943.11 | | 2020-01-29 | A | | | 2020-01-30 | A | | | 2020-01-31 | A | | | 2020-02-01 | A | | | 2020-02-02 | A | | | 2020-02-03 | A | | | 2020-02-04 | A | | | 2020-01-01 | B | 8477.22 | | 2020-01-02 | B | 7210.29 | | 24 more historical rows | | | | 2020-01-27 | B | 7400.21 | | 2020-01-28 | B | 8844.71 | | 2020-01-29 | B | | | 2020-01-30 | B | | | 2020-01-31 | B | | | 2020-02-01 | B | | | 2020-02-02 | B | | | 2020-02-03 | B | | | 2020-02-04 | B | | --- # Troubleshooting Batch Prediction jobs URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/batch-pred-tshoot.html > A list of common issues that occur with Batch Prediction jobs, and how to resolve them. The following lists some common issues and how to resolve them. ## A job is stuck in INITIALIZING If using local file intake, make sure you have made a `PUT` request with the scoring data for the job after the initial `POST` request. DataRobot only processes one job at a time per prediction instance, so your job may be queued behind other jobs. Check the job log for details: ``` curl -X GET https://app.datarobot.com/api/v2/batchPredictions/:id/ \ -H 'Authorization: Bearer ' ``` ## A job is stuck in RUNNING The job may be running slowly, either because of a slow model or because the scoring data contains errors that the API is trying to identify. You can follow the progress of a job by requesting the job status: ``` curl -X GET https://app.datarobot.com/api/v2/batchPredictions/:id/ \ -H 'Authorization: Bearer ' ``` ## A job was ABORTED When a job is aborted, DataRobot logs the reason to the job status. You can check job status from an individual job URL: ``` curl -X GET https://app.datarobot.com/api/v2/batchPredictions/:id/ \ -H 'Authorization: Bearer ' ``` Or from the listing view of all jobs: ``` curl -X GET https://app.datarobot.com/api/v2/batchPredictions/ \ -H 'Authorization: Bearer ' ``` ## HTTP 406 was returned when uploading a CSV file for local file intake You are missing the `Content-Type: text/csv` header. ## HTTP 422 was returned when uploading a CSV file for local file intake You either: - Already pushed CSV data for this job. To submit new data, create a new job. - Tried to push CSV data for a job that does not require you to push data (e.g., S3 intake). - Didn't encode your CSV data in the UTF-8 character set and didn't specify a custom encoding in csvSettings . - Didn't encode your CSV data in the proper CSV format and didn't specify a custom format in csvSettings . - Tried to push an empty file. In any of the above cases, the response and the job log will contain an explanation. ## Intake stream error due to date format mismatch in Oracle JDBC scoring data Oracle's DATE type contains a time component, which can cause issues with scoring time series data. A model trained using the date format `yyyy-mm-dd` can result in an error for Oracle JDBC scoring data due to Oracle's DATE format. When DataRobot reads dates from Oracle, the dates are returned in the format `yyyy-mm-dd hh:mm:ss` by default. This can cause an error when passed to a model expecting a different format. Use one of the following workarounds to avoid this issue: - Train the model using Oracle as the data source to ensure that the time format is the same when scored from Oracle. - Use the query option instead of table and schema to allow for the use of SQL functions. Oracle's TO_CHAR function can be used to parse time columns before the data is scored. ## The network connection broke while uploading a dataset for local file intake Create a new job and re-upload the dataset. Failed uploads cannot be resumed and will eventually time out. ## The network connection became unavailable while downloading the scoring data for local file output Re-download the job again. The scored data is available for 48 hours on the managed AI Platform (SaaS) and for 48 hours (but configurable) on the Self-Managed AI Platform (VPC or on-prem). ## HTTP 404 was returned while trying to download scored data You either: - Tried to download the scored data for a job that does not have scored data available for download (e.g., S3 output). - Started the download before the job had started scoring. In that case, wait until the download link becomes available in the job links and try again. ## HTTP 406 was returned when trying to download scored data Your client sent an `Accept` header that did not include `text/csv`. Either do not send the `Accept` header or include `text/csv` in it. ## CREATE_TABLE scoring fails due to unsupported output column name formats You may be using a target database as your output adapter that does not support the way DataRobot generates the [output format](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html) column names. Column names such as `name (actual)_PREDICTION` when scoring Time Series models might not be supported with all databases. To work around this issue, you can utilize the [Column Name Remapping](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html#column-name-remapping) functionality to re-write the output column name to some form your target database supports. For instance, if you want to remove the spaces from a column name, you can make a request adding `columnNamesRemapping` as such: ``` { "deploymentId":"", "passthroughColumnsSet":"all", "includePredictionStatus":true, "intakeSettings":{ "type":"localFile" }, "outputSettings":{ "type":"jdbc", "dataStoreId":"", "credentialId":"", "table":"table_name_of_database", "schema":"dbo", "catalog":"test", "statementType":"create_table" }, "columnNamesRemapping":{ "name (actual)_PREDICTION":"name_actual_PREDICTION" } } ``` ## Possible causes for HTTP 422 on job creation These are the possible causes for an `HTTP 422` reply when creating a new Batch Prediction job: - You sent an unknown job parameter - You specified a job parameter with an unexpected type or value - You specified an unknown credential ID in either your intake or output settings - You are attempting to score from/to the same S3/Azure/GCP URL (not supported) - You are attempting to ingest data from the AI Catalog , but your account does not have access to the AI Catalog - You are attempting to ingest data from the AI Catalog and the AI Catalog dataset is not snapshotted (required for predictions) or has not been successfully ingested - You are attempting to use a time series custom model (not currently supported) - You are attempting to use a traditional time series (ARIMA) model (not currently supported) - You requested Prediction Explanations for a multiclass or time series project (not currently supported) - You requested prediction warnings for a project other than a regression project (not currently supported) - You requested prediction warnings for a project that is not properly configured with prediction boundaries --- # Batch Prediction API URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html > The Batch Prediction API provides flexible options for scoring large datasets using the prediction servers you have already deployed. The Batch Prediction API provides flexible options for intake and output when scoring large datasets using the prediction servers you have already deployed. The API is exposed through the DataRobot Public API. The API can be consumed using either any REST-enabled client or the [DataRobot Python Public API bindings](https://datarobot-public-api-client.readthedocs-hosted.com/page/). For more information about Batch Prediction REST API routes, view the [DataRobot REST API reference documentation](https://docs.datarobot.com/en/docs/api/reference/public-api/index.html). The main features of the API are: - Flexible options for intake and output: - Protection against prediction server overload with a concurrency control level option. - Inclusion of Prediction Explanations (with an option to add thresholds). - Support for passthrough columns to correlate scored data with source data. - Addition of prediction warnings in the output. - The ability to make predictions with files greater than 1GB via the API . For more information about making batch prediction settings for time series, [reference the time series documentation](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/batch-pred-ts.html). ## Limits | Item | AI Platform (SaaS) | Self-managed AI Platform (VPC or on-prem) | | --- | --- | --- | | Job runtime limit | 4 hours* | Unlimited | | Local file intake size | Unlimited | Unlimited | | Local file write size | Unlimited | Unlimited | | S3 intake size | Unlimited | Unlimited | | S3 write size | 100GB | 100GB (configurable) | | Azure intake size | 4.75TB | 4.75TB | | Azure write size | 195GB | 195GB | | GCP intake size | 5TB | 5TB | | GCP write size | 5TB | 5TB | | JDBC intake size | Unlimited | Unlimited | | JDBC output size | Unlimited | Unlimited | | Concurrent jobs | 1 per prediction instance | 1 per installation | | Stored data retention time For local file adapters | 48 hours | 48 hours (configurable) | * Feature Discovery projects have a job runtime limit of 6 hours. ## Concurrent jobs To ensure that the prediction server does not get overloaded, DataRobot will only run one job per prediction instance. Further jobs are queued and started as soon as previous jobs complete. ## Data pipeline A Batch Prediction job is a data pipeline consisting of: > Data Intake > Concurrent Scoring > Data Output On creation, the job's `intakeSettings` and `outputSettings` define the data intake and data output part of the pipeline. You can configure any combination of intake and output options. For both, the defaults are local file intake and output, meaning you will have to issue a separate `PUT` request with the data to score and subsequently download the scored data. ### Data sources supported for batch predictions The following table shows the data source support for batch predictions. | Name | Driver version | Intake support | Output support | DataRobot version validated | | --- | --- | --- | --- | --- | | AWS Athena 2.0 | 2.0.35 | yes | no | 7.3 | | AWS S3 | 2022.1.1670354484 | yes | yes | - | | Alibaba Cloud MaxCompute¹ | 3.6.0 | yes | yes | 11.1 | | Databricks² | 2.6.40 | yes | yes | 9.2 | | Exasol | 7.0.14 | yes | yes | 8.0 | | Google BigQuery | 1.2.4 | yes | yes | 7.3 | | InterSystems | 3.2.0 | yes | no | 7.3 | | kdb+ | - | yes | yes | 7.3 | | Microsoft SQL Server | 12.2.0 | yes | yes | 6.0 | | MySQL | 8.0.32 | yes | yes | 6.0 | | Oracle | 11.2.0 | yes | yes | 7.3 | | PostgreSQL | 42.5.1 | yes | yes | 6.0 | | Presto³ | 0.216 | yes | yes | 8.0 | | Redshift | 2.1.0.14 | yes | yes | 6.0 | | SAP HANA | 2.20.17 | yes | yes | 7.3 (intake support only) 10.1 (intake and output support) | | Snowflake | 3.15.1 | yes | yes | 6.2 | | Synapse | 12.4.1 | yes | yes | 7.3 | | Teradata⁴ | 17.10.00.23 | yes | yes | 7.3 | | TreasureData | 0.5.10 | yes | no | 7.3 | ¹ Only the "insert" write strategy is supported. Data table and column names cannot contain special characters. These names can contain letters, digits, and underscores (_); however, they must start with a letter and cannot exceed 128 bytes in length. For more information, see the [feature considerations](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/data-sources/wb-maxcompute.html#feature-considerations). ² Only the [Databricks JDBC driver](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/data-sources/dc-databricks.html) supports batch predictions. ³ Presto requires the use of `auto commit: true` for many of the underlying connectors which can delay writes. ⁴ For output to Teradata, DataRobot only supports ANSI mode. For further information, see: - Supported intake options - Supported output options - Output format schema ## Concurrent scoring When scoring, the data you supply is split into chunks and scored concurrently on the prediction instance specified by the deployment. To control the level of concurrency, modify the `numConcurrent` parameter at job creation. ## Job states When working with batch predictions, each prediction job can be in one of four states: - INITIALIZING : The job has been successfully created and is either: - RUNNING : Scoring the dataset on prediction servers has started. - ABORTED : The job was aborted because either: - COMPLETED : The dataset has been scored and: ## Store credentials securely Some sources or targets for scoring may require DataRobot to authenticate on your behalf (for example, if your database requires that you pass a username and password for login). To ensure proper storage of these credentials, you must have [data credentials](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html) enabled. DataRobot uses the following credential types and properties: | Adapter | Credential Type | Property | | --- | --- | --- | | S3 intake / output | s3 | awsAccessKeyId awsSecretAccessKey awsSessionToken (optional) | | JDBC intake / output | basic | username password | To use a stored credential, you must pass the associated `credentialId` in either `intakeSettings` or `outputSettings` as described below for each of the adapters. ## CSV format For any intake or output options that deal with reading or writing CSV files, you can use a custom format by specifying the following in `csvSettings`: | Parameter | Example | Description | | --- | --- | --- | | delimiter | , | (Optional) The delimiter character to use. Default: , (comma). To specify TAB as a delimiter, use the string tab. | | quotechar | " | (Optional) The character to use for quoting fields containing the delimiter. Default: ". | | encoding | utf-8 | (Optional) Encoding for the CSV file. For example (but not limited to): shift_jis, latin_1 or mskanji. Default: utf-8. Any Python supported encoding can be used. | The same format will be used for both intake and output. See a [complete example](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-of-csv-files-from-local-files). ## Model monitoring The Batch Prediction API integrates well with DataRobot's model monitoring capabilities: - If you have enabled data drift tracking for your deployment, any predictions run through the Batch Prediction API will be tracked. - If you have enabled target drift tracking for your deployment, the output will contain the desired association ID to be used for reporting actuals. Should you need to run a non-production dataset against your deployment, you can turn off drift and accuracy tracking for a single job by providing the following parameter: | Parameter | Example | Description | | --- | --- | --- | | skipDriftTracking | true | (Optional) Skip data drift, target drift, and accuracy tracking for this job. Default: false. | ## Override the default prediction instance Under normal circumstances, the prediction server used for scoring will be the default prediction server that your model was deployed to. It is however possible to override it If you have access to multiple prediction servers, you can override the default behavior by using the following properties in the `predictionInstance` option: | Parameter | Example | Description | | --- | --- | --- | | hostName | 192.0.2.4 | Sets the hostname to use instead of the default hostname from the prediction server the model was deployed to. | | sslEnabled | false | (Optional) Use SSL (HTTPS) to access the prediction server. Default: true. | | apiKey | NWU...IBn2w | (Optional) Use an API key different from the job creator's key to authenticate against the new prediction server. | | datarobotKey | 154a8abb-cbde-4e73-ab3b-a46c389c337b | (Optional) If running in a managed AI Platform environment, specify the per-organization DataRobot key for the prediction server. Find the key on the Deployments> Predictions > Prediction API tab or by contacting your DataRobot representative. | Here's a complete example: ``` job_details = { 'deploymentId': deployment_id, 'intakeSettings': {'type': 'localFile'}, 'outputSettings': {'type': 'localFile'}, 'predictionInstance': { 'hostName': '192.0.2.4', 'sslEnabled': False, 'apiKey': 'NWUQ9w21UhGgerBtOC4ahN0aqjbjZ0NMhL1e5cSt4ZHIBn2w', 'datarobotKey': '154a8abb-cbde-4e73-ab3b-a46c389c337b', }, } ``` ## Consistent scoring with updated model If you deploy a new model after a job has been queued, DataRobot will still use the model that was deployed at the time of job creation for the entire job. Every row will be scored with the same model. ## Template variables Sometimes it can be useful to specify dynamic parameters in your batch jobs, such as in [Job Definitions](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/job-definitions.html). You can use [jinja's variable syntax](https://jinja.palletsprojects.com/en/3.0.x/templates/#variables) (double curly braces) to print the value of the following parameters: | Variable | Description | | --- | --- | | current_run_time | datetime object for current UTC time (datetime.utcnow()) | | current_run_timestamp | Milliseconds from Unix epoch (integer) | | last_scheduled_run_time | datetime object for the start of last job instantiated from the same job definition | | next_scheduled_run_time | datetime object for the next scheduled start of job from the same job definition | | last_completed_run_time | datetime object for when the previously scheduled job finished scoring | The above variables can be used in the following fields: | Field | Condition | | --- | --- | | intake_settings.query | For JDBC, Synapse, and Snowflake adapters | | output_settings.table | For JDBC, Synapse, Snowflake, and BigQuery adapters, when statement type is create_table or create_table_if_not_exists is marked true | | output_settings.url | For S3, GCP, and Azure adapters | You should specify the URL as: `gs://bucket/output-.csv`. > [!NOTE] Note > To ensure that most databases understand the replacements mentioned above, DataRobot strips microseconds off the ISO-8601 format timestamps. ## API Reference ### The Public API The Batch Prediction API is part of the [DataRobot REST API](https://docs.datarobot.com/en/docs/api/reference/public-api/batch_predictions.html). Reference this documentation for more information about how to work with batch predictions. ### The Python API Client You can use the [Python Public API Client](https://datarobot-public-api-client.readthedocs-hosted.com/) to interface with the Batch Prediction API. --- # Prediction intake options URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html > Configure batch prediction data sources (intake) with the Job Definitions UI or the Batch Prediction API. You can configure a prediction source using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-sources) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html). This topic describes both the UI and API intake options. > [!NOTE] Note > For a complete list of supported intake options, see the [data sources supported for batch predictions](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html#data-sources-supported-for-batch-predictions). | Intake option | Description | | --- | --- | | Local file streaming | Stream input data through a URL endpoint for immediate processing when the job moves to a running state. | | HTTP scoring | Stream input data from an absolute URL for scoring. This option can read data from pre-signed URLs for Amazon S3, Azure, and Google Cloud Platform. | | Database connections | | | JDBC scoring | Read prediction data from a JDBC-compatible database with data source details supplied through a job definition or the Batch Prediction API. | | SAP Datasphere scoring | Read prediction data from a SAP Datasphere database with data source details supplied through a job definition or the Batch Prediction API. | | Trino scoring | Read prediction data from a Trino database with data source details supplied through a job definition or the Batch Prediction API. | | Cloud storage connections | | | Azure Blob Storage scoring | Read input data from Azure Blob Storage with DataRobot credentials consisting of an Azure Connection String. | | Google Cloud Storage scoring (GCP) | Read input data from Google Cloud Storage with DataRobot credentials consisting of a JSON-formatted account key. | | Amazon S3 scoring | Read input data from public or private S3 buckets with DataRobot credentials consisting of an access key (ID and key) and a session token (Optional). This is the preferred intake option for larger files. | | Data warehouse connections | | | BigQuery scoring | Score data using BigQuery with data source details supplied through a job definition or the Batch Prediction API. | | Snowflake scoring | Score data using Snowflake with data source details supplied through a job definition or the Batch Prediction API. | | Azure Synapse scoring | Score data using Synapse with data source details supplied through a job definition or the Batch Prediction API. | | Other connections | | | AI Catalog / Data Registry dataset scoring | Read input data from a dataset snapshot in the DataRobot AI Catalog / Data Registry. | | Wrangler Recipe scoring | Read input data from a wrangler recipe created in the DataRobot Workbench from a Snowflake data connection. | If you are using a custom [CSV format](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html#csv-format), any intake option dealing with CSV will adhere to that format. ## Local file streaming Local file intake does not have any special options. This intake option requires you to upload the job's scoring data using a `PUT` request to the URL specified in the `csvUpload` link in the job data. This starts the job (or queues it for processing if the prediction instance is already occupied). If there is no other queued job for the selected prediction instance, scoring will start while you are still uploading. Refer to [this sample use case](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-of-csv-files-from-local-files). > [!NOTE] Note > If you forget to send scoring data, the job remains in the INITIALIZING state. ### Multipart upload Because the local file intake process requires that you upload scoring data for a job using a `PUT` request to the URL specified in the `csvUpload` parameter, by default, a single `PUT` request starts the job (or queues it for processing if the prediction instance is occupied). Multipart upload for batch predictions allows you to override the default behavior to upload scoring data through multiple files. This upload process requires multiple `PUT` requests followed by a single `POST` request ( `finalizeMultipart`) to finalize the multipart upload manually. This feature can be helpful when you want to upload large datasets over a slow connection or if you experience frequent network instability. > [!NOTE] Note > For more information on the batch prediction API and local file intake, see [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) and [Prediction intake options](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html#local-file-streaming). #### Multipart upload endpoints This feature adds the following multipart upload endpoints to the batch prediction API: | Endpoint | Description | | --- | --- | | PUT /api/v2/batchPredictions/:id/csvUpload/part/0/ | Upload scoring data in multiple parts to the URL specified by csvUpload. Increment 0 by 1 in sequential order for each part of the upload. | | POST /api/v2/batchPredictions/:id/csvUpload/finalizeMultipart/ | Finalize the multipart upload process. Make sure each part of the upload has finished before finalizing. | #### Local file intake settings The intake settings for the local file adapter added two new properties to support multipart upload for the batch prediction API: | Property | Type | Default | Description | | --- | --- | --- | --- | | intakeSettings.multipart | boolean | false | true: Requires you to submit multiple files via a PUT request and finalize the process manually via a POST request (finalizeMultipart).false: Finalizes intake after one file is submitted via a PUT request. | | intakeSettings.async | boolean | true | true: Starts the scoring job when the initial PUT request for file intake is made.false: Postpones the scoring job until the PUT request resolves or the POST request for finalizeMultipart resolves. | ##### Multipart intake setting To enable the new multipart upload workflow, configure the `intakeSettings` for the `localFile` adapter as shown in the following sample request: ``` { "intakeSettings": { "type": "localFile", "multipart": true } } ``` - Upload any number of sequentially numbered files. - Finalize the upload to indicate that all required files uploaded successfully. ##### Async intake setting To enable the new multipart upload workflow with async enabled, configure the `intakeSettings` for the `localFile` adapter as shown in the following sample request: > [!NOTE] Note > You can also use the `async` intake setting independently of the `multipart` setting. ``` { "intakeSettings": { "type": "localFile", "multipart": true, "async": false } } ``` A defining feature of batch predictions is that the scoring job starts on the initial file upload, and only one batch prediction job at a time can run for any given prediction instance. This functionality may cause issues when uploading large datasets over a slow connection. In these cases, the client's upload speed could create a bottleneck and block the processing of other jobs. To avoid this potential bottleneck, you can set `async` to `false`, as shown in the example above. This configuration postpones submitting the batch prediction job to the queue. When `"async": false`, the point at which a job enters the batch prediction queue depends on the `multipart` setting: - If"multipart": true, the job is submitted to the queue after thePOSTrequest forfinalizeMultipartresolves. - If"multipart": false, the job is submitted to the queue after the initial file intakePUTrequest resolves. #### Example multipart upload requests The batch prediction API requests required to upload a 3 part multipart batch prediction job would be: ``` PUT /api/v2/batchPredictions/:id/csvUpload/part/0/ PUT /api/v2/batchPredictions/:id/csvUpload/part/1/ PUT /api/v2/batchPredictions/:id/csvUpload/part/2/ POST /api/v2/batchPredictions/:id/csvUpload/finalizeMultipart/ ``` Each uploaded part is a complete CSV file with a header. #### Abort a multipart upload If you start a multipart upload that you don't want to finalize, you can use a `DELETE` request to the existing `batchPredictions` abort route: ``` DELETE /api/v2/batchPredictions/:id/ ``` ## HTTP scoring In addition to the cloud storage adapters, you can also point batch predictions to a regular URL so DataRobot can stream the data for scoring: | Parameter | Example | Description | | --- | --- | --- | | type | http | Use HTTP for intake. | | url | https://example.com/datasets/scoring.csv | An absolute URL for the file to be scored. | The URL can optionally contain a username and password, such as `https://username:password@example.com/datasets/scoring.csv`. The `http` adapter can be used for ingesting data from pre-signed URLs from either [S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html), [Azure](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview), or [GCP](https://cloud.google.com/storage/docs/access-control/signed-urls). ## JDBC scoring DataRobot supports reading from any JDBC-compatible database for Batch Predictions. To use JDBC with the Batch Prediction API, specify `jdbc` as the intake type. Since no file is needed for a `PUT` request, scoring will start immediately, transitioning the job to RUNNING if preliminary validation succeeds. To support this, the Batch Prediction API integrates with [external data sources](https://docs.datarobot.com/en/docs/classic-ui/data/connect-data/data-conn.html#add-data-sources) using credentials securely stored in [data credentials](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html). Supply data source details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-sources) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `intakeSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | jdbc | Use a JDBC data store for intake. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | The ID of an external data source. In the UI, select a data connection or click add a new data connection. Complete account and authorization fields. | | Enter credentials | credentialId | 5e4bc5555e6e763beb9db147 | The ID of a stored credential. Refer to storing credentials securely. | | Schemas | schema | public | (Optional) The name of the schema containing the table to be scored. | | Tables | table | scoring_data | (Optional) The name of the database table containing data to be scored. | | SQL query | query | SELECT feature1, feature2, feature3 AS readmitted FROM diabetes | (Optional) A custom query to run against the database. | | Deprecated option | | | | | Fetch size | fetchSize (deprecated) | 1000 | Deprecated: fetchSize is now inferred dynamically for optimal throughput and is no longer needed. (Optional) To balance throughput and memory usage, sets a custom fetchSize (number of rows read at a time). Must be in range [1, 100000]; default 1000. | > [!NOTE] Note > You must specify either `table` and `schema` or `query`. Refer to the [example section](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-from-a-jdbc-postgresql-database) for a complete API example. > [!NOTE] Data warehouse connections > Using JDBC to transfer data can be costly in terms of IOPS (input/output operations per second) and expense for data warehouses. The data warehouse adapters reduce the load on database engines during prediction scoring by using cloud storage and bulk insert to create a hybrid JDBC-cloud storage solution. For more information, see the [BigQuery](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html#bigquery-scoring), [Snowflake](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html#snowflake-scoring), and [Synapse](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html#azure-synapse-scoring) data warehouse adapter sections. ### Allowed source IP addresses Any connection initiated from DataRobot originates from an allowed IP addresses. See a full list at [Allowed source IP addresses](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/allowed-ips.html). ## SAP Datasphere scoring > [!NOTE] Premium > Support for SAP Datasphere is off by default. Contact your DataRobot representative or administrator for information on enabling the feature. > > Feature flag(s): Enable SAP Datasphere Connector, Enable SAP Datasphere Batch Predictions Integration To use SAP Datasphere for scoring, supply data source details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-sources) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `intakeSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | datasphere | Use a SAP Datasphere database for intake. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | The ID of an external data source. In the UI, select a data connection or click add a new data connection. Refer to the SAP Datasphere connection documentation. | | Enter credentials | credentialId | 5e4bc5555e6e763beb9db147 | The ID of a stored credential for Datasphere. Refer to storing credentials securely. | | | catalog | / | The name of the database catalog containing the table to be scored. | | Schemas | schema | public | The name of the database schema containing the table to be scored. | | Tables | table | scoring_data | The name of the database table containing data to be scored. | ## Databricks scoring To use the [Databricks connector](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/data-sources/wb-databricks.html) for scoring, supply data source details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-sources) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `intakeSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | databricks | Use a Databricks database for intake. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | The ID of an external data source. In the UI, select a data connection or click add a new data connection. | | + Add credentials | credentialId | 5e96092ef7e8773ddbdbabed | The ID of stored credentials for the external Databricks database connection. | | Catalog | catalog | default | (Optional) The Databricks database catalog containing the source table. | | Schema | schema | public | The Databricks schema containing the source table. | | Table | table | kickcars | The Databricks table from which to read input data. | ## Trino scoring To use the [Trino connector](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/data-sources/dc-trino.html) for scoring, supply data source details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-sources) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `intakeSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | trino | Use a Trino database for intake. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | The ID of an external data source. In the UI, select a data connection or click add a new data connection. | | + Add credentials | credentialId | 5e96092ef7e8773ddbdbabed | The ID of stored credentials for the external Trino database connection. | | Catalog | catalog | starburst_catalog | The Trino database catalog containing the source table. | | Schema | schema | analytics | The Trino schema containing the source table. | | Table | table | input_data_table | The Trino table from which to read input data. | > [!WARNING] Trino column name case requirement > Use lowercase only for column names in the dataset used to train a project. Trino sanitizes column names automatically (unquoted identifiers are lowercased), so mixed-case or uppercase column names can cause column inconsistency errors when reading from Trino for batch scoring. This applies even when creating tables with quoted column names—Trino still stores them as lowercase. For more information, see [trinodb/trino#17](https://github.com/trinodb/trino/issues/17). ## Azure Blob Storage scoring A scoring option for large files is Azure. To score from Azure Blob Storage, you must configure credentials with DataRobot using an Azure Connection String. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | azure | Use Azure Blob Storage for intake. | | URL | url | https://myaccount.blob.core.windows.net/datasets/scoring.csv | An absolute URL for the file to be scored. | | Format | format | csv | (Optional) Select CSV (csv) or Parquet (parquet). Default value: CSV | | + Add credentials | credentialId | 5e4bc5555e6e763beb488dba | In the UI, enable the + Add credentials field by selecting This URL requires credentials. Refer to storing credentials securely. | Azure credentials are encrypted and are only decrypted when used to set up the client for communication with Azure during scoring. ## Google Cloud Storage scoring DataRobot supports the Google Cloud Storage adapter. To score from Google Cloud Storage, you must set up a credential with DataRobot consisting of a JSON-formatted account key. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | gcp | Use Google Cloud Storage for intake. | | URL | url | gcs://bucket-name/datasets/scoring.csv | An absolute URL for the file to be scored. | | Format | format | csv | (Optional) Select CSV (csv) or Parquet (parquet). Default value: CSV | | + Add credentials | credentialId | 5e4bc5555e6e763beb488dba | In the UI, enable the + Add credentials field by selecting This URL requires credentials. Required if explicit access credentials for this URL are required, otherwise optional. Refer to storing credentials securely. | GCP credentials are encrypted and are only decrypted when used to set up the client for communication with GCP during scoring. ## Amazon S3 scoring For larger files, S3 is the preferred method for intake. DataRobot can ingest files from both public and private buckets. To score from Amazon S3, you must set up a credential with DataRobot consisting of an access key (ID and key) and, optionally, a session token. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | s3 | DataRobot recommends S3 for intake. | | URL | url | s3://bucket-name/datasets/scoring.csv | An absolute URL for the file to be scored. | | Format | format | csv | (Optional) Select CSV (csv) or Parquet (parquet). Default value: CSV | | + Add credentials | credentialId | 5e4bc5555e6e763beb488dba | In the UI, enable the + Add credentials field by selecting This URL requires credentials. Required if explicit access credentials for this URL are required. Refer to storing credentials securely. | AWS credentials are encrypted and only decrypted when used to set up the client for communication with AWS during scoring. > [!NOTE] Note > If running a Private AI Cloud within AWS, it is possible to provide implicit credentials for your application instances using an IAM Instance Profile to access your S3 buckets without supplying explicit credentials in the job data. For more information, see the [AWS documentation](https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html). ## BigQuery scoring To use BigQuery for scoring, supply data source details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-sources) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `intakeSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | bigquery | Use the BigQuery API to unload data to Google Cloud Storage and use it as intake. | | Dataset | dataset | my_dataset | The BigQuery dataset to use. | | Table | table | my_table | The BigQuery table or view from the dataset used as intake. | | Bucket | bucket | my-bucket-in-gcs | Bucket where data should be exported. | | + Add credentials | credentialId | 5e4bc5555e6e763beb488dba | Required if explicit access credentials for this bucket are required (otherwise optional).In the UI, enable the + Add credentials field by selecting This connection requires credentials. Refer to storing credentials securely. | Refer to the [example section](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-with-bigquery) for a complete API example. ## Snowflake scoring To use Snowflake for scoring, supply data source details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-sources) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `intakeSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | snowflake | Adapter type. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | ID of Snowflake data source. In the UI, select a Snowflake data connection or click add a new data connection. Complete account and authorization fields. | | Enter credentials | credentialId | 5e4bc5555e6e763beb9db147 | The ID of a stored credential for Snowflake. | | Tables | table | SCORING_DATA | (Optional) Name of the Snowflake table containing data to be scored. | | Schemas | schema | PUBLIC | (Optional) Name of the schema containing the table to be scored. | | SQL query | query | SELECT feature1, feature2, feature3 FROM diabetes | (Optional) Custom query to run against the database. | | Cloud storage type | cloudStorageType | s3 | Type of cloud storage backend used in Snowflake external stage. Can be one of 3 cloud storage providers: s3/azure/gcp. Default is s3 | | External stage | externalStage | my_s3_stage | Snowflake external stage. In the UI, toggle on Use external stage to enable the External stage field. | | + Add credentials | cloudStorageCredentialId | 6e4bc5541e6e763beb9db15c | ID of stored credentials for a storage backend (S3/Azure/GCS) used in Snowflake stage. In the UI, enable the + Add credentials field by selecting This URL requires credentials. | Refer to the [example section](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-with-snowflake) for a complete API example. ## Azure Synapse scoring To use Synapse for scoring, supply data source details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-sources) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `intakeSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | synapse | Adapter type. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | ID of Synapse data source. In the UI, select a Synapse data connection or click add a new data connection. Complete account and authorization fields. | | External data source | externalDatasource | my_data_source | Name of the Synapse external data source. | | Tables | table | SCORING_DATA | (Optional) Name of the Synapse table containing data to be scored. | | Schemas | schema | dbo | (Optional) Name of the schema containing the table to be scored. | | SQL query | query | SELECT feature1, feature2, feature3 FROM diabetes | (Optional) Custom query to run against the database. | | Enter credentials | credentialId | 5e4bc5555e6e763beb9db147 | The ID of a stored credential for Synapse. Credentials are required if explicit access credentials for this URL are required, otherwise optional. Refer to storing credentials securely. | | + Add credentials | cloudStorageCredentialId | 6e4bc5541e6e763beb9db15c | ID of a stored credential for Azure Blob storage. In the UI, enable the + Add credentials field by selecting This external data source requires credentials. | Refer to the [example section](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-with-synapse) for a complete API example. > [!NOTE] Note > Synapse supports fewer collations than the default Microsoft SQL Server. For more information, reference the [Synapse documentation](https://docs.microsoft.com/en-us/azure/synapse-analytics/sql/reference-collation-types). ## AI Catalog / Data Registry dataset scoring To read input data from an [AI Catalog/Data Registry](https://docs.datarobot.com/en/docs/classic-ui/data/ai-catalog/catalog.html) dataset, the following options are available: | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | dataset | In the UI, select Data Registry or AI Catalog (in Classic). | | + Select source from AI Catalog | datasetId | 5e4bc5b35e6e763beb9db14a | The AI Catalog dataset ID.In the UI, search for the dataset, select the dataset, then click Use the dataset (or Confirm in Workbench). | | + Select version | datasetVersionId | 5e4bc5555e6e763beb488dba | The AI Catalog dataset version ID (Optional)In the UI, enable the + Select version field by selecting the Use specific version check box. Search for and select the version. If datasetVersionId is not specified, it defaults to the latest version for the specified dataset. | > [!NOTE] Note > For the specified AI Catalog dataset, the version to be scored must have been successfully ingested, and it must be a snapshot. ## Wrangler recipe dataset scoring The following options are available to read input data from a wrangler recipe created in [Workbench](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/dataprep/wrangle-data/index.html) from a [Snowflake data connection](https://docs.datarobot.com/en/docs/workbench/nxt-workbench/dataprep/add-data-usecase.html): > [!NOTE] Wrangler data connection > Wrangler recipes for batch prediction jobs support data wrangled from a Snowflake data connection or the AI Catalog/Data Registry. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Source type | type | recipe | In the UI, select Wrangler Recipe. | | + Select wrangler recipe | recipeId | 65fb040a42c170ee46230133 | The Wrangler Recipe dataset ID.In the prediction jobs UI, search for the wrangled dataset, select the dataset, then click Confirm. | --- # Batch Prediction job definitions URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/job-definitions.html > How to submit a working Batch Prediction job. You must supply a variety of elements to the POST request payload depending on the type of prediction. To submit a working Batch Prediction job, you must supply a variety of elements to the `POST` request payload depending on what type of prediction is required. Additionally, you must consider the type of intake and output adapters used for a given job. For more information about Batch Prediction REST API routes, view the [DataRobot REST API reference documentation](https://docs.datarobot.com/en/docs/api/reference/public-api/batch_predictions.html). Every time you make a Batch Prediction, the prediction information is stored outside DataRobot and re-submitted for each prediction request, as described in detail in the [sample use cases section](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html). One such request could be as follows: `POST https://app.datarobot.com/api/v2/batchPredictions` ``` { "deploymentId": "", "intakeSettings": { "type": "dataset", "datasetId": "" }, "outputSettings": { "type": "jdbc", "statementType": "insert", "credentialId": "", "dataStoreId": "", "schema": "public", "table": "example_table", "createTableIfNotExists": false }, "includeProbabilities": true, "includePredictionStatus": true, "passthroughColumnsSet": "all" } ``` ## Job Definitions API If your use case requires the same, or close to the same, type of prediction to be done multiple times, you can choose to create a Job Definition of the Batch Prediction job and store this inside DataRobot for future use. The API for job definitions is identical to the existing `/batchPredictions/` endpoint, and can be used interchangeably by changing the `POST` endpoint to `/batchPredictionJobDefinitions`: `POST https://app.datarobot.com/api/v2/batchPredictionJobDefinitions` ``` { "deploymentId": "", "intakeSettings": { "type": "dataset", "datasetId": "" }, "outputSettings": { "type": "jdbc", "statementType": "insert", "credentialId": "", "dataStoreId": "", "schema": "public", "table": "example_table", "createTableIfNotExists": false }, "includeProbabilities": true, "includePredictionStatus": true, "passthroughColumnsSet": "all" } ``` This definition endpoint will return an accepted payload that verifies the successful storing of the definition to DataRobot. (Optional) You can supply a `name` parameter for easier identification. If you don't supply one, DataRobot will create one for you. > [!WARNING] Warning > The `name` parameter must be unique across your organization. If you attempt to create multiple definitions with the same name, the request will fail. If you wish to free up a name, you must first send a `DELETE` request with the existing job definition ID you wish to delete. ## Execute a Job Definition If you wish to submit a stored job definition for scoring, you can either choose to do so on a scheduled basis, described [here](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/job-scheduling.html), or by manually submitting the definition ID to the endpoint `/batchPredictions/fromJobDefinition` and with the definition ID as the payload, as such: `POST https://app.datarobot.com/api/v2/batchPredictions/fromJobDefinition` ``` { "jobDefinitionId": "" } ``` The endpoint supports regular the CRUD operations, `GET`, `POST`, `DELETE` and `PATCH`. --- # Schedule Batch Prediction jobs URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/job-scheduling.html > How to create a definition and schedule the execution of a Batch Prediction job. After [creating a job definition](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/job-definitions.html), you can choose to execute job definitions on a scheduled basis instead of manually doing so through the `/batchPredictions/fromJobDefinition` endpoint. A Scheduled Batch Prediction job works just like a regular Batch Prediction job, except DataRobot handles the execution of the job. In order to schedule the execution of a Batch Prediction job, a definition must first be created, as described [here](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/job-definitions.html). For more information about Batch Prediction REST API routes, view the [DataRobot REST API reference documentation](https://docs.datarobot.com/en/docs/api/reference/public-api/batch_predictions.html). ## Schedule a job definition The API accepts the keywords `enabled` as well as a `schedule` object, as such: `POST https://app.datarobot.com/api/v2/batchPredictionJobDefinitions` ``` { "deploymentId": "", "intakeSettings": { "type": "dataset", "datasetId": "" }, "outputSettings": { "type": "jdbc", "statementType": "insert", "credentialId": "", "dataStoreId": "", "schema": "public", "table": "example_table", "createTableIfNotExists": false }, "includeProbabilities": true, "includePredictionStatus": true, "passthroughColumnsSet": "all" "enabled": false, "schedule": { "minute": [0], "hour": [1], "month": ["*"] "dayOfWeek": ["*"], "dayOfMonth": ["*"], } } ``` ### Schedule payload The `schedule` payload defines at what intervals the job should run, which can be combined in various ways to construct complex scheduling terms if needed. In all of the elements in the objects, you can supply either an asterisk `["*"]` denoting "every" time denomination or an array of integers (e.g.`[1, 2, 3]`) to define a specific interval. | Key | Possible values | Example | Description | | --- | --- | --- | --- | | minute | ["*"] or [0 ... 59] | [15, 30, 45] | The job will run at these minute values for every hour of the day. | | hour | ["*"] or [0 ... 23] | [12,23] | The hour(s) of the day that the job will run. | | month | ["*"] or [1 ... 12] | ["jan"] | Strings, either 3-letter abbreviations or the full name of the month, can be used interchangeably (e.g., "jan" or "october"). Months that are not compatible with dayOfMonth are ignored, for example {"dayOfMonth": [31], "month":["feb"]}. | | dayOfWeek | ["*"] or [0 ... 6] where (Sunday=0) | ["sun"] | The day(s) of the week that the job will run. Strings, either 3-letter abbreviations or the full name of the day, can be used interchangeably (e.g., "sunday", "Sunday", "sun", or "Sun", all map to [0]). NOTE: This field is additive with dayOfMonth, meaning the job will run both on the date specified by dayOfMonth and the day defined in this field. | | dayOfMonth | ["*"] or [1 ... 31] | [1, 25] | The date(s) of the month that the job will run. Allowed values are either [1 ... 31] or ["*"] for all days of the month. NOTE: This field is additive with dayOfWeek, meaning the job will run both on the date(s) defined in this field and the day specified by dayOfWeek (for example, dates 1st, 2nd, 3rd, plus every Tuesday). If dayOfMonth is set to ["*"] and dayOfWeek is defined, the scheduler will trigger on every day of the month that matches dayOfWeek (for example, Tuesday the 2nd, 9th, 16th, 23rd, 30th). Invalid dates such as February 31st are ignored. | > [!NOTE] Note > When specifying a time of day to run jobs, you must use UTC in the `schedule` payload—local time zones are not supported. > To account for DST (daylight savings time), update the schedule according to your local time. ### Examples | Interval | Example | Description | | --- | --- | --- | | Run every 5 minutes | "schedule": { "minute": [0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55], "hour": ["*"], "month": ["*"] "dayOfWeek": ["*"], "dayOfMonth": ["*"], } | Executes every time the minute dial of a clock reaches the number(s) defined in minute, since all other fields are with asterisks. | | Run every full hour | "schedule": { "minute": [0], "hour": ["*"], "month": ["*"] "dayOfWeek": ["*"], "dayOfMonth": ["*"], } | Executes every time the clock reaches the minute(s) defined in minute. This example executes every day at 1:00 AM, 2:00 AM, 3:00 AM, and so forth. | | Run right before noon every day | "schedule": { "minute": [59], "hour": [11], "month": ["*"] "dayOfWeek": ["*"], "dayOfMonth": ["*"], } | Executes every time the minute dial of a clock reaches the minutes(s) defined in minute, and the same when the hour dial reaches the number(s) defined in hour. This example executes every day at 11:59 AM. | | Run every full hour once every half year | "schedule": { "minute": [0], "hour": ["*"], "month": [1, 6] "dayOfWeek": ["*"], "dayOfMonth": ["*"], } | Executes every time the minute dial of a clock reaches the minute(s) defined in minute, and only when the month is January (1) or June (6). | | Run every full hour once every half year and only on Mondays and Saturdays | "schedule": { "minute": [0], "hour": ["*"], "month": [1, 6] "dayOfWeek": ["mon", "sun"], "dayOfMonth": ["*"], } | Same as above, but with dayOfWeek specified, the interval is only executed on the days specified. | | Run every full hour once every half year and only on Mondays and Saturdays, but also on the 1st and 10th of the month | "schedule": { "minute": [0], "hour": ["*"], "month": [1, 6] "dayOfWeek": ["mon", "sun"], "dayOfMonth": [1, 10], } | Same as above, but with both dayOfWeek and dayOfMonth specified, these values add to each other, not excluding. This example executes on both the times defined in dayOfWeek and dayOfMonth, and not, as could be believed, only on those years where the 1st and 10th are Mondays and Sundays. | ## Disable a scheduled job Job definitions are only be executed by the scheduler if `enabled` is set to `True`. If you have a job definition that was previously running as a scheduled job, but should now be stopped, simply `PATCH` the endpoint with `enabled` set to `False`. If a job is currently running, this will finish execution regardless. `PATCH https://app.datarobot.com/api/v2/batchPredictionJobDefinitions/` ``` { "enabled": false } ``` ## Limitations The Scheduler has limitations set to how often a job can run and how many jobs can run at once. ### Total runs per day Each organization is limited to a number of job executions per day. If you are a Self-Managed AI Platform user, you can change this limitation by changing the environment variable `BATCH_PREDICTIONS_JOB_SCHEDULER_MAX_NUMBER_OF_RUNS_PER_DAY_PER_ORGANIZATION`. On cloud, this limit is `1000` by default. Note that the limitation is across all scheduled jobs per an organization, so if one scheduled job has a maximum run time of `1000` per day, no more scheduled jobs can be activated by that organization. ### Schedules are best-effort Depending on the load of different definitions running at the same time across the organization, the scheduler cannot guarantee to execute all jobs at the exact second of the schedule. However, in most cases, the scheduler will have resources to trigger the job within 5 seconds of the schedule. ### Running the same definition simultaneously One job definition cannot run more than once on a scheduled basis. This means that if a schedule job is taking long to execute, causing the next interval to trigger before the first one finished, the job will be rejected and aborted. This will continue to happen until the running job finishes. ### Automatic disablement of failing jobs If a user has created a job definition that cannot execute due to misconfiguration and is aborted, this will cause the `enabled` feature to be auto-disabled after `5` consecutive failures. It is therefore recommended that you use the existing `/batchPredictions` endpoint to test if the solution works, before `POST` ing the identical, confirmed working payload to the `/batchPredictionJobDefinitions`. For Self-Managed AI Platform customers, this cut-off point of consecutive failures can be adjusted by changing the `BATCH_PREDICTIONS_JOB_SCHEDULER_FAILURES_BEFORE_ABORT` environment variable. --- # Predictions on large datasets URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/large-preds-api.html > Walk through an example of making predictions on a large dataset using the Batch Prediction API. [File size limits](https://docs.datarobot.com/en/docs/classic-ui/predictions/pred-file-limits.html) vary depending on the prediction method—for predictions on large datasets, use the Batch Prediction API or real-time Prediction API. The following example shows how to make predictions on a large dataset using the Batch Prediction API. See the [Prediction API](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html) for real-time predictions. In this example, the prediction dataset is stored in the AI Catalog. The Batch Prediction API also supports predicting on data sourced from [other locations](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html). Note that for predicting with a dataset from the AI Catalog, the dataset must be snapshotted. In addition to the API key sent in the header of all API requests, you need the following to use the Batch Prediction API: 1. : The deployment ID for the model being used to make predictions against. 2. : The dataset ID of the snapshotted AI Catalog dataset used by the model . The following steps show how to work with files greater than 100MB using the `batchPredictions` API endpoint. In summary, you will: 1. Create a BatchPrediction job indicating the deployed model and dataset to use. 2. Check the status of that BatchPrediction job until it is complete. 3. Download the results. ### 1. Create a Batch Prediction job `POST https://app.datarobot.com/api/v2/batchPredictions` Sample request: ``` { "deploymentId": "", "intakeSettings": { "type": "dataset", "datasetId": "" } } ``` Sample time series request (requires enabling the time series product and the Batch Predictions for time series preview flag): ``` { "deploymentId": "", "intakeSettings": { "type": "dataset", "datasetId": "" }, "timeseriesSettings": { "type": "forecast" } } ``` Sample response: The `links.self` property of the response contains the URL used for the next two steps. ``` { "status": "INITIALIZING", "skippedRows": 0, "failedRows": 0, "elapsedTimeSec": 0, "logs": [ "Job created by user@example.com from 10.1.2.1 at 2020-02-19 22:41:00.865000" ], "links": { "download": null, "self": "https://app.datarobot.com/api/v2/batchPredictions/a1b2c3d4x5y6z7/" }, "jobIntakeSize": null, "scoredRows": 0, "jobOutputSize": null, "jobSpec": { "includeProbabilitiesClasses": [], "maxExplanations": 0, "predictionWarningEnabled": null, "numConcurrent": 4, "thresholdHigh": null, "passthroughColumnsSet": null, "csvSettings": { "quotechar": "\"", "delimiter": ",", "encoding": "utf-8" }, "thresholdLow": null, "outputSettings": { "type": "localFile" }, "includeProbabilities": true, "columnNamesRemapping": {}, "deploymentId": "", "abortOnError": true, "intakeSettings": { "type": "dataset", "datasetId": "" }, "includePredictionStatus": false, "skipDriftTracking": false, "passthroughColumns": null }, "statusDetails": "Job created by user@example.com from 10.1.2.1 at 2020-02-19 22:41:00.865000", "percentageCompleted": 0.0 } ``` The `links.self` property `https://app.datarobot.com/api/v2/batchPredictions/a1b2c3d4x5y6z7/` is the variable `` in the Step 2 GET call, below. ### 2. Check the status of the batch prediction job `GET ` Sample response: ``` { "status": "INITIALIZING", "skippedRows": 0, "failedRows": 0, "elapsedTimeSec": 352, "logs": [ "Job created by user@example.com from 10.1.2.1 at 2020-02-19 22:41:00.865000", "Job started processing at 2020-02-19 22:41:16.192000" ], "links": { "download": "https://app.datarobot.com/api/v2/batchPredictions/a1b2c3d4x5y6z7/download/", "self": "https://app.datarobot.com/api/v2/batchPredictions/a1b2c3d4x5y6z7/" }, "jobIntakeSize": null, "scoredRows": 1982300, "jobOutputSize": null, "jobSpec": { "includeProbabilitiesClasses": [], "maxExplanations": 0, "predictionWarningEnabled": null, "numConcurrent": 4, "thresholdHigh": null, "passthroughColumnsSet": null, "csvSettings": { "quotechar": "\"", "delimiter": ",", "encoding": "utf-8" }, "thresholdLow": null, "outputSettings": { "type": "localFile" }, "includeProbabilities": true, "columnNamesRemapping": {}, "deploymentId": "", "abortOnError": true, "intakeSettings": { "type": "dataset", "datasetId": "" }, "includePredictionStatus": false, "skipDriftTracking": false, "passthroughColumns": null }, "statusDetails": "Job started processing at 2020-02-19 22:41:16.192000", "percentageCompleted": 0.0 } ``` The `links.download` property `https://app.datarobot.com/api/v2/batchPredictions/a1b2c3d4x5y6z7/download/` is the variable `` in the Step 3 GET call, below. ### 3. Download the results of the batch prediction job Continue polling the status URL above until the job status is COMPLETED and error-free. At that point, predictions can be downloaded. `GET ` --- # Output formats URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html > Review the output formats for the predictions DataRobot returns in a columnar table format. DataRobot returns predictions in a columnar table format. Each example value is followed by the data type it belongs to. The columns returned are determined by model type, as described below. The output schema shares the same format for real-time and batch predictions. > [!NOTE] Note > DataRobot allows prediction output to many different databases that all have unique versions of a string (e.g., some may call it `TEXT` while others may call it `VARCHAR`). > As a result, DataRobot cannot provide implementation-specific data types. ## Regression models | Column name | _PREDICTION | | Data type | Numeric | | Example name | revenue_PREDICTION | | Example value | 493822.12 | | Description | The predicted value. | ## Binary classification models | Positive label | | --- | | Column name | | Data type | | Example name | | Example value | | Description | | Negative label | | --- | | Column name | | Data type | | Example name | | Example value | | Description | | Column name | _PREDICTION | | Data type | Text | | Example name | isbadbuy_PREDICTION | | Example value | 0 | | Description | The predicted label of the classification. | | Column name | THRESHOLD | | Data type | Numeric | | Example name | THRESHOLD | | Example value | 0.5 | | Description | The float prediction threshold used for determining the label. | | Positive class label | | --- | | Column name | | Data type | | Example name | | Example value | | Description | ## Multiclass classification models | Column name | _PREDICTION | | Data type | Text | | Example name | species_PREDICTION | | Example value | lion | | Description | The predicted label of the classification. | | Column name | __PREDICTION | | Data type | Numeric | | Description | The float probability for each class. | | Example name | Example value | | --- | --- | | species_cat_PREDICTION | 0.28 | | species_lion_PREDICTION | 0.24 | | species_lynx_PREDICTION | 0.48 | ## Time series models > [!NOTE] Note > These output columns are available for time series regression, classification, and anomaly detection models. | Time series model columns | Description | Data type | | --- | --- | --- | | | Contains the series ID the row belongs to. Functions as a passthrough column and returns the unaltered column name and values provided in the scoring data. | Text | | FORECAST_POINT | Contains the forecast point timestamp.Unless you request historical time series predictions, the output value is the same for all rows with the same forecast point (but different for each unique forecast distance). | Date | | | Contains the time series timestamp.Functions as a passthrough column and returns the unaltered column name and values provided in the scoring data. (This returns the same value as the originalFormatTimestamp field returned by time series models.) | Date | | FORECAST_DISTANCE | Contains the numeric forecast distance returned by time series models. | Numeric | ## Prediction status | Column name | prediction_status | | Data type | Text | | Description | A row-by-row status containing either OK or a string error message describing why the prediction did not succeed. | | Example value | Could not convert date field to date format YYYY-MM-DD | | Example value | OK | ## Prediction warnings If prediction warnings are enabled for your job, DataRobot returns an additional column. | Column name | IS_OUTLIER_PREDICTION | | Data type | Text | | Description | Whether the prediction is outside the calculated prediction boundaries. | | Column | Example value | | --- | --- | | Data type | Text | | IS_OUTLIER_PREDICTION | True | | IS_OUTLIER_PREDICTION | False | ## Deployment approval status If the approval workflow is enabled for your deployment, the output schema will contain an extra column showing the deployment approval status. | Column name | DEPLOYMENT_APPROVAL_STATUS | | Data type | Text/td> | | Description | Whether the deployment was approved. | | Example value | PENDING | ## Prediction Explanations You can request Prediction Explanations be returned with your predictions by setting the `maxExplanations` job parameter to a non-zero value. You can also set thresholds for computing explanations. If you do not configure a threshold, DataRobot computes explanations for every row. | Job parameter | Description | Example value | Data type | | --- | --- | --- | --- | | maxExplanations | (Optional) Compute up to this number of explanations. | 10 | Integer | | thresholdHigh | (Optional) Limit explanations to predictions above this threshold. | 0.5 | Float | | thresholdLow | (Optional) Limit explanations to predictions below this threshold. | 0.15 | Float | If Prediction Explanations are requested, DataRobot returns four extra columns for each explanation in the format `EXPLANATION__IDENTIFIER` (where `n` is the feature explanation index, from 1 to the maximum number of explanations requested). The returned columns are: | Column | Description | Data type | | --- | --- | --- | | EXPLANATION__FEATURE_NAME | The feature name this explanation covers. | Text | | EXPLANATION__STRENGTH | The feature strength as a float. | Numeric | | EXPLANATION__QUALITATIVE_STRENGTH | The feature strength as a string, a plus or minus indicator from +++ to ---. | Text | | EXPLANATION__ACTUAL_VALUE | The feature associated with this explanation. | Text | ### Prediction Explanation examples | Name | Value | | --- | --- | | EXPLANATION_1_FEATURE_NAME | loan_status | | EXPLANATION_1_ACTUAL_VALUE | Charged Off | | EXPLANATION_1_STRENGTH | 1.380291221709652 | | EXPLANATION_1_QUALITATIVE_STRENGTH | +++ | | Name | Value | | --- | --- | | EXPLANATION_1_FEATURE_NAME | loan_status | | EXPLANATION_1_ACTUAL_VALUE | Fully Paid | | EXPLANATION_1_STRENGTH | -1.2145340858375335 | | EXPLANATION_1_QUALITATIVE_STRENGTH | --- | ## Passthrough columns Passthrough columns you request are passed verbatim. If they conflict with any of the above names, the job is rejected. ## Association ID If your deployment was configured with an [association ID for accuracy](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment-settings/accuracy-settings.html), all result sets will have that column passed through from the source data automatically. ## Output filters Use the following job configuration properties to control whether to display only specific class probabilities or none at all. | Job parameter | Description | Example value | Data type | | --- | --- | --- | --- | | includeProbabilities | (Optional) Include probabilities for all classes; defaults to true. | true | Boolean | | includeProbabilitiesClasses | (Optional) Include only probabilities for classes listed in the given array; defaults to an empty array []. | ['setosa', 'versicolor'] | Boolean | | includePredictionStatus | (Optional) Include the prediction_status column in the output; defaults to false. | true | Boolean | > [!NOTE] Note > For binary classification, `includeProbabilities` also controls the `THRESHOLD` and `POSITIVE_CLASS` columns. ## Column name remapping If your use case has a strict output schema that does not match the DataRobot output, you can rename and remove any columns from the output using the `columnNamesRemapping` job configuration property. | Job parameter | Description | Example value | | --- | --- | --- | | columnNamesRemapping | (Optional) Provide a list of items to remap (rename or remove columns from) the output from this job. Set an outputName for the column to null or false to ignore it. | [{'inputName': 'isbadbuy_1_PREDICTION', 'outputName':'prediction'}, {'inputName': 'isbadbuy_0_PREDICTION', 'outputName': null}] | --- # Prediction output options URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-options.html > Configure batch prediction destinations (output) with the Job Definitions UI or the Batch Prediction API. You can configure a prediction destination using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-destinations) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html). This topic describes both the UI and API output options. > [!NOTE] Note > For a complete list of supported output options, see the [data sources supported for batch predictions](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html#data-sources-supported-for-batch-predictions). | Output option | Description | | --- | --- | | Local file streaming | Stream scored data through a URL endpoint for immediate download when the job moves to a running state. | | HTTP write | Stream scored data to an absolute URL for writing. This option can write data to pre-signed URLs for Amazon S3, Azure, and Google Cloud Platform. | | Database connections | | | JDBC write | Write prediction results back to a JDBC data source with data destination details supplied through a job definition or the Batch Prediction API. | | SAP Datasphere write | Write prediction results back to a SAP Datasphere data source with data destination details supplied through a job definition or the Batch Prediction API. | | Trino write | Write prediction results back to a Trino database with data destination details supplied through a job definition or the Batch Prediction API. | | Cloud storage connections | | | Azure Blob Storage write | Write scored data to Azure Blob Storage with a DataRobot credential consisting of an Azure Connection String. | | Google Cloud Storage write | Write scored data to Google Cloud Storage with a DataRobot credential consisting of a JSON-formatted account key. | | Amazon S3 write | Write scored data to public or private S3 buckets with a DataRobot credential consisting of an access key (ID and key) and a session token (Optional) | | Data warehouse connections | | | BigQuery write | Write prediction results to BigQuery with data destination details supplied through a job definition or the Batch Prediction API. | | Snowflake write | Write prediction results to Snowflake with data destination details supplied through a job definition or the Batch Prediction API. | | Azure Synapse write | Write prediction results to Synapse with data destination details supplied through a job definition or the Batch Prediction API. | If you are using a custom [CSV format](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html#csv-format), any output option dealing with CSV will adhere to that format. The columns that appear in the output are documented in the section on [output format](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html). ## Local file streaming If your job is configured with local file streaming as the output option, you can start downloading the scored data as soon as the job moves to a `RUNNING` state. In the example job data JSON below, the URL needed to make the local file streaming request is available in the `download` key of the `links` object: ``` { "elapsedTimeSec": 97, "failedRows": 0, "jobIntakeSize": 1150602342, "jobOutputSize": 107791140, "jobSpec": { "deploymentId": "5dc1a6a9865d6c004dd881ef", "maxExplanations": 0, "numConcurrent": 4, "passthroughColumns": null, "passthroughColumnsSet": null, "predictionWarningEnabled": null, "thresholdHigh": null, "thresholdLow": null }, "links": { "download": "https://app.datarobot.com/api/v2/batchPredictions/5dc45e583c36a100e45276da/download/", "self": "https://app.datarobot.com/api/v2/batchPredictions/5dc45e583c36a100e45276da/" }, "logs": [ "Job created by user@example.org from 203.0.113.42 at 2019-11-07 18:11:36.870000", "Job started processing at 2019-11-07 18:11:49.781000", "Job done processing at 2019-11-07 18:13:14.533000" ], "percentageCompleted": 0.0, "scoredRows": 3000000, "status": "COMPLETED", "statusDetails": "Job done processing at 2019-11-07 18:13:14.533000" } ``` If you download faster than DataRobot can ingest and score your data, the download may appear sluggish because DataRobot streams the scored data as soon as it arrives (in chunks). Refer to the [this sample use case](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-of-csv-files-from-local-files) for a complete example. ## HTTP write You can point Batch Predictions at a regular URL, and DataRobot streams the data: | Parameter | Example | Description | | --- | --- | --- | | type | http | Use HTTP for output. | | url | https://example.com/datasets/scored.csv | An absolute URL that designates where the file is written. | The URL can optionally contain a username and password such as: `https://username:password@example.com/datasets/scored.csv`. The `http` adapter can be used for writing data to pre-signed URLs from either [S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html), [Azure](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview), or [GCP](https://cloud.google.com/storage/docs/access-control/signed-urls). ## JDBC write DataRobot supports writing prediction results back to a JDBC data source. For this, the Batch Prediction API integrates with [external data sources](https://docs.datarobot.com/en/docs/classic-ui/data/connect-data/data-conn.html#add-data-sources) using [securely stored credentials](https://docs.datarobot.com/en/docs/platform/acct-settings/stored-creds.html). Supply data destination details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-destinations) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `outputSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | jdbc | Use a JDBC data store as output. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | The external data source ID. | | Enter credentials | credentialId | 5e4bc5555e6e763beb9db147 | (Optional) The ID of a stored credential. Refer to storing credentials securely. | | Schemas | schema | public | (Optional) The name of the schema where scored data will be written. | | Tables | table | scoring_data | The name of the database table where scored data will be written. | | Database | catalog | output_data | (Optional) The name of the specified database catalog to write output data to. | | Write strategy options | | | | | Write strategy | statementType | update | The statement type, insert, update, or insertUpdate. | | Create table if it does not exist (for Insert or Insert + Update) | create_table_if_not_exists | true | (Optional) If no existing table is detected, attempt to create it before writing data with the strategy defined in the statementType parameter. | | Row identifier (for Update or Insert + Update) | updateColumns | ['index'] | (Optional) A list of strings containing the column names to be updated when statementType is set to update or insertUpdate. | | Row identifier (for Update or Insert + Update) | where_columns | ['refId'] | (Optional) A list of strings containing the column names to be selected when statementType is set to update or insertUpdate. | | Advanced options | | | | | Commit interval | commitInterval | 600 | (Optional) Defines a time interval, in seconds, between commits to the target database. If set to 0, the batch prediction operation will write the entire job before committing. Default: 600 | > [!NOTE] Note > If your target database doesn't support the column naming conventions of DataRobot's [output format](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html), you can use [Column Name Remapping](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html#column-name-remapping) to re-write the output column names to a format your target database supports (e.g., remove spaces from the name). ### Statement types When dealing with Write strategy options, you can use the following statement types to write data, depending on the situation: | Statement type | Description | | --- | --- | | insert | Scored data rows are inserted in the target database as a new entry. Suitable for writing to an empty table. | | update | Scored data entries in the target database matching the row identifier of a result row are updated with the new result (columns identified in updateColumns). Suitable for writing to an existing table. | | insertUpdate | Entries in the target database matching the row identifier of a result row (where_columns) are updated with the new result (update queries). All other result rows are inserted as new entries (insert queries). | | createTable (deprecated) | DataRobot no longer recommends createTable. Use a different option with create_table_if_not_exists set to True. If used, scored data rows are saved to a new table using INSERT queries. The table must not exist before writing. | ### Allowed source IP addresses Any connection initiated from DataRobot originates from an allowed IP addresses. See a full list at [Allowed source IP addresses](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/allowed-ips.html). ## SAP Datasphere write > [!NOTE] Premium > Support for SAP Datasphere is off by default. Contact your DataRobot representative or administrator for information on enabling the feature. > > Feature flag(s): Enable SAP Datasphere Connector, Enable SAP Datasphere Batch Predictions Integration To use SAP Datasphere, supply data destination details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-destinations) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `outputSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | datasphere | Use a SAP Datasphere database for output. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | The ID of an external data source. In the UI, select a data connection or click add a new data connection. Refer to the SAP Datasphere connection documentation. | | Enter credentials | credentialId | 5e4bc5555e6e763beb9db147 | The ID of a stored credential for Datasphere. Refer to storing credentials securely. | | | catalog | / | The name of the database catalog containing the table to write to. | | Schemas | schema | public | The name of the database schema containing the table to write to. | | Tables | table | scoring_data | The name of the database table containing data to write to. In the UI, select a table or click Create a table. | ## Databricks write To use the [Databricks connector](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/data-sources/wb-databricks.html) for output, supply data destination details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-destinations) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `outputSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | databricks | Use a Databricks database for output. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | The ID of an external data source. In the UI, select a data connection or click add a new data connection. | | + Add credentials | credentialId | 5e96092ef7e8773ddbdbabed | The ID of stored credentials for the external Databricks database connection. | | Catalog | catalog | default | (Optional) The Databricks database catalog containing the destination table. | | Schema | schema | public | The Databricks schema containing the destination table. | | Table | table | kickcars_predictions | The Databricks table in which to write output data. | ## Trino write To use the [Trino connector](https://docs.datarobot.com/en/docs/reference/data-ref/connectivity/data-sources/dc-trino.html) for output, supply data destination details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-destinations) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `outputSettings`): | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | trino | Use a Trino database for output. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | The ID of an external data source. In the UI, select a data connection or click add a new data connection. | | + Credentials | credentialId | 5e96092ef7e8773ddbdbabed | The credentials to use for the external Trino database connection. | | Catalog | catalog | starburst_catalog | The Trino database catalog to store the output table. | | Schema | schema | analytics | The Trino schema to store the output table. | | Table | table | prediction_results | The Trino table in which to write output data. | | Advanced options | | | | | Chunk size | chunkSize | 500000 | An explicit numeric chunk size in bytes. Must be a positive integer no greater than 1000000 (1MB). Named strategies (auto, dynamic, fixed) are not supported and will cause the job to fail. See the note below. | > [!WARNING] Trino chunk size requirement > Trino enforces a default [query.max-length](https://trino.io/docs/current/admin/properties-query-management.html#query-max-length) of 1MB (1,000,000 bytes). Because DataRobot generates SQL `INSERT` statements for each chunk of rows sent to Trino, the `chunkSize` parameter must be set to an explicit numeric value and must not exceed 1,000,000 bytes. Using named chunk strategies ( `auto`, `dynamic`, or `fixed`) or a value larger than `1000000` will cause the batch prediction job to fail. > [!WARNING] Trino column name case requirement > Use lowercase only for column names in the dataset used to train a project. Trino sanitizes column names automatically (unquoted identifiers are lowercased), so mixed-case or uppercase column names can cause column inconsistency errors when reading from Trino for batch scoring. This applies even when creating tables with quoted column names—Trino still stores them as lowercase. For more information, see [trinodb/trino#17](https://github.com/trinodb/trino/issues/17). ## Azure Blob Storage write Azure Blob Storage is an option for writing large files. To save a dataset to Azure Blob Storage, you must set up a credential with DataRobot consisting of an Azure Connection String. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | azure | Use Azure Blob Storage for output. | | URL | url | https://myaccount.blob.core.windows.net/datasets/scored.csv | An absolute URL for the file to be written. | | Format | format | csv | (Optional) Select CSV (csv) or Parquet (parquet). Default value: CSV | | + Add credentials | credentialId | 5e4bc5555e6e763beb488dba | In the UI, enable the + Add credentials field by selecting This URL requires credentials. Required if explicit access to credentials for this URL are necessary (optional otherwise). Refer to storing credentials securely. | Azure credentials are encrypted and only decrypted when used to set up the client for communication with Azure when writing. ## Google Cloud Storage write DataRobot supports the Google Cloud Storage adapter. To save a dataset to Google Cloud Storage, you must set up a credential with DataRobot consisting of a JSON-formatted account key. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | gcp | Use Google Cloud Storage for output. | | URL | url | gcs://bucket-name/datasets/scored.csv | An absolute URL designating where the file is written. | | Format | format | csv | (Optional) Select CSV (csv) or Parquet (parquet). Default value: CSV | | + Add credentials | credentialId | 5e4bc5555e6e763beb488dba | Required if explicit access credentials for this URL are required, otherwise (Optional) Refer to storing credentials securely. | GCP credentials are encrypted and are only decrypted when used to set up the client for communication with GCP when writing. ## Amazon S3 write DataRobot can save scored data to both public and private buckets. To write to S3, you must set up a credential with DataRobot consisting of an access key (ID and key) and optionally a session token. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | s3 | Use S3 for output. | | URL | url | s3://bucket-name/results/scored.csv | An absolute URL for the file to be written. DataRobot only supports directory scoring when scoring from cloud to cloud. Provide a directory in S3 (or another cloud provider) for the input and a directory ending with / for the output. Using this configuration, all files in the input directory are scored and the results are written to the output directory with the original filenames. When a single file is specified for both the input and the output, the file is overwritten each time the job runs. If you do not wish to overwrite the file, specify a filename template such as s3://bucket-name/results/scored_{{ current_run_time }}.csv. You can review template variable definitions in the documentation. | | Format | format | csv | (Optional) Select CSV (csv) or Parquet (parquet). Default value: CSV | | + Add credentials | credentialId | 5e4bc5555e6e763beb9db147 | In the UI, enable the + Add credentials field by selecting This URL requires credentials. Required if explicit access credentials for this URL are required. Refer to storing credentials securely. | | Advanced options | | | | | Endpoint URL | endpointUrl | https://s3.us-east-1.amazonaws.com | (Optional) Override the endpoint used to connect to S3, for example, to use an API gateway or another S3-compatible storage service. | AWS credentials are encrypted and only decrypted when used to set up the client for communication with AWS when writing. > [!NOTE] Note > If running a Private AI Cloud within AWS, you can provide implicit credentials for your application instances using an IAM Instance Profile to access your S3 buckets without supplying explicit credentials in the job data. For more information, see the AWS article, [Create an IAM Instance Profile](https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html). ## BigQuery write To use BigQuery, supply data destination details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-destinations) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `outputSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | bigquery | Use Google Cloud Storage for output and the batch loading job to ingest data from GCS into a BigQuery table. | | Dataset | dataset | my_dataset | The BigQuery dataset to use. | | Table | table | my_table | The BigQuery table from the dataset to use for output. | | Bucket name | bucket | my-bucket-in-gcs | The GCP bucket where data files are stored to be loaded into or unloaded from a BiqQuery table. | | + Add credentials | credentialId | 5e4bc5555e6e763beb488dba | Required if explicit access credentials for this bucket are necessary (otherwise optional). In the UI, enable the + Add credentials field by selecting This connection requires credentials. Refer to storing credentials securely. | > [!NOTE] BigQuery output write strategy > The write strategy for BigQuery output is `insert`. First, the output adapter checks if a BigQuery table exists. If a table exists, the data is inserted. If a table doesn't exist, a table is created and then the data is inserted. Refer to the [example section](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-with-bigquery) for a complete API example. ## Snowflake write To use Snowflake, supply data destination details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-destinations) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `outputSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | snowflake | Adapter type. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | ID of Snowflake data source. | | Enter credentials | credentialId | 5e4bc5555e6e763beb9db147 | (Optional) The ID of a stored credential for Snowflake. | | Tables | table | RESULTS | Name of the Snowflake table to store results. | | Schemas | schema | PUBLIC | (Optional) The name of the schema containing the table where results are written. | | Database | catalog | OUTPUT | (Optional) The name of the specified database catalog to write output data to. | | Use external stage options | | | | | Cloud storage type | cloudStorageType | s3 | (Optional) Type of cloud storage backend used in Snowflake external stage. Can be one of 3 cloud storage providers: s3/azure/gcp. The default is s3. In the UI, select Use external stage to enable the Cloud storage type field. | | External stage | externalStage | my_s3_stage | Snowflake external stage. In the UI, select Use external stage to enable the External stage field. | | Endpoint URL (for S3 only) | endpointUrl | https://www.example.com/datasets/ | (Optional) Override the endpoint used to connect to S3, for example, to use an API gateway or another S3-compatible storage service. In the UI, for the S3 option in Cloud storage type click Show advanced options to reveal the Endpoint URL field. | | + Add credentials | cloudStorageCredentialId | 6e4bc5541e6e763beb9db15c | (Optional) ID of stored credentials for a storage backend (S3/Azure/GCS) used in Snowflake stage. In the UI, enable the + Add credentials field by selecting This URL requires credentials. | | Write strategy options (for fallback JDBC connection) | | | | | Write strategy | statementType | insert | If you're using a Snowflake external stage the statementType is insert. However, in the UI you have two configuration options: If you haven't configured an external stage, the connection defaults to JDBC and you can select Insert or Update. If you select Update, you can provide a Row identifier.If you selected Use external stage, the Insert option is required. | | Create table if it does not exist (for Insert) | create_table_if_not_exists | true | (Optional) If no existing table is detected, attempt to create one. | | Advanced options | | | | | Commit interval | commitInterval | 600 | (Optional) Defines a time interval, in seconds, between commits to the target database. If set to 0, the batch prediction operation will write the entire job before committing. Default: 600 | Refer to the [example section](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-with-snowflake) for a complete API example. ## Azure Synapse write To use Azure Synapse, supply data destination details using the [Predictions > Job Definitions](https://docs.datarobot.com/en/docs/classic-ui/predictions/batch/batch-dep/batch-pred-jobs.html#set-up-prediction-destinations) tab or the [Batch Prediction API](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/index.html) ( `outputSettings`) as described in the table below. | UI field | Parameter | Example | Description | | --- | --- | --- | --- | | Destination type | type | synapse | Adapter type. | | Data connection parameters | | | | | + Select connection | dataStoreId | 5e4bc5b35e6e763beb9db14a | ID of Synapse data source. | | Enter credentials | credentialId | 5e4bc5555e6e763beb9db147 | (Optional) The ID of a stored credential for Synapse. | | Tables | table | RESULTS | Name of the Synapse table to keep results in. | | Schemas | schema | dbo | (Optional) Name of the schema containing the table where results are written. | | Use external stage options | | | | | External data source | externalDatasource | my_data_source | Name of the identifier created in Synapse for the external data source. | | + Add credentials | cloudStorageCredentialId | 6e4bc5541e6e763beb9db15c | (Optional) ID of a stored credential for Azure Blob storage. | | Write strategy options (for fallback JDBC connection) | | | | | Write strategy | statementType | insert | If you're using a Synapse external stage the statementType is insert. However, in the UI you have two configuration options: If you haven't configured an external stage, the connection defaults to JDBC and you can select Insert, Update, or Insert + Update. If you select Update or Insert + Update, you can provide a Row identifier.If you selected Use external stage, the Insert option is required. | | Create table if it does not exist (for Insert or Insert + Update) | create_table_if_not_exists | true | (Optional) If no existing table is detected, attempt to create it before writing data with the strategy defined in the statementType parameter. | | Create table if it does not exist | create_table_if_not_exists | true | (Optional) Attempt to create the table first if no existing one is detected. | | Advanced options | | | | | Commit interval | commitInterval | 600 | (Optional) Defines a time interval, in seconds, between commits to the target database. If set to 0, the batch prediction operation will write the entire job before committing. Default: 600 | Refer to the [example section](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html#end-to-end-scoring-with-synapse) for a complete API example. --- # Batch prediction use cases URL: https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/pred-examples.html > Examine several end-to-end examples of scoring with API code for both CSV files and external services. The following provides several end-to-end examples of scoring with API code for both CSV files and external services. - End-to-end scoring of CSV files from local files - End-to-end scoring of CSV files on S3 - AI Catalog-to-CSV file scoring - End-to-end scoring from a JDBC PostgreSQL database - End-to-end scoring with Snowflake - End-to-end scoring with Synapse - End-to-end scoring with BigQuery > [!NOTE] Note > These use cases require the [DataRobot](https://datarobot-public-api-client.readthedocs-hosted.com/) API client to be installed. ## End-to-end scoring of CSV files from local files The following example scores a local CSV file, waits for processing to start, and then initializes the download. ``` import datarobot as dr dr.Client( endpoint="https://app.datarobot.com/api/v2", token="...", ) deployment_id = "..." input_file = "to_predict.csv" output_file = "predicted.csv" job = dr.BatchPredictionJob.score_to_file( deployment_id, input_file, output_file, passthrough_columns_set="all" ) print("started scoring...", job) job.wait_for_completion() ``` ### Prediction Explanations You can include Prediction Explanations by adding the desired [Prediction Explanation parameters](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html#prediction-explanations) to the job configuration: ``` job = dr.BatchPredictionJob.score_to_file( deployment_id, input_file, output_file, max_explanations=10, threshold_high=0.5, threshold_low=0.15, ) ``` ### Custom CSV format If your CSV files does not match the default CSV format, you can modify the expected CSV format by setting `csvSettings`: ``` job = dr.BatchPredictionJob.score_to_file( deployment_id, input_file, output_file, csv_settings={ 'delimiter': ';', 'quotechar': '\'', 'encoding': 'ms_kanji', }, ) ``` ## End-to-end scoring of CSV files on S3 ``` import datarobot as dr dr.Client( endpoint="https://app.datarobot.com/api/v2", token="...", ) deployment_id = "616d01a8ddbd17fc2c75caf4" credential_id = "..." s3_csv_input_file = 's3://my-bucket/data/to_predict.csv' s3_csv_output_file = 's3://my-bucket/data/predicted.csv' job = dr.BatchPredictionJob.score_s3( deployment_id, source_url=s3_csv_input_file, destination_url=s3_csv_output_file, credential=credential_id ) print("started scoring...", job) job.wait_for_completion() ``` The same functionality is available for `score_azure` and `score_gcp`. You can also specify the `credential` object itself, instead of a credential ID: ``` credentials = dr.Credential.get(credential_id) job = dr.BatchPredictionJob.score_s3( deployment_id, source_url=s3_csv_input_file, destination_url=s3_csv_output_file, credential=credentials, ) ``` ### Prediction Explanations You can include Prediction Explanations by adding the desired [Prediction Explanation parameters](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html#prediction-explanations) to the job configuration: ``` job = dr.BatchPredictionJob.score_s3( deployment_id, source_url=s3_csv_input_file, destination_url=s3_csv_output_file, credential=credential_id, max_explanations=10, threshold_high=0.5, threshold_low=0.15, ) ``` ## AI Catalog-to-CSV file scoring When using the [AI Catalog](https://docs.datarobot.com/en/docs/classic-ui/data/ai-catalog/catalog.html) for intake, you need the `dataset_id` of an already created dataset. ``` import datarobot as dr dr.Client( endpoint="https://app.datarobot.com/api/v2", token="...", ) deployment_id = "616d01a8ddbd17fc2c75caf4" credential_id = "..." dataset_id = "..." dataset = dr.Dataset.get(dataset_id) job = dr.BatchPredictionJob.score( deployment_id, intake_settings={ 'type': 'dataset', 'dataset_id': dataset, }, output_settings={ 'type': 'localFile', }, ) job.wait_for_completion() ``` ## End-to-end scoring from a JDBC PostgreSQL database The following reads a scoring dataset from the table `public.scoring_data` and saves the scored data back to `public.scored_data` (assuming that table already exists). ``` import datarobot as dr dr.Client( endpoint="https://app.datarobot.com/api/v2", token="...", ) deployment_id = "616d01a8ddbd17fc2c75caf4" credential_id = "..." datastore_id = "..." intake_settings = { 'type': 'jdbc', 'table': 'scoring_data', 'schema': 'public', 'data_store_id': datastore_id, 'credential_id': credential_id, } output_settings = { 'type': 'jdbc', 'table': 'scored_data', 'schema': 'public', 'data_store_id': datastore_id, 'credential_id': credential_id, 'statement_type': 'insert' } job = dr.BatchPredictionJob.score( deployment_id, passthrough_columns_set='all', intake_settings=intake_settings, output_settings=output_settings, ) print("started scoring...", job) job.wait_for_completion() ``` More details about JDBC scoring can be found [here](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html#jdbc-scoring). ## End-to-end scoring with Snowflake The following example reads a scoring dataset from the table `public.SCORING_DATA` and saves the scored data back to `public.SCORED_DATA` (assuming that table already exists). ``` import datarobot as dr dr.Client( endpoint="https://app.datarobot.com/api/v2", token="...", ) deployment_id = "616d01a8ddbd17fc2c75caf4" credential_id = "..." cloud_storage_credential_id = "..." datastore_id = "..." intake_settings = { 'type': 'snowflake', 'table': 'SCORING_DATA', 'schema': 'PUBLIC', 'external_stage': 'my_s3_stage_in_snowflake', 'data_store_id': datastore_id, 'credential_id': credential_id, 'cloud_storage_type': 's3', 'cloud_storage_credential_id': cloud_storage_credential_id } output_settings = { 'type': 'snowflake', 'table': 'SCORED_DATA', 'schema': 'PUBLIC', 'statement_type': 'insert' 'external_stage': 'my_s3_stage_in_snowflake', 'data_store_id': datastore_id, 'credential_id': credential_id, 'cloud_storage_type': 's3', 'cloud_storage_credential_id': cloud_storage_credential_id } job = dr.BatchPredictionJob.score( deployment_id, passthrough_columns_set='all', intake_settings=intake_settings, output_settings=output_settings, ) print("started scoring...", job) job.wait_for_completion() ``` More details about Snowflake scoring can be found in [intake](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html#snowflake-scoring) and [output](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-options.html#snowflake-write) documentation. ## End-to-end scoring with Synapse The following example reads a scoring dataset from the table `public.scoring_data` and saves the scored data back to `public.scored_data` (assuming that table already exists). ``` import datarobot as dr dr.Client( endpoint="https://app.datarobot.com/api/v2", token="...", ) deployment_id = "616d01a8ddbd17fc2c75caf4" credential_id = "..." cloud_storage_credential_id = "..." datastore_id = "..." intake_settings = { 'type': 'synapse', 'table': 'SCORING_DATA', 'schema': 'PUBLIC', 'external_data_source': 'some_datastore', 'data_store_id': datastore_id, 'credential_id': credential_id, 'cloud_storage_credential_id': cloud_storage_credential_id } output_settings = { 'type': 'synapse', 'table': 'SCORED_DATA', 'schema': 'PUBLIC', 'statement_type': 'insert' 'external_data_source': 'some_datastore', 'data_store_id': datastore_id, 'credential_id': credential_id, 'cloud_storage_credential_id': cloud_storage_credential_id } job = dr.BatchPredictionJob.score( deployment_id, passthrough_columns_set='all', intake_settings=intake_settings, output_settings=output_settings, ) print("started scoring...", job) job.wait_for_completion() ``` More details about Synapse scoring can be found in the [intake](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html#synapse-scoring) and [output](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-options.html#synapse-write) documentation. ## End-to-end scoring with BigQuery The following example scores data from a BigQuery table and sends results to a BigQuery table. ``` import datarobot as dr dr.Client( endpoint="https://app.datarobot.com/api/v2", token="...", ) deployment_id = "616d01a8ddbd17fc2c75caf4" gcs_credential_id = "6166c01ee91fb6641ecd28bd" intake_settings = { 'type': 'bigquery', 'dataset': 'my-dataset', 'table': 'intake-table', 'bucket': 'my-bucket', 'credential_id': gcs_credential_id, } output_settings = { 'type': 'bigquery', 'dataset': 'my-dataset', 'table': 'output-table', 'bucket': 'my-bucket', 'credential_id': gcs_credential_id, } job = dr.BatchPredictionJob.score( deployment=deployment_id, intake_settings=intake_settings, output_settings=output_settings, include_prediction_status=True, passthrough_columns=["some_col_name"], ) print("started scoring...", job) job.wait_for_completion() ``` More details about BigQuery scoring can be found in the [intake](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/intake-options.html#bigquery-scoring) and [output](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-options.html#bigquery-write) documentation. --- # API changelogs URL: https://docs.datarobot.com/en/docs/api/reference/changelogs/index.html > Reference the changes introduced to new versions of DataRobot's Python client, R client, and REST API. Changelogs contain curated, ordered lists of notable changes for each versioned release for DataRobot's SDKs and REST API. Reference the table below to view changes for DataRobot's newest versions. | Topic | Description | | --- | --- | | REST API changelog | Changes introduced to new versions of the DataRobot REST API. | | Python client changelog | Changes introduced to new versions of the DataRobot Python client. | | R client changelog | Changes introduced to new versions of the DataRobot R client. | --- # Python client changelog URL: https://docs.datarobot.com/en/docs/api/reference/changelogs/py-changelog/index.html ## 3.19.0 ### New features - Promoted the query-engine package extra out of experimental. QueryEngine , the SQLAlchemy-compatible wrapper, is available at datarobot[query-engine] . - Promoted methods preview_query and execute_update from experimental. These methods allow execution of SQL statements against data stores. - Promoted methods preview and execute_update from experimental. These methods allow execution SQL statements against a database via a JDBC connection. - Updated OtelTrace to retrieve and delete OpenTelemetry traces. - Updated ExecutionEnvironmentAssignPayload to allow for sending pin_version boolean. - Updated Notebooks Image with proper validation for recent changes to API. - Added a dr-dev console script, letting the DataRobot CLI run drdev as dr dev . The drdev command is unchanged. - Added the [application-utils] install extra ( pip install datarobot[application-utils] , Python 3.11+). datarobot.application_utils.persistence is a light async ORM over the DataRobot Agentic Memory Service ( DRMemorySpace , DRSession , DREvent ); datarobot.application_utils.chat_history , built on it, adds Chat / Message domain models and repositories plus an AGUIStorageAgent state machine and StreamPersistenceManager / RunHandle pair for persisting and resuming AG-UI agent streams. - Added typed rate-limit and transport errors to the Memory Service ORM client: HTTP 429 raises DRMemoryRateLimitError carrying retry_after whole seconds parsed from the Retry-After header (both delta-seconds and HTTP-date forms), and request timeouts or transport failures raise DRMemoryUnavailableError with the original httpx exception as __cause__ — both derive from DRMemoryServiceError , so existing DRMemoryServiceError handlers keep working — but httpx exceptions no longer escape request() , so code catching those around ORM calls should catch DRMemoryUnavailableError instead. Also documented multi-principal usage (one lightweight client per user token over a shared identity-free http_client — no client-level auth, non-storing cookie jar) as a supported pattern. ### Enhancements - Added prompt_column_name and response_column_name to SearchStudy.create and SearchStudy , letting you point a syftr search at evaluation dataset columns other than the previously hardcoded promptText and responseText . - Added --timeout / -t and the DRDEV_STARTUP_TIMEOUT environment variable to drdev , allowing users to raise the default 120-second wait for a service. - Updated Connector.create to read the created connector straight from the POST /externalConnectors/ response. The route is synchronous as of DataRobot 11.12, so against those servers the client no longer polls a status object and then re-fetches the connector, cutting three requests down to one. Older servers, which still register connectors asynchronously, keep working. ### Bugfixes - Fixed DeploymentListFilters incorrectly formatting tag keys and tag values, which previously would result in no matches when multiple values were specified. - Surfaced parameter read_timeout on preview() , execute_update , preview_query and execute_update to allow for configuring timeout. - Fixed drdev never probing a service at the end of its startup window, which resulted in the service reporting as timed out if it became ready in the final second. - Fixed drdev hanging instead of exiting when a stopped service left behind a child holding its output pipe open. - Fixed drdev terminating unrelated processes bound to a service port. It now only stops processes carrying this run’s DRDEV_MANAGED marker, and otherwise stops the run unless --force is passed. The marker names the service and its resolved directory, so a leftover from another service or directory needs --force . - Fixed drdev treating a process as the owner of a service port when the port was merely the local port of one of its sockets. Only listening sockets count now. - Fixed drdev starting a service whose port had not come free. It now checks the port after cleanup and names whatever still holds it. - Fixed drdev failing the whole session when a service exited cleanly after finding its port already served by another task. It now checks the port before treating an exit as fatal, so only a genuine crash, or a clean exit that never reached the port, still fails. - Fixed drdev ’s Windows shutdown, where CTRL_BREAK_EVENT always failed and fell back to hard-killing only the top-level task process. - Fixed drdev to reject an out-of-range port while parsing arguments and configuration, instead of silently dropping the service or failing later with a misleading start-up timeout. ### Documentation changes - Revised docstrings across datarobot.models and datarobot.insights to align with the SDK API documentation style guidelines (articles, abbreviations, punctuation, and parameter description formatting). These are documentation-only updates; no API behavior was changed. ### Experimental changes - Added python_version and gpu parameters to experimental pipeline image creation and update — PipelineImage.create , PipelineImage.update , and the Pipelines.create_image / Pipelines.update_image facade — letting you select the Python interpreter version (e.g. "3.11" ) and request GPU support. The python_base_image parameter is deprecated in favour of python_version . ## 3.18.0 ### New features - The DataRobot Python Client now supports Python 3.14. - Added ExecutionEnvironmentVersion.upload to allow uploading of a pre-built Docker image. - Added the output-only system_prompts field to LlmConfig , exposing the set of system prompts explored by a syftr search. The field is populated by the server on the returned search space and is not user-set on create. - Added LLMConfig and LLMType to the datarobot[core] extra. LLMConfig is a self-contained value object holding one LLM instance’s resolved connection parameters (DataRobot endpoint and token plus that instance’s routing fields), so an app can configure several LLMs independently instead of relying on a single global config. LLMConfig.get_llm_type reports how the instance routes (LLM gateway, deployment, NIM, or an external provider) and LLMConfig.to_litellm_params renders it as a litellm_params dict for litellm.Router . DataRobotAppFrameworkBaseSettings.resolve_llm_config builds one from a settings object’s {name}_* fields. - Add backwards compatibility to LLMConfig . ### Enhancements - Updated TemplateMetadata with additional fields provided by server for environment_variables , readiness_probe , liveness_probe , and startup_probe . ### Bugfixes - Fixed Application deserialization to accept a blank description field returned by the API. - Fixed the datarobot build to ship the drdev console script (previously only in the early-access and weekly builds). ### Experimental changes - Updated the experimental Pipelines facade to match the current pipelines-api surface: - Pipelines.run now requires an image_id (the dispatch executes on that execution image), and Pipelines.create_schedule now requires image_id and image_version . Dispatch responses carry image_id / image_version . - Schedules are now flat under the pipeline: Pipelines.list_schedules and Pipelines.delete_schedule no longer take version_id . - Pipelines.create_image and Pipelines.update_image accept python_base_image ; update_image is a full redefinition. PipelineImageVersion now exposes the round-tripped definition and built image_uri . - Pipelines.create accepts optional name / image_id , and Pipelines.update accepts optional file / name / description / image_id (all independent — the file is now optional, so a draft can be renamed or re-described without re-uploading its source). Pipeline detail surfaces input_set_template , image_id , and linked_image . - Added task-execution introspection to Pipelines , backed by the new PipelineTaskExecution : Pipelines.list_tasks , Pipelines.get_task_execution , Pipelines.get_task_result (result download URL + JSON preview), Pipelines.get_task_logs (live pod logs), and Pipelines.get_task_durable_log (durable S3 stdout / stderr ). - Added Pipelines.get_source to fetch the raw pipeline source.py for a draft or locked version, and Pipelines.get_image_logs for image build logs. - Added Pipelines.clone (and the underlying Pipeline.clone ) to clone an existing pipeline into a new draft — copying its source, description, latest input params, and assigned image — backed by the new pipelines-api POST /pipelines/{id}/clone endpoint. The clone name defaults to Clone of and can be overridden via the name argument. - Added the PREPARING value to PipelineDispatchStatus and a new TaskExecutionStatus enum. - Fixed locked-version writes on PipelineInput and PipelineDispatch to address the version by its number in the URL rather than the internal row id returned in the response body, so update / delete / cancel / get_status target the correct version-scoped path. - Fixed Pipeline.list_versions and Pipeline.get_version to run version payloads through the SDK’s camelCase-to-snake_case conversion; previously task_names , python_version , resource_bundle , error_detail , and created_at silently came back None . The Pipelines.list_versions and Pipelines.get facade dicts now also surface python_version , resource_bundle , and error_detail . - Fixed the experimental pipelines list_* methods ( Pipelines.list , Pipelines.list_versions , Pipelines.list_inputs , Pipelines.list_runs , Pipelines.list_schedules , and Pipelines.list_images ) to follow pagination and return the complete result set instead of silently truncating at the first page. The now-unused offset / limit parameters were removed. - Standardized the locked-version keyword on the Pipelines facade to version (was version_id on create_input , list_inputs , get_input , delete_input , and create_schedule ). - Added node_id support to the experimental pipelines per-task read methods ( Pipelines.get_task_execution , Pipelines.get_task_result , Pipelines.get_task_logs , Pipelines.get_task_durable_log ) to address a specific invocation of a fan-out task (the same @task run at multiple graph nodes), which pipelines-api now serves via ?nodeId= and answers with a 409 for an ambiguous bare task_id . PipelineTaskExecution and Pipelines.list_tasks now surface node_id and graph_node_id so callers can discover the invocations to disambiguate. - Fixed bug in QueryEngine.execute where newlines in queries with leading comments could be removed, turning the whole query into a no-op. ## 3.17.0 ### New features - Added the [pipelines] install extra for datarobot-early-access : pip install datarobot-early-access[pipelines] . The extra bundles covalent and covalent-cloud directly in the wheel — no additional package index required. - Added SupportedOpenAIEmbeddingModel to be used in SupportedEmbeddings . - Added OpenAIEmbeddingModelValidation to provide a basic CRUD functionality for OpenAI-compatible embedding models. - Create classes for working with user MCP servers: - Created file-like class DataRobotFile to encapsulate and support reading and writing operations on files in the DataRobot file system. - Created user MCP server class ToolInUserMCPServerDeployment to save/list/delete tool metadata of one user MCP server deployment. - Created user MCP server class PromptInUserMCPServerDeployment to save/list/delete prompt metadata of one user MCP server deployment. - Created user MCP server class ResourceInUserMCPServerDeployment to save/list/delete resource metadata of one user MCP server deployment. - Created user MCP server class ToolInUserMCPServerVersion to list tool metadata of one user MCP server version. - Created user MCP server class PromptInUserMCPServerVersion to list prompt metadata of one user MCP server version. - Created user MCP server class ResourceInUserMCPServerVersion to list resource metadata of one user MCP server version. - Moved CustomScoringMetric , CustomScoringMetricVersion , and CustomScoringMetricVersionFileContents out of the _experimental module into datarobot.models . ### Enhancements - Added field openai_embedding_validations to SupportedEmbeddings . - Updated field embedding_validation in {class} ChunkingParameters to support OpenAIEmbeddingModelValidation . - Added deduplication_key to Session.create to prevent duplicate session creations. When a live session with the same key already exists in the memory space, the server returns 409 and the SDK raises MemorySessionDeduplicationError , which exposes existing_session_id so concurrent callers can adopt the winning session without a follow-up lookup. - Added deduplication_key to MemorySpace.create and a deduplication_key filter to MemorySpace.list to prevent duplicate memory space creations. When a live memory space with the same key already exists for the user, the server returns 409 and the SDK raises MemorySpaceDeduplicationError , which exposes existing_memory_space_id so concurrent callers can adopt the winning memory space without a follow-up lookup. - Added Session.post_events and Session.update_events to batch-insert or batch-update up to 200 events atomically within a session. - Documented that lifecycle_strategies can be attached to Session . - Added more type annotations and mypy coverage. - Added Multilabel as an option for Custom Model Target Types . - Added CustomScoringMetricVersionFileContents and CustomScoringMetricVersion.get_file_contents to retrieve the raw metadata.yml and custom_metrics.py contents backing a custom scoring metric version. An instance shortcut CustomScoringMetricVersion.get_files is also provided. - Fixed CustomScoringMetric.list so isArchived is omitted when listing non-archived metrics ( is_archived=False ); the API returns 400 if isArchived=false is sent explicitly. - Removed target_type from CustomScoringMetricVersion . ### Documentation changes - Updated text for examples of private_key_str credentials to avoid triggering false positive on security scans. - ### Experimental changes - Renamed PipelineEnvironment / PipelineEnvironmentVersion to PipelineImage / PipelineImageVersion , PipelineEnvironmentStatus to PipelineImageStatus , and field environment_id to image_id to align with the updated API terminology. The URL path /pipelines/environments is now /pipelines/images . - Renamed the Pipelines facade methods create_env / list_envs / update_env / delete_env to create_image / list_images / update_image / delete_image (and the env_id parameter and return key to image_id ), completing the pipeline environment→image rename. - Added preview_table to support previewing data from a table in a data store. - Extend JdbcPreviewData with df property to parse data preview response to pandas DataFrame. - Added preview_query to support executing arbitrary SQL statement query against a data store. - Added execute_update to support executing arbitrary SQL update statement against a data store. - Update JdbcPreview.preview to use the new experimental endpoint, /jdbcPreviewQuery . - Added experimental Pipeline module for authoring, dispatching, and managing DataRobot Pipelines. Includes support for pipeline inputs, environments, scheduling, and result retrieval. - Added Pipelines.get_task and the backing PipelineTask / TaskParameter classes to fetch per-task detail (source, function signature, and pipeline inputs) for a draft or locked-version task. Task IDs are discovered from the numeric taskId field now present on each graph node. - Added JdbcPreview.execute_update to support executing SQL update statements against JDBC databases without a data store. - Removed user MCP server related APIs out from _experimental module and moved them to datarobot.models module. - Updated JdbcPreview.preview , JdbcPreview.execute_update , preview_query and execute_update to support parameter binding with new parameter bind_parameters . - Created package extra datarobot[query-engine] with ease-of-use functionality for querying against data stores, JDBC connections, and integration with SQLAlchemy . - Added experimental class QueryEngine in package extra datarobot[query-engine] as a wrapper for executing SQL statements against data stores and JDBC connections. ### Deprecation summary - Remove deprecated Connector.update . - Remove deprecated parameter file_path on Connector.update . ## 3.16.0 ### New features - Added create_dr_resource to build an OpenTelemetry Resource with DataRobot-standard attributes ( datarobot.service.priority , datarobot.application.id , k8s.pod.name , service.version ). Available via the new datarobot[otel] extra. - Added get_shared_roles to fetch list roles for users, groups, and organizations for a dataset. - Added modify_shared_roles to grant, remove, or update roles for users, groups, or organizations for a dataset. - Added get_shared_roles to fetch list roles for users, groups, and organizations for a files container. - Added modify_shared_roles to grant, remove, or update roles for users, groups, or organizations for a files container. ### Enhancements - UseCase.share now supports sharing with organizations and groups in addition to users. Pass share_recipient_type=SHARING_RECIPIENT_TYPE.ORGANIZATION or SHARING_RECIPIENT_TYPE.GROUP in a SharingRole to grant access accordingly. - Added custom_instructions to MemorySpace to support custom fact-extraction prompts; pass None to revert to the default memory extraction prompt. - Added llm_base_url to MemorySpace to configure the chat API URL used for memory extraction. By default, the memory service uses the DataRobot LLM gateway; set this only when the default does not work — for example, in air-gapped environments or when the LLM model you need is not provided by the gateway and cannot be added. - Updated the docstring for {meth} get , and updated {meth} from_server_data for adding a guard for empty response from the server. - SearchStudy.create now returns immediately in RUNNING state by default ( wait_for_completion=False ) instead of blocking with a hard 600-second timeout. Pass wait_for_completion=True with an optional max_wait (default 10800 seconds) to block until the study finishes, or call wait_for_completion on the returned object to wait later. - Added new pruner parameters to SearchStudy and related objects to handle search pruning. - Improved put to use stages when uploading multiple local files at once. - Improved pipe_file to avoid making an extra call to exists . - Added prediction environments and prediction instances as entity types that key-value pairs can be added to. ### Experimental changes - Added CustomScoringMetric and CustomScoringMetricVersion for versioned custom scoring metrics (list/create/get/update/delete metric containers and list/create/get/delete uploaded versions using HTTP form file uploads). tags use [{"name": str, "value": str}, ...] structures; CustomScoringMetric.list supports an optional tags query filter. ### Bugfixes - Fixed SearchStudy failing with a DataError when the server returns a blank user_name (e.g. for service accounts or users with no display name configured). - Relaxed validation for MetricMetadata to make value optional. - Fixed datarobot.auth.authlib.oauth.AsyncOAuth.get_user_info failing under authlib 1.7.0+ by removing the expires_at=0 sentinel from the token constructed for the userinfo call; also removed the temporary authlib<1.7.0 upper-bound pin. - Removed ‘citations’ from the list of supported objectives for syftr. - Fix asynchronous status resolution in syftr search. - Fixed getenv() raising TypeError: argument of type 'bool' is not iterable when a runtime parameter has type: "boolean" (e.g. MLOPS_RUNTIME_PARAM_* set to {"type": "boolean", "payload": false} ). - Fixed AnomalyAssessmentRecord.list not filtering by backtest when backtest=0 was specified. ### Documentation changes - Updated genai_example.rst to use the correct imports and class names. - Update documentation for various methods on DataRobotFileSystem . ## 3.15.0 ### New features - Added MemorySpace to provide a basic CRUD functionality for Memory Spaces. - Added Session and Event to simplify building agentic applications with chat interfaces. - Added SearchStudy to run LLM blueprint searches. ## 3.14.0 ### New features - Extended the project advanced options available when setting a target to include new parameter custom_metrics_losses_info (part of the AdvancedOptions object; API field customMetricsLossesInfo ), for the Mongo ID of custom metrics / losses metadata. - Added JdbcPreview for previewing data from a JDBC URL by executing SQL without creating a data store; use preview to get a JdbcPreviewData . - Promoted DataRobotFileSystem out of experimental. - Promoted FilesDetails out of experimental. - Promoted experimental methods from Files out of experimental. - Added OtelStats to retrieve OpenTelemetry record counts by service. - Added FEATURE_DISCOVERY_PRIVATE_PREVIEW to RecipeType for backward compatibility with legacy recipes that could not be migrated to the new version. ### Enhancements - Added checksum to File . - Added SAP_AI_CORE to PredictionEnvironmentPlatform . ### Bugfixes - Fixed an issue which caused an error when Feature Lists had an empty description in get_relationships_configuration - Fixed deserialization of ProjectOptions when loading project options from the server: feature_engineering_prediction_point , user_partition_col , and each entry in external_predictions are column names returned as strings by the API (the client previously expected Feature -shaped data, which could raise validation errors). ## 3.13.0 ### New features - Added OtelMetrics to retrieve and delete OpenTelemetry metric data. - Added delete to delete OpenTelemetry log entries. - Added the code_challenge_method property in OAuthProviderConfig to support the PKCE authorization code flow. client_secret was also made optional, as some providers do not require it in the PKCE flow. - Added support for Box JWT credentials via Credential.create_box_jwt and the CredentialTypes.BOX_JWT credential type. - Added get_agent_card , upload_agent_card , and delete_agent_card methods to Deployment for managing A2A agent cards. - Added is_a2a_agent filter parameter to DeploymentListFilters to filter deployments by whether they are A2A agents. ### Enhancements - Added MySQL support to the DataWranglingDialect enum. - Improved typing support for RecipePreview attribute result_schema . - Added the runtime_parameters field to CustomModelVersion.create_from_previous and Job.update methods. This supports a new API attribute that enables the creation of runtime parameters without requiring schema definitions in metadata files. - Added created_at to File . ### Bugfixes - Fixed an issue where VectorDatabase.deploy was not sending its parameters to the DataRobot API. - Fixed an issue where OAuthToken.from_dict was not populating id_token . - Fixed an issue where BatchPredictionJob.score sent improperly serialized columnNamesRemapping to the DataRobot API. ### API changes - Revert handling of deprecated parameter inputs on Recipe.from_dataset when an empty list is passed. Prior to release 3.10.0, an empty list was silently ignored, afterwards an empty list would cause an error. This release reverts the prior behavior. - CustomApplicationSource and CustomApplication no longer require the fields creator_first_name and creator_last_name to be set. ### Documentation changes - Add experimental documentation for DataRobotFileSystem , DataRobotFile and DataRobotFSMap . ### Experimental changes - Implement additional methods for DataRobot file system and Files API with fsspec implementation DataRobotFileSystem : - mv : Move files or directories from one path to another. Supports glob patterns, recursive expansion, and list-of-sources. Same-catalog moves use PATCH when possible; otherwise copy then delete. - mv_file : Move a single file or directory from path1 to path2. - glob : Finds files and directories by glob-matching. - sign : Returns a signed URL for a file path. - open : Open a file in the DataRobot file system for reading or writing. - touch : Create or overwrite an empty file at a given path in the DataRobot file system. - cat : Retrieve contents of one or more files. - put_file : Copy file from local to the DataRobot file system. - put_from_url : Load file(s) from a URL into a directory in the DataRobot file system (supports archives and optional subfolder path). - Added upload_from_url to FilesExperimental to load file(s) from a URL into a catalog item. - clone_catalog_item_dir : Clone a catalog item directory to create a new catalog item directory. - get_mapper : Get a mutable mapping object DataRobotFSMap to interact with the DataRobot file system using a key-value pattern. - copy : Copy one or more paths between locations in the DataRobot file system. - put_from_data_source : Upload file or folder from a DataRobot data source DataSource . - created : Get created timestamp for a file. - Create classes for working with user MCP servers: - Created file-like class DataRobotFile to encapsulate and support reading and writing operations on files in the DataRobot file system. - Create user MCP server class ToolInUserMCPServerDeployment to save/list/delete tool metadata of one user MCP server deployment. - Create user MCP server class PromptInUserMCPServerDeployment to save/list/delete prompt metadata of one user MCP server deployment. - Create user MCP server class ResourceInUserMCPServerDeployment to save/list/delete resource metadata of one user MCP server deployment. - Create user MCP server class ToolInUserMCPServerVersion to list tool metadata of one user MCP server version. - Create user MCP server class PromptInUserMCPServerVersion to list prompt metadata of one user MCP server version. - Create user MCP server class ResourceInUserMCPServerVersion to list resource metadata of one user MCP server version. ## 3.12.0 ### Configuration changes - Added pytz and python-dateutil as dependencies. These were implicitly required by the client for date and time handling. - Updated datarobot[core] to require psutil>=7.2.1 (previously unpinned). This ensures compatibility in environments where an older psutil version may already be installed. ### New features - Added OtelSingleMetricValue to retrieve OpenTelemetry metric data for a single metric without configuration. - Added OtelMetricAggregatedValues to retrieve the latest OpenTelemetry metric data for configured metrics. - Added a new class ConfusionMatrix to interact with confusion matrix insights. ### Experimental changes - Added experimental package datarobot._experimental.fs to store DataRobot file system functionality. - Configured datarobot-early-access[fs] as an optional extra add-on to the datarobot-early-access package. - Added initial support for DataRobot file system and Files API with fsspec implementation DataRobotFileSystem . Added implementations for the following methods: - ls : Lists files and directories under a directory path. - info : Gets information about a file or folder. - cp_file : Copies file or directory to another path. - rm_file : Recursively deletes one or more files or folders. - rm : Deletes one or more file or folder paths. Supports use of glob patterns, recursive, and non-recursive search. ### Documentation changes - Added API reference documentation for ConfusionMatrix . - Added usage examples for model performance insights in a new guide showing how to use ConfusionMatrix.get() and ConfusionMatrix.compute() methods. ## 3.11.0 ### New features - Added Recipe.publish_to_dataset to recipe API for easy publishing to dataset. ### Enhancements - Added tag data and create/update/delete functions to Deployment . - Added PromptTemplateVersion.list_all to list prompt template versions across multiple templates with optional filtering by template IDs. - Extended LLM Settings to support creating LLM Blueprints for Agentic Workflows in LLMBlueprint llm_setting parameter now supports - Added the available_litellm_endpoints field to LLMGatewayCatalogEntry to define supported endpoints for each LLM gateway model (includes supports_chat_completions and supports_responses which correspond to /chat/completions and /responses ). - Edited LLMGatewayCatalog.list to include the optional parameter chat_completions_supported_only which filters to only list models that support the /chat/completions route. - Added support for external OAuth provider credentials, including the new CredentialTypes.EXTERNAL_OAUTH_PROVIDER enum value and the creation helper Credential.create_external_oauth_provider . - custom_model_version_id ### Bugfixes - Made the prediction environment in Challenger optional, since it is not always provided by the server. - Restore multipart file uploads via client.request() broken in v3.10.0. ### Documentation changes - Added API reference documentation for RocCurve , LiftChart , and Residuals classes that were previously added in 3.7.0. They replace the deprecated methods Model.request_lift_chart and similar. - Added comprehensive usage examples for model performance insights in a new guide showing how to use RocCurve.get() , LiftChart.get() , and Residuals.get() methods. ## 3.10.0 ### New features - Added PromptTemplate to manage prompt templates with versioning support in the GenAI namespace. - Added PromptTemplateVersion to manage specific versions of prompt templates, including the ability to render prompts with variable substitution. - Added Dataset.get_raw_sample_data to dataset raw data as Pandas DataFrame. - Added the following data wrangling capabilities: - Method Recipe.update to update a recipe using an instance method. Previously, you had to use a class method and explicitly specify a recipe ID. - Method Recipe.list class method to list recipes. - Method Recipe.generate_sql_for_operations class method for generating SQL from an arbitrary list of operations. - Wrangling Operation DedupeRowsOperation to support removing duplicate rows for recipes. - Wrangling Operation FindAndReplaceOperation to support find and replace operations in recipes. - Sampling Operation LimitSamplingOperation to support sampling the first n rows of data from an input. - Sampling Operation TableSampleSamplingOperation to support randomly sampling x% of data from an input using a table sampling method. - Method RecipeDatasetInput.from_dataset to quickly create recipe dataset input using a dataset. - Wrangling Operation AggregationOperation to support applying aggregate transformations in a recipe. - Wrangling Operation JoinOperation to support joining datasets to the data in a recipe. - Method Recipe.set_settings to update the settings of a recipe. Also added support for instance method Recipe.update to update settings with settings parameter. ### Enhancements - Added PromptTemplateVersion.to_fstring to convert prompt templates from {{ variable }} format to Python f-string {variable} format for use with native Python string formatting. - Updated ExecutionEnvironment.list to accept additional parameter for is_public . - Updated MetricInsights.list to add additional parameters with_aggregation_types_only , production_only , and completed_only . - Exposed Recipe for import directly from the datarobot package. - Added RandomDownsamplingOperation and SmartDownsamplingOperation as wrappers to use when setting downsampling on a recipe. - Updated get_authorization_context to return an empty context if no authorization context is set instead of raising an error. - Made variables parameter optional in PromptTemplateVersion.create and PromptTemplate.create_version . Defaults to None , which sends an empty list to the API. ### Bugfixes - Fixed a bug on updating Recipe recipe_type . - Fixed a bug on updating Recipe downsampling to None . - Always camelize PUT request json body under the hood. ### API changes - Warning: Multipart file uploads via client.request() are no longer possible and will be addressed in a future release. - Removed pagination parameters from OtelMetricSummary.list . - Removed pagination values (e.g., count , next , and previous ) from OtelMetricSummary . - Added the name attribute to the Recipe class. - Deprecated the parameter operations from Recipe.get_sql . This should avoid confusion when converting arbitrary operations. This functionality has been duplicated to Recipe.generate_sql_for_operations . The operations parameter will be removed in 3.12. - Deprecated Recipe.retrieve_preview in favor of Recipe.get_preview . Recipe.get_preview will return the wrapper object RecipePreview . The RecipePreview object has a .df attribute containing the preview data as a Pandas DataFrame. - Added optional trace_id and span_id query parameters to OtelLogEntry.list . - Added optional trace_id and span_id to OtelLogEntry . - Deprecated parameter inputs on Recipe.from_dataset . Added new parameter sampling to use instead. ### Configuration changes - Removed black and pylint as dev dependencies. ruff is now used instead. ### Documentation changes - Updated documentation page for Recipes to include new methods. Created the new sections “Recipe Inputs”, “Recipe Operations”, and “Enums and Helpers” to better organize the Recipe components. - Updated documentation for recipe operations, related enums, and helper classes to document arguments and provide examples. - Improved documentation for recipe sampling operations. - Improved documentation for recipe creation methods Recipe.from_dataset and Recipe.from_data_store . - Improved documentation for the recipe input wrapper classes RecipeDatasetInput and JDBCTableDataSourceInput . ## 3.10.0b0 This was an experimental release and is subsumed by 3.10.0rc0. ## 3.9.1 ### New features - Added CustomApplication to manage custom applications with detailed resource information and operational controls. - Added CustomApplicationSource to manage custom application sources (templates for creating custom applications). - Added optional parameter version_id=None to Files.download and Files.list_contained_files to allow non-blocking upload. - Promoted FilesStage out of experimental. - Added Files.clone , Files.create_stage , Files.apply_stage and Files.copy , which were previously experimental. - Added DataRobotAppFrameworkBaseSettings as a Pydantic Settings class for managing configurations for agentic workflows and applications. ### Enhancements - Improved the string representation of RESTClientObject to include endpoint and client version. ## 3.9.0 ### New features - Added OtelMetricConfig to manage and control the display of OpenTelemetry metric configurations for an entity. - Added OtelLogEntry to list the OpenTelemetry logs associated with an entity. - Added OtelMetricSummary to list the reported OpenTelemetry metrics associated with an entity. - Added OtelMetricValue to list the OpenTelemetry metric values associated with an entity. - Added LLMGatewayCatalog to get available LLMs from the LLM Gateway. - Added ResourceBundle to list defined resource bundles. ### Enhancements - Added CustomTemplate.create to allow users to create a new CustomTemplate . - Updated CustomTemplate.list to accept additional parameters for publisher , category , and show_hidden for improved queries. - Updated CustomTemplate.update to accept additional parameters for file , enabled , and is_hidden to set additional fields. - Added is_hidden member to CustomTemplate to align with server data. - Added custom_metric_metadata member to TemplateMetadata to allow creating custom-metric templates using APIObjects. - Added CustomTemplate.download_content to allow users to download content associated with an items file. - Added new attribute spark_instance_size to RecipeSettings . - Added set_default_credential to DataStore.test to set the provided credential as default when the connection test is successful. - Added optional parameter data_type to Connector.list to list connectors which support specified data type. - Added optional parameter data_type to DataStore.list to list data stores which support specified data type. - Added optional parameters retrieval_mode and maximal_marginal_relevance_lambda to VectorDatabaseSettings to select the retrieval mode. - Added optional parameter wait_for_completion=True to Files.upload and Files.create_from_url to allow non-blocking upload. - Added support for file to UseCase.add and UseCase.remove . - Added GridSearchArguments with GridSearchArguments.to_api_payload for creating grid search arguments for advanced tuning jobs. - Added GridSearchSearchType . An enum to define supported grid search types. - Added GridSearchAlgorithm . An enum to define supported grid search algorithms. - Added optional parameters include_agentic , is_agentic , for_playground , and for_production to ModerationTemplate.list to include/filter for agentic templates and to fetch templates specific to playground or production. - Improved equality comparison for APIObject to only look at API related fields. - Added optional parameters tag_keys and tag_values to Deployment.list to filter search results by tags. ### Bugfixes - Fixed validator errors in EvaluationDatasetMetricAggregation . - Fixed Files.download() to download a correct file from the files container. - Fixed the enum values of ModerationGuardOotbType . - Fixed a bug where ModerationTemplate.find was unable to find a template when given a name. - Fixed a bug where OverallModerationConfig.find was unable to find a given entity’s overall moderation config. ### Documentation changes - Added OCREngineSpecificParameters, DataRobotOCREngineType and DataRobotArynOutputFormat to OCR job resources section ## 3.8.0 This release adds support for Notebooks and Codespaces and unstructured data in the Data Registry, and the Chunking Service v2. There are improvements related to playgrounds, vector databases, agentic workflows, incremental learning, and datasets. This release focuses heavily on file management capabilities. There are two new package extras: `auth` and `auth-authlib`. The `auth` extra provides OAuth2 support, while the `auth-authlib` extra provides OAuth2 support using the Authlib library. ### New features #### GenAI - Added AGENTIC_WORKFLOW target type. - Added VectorDatabase.send_to_custom_model_workshop to create a new custom model from a vector database. - Added VectorDatabase.deploy to create a new deployment from a vector database. - Added optional parameters vector_database_default_prediction_server_id , vector_database_prediction_environment_id , vector_database_maximum_memory , vector_database_resource_bundle_id , vector_database_replicas , and vector_database_network_egress_policy to LLMBlueprint.register_custom_model to allow specifying resources in cases where we automatically deploy a vector database when this function is called. - Added ReferenceToolCall for creating tool calls in the evaluation dataset. - Added ReferenceToolCalls to represent a list of tool calls in the evaluation dataset. - Added VectorDatabase.update_connected to add a dataset and optional additional metadata to a connected vector database. #### Notebooks and Codespaces The Notebook and Codespace APIs are now GA and the related classes have been promoted to the stable client. - Changed name of the Notebook run() method to be Notebook.run_as_job . - Added support for Codespaces to the Notebook.is_finished_executing method. - Added the NotebookKernel.get method. - Added the NotebookScheduledJob.cancel method. - Added the NotebookScheduledJob.list method. - Added the Notebook.list_schedules method. #### Unstructured Data The client now supports unstructured data in the Data Registry. - Added the Files class to manage files on the DataRobot platform. The class supports file metadata including the description, creation date, and creator information. - Use Files.get to retrieve file information. - Use Files.upload as a convenient facade method to upload files from URLs, file paths, or file objects (does not support DataFrames). - Use Files.create_from_url to upload a new file from a URL. - Use Files.create_from_file to upload a new file from a local file or file-like object. - Use Files.create_from_data_source to create a new file from a DataSource. - Use Files.list_files to retrieve all individual files contained within a Files object. This is useful for Files objects ingested from archives that contain multiple files. - Use Files.download to download a file’s contents. - Use Files.modify to update a file’s name, description, and/or tags. - Use Files.update to refresh a file object with the latest information from the server. - Use Files.delete to soft-delete a file. - Use Files.un_delete to restore a previously deleted file. - Use Files.search_catalog to search for files in the catalog based on name, tags, or other criteria. - Added the FilesCatalogSearch class to represent file catalog search results with metadata such as catalog name, creator, and tags. - Added the File class to represent individual files within a Files archive. The class provides information about individual files such as name, size, and path within the archive. #### OAuth The client provides better support for OAuth2 authorization workflows in applications using the DataRobot platform. These features are available in the datarobot.auth module. - Added the methods set_authorization_context and get_authorization_context to handle context needed for OAuth access token management. - Added the decorator datarobot_tool_auth to inject OAuth access tokens into the agent tool functions. #### Other Features - Introduced support for Chunking Service V2. The chunking_service_v2 classes have been moved out of the experimental directory and are now available to all users. - Added Model.continue_incremental_learning_from_incremental_model to continue training of the incremental learning model. - Added optional parameter chunk_definition_id in Model.start_incremental_learning_from_sample to begin training using new chunking service. - Added a new attribute snapshot_policy to datarobot.models.RecipeDatasetInput to specify the snapshot policy to use. - Added a new attribute dataset_id to datarobot.models.JDBCTableDataSourceInput to specify the exact dataset ID to use. - Added Dataset.create_version_from_recipe to create a new dataset version based on the Recipe. ### Enhancements - Added the use_tcp_keepalive parameter to Client to enable TCP keep-alive packets when connections are timing out, enabled by default. - Enabled Playground to create agentic playgrounds via input param playground_type=PlaygroundType.AGENTIC . - Extended PlaygroundOOTBMetricConfiguration.create with additional reference column names for agentic metrics. - Updated CustomTemplate.list to return all custom templates when no offset is specified. - Extended MetricInsights.list with the option to pass llm_blueprint_ids . - Extended OOTBMetricConfigurationRequest and OOTBMetricConfigurationResponse with support for extra_metric_settings , which provides an additional configuration option for the Tool Call Accuracy metric. - Extended VectorDatabase.create to support creation of connected vector databases via input param external_vector_database_connection . - Extended VectorDatabase.create to support an additional metadata dataset via input params metadata_dataset_id and metadata_combination_strategy . - Extended VectorDatabase.update to support updating the credential used to access a connected vector database via input param credential_id . - Extended VectorDatabase.download_text_and_embeddings_asset to support downloading additional files via input param part . - Added a new attribute engine_specific_parameters to datarobot.models.OCRJobResource to specify OCR engine specific parameters. - Added docker_image_uri to datarobot.ExecutionEnvironmentVersion . - Added optional parameter docker_image_uri to ExecutionEnvironmentVersion.create . - Changed parameter docker_context_path in ExecutionEnvironmentVersion.create to be optional. - Added a new attribute image_id to datarobot.ExecutionEnvironmentVersion . ### Bugfixes - Fixed PlaygroundOOTBMetricConfiguration.create by using the right payload for customModelLLMValidationId instead of customModelLlmValidationId . - Fixed datarobot.models.RecipeDatasetInput to use correct fields for to_api . - Fixed EvaluationDatasetConfiguration.create to use the correct payload for is_synthetic_dataset . ### Deprecation summary - Remove unreleased Insight configuration routes. These were replaced with the new MetricInsights class, and insight specific configurations. - Deployment.create_from_learning_model method is deprecated. Please first register the leaderboard model with RegisteredModelVersion.create_for_leaderboard_item , then create a deployment with Deployment.create_from_registered_model_version . ### Documentation changes - Updated the example for GenAI to show creation of a metric aggregation job. ### Experimental changes - Added VectorDatabase with a new attribute external_vector_database_connection added to the VectorDatabase.create() method. - Added attribute version to DatasetInfo to identify the analysis version. - Added attribute dataset_definition_info_version to ChunkDefinition to identify the analysis information version. - Added a version query parameter to the DatasetDefinition class, allowing users to specify the analysis version in the get method. - Added DatasetDefinitionInfoHistory with the DatasetDefinitionInfoHistory.list method to retrieve a list of dataset information history records. - Added the DatasetDefinitionInfoHistory.list_versions method to retrieve a list of dataset information records. ## 3.7.0 ### New features - The DataRobot Python Client now supports Python 3.12 and Python 3.13. - Added Deployment.get_retraining_settings to retrieve retraining settings. - Added Deployment.update_retraining_settings to update retraining settings. - Updated RESTClientObject to retry requests when the server returns a 104 connection reset error. - Added support for datasphere as an intake and output type in batch predictions. - Added Deployment.get_accuracy_metrics_settings to retrieve accuracy metrics settings. - Added Deployment.update_accuracy_metrics_settings to update accuracy metrics settings. - Added CustomMetricValuesOverSpace to retrieve custom metric values over space. - Added CustomMetric.get_values_over_space to retrieve custom metric values over space. - Created ComplianceDocTemplateProjectType , an enum to define project type supported by the compliance documentation custom template. - Added attribute project_type to ComplianceDocTemplate to identify the template supported project type. - Added optional parameter project_type in ComplianceDocTemplate.get_default to retrieve the project type’s default template. - Added optional parameter project_type in ComplianceDocTemplate.create to specify the project type supported by the template to create. - Added optional parameter project_type in ComplianceDocTemplate.create_from_json_file to specify the project type supported by the template to create. - Added optional parameter project_type in ComplianceDocTemplate.update to allow updating an existing template’s project type. - Added optional parameter project_type in ComplianceDocTemplate.list to allow to filtering/searching by template’s project type. - Added ShapMatrix.get_as_csv to retrieve SHAP matrix results as a CSV file. - Added ShapMatrix.get_as_dataframe to retrieve SHAP matrix results as a dataframe. - Added a new class LiftChart to interact with lift chart insights. - Added a new class RocCurve to interact with ROC curve insights. - Added a new class Residuals to interact with residuals insights. - Added Project.create_from_recipe to create Feature Discovery projects using recipes. - Added an optional parameter recipe_type to datarobot.models.Recipe.from_dataset() to create Wrangling recipes. - Added an optional parameter recipe_type to datarobot.models.Recipe.from_data_store() to create Wrangling recipes. - Added Recipe.set_recipe_metadata to update recipe metadata. - Added an optional parameter snapshot_policy to datarobot.models.Recipe.from_dataset() to specify the snapshot policy to use. - Added new attributes prediction_point , relationships_configuration_id and feature_discovery_supervised_feature_reduction to RecipeSettings . - Added several optional parameters to ExecutionEnvironment for list , create and update methods. - Added optional parameter metadata_filter to ComparisonPrompt.create . - Added CustomInferenceModel.share to update access control settings for a custom model. - Added CustomInferenceModel.get_access_list to retrieve access control settings for a custom model. - Added new attribute latest_successful_version to ExecutionEnvironment . - Added Dataset.create_from_project to create datasets from project data. - Added ExecutionEnvironment.share to update access control settings for an execution environment. - Added ExecutionEnvironment.get_access_list to retrieve access control settings an execution environment. - Created ModerationTemplate to interact with LLM moderation templates. - Created ModerationConfiguration to interact with LLM moderation configuration. - Created CustomTemplate to interact with custom-templates elements. - Extended the advanced options available when setting a target to include parameter: ‘feature_engineering_prediction_point’(part of the AdvancedOptions object). - Added optional parameter substitute_url_parameters to DataStore for list and get methods. - Added Model.start_incremental_learning_from_sample to initialize the incremental learning model and begin training using the chunking service. Requires the “Project Creation from a Dataset Sample” feature flag. - Added NonChatAwareCustomModelValidation as the base class for CustomModelVectorDatabaseValidation and CustomModelEmbeddingValidation . In contrast, CustomModelLLMValidation now implements the create and update methods differently to interact with the deployments that support the chat completion API. - Added optional parameter chat_model_id to CustomModelLLMValidation.create and CustomModelLLMValidation.update to allow adding deployed LLMs that support the chat completion API. - Fixed ComparisonPrompt not being able to load errored comparison prompt results. - Added optional parameters retirement_date , is_deprecated , and is_active to LLMDefinition and added an optional parameter llm_is_deprecated to the MetricMetadata to expose LLM deprecation and retirement-related information. ### Enhancements - Added Deployment.share as an alias for Deployment.update_shared_roles . - Internally use the existing input argument max_wait in CustomModelVersion.clean_create , to set the READ request timeout. ### Bugfixes - Made user_id and username fields in management_meta optional for PredictionEnvironment to support API responses without these fields. - Fixed the enum values of ComplianceDocTemplateType . - Fixed the enum values of WranglingOperations . - Fixed the enum values of DataWranglingDialect . - Playground id parameter is no longer optional in EvaluationDatasetConfiguration.list - Copy insights path fixed in MetricInsights.copy_to_playground - Missing fields for prompt_type and warning were added to PromptTrace . - Fixed a query parameter name in SidecarModelMetricValidation.list . - Fix typo in attribute VectorDatabase : metadata_columns which was metada_columns - Do not camelCase metadata_filter dict in ChatPrompt.create - Fixed a Use Case query parameter name in CustomModelLLMValidation.list , CustomModelEmbeddingValidation.list , and CustomModelVectorDatabaseValidation.list . - Fixed featureDiscoverySettings parameter name in RelationshipsConfiguration.create and RelationshipsConfiguration.replace . ### API changes - Method CustomModelLLMValidation.create no longer requires the prompt_column_name and target_column_name parameters, and can accept an optional chat_model_id parameter. The parameter order has changed. If the custom model LLM deployment supports the chat completion API, it is recommended to use chat_model_id now instead of (or in addition to) specifying the column names. ### Deprecation summary - Removed the deprecated capabilities attribute of Deployment . - Method Model.request_lift_chart is deprecated and will be removed in favor of LiftChart.compute . - Method Model.get_lift_chart is deprecated and will be removed in favor of LiftChart.get . - Method Model.get_all_lift_charts is deprecated and will be removed in favor of LiftChart.list . - Method Model.request_roc_curve is deprecated and will be removed in favor of RocCurve.compute . - Method Model.get_roc_curve is deprecated and will be removed in favor of RocCurve.get . - Method Model.get_all_roc_curves is deprecated and will be removed in favor of RocCurve.list . - Method Model.request_residuals_chart is deprecated and will be removed in favor of Residuals.compute . - Method Model.get_residuals_chart is deprecated and will be removed in favor of Residuals.get . - Method Model.get_all_residuals_charts is deprecated and will be removed in favor of Residuals.list . ### Documentation changes - Starting with this release, Python client documentation will be available at https://docs.datarobot.com/ as well as on ReadTheDocs. Content has been reorganized to support this change. - Removed numpydoc as a dependency. Docstring parsing has been handled by sphinx.ext.napoleon since 3.6.0. - Fix issues with how the Table of Contents is rendered on ReadTheDocs. sphinx-external-toc is now a dev dependency. - Fix minor issues with formatting across the ReadTheDocs site. - Updated docs on Anomaly Assessment objects to remove duplicate information. ### Experimental changes - Added use_case and deployment_id properties to RetrainingPolicy class. - Added create and update_use_case methods to RetrainingPolicy class. - Renamed the method ‘train_first_incremental_from_sample’ to ‘start_incremental_learning_from_sample’. Added new parameters : ‘early_stopping_rounds’ and ‘first_iteration_only’. - Added the credentials_id parameter to the create method in ChunkDefinition . - Bugfix the next_run_time property of the NotebookScheduledJob class to be nullable. - Added the highlight_whitespace property to the NotebookSettings . - Create new directory specifically for notebooks in the experimental portion of the client. - Added methods to the Notebook class to work with session: start_session() , stop_session() , get_session_status() , is_running() . - Added methods to the Notebook in order to execute and check related execution status: execute() , get_execution_status() , is_finished_executing() . - Added Notebook.create_revision to the Notebook class in order to create revisions. - Moved ModerationTemplate class to ModerationTemplate . - Moved ModerationConfiguration class to ModerationConfiguration to interact with LLM moderation configuration. - Updates to Notebook.run method in the Notebook class in order to encourage proper usage as well as add more descriptive TypedDict as annotation. - Added NotebookScheduledJob.get_most_recent_run to the NotebookScheduledJob class to aid in more idiomatic code when dealing with manual runs. - Updates to Notebook.run method in the Notebook class in order to support Codespace Notebook execution as well as multiple related new classes and methods to expand API coverage which is needed for the underlying execution. - Added ExecutionEnvironment.assign_environment to the ExecutionEnvironment class, which gives the ability to assign or update a notebook’s environment. - Removed deprecated experimental method Model.get_incremental_learning_metadata . - Removed deprecated experimental method Model.start_incremental_learning . ## 3.6.0 ### New features - Added OCRJobResource for running OCR jobs. - Added new Jina V2 embedding model in VectorDatabaseEmbeddingModel. - Added new Small MultiLingual Embedding Model in VectorDatabaseEmbeddingModel. - Added Deployment.get_segment_attributes to retrieve segment attributes. - Added Deployment.get_segment_values to retrieve segment values. - Added AutomatedDocument.list_all_available_document_types to return a list of document types. - Added Model.request_per_class_fairness_insights to return per-class bias & fairness insights. - Added MLOpsEvent to report MLOps Events. Currently supporting moderation MLOps events only - Added Deployment.get_moderation_events to retrieve moderation events for that deployment. - Extended the advanced options available when setting a target to include new parameter: ‘number_of_incremental_learning_iterations_before_best_model_selection’(part of the AdvancedOptions object). This parameter allows you to specify how long top 5 models will run for prior to best model selection. - Add support for ‘connector_type’ in Connector.create . - Deprecate file_path for Connector.create and Connector.update . - Added DataQualityExport and Deployment.list_data_quality_exports to retrieve a list of data quality records. - Added secure config support for Azure Service Principal credentials. - Added support for categorical custom metrics in CustomMetric . - Added NemoConfiguration to manage Nemo configurations. - Added NemoConfiguration.create to create or update a Nemo configuration. - Added NemoConfiguration.get to retrieve a Nemo configuration. - Added a new class ShapDistributions to interact with SHAP distribution insights. - Added the MODEL_COMPLIANCE_GEN_AI value to the attribute document_type from DocumentOption to generate compliance documentation for LLMs in the Registry. - Added new attribute prompts_count to Chat . - Added Recipe modules for Data Wrangling. - Added RecipeOperation and a set of subclasses to represent a single Recipe.operations operation. - Added new attribute similarity_score to Citation . - Added new attributes retriever and add_neighbor_chunks to VectorDatabaseSettings . - Added new attribute metadata to Citation . - Added new attribute metadata_filter to ChatPrompt . - Added new attribute metadata_filter to ComparisonPrompt . - Added new attribute custom_chunking to ChunkingParameters . - Added new attribute custom_chunking to VectorDatabase . - Added a new class LLMTestConfiguration for LLM test configurations. - LLMTestConfiguration.get to retrieve a hosted LLM test configuration. - LLMTestConfiguration.list to list hosted LLM test configurations. - LLMTestConfiguration.create to create an LLM test configuration. - LLMTestConfiguration.update to update an LLM test configuration. - LLMTestConfiguration.delete to delete an LLM test configuration. - Added a new class LLMTestConfigurationSupportedInsights for LLM test configuration supported insights. - LLMTestConfigurationSupportedInsights.list to list hosted LLM test configuration supported insights. - Added a new class LLMTestResult for LLM test results. - LLMTestResult.get to retrieve a hosted LLM test result. - LLMTestResult.list to list hosted LLM test results. - LLMTestResult.create to create an LLM test result. - LLMTestResult.delete to delete an LLM test result. - Added new attribute dataset_name to OOTBDatasetDict . - Added new attribute rows_count to OOTBDatasetDict . - Added new attribute max_num_prompts to DatasetEvaluationDict . - Added new attribute prompt_sampling_strategy to DatasetEvaluationDict . - Added a new class DatasetEvaluationRequestDict for Dataset Evaluations in create/edit requests. - Added new attribute evaluation_dataset_name to InsightEvaluationResult . - Added new attribute chat_name to InsightEvaluationResult . - Added new attribute llm_test_configuration_name to LLMTestResult . - Added new attribute creation_user_name to LLMTestResult . - Added new attribute pass_percentage to LLMTestResult . - Added new attribute evaluation_dataset_name to DatasetEvaluation . - Added new attribute datasets_compatibility to LLMTestConfigurationSupportedInsights . - Added a new class NonOOTBDataset for non out-of-the-box (OOTB) dataset entities. - NonOOTBDataset.list to retrieve non OOTB datasets for compliance testing. - Added a new class OOTBDataset for OOTB dataset entities. - OOTBDataset.list to retrieve OOTB datasets for compliance testing. - Added a new class TraceMetadata to retrieve trace metadata. - Add new attributes to VectorDatabase : parent_id , family_id , metadata_columns , added_dataset_ids , added_dataset_names , and`version`. - VectorDatabase.get_supported_retrieval_settings to retrieve supported retrieval settings. - VectorDatabase.submit_export_dataset_job to submit the vector database as dataset to the AI catalog. - Updated the method VectorDatabase.create to create a new vector database version. - Added a new class SupportedRetrievalSettings for supported vector database retrieval settings. - Added a new class SupportedRetrievalSetting for supported vector database retrieval setting. - Added a new class VectorDatabaseDatasetExportJob for vector database dataset export jobs. - Added new attribute playground_id to CostMetricConfiguration . - Added new attribute name to CostMetricConfiguration . - Added a new class SupportedInsights to support lists. - SupportedInsights.list to list supported insights. - Added a new class MetricInsights for the new metric insights routes. - MetricInsights.list to list metric insights. - MetricInsights.copy_to_playground to copy metrics to another playground. - Added a new class PlaygroundOOTBMetricConfiguration for OOTB metric configurations. - Updated the schema for EvaluationDatasetMetricAggregation to include the new attributes ootb_dataset_name , dataset_id and dataset_name . - Updated the method EvaluationDatasetMetricAggregation.list with additional optional filter parameters. - Added new attribute warning to OOTBDataset . - Added new attribute warning to OOTBDatasetDict . - Added new attribute warnings to LLMTestConfiguration . - Added a new parameter playground_id to SidecarModelMetricValidation.create to support sidecar model metrics transition to playground. - Updated the schema for NemoConfiguration to include the new attributes prompt_pipeline_template_id and response_pipeline_template_id . - Added new attributes to EvaluationDatasetConfiguration : rows_count , playground_id . - Fix retrieving shap_remaining_total when requesting predictions with SHAP insights. This should return the remaining shap values when present. ### API changes - Updated ServerError ’s exc_message to be constructed with a request ID to help with debugging. - Added method Deployment.get_capabilities to retrieve a list of Capability objects containing capability details. - Advanced options parameters: modelGroupId , modelRegimeId , and modelBaselines were renamed into seriesId , forecastDistance , and forecastOffsets . - Added the parameter use_sample_from_dataset from Project.create_from_dataset . This parameter, when set, uses the EDA sample of the dataset to start the project. - Added the parameter quick_compute to functions in the classes ShapMatrix , ShapImpact , and ShapPreview . - Added the parameter copy_insights to Playground.create to copy the insights from existing Playground to the new one. - Added the parameter llm_test_configuration_ids , LLMBlueprint.register_custom_model , to run LLM compliance tests when a blueprint is sent to the custom model workshop. ### Enhancements - Added standard pagination parameters (e.g. limit , offset ) to Deployment.list , allowing you to get deployment data in smaller chunks. - Added the parameter base_path to get_encoded_file_contents_from_paths and get_encoded_image_contents_from_paths , allowing you to better control script behavior when using relative file paths. ### Bugfixes - Fixed field in CustomTaskVersion for controlling network policies. This is changed from outgoing_network_policy to outbound_network_policy . When performing a GET action, this field was incorrect and always resolved to None . When attempting a POST or PATCH action, the incorrect field would result in a 422. Also changed the name of datarobot.enums.CustomTaskOutgoingNetworkPolicy to datarobot.enums.CustomTaskOutboundNetworkPolicy to reflect the proper field name. - Fixed schema for DataSliceSizeInfo , so it now allows an empty list for the messages field. ### Deprecation summary - Removed the parameter in_use from ImageAugmentationList.create . This parameter was deprecated in v3.1.0. - Deprecated AutomatedDocument.list_available_document_types . Please use AutomatedDocument.list_all_available_document_types instead. - Deprecated Model.request_fairness_insights . Please use Model.request_per_class_fairness_insights instead, to return StatusCheckJob instead of status_id . - Deprecated Model.get_prime_eligibility . Prime models are no longer supported. - eligibleForPrime field will no longer be returned from Model.get_supported_capabilities and will be removed after version 3.8 is released. - Deprecated the property ShapImpact.row_count and it will be removed after version 3.7 is released. - Advanced options parameters: modelGroupId , modelRegimeId , and modelBaselines were renamed into seriesId , forecastDistance , and forecastOffsets and are deprecated and they will be removed after version 3.6 is released. - Renamed datarobot.enums.CustomTaskOutgoingNetworkPolicy to datarobot.enums.CustomTaskOutboundNetworkPolicy to reflect bug fix changes. The original enum was unusable. - Removed parameter user_agent_suffix in datarobot.Client . Please use trace_context instead. - Removed deprecated method DataStore.get_access_list . Please use DataStore.get_shared_roles instead. - Removed support for SharingAccess instances in DataStore.update_access_list . Please use SharingRole instances instead. ### Configuration changes - Removed upper bound pin on urllib3 package to allow versions 2.0.2 and above. - Upgraded the Pillow library to version 10.3.0. Users installing DataRobot with the “images” extra ( pip install datarobot[images] ) should note that this is a required library. ### Documentation changes - The API Reference page has been split into multiple sections for better usability. - Fixed docs for Project.refresh to clarify that it does not return a value. - Fixed code example for ExternalScores . - Added copy button to code examples in ReadTheDocs documentation, for convenience. - Removed the outdated ‘examples’ section from the documentation. Please refer to DataRobot’s API Documentation Home for more examples. - Removed the duplicate ‘getting started’ section from the documentation. - Updated to Sphinx RTD Theme v3. - Updated the description for the parameter: ‘number_of_incremental_learning_iterations_before_best_model_selection’ (part of the AdvancedOptions object). ### Experimental changes - Added the force_update parameter to the update method in ChunkDefinition . - Removed attribute select_columns from ChunkDefinition - Added initial experimental support for Chunking Service V2 - DatasetDefinition - DatasetProps - DatasetInfo - DynamicDatasetProps - RowsChunkDefinition - FeaturesChunkDefinition - ChunkDefinitionStats - ChunkDefinition - Added new method update to ChunkDefinition - Added experimental support for time series wrangling, including usage template: - datarobot._experimental.models.time_series_wrangling_template.user_flow_template Experimental changes offer automated time series feature engineering for the data in Snowflake or Postgres. - Added the ability to use the Spark dialect when creating a recipe, allowing data wrangling support for files. - Added new attribute warning to Chat . - Moved all modules from datarobot._experimental.models.genai to datarobot.models.genai . - Added a new method ‘Model.train_first_incremental_from_sample’ that will train first incremental learning iteration from existing sample model. Requires “Project Creation from a Dataset Sample” feature flag. ## 3.5.0 ### New features - Added support for BYO LLMs using serverless predictions in CustomModelLLMValidation . - Added attribute creation_user_name to LLMBlueprint . - Added a new class HostedCustomMetricTemplate for hosted custom metrics templates. - HostedCustomMetricTemplate.get to retrieve a hosted custom metric template. - HostedCustomMetricTemplate.list to list hosted custom metric templates. - Added Job.create_from_custom_metric_gallery_template to create a job from a custom metric gallery template. - Added a new class HostedCustomMetricTemplate for hosted custom metrics. - HostedCustomMetric.list to list hosted custom metrics. - HostedCustomMetric.update to update a hosted custom metrics. - HostedCustomMetric.delete to delete a hosted custom metric. - HostedCustomMetric.create_from_custom_job to create a hosted custom metric from existing custom job. - HostedCustomMetric.create_from_template to create hosted custom metric from template. - Added a new class datarobot.models.deployment.custom_metrics.HostedCustomMetricBlueprint for hosted custom metric blueprints. - HostedCustomMetricBlueprint.get to get a hosted custom metric blueprint. - HostedCustomMetricBlueprint.create to create a hosted custom metric blueprint. - HostedCustomMetricBlueprint.update to update a hosted custom metric blueprint. - Added Job.list_schedules to list job schedules. - Added a new class JobSchedule for the registry job schedule. - JobSchedule.create to create a job schedule. - JobSchedule.update to update a job schedule. - JobSchedule.delete to delete a job schedule. - Added attribute credential_type to RuntimeParameter . - Added a new class EvaluationDatasetConfiguration for configuration of evaluation datasets. - EvaluationDatasetConfiguration.get to get an evaluation dataset configuration. - EvaluationDatasetConfiguration.list to list the evaluation dataset configurations for a Use Case. - EvaluationDatasetConfiguration.create to create an evaluation dataset configuration. - EvaluationDatasetConfiguration.update to update an evaluation dataset configuration. - EvaluationDatasetConfiguration.delete to delete an evaluation dataset configuration. - Added a new class EvaluationDatasetMetricAggregation for metric aggregation results. - EvaluationDatasetMetricAggregation.list to get the metric aggregation results. - EvaluationDatasetMetricAggregation.create to create the metric aggregation job. - EvaluationDatasetMetricAggregation.delete to delete metric aggregation results. - Added a new class SyntheticEvaluationDataset for synthetic dataset generation. Use SyntheticEvaluationDataset.create to create a synthetic evaluation dataset. - Added a new class SidecarModelMetricValidation for sidecar model metric validations. - SidecarModelMetricValidation.create to create a sidecar model metric validation. - SidecarModelMetricValidation.list to list sidecar model metric validations. - SidecarModelMetricValidation.get to get a sidecar model metric validation. - SidecarModelMetricValidation.revalidate to rerun a sidecar model metric validation. - SidecarModelMetricValidation.update to update a sidecar model metric validation. - SidecarModelMetricValidation.delete to delete a sidecar model metric validation. - Added experimental support for Chunking Service: - Added a new attribute, is_descending_order to: ### Bugfixes - Updated the trafaret column prediction from TrainingPredictionsIterator for supporting extra list of strings. ### Configuration changes - Updated black version to 23.1.0. - Removes dependency on package mock , since it is part of the standard library. ### Documentation changes - Removed incorrect can_share parameters in Use Case sharing example - Added usage of external_llm_context_size in llm_settings in genai_example.rst . - Updated doc string for llm_settings to include attribute external_llm_context_size for external LLMs. - Updated genai_example.rst to link to DataRobot doc pages for external vector database and external LLM deployment creation. ### API changes - Remove ImportedModel object since it was API for SSE (standalone scoring engine) which is not part of DataRobot anymore. - Added number_of_clusters parameter to Project.get_model_records to filter models by number of clusters in unsupervised clustering projects. - Remove an unsupported NETWORK_EGRESS_POLICY.DR_API_ACCESS value for custom models. This value was used by a feature that was never released as a GA and is not supported in the current API. - Implemented support for dr-connector-v1 to DataStore and DataSource . - Added a new parameter name to DataStore.list for searching data stores by name. - Added a new parameter entity_type to the compute and create methods of the classes ShapMatrix , ShapImpact , ShapPreview . Insights can be computed for custom models if the parameter entity_type="customModel" is passed. See also the User Guide: :ref: SHAP insights overview . ### Experimental changes - Added experimental api support for Data Wrangling. See Recipe . - Recipe.from_data_store to create a Recipe from data store. - Recipe.retrieve_preview to get a sample of the data after recipe is applied. - Recipe.set_inputs to set inputs to the recipe. - Recipe.set_operations to set operations to the recipe. - Added new experimental DataStore that adds get_spark_session for Databricks databricks-v1 data stores to get a Spark session. - Added attribute chunking_type to DatasetChunkDefinition . - Added OTV attributes to DatasourceDefinition . - Added DatasetChunkDefinition.patch_validation_dates to patch validation dates of OTV datasource definitions after sampling job. ## 3.4.1 ### New features ### Enhancements ### Bugfixes - Updated the validation logic of RelationshipsConfiguration to work with native database connections ### API changes ### Deprecation summary ### Configuration changes ### Documentation changes ### Experimental changes ## 3.4.0 ### New features - Added the following classes for generative AI. Importing these from datarobot._experimental.models.genai is deprecated and will be removed by the release of DataRobot 10.1 and API Client 3.5. - Playground to manage generative AI playgrounds. - LLMDefinition to get information about supported LLMs. - LLMBlueprint to manage LLM blueprints. - Chat to manage chats for LLM blueprints. - ChatPrompt to submit prompts within a chat. - ComparisonChat to manage comparison chats across multiple LLM blueprints within a playground. - ComparisonPrompt to submit a prompt to multiple LLM blueprints within a comparison chat. - VectorDatabase to create vector databases from datasets in the AI Catalog for retrieval augmented generation with an LLM blueprint. - CustomModelVectorDatabaseValidation to validate a deployment for use as a vector database. - CustomModelLLMValidation to validate a deployment for use as an LLM. - UserLimits to get counts of vector databases and LLM requests for a user. - Extended the advanced options available when setting a target to include new parameter: incrementalLearningEarlyStoppingRounds (part of the AdvancedOptions object). This parameter allows you to specify when to stop for incremental learning automation. - Added experimental support for Chunking Service: - DatasetChunkDefinition for defining how chunks are created from a data source. - DatasetChunkDefinition.create to create a new dataset chunk definition. - DatasetChunkDefinition.get to get a specific dataset chunk definition. - DatasetChunkDefinition.list to list all dataset chunk definitions. - DatasetChunkDefinition.get_datasource_definition to retrieve the data source definition. - DatasetChunkDefinition.get_chunk to get specific chunk metadata belonging to a dataset chunk definition. - DatasetChunkDefinition.list_chunks to list all chunk metadata belonging to a dataset chunk definition. - DatasetChunkDefinition.create_chunk to submit a job to retrieve the data from the origin data source. - DatasetChunkDefinition.create_chunk_by_index to submit a job to retrieve data from the origin data source by index. - OriginStorageType - Chunk - ChunkStorageType - ChunkStorage - DatasourceDefinition - DatasourceAICatalogInfo to define the datasource AI catalog information to create a new dataset chunk definition. - DatasourceDataWarehouseInfo to define the datasource data warehouse (snowflake, big query, etc) information to create a new dataset chunk definition. - RuntimeParameter for retrieving runtime parameters assigned to CustomModelVersion . - RuntimeParameterValue to define runtime parameter override value, to be assigned to CustomModelVersion . - Added Snowflake Key Pair authentication for uploading datasets from Snowflake or creating a project from Snowflake data - Added Project.get_model_records to retrieve models. Method Project.get_models is deprecated and will be removed soon in favor of Project.get_model_records . - Extended the advanced options available when setting a target to include new parameter: chunkDefinitionId (part of the AdvancedOptions object). This parameter allows you to specify the chunking definition needed for incremental learning automation. - Extended the advanced options available when setting a target to include new Autopilot parameters: incrementalLearningOnlyMode and incrementalLearningOnBestModel (part of the AdvancedOptions object). These parameters allow you to specify how Autopilot is performed with the chunking service. - Added a new method DatetimeModel.request_lift_chart to support Lift Chart calculations for datetime partitioned projects with support of Sliced Insights. - Added a new method DatetimeModel.get_lift_chart to support Lift chart retrieval for datetime partitioned projects with support of Sliced Insights. - Added a new method DatetimeModel.request_roc_curve to support ROC curve calculation for datetime partitioned projects with support of Sliced Insights. - Added a new method DatetimeModel.get_roc_curve to support ROC curve retrieval for datetime partitioned projects with support of Sliced Insights. - Update method DatetimeModel.request_feature_impact to support use of Sliced Insights. - Update method DatetimeModel.get_feature_impact to support use of Sliced Insights. - Update method DatetimeModel.get_or_request_feature_impact to support use of Sliced Insights. - Update method DatetimeModel.request_feature_effect to support use of Sliced Insights. - Update method DatetimeModel.get_feature_effect to support use of Sliced Insights. - Update method DatetimeModel.get_or_request_feature_effect to support use of Sliced Insights. - Added a new method FeatureAssociationMatrix.create to support the creation of FeatureAssociationMatricies for Featurelists. - Introduced a new method Deployment.perform_model_replace as a replacement for Deployment.replace_model . - Introduced a new property, model_package , which provides an overview of the currently used model package in datarobot.models.Deployment . - Added new parameter prediction_threshold to BatchPredictionJob.score_with_leaderboard_model and BatchPredictionJob.score that automatically assigns the positive class label to any prediction exceeding the threshold. - Added two new enum values to datarobot.models.data_slice.DataSlicesOperators , “BETWEEN” and “NOT_BETWEEN”, which are used to allow slicing. - Added a new class Challenger for interacting with DataRobot challengers to support the following methods: Challenger.get to retrieve challenger objects by ID. Challenger.list to list all challengers. Challenger.create to create a new challenger. Challenger.update to update a challenger. Challenger.delete to delete a challenger. - Added a new method Deployment.get_challenger_replay_settings to retrieve the challenger replay settings of a deployment. - Added a new method Deployment.list_challengers to retrieve the challengers of a deployment. - Added a new method Deployment.get_champion_model_package to retrieve the champion model package from a deployment. - Added a new method Deployment.list_prediction_data_exports to retrieve deployment prediction data exports. - Added a new method Deployment.list_actuals_data_exports to retrieve deployment actuals data exports. - Added a new method Deployment.list_training_data_exports to retrieve deployment training data exports. - Manage deployment health settings with the following methods: - Get health settings Deployment.get_health_settings - Update health settings Deployment.update_health_settings - Get default health settings Deployment.get_default_health_settings - Added new enum value to datarobot.enums._SHARED_TARGET_TYPE to support Text Generation use case. - Added new enum value datarobotServerless to datarobot.enums.PredictionEnvironmentPlatform to support DataRobot Serverless prediction environments. - Added new enum value notApplicable to datarobot.enums.PredictionEnvironmentHealthType to support new health status from DataRobot API. - Added new enum value to datarobot.enums.TARGET_TYPE and datarobot.enums.CUSTOM_MODEL_TARGET_TYPE to support text generation custom inference models. - Updated datarobot.CustomModel to support the creation of text generation custom models. - Added a new class CustomMetric for interacting with DataRobot custom metrics to support the following methods: - CustomMetric.get to retrieve a custom metric object by ID from a given deployment. - CustomMetric.list to list all custom metrics from a given deployment. - CustomMetric.create to create a new custom metric for a given deployment. - CustomMetric.update to update a custom metric for a given deployment. - CustomMetric.delete to delete a custom metric for a given deployment. - CustomMetric.unset_baseline to remove baseline for a given custom metric. - CustomMetric.submit_values to submit aggregated custom metrics values from code. The provided data should be in the form of a dict or a Pandas DataFrame. - CustomMetric.submit_single_value to submit a single custom metric value. - CustomMetric.submit_values_from_catalog to submit aggregated custom metrics values from a dataset via the AI Catalog. - CustomMetric.get_values_over_time to retrieve values of a custom metric over a time period. - CustomMetric.get_summary to retrieve the summary of a custom metric over a time period. - CustomMetric.get_values_over_batch to retrieve values of a custom metric over batches. - CustomMetric.get_batch_summary to retrieve the summary of a custom metric over batches. - Added CustomMetricValuesOverTime to retrieve custom metric over time information. - Added CustomMetricSummary to retrieve custom metric over time summary. - Added CustomMetricValuesOverBatch to retrieve custom metric over batch information. - Added CustomMetricBatchSummary to retrieve custom metric batch summary. - Added Job and JobRun to create, read, update, run, and delete jobs in the Registry. - Added KeyValue to create, read, update, and delete key values. - Added a new class PredictionDataExport for interacting with DataRobot deployment data export to support the following methods: - PredictionDataExport.get to retrieve a prediction data export object by ID from a given deployment. - PredictionDataExport.list to list all prediction data exports from a given deployment. - PredictionDataExport.create to create a new prediction data export for a given deployment. - PredictionDataExport.fetch_data to retrieve a prediction export data as a DataRobot dataset. - Added a new class ActualsDataExport for interacting with DataRobot deployment data export to support the following methods: - ActualsDataExport.get to retrieve an actuals data export object by ID from a given deployment. - ActualsDataExport.list to list all actuals data exports from a given deployment. - ActualsDataExport.create to create a new actuals data export for a given deployment. - ActualsDataExport.fetch_data to retrieve an actuals export data as a DataRobot dataset. - Added a new class TrainingDataExport for interacting with DataRobot deployment data export to support the following methods: - TrainingDataExport.get to retrieve a training data export object by ID from a given deployment. - TrainingDataExport.list to list all training data exports from a given deployment. - TrainingDataExport.create to create a new training data export for a given deployment. - TrainingDataExport.fetch_data to retrieve a training export data as a DataRobot dataset. - Added a new parameter base_environment_version_id to CustomModelVersion.create_clean for overriding the default environment version selection behavior. - Added a new parameter base_environment_version_id to CustomModelVersion.create_from_previous for overriding the default environment version selection behavior. - Added a new class PromptTrace for interacting with DataRobot prompt trace to support the following methods: - PromptTrace.list to list all prompt traces from a given playground. - PromptTrace.export_to_ai_catalog to export prompt traces for the playground to AI catalog. - Added a new class InsightsConfiguration for describing available insights and configured insights for a playground. InsightsConfiguration.list to list the insights that are available to be configured. - Added a new class Insights for configuring insights for a playground. Insights.get to get the current insights configuration for a playground. Insights.create to create or update the insights configuration for a playground. - Added a new class CostMetricConfiguration for describing available cost metrics and configured cost metrics for a Use Case. CostMetricConfiguration.get to get the cost metric configuration. CostMetricConfiguration.create to create a cost metric configuration. CostMetricConfiguration.update to update the cost metric configuration. CostMetricConfiguration.delete to delete the cost metric configuration.Key - Added a new class LLMCostConfiguration for the cost configuration of a specific llm within a Use Case. - Added new classes ShapMatrix , ShapImpact , ShapPreview to interact with SHAP-based insights. See also the User Guide: :ref: SHAP insights overview ### API changes - Parameter Overrides: Users can now override most of the previously set configuration values directly through parameters when initializing the Client. Exceptions: The endpoint and token values must be initialized from one source (client params, environment, or config file) and cannot be overridden individually, for security and consistency reasons. The new configuration priority is as follows: - Client Params - Client config_path param - Environment Variables - Default to reading YAML config file from ~/.config/datarobot/drconfig.yaml - DATAROBOT_API_CONSUMER_TRACKING_ENABLED now always defaults to True . - Added Databricks personal access token and service principal (also shared credentials via secure config) authentication for uploading datasets from Databricks or creating a project from Databricks data. - Added secure config support for AWS long term credentials. - Implemented support for dr-database-v1 to DataStore , DataSource , and DataDriver . Added enum classes to support the changes. - You can retrieve the canonical URI for a Use Case using UseCase.get_uri . - You can open a Use Case in a browser using UseCase.open_in_browser . ### Enhancements - Added a new parameter to Dataset.create_from_url to support fast dataset registration: - sample_size - Added a new parameter to Dataset.create_from_data_source to support fast dataset registration: - sample_size - Job.get_result_when_complete returns datarobot.models.DatetimeModel instead of the datarobot.models.Model if a datetime model was trained. - Dataset.get_as_dataframe can handle downloading parquet files as well as csv files. - Implement support for dr-database-v1 in DataStore - Added two new parameters to BatchPredictionJobDefinition.list for paginating long job definitions lists: - offset - limit - Added two new parameters to BatchPredictionJobDefinition.list for filtering the job definitions: - deployment_id - search_name - Added new parameter to Deployment.validate_replacement_model to support replacement validation based on model package ID: - new_registered_model_version_id - Added support for Native Connectors to Connector for everything other than Connector.create and Connector.update ### Deprecation summary - Removed Model.get_leaderboard_ui_permalink and Model.open_model_browser - Deprecated Project.get_models in favor of Project.get_model_records . - BatchPredictionJobDefinition.list will no longer return all job definitions after version 3.6 is released. To preserve current behavior please pass limit=0. - new_model_id parameter in Deployment.validate_replacement_model will be removed after version 3.6 is released. - Deployment.replace_model will be removed after version 3.6 is released. Method Deployment.perform_model_replace should be used instead. - CustomInferenceModel.assign_training_data was marked as deprecated in v3.2. The deprecation period has been extended, and the feature will now be removed in v3.5. Use CustomModelVersion.create_clean and CustomModelVersion.create_from_previous instead. ### Documentation changes - Updated genai_example.rst to utilize latest genAI features and methods introduced most recently in the API client. ### Experimental changes - Added new attribute, prediction_timeout to CustomModelValidation . - Added new attributes, feedback_result , metrics , and final_prompt to ResultMetadata . - Added use_case_id to CustomModelValidation . - Added llm_blueprints_count and user_name to Playground . - Added custom_model_embedding_validations to SupportedEmbeddings . - Added embedding_validation_id and is_separator_regex to VectorDatabase . - Added optional parameters, use_case , name , and model to CustomModelValidation.create . - Added a method CustomModelValidation.list , to list custom model validations available to a user with several optional parameters to filter the results. - Added a method CustomModelValidation.update , to update a custom model validation. - Added an optional parameter, use_case , to LLMDefinition.list , to include in the returned LLMs the external LLMs available for the specified use_case as well. - Added optional parameter, playground to VectorDatabase.list to list vector databases by playground. - Added optional parameter, comparison_chat , to ComparisonPrompt.list , to list comparison prompts by comparison chat. - Added optional parameter, comparison_chat , to ComparisonPrompt.create , to specify the comparison chat to create the comparison prompt in. - Added optional parameter, feedback_result , to ComparisonPrompt.update , to update a comparison prompt with feedback. - Added optional parameters, is_starred to LLMBlueprint.update to update the LLM blueprint’s starred status. - Added optional parameters, is_starred to LLMBlueprint.list to filter the returned LLM blueprints to those matching is_starred . - Added a new enum PromptType, PromptType to identify the LLMBlueprint’s prompting type. - Added optional parameters, prompt_type to LLMBlueprint.create , to specify the LLM blueprint’s prompting type. This can be set with PromptType . - Added optional parameters, prompt_type to LLMBlueprint.update , to specify the updated LLM blueprint’s prompting type. This can be set with PromptType . - Added a new class, ComparisonChat , for interacting with DataRobot generative AI comparison chats. ComparisonChat.get retrieves a comparison chat object by ID. ComparisonChat.list lists all comparison chats available to the user. ComparisonChat.create creates a new comparison chat. ComparisonChat.update updates the name of a comparison chat. ComparisonChat.delete deletes a single comparison chat. - Added optional parameters, playground and chat to ChatPrompt.list , to list chat prompts by playground and chat. - Added optional parameter, chat to ChatPrompt.create , to specify the chat to create the chat prompt in. - Added a new method, ChatPrompt.update , to update a chat prompt with custom metrics and feedback. - Added a new class, Chat , for interacting with DataRobot generative AI chats. Chat.get retrieves a chat object by ID. Chat.list lists all chats available to the user. Chat.create creates a new chat. Chat.update updates the name of a chat. Chat.delete deletes a single chat. - Removed the model_package module. Use RegisteredModelVersion instead. - Added new class UserLimits - Added support to get the count of users’ LLM API requests. UserLimits.get_llm_requests_count - Added support to get the count of users’ vector databases. UserLimits.get_vector_database_count - Added new methods to the class Notebook which includes Notebook.run and Notebook.download_revision . See the documentation for example usage. - Added new class NotebookScheduledJob . - Added new class NotebookScheduledRun . - Added a new method Model.get_incremental_learning_metadata that retrieves incremental learning metadata for a model. - Added a new method Model.start_incremental_learning that starts incremental learning for a model. - Updated the API endpoint prefix for all GenerativeAI routes to align with the publicly documented routes. ### Bugfixes - Fixed how async url is build in Model.get_or_request_feature_impact - Fixed setting ssl_verify by env variables. - Resolved a problem related to tilde-based paths in the Client’s ‘config_path’ attribute. - Changed the force_size default of ImageOptions to apply the same transformations by default, which are applied when image archive datasets are uploaded to DataRobot. ## 3.3.0 ### New features - Added support for Python 3.11. - Added new library strenum to add StrEnum support while maintaining backwards compatibility with Python 3.7-3.10. DataRobot does not use the native StrEnum class in Python 3.11. - Added a new class PredictionEnvironment for interacting with DataRobot Prediction environments. - Extended the advanced options available when setting a target to include new parameters: modelGroupId , modelRegimeId , and modelBaselines (part of the AdvancedOptions object). These parameters allow you to specify the user columns required to run time series models without feature derivation in OTV projects. - Added a new method PredictionExplanations.create_on_training_data , for computing prediction explanation on training data. - Added a new class RegisteredModel for interacting with DataRobot registered models to support the following methods: - RegisteredModel.get to retrieve RegisteredModel object by ID. - RegisteredModel.list to list all registered models. - RegisteredModel.archive to permanently archive registered model. - RegisteredModel.update to update registered model. - RegisteredModel.get_shared_roles to retrieve access control information for registered model. - RegisteredModel.share to share a registered model. - RegisteredModel.get_version to retrieve RegisteredModelVersion object by ID. - RegisteredModel.list_versions to list registered model versions. - RegisteredModel.list_associated_deployments to list deployments associated with a registered model. - Added a new class RegisteredModelVersion for interacting with DataRobot registered model versions (also known as model packages) to support the following methods: - RegisteredModelVersion.create_for_external to create a new registered model version from an external model. - RegisteredModelVersion.list_associated_deployments to list deployments associated with a registered model version. - RegisteredModelVersion.create_for_leaderboard_item to create a new registered model version from a Leaderboard model. - RegisteredModelVersion.create_for_custom_model_version to create a new registered model version from a custom model version. - Added a new method Deployment.create_from_registered_model_version to support creating deployments from registered model version. - Added a new method Deployment.download_model_package_file to support downloading model package files (.mlpkg) of the currently deployed model. - Added support for retrieving document thumbnails: - DocumentThumbnail - DocumentPageFile - Added support to retrieve document text extraction samples using: - DocumentTextExtractionSample - DocumentTextExtractionSamplePage - DocumentTextExtractionSampleDocument - Added new fields to CustomTaskVersion for controlling network policies. The new fields were also added to the response. This can be set with datarobot.enums.CustomTaskOutgoingNetworkPolicy . - Added a new method BatchPredictionJob.score_with_leaderboard_model to run batch predictions using a Leaderboard model instead of a deployment. - Set IntakeSettings and OutputSettings to use IntakeAdapters and OutputAdapters enum values respectively for the property type . - Added method Deployment.get_predictions_vs_actuals_over_time to retrieve a deployment’s predictions vs actuals over time data. ### Bugfixes - Payload property subset renamed to source in Model.request_feature_effect - Fixed an issue where Context.trace_context was not being set from environment variables or DR config files. - Project.refresh no longer sets Project.advanced_options to a dictionary. - Fixed Dataset.modify to clarify behavior of when to preserve or clear categories. - Fixed an issue with enums in f-strings resulting in the enum class and property being printed instead of the enum property’s value in Python 3.11 environments. ### Deprecation summary - Project.refresh will no longer set Project.advanced_options to a dictionary after version 3.5 is released. : All interactions with Project.advanced_options should be expected to be through the AdvancedOptions class. ### Experimental changes - Added a new class, VectorDatabase , for interacting with DataRobot vector databases. - VectorDatabase.get retrieves a VectorDatabase object by ID. - VectorDatabase.list lists all VectorDatabases available to the user. - VectorDatabase.create creates a new VectorDatabase. - VectorDatabase.create allows you to use a validated deployment of a custom model as your own Vector Database. - VectorDatabase.update updates the name of a VectorDatabase. - VectorDatabase.delete deletes a single VectorDatabase. - VectorDatabase.get_supported_embeddings retrieves all supported embedding models. - VectorDatabase.get_supported_text_chunkings retrieves all supported text chunking configurations. - VectorDatabase.download_text_and_embeddings_asset download a parquet file with internal vector database data. - Added a new class, CustomModelVectorDatabaseValidation , for validating custom model deployments for use as a vector database. - CustomModelVectorDatabaseValidation.get retrieves a CustomModelVectorDatabaseValidation object by ID. - CustomModelVectorDatabaseValidation.get_by_values retrieves a CustomModelVectorDatabaseValidation object by field values. - CustomModelVectorDatabaseValidation.create starts validation of the deployment. - CustomModelVectorDatabaseValidation.revalidate repairs an unlinked external vector database. - Added a new class, Playground , for interacting with DataRobot generative AI playgrounds. - Playground.get retrieves a playground object by ID. - Playground.list lists all playgrounds available to the user. - Playground.create creates a new playground. - Playground.update updates the name and description of a playground. - Playground.delete deletes a single playground. - Added a new class, LLMDefinition , for interacting with DataRobot generative AI LLMs. - LLMDefinition.list lists all LLMs available to the user. - Added a new class, LLMBlueprint , for interacting with DataRobot generative AI LLM blueprints. - LLMBlueprint.get retrieves an LLM blueprint object by ID. - LLMBlueprint.list lists all LLM blueprints available to the user. - LLMBlueprint.create creates a new LLM blueprint. - LLMBlueprint.create_from_llm_blueprint creates a new LLM blueprint from an existing one. - LLMBlueprint.update updates an LLM blueprint. - LLMBlueprint.delete deletes a single LLM blueprint. - Added a new class, ChatPrompt , for interacting with DataRobot generative AI chat prompts. - ChatPrompt.get retrieves a chat prompt object by ID. - ChatPrompt.list lists all chat prompts available to the user. - ChatPrompt.create creates a new chat prompt. - ChatPrompt.delete deletes a single chat prompt. - Added a new class, CustomModelLLMValidation , for validating custom model deployments for use as a custom model LLM. - CustomModelLLMValidation.get retrieves a CustomModelLLMValidation object by ID. - CustomModelLLMValidation.get_by_values retrieves a CustomModelLLMValidation object by field values. - CustomModelLLMValidation.create starts validation of the deployment. - CustomModelLLMValidation.revalidate repairs an unlinked external custom model LLM. - Added a new class, ComparisonPrompt , for interacting with DataRobot generative AI comparison prompts. - ComparisonPrompt.get retrieves a comparison prompt object by ID. - ComparisonPrompt.list lists all comparison prompts available to the user. - ComparisonPrompt.create creates a new comparison prompt. - ComparisonPrompt.update updates a comparison prompt. - ComparisonPrompt.delete deletes a single comparison prompt. - Extended UseCase , adding two new fields to represent the count of vector databases and playgrounds. - Added a new method, ChatPrompt.create_llm_blueprint , to create an LLM blueprint from a chat prompt. - Added a new method, CustomModelLLMValidation.delete , to delete a custom model LLM validation record. - Added a new method, LLMBlueprint.register_custom_model , for registering a custom model from a generative AI LLM blueprint. ## 3.2.0 ### New features - Added new methods to trigger batch monitoring jobs without providing a job definition. - BatchMonitoringJob.run - BatchMonitoringJob.get_status - BatchMonitoringJob.cancel - BatchMonitoringJob.download - Added Deployment.submit_actuals_from_catalog_async to submit actuals from the AI Catalog. - Added a new class StatusCheckJob which represents a job for a status check of submitted async jobs. - Added a new class JobStatusResult represents the result for a status check job of a submitted async task. - Added DatetimePartitioning.datetime_partitioning_log_retrieve to download the datetime partitioning log. - Added method DatetimePartitioning.datetime_partitioning_log_list to list the datetime partitioning log. - Added DatetimePartitioning.get_input_data to retrieve the input data used to create an optimized datetime partitioning. - Added DatetimePartitioningId , which can be passed as a partitioning_method to Project.analyze_and_model . - Added the ability to share deployments. See :ref: deployment sharing for more information on sharing deployments. - Added new methods get_bias_and_fairness_settings and update_bias_and_fairness_settings to retrieve or update bias and fairness settings. - Deployment.get_bias_and_fairness_settings - Deployment.update_bias_and_fairness_settings - Added a new class UseCase for interacting with the DataRobot Use Cases API. - Added a new class Application for retrieving DataRobot Applications available to the user. - Added a new class SharingRole to hold user or organization access rights. - Added a new class BatchMonitoringJob for interacting with batch monitoring jobs. - Added a new class BatchMonitoringJobDefinition for interacting with batch monitoring jobs definitions. - Added a new methods for handling monitoring job definitions: list, get, create, update, delete, run_on_schedule and run_once - BatchMonitoringJobDefinition.list - BatchMonitoringJobDefinition.get - BatchMonitoringJobDefinition.create - BatchMonitoringJobDefinition.update - BatchMonitoringJobDefinition.delete - BatchMonitoringJobDefinition.run_on_schedule - BatchMonitoringJobDefinition.run_once - Added a new method to retrieve a monitoring job - BatchMonitoringJob.get - Added the ability to filter return objects by a Use Case ID passed to the following methods: - Dataset.list - Project.list - Added the ability to automatically add a newly created dataset or project to a Use Case by passing a UseCase, list of UseCase objects, UseCase ID or list of UseCase IDs using the keyword argument use_cases to the following methods: - Dataset.create_from_file - Dataset.create_from_in_memory_data - Dataset.create_from_url - Dataset.create_from_data_source - Dataset.create_from_query_generator - Dataset.create_project - Project.create - Project.create_from_data_source - Project.create_from_dataset - Project.create_segmented_project_from_clustering_model - Project.start - Added the ability to set a default UseCase for requests. It can be set in several ways. - If the user configures the client via Client(...) , then invoke Client(..., default_use_case = ) . - If the user configures the client via dr.config.yaml, then add the property default_use_case: . - If the user configures the client via env vars, then set the env var DATAROBOT_DEFAULT_USE_CASE . - The default use case can also be set programmatically as a context manager via with UseCase.get(): . - Added the ability to configure the collection of client usage metrics to send to DataRobot. Note that this feature only tracks which DataRobot package methods are called and does not collect any user data. You can configure collection with the following settings: - If the user configures the client via Client(...) , then invoke Client(..., enable_api_consumer_tracking = ) . - If the user configures the client via dr.config.yaml, then add the property enable_api_consumer_tracking: . - If the user configures the client via env vars, then set the env var DATAROBOT_API_CONSUMER_TRACKING_ENABLED . Currently the default value for `enable_api_consumer_tracking` is `True`. - Added method [Deployment.get_predictions_over_time](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_predictions_over_time) to retrieve deployment predictions over time data. - Added a new class [FairnessScoresOverTime](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.bias_and_fairness.FairnessScoresOverTime) to retrieve fairness over time information. - Added a new method [Deployment.get_fairness_scores_over_time](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_fairness_scores_over_time) to retrieve fairness scores over time of a deployment. - Added a new `use_gpu` parameter to the method [Project.analyze_and_model](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project.analyze_and_model) to set whether the project should allow usage of GPU - Added a new `use_gpu` parameter to the class [Project](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project) with information whether project allows usage of GPU - Added a new class [TrainingData](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.models.custom_model_version.TrainingData) for retrieving TrainingData assigned to [CustomModelVersion](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.CustomModelVersion). - Added a new class [HoldoutData](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.models.custom_model_version.HoldoutData) for retrieving HoldoutData assigned to [CustomModelVersion](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.CustomModelVersion). - Added the ability to retrieve the model and blueprint json using the following methods: - [Model.get_model_blueprint_json](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_model_blueprint_json) - [Blueprint.get_json](https://docs.datarobot.com/en/docs/api/reference/public-api/blueprints.html#datarobot.models.Blueprint.get_json) - Added [Credential.update](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#datarobot.models.Credential.update) which allows you to update existing credential resources. - Added a new optional parameter `trace_context` to `datarobot.Client` to provide additional information on the DataRobot code being run. This parameter defaults to `None`. - Updated methods in [Model](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model) to support use of Sliced Insights: - [Model.get_feature_effect](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_feature_effect) - [Model.request_feature_effect](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_feature_effect) - [Model.get_or_request_feature_effect](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_or_request_feature_effect) - [Model.get_lift_chart](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_lift_chart) - [Model.get_all_lift_charts](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_all_lift_charts) - [Model.get_residuals_chart](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_residuals_chart) - [Model.get_all_residuals_charts](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_all_residuals_charts) - [Model.request_lift_chart](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_lift_chart) - [Model.request_residuals_chart](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_residuals_chart) - [Model.get_roc_curve](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_roc_curve) - [Model.get_feature_impact](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_feature_impact) - [Model.request_feature_impact](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_feature_impact) - [Model.get_or_request_feature_impact](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_or_request_feature_impact) - Added support for [SharingRole](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.sharing.SharingRole) to the following methods: - [DataStore.share](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore.share) - Added new methods for retrieving [SharingRole](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.sharing.SharingRole) information for the following classes: - [DataStore.get_shared_roles](https://docs.datarobot.com/en/docs/api/reference/sdk/data-connectivity.html#datarobot.DataStore.get_shared_roles) - Added new method for calculating sliced roc curve [Model.request_roc_curve](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_roc_curve) - Added new [DataSlice](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.data_slice.DataSlice) to support the following slices methods: - [DataSlice.list](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.data_slice.DataSlice.list) to retrieve all data slices in a project. - [DataSlice.create](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.data_slice.DataSlice.create) to create a new data slice. - [DataSlice.delete](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.data_slice.DataSlice.delete) to delete the data slice calling this method. - [DataSlice.request_size](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.data_slice.DataSlice.request_size) to submit a request to calculate a data slice size on a source. - [DataSlice.get_size_info](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.data_slice.DataSlice.get_size_info) to get the data slice’s info when applied to a source. - [DataSlice.get](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.data_slice.DataSlice.get) to retrieve a specific data slice. - Added new [DataSliceSizeInfo](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.data_slice.DataSliceSizeInfo) to define the result of a data slice applied to a source. - Added new method for retrieving all available feature impacts for the model [Model.get_all_feature_impacts](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_all_feature_impacts). - Added new method for StatusCheckJob to wait and return the completed object once it is generated [datarobot.models.StatusCheckJob.get_result_when_complete()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.StatusCheckJob.get_result_when_complete) ### Enhancements - Improve error message of SampleImage.list to clarify that a selected parameter cannot be used when a project has not proceeded to the correct stage prior to calling this method. - Extended SampleImage.list by two parameters to filter for a target value range in regression projects. - Added text explanations data to PredictionExplanations and made sure it is returned in both datarobot.PredictionExplanations.get_all_as_dataframe() and datarobot.PredictionExplanations.get_rows() method. - Added two new parameters to Project.upload_dataset_from_catalog : - credential_id - credential_data - Implemented training and holdout data assignment for Custom Model Version creation APIs: - CustomModelVersion.create_clean - CustomModelVersion.create_from_previous - The parameters added to both APIs are: - Extended CustomInferenceModel.create and CustomInferenceModel.update with the parameter is_training_data_for_versions_permanently_enabled . - Added value DR_API_ACCESS to the NETWORK_EGRESS_POLICY enum. - Added new parameter low_memory to Dataset.get_as_dataframe to allow a low memory mode for larger datasets - Added two new parameters to Project.list for paginating long project lists: - offset - limit ### Bugfixes - Fixed incompatibilities with Pandas 2.0 in DatetimePartitioning.to_dataframe . - Fixed a crash when using non-“latin-1” characters in Panda’s DataFrame used as prediction data in BatchPredictionJob.score . - Fixed an issue where failed authentication when invoking datarobot.client.Client() raises a misleading error about client-server compatibility. - Fixed incompatibilities with Pandas 2.0 in AccuracyOverTime.get_as_dataframe . The method will now throw a ValueError if an empty list is passed to the parameter metrics . ### API changes - Added parameter unsupervised_type to the class DatetimePartitioning . - The sliced insight API endpoint GET: api/v2/insights// returns a paginated response. This means that it returns an empty response if no insights data is found, unlike GET: api/v2/projects//models/// , which returns 404 NOT FOUND in this case. To maintain backwards-compatibility, all methods that retrieve insights data raise 404 NOT FOUND if the insights API returns an empty response. ### Deprecation summary - Model.get_feature_fit_metadata() has been removed. Use Model.get_feature_effect_metadata instead. - DatetimeModel.get_feature_fit_metadata() has been removed. Use DatetimeModel.get_feature_effect_metadata instead. - Model.request_feature_fit has been removed. Use Model.request_feature_effect instead. - DatetimeModel.request_feature_fit has been removed. Use DatetimeModel.request_feature_effect instead. - Model.get_feature_fit has been removed. Use Model.get_feature_effect instead. - DatetimeModel.get_feature_fit has been removed. Use DatetimeModel.get_feature_effect instead. - Model.get_or_request_feature_fit has been removed. Use Model.get_or_request_feature_effect instead. - DatetimeModel.get_or_request_feature_fit has been removed. Use DatetimeModel.get_or_request_feature_effect instead. - Deprecated the use of SharingAccess in favor of SharingRole for sharing in the following classes: - DataStore.share - Deprecated the following methods for retrieving SharingAccess information. - DataStore.get_access_list . Please use DataStore.get_shared_roles instead. - CustomInferenceModel.assign_training_data was marked as deprecated and will be removed in v3.4. Use CustomModelVersion.create_clean and CustomModelVersion.create_from_previous instead. ### Configuration changes - Pins dependency on package urllib3 to be less than version 2.0.0. ### Deprecation summary - Deprecated parameter user_agent_suffix in datarobot.Client . user_agent_suffix will be removed in v3.4. Please use trace_context instead. ### Documentation changes - Fixed in-line documentation of DataRobotClientConfig . - Fixed documentation around client configuration from environment variables or config file. ### Experimental changes - Added experimental support for data matching: - DataMatching - DataMatchingQuery - Added new method DataMatchingQuery.get_result for returning data matching query results as pandas dataframes to DataMatchingQuery . - Changed behavior for returning results in the DataMatching . Instead of saving the results as a file, a pandas dataframe will be returned in the following methods: - DataMatching.get_closest_data - DataMatching.get_closest_data_for_model - DataMatching.get_closest_data_for_featurelist - Added experimental support for model lineage: ModelLineage - Changed behavior for methods that search for the closest data points in DataMatching . If the index is missing, instead of throwing the error, methods try to create the index and then query it. This is enabled by default, but if this is not the intended behavior it can be changed by passing False to the new build_index parameter added to the methods: - DataMatching.get_closest_data - DataMatching.get_closest_data_for_model - DataMatching.get_closest_data_for_featurelist - Added a new class Notebook for retrieving DataRobot Notebooks available to the user. - Added experimental support for data wrangling: - Recipe ## 3.1.1 ### Configuration changes - Removes dependency on package contextlib2 since the package is Python 3.7+. - Update typing-extensions to be inclusive of versions from 4.3.0 to < 5.0.0. ## 3.1.0 ### Enhancements - Added new methods BatchPredictionJob.apply_time_series_data_prep_and_score and BatchPredictionJob.apply_time_series_data_prep_and_score_to_file that apply time series data prep to a file or dataset and make batch predictions with a deployment. - Added new methods DataEngineQueryGenerator.prepare_prediction_dataset and DataEngineQueryGenerator.prepare_prediction_dataset_from_catalog that apply time series data prep to a file or catalog dataset and upload the prediction dataset to a project. - Added new max_wait parameter to method Project.create_from_dataset . Values larger than the default can be specified to avoid timeouts when creating a project from Dataset. - Added new method for creating a segmented modeling project from an existing clustering project and model Project.create_segmented_project_from_clustering_model . Please switch to this function if you are previously using ModelPackage for segmented modeling purposes. - Added new method is_unsupervised_clustering_or_multiclass for checking whether the clustering or multiclass parameters are used, quick and efficient without extra API calls. PredictionExplanations.is_unsupervised_clustering_or_multiclass - Retry idempotent requests which result in HTTP 502 and HTTP 504 (in addition to the previous HTTP 413, HTTP 429 and HTTP 503) - Added value PREPARED_FOR_DEPLOYMENT to the RECOMMENDED_MODEL_TYPE enum - Added two new methods to the ImageAugmentationList class: - ImageAugmentationList.list , - ImageAugmentationList.update ### Bugfixes - Added format key to Batch Prediction intake and output settings for S3, GCP and Azure ### API changes - The method PredictionExplanations.is_multiclass now adds an additional API call to check for multiclass target validity, which adds a small delay. - AdvancedOptions parameter blend_best_models defaults to false - AdvancedOptions parameter consider_blenders_in_recommendation defaults to false - DatetimePartitioning has parameter unsupervised_mode ### Deprecation summary - Deprecated method Project.create_from_hdfs . - Deprecated method DatetimePartitioning.generate . - Deprecated parameter in_use from ImageAugmentationList.create as DataRobot will take care of it automatically. - Deprecated property Deployment.capabilities from Deployment . - ImageAugmentationSample.compute was removed in v3.1. You can get the same information with the method ImageAugmentationList.compute_samples . - sample_id parameter removed from ImageAugmentationSample.list . Please use auglist_id instead. ### Documentation changes - Update the documentation to suggest that setting use_backtest_start_end_format of DatetimePartitioning.to_specification to True will mirror the same behavior as the Web UI. - Update the documentation to suggest setting use_start_end_format of Backtest.to_specification to True will mirror the same behavior as the Web UI. ## 3.0.3 ### Bugfixes - Fixed an issue affecting backwards compatibility in datarobot.models.DatetimeModel , where an unexpected keyword from the DataRobot API would break class deserialization. ## 3.0.2 ### Bugfixes - Restored Model.get_leaderboard_ui_permalink , Model.open_model_browser , These methods were accidentally removed instead of deprecated. - Fix for ipykernel < 6.0.0 which does not persist contextvars across cells ### Deprecation summary - Deprecated method Model.get_leaderboard_ui_permalink . Please use Model.get_uri instead. - Deprecated method Model.open_model_browser . Please use Model.open_in_browser instead. ## 3.0.1 ### Bugfixes - Added typing-extensions as a required dependency for the DataRobot Python API client. ## 3.0.0 ### New features - Version 3.0 of the Python client does not support Python 3.6 and earlier versions. Version 3.0 currently supports Python 3.7+. - The default Autopilot mode for project.start_autopilot has changed to Quick mode. - For datetime-aware models, you can now calculate and retrieve feature impact for backtests other than zero and holdout: - DatetimeModel.get_feature_impact - DatetimeModel.request_feature_impact - DatetimeModel.get_or_request_feature_impact - Added a backtest field to feature impact metadata: Model.get_or_request_feature_impact . This field is null for non-datetime-aware models and greater than or equal to zero for holdout in datetime-aware models. - You can use a new method to retrieve the canonical URI for a project, model, deployment, or dataset: - Project.get_uri - Model.get_uri - Deployment.get_uri - Dataset.get_uri - You can use a new method to open a class in a browser based on their URI (project, model, deployment, or dataset): - Project.open_in_browser - Model.open_in_browser - Deployment.open_in_browser - Dataset.open_in_browser - Added a new method for opening DataRobot in a browser: datarobot.rest.RESTClientObject.open_in_browser() . Invoke the method via dr.Client().open_in_browser() . - Altered method Project.create_featurelist to accept five new parameters (please see documentation for information about usage): - starting_featurelist - starting_featurelist_id - starting_featurelist_name - features_to_include - features_to_exclude - Added a new method to retrieve a feature list by name: Project.get_featurelist_by_name . - Added a new convenience method to create datasets: Dataset.upload . - Altered the method Model.request_predictions to accept four new parameters: - dataset - file - file_path - dataframe - Note that the method already supports the parameter dataset_id and all data source parameters are mutually exclusive. - Added a new method to datarobot.models.Dataset , Dataset.get_as_dataframe , which retrieves all the originally uploaded data in a pandas DataFrame. - Added a new method to datarobot.models.Dataset , Dataset.share , which allows the sharing of a dataset with another user. - Added new convenience methods to datarobot.models.Project for dealing with partition classes. Both methods should be called before Project.analyze_and_model . - Project.set_partitioning_method intelligently creates the correct partition class for a regular project, based on input arguments. - Project.set_datetime_partitioning creates the correct partition class for a time series project. - Added a new method to datarobot.models.Project Project.get_top_model which returns the highest scoring model for a metric of your choice. - Use the new method Deployment.predict_batch to pass a file, file path, or DataFrame to datarobot.models.Deployment to easily make batch predictions and return the results as a DataFrame. - Added support for passing in a credentials ID or credentials data to Project.create_from_data_source as an alternative to providing a username and password. - You can now pass in a max_wait value to AutomatedDocument.generate . - Added a new method to datarobot.models.Project Project.get_dataset which retrieves the dataset used during creation of a project. - Added two new properties to datarobot.models.Project : - catalog_id - catalog_version_id - Added a new Autopilot method to datarobot.models.Project Project.analyze_and_model which allows you to initiate Autopilot or data analysis against data uploaded to DataRobot. - Added a new convenience method to datarobot.models.Project Project.set_options which allows you to save AdvancedOptions values for use in modeling. - Added a new convenience method to datarobot.models.Project Project.get_options which allows you to retrieve saved modeling options. ### Enhancements - Refactored the global singleton client connection ( datarobot.client.Client() ) to use ContextVar instead of a global variable for better concurrency support. - Added support for creating monotonic feature lists for time series projects. Set skip_datetime_partition_column to True to create monotonic feature list. For more information see datarobot.models.Project.create_modeling_featurelist() . - Added information about vertex to advanced tuning parameters datarobot.models.Model.get_advanced_tuning_parameters() . - Added the ability to automatically use saved AdvancedOptions set using Project.set_options in Project.analyze_and_model . ### Bugfixes - Dataset.list no longer throws errors when listing datasets with no owner. - Fixed an issue with the creation of BatchPredictionJobDefinitions containing a schedule. - Fixed error handling in datarobot.helpers.partitioning_methods.get_class . - Fixed issue with portions of the payload not using camelCasing in Project.upload_dataset_from_catalog . ### API changes - The Python client now outputs a DataRobotProjectDeprecationWarning when you attempt to access certain resources (projects, models, deployments, etc.) that are deprecated or disabled as a result of the DataRobot platform’s migration to Python 3. - The Python client now raises a TypeError when you try to retrieve a labelwise ROC on a binary model or a binary ROC on a multilabel model. - The method Dataset.create_from_data_source now raises InvalidUsageError if username and password are not passed as a pair together. ### Deprecation summary - Model.get_leaderboard_ui_permalink has been removed. Use Model.get_uri instead. - Model.open_model_browser has been removed. Use Model.open_in_browser instead. - Project.get_leaderboard_ui_permalink has been removed. Use Project.get_uri instead. - Project.open_leaderboard_browser has been removed. Use Project.open_in_browser instead. - Enum VARIABLE_TYPE_TRANSFORM.CATEGORICAL has been removed - Instantiation of Blueprint using a dict has been removed. Use Blueprint.from_data instead. - Specifying an environment to use for testing with CustomModelTest has been removed. - CustomModelVersion ’s required_metadata parameter has been removed. Use required_metadata_values instead. - CustomTaskVersion ’s required_metadata parameter has been removed. Use required_metadata_values instead. - Instantiation of Feature using a dict has been removed. Use Feature.from_data instead. - Instantiation of Featurelist using a dict has been removed. Use Featurelist.from_data instead. - Instantiation of Model using a dict, tuple, or the data parameter has been removed. Use Model.from_data instead. - Instantiation of Project using a dict has been removed. Use Project.from_data instead. - Project ’s quickrun parameter has been removed. Pass AUTOPILOT_MODE.QUICK as the mode instead. - Project ’s scaleout_max_train_pct and scaleout_max_train_rows parameters have been removed. - ComplianceDocumentation has been removed. Use AutomatedDocument instead. - The Deployment method create_from_custom_model_image was removed. Use Deployment.create_from_custom_model_version instead. - PredictJob.create has been removed. Use Model.request_predictions instead. - Model.fetch_resource_data has been removed. Use Model.get instead. - The class CustomInferenceImage was removed. Use CustomModelVersion with base_environment_id instead. - Project.set_target has been deprecated. Use Project.analyze_and_model instead. ### Configuration changes - Added a context manager client_configuration that can be used to change the connection configuration temporarily, for use in asynchronous or multithreaded code. - Upgraded the Pillow library to version 9.2.0. Users installing DataRobot with the “images” extra ( pip install datarobot[images] ) should note that this is a required library. ### Experimental changes - Added experimental support for retrieving document thumbnails: - DocumentThumbnail - DocumentPageFile - Added experimental support to retrieve document text extraction samples using: - DocumentTextExtractionSample - DocumentTextExtractionSamplePage - DocumentTextExtractionSampleDocument - Added experimental deployment improvements: - RetrainingPolicy can be used to manage retraining policies associated with a deployment. - Added an experimental deployment improvement: - Use RetrainingPolicyRun to manage retraining policies run for a retraining policy associated with a deployment. - Added new methods to RetrainingPolicy : - Use RetrainingPolicy.get to get a retraining policy associated with a deployment. - Use RetrainingPolicy.delete to delete a retraining policy associated with a deployment. ## 2.29.0 ### New features - Added support to pass max_ngram_explanations parameter in batch predictions that will trigger the compute of text prediction explanations. - BatchPredictionJob.score - Added support to pass calculation mode to prediction explanations ( mode parameter in PredictionExplanations.create ) as well as batch scoring ( explanations_mode in BatchPredictionJob.score ) for multiclass models. Supported modes: - TopPredictionsMode - ClassListMode - Added method datarobot.CalendarFile.create_calendar_from_dataset() to the calendar file that allows us to create a calendar from a dataset. - Added experimental support for n_clusters parameter in Model.train_datetime and DatetimeModel.retrain that allows to specify number of clusters when creating models in Time Series Clustering project. - Added new parameter clone to datarobot.CombinedModel.set_segment_champion() that allows to set a new champion model in a cloned model instead of the original one, leaving latter unmodified. - Added new property is_active_combined_model to datarobot.CombinedModel that indicates if the selected combined model is currently the active one in the segmented project. - Added new datarobot.models.Project.get_active_combined_model() that allows users to get the currently active combined model in the segmented project. - Added new parameters read_timeout to method ShapMatrix.get_as_dataframe . Values larger than the default can be specified to avoid timeouts when requesting large files. ShapMatrix.get_as_dataframe - Added support for bias mitigation with the following methods - Project.get_bias_mitigated_models - Project.apply_bias_mitigation - Project.request_bias_mitigation_feature_info - Project.get_bias_mitigation_feature_info and by adding new bias mitigation params - bias_mitigation_feature_name - bias_mitigation_technique - include_bias_mitigation_feature_as_predictor_variable to the existing method - Project.start and by adding this enum to supply params to some of the above functionality datarobot.enums.BiasMitigationTechnique - Added new property status to datarobot.models.Deployment that represents model deployment status. - Added new Deployment.activate and Deployment.deactivate that allows deployment activation and deactivation - Added new Deployment.delete_monitoring_data to delete deployment monitoring data. ### Enhancements - Added support for specifying custom endpoint URLs for S3 access in batch predictions: - BatchPredictionJob.score - BatchPredictionJob.score See: `endpoint_url` parameter. - Added guide on :ref: `working with binary data ` - Added multithreading support to binary data helper functions. - Binary data helpers image defaults aligned with application’s image preprocessing. - Added the following accuracy metrics to be retrieved for a deployment - TPR, PPV, F1 and MCC :ref: `Deployment monitoring ` ### Bugfixes - Don’t include holdout start date, end date, or duration in datetime partitioning payload when holdout is disabled. - Removed ICE Plot capabilities from Feature Fit. - Handle undefined calendar_name in CalendarFile.create_calendar_from_dataset - Raise ValueError for submitted calendar names that are not strings ### API changes - version field is removed from ImportedModel object ### Deprecation summary - Reason Codes objects deprecated in 2.13 version were removed. Please use Prediction Explanations instead. ### Configuration changes - The upper version constraint on pandas has been removed. ### Documentation changes - Fixed a minor typo in the example for Dataset.create_from_data_source. - Update the documentation to suggest that feature_derivation_window_end of datarobot.DatetimePartitioningSpecification class should be a negative or zero. ## 2.28.0 ### New features - Added new parameter upload_read_timeout to BatchPredictionJob.score and BatchPredictionJob.score_to_file to indicate how many seconds to wait until intake dataset uploads to server. Default value 600s. - Added the ability to turn off supervised feature reduction for Time Series projects. Option use_supervised_feature_reduction can be set in AdvancedOptions . - Allow maximum_memory to be input for custom tasks versions. This will be used for setting the limit to which a custom task prediction container memory can grow. - Added method datarobot.models.Project.get_multiseries_names() to the project service which will return all the distinct entries in the multiseries column - Added new segmentation_task_id attribute to datarobot.models.Project.set_target() that allows to start project as Segmented Modeling project. - Added new property is_segmented to datarobot.models.Project that indicates if project is a regular one or Segmented Modeling project. - Added method datarobot.models.Project.restart_segment() to the project service that allows to restart single segment that hasn’t reached modeling phase. - Added the ability to interact with Combined Models in Segmented Modeling projects. Available with new class: datarobot.CombinedModel . Functionality: - [datarobot.CombinedModel.get()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.CombinedModel.get) - [datarobot.CombinedModel.get_segments_info()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.CombinedModel.get_segments_info) - [datarobot.CombinedModel.get_segments_as_dataframe()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.CombinedModel.get_segments_as_dataframe) - [datarobot.CombinedModel.get_segments_as_csv()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.CombinedModel.get_segments_as_csv) - [datarobot.CombinedModel.set_segment_champion()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.CombinedModel.set_segment_champion) - Added the ability to create and retrieve segmentation tasks used in Segmented Modeling projects. Available with new class: [datarobot.SegmentationTask](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.SegmentationTask). Functionality: - [datarobot.SegmentationTask.create()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.SegmentationTask.create) - [datarobot.SegmentationTask.list()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.SegmentationTask.list) - [datarobot.SegmentationTask.get()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.SegmentationTask.get) - Added new class: [datarobot.SegmentInfo](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.SegmentInfo) that allows to get information on all segments of Segmented modeling projects, i.e. segment project ID, model counts, autopilot status. Functionality: - [datarobot.SegmentInfo.list()](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.SegmentInfo.list) - Added new methods to base `APIObject` to assist with dictionary and json serialization of child objects. Functionality: - `APIObject.to_dict` - `APIObject.to_json` - Added new methods to `ImageAugmentationList` for interacting with image augmentation samples. Functionality: - `ImageAugmentationList.compute_samples` - `ImageAugmentationList.retrieve_samples` - Added the ability to set a prediction threshold when creating a deployment from a learning model. - Added support for governance, owners, predictionEnvironment, and fairnessHealth fields when querying for a Deployment object. - Added helper methods for working with files, images and documents. Methods support conversion of file contents into base64 string representations. Methods for images provide also image resize and transformation support. Functionality: - [get_encoded_file_contents_from_urls](https://docs.datarobot.com/en/docs/api/reference/sdk/binary_data_helpers.html#datarobot.helpers.binary_data_utils.get_encoded_file_contents_from_urls) - [get_encoded_file_contents_from_paths](https://docs.datarobot.com/en/docs/api/reference/sdk/binary_data_helpers.html#datarobot.helpers.binary_data_utils.get_encoded_file_contents_from_paths) - [get_encoded_image_contents_from_paths](https://docs.datarobot.com/en/docs/api/reference/sdk/binary_data_helpers.html#datarobot.helpers.binary_data_utils.get_encoded_image_contents_from_paths) - [get_encoded_image_contents_from_urls](https://docs.datarobot.com/en/docs/api/reference/sdk/binary_data_helpers.html#datarobot.helpers.binary_data_utils.get_encoded_image_contents_from_urls) ### Enhancements - Requesting metadata instead of actual data of datarobot.PredictionExplanations to reduce the amount of data transfer ### Bugfixes - Fix a bug in Job.get_result_when_complete for Prediction Explanations job type to populate all attribute of of datarobot.PredictionExplanations instead of just one - Fix a bug in datarobot.models.ShapImpact where row_count was not optional - Allow blank value for schema and catalog in RelationshipsConfiguration response data - Fix a bug where credentials were incorrectly formatted in Project.upload_dataset_from_catalog and Project.upload_dataset_from_data_source - Rejecting downloads of Batch Prediction data that was not written to the localfile output adapter - Fix a bug in datarobot.models.BatchPredictionJobDefinition.create() where schedule was not optional for all cases ### API changes - User can include ICE plots data in the response when requesting Feature Effects/Feature Fit. Extended methods are - Model.get_feature_effect , - Model.get_feature_fit , - DatetimeModel.get_feature_effect and - DatetimeModel.get_feature_fit . ### Deprecation summary - attrs library is removed from library dependencies - ImageAugmentationSample.compute was marked as deprecated and will be removed in v2.30. You can get the same information with newly introduced method ImageAugmentationList.compute_samples - ImageAugmentationSample.list using sample_id - Deprecating scaleout parameters for projects / models. Includes scaleout_modeling_mode , scaleout_max_train_pct , and scaleout_max_train_rows ### Configuration changes - pandas upper version constraint is updated to include version 1.3.5. ### Documentation changes - Fixed “from datarobot.enums” import in Unsupervised Clustering example provided in docs. ## 2.27.0 ### New features - datarobot.UserBlueprint is now mature with full support of functionality. Users are encouraged to use the Blueprint Workshop instead of this class directly. - Added the arguments attribute in datarobot.CustomTaskVersion . - Added the ability to retrieve detected errors in the potentially multicategorical feature types that prevented the feature to be identified as multicategorical. Project.download_multicategorical_data_format_errors - Added the support of listing/updating user roles on one custom task. - datarobot.CustomTask.get_access_list() - datarobot.CustomTask.share() - Added a method datarobot.models.Dataset.create_from_query_generator() . This creates a dataset in the AI catalog from a datarobot.DataEngineQueryGenerator . - Added the new functionality of creating a user blueprint with a custom task version id. datarobot.UserBlueprint.create_from_custom_task_version_id() . - The DataRobot Python Client is no longer published under the Apache-2.0 software license, but rather under the terms of the DataRobot Tool and Utility Agreement. - Added a new class: datarobot.DataEngineQueryGenerator . This class generates a Spark SQL query to apply time series data prep to a dataset in the AI catalog. Functionality: - [datarobot.DataEngineQueryGenerator.create()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.DataEngineQueryGenerator.create) - [datarobot.DataEngineQueryGenerator.get()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.DataEngineQueryGenerator.get) - [datarobot.DataEngineQueryGenerator.create_dataset()](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.DataEngineQueryGenerator.create_dataset) See the :ref: `time series data prep documentation ` for more information. - Added the ability to upload a prediction dataset into a project from the AI catalog [Project.upload_dataset_from_catalog](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project.upload_dataset_from_catalog). - Added the ability to specify the number of training rows to use in SHAP based Feature Impact computation. Extended method: - `ShapImpact.create` - Added the ability to retrieve and restore features that have been reduced using the time series feature generation and reduction functionality. The functionality comes with a new class: [datarobot.models.restore_discarded_features.DiscardedFeaturesInfo](https://docs.datarobot.com/en/docs/api/reference/public-api/features.html#datarobot.models.restore_discarded_features.DiscardedFeaturesInfo). Functionality: - [datarobot.models.restore_discarded_features.DiscardedFeaturesInfo.retrieve()](https://docs.datarobot.com/en/docs/api/reference/public-api/features.html#datarobot.models.restore_discarded_features.DiscardedFeaturesInfo.retrieve) - [datarobot.models.restore_discarded_features.DiscardedFeaturesInfo.restore()](https://docs.datarobot.com/en/docs/api/reference/public-api/features.html#datarobot.models.restore_discarded_features.DiscardedFeaturesInfo.restore) - Added the ability to control class mapping aggregation in multiclass projects via [ClassMappingAggregationSettings](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.helpers.ClassMappingAggregationSettings) passed as a parameter to [Project.set_target](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project.set_target) - Added support for :ref: `unsupervised clustering projects` - Added the ability to compute and retrieve Feature Effects for a Multiclass model using [datarobot.models.Model.request_feature_effects_multiclass()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.request_feature_effects_multiclass), [datarobot.models.Model.get_feature_effects_multiclass()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_feature_effects_multiclass) or [datarobot.models.Model.get_or_request_feature_effects_multiclass()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_or_request_feature_effects_multiclass) methods. For datetime models use following methods [datarobot.models.DatetimeModel.request_feature_effects_multiclass()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.request_feature_effects_multiclass), [datarobot.models.DatetimeModel.get_feature_effects_multiclass()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_feature_effects_multiclass) or [datarobot.models.DatetimeModel.get_or_request_feature_effects_multiclass()](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_or_request_feature_effects_multiclass) with `backtest_index` specified - Added the ability to get and update challenger model settings for deployment class: [datarobot.models.Deployment](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment) Functionality: - [datarobot.models.Deployment.get_challenger_models_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_challenger_models_settings) - [datarobot.models.Deployment.update_challenger_models_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_challenger_models_settings) - Added the ability to get and update segment analysis settings for deployment class: [datarobot.models.Deployment](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment) Functionality: - [datarobot.models.Deployment.get_segment_analysis_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_segment_analysis_settings) - [datarobot.models.Deployment.update_segment_analysis_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_segment_analysis_settings) - Added the ability to get and update predictions by forecast date settings for deployment class: [datarobot.models.Deployment](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment) Functionality: - [datarobot.models.Deployment.get_predictions_by_forecast_date_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.get_predictions_by_forecast_date_settings) - [datarobot.models.Deployment.update_predictions_by_forecast_date_settings()](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update_predictions_by_forecast_date_settings) - Added the ability to specify multiple feature derivation windows when creating a Relationships Configuration using [RelationshipsConfiguration.create](https://docs.datarobot.com/en/docs/api/reference/public-api/features.html#datarobot.models.RelationshipsConfiguration.create) - Added the ability to manipulate a legacy conversion for a custom inference model, using the class: [CustomModelVersionConversion](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.models.CustomModelVersionConversion) Functionality: - [CustomModelVersionConversion.run_conversion](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.models.CustomModelVersionConversion.run_conversion) - [CustomModelVersionConversion.stop_conversion](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.models.CustomModelVersionConversion.stop_conversion) - [CustomModelVersionConversion.get](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.models.CustomModelVersionConversion.get) - [CustomModelVersionConversion.get_latest](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.models.CustomModelVersionConversion.get_latest) - [CustomModelVersionConversion.list](https://docs.datarobot.com/en/docs/api/reference/sdk/custom-models.html#datarobot.models.CustomModelVersionConversion.list) ### Enhancements - Project.get returns the query_generator_id used for time series data prep when applicable. - Feature Fit & Feature Effects can return datetime instead of numeric for feature_type field for numeric features that are derived from dates. - These methods now provide additional field rowCount in SHAP based Feature Impact results. - ShapImpact.create - ShapImpact.get - Improved performance when downloading prediction dataframes for Multilabel projects using: - Predictions.get_all_as_dataframe - PredictJob.get_predictions - Job.get_result ### Bugfixes - Fix datarobot.CustomTaskVersion and datarobot.CustomModelVersion to correctly format required_metadata_values before sending them via API - Fixed response validation that could cause DataError when using datarobot.models.Dataset for a dataset with a description that is an empty string. ### API changes - RelationshipsConfiguration.create will include a new key data_source_id in data_source field when applicable ### Deprecation summary - Model.get_all_labelwise_roc_curves has been removed. You can get the same information with multiple calls of Model.get_labelwise_roc_curves , one per data source. - Model.get_all_multilabel_lift_charts has been removed. You can get the same information with multiple calls of Model.get_multilabel_lift_charts , one per data source. ### Documentation changes - This release introduces a new documentation organization. The organization has been modified to better reflect the end-to-end modeling workflow. The new “Tutorials” section has 5 major topics that outline the major components of modeling: Data, Modeling, Predictions, MLOps, and Administration. - The Getting Started workflow is now hosted at DataRobot’s API Documentation Home . - Added an example of how to set up optimized datetime partitioning for time series projects. ## 2.26.0 ### New features - Added the ability to use external baseline predictions for time series project. External dataset can be validated using datarobot.models.Project.validate_external_time_series_baseline() . Option can be set in AdvancedOptions to scale datarobot models’ accuracy performance using external dataset’s accuracy performance. See the :ref: external baseline predictions documentation for more information. - Added the ability to generate exponentially weighted moving average features for time series project. Option can be set in AdvancedOptions and controls the alpha parameter used in exponentially weighted moving average operation. - Added the ability to request a specific model be prepared for deployment using Project.start_prepare_model_for_deployment . - Added a new class: datarobot.CustomTask . This class is a custom task that you can use as part (or all) of your blue print for training models. It needs datarobot.CustomTaskVersion before it can properly be used. - Functionality: - Added a new class: datarobot.CustomTaskVersion . This class is for management of specific versions of a custom task. - Functionality: - Added the ability compute batch predictions for an in-memory DataFrame using BatchPredictionJob.score - Added the ability to specify feature discovery settings when creating a Relationships Configuration using RelationshipsConfiguration.create ### Enhancements - Improved performance when downloading prediction dataframes using: - Predictions.get_all_as_dataframe - PredictJob.get_predictions - Job.get_result - Added new max_wait parameter to methods: - Dataset.create_from_url - Dataset.create_from_in_memory_data - Dataset.create_from_data_source - Dataset.create_version_from_in_memory_data - Dataset.create_version_from_url - Dataset.create_version_from_data_source ### Bugfixes - Model.get will return a DatetimeModel instead of Model whenever the project is datetime partitioned. This enables the ModelRecommendation.get_model to return a DatetimeModel instead of Model whenever the project is datetime partitioned. - Try to read Feature Impact result if existing jobId is None in Model.get_or_request_feature_impact . - Set upper version constraints for pandas. - RelationshipsConfiguration.create will return a catalog in data_source field - Argument required_metadata_keys was not properly being sent in the update and create requests for datarobot.ExecutionEnvironment . - Fix issue with datarobot.ExecutionEnvironment create method failing when used against older versions of the application - datarobot.CustomTaskVersion was not properly handling required_metadata_values from the API response ### API changes - Updated Project.start to use AUTOPILOT_MODE.QUICK when the autopilot_on param is set to True. This brings it in line with Project.set_target . - Updated project.start_autopilot to accept the following new GA parameters that are already in the public API: consider_blenders_in_recommendation , run_leakage_removed_feature_list ### Deprecation summary - The required_metadata property of datarobot.CustomModelVersion has been deprecated. required_metadata_values should be used instead. - The required_metadata property of datarobot.CustomTaskVersion has been deprecated. required_metadata_values should be used instead. ### Configuration changes - Now requires dependency on package scikit-learn rather than sklearn . Note: This dependency is only used in example code. See this scikit-learn issue for more information. - Now permits dependency on package attrs to be less than version 21. This fixes compatibility with apache-airflow. - Allow to setup Authorization: type header for OAuth2 Bearer tokens. ### Documentation changes - Update the documentation with respect to the permission that controls AI Catalog dataset snapshot behavior. ## 2.25.0 ### New features - There is a new AnomalyAssessmentRecord object that implements public API routes to work with anomaly assessment insight. This also adds explanations and predictions preview classes. The insight is available for anomaly detection models in time series unsupervised projects which also support calculation of Shapley values. - AnomalyAssessmentPredictionsPreview - AnomalyAssessmentExplanations Functionality: - Initialize an anomaly assessment insight for the specified subset. - [DatetimeModel.initialize_anomaly_assessment](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.initialize_anomaly_assessment) - Get anomaly assessment records, shap explanations, predictions preview: - [DatetimeModel.get_anomaly_assessment_records](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_anomaly_assessment_records) list available records - [AnomalyAssessmentRecord.get_predictions_preview](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.anomaly_assessment.AnomalyAssessmentRecord.get_predictions_preview) get predictions preview for the record - [AnomalyAssessmentRecord.get_latest_explanations](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.anomaly_assessment.AnomalyAssessmentRecord.get_latest_explanations) get latest predictions along with shap explanations for the most anomalous records. - [AnomalyAssessmentRecord.get_explanations](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.anomaly_assessment.AnomalyAssessmentRecord.get_explanations) get predictions along with shap explanations for the most anomalous records for the specified range. - Delete anomaly assessment record: - [AnomalyAssessmentRecord.delete](https://docs.datarobot.com/en/docs/api/reference/public-api/insights.html#datarobot.models.anomaly_assessment.AnomalyAssessmentRecord.delete) delete record - Added an ability to calculate and retrieve Datetime trend plots for [DatetimeModel](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel). This includes Accuracy over Time, Forecast vs Actual, and Anomaly over Time. Plots can be calculated using a common method: - [DatetimeModel.compute_datetime_trend_plots](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.compute_datetime_trend_plots) Metadata for plots can be retrieved using the following methods: - [DatetimeModel.get_accuracy_over_time_plots_metadata](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_accuracy_over_time_plots_metadata) - [DatetimeModel.get_forecast_vs_actual_plots_metadata](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_forecast_vs_actual_plots_metadata) - [DatetimeModel.get_anomaly_over_time_plots_metadata](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_anomaly_over_time_plots_metadata) Plots can be retrieved using the following methods: - [DatetimeModel.get_accuracy_over_time_plot](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_accuracy_over_time_plot) - [DatetimeModel.get_forecast_vs_actual_plot](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_forecast_vs_actual_plot) - [DatetimeModel.get_anomaly_over_time_plot](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_anomaly_over_time_plot) Preview plots can be retrieved using the following methods: - [DatetimeModel.get_accuracy_over_time_plot_preview](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_accuracy_over_time_plot_preview) - [DatetimeModel.get_forecast_vs_actual_plot_preview](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_forecast_vs_actual_plot_preview) - [DatetimeModel.get_anomaly_over_time_plot_preview](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel.get_anomaly_over_time_plot_preview) - Support for Batch Prediction Job Definitions has now been added through the following class: [BatchPredictionJobDefinition](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition). You can create, update, list and delete definitions using the following methods: - [BatchPredictionJobDefinition.list](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.list) - [BatchPredictionJobDefinition.create](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.create) - [BatchPredictionJobDefinition.update](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.update) - [BatchPredictionJobDefinition.delete](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.BatchPredictionJobDefinition.delete) ### Enhancements - Added a new helper function to create Dataset Definition, Relationship and Secondary Dataset used by Feature Discovery Project. They are accessible via DatasetDefinition Relationship SecondaryDataset - Added new helper function to projects to retrieve the recommended model. Project.recommended_model - Added method to download feature discovery recipe SQLs (limited beta feature). Project.download_feature_discovery_recipe_sqls . - Added docker_context_size and docker_image_size to datarobot.ExecutionEnvironmentVersion ### Bugfixes - Remove the deprecation warnings when using with latest versions of urllib3. - FeatureAssociationMatrix.get is now using correct query param name when featurelist_id is specified. - Handle scalar values in shapBaseValue while converting a predictions response to a data frame. - Ensure that if a configured endpoint ends in a trailing slash, the resulting full URL does not end up with double slashes in the path. - Model.request_frozen_datetime_model is now implementing correct validation of input parameter training_start_date . ### API changes - Arguments secondary_datasets now accept SecondaryDataset to create secondary dataset configurations - SecondaryDatasetConfigurations.create - Arguments dataset_definitions and relationships now accept DatasetDefinition Relationship to create and replace relationships configuration - RelationshipsConfiguration.create creates a new relationships configuration between datasets - RelationshipsConfiguration.retrieve retrieve the requested relationships configuration - Argument required_metadata_keys has been added to datarobot.ExecutionEnvironment . This should be used to define a list of RequiredMetadataKey . datarobot.CustomModelVersion that use a base environment with required_metadata_keys must define values for these fields in their respective required_metadata - Argument required_metadata has been added to datarobot.CustomModelVersion . This should be set with relevant values defined by the base environment’s required_metadata_keys ## 2.24.0 ### New features - Partial history predictions can be made with time series time series multiseries models using the allow_partial_history_time_series_predictions attribute of the datarobot.DatetimePartitioningSpecification . See the :ref: Time Series documentation for more info. - Multicategorical Histograms are now retrievable. They are accessible via MulticategoricalHistogram or Feature.get_multicategorical_histogram . - Add methods to retrieve per-class lift chart data for multilabel models: Model.get_multilabel_lift_charts and Model.get_all_multilabel_lift_charts . - Add methods to retrieve labelwise ROC curves for multilabel models: Model.get_labelwise_roc_curves and Model.get_all_labelwise_roc_curves . - Multicategorical Pairwise Statistics are now retrievable. They are accessible via PairwiseCorrelations , PairwiseJointProbabilities and PairwiseConditionalProbabilities or Feature.get_pairwise_correlations , Feature.get_pairwise_joint_probabilities and Feature.get_pairwise_conditional_probabilities . - Add methods to retrieve prediction results of a deployment: : - Deployment.get_prediction_results - Add method to download scoring code of a deployment using Deployment.download_scoring_code . - Added Automated Documentation: now you can automatically generate documentation about various entities within the platform, such as specific models or projects. Check out the - ref: Automated Documentation overview and also refer to the :ref: API Reference for more details. - Create a new Dataset version for a given dataset by uploading from a file, URL or in-memory datasource. : - Dataset.create_version_from_file ### Enhancements - Added a new status called FAILED to from BatchPredictionJob as this is a new status coming to Batch Predictions in an upcoming version of DataRobot. - Added base_environment_version_id to datarobot.CustomModelVersion . - Support for downloading feature discovery training or prediction dataset using Project.download_feature_discovery_dataset . - Added datarobot.models.FeatureAssociationMatrix , datarobot.models.FeatureAssociationMatrixDetails and datarobot.models.FeatureAssociationFeaturelists that can be used to retrieve feature associations data as an alternative to Project.get_associations , Project.get_association_matrix_details and Project.get_association_featurelists methods. ### Bugfixes - Fixed response validation that could cause DataError when using TrainingPredictions.list and TrainingPredictions.get_all_as_dataframe methods if there are training predictions computed with explanation_algorithm . ### API changes - Remove desired_memory param from the following classes: datarobot.CustomInferenceModel , datarobot.CustomModelVersion , datarobot.CustomModelTest - Remove desired_memory param from the following methods: CustomInferenceModel.create , CustomModelVersion.create_clean , CustomModelVersion.create_from_previous , CustomModelTest.create and CustomModelTest.create ### Deprecation summary - class ComplianceDocumentation will be deprecated in v2.24 and will be removed entirely in v2.27. Use AutomatedDocument instead. To start off, see the - ref: Automated Documentation overview for details. ### Documentation changes - Remove reference to S3 for Project.upload_dataset since it is not supported by the server ## 2.23.0 ### New features - Calendars for time series projects can now be automatically generated by providing a country code to the method CalendarFile.create_calendar_from_country_code . A list of allowed country codes can be retrieved using CalendarFile.get_allowed_country_codes For more information, see the :ref: calendar documentation . - Added calculate_all_series`` param to [ DatetimeModel.compute_series_accuracy`](../../sdk/datarobot-models.md#datarobot.models.DatetimeModel.compute_series_accuracy). This option allows users to compute series accuracy for all available series at once, while by default it is computed for first 1000 series only. - Added ability to specify sampling method when setting target of OTV project. Option can be set in AdvancedOptions and changes a way training data is defined in autopilot steps. - Add support for custom inference model k8s resources management. This new feature enables users to control k8s resources allocation for their executed model in the k8s cluster. It involves in adding the following new parameters: network_egress_policy , desired_memory , maximum_memory , replicas to the following classes: datarobot.CustomInferenceModel , datarobot.CustomModelVersion , datarobot.CustomModelTest - Add support for multiclass custom inference and training models. This enables users to create classification custom models with more than two class labels. The datarobot.CustomInferenceModel class can now use datarobot.TARGET_TYPE.MULTICLASS for their target_type parameter. Class labels for inference models can be set/updated using either a file or as a list of labels. - Support for Listing all the secondary dataset configuration for a given project: : - SecondaryDatasetConfigurations.list - Add support for unstructured custom inference models. The datarobot.CustomInferenceModel class can now use datarobot.TARGET_TYPE.UNSTRUCTURED for its target_type parameter. target_name parameter is optional for UNSTRUCTURED target type. - All per-class lift chart data is now available for multiclass models using Model.get_multiclass_lift_chart . - AUTOPILOT_MODE.COMPREHENSIVE , a new mode , has been added to Project.set_target . - Add support for anomaly detection custom inference models. The datarobot.CustomInferenceModel class can now use datarobot.TARGET_TYPE.ANOMALY for its target_type parameter. target_name parameter is optional for ANOMALY target type. - Support for Updating and retrieving the secondary dataset configuration for a Feature discovery deployment: : - Deployment.update_secondary_dataset_config - Add support for starting and retrieving Feature Impact information for datarobot.CustomModelVersion - Search for interaction features and Supervised Feature reduction for feature discovery project can now be specified : in AdvancedOptions . - Feature discovery projects can now be created using the Project.start method by providing relationships_configuration_id . - Actions applied to input data during automated feature discovery can now be retrieved using FeatureLineage.get Corresponding feature lineage id is available as a new datarobot.models.Feature field feature_lineage_id . - Lift charts and ROC curves are now calculated for backtests 2+ in time series and OTV models. The data can be retrieved for individual backtests using Model.get_lift_chart and Model.get_roc_curve . - The following methods now accept a new argument called credential_data, the credentials to authenticate with the database, to use instead of user/password or credential ID: : - Dataset.create_from_data_source - Add support for DataRobot Connectors, datarobot.Connector provides a simple implementation to interface with connectors. ### Enhancements - Running Autopilot on Leakage Removed feature list can now be specified in AdvancedOptions . By default, Autopilot will always run on Informative Features - Leakage Removed feature list if it exists. If the parameter run_leakage_removed_feature_list is set to False, then Autopilot will run on Informative Features or available custom feature list. - Method Project.upload_dataset and Project.upload_dataset_from_data_source support new optional parameter secondary_datasets_config_id for Feature discovery project. ### Bugfixes - added disable_holdout param in datarobot.DatetimePartitioning - Using Credential.create_gcp produced an incompatible credential - SampleImage.list now supports Regression & Multilabel projects - Using BatchPredictionJob.score could in some circumstances result in a crash from trying to abort the job if it fails to start - Using BatchPredictionJob.score or BatchPredictionJob.score would produce incomplete results in case a job was aborted while downloading. This will now raise an exception. ### API changes - New sampling_method param in Model.train_datetime , Project.train_datetime , Model.train_datetime and Model.train_datetime . - New target_type param in datarobot.CustomInferenceModel - New arguments secondary_datasets , name , creator_full_name , creator_user_id , created , : featurelist_id , credentials_ids , project_version and is_default in datarobot.models.SecondaryDatasetConfigurations - New arguments secondary_datasets , name , featurelist_id to : SecondaryDatasetConfigurations.create - Class FeatureEngineeringGraph has been removed. Use datarobot.models.RelationshipsConfiguration instead. - Param feature_engineering_graphs removed from Project.set_target . - Param config removed from SecondaryDatasetConfigurations.create . ### Deprecation summary - supports_binary_classification and supports_regression are deprecated : for datarobot.CustomInferenceModel and will be removed in v2.24 - Argument config and supports_regression are deprecated : for datarobot.models.SecondaryDatasetConfigurations and will be removed in v2.24 - CustomInferenceImage has been deprecated and will be removed in v2.24. : datarobot.CustomModelVersion with base_environment_id should be used in their place. - environment_id and environment_version_id are deprecated for CustomModelTest.create ### Documentation changes - feature_lineage_id is added as a new parameter in the response for retrieval of a datarobot.models.Feature created by automated feature discovery or time series feature derivation. This id is required to retrieve a datarobot.models.FeatureLineage instance. ## 2.22.1 ### New features - Batch Prediction jobs now support :ref: dataset as intake settings for BatchPredictionJob.score . - Create a Dataset from DataSource: Dataset.create_from_data_sourceDataSource.create_dataset - Added support for Custom Model Dependency Management. Please see :ref: custom model documentation . New features added: Added new argumentbase_environment_idto methodsCustomModelVersion.create_cleanandCustomModelVersion.create_from_previousNew fieldsbase_environment_idanddependenciesto classdatarobot.CustomModelVersionNew classdatarobot.CustomModelVersionDependencyBuildto prepare custom model versions with dependencies.Made argumentenvironment_idofCustomModelTest.createoptional to enable using custom model versions with dependenciesNew fieldimage_typeadded to classdatarobot.CustomModelTestDeployment.create_from_custom_model_versioncan be used to create a deployment from a custom model version. - Added new parameters for starting and re-running Autopilot with customizable settings within Project.start_autopilot . - Added a new method to trigger Feature Impact calculation for a Custom Inference Image: CustomInferenceImage.calculate_feature_impact - Added new method to retrieve number of iterations trained for early stopping models. Currently supports only tree-based models. Model.get_num_iterations_trained . ### Enhancements - A description can now be added or updated for a project. Project.set_project_description . - Added new parameters read_timeout and max_wait to method Dataset.create_from_file . Values larger than the default can be specified for both to avoid timeouts when uploading large files. - Added new parameter metric to datarobot.models.deployment.TargetDrift , datarobot.models.deployment.FeatureDrift , Deployment.get_target_drift and Deployment.get_feature_drift . - Added new parameter timeout to BatchPredictionJob.download to indicate how many seconds to wait for the download to start (in case the job doesn’t start processing immediately). Set to -1 to disable. This parameter can also be sent as download_timeout to BatchPredictionJob.score and BatchPredictionJob.score . If the timeout occurs, the pending job will be aborted. - Added new parameter read_timeout to BatchPredictionJob.download to indicate how many seconds to wait between each downloaded chunk. This parameter can also be sent as download_read_timeout to BatchPredictionJob.score and BatchPredictionJob.score . - Added parameter catalog to BatchPredictionJob to both intake and output adapters for type jdbc . - Consider blenders in recommendation can now be specified in AdvancedOptions . Blenders will be included when autopilot chooses a model to prepare and recommend for deployment. - Added optional parameter max_wait to Deployment.replace_model to indicate the maximum time to wait for model replacement job to complete before erroring. ### Bugfixes - Handle null values in predictionExplanationMetadata["shapRemainingTotal"] while converting a predictions response to a data frame. - Handle null values in customModel["latestVersion"] - Removed an extra column status from BatchPredictionJob as it caused issues with never version of Trafaret validation. - Make predicted_vs_actual optional in Feature Effects data because a feature may have insufficient qualified samples. - Make jdbc_url optional in Data Store data because some data stores will not have it. - The method Project.get_datetime_models now correctly returns all DatetimeModel objects for the project, instead of just the first 100. - Fixed a documentation error related to snake_case vs camelCase in the JDBC settings payload. - Make trafaret validator for datasets use a syntax that works properly with a wider range of trafaret versions. - Handle extra keys in CustomModelTests and CustomModelVersions - ImageEmbedding and ImageActivationMap now supports regression projects. ### API changes - The default value for the mode param in Project.set_target has been changed from AUTOPILOT_MODE.FULL_AUTO to AUTOPILOT_MODE.QUICK ### Documentation changes - Added links to classes with duration parameters such as validation_duration and holdout_duration to provide duration string examples to users. - The :ref: models documentation has been revised to include section on how to train a new model and how to run cross-validation or backtesting for a model. ## 2.21.0 ### New features - Added new arguments explanation_algorithm and max_explanations to method Model.request_training_predictions . New fields explanation_algorithm , max_explanations and shap_warnings have been added to class TrainingPredictions . New fields prediction_explanations and shap_metadata have been added to class TrainingPredictionsIterator that is returned by method TrainingPredictions.iterate_rows . - Added new arguments explanation_algorithm and max_explanations to method Model.request_predictions . New fields explanation_algorithm , max_explanations and shap_warnings have been added to class Predictions . Method Predictions.get_all_as_dataframe has new argument serializer that specifies the retrieval and results validation method ( json or csv ) for the predictions. - Added possibility to compute ShapImpact.create and request ShapImpact.get SHAP impact scores for features in a model. - Added support for accessing Visual AI images and insights. See the DataRobot Python Package documentation, Visual AI Projects, section for details. - User can specify custom row count when requesting Feature Effects. Extended methods are Model.request_feature_effect and Model.get_or_request_feature_effect . - Users can request SHAP based predictions explanations for a models that support SHAP scores using ShapMatrix.create . - Added two new methods to Dataset to lazily retrieve paginated responses. - Dataset.iterate returns an iterator of the datasets that a user can view. - Dataset.iterate_all_features returns an iterator of the features of a dataset. - It’s possible to create an Interaction feature by combining two categorical features together using Project.create_interaction_feature . Operation result represented by models.InteractionFeature. . Specific information about an interaction feature may be retrieved by its name using models.InteractionFeature.get - Added the DatasetFeaturelist class to support featurelists on datasets in the AI Catalog. DatasetFeaturelists can be updated or deleted. Two new methods were also added to Dataset to interact with DatasetFeaturelists. These are Dataset.get_featurelists and Dataset.create_featurelist which list existing featurelists and create new featurelists on a dataset, respectively. - Added model_splits to DatetimePartitioningSpecification and to DatetimePartitioning . This will allow users to control the jobs per model used when building models. A higher number of model_splits will result in less downsampling, allowing the use of more post-processed data. - Added support for :ref: unsupervised projects . - Added support for external test set. Please see :ref: testset documentation - A new workflow is available for assessing models on external test sets in time series unsupervised projects. More information can be found in the :ref: documentation . - Project.upload_dataset and Model.request_predictions now accept actual_value_column - name of the actual value column, can be passed only with date range. - PredictionDataset objects now contain the following new fields: - New warning is added to data_quality_warnings of datarobot.models.PredictionDataset : single_class_actual_value_column . - Scores and insights on external test sets can be retrieved using ExternalScores , ExternalLiftChart , ExternalRocCurve . - Users can create payoff matrices for generating profit curves for binary classification projects using PayoffMatrix.create . - Deployment Improvements: - datarobot.models.deployment.TargetDrift can be used to retrieve target drift information. - datarobot.models.deployment.FeatureDrift can be used to retrieve feature drift information. - Deployment.submit_actuals will submit actuals in batches if the total number of actuals exceeds the limit of one single request. - Deployment.create_from_custom_model_image can be used to create a deployment from a custom model image. - Deployments now support predictions data collection that enables prediction requests and results to be saved in Predictions Data Storage. See Deployment.get_predictions_data_collection_settings and Deployment.update_predictions_data_collection_settings for usage. - New arguments send_notification and include_feature_discovery_entities are added to Project.share . - Now it is possible to specify the number of training rows to use in feature impact computation on supported project types (that is everything except unsupervised, multi-class, time-series). This does not affect SHAP based feature impact. Extended methods: - Model.request_feature_impact - Model.get_or_request_feature_impact - A new class FeatureImpactJob is added to retrieve Feature Impact records with metadata. The regular Job still works as before. - Added support for custom models. Please see :ref: custom model documentation . Classes added: - datarobot.ExecutionEnvironment and datarobot.ExecutionEnvironmentVersion to create and manage custom model executions environments - datarobot.CustomInferenceModel and datarobot.CustomModelVersion to create and manage custom inference models - datarobot.CustomModelTest to perform testing of custom models - Batch Prediction jobs now support forecast and historical Time Series predictions using the new argument timeseries_settings for BatchPredictionJob.score . - Batch Prediction jobs now support scoring to Azure and Google Cloud Storage with methods BatchPredictionJob.score_azure and BatchPredictionJob.score_gcp . - Now it’s possible to create Relationships Configurations to introduce secondary datasets to projects. A configuration specifies additional datasets to be included to a project and how these datasets are related to each other, and the primary dataset. When a relationships configuration is specified for a project, Feature Discovery will create features automatically from these datasets. - RelationshipsConfiguration.create creates a new relationships configuration between datasets - RelationshipsConfiguration.retrieve retrieve the requested relationships configuration - RelationshipsConfiguration.replace replace the relationships configuration details with new one - RelationshipsConfiguration.delete delete the relationships configuration ### Enhancements - Made creating projects from a dataset easier through the new Dataset.create_project . - These methods now provide additional metadata fields in Feature Impact results if called with with_metadata=True . Fields added: rowCount , shapBased , ranRedundancyDetection , count . - Model.get_feature_impact - Model.request_feature_impact - Model.get_or_request_feature_impact - Secondary dataset configuration retrieve and deletion is easier now though new SecondaryDatasetConfigurations.delete soft deletes a Secondary dataset configuration. SecondaryDatasetConfigurations.get retrieve a Secondary dataset configuration. - Retrieve relationships configuration which is applied on the given feature discovery project using Project.get_relationships_configuration . ### Bugfixes - An issue with input validation of the Batch Prediction module - parent_model_id was not visible for all frozen models - Batch Prediction jobs that used other output types than local_file failed when using .wait_for_completion() - A race condition in the Batch Prediction file scoring logic ### API changes - Three new fields were added to the Dataset object. This reflects the updated fields in the public API routes at api/v2/datasets/ . The added fields are: - processing_state: Current ingestion process state of the dataset - row_count: The number of rows in the dataset. - size: The size of the dataset as a CSV in bytes. ### Deprecation summary - datarobot.enums.VARIABLE_TYPE_TRANSFORM.CATEGORICAL for is deprecated for the following and will be removed in v2.22. - Project.batch_features_type_transform() - Project.create_type_transform_feature() ## 2.20.0 ### New features - There is a new Dataset object that implements some of the public API routes at api/v2/datasets/ . This also adds two new feature classes and a details class. - DatasetFeature - DatasetFeatureHistogram - DatasetDetails Functionality: - Create a Dataset by uploading from a file, URL or in-memory datasource. - [Dataset.create_from_file](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.create_from_file) - [Dataset.create_from_in_memory_data](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.create_from_in_memory_data) - [Dataset.create_from_url](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.create_from_url) - Get Datasets or elements of Dataset with: - [Dataset.list](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.list) lists available Datasets - [Dataset.get](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.get) gets a specified Dataset - [Dataset.update](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.get) updates the Dataset with the latest server information. - [Dataset.get_details](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.get_details) gets the DatasetDetails of the Dataset. - [Dataset.get_all_features](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.get_all_features) gets a list of the Dataset’s Features. - [Dataset.get_file](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.get_file) downloads the Dataset as a csv file. - [Dataset.get_projects](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.get_projects) gets a list of Projects that use the Dataset. - Modify, delete or un-delete a Dataset: - [Dataset.modify](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.modify) Changes the name and categories of the Dataset - [Dataset.delete](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.delete) soft deletes a Dataset. - [Dataset.un_delete](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.Dataset.un_delete) un-deletes the Dataset. You cannot retrieve the IDs of deleted Datasets, so if you want to un-delete a Dataset, you need to store its ID before deletion. - You can also create a Project using a `Dataset` with: - [Project.create_from_dataset](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project.create_from_dataset) - It is possible to create an alternative configuration for the secondary dataset which can be used during the prediction - [SecondaryDatasetConfigurations.create](https://docs.datarobot.com/en/docs/api/reference/sdk/data-registry.html#datarobot.models.SecondaryDatasetConfigurations.create) allow to create secondary dataset configuration - You can now filter the deployments returned by the [Deployment.list](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.list) command. You can do this by passing an instance of the [DeploymentListFilters](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.deployment.DeploymentListFilters) class to the `filters` keyword argument. The currently supported filters are: - `role` - `service_health` - `model_health` - `accuracy_health` - `execution_environment_type` - `materiality` - A new workflow is available for making predictions in time series projects. To that end, [PredictionDataset](https://docs.datarobot.com/en/docs/api/reference/sdk/batch-predictions.html#datarobot.models.PredictionDataset) objects now contain the following new fields: - `forecast_point_range`: The start and end date of the range of dates available for use as the forecast point, detected based on the uploaded prediction dataset - `data_start_date`: A datestring representing the minimum primary date of the prediction dataset - `data_end_date`: A datestring representing the maximum primary date of the prediction dataset - `max_forecast_date`: A datestring representing the maximum forecast date of this prediction dataset Additionally, users no longer need to specify a `forecast_point` or `predictions_start_date` and `predictions_end_date` when uploading datasets for predictions in time series projects. More information can be found in the :ref: `time series predictions` documentation. - Per-class lift chart data is now available for multiclass models using [Model.get_multiclass_lift_chart](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_multiclass_lift_chart). - Unsupervised projects can now be created using the [Project.start](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project.start) and [Project.set_target](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project.set_target) methods by providing `unsupervised_mode=True`, provided that the user has access to unsupervised machine learning functionality. Contact support for more information. - A new boolean attribute `unsupervised_mode` was added to [datarobot.DatetimePartitioningSpecification](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.DatetimePartitioningSpecification). When it is set to True, datetime partitioning for unsupervised time series projects will be constructed for nowcasting: `forecast_window_start=forecast_window_end=0`. - Users can now configure the start and end of the training partition as well as the end of the validation partition for backtests in a datetime-partitioned project. More information and example usage can be found in the * ref: `backtesting documentation `. ### Enhancements - Updated the user agent header to show which python version. - Model.get_frozen_child_models can be used to retrieve models that are frozen from a given model - Added datarobot.enums.TS_BLENDER_METHOD to make it clearer which blender methods are allowed for use in time series projects. ### Bugfixes - An issue where uploaded CSV’s would loose quotes during serialization causing issues when columns containing line terminators where loaded in a dataframe, has been fixed - Project.get_association_featurelists is now using the correct endpoint name, but the old one will continue to work - Python API PredictionServer supports now on-premise format of API response. ## 2.19.0 ### New features - Projects can be cloned using Project.clone_project - Calendars used in time series projects now support having series-specific events, for instance if a holiday only affects some stores. This can be controlled by using new argument of the CalendarFile.create method. If multiseries id columns are not provided, calendar is considered to be single series and all events are applied to all series. - We have expanded prediction intervals availability to the following use-cases: - Time series model deployments now support prediction intervals. See Deployment.get_prediction_intervals_settings and Deployment.update_prediction_intervals_settings for usage. - Prediction intervals are now supported for model exports for time series. To that end, a new optional parameter prediction_intervals_size has been added to Model.request_transferable_export . More details on prediction intervals can be found in the :ref: `prediction intervals documentation `. - Allowed pairwise interaction groups can now be specified in [AdvancedOptions](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.helpers.AdvancedOptions). They will be used in GAM models during training. - New deployments features: - Update the label and description of a deployment using [Deployment.update](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Deployment.update). - * ref: `Association ID setting` can be retrieved and updated. - Regression deployments now support :ref: `prediction warnings`. - For multiclass models now it’s possible to get feature impact for each individual target class using [Model.get_multiclass_feature_impact](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.Model.get_multiclass_feature_impact) - Added support for new :ref: `Batch Prediction API `. - It is now possible to create and retrieve basic, oauth and s3 credentials with [Credential](https://docs.datarobot.com/en/docs/api/dev-learning/python/admin/credentials.html#datarobot.models.Credential). - It’s now possible to get feature association statuses for featurelists using [Project.get_association_featurelists](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project.get_association_featurelists) - You can also pass a specific featurelist_id into [Project.get_associations](https://docs.datarobot.com/en/docs/api/reference/public-api/projects.html#datarobot.models.Project.get_associations) ### Enhancements - Added documentation to Project.get_metrics to detail the new ascending field that indicates how a metric should be sorted. - Retraining of a model is processed asynchronously and returns a ModelJob immediately. - Blender models can be retrained on a different set of data or a different feature list. - Word cloud ngrams now has variable field representing the source of the ngram. - Method WordCloud.ngrams_per_class can be used to split ngrams for better usability in multiclass projects. - Method Project.set_target support new optional parameters featureEngineeringGraphs and credentials . - Method Project.upload_dataset and Project.upload_dataset_from_data_source support new optional parameter credentials . - Series accuracy retrieval methods ( DatetimeModel.get_series_accuracy_as_dataframe and DatetimeModel.download_series_accuracy_as_csv ) for multiseries time series projects now support additional parameters for specifying what data to retrieve, including: - metric : Which metric to retrieve scores for - multiseries_value : Only returns series with a matching multiseries ID - order_by : An attribute by which to sort the results ### Bugfixes - An issue when using Feature.get and ModelingFeature.get to retrieve summarized categorical feature has been fixed. ### API changes - The datarobot package is now no longer a namespace package . - datarobot.enums.BLENDER_METHOD.FORECAST_DISTANCE is removed (deprecated in 2.18.0). ### Documentation changes - Updated :ref: Residuals charts documentation to reflect that the data rows include row numbers from the source dataset for projects created in DataRobot 5.3 and newer. ## 2.18.0 ### New features - - - - Deployment.submit_actuals can now be used to submit data about actual results from a deployed model, which can be used to calculate accuracy metrics. ### Enhancements - Monotonic constraints are now supported for OTV projects. To that end, the parameters monotonic_increasing_featurelist_id and monotonic_decreasing_featurelist_id can be specified in calls to Model.train_datetime or Project.train_datetime . - When retrieving information about features , information about summarized categorical variables is now available in a new keySummary . - For Word Clouds in multiclass projects, values of the target class for corresponding word or ngram can now be passed using the new class parameter. - Listing deployments using Deployment.list now support sorting and searching the results using the new order_by and search parameters. - You can now get the model associated with a model job by getting the model variable on the model job object . - The Blueprint class can now retrieve the recommended_featurelist_id , which indicates which feature list is recommended for this blueprint. If the field is not present, then there is no recommended feature list for this blueprint. - The Model class now can be used to retrieve the model_number . - The method Model.get_supported_capabilities now has an extra field supportsCodeGeneration to explain whether the model supports code generation. - Calls to Project.start and Project.upload_dataset now support uploading data via S3 URI and pathlib.Path objects. - Errors upon connecting to DataRobot are now clearer when an incorrect API Token is used. - The datarobot package is now a namespace package . ### Deprecation summary - datarobot.enums.BLENDER_METHOD.FORECAST_DISTANCE is deprecated and will be removed in 2.19. Use FORECAST_DISTANCE_ENET instead. ### Documentation changes - Various typo and wording issues have been addressed. - A new notebook showing regression-specific features is now been added to the examples_index. - Documentation for :ref: Access lists has been added. ## 2.17.0 ### New features - - Users can now list available prediction servers using PredictionServer.list . - When specifying datetime partitioning settings , :ref: time series projects can now mark individual features as excluded from feature derivation using the FeatureSettings.do_not_derive attribute. Any features not specified will be assigned according to the DatetimePartitioningSpecification.default_to_do_not_derive value. - Users can now submit multiple feature type transformations in a single batch request using Project.batch_features_type_transform . - - Information on feature clustering and the association strength between pairs of numeric or categorical features is now available. Project.get_associations can be used to retrieve pairwise feature association statistics and Project.get_association_matrix_details can be used to get a sample of the actual values used to measure association strength. ### Enhancements - number_of_do_not_derive_features has been added to the datarobot.DatetimePartitioning class to specify the number of features that are marked as excluded from derivation. - Users with PyYAML>=5.1 will no longer receive a warning when using the datarobot package - It is now possible to use files with unicode names for creating projects and prediction jobs. - Users can now embed DataRobot-generated content in a ComplianceDocTemplate using keyword tags. :ref: See here for more details. - The field calendar_name has been added to datarobot.DatetimePartitioning to display the name of the calendar used for a project. - - Previously, all backtests had to be run before :ref: prediction intervals for a time series project could be requested with predictions. Now, backtests will be computed automatically if needed when prediction intervals are requested. ### Bugfixes - An issue affecting time series project creation for irregularly spaced dates has been fixed. - ComplianceDocTemplate now supports empty text blocks in user sections. - An issue when using Predictions.get to retrieve predictions metadata has been fixed. ### Documentation changes - An overview on working with class ComplianceDocumentation and ComplianceDocTemplate has been created. :ref: See here for more details. ## 2.16.0 ### New features - Three new methods for Series Accuracy have been added to the DatetimeModel class. - Start a request to calculate Series Accuracy with DatetimeModel.compute_series_accuracy - Once computed, Series Accuracy can be retrieved as a pandas.DataFrame using DatetimeModel.get_series_accuracy_as_dataframe - Or saved as a CSV using DatetimeModel.download_series_accuracy_as_csv - Users can now access :ref: prediction intervals data for each prediction with a DatetimeModel . For each model, prediction intervals estimate the range of values DataRobot expects actual values of the target to fall within. They are similar to a confidence interval of a prediction, but are based on the residual errors measured during the backtesting for the selected model. ### Enhancements - Information on the effective feature derivation window is now available for :ref: time series projects to specify the full span of historical data required at prediction time. It may be longer than the feature derivation window of the project depending on the differencing settings used. Additionally, more of the project partitioning settings are also available on the [DatetimeModel](https://docs.datarobot.com/en/docs/api/reference/sdk/datarobot-models.html#datarobot.models.DatetimeModel) class. The new attributes are: - `effective_feature_derivation_window_start` - `effective_feature_derivation_window_end` - `forecast_window_start` - `forecast_window_end` - `windows_basis_unit` - Prediction metadata is now included in the return of [Predictions.get](https://docs.datarobot.com/en/docs/api/reference/sdk/deployment-management.html#datarobot.models.Predictions.get) ### Documentation changes - Various typo and wording issues have been addressed. - The example data that was meant to accompany the Time Series examples has been added to the zip file of the download in the examples_index. ## 2.15.1 ### Enhancements - CalendarFile.get_access_list has been added to the CalendarFile class to return a list of users with access to a calendar file. - A role attribute has been added to the CalendarFile class to indicate the access level a current user has to a calendar file. For more information on the specific access levels, see the :ref: sharing documentation. ### Bugfixes - Previously, attempting to retrieve the calendar_id of a project without a set target would result in an error. This has been fixed to return None instead. ## 2.15.0 ### New features - Previously available for only Eureqa models, Advanced Tuning methods and objects, including Model.start_advanced_tuning_session , Model.get_advanced_tuning_parameters , Model.advanced_tune , and AdvancedTuningSession , now support all models other than blender, open source, and user-created models. Use of Advanced Tuning via API for non-Eureqa models is in beta and not available by default, but can be enabled. - Calendar Files for time series projects can now be created and managed through the CalendarFile class. ### Enhancements - The dataframe returned from datarobot.PredictionExplanations.get_all_as_dataframe() will now have each class label class_X be the same from row to row. - The client is now more robust to networking issues by default. It will retry on more errors and respects Retry-After headers in HTTP 413, 429, and 503 responses. - Added Forecast Distance blender for Time-Series projects configured with more than one Forecast Distance. It blends the selected models creating separate linear models for each Forecast Distance. - Project can now be :ref: shared with other users. - Project.upload_dataset and Project.upload_dataset_from_data_source will return a PredictionDataset with data_quality_warnings if potential problems exist around the uploaded dataset. - relax_known_in_advance_features_check has been added to Project.upload_dataset and Project.upload_dataset_from_data_source to allow missing values from the known in advance features in the forecast window at prediction time. - cross_series_group_by_columns has been added to datarobot.DatetimePartitioning to allow users the ability to indicate how to further split series into related groups. - Information retrieval for ROC Curve has been extended to include fraction_predicted_as_positive , fraction_predicted_as_negative , lift_positive and lift_negative ### Bugfixes - Fixes an issue where the client would not be usable if it could not be sure it was compatible with the configured server ### API changes - Methods for creating datarobot.models.Project : create_from_mysql , create_from_oracle , and create_from_postgresql , deprecated in 2.11, have now been removed. Use datarobot.models.Project.create_from_data_source() instead. - datarobot.FeatureSettings attribute apriori , deprecated in 2.11, has been removed. Use datarobot.FeatureSettings.known_in_advance instead. - datarobot.DatetimePartitioning attribute default_to_a_priori , deprecated in 2.11, has been removed. Use datarobot.DatetimePartitioning.known_in_advance instead. - datarobot.DatetimePartitioningSpecification attribute default_to_a_priori , deprecated in 2.11, has been removed. Use datarobot.DatetimePartitioningSpecification.known_in_advance instead. ### Configuration changes - Now requires dependency on package requests to be at least version 2.21. - Now requires dependency on package urllib3 to be at least version 1.24. ### Documentation changes - Advanced model insights notebook extended to contain information on visualization of cumulative gains and lift charts. ## 2.14.2 ### Bugfixes - Fixed an issue where searches of the HTML documentation would sometimes hang indefinitely ### Documentation changes - Python3 is now the primary interpreter used to build the docs (this does not affect the ability to use the package with Python2) ## 2.14.1 ### Documentation changes - Documentation for the Model Deployment interface has been removed after the corresponding interface was removed in 2.13.0. ## 2.14.0 ### New features - The new method Model.get_supported_capabilities retrieves a summary of the capabilities supported by a particular model, such as whether it is eligible for Prime and whether it has word cloud data available. - New class for working with model compliance documentation feature of DataRobot: class ComplianceDocumentation - New class for working with compliance documentation templates: ComplianceDocTemplate - New class FeatureHistogram has been added to retrieve feature histograms for a requested maximum bin count - Time series projects now support binary classification targets. - Cross series features can now be created within time series multiseries projects using the use_cross_series_features and aggregation_type attributes of the datarobot.DatetimePartitioningSpecification . See the :ref: Time Series documentation for more info. ### Enhancements - Client instantiation now checks the endpoint configuration and provides more informative error messages. It also automatically corrects HTTP to HTTPS if the server responds with a redirect to HTTPS. - Project.upload_dataset and Project.create now accept an optional parameter of dataset_filename to specify a file name for the dataset. This is ignored for url and file path sources. - New optional parameter fallback_to_parent_insights has been added to Model.get_lift_chart , Model.get_all_lift_charts , Model.get_confusion_chart , Model.get_all_confusion_charts , Model.get_roc_curve , and Model.get_all_roc_curves . When True , a frozen model with missing insights will attempt to retrieve the missing insight data from its parent model. - New number_of_known_in_advance_features attribute has been added to the datarobot.DatetimePartitioning class. The attribute specifies number of features that are marked as known in advance. - Project.set_worker_count can now update the worker count on a project to the maximum number available to the user. - - Timeseries projects can now accept feature derivation and forecast windows intervals in terms of number of the rows rather than a fixed time unit. DatetimePartitioningSpecification and Project.set_target support new optional parameter windowsBasisUnit , either ‘ROW’ or detected time unit. - Timeseries projects can now accept feature derivation intervals, forecast windows, forecast points and prediction start/end dates in milliseconds. - DataSources and DataStores can now be :ref: shared with other users. - Training predictions for datetime partitioned projects now support the new data subset dr.enums.DATA_SUBSET.ALL_BACKTESTS for requesting the predictions for all backtest validation folds. ### API changes - The model recommendation type “Recommended” (deprecated in version 2.13.0) has been removed. ### Documentation changes - Example notebooks have been updated: - Notebooks now work in Python 2 and Python 3 - A notebook illustrating time series capability has been added - The financial data example has been replaced with an updated introductory example. - To supplement the embedded Python notebooks in both the PDF and HTML docs bundles, the notebook files and supporting data can now be downloaded from the HTML docs bundle. - Fixed a minor typo in the code sample for get_or_request_feature_impact ## 2.13.0 ### New features - The new method Model.get_or_request_feature_impact functionality will attempt to request feature impact and return the newly created feature impact object or the existing object so two calls are no longer required. - New methods and objects, including Model.start_advanced_tuning_session , Model.get_advanced_tuning_parameters , Model.advanced_tune , and AdvancedTuningSession , were added to support the setting of Advanced Tuning parameters. This is currently supported for Eureqa models only. - New is_starred attribute has been added to the Model class. The attribute specifies whether a model has been marked as starred by user or not. - Model can be marked as starred or being unstarred with Model.star_model and Model.unstar_model . - When listing models with Project.get_models , the model list can now be filtered by the is_starred value. - A custom prediction threshold may now be configured for each model via Model.set_prediction_threshold . When making predictions in binary classification projects, this value will be used when deciding between the positive and negative classes. - Project.check_blendable can be used to confirm if a particular group of models are eligible for blending as some are not, e.g. scaleout models and datetime models with different training lengths. - Individual cross validation scores can be retrieved for new models using Model.get_cross_validation_scores . ### Enhancements - Python 3.7 is now supported. - Feature impact now returns not only the impact score for the features but also whether they were detected to be redundant with other high-impact features. - A new is_blocked attribute has been added to the Job class, specifying whether a job is blocked from execution because one or more dependencies are not yet met. - The Featurelist object now has new attributes reporting its creation time, whether it was created by a user or by DataRobot, and the number of models using the featurelist, as well as a new description field. - Featurelists can now be renamed and have their descriptions updated with Featurelist.update and ModelingFeaturelist.update . - Featurelists can now be deleted with Featurelist.delete and ModelingFeaturelist.delete . - ModelRecommendation.get now accepts an optional parameter of type datarobot.enums.RECOMMENDED_MODEL_TYPE which can be used to get a specific kind of recommendation. - Previously computed predictions can now be listed and retrieved with the Predictions class, without requiring a reference to the original PredictJob . ### Bugfixes - The Model Deployment interface which was previously visible in the client has been removed to allow the interface to mature, although the raw API is available as a “beta” API without full backwards compatibility support. ### API changes - Added support for retrieving the Pareto Front of a Eureqa model. See ParetoFront . - A new recommendation type “Recommended for Deployment” has been added to ModelRecommendation which is now returns as the default recommended model when available. See :ref: model_recommendation . ### Deprecation summary - The feature previously referred to as “Reason Codes” has been renamed to “Prediction Explanations”, to provide increased clarity and accessibility. The old ReasonCodes interface has been deprecated and replaced with PredictionExplanations . - The recommendation type “Recommended” is deprecated and will no longer be returned in v2.14 of the API. ### Documentation changes - Added a new documentation section :ref: model_recommendation . - Time series projects support multiseries as well as single series data. They are now documented in the :ref: Time Series Projects documentation. ## 2.12.0 ### New features - Some models now have Missing Value reports allowing users with access to uncensored blueprints to retrieve a detailed breakdown of how numeric imputation and categorical converter tasks handled missing values. See the :ref: documentation for more information on the report. ## 2.11.0 ### New features - The new ModelRecommendation class can be used to retrieve the recommended models for a project. - A new helper method cross_validate was added to class Model. This method can be used to request Model’s Cross Validation score. - Training a model with monotonic constraints is now supported. Training with monotonic constraints allows users to force models to learn monotonic relationships with respect to some features and the target. This helps users create accurate models that comply with regulations (e.g. insurance, banking). Currently, only certain blueprints (e.g. xgboost) support this feature, and it is only supported for regression and binary classification projects. - DataRobot now supports “Database Connectivity”, allowing databases to be used as the source of data for projects and prediction datasets. The feature works on top of the JDBC standard, so a variety of databases conforming to that standard are available; a list of databases with tested support for DataRobot is available in the user guide in the web application. See :ref: Database Connectivity for details. - Added a new feature to retrieve feature logs for time series projects. Check datarobot.DatetimePartitioning.feature_log_list() and datarobot.DatetimePartitioning.feature_log_retrieve() for details. ### API changes - New attributes supporting monotonic constraints have been added to the AdvancedOptions , Project , Model , and Blueprint classes. See :ref: monotonic constraints for more information on how to configure monotonic constraints. - New parameters predictions_start_date and predictions_end_date added to Project.upload_dataset to support bulk predictions upload for time series projects. ### Deprecation summary - Methods for creating datarobot.models.Project : create_from_mysql , create_from_oracle , and create_from_postgresql , have been deprecated and will be removed in 2.14. Use datarobot.models.Project.create_from_data_source() instead. - datarobot.FeatureSettings attribute apriori , has been deprecated and will be removed in 2.14. Use datarobot.FeatureSettings.known_in_advance instead. - datarobot.DatetimePartitioning attribute default_to_a_priori , has been deprecated and will be removed in 2.14. datarobot.DatetimePartitioning.known_in_advance instead. - datarobot.DatetimePartitioningSpecification attribute default_to_a_priori , has been deprecated and will be removed in 2.14. Use datarobot.DatetimePartitioningSpecification.known_in_advance instead. ### Configuration changes - Retry settings compatible with those offered by urllib3’s Retry interface can now be configured. By default, we will now retry connection errors that prevented requests from arriving at the server. ### Documentation changes - “Advanced Model Insights” example has been updated to properly handle bin weights when rebinning. ## 2.9.0 ### New features - New ModelDeployment class can be used to track status and health of models deployed for predictions. ### Enhancements - DataRobot API now supports creating 3 new blender types - Random Forest, TensorFlow, LightGBM. - Multiclass projects now support blenders creation for 3 new blender types as well as Average and ENET blenders. - Models can be trained by requesting a particular row count using the new training_row_count argument with Project.train , Model.train and Model.request_frozen_model in non-datetime partitioned projects, as an alternative to the previous option of specifying a desired percentage of the project dataset. Specifying model size by row count is recommended when the float precision of sample_pct could be problematic, e.g. when training on a small percentage of the dataset or when training up to partition boundaries. - New attributes max_train_rows , scaleout_max_train_pct , and scaleout_max_train_rows have been added to Project . max_train_rows specified the equivalent value to the existing max_train_pct as a row count. The scaleout fields can be used to see how far scaleout models can be trained on projects, which for projects taking advantage of scalable ingest may exceed the limits on the data available to non-scaleout blueprints. - Individual features can now be marked as a priori or not a priori using the new feature_settings attribute when setting the target or specifying datetime partitioning settings on time series projects. Any features not specified in the feature_settings parameter will be assigned according to the default_to_a_priori value. - Three new options have been made available in the datarobot.DatetimePartitioningSpecification class to fine-tune how time-series projects derive modeling features. treat_as_exponential can control whether data is analyzed as an exponential trend and transformations like log-transform are applied. differencing_method can control which differencing method to use for stationary data. periodicities can be used to specify periodicities occurring within the data. All are optional and defaults will be chosen automatically if they are unspecified. ### API changes - Now training_row_count is available on non-datetime models as well as rowCount based datetime models. It reports the number of rows used to train the model (equivalent to sample_pct ). - Features retrieved from Feature.get now include target_leakage . ## 2.8.1 ### Bugfixes - The documented default connect_timeout will now be correctly set for all configuration mechanisms, so that requests that fail to reach the DataRobot server in a reasonable amount of time will now error instead of hanging indefinitely. If you observe that you have started seeing ConnectTimeout errors, please configure your connect_timeout to a larger value. - Version of trafaret library this package depends on is now pinned to trafaret>=0.7,<1.1 since versions outside that range are known to be incompatible. ## 2.8.0 ### New features - The DataRobot API supports the creation, training, and predicting of multiclass classification projects. DataRobot, by default, handles a dataset with a numeric target column as regression. If your data has a numeric cardinality of fewer than 11 classes, you can override this behavior to instead create a multiclass classification project from the data. To do so, use the set_target function, setting target_type=‘Multiclass’. If DataRobot recognizes your data as categorical, and it has fewer than 11 classes, using multiclass will create a project that classifies which label the data belongs to. - The DataRobot API now includes Rating Tables. A rating table is an exportable csv representation of a model. Users can influence predictions by modifying them and creating a new model with the modified table. See the :ref: documentation for more information on how to use rating tables. - scaleout_modeling_mode has been added to the AdvancedOptions class used when setting a project target. It can be used to control whether scaleout models appear in the autopilot and/or available blueprints. Scaleout models are only supported in the Hadoop environment with the corresponding user permission set. - A new premium add-on product, Time Series, is now available. New projects can be created as time series projects which automatically derive features from past data and forecast the future. See the - ref: time series documentation for more information. - The Feature object now returns the EDA summary statistics (i.e., mean, median, minimum, maximum, and standard deviation) for features where this is available (e.g., numeric, date, time, currency, and length features). These summary statistics will be formatted in the same format as the data it summarizes. - The DataRobot API now supports Training Predictions workflow. Training predictions are made by a model for a subset of data from original dataset. User can start a job which will make those predictions and retrieve them. See the :ref: documentation for more information on how to use training predictions. - DataRobot now supports retrieving a :ref: model blueprint chart and a - ref: model blueprint docs . - With the introduction of Multiclass Classification projects, DataRobot needed a better way to explain the performance of a multiclass model so we created a new Confusion Chart. The API now supports retrieving and interacting with confusion charts. ### Enhancements - DatetimePartitioningSpecification now includes the optional disable_holdout flag that can be used to disable the holdout fold when creating a project with datetime partitioning. - When retrieving reason codes on a project using an exposure column, predictions that are adjusted for exposure can be retrieved. - File URIs can now be used as source data when creating a project or uploading a prediction dataset. The file URI must refer to an allowed location on the server, which is configured as described in the user guide documentation. - The advanced options available when setting the target have been extended to include the new parameter ‘events_count’ as a part of the AdvancedOptions object to allow specifying the events count column. See the user guide documentation in the web app for more information on events count. - PredictJob.get_predictions now returns predicted probability for each class in the dataframe. - PredictJob.get_predictions now accepts prefix parameter to prefix the classes name returned in the predictions dataframe. ### API changes - Add target_type parameter to set_target() and start(), used to override the project default. ## 2.7.2 ### Documentation changes - Updated link to the publicly hosted documentation. ## 2.7.1 ### Documentation changes - Online documentation hosting has migrated from PythonHosted to Read The Docs. Minor code changes have been made to support this. ## 2.7.0 ### New features - Lift chart data for models can be retrieved using the Model.get_lift_chart and Model.get_all_lift_charts methods. - ROC curve data for models in classification projects can be retrieved using the Model.get_roc_curve and Model.get_all_roc_curves methods. - Semi-automatic autopilot mode is removed. - Word cloud data for text processing models can be retrieved using Model.get_word_cloud method. - Scoring code JAR file can be downloaded for models supporting code generation. ### Enhancements - A __repr__ method has been added to the PredictionDataset class to improve readability when using the client interactively. - Model.get_parameters now includes an additional key in the derived features it includes, showing the coefficients for individual stages of multistage models (e.g. Frequency-Severity models). - When training a DatetimeModel on a window of data, a time_window_sample_pct can be specified to take a uniform random sample of the training data instead of using all data within the window. - Installing of DataRobot package now has an “Extra Requirements” section that will install all of the dependencies needed to run the example notebooks. ### Documentation changes - A new example notebook describing how to visualize some of the newly available model insights including lift charts, ROC curves, and word clouds has been added to the examples section. - A new section for Common Issues has been added to Getting Started to help debug issues related to client installation and usage. ## 2.6.1 ### Bugfixes - Fixed a bug with Model.get_parameters raising an exception on some valid parameter values. ### Documentation changes - Fixed sorting order in Feature Impact example code snippet. ## 2.6.0 ### New features - A new partitioning method (datetime partitioning) has been added. The recommended workflow is to preview the partitioning by creating a DatetimePartitioningSpecification and passing it into DatetimePartitioning.generate , inspect the results and adjust as needed for the specific project dataset by adjusting the DatetimePartitioningSpecification and re-generating, and then set the target by passing the final DatetimePartitioningSpecification object to the partitioning_method parameter of Project.set_target . - When interacting with datetime partitioned projects, DatetimeModel can be used to access more information specific to models in datetime partitioned projects. See - ref: the documentation for more information on differences in the modeling workflow for datetime partitioned projects. - The advanced options available when setting the target have been extended to include the new parameters ‘offset’ and ‘exposure’ (part of the AdvancedOptions object) to allow specifying offset and exposure columns to apply to predictions generated by models within the project. See the user guide documentation in the web app for more information on offset and exposure columns. - Blueprints can now be retrieved directly by project_id and blueprint_id via Blueprint.get . - Blueprint charts can now be retrieved directly by project_id and blueprint_id via BlueprintChart.get . If you already have an instance of Blueprint you can retrieve its chart using Blueprint.get_chart . - Model parameters can now be retrieved using ModelParameters.get . If you already have an instance of Model you can retrieve its parameters using Model.get_parameters . - Blueprint documentation can now be retrieved using Blueprint.get_documents . It will contain information about the task, its parameters and (when available) links and references to additional sources. - The DataRobot API now includes Reason Codes. You can now compute reason codes for prediction datasets. You are able to specify thresholds on which rows to compute reason codes for to speed up computation by skipping rows based on the predictions they generate. See the reason codes - ref: documentation for more information. ### Enhancements - A new parameter has been added to the AdvancedOptions used with Project.set_target . By specifying accuracyOptimizedMb=True when creating AdvancedOptions , longer-running models that may have a high accuracy will be included in the autopilot and made available to run manually. - A new option for Project.create_type_transform_feature has been added which explicitly truncates data when casting numerical data as categorical data. - Added 2 new blenders for projects that use MAD or Weighted MAD as a metric. The MAE blender uses BFGS optimization to find linear weights for the blender that minimize mean absolute error (compared to the GLM blender, which finds linear weights that minimize RMSE), and the MAEL1 blender uses BFGS optimization to find linear weights that minimize MAE + a L1 penalty on the coefficients (compared to the ENET blender, which minimizes RMSE + a combination of the L1 and L2 penalty on the coefficients). ### Bugfixes - Fixed a bug (affecting Python 2 only) with printing any model (including frozen and prime models) whose model_type is not ascii. - FrozenModels were unable to correctly use methods inherited from Model. This has been fixed. - When calling get_result for a Job, ModelJob, or PredictJob that has errored, AsyncProcessUnsuccessfulError will now be raised instead of JobNotFinished , consistently with the behavior of get_result_when_complete . ### Deprecation summary - Support for the experimental Recommender Problems projects has been removed. Any code relying on RecommenderSettings or the recommender_settings argument of Project.set_target and Project.start will error. - Project.update , deprecated in v2.2.32, has been removed in favor of specific updates: rename , unlock_holdout , set_worker_count . ### Documentation changes - The link to Configuration from the Quickstart page has been fixed. ## 2.5.1 ### Bugfixes - Fixed a bug (affecting Python 2 only) with printing blueprints whose names are not ascii. - Fixed an issue where the weights column (for weighted projects) did not appear in the advanced_options of a Project . ## 2.5.0 ### New features - Methods to work with blender models have been added. Use Project.blend method to create new blenders, Project.get_blenders to get the list of existing blenders and BlenderModel.get to retrieve a model with blender-specific information. - Projects created via the API can now use smart downsampling when setting the target by passing smart_downsampled and majority_downsampling_rate into the AdvancedOptions object used with Project.set_target . The smart sampling options used with an existing project will be available as part of Project.advanced_options . - Support for frozen models, which use tuning parameters from a parent model for more efficient training, has been added. Use Model.request_frozen_model to create a new frozen model, Project.get_frozen_models to get the list of existing frozen models and FrozenModel.get to retrieve a particular frozen model. ### Enhancements - The inferred date format (e.g. “%Y-%m-%d %H:%M:%S”) is now included in the Feature object. For non-date features, it will be None. - When specifying the API endpoint in the configuration, the client will now behave correctly for endpoints with and without trailing slashes. ## 2.4.0 ### New features - The premium add-on product DataRobot Prime has been added. You can now approximate a model on the leaderboard and download executable code for it. See documentation for further details, or talk to your account representative if the feature is not available on your account. - (Only relevant for on-premise users with a Standalone Scoring cluster.) Methods ( request_transferable_export and download_export ) have been added to the Model class for exporting models (which will only work if model export is turned on). There is a new class ImportedModel for managing imported models on a Standalone Scoring cluster. - It is now possible to create projects from a WebHDFS, PostgreSQL, Oracle or MySQL data source. For more information see the documentation for the relevant Project classmethods: create_from_hdfs , create_from_postgresql , create_from_oracle and create_from_mysql . - Job.wait_for_completion , which waits for a job to complete without returning anything, has been added. ### Enhancements - The client will now check the API version offered by the server specified in configuration, and give a warning if the client version is newer than the server version. The DataRobot server is always backwards compatible with old clients, but new clients may have functionality that is not implemented on older server versions. This issue mainly affects users with on-premise deployments of DataRobot. ### Bugfixes - Fixed an issue where Model.request_predictions might raise an error when predictions finished very quickly instead of returning the job. ### API changes - To set the target with quickrun autopilot, call Project.set_target with mode=AUTOPILOT_MODE.QUICK instead of specifying quickrun=True . ### Deprecation summary - Semi-automatic mode for autopilot has been deprecated and will be removed in 3.0. Use manual or fully automatic instead. - Use of the quickrun argument in Project.set_target has been deprecated and will be removed in 3.0. Use mode=AUTOPILOT_MODE.QUICK instead. ### Configuration changes - It is now possible to control the SSL certificate verification by setting the parameter ssl_verify in the config file. ### Documentation changes - The “Modeling Airline Delay” example notebook has been updated to work with the new 2.3 enhancements. - Documentation for the generic Job class has been added. - Class attributes are now documented in the API Reference section of the documentation. - The changelog now appears in the documentation. - There is a new section dedicated to configuration, which lists all of the configuration options and their meanings. ## 2.3.0 ### New features - The DataRobot API now includes Feature Impact, an approach to measuring the relevance of each feature that can be applied to any model. The Model class now includes methods request_feature_impact (which creates and returns a feature impact job) and get_feature_impact (which can retrieve completed feature impact results). - A new improved workflow for predictions now supports first uploading a dataset via Project.upload_dataset , then requesting predictions via Model.request_predictions . This allows us to better support predictions on larger datasets and non-ascii files. - Datasets previously uploaded for predictions (represented by the PredictionDataset class) can be listed from Project.get_datasets and retrieve and deleted via PredictionDataset.get and PredictionDataset.delete . - You can now create a new feature by re-interpreting the type of an existing feature in a project by using the Project.create_type_transform_feature method. - The Job class now includes a get method for retrieving a job and a cancel method for canceling a job. - All of the jobs classes ( Job , ModelJob , PredictJob ) now include the following new methods: refresh (for refreshing the data in the job object), get_result (for getting the completed resource resulting from the job), and get_result_when_complete (which waits until the job is complete and returns the results, or times out). - A new method Project.refresh can be used to update Project objects with the latest state from the server. - A new function datarobot.async.wait_for_async_resolution can be used to poll for the resolution of any generic asynchronous operation on the server. ### Enhancements - The JOB_TYPE enum now includes FEATURE_IMPACT . - The QUEUE_STATUS enum now includes ABORTED and COMPLETED . - The Project.create method now has a read_timeout parameter which can be used to keep open the connection to DataRobot while an uploaded file is being processed. For very large files this time can be substantial. Appropriately raising this value can help avoid timeouts when uploading large files. - The method Project.wait_for_autopilot has been enhanced to error if the project enters a state where autopilot may not finish. This avoids a situation that existed previously where users could wait indefinitely on their project that was not going to finish. However, users are still responsible to make sure a project has more than zero workers, and that the queue is not paused. - Feature.get now supports retrieving features by feature name. (For backwards compatibility, feature IDs are still supported until 3.0.) - File paths that have unicode directory names can now be used for creating projects and PredictJobs. The filename itself must still be ascii, but containing directory names can have other encodings. - Now raises more specific JobAlreadyRequested exception when we refuse a model fitting request as a duplicate. Users can explicitly catch this exception if they want it to be ignored. - A file_name attribute has been added to the Project class, identifying the file name associated with the original project dataset. Note that if the project was created from a data frame, the file name may not be helpful. - The connect timeout for establishing a connection to the server can now be set directly. This can be done in the yaml configuration of the client, or directly in the code. The default timeout has been lowered from 60 seconds to 6 seconds, which will make detecting a bad connection happen much quicker. ### Bugfixes - Fixed a bug (affecting Python 2 only) with printing features and featurelists whose names are not ascii. ### API changes - Job class hierarchy is rearranged to better express the relationship between these objects. See documentation for datarobot.models.job for details. - Featurelist objects now have a project_id attribute to indicate which project they belong to. Directly accessing the project attribute of a Featurelist object is now deprecated - Support INI-style configuration, which was deprecated in v2.1, has been removed. yaml is the only supported configuration format. - The method Project.get_jobs method, which was deprecated in v2.1, has been removed. Users should use the Project.get_model_jobs method instead to get the list of model jobs. ### Deprecation summary - PredictJob.create has been deprecated in favor of the alternate workflow using Model.request_predictions . - Feature.converter (used internally for object construction) has been made private. - Model.fetch_resource_data has been deprecated and will be removed in 3.0. To fetch a model from its ID, use Model.get. - The ability to use Feature.get with feature IDs (rather than names) is deprecated and will be removed in 3.0. - Instantiating a Project , Model , Blueprint , Featurelist , or Feature instance from a dict of data is now deprecated. Please use the from_data classmethod of these classes instead. Additionally, instantiating a Model from a tuple or by using the keyword argument data is also deprecated. - Use of the attribute Featurelist.project is now deprecated. You can use the project_id attribute of a Featurelist to instantiate a Project instance using Project.get . - Use of the attributes Model.project , Model.blueprint , and Model.featurelist are all deprecated now to avoid use of partially instantiated objects. Please use the ids of these objects instead. - Using a Project instance as an argument in Featurelist.get is now deprecated. Please use a project_id instead. Similarly, using a Project instance in Model.get is also deprecated, and a project_id should be used in its place. ### Configuration changes - Previously it was possible (though unintended) that the client configuration could be mixed through environment variables, configuration files, and arguments to datarobot.Client . This logic is now simpler - please see the Getting Started section of the documentation for more information. ## 2.2.33 ### Bugfixes - Fixed a bug with non-ascii project names using the package with Python 2. - Fixed an error that occurred when printing projects that had been constructed from an ID only or printing printing models that had been constructed from a tuple (which impacted printing PredictJobs). - Fixed a bug with project creation from non-ascii file names. Project creation from non-ascii file names is not supported, so this now raises a more informative exception. The project name is no longer used as the file name in cases where we do not have a file name, which prevents non-ascii project names from causing problems in those circumstances. - Fixed a bug (affecting Python 2 only) with printing projects, features, and featurelists whose names are not ascii. ## 2.2.32 ### New features - Project.get_features and Feature.get methods have been added for feature retrieval. - A generic Job entity has been added for use in retrieving the entire queue at once. Calling Project.get_all_jobs will retrieve all (appropriately filtered) jobs from the queue. Those can be cancelled directly as generic jobs, or transformed into instances of the specific job class using ModelJob.from_job and PredictJob.from_job , which allow all functionality previously available via the ModelJob and PredictJob interfaces. - Model.train now supports featurelist_id and scoring_type parameters, similar to Project.train . ### Enhancements - Deprecation warning filters have been updated. By default, a filter will be added ensuring that usage of deprecated features will display a warning once per new usage location. In order to hide deprecation warnings, a filter like warnings.filterwarnings('ignore', category=DataRobotDeprecationWarning) can be added to a script so no such warnings are shown. Watching for deprecation warnings to avoid reliance on deprecated features is recommended. - If your client is misconfigured and does not specify an endpoint, the cloud production server is no longer used as the default as in many cases this is not the correct default. - This changelog is now included in the distributable of the client. ### Bugfixes - Fixed an issue where updating the global client would not affect existing objects with cached clients. Now the global client is used for every API call. - An issue where mistyping a filepath for use in a file upload has been resolved. Now an error will be raised if it looks like the raw string content for modeling or predictions is just one single line. ### API changes - Use of username and password to authenticate is no longer supported - use an API token instead. - Usage of start_time and finish_time parameters in Project.get_models is not supported both in filtering and ordering of models - Default value of sample_pct parameter of Model.train method is now None instead of 100 . If the default value is used, models will be trained with all of the available training data based on project configuration, rather than with entire dataset including holdout for the previous default value of 100 . - order_by parameter of Project.list which was deprecated in v2.0 has been removed. - recommendation_settings parameter of Project.start which was deprecated in v0.2 has been removed. - Project.status method which was deprecated in v0.2 has been removed. - Project.wait_for_aim_stage method which was deprecated in v0.2 has been removed. - Delay , ConstantDelay , NoDelay , ExponentialBackoffDelay , RetryManager classes from retry module which were deprecated in v2.1 were removed. - Package renamed to datarobot . ### Deprecation summary - Project.update deprecated in favor of specific updates: rename , unlock_holdout , set_worker_count . ### Documentation changes - A new use case involving financial data has been added to the examples directory. - Added documentation for the partition methods. ## 2.1.31 ### Bugfixes - In Python 2, using a unicode token to instantiate the client will now work correctly. ## 2.1.30 ### Bugfixes - The minimum required version of trafaret has been upgraded to 0.7.1 to get around an incompatibility between it and setuptools . ## 2.1.29 ### Enhancements - Minimal used version of requests_toolbelt package changed from 0.4 to 0.6 ## 2.1.28 ### New features - Default to reading YAML config file from ~/.config/datarobot/drconfig.yaml - Allow config_path argument to client - wait_for_autopilot method added to Project. This method can be used to block execution until autopilot has finished running on the project. - Support for specifying which featurelist to use with initial autopilot in Project.set_target - Project.get_predict_jobs method has been added, which looks up all prediction jobs for a project - Project.start_autopilot method has been added, which starts autopilot on specified featurelist - The schema for PredictJob in DataRobot API v2.1 now includes a message . This attribute has been added to the PredictJob class. - PredictJob.cancel now exists to cancel prediction jobs, mirroring ModelJob.cancel - Project.from_async is a new classmethod that can be used to wait for an async resolution in project creation. Most users will not need to know about it as it is used behind the scenes in Project.create and Project.set_target , but power users who may run into periodic connection errors will be able to catch the new ProjectAsyncFailureError and decide if they would like to resume waiting for async process to resolve ### Enhancements - AUTOPILOT_MODE enum now uses string names for autopilot modes instead of numbers ### Deprecation summary - ConstantDelay , NoDelay , ExponentialBackoffDelay , and RetryManager utils are now deprecated - INI-style config files are now deprecated (in favor of YAML config files) - Several functions in the utils submodule are now deprecated (they are being moved elsewhere and are not considered part of the public interface) - Project.get_jobs has been renamed Project.get_model_jobs for clarity and deprecated - Support for the experimental date partitioning has been removed in DataRobot API, so it is being removed from the client immediately. ### API changes - In several places where AppPlatformError was being raised, now TypeError , ValueError or InputNotUnderstoodError are now used. With this change, one can now safely assume that when catching an AppPlatformError it is because of an unexpected response from the server. - AppPlatformError has gained a two new attributes, status_code which is the HTTP status code of the unexpected response from the server, and error_code which is a DataRobot-defined error code. error_code is not used by any routes in DataRobot API 2.1, but will be in the future. In cases where it is not provided, the instance of AppPlatformError will have the attribute error_code set to None . - Two new subclasses of AppPlatformError have been introduced, ClientError (for 400-level response status codes) and ServerError (for 500-level response status codes). These will make it easier to build automated tooling that can recover from periodic connection issues while polling. - If a ClientError or ServerError occurs during a call to Project.from_async , then a ProjectAsyncFailureError (a subclass of AsyncFailureError) will be raised. That exception will have the status_code of the unexpected response from the server, and the location that was being polled to wait for the asynchronous process to resolve. ## 2.0.27 ### New features - PredictJob class was added to work with prediction jobs - wait_for_async_predictions function added to predict_job module ### Deprecation summary - The order_by parameter of the Project.list is now deprecated. ## 0.2.26 ### Enhancements - Project.set_target will re-fetch the project data after it succeeds, keeping the client side in sync with the state of the project on the server - Project.create_featurelist now throws DuplicateFeaturesError exception if passed list of features contains duplicates - Project.get_models now supports snake_case arguments to its order_by keyword ### Deprecation summary - Project.wait_for_aim_stage is now deprecated, as the REST Async flow is a more reliable method of determining that project creation has completed successfully - Project.status is deprecated in favor of Project.get_status - recommendation_settings parameter of Project.start is deprecated in favor of recommender_settings ### Bugfixes - Project.wait_for_aim_stage changed to support Python 3 - Fixed incorrect value of SCORING_TYPE.cross_validation - Models returned by Project.get_models will now be correctly ordered when the order_by keyword is used ## 0.2.25 - Pinned versions of required libraries ## 0.2.24 Official release of v0.2 ## 0.1.24 - Updated documentation - Renamed parameter name of Project.create and Project.start to project_name - Removed Model.predict method - wait_for_async_model_creation function added to modeljob module - wait_for_async_status_service of Project class renamed to _wait_for_async_status_service - Can now use auth_token in config file to configure API Client ## 0.1.23 - Fixes a method that pointed to a removed route ## 0.1.22 - Added featurelist_id attribute to ModelJob class ## 0.1.21 - Removes model attribute from ModelJob class ## 0.1.20 - Project creation raises AsyncProjectCreationError if it was unsuccessful - Removed Model.list_prime_rulesets and Model.get_prime_ruleset methods - Removed Model.predict_batch method - Removed Project.create_prime_model method - Removed PrimeRuleSet model - Adds backwards compatibility bridge for ModelJob async - Adds ModelJob.get and ModelJob.get_model ## 0.1.19 - Minor bugfixes in wait_for_async_status_service ## 0.1.18 - Removes submit_model from Project until server-side implementation is improved - Switches training URLs for new resource-based route at /projects/ /models/ - Job renamed to ModelJob, and using modelJobs route - Fixes an inconsistency in argument order for train methods ## 0.1.17 - wait_for_async_status_service timeout increased from 60s to 600s ## 0.1.16 - Project.create will now handle both async/sync project creation ## 0.1.15 - All routes pluralized to sync with changes in API - Project.get_jobs will request all jobs when no param specified - dataframes from predict method will have pythonic names - Project.get_status created, Project.status now deprecated - Project.unlock_holdout created. - Added quickrun parameter to Project.set_target - Added modelCategory to Model schema - Add permalinks feature to Project and Model objects. - Project.create_prime_model created ## 0.1.14 - Project.set_worker_count fix for compatibility with API change in project update. ## 0.1.13 - Add positive class to set_target . - Change attributes names of Project , Model , Job and Blueprint - features in Model , Job and Blueprint are now processes - dataset_id and dataset_name migrated to featurelist_id and featurelist_name . - samplepct -> sample_pct - Model has now blueprint , project , and featurelist attributes. - Minor bugfixes. ## 0.1.12 - Minor fixes regarding rename Job attributes. features attributes now named processes , samplepct now is sample_pct . ## 0.1.11 (May 27, 2015) - Minor fixes regarding migrating API from under_score names to camelCase. ## 0.1.10 (May 20, 2015) - Remove Project.upload_file , Project.upload_file_from_url and Project.attach_file methods. Moved all logic that uploading file to Project.create method. ## 0.1.9 (May 15, 2015) - Fix uploading file causing a lot of memory usage. Minor bugfixes. --- # R client changelog URL: https://docs.datarobot.com/en/docs/api/reference/changelogs/r-log.html > Reference the changes introduced to new versions of DataRobot's R client. Reference the changes introduced to new versions of DataRobot's R client. ## R client v2.18.7 Version v2.18.7 of the R client is now generally available. It can be accessed via [CRAN](https://cran.r-project.org/web/packages/datarobot/index.html) or [GitHub](https://github.com/datarobot/rsdk/releases/tag/v2.18.7). This is a maintenance release to ensure package compatibility with future versions of R and testthat. ### Enhancements - Test suite updated to replace the deprecated testthat::with_mock() with testthat::local_mocked_bindings() and testthat::with_mocked_bindings() . ## R client v2.18.6 Version v2.18.6 of the R client is now generally available. It can be accessed via [CRAN](https://cran.r-project.org/web/packages/datarobot/index.html) or [GitHub](https://github.com/datarobot/rsdk/releases/tag/v2.18.6). This is a maintenance release to ensure package compatibility with future versions of R. ### Bugfixes - Fixed a small issue with the metadata for the "Introduction to Multiclass" vignette. - Fixed some outstanding code formatting issues in various roxygen docs. ## R client v2.18.5 Version v2.18.5 of the R client is now generally available. It can be accessed via [CRAN](https://cran.r-project.org/web/packages/datarobot/index.html) or [GitHub](https://github.com/datarobot/rsdk/releases/tag/v2.18.5). This is a maintenance release. ### Bugfixes - The functions ListProjects now has NULL default values for limit and offset arguments to maintain backwards compatibility. This fixes compatibility issues with versions of DataRobot before 9.x. ## R client v2.18.4 Version v2.18.4 of the R client is now generally available. It can be accessed via [CRAN](https://cran.r-project.org/web/packages/datarobot/index.html) or [GitHub](https://github.com/datarobot/rsdk/releases/tag/v2.18.4). The `datarobot` package is now dependent on R >= 3.5. ### New features - The R client will now output a warning when you attempt to access certain resources (projects, models, deployments, etc.) that are deprecated or disabled by the DataRobot platform migration to Python 3. - Added support for comprehensive autopilot: usemode = AutopilotMode.Comprehensive. ### Enhancements - The functionRequestFeatureImpactnow accepts arowCountargument, which will change the sample size used for Feature Impact calculations. - The un-exported functiondatarobot:::UploadDatanow takes an optional argumentfileName. ### Bugfixes - Fixed an issue where an undocumented feature incurl==5.0.1is installed that caused any invocation ofdatarobot:::UploadData(i.e.,SetupProject) to fail with the errorNo method asJSON S3 class: form_file. - Loading thedatarobotpackage withsuppressPackageStartupMessages()will now suppress all messages. ### API changes - The functionsListProjectsandas.data.frame.projectSummaryListno longer return fields related to recommender models, which were removed in v2.5.0. - The functionSetTargetnow sets autopilot mode to Quick by default. Additionally, when Quick is passed, the underlying/aimendpoint will no longer be invoked with Auto. ### Deprecations - Thequickrunargument is removed from the functionSetTarget. Users should setmode = AutopilotMode.Quickinstead. - Compliance Documentation was deprecated in favor of the Automated Documentation API. ### Dependency changes - Thedatarobotpackage is now dependent on R >= 3.5 due to changes in the updated "Introduction to DataRobot" vignette. - Added dependency onAmesHousingpackage for updated "Introduction to DataRobot" vignette. - Removed dependency onMASSpackage. - Client documentation is now explicitly generated with Roxygen2 v7.2.3. ### Documentation changes - Updated the "Introduction to DataRobot" vignette to use Ames, Iowa housing data instead of the Boston housing dataset. ## R client v2.31 Version v2.31 of the R client is available for preview. It can be installed via [GitHub](https://github.com/datarobot/rsdk/releases/tag/v2.31.0.9000). This version of the R client addresses an issue where a new feature in the `curl==5.0.1` package caused any invocation of `datarobot:::UploadData` (i.e., `SetupProject`) to fail with the error `No method asJSON S3 class: form_file`. ### Enhancements The unexported function `datarobot:::UploadData` now takes an optional argument `fileName`. ### Bugfixes Loading the `datarobot` package with `suppressPackageStartupMessages()` will now suppress all messages. ### Deprecations - CreateProjectsDatetimeModelsFeatureFit has been removed. Use CreateProjectsDatetimeModelsFeatureEffects instead. - ListProjectsDatetimeModelsFeatureFit has been removed. Use ListProjectsDatetimeModelsFeatureEffects instead. - ListProjectsDatetimeModelsFeatureFitMetadata has been removed. Use ListProjectsDatetimeModelsFeatureEffectsMetadata instead. - CreateProjectsModelsFeatureFit has been removed. Use CreateProjectsModelsFeatureEffects instead. - ListProjectsModelsFeatureFit has been removed. Use ListProjectsModelsFeatureEffects instead. - ListProjectsModelsFeatureFitMetadata has been removed. Use ListProjectsModelsFeatureEffectsMetadata instead. ### Dependency changes Client documentation is now explicitly generated with Roxygen2 v7.2.3. Added Suggests: mockery to improve unit test development experience. --- # REST API changelogs URL: https://docs.datarobot.com/en/docs/api/reference/changelogs/rest-changelog/index.html > Reference the changes introduced to new versions of DataRobot's REST API. Changelogs contain curated, ordered lists of notable changes for each versioned release for [DataRobot's REST API client](https://docs.datarobot.com/en/docs/api/reference/public-api/index.html). Reference the changelog below to view changes for DataRobot's newest version, and view previous versions in the table of contents. ## v2.48 changelog Reference the changes introduced to version 2.48 of DataRobot's REST API. ### New features - Added endpoint for activity event descriptions:GET /api/v2/activityEventDescriptions/ - Added new endpoints for Custom Application Workload Migration. Migrate custom applications to a workload or revert them back to LRS hosting. The feature comprises the following endpoints: - Added new endpoints for File Refresh Jobs. Schedule, manage, and monitor refresh jobs for files in the data registry. The feature comprises the following endpoints: - Added new endpoints for Meta Memory Space Sharing. View and manage role-based access sharing for meta memory spaces. The feature comprises the following endpoints: - Added an endpoint for OCR job progress:GET /api/v2/ocrJobResources/{jobResourceId}/jobProgress/ - Added an endpoint for pipeline cloning:POST /api/v2/pipelines/{pipeline_id}/clone ### API changes - The new optional parameterownershiphas been added toGET /. - Description updated forGET /{memory_space_id}/. - Added response code 403 forPATCH /{memory_space_id}/. - Description updated forDELETE /{memory_space_id}/. - Parameter 'participants' (query) schema updated forGET /{memory_space_id}/sessions/. - The new optional parameterworkloadIdshas been added toGET /api/v2/agentCards/. - Parameter 'types' (query) schema updated forGET /api/v2/credentials/. - Added response code 403 forPOST /api/v2/credentials/. - Added response code 403 forPATCH /api/v2/credentials/{credentialId}/. - Parameter 'types' (query) schema updated forGET /api/v2/credentials/{credentialId}/associations/. - The new optional parameteruseCaseIdshas been added toGET /api/v2/datasets/. - Parameter 'event' (query) schema updated forGET /api/v2/eventLogs/. - Parameter 'types' (query) schema updated forPOST /api/v2/externalDataStores/{dataStoreId}/columnsInfo/. - Parameter 'types' (query) schema updated forGET /api/v2/externalDataStores/{dataStoreId}/credentials/. - Response 422 description updated forPOST /api/v2/ocrJobResources/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/logs/. - Parameter 'entityType' (path) schema updated forDELETE /api/v2/otel/{entityType}/{entityId}/logs/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/logs/podInfo/. - Parameter 'entityType' (path) schema updated forDELETE /api/v2/otel/{entityType}/{entityId}/metrics/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/autocollectedValues/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/configs/. - Parameter 'entityType' (path) schema updated forPOST /api/v2/otel/{entityType}/{entityId}/metrics/configs/. - Parameter 'entityType' (path) schema updated forPUT /api/v2/otel/{entityType}/{entityId}/metrics/configs/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/configs/{otelMetricId}/. - Parameter 'entityType' (path) schema updated forPATCH /api/v2/otel/{entityType}/{entityId}/metrics/configs/{otelMetricId}/. - Parameter 'entityType' (path) schema updated forDELETE /api/v2/otel/{entityType}/{entityId}/metrics/configs/{otelMetricId}/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/podInfo/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/summary/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/valueOverTime/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/values/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/values/segments/{segmentAttribute}/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/valuesOverTime/. - Parameter 'entityType' (path) schema updated forPOST /api/v2/otel/{entityType}/{entityId}/metrics/valuesOverTime/segments/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/valuesOverTime/segments/{segmentAttribute}/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/traces/. - Parameter 'entityType' (path) schema updated forDELETE /api/v2/otel/{entityType}/{entityId}/traces/. - Parameter 'entityType' (path) schema updated forGET /api/v2/otel/{entityType}/{entityId}/traces/{traceId}/. - Description updated forPOST /api/v2/pipelines/images. - Description updated forPATCH /api/v2/pipelines/images/{image_id}. - Description updated forPATCH /api/v2/pipelines/{pipeline_id}. - The new optional parameternodeIdhas been added toGET /api/v2/pipelines/{pipeline_id}/dispatches/{dispatch_id}/tasks/{task_id}. - The new optional parameternodeIdhas been added toGET /api/v2/pipelines/{pipeline_id}/dispatches/{dispatch_id}/tasks/{task_id}/logs. - The new optional parameternodeIdhas been added toGET /api/v2/pipelines/{pipeline_id}/dispatches/{dispatch_id}/tasks/{task_id}/logs/{stream}. - The new optional parameternodeIdhas been added toGET /api/v2/pipelines/{pipeline_id}/dispatches/{dispatch_id}/tasks/{task_id}/result. - Added response code 403 forPOST /api/v2/secureConfigs/. - Added response code 403 forPATCH /api/v2/secureConfigs/{secureConfigId}/. - Parameter 'workloadCategory' (query) schema updated forGET /api/v2/tenantUsageResources/. - Parameter 'workloadCategory' (query) schema updated forGET /api/v2/tenantUsageResources/deployments/. - Parameter 'workloadCategory' (query) schema updated forGET /api/v2/tenantUsageResources/export/. - Parameter 'workloadCategory' (query) schema updated forGET /api/v2/tenants/{tenantId}/usage/. - Parameter 'workloadCategory' (query) schema updated forGET /api/v2/tenants/{tenantId}/usageExport/. - Summary changed from 'Retrieve the list of use cases.' to 'Retrieve the list of Use Cases.' forGET /api/v2/useCases/. - Summary changed from 'Get a use case.' to 'Get a Use Case.' forPOST /api/v2/useCases/. - Description updated forGET /api/v2/useCases/notebooks/. - Summary changed from 'Get a use case by use case ID' to 'Get a Use Case by use case ID' forGET /api/v2/useCases/{useCaseId}/. - Response 404 description updated forPATCH /api/v2/useCases/{useCaseId}/. - Summary changed from 'Delete a use case by use case ID' to 'Delete a Use Case by use case ID' forDELETE /api/v2/useCases/{useCaseId}/. - Summary changed from 'The list of the custom applications referenced by a use case by use case ID' to 'The list of the custom applications referenced by a Use Case by use case ID' forGET /api/v2/useCases/{useCaseId}/customApplications/. - Description updated forGET /api/v2/useCases/{useCaseId}/deployments/. - Description updated forPOST /api/v2/useCases/{useCaseId}/multilink/. - Description updated forGET /api/v2/useCases/{useCaseId}/notebooks/. - Summary changed from 'The list of the registered models referenced by a use case by use case ID' to 'The list of the registered models referenced by a Use Case by use case ID' forGET /api/v2/useCases/{useCaseId}/registeredModels/. - Description updated forGET /api/v2/useCases/{useCaseId}/resources/. - Summary changed from 'Get the use case's access control list by use case ID' to 'Get the Use Case's access control list by use case ID' forGET /api/v2/useCases/{useCaseId}/sharedRoles/. - Description updated forPOST /api/v2/useCases/{useCaseId}/{referenceCollectionType}/{entityId}/. - Parameter 'referenceCollectionType' (path) schema updated forPATCH /api/v2/useCases/{useCaseId}/{referenceCollectionType}/{entityId}/. - Description updated forDELETE /api/v2/useCases/{useCaseId}/{referenceCollectionType}/{entityId}/. - Description updated forGET /api/v2/useCasesWithShortenedInfo/. - The new optional parameterenclaveIdhas been added toGET /api/v2/workloads. - The new optional parameterenclaveIdhas been added toGET /api/v2/workloads/stats. ## v2.47 changelog Reference the changes introduced to version 2.47 of DataRobot's REST API. ### New features - Added new endpoints for Custom Scoring Metrics. Manage custom scoring metrics and their versioned file contents, including creation, retrieval, update, deletion, and version uploads. The feature comprises the following endpoints: - Added new endpoints for Inbound OAuth (SSO) Configuration. Create, retrieve, and update global and per-organization Inbound OAuth configuration for external application SSO. The feature comprises the following endpoints: - Added endpoint for file bulk actions:PATCH /api/v2/files/ - Added new endpoints for Pipeline Management. Create, retrieve, update, delete, and manage versions, graphs, tasks, and source definitions for pipelines. The feature comprises the following endpoints: - Added new endpoints for Pipeline Execution Images. Manage container images used for pipeline task execution, including creating, updating, deleting images and versions, and retrieving build logs. The feature comprises the following endpoints: - Added new endpoints for Pipeline Dispatches. Create, monitor, cancel, and retrieve logs and results for pipeline run dispatches across both draft and locked pipeline versions. The feature comprises the following endpoints: - Added new endpoints for Pipeline Inputs. Manage input definitions for both draft and locked pipeline versions, including creation, retrieval, update, and deletion. The feature comprises the following endpoints: - Added new endpoints for Pipeline Schedules. Create, retrieve, update, delete, and trigger scheduled executions for pipelines. The feature comprises the following endpoints: - Added new endpoints for Pipeline Internal Task Callbacks. Internal callbacks for notifying the pipeline orchestrator when a task graph is ready or a task execution completes. The feature comprises the following endpoints: - Added new endpoints for SCIM User & Group Provisioning. Provision, update, and deprovision users and groups via the SCIM protocol for identity-provider-managed organizations. The feature comprises the following endpoints: - Added new endpoints for SCIM Configuration & Metadata. Retrieve and update SCIM service provider configuration, supported schemas, resource types, and per-org SCIM settings. The feature comprises the following endpoints: - Added endpoint for tenant usage resources:GET /api/v2/tenantUsageResources/deployments/ - Added endpoint for health check:GET /health ## v2.46 changelog Reference the changes introduced to version 2.46 of DataRobot's REST API. ### New features - Added new endpoints to the REST API: - Added new endpoints for File Management. Endpoints for file upload, download, and management. The feature is comprised of the following endpoints: - Added endpoint for tenant management:GET /api/v2/tenantUsageResources/usageOverTime/ - Added new endpoints for User Management. Endpoints for user, role, and permission management. The feature is comprised of the following endpoints: ### API changes - Parameter 'datasourceType' schema updated forGET /api/v2/catalogItems/. - Summary changed from 'Check if a license is valid' to 'Check if a license is valid.' forPOST /api/v2/clusterLicenseValidation/. - Description updated forPATCH /api/v2/credentials/{credentialId}/associations/. - Parameter 'eventGroup' schema updated forGET /api/v2/entityNotificationPolicies/{relatedEntityType}/{relatedEntityId}/. - Parameter 'eventGroup' schema updated forGET /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/. - Parameter 'event' schema updated forGET /api/v2/eventLogs/. - Summary changed from 'List pods and containers found by entitytype' to 'Retrieve pod info by id' forGET /api/v2/otel/{entityType}/{entityId}/metrics/podInfo/. - The new optional parameteroffsethas been added toGET /api/v2/otel/{entityType}/{entityId}/traces/{traceId}/. - Added response code 409 forPOST /new/. - The new optional parameterdeduplicationKeyhas been added toGET /{memory_space_id}/sessions/. - Added response code 409 forPOST /{memory_space_id}/sessions/. - Added response code 409 forPOST /{memory_space_id}/v1/memories/. - Response 422 description updated forPOST /{memory_space_id}/v2/memories/. - Response 422 description updated forPOST /{memory_space_id}/v2/memories/search/. - Parameter 'eventGroup' schema updated forGET /api/v2/notificationPolicies/. - Added response code 201 forPOST /api/v2/notificationPolicyMutes/. - Added response code 204 forDELETE /api/v2/notificationPolicyMutes/{muteId}/. - Response 200 content schema updated forGET /api/v2/projectCleanupJobs/{statusId}/download/. - Parameter 'workloadCategory' schema updated forGET /api/v2/tenantUsageResources/. - Parameter 'workloadCategory' schema updated forGET /api/v2/tenantUsageResources/export/. - Parameter 'workloadCategory' schema updated forGET /api/v2/tenants/{tenantId}/usage/. - Parameter 'workloadCategory' schema updated forGET /api/v2/tenants/{tenantId}/usageExport/. - Parameter 'entityType' schema updated forGET /api/v2/useCases/. - Parameter 'referenceCollectionType' schema updated forPOST /api/v2/useCases/{useCaseId}/{referenceCollectionType}/{entityId}/. - Parameter 'referenceCollectionType' schema updated forPATCH /api/v2/useCases/{useCaseId}/{referenceCollectionType}/{entityId}/. - Parameter 'referenceCollectionType' schema updated forDELETE /api/v2/useCases/{useCaseId}/{referenceCollectionType}/{entityId}/. - Parameter 'entityType' schema updated forGET /api/v2/useCasesWithShortenedInfo/. - Summary changed from 'Get a list of models from projects associated with a Use Case by use case ID' to 'Gets a list of models from projects associated with a Use Case by use case ID' forGET /api/v2/useCases/{useCaseId}/modelsForComparison/. ### API changes - Added response code 429 forGET /. - Added response code 429 forPOST /new/. - Added response code 429 forGET /{memory_space_id}/. - Added response code 429 forPATCH /{memory_space_id}/. - Added response code 429 forGET /{memory_space_id}/sessions/. - Added response code 429 forPOST /{memory_space_id}/sessions/. - Added response code 429 forGET /{memory_space_id}/sessions/{session_id}/. - Added response code 429 forPATCH /{memory_space_id}/sessions/{session_id}/. - Added response code 429 forGET /{memory_space_id}/sessions/{session_id}/events/. - Added response code 429 forPOST /{memory_space_id}/sessions/{session_id}/events/. - Added response code 429 forPOST /{memory_space_id}/sessions/{session_id}/events/batch/. - Added response code 429 forPATCH /{memory_space_id}/sessions/{session_id}/events/batch/. - Added response code 429 forPATCH /{memory_space_id}/sessions/{session_id}/events/{sequence_id}/. - Added response code 429 forGET /{memory_space_id}/v1/memories/. - Added response code 429 forPOST /{memory_space_id}/v1/memories/. - Added response code 429 forPOST /{memory_space_id}/v1/memories/search/. - Added response code 429 forGET /{memory_space_id}/v1/memories/{memory_id}/. - Added response code 429 forPUT /{memory_space_id}/v1/memories/{memory_id}/. - Added response code 429 forGET /{memory_space_id}/v1/memories/{memory_id}/history/. - Added response code 429 forPOST /{memory_space_id}/v2/memories/. - Added response code 429 forPOST /{memory_space_id}/v2/memories/search/. - Parameter 'typeId' (query) schema updated forGET /api/v2/scheduledJobs/. - Description updated forGET /api/v2/useCases/{useCaseId}/datasets/. - Response 200 content schema updated forGET /api/v2/useCases/{useCaseId}/filterMetadata/. - Description updated forGET /api/v2/useCases/{useCaseId}/projects/. ## v2.45 changelog Reference the changes introduced to version 2.45 of DataRobot's REST API. ### New features - Added new endpoints for General API. General API endpoints. The feature is comprised of the following endpoints: - Added new endpoints for File Management. Endpoints for file upload, download, and management. The feature is comprised of the following endpoints: ### API changes - Parameter 'entityType' schema updated forGET /api/v2/comments/{entityType}/{entityId}/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/complianceDocTemplates/{templateId}/sharedRoles/. - Parameter 'types' schema updated forGET /api/v2/credentials/. - Parameter 'types' schema updated forGET /api/v2/credentials/{credentialId}/associations/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/customApplicationSources/{appSourceId}/sharedRoles/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/customApplications/{applicationId}/sharedRoles/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/customJobs/{customJobId}/sharedRoles/. - Parameter 'targetType' schema updated forGET /api/v2/customModels/. - Parameter 'targetTypes' schema updated forGET /api/v2/customTrainingBlueprints/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/{policyId}/sharedRoles/. - Parameter 'event' schema updated forGET /api/v2/eventLogs/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/externalDataSources/{dataSourceId}/sharedRoles/. - Parameter 'types' schema updated forPOST /api/v2/externalDataStores/{dataStoreId}/columnsInfo/. - Parameter 'types' schema updated forGET /api/v2/externalDataStores/{dataStoreId}/credentials/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/externalDataStores/{dataStoreId}/sharedRoles/. - The new optional parametergroupIdshas been added toGET /api/v2/groups/. - The new optional parametersearchKeyshas been added toGET /api/v2/otel/{entityType}/{entityId}/logs/. - The new optional parametersearchKeyshas been added toDELETE /api/v2/otel/{entityType}/{entityId}/logs/. - The new optional parametersearchKeyshas been added toDELETE /api/v2/otel/{entityType}/{entityId}/metrics/. - The new optional parametersearchKeyshas been added toGET /api/v2/otel/{entityType}/{entityId}/metrics/autocollectedValues/. - Parameter 'entityType' schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/configs/. - Parameter 'entityType' schema updated forPOST /api/v2/otel/{entityType}/{entityId}/metrics/configs/. - Parameter 'entityType' schema updated forPUT /api/v2/otel/{entityType}/{entityId}/metrics/configs/. - Parameter 'entityType' schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/configs/{otelMetricId}/. - Parameter 'entityType' schema updated forPATCH /api/v2/otel/{entityType}/{entityId}/metrics/configs/{otelMetricId}/. - Parameter 'entityType' schema updated forDELETE /api/v2/otel/{entityType}/{entityId}/metrics/configs/{otelMetricId}/. - The new optional parametersearchKeyshas been added toGET /api/v2/otel/{entityType}/{entityId}/metrics/podInfo/. - Parameter 'entityType' schema updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/summary/. - The new optional parametersearchKeyshas been added toGET /api/v2/otel/{entityType}/{entityId}/metrics/valueOverTime/. - The new optional parametersearchKeyshas been added toGET /api/v2/otel/{entityType}/{entityId}/metrics/values/. - The new optional parametersearchKeyshas been added toGET /api/v2/otel/{entityType}/{entityId}/metrics/values/segments/{segmentAttribute}/. - The new optional parametersearchKeyshas been added toGET /api/v2/otel/{entityType}/{entityId}/metrics/valuesOverTime/. - Parameter 'entityType' schema updated forPOST /api/v2/otel/{entityType}/{entityId}/metrics/valuesOverTime/segments/. - The new optional parametersearchKeyshas been added toGET /api/v2/otel/{entityType}/{entityId}/metrics/valuesOverTime/segments/{segmentAttribute}/. - The new optional parametertraceTypehas been added toGET /api/v2/otel/{entityType}/{entityId}/traces/. - The new optional parametersearchKeyshas been added toDELETE /api/v2/otel/{entityType}/{entityId}/traces/. - Parameter 'entityType' schema updated forGET /api/v2/otel/{entityType}/{entityId}/traces/{traceId}/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/registeredModels/{registeredModelId}/sharedRoles/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/secureConfigs/{secureConfigId}/sharedRoles/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/valueTrackers/{valueTrackerId}/sharedRoles/. - The new optional parameterIf-Matchhas been added toPATCH /{memory_space_id}/sessions/{session_id}/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/datasets/{datasetId}/sharedRoles/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/deployments/{deploymentId}/sharedRoles/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/modelPackages/{modelPackageId}/sharedRoles/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/notificationChannelTemplates/{channelId}/sharedRoles/. - Parameter 'offset' description updated forGET /api/v2/useCases/notebooks/. - Parameter 'offset' description updated forGET /api/v2/useCases/{useCaseId}/notebooks/. - Parameter 'offset' description updated forGET /api/v2/useCases/{useCaseId}/playgrounds/. - Parameter 'offset' description updated forGET /api/v2/useCases/{useCaseId}/vectorDatabases/{vectorDatabaseId}/relatedCustomModels/. - Parameter 'targetType' schema updated forGET /api/v2/useCases/{useCaseId}/vectorDatabases/{vectorDatabaseId}/relatedDeployments/. - Parameter 'offset' description updated forGET /api/v2/useCases/{useCaseId}/vectorDatabases/{vectorDatabaseId}/relatedRegisteredModels/. - Parameter 'shareRecipientType' schema updated forGET /api/v2/userBlueprints/{userBlueprintId}/sharedRoles/. - Added response code 204 forDELETE /api/v2/userCleanupPreviews/{reportId}/. - Added response code 204 forDELETE /api/v2/userCleanupSummaries/{reportId}/. - Response 200 content schema updated forGET /api/v2/users/. - Response 202 content schema updated forPOST /api/v2/users/invite/. - Response 200 content schema updated forGET /api/v2/users/{userId}/. ### Deprecation summary - The API endpoint GET /api/v2/tracing/{entityType}/{entityId}/ is deprecated. - The API endpoint GET /api/v2/tracing/{entityType}/{entityId}/{traceId}/ is deprecated. - The API endpoint GET /health/live/ is deprecated. - The API endpoint GET /health/ready/ is deprecated. ## v2.44 changelog Reference the changes introduced to version 2.44 of DataRobot's REST API. ### New features - GET /api/v2/customApplications/{applicationId}/usages/download/ - GET /api/v2/externalOAuth/authorizedProviders/ - GET /api/v2/genai/promptTemplates/versions/ - GET /api/v2/genai/syftrSearch/ - POST /api/v2/genai/syftrSearch/ - GET /api/v2/genai/syftrSearch/{searchStudyId}/ - PATCH /api/v2/genai/syftrSearch/{searchStudyId}/ - DELETE /api/v2/genai/syftrSearch/{searchStudyId}/ - GET /health/live/ - GET /health/ready/ - POST /new/ - GET /{memory_space_id}/ - PATCH /{memory_space_id}/ - DELETE /{memory_space_id}/ - GET /{memory_space_id}/sessions/ - POST /{memory_space_id}/sessions/ - GET /{memory_space_id}/sessions/{session_id}/ - PATCH /{memory_space_id}/sessions/{session_id}/ - DELETE /{memory_space_id}/sessions/{session_id}/ - GET /{memory_space_id}/sessions/{session_id}/events/ - POST /{memory_space_id}/sessions/{session_id}/events/ - PATCH /{memory_space_id}/sessions/{session_id}/events/{sequence_id}/ - GET /{memory_space_id}/v1/memories/ - POST /{memory_space_id}/v1/memories/ - DELETE /{memory_space_id}/v1/memories/ - POST /{memory_space_id}/v1/memories/search/ - GET /{memory_space_id}/v1/memories/{memory_id}/ - PUT /{memory_space_id}/v1/memories/{memory_id}/ - DELETE /{memory_space_id}/v1/memories/{memory_id}/ - GET /{memory_space_id}/v1/memories/{memory_id}/history/ - GET /{memory_space_id}/v1/ping/ - POST /{memory_space_id}/v1/reset/ - POST /{memory_space_id}/v2/memories/ - POST /{memory_space_id}/v2/memories/search/ - Added new endpoints for Deployments. Endpoints for model deployment and management. The feature is comprised of the following endpoints: - Added endpoint for file management:GET /api/v2/files/{catalogId}/versions/ ### API changes - Parameter 'entityType' schema updated forGET /api/v2/comments/{entityType}/{entityId}/. - Parameter 'types' schema updated forGET /api/v2/credentials/. - Parameter 'types' schema updated forGET /api/v2/credentials/{credentialId}/associations/. - The new optional parametercreatedByhas been added toGET /api/v2/customApplicationSources/. - The new optional parametercreatedByhas been added toGET /api/v2/customApplications/. - The new optional parameterstarthas been added toGET /api/v2/customApplications/{applicationId}/usages/. - Parameter 'event' schema updated forGET /api/v2/eventLogs/. - Parameter 'types' schema updated forPOST /api/v2/externalDataStores/{dataStoreId}/columnsInfo/. - Parameter 'types' schema updated forGET /api/v2/externalDataStores/{dataStoreId}/credentials/. - The new optional parameterchatCompletionsSupportedOnlyhas been added toGET /api/v2/genai/llms/. - The new optional parameteroffsethas been added toGET /api/v2/otel/{entityType}/{entityId}/traces/. - The new optional parameterconsumerhas been added toGET /api/v2/secureConfigs/{secureConfigId}/values/. - The new optional parameteroffsethas been added toGET /api/v2/tracing/{entityType}/{entityId}/. - The new optional parameterisA2AAgenthas been added toGET /api/v2/deployments/. - Summary changed from 'Retrieve LLM API call count' to '[DEPRECATED] Retrieve LLM API call count' forGET /api/v2/genai/userLimits/llmApiCalls/. - The new optional parameterquickComputehas been added toGET /api/v2/insights/shapImpact/models/{entityId}/. - Parameter 'workloadCategory' schema updated forGET /api/v2/tenantUsageResources/. - Parameter 'workloadCategory' schema updated forGET /api/v2/tenantUsageResources/export/. - Parameter 'workloadCategory' schema updated forGET /api/v2/tenants/{tenantId}/usage/. - Parameter 'workloadCategory' schema updated forGET /api/v2/tenants/{tenantId}/usageExport/. - Parameter 'entityType' schema updated forGET /api/v2/useCases/. - Description updated forGET /api/v2/useCases/{useCaseId}/applications/. - Parameter 'offset' description updated forGET /api/v2/useCases/{useCaseId}/sharedRoles/. - Parameter 'referenceCollectionType' schema updated forPOST /api/v2/useCases/{useCaseId}/{referenceCollectionType}/{entityId}/. - Parameter 'referenceCollectionType' schema updated forPATCH /api/v2/useCases/{useCaseId}/{referenceCollectionType}/{entityId}/. - Parameter 'referenceCollectionType' schema updated forDELETE /api/v2/useCases/{useCaseId}/{referenceCollectionType}/{entityId}/. - Parameter 'entityType' schema updated forGET /api/v2/useCasesWithShortenedInfo/. ## v2.43 changelog Reference for the changes introduced to version 2.43 of DataRobot's REST API. ### New features - Added new endpoints for the REST API: ## v2.42 changelog Reference for the changes introduced to version 2.42 of DataRobot's REST API. ### New features - Added new endpoints for general API usage: - Added an endpoint for limits on deployments:GET /api/v2/deployments/limits/ - Added new endpoints for File Management. Use the endpoints for file uploads, downloads, and management. The feature is comprised of the following endpoints: - Added an endpoint for insights & analytics:POST /api/v2/insights/confusionMatrix/ - Added an endpoint for confusion matrices:GET /api/v2/insights/confusionMatrix/models/{entityId}/ ### API changes - ParameterintakeTypeschema updated forGET /api/v2/batchJobs/. - ParameterintakeTypeschema updated forGET /api/v2/batchPredictions/. - ParameterhardDeleteschema updated forDELETE /api/v2/customApplicationSources/{appSourceId}/. - ParameterhardDeleteschema updated forDELETE /api/v2/customApplications/{applicationId}/. - Parameterreplicasschema updated forGET /api/v2/customModelTests/. - Description updated forGET /api/v2/customTasks/. - Description updated forPOST /api/v2/customTasks/. - Parametereventschema updated forGET /api/v2/eventLogs/. - Parameteridsschema updated forGET /api/v2/externalOAuth/providers/. - Removed parameterskip_consent(query) forPOST /api/v2/externalOAuth/providers/{providerId}/authorize/. - ParameteruseCasesschema updated forGET /api/v2/mlops/compute/bundles/. - Description updated forPOST /api/v2/deployments/fromLearningModel/. - Removed parameterhardDelete(query) forDELETE /api/v2/executionEnvironments/{environmentId}/. - Description updated forGET /api/v2/tenantUsageResources/. - Description updated forGET /api/v2/tenantUsageResources/activeUsers/. - Description updated forGET /api/v2/tenantUsageResources/categories/. - Description updated forGET /api/v2/tenantUsageResources/export/. - Description updated forGET /api/v2/tenants/{tenantId}/resourceCategories/. - ParameterbinarySortMetricschema updated forGET /api/v2/useCases/{useCaseId}/modelsForComparison/. ### Deprecation summary - The API endpoint POST /api/v2/externalOAuth/authorizedProviders/{authorizedProviderId}/accessToken/ is deprecated. ## v2.41 changelog Reference for the changes introduced to version 2.41 of DataRobot's REST API. ### New features - Added an endpoint for user invitations: POST /api/v2/users/invite/ ### API changes - Parameter 'types' schema updated forGET /api/v2/credentials/. - Parameter 'types' schema updated forGET /api/v2/credentials/{credentialId}/associations/. - Parameter 'relatedEntityType' schema updated forGET /api/v2/entityNotificationChannels/{relatedEntityType}/{relatedEntityId}/. - Parameter 'relatedEntityType' schema updated forGET /api/v2/entityNotificationChannels/{relatedEntityType}/{relatedEntityId}/{channelId}/. - Parameter 'relatedEntityType' schema updated forPUT /api/v2/entityNotificationChannels/{relatedEntityType}/{relatedEntityId}/{channelId}/. - Parameter 'relatedEntityType' schema updated forDELETE /api/v2/entityNotificationChannels/{relatedEntityType}/{relatedEntityId}/{channelId}/. - Parameter 'relatedEntityType' schema updated forGET /api/v2/entityNotificationPolicies/{relatedEntityType}/{relatedEntityId}/. - Parameter 'relatedEntityType' schema updated forGET /api/v2/entityNotificationPolicies/{relatedEntityType}/{relatedEntityId}/{policyId}/. - Parameter 'relatedEntityType' schema updated forPUT /api/v2/entityNotificationPolicies/{relatedEntityType}/{relatedEntityId}/{policyId}/. - Parameter 'relatedEntityType' schema updated forDELETE /api/v2/entityNotificationPolicies/{relatedEntityType}/{relatedEntityId}/{policyId}/. - Parameter 'relatedEntityType' schema updated forGET /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/. - Parameter 'relatedEntityType' schema updated forGET /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/{policyId}/. - Parameter 'relatedEntityType' schema updated forPUT /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/{policyId}/. - Parameter 'relatedEntityType' schema updated forDELETE /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/{policyId}/. - Parameter 'relatedEntityType' schema updated forGET /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/{policyId}/relatedPolicies/. - Parameter 'relatedEntityType' schema updated forGET /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/{policyId}/sharedRoles/. - Parameter 'relatedEntityType' schema updated forPATCH /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/{policyId}/sharedRoles/. - Parameter 'event' schema updated forGET /api/v2/eventLogs/. - Parameter 'types' schema updated forPOST /api/v2/externalDataStores/{dataStoreId}/columnsInfo/. - Parameter 'types' schema updated forGET /api/v2/externalDataStores/{dataStoreId}/credentials/. - The new optional parameterprefixhas been added toGET /api/v2/files/{catalogId}/allFiles/. - The new optional parameterprefixhas been added toGET /api/v2/files/{catalogId}/versions/{catalogVersionId}/allFiles/. - Description updated forPOST /api/v2/notificationChannelTemplates/. - Parameter 'relatedEntityType' schema updated forGET /api/v2/notificationEvents/. - Parameter 'startTime' description updated forGET /api/v2/otel/{entityType}/{entityId}/logs/. - Parameter 'startTime' description updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/autocollectedValues/. - Parameter 'startTime' description updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/podInfo/. - Parameter 'segmentValue' description updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/values/segments/{segmentAttribute}/. - Response 403 description updated forPOST /api/v2/otel/{entityType}/{entityId}/metrics/valuesOverTime/segments/. - Parameter 'segmentValue' description updated forGET /api/v2/otel/{entityType}/{entityId}/metrics/valuesOverTime/segments/{segmentAttribute}/. - Parameter 'startTime' description updated forGET /api/v2/otel/{entityType}/{entityId}/traces/. - Parameter 'traceId' description updated forGET /api/v2/otel/{entityType}/{entityId}/traces/{traceId}/. - The new optional parametertagFiltershas been added toGET /api/v2/registeredModels/. - Parameter 'startTime' description updated forGET /api/v2/tracing/{entityType}/{entityId}/. - Parameter 'traceId' description updated forGET /api/v2/tracing/{entityType}/{entityId}/{traceId}/. ## v2.40 changelog Reference for the changes introduced to version 2.40 of DataRobot's REST API. ### New features - Added an endpoint for deployment quota tracking:GET /api/v2/deployments/{deploymentId}/quotaConsumers/ - Added an endpoint for execution environment build management:PATCH /api/v2/executionEnvironments/{environmentId}/versions/{environmentVersionId}/cancelBuild/ - Added new endpoints for OpenTelemetry Metrics Segmentation. Retrieve OpenTelemetry metrics grouped by specific segment attributes for detailed analysis of metric values across different dimensions. This functionality enables users to analyze metrics by custom attributes such as deployment, model version, or other entity characteristics, both for single time periods and over time. Helps identify patterns, anomalies, and performance differences across different segments of operations, enabling more granular observability and troubleshooting. The feature is comprised of the following endpoints: - GET /api/v2/otel/{entityType}/{entityId}/metrics/values/segments/{segmentAttribute}/ - GET /api/v2/otel/{entityType}/{entityId}/metrics/valuesOverTime/segments/{segmentAttribute}/ - Added new endpoints for Tenant Usage and Resource Utilization. Monitor tenant activity and CPU/GPU resource utilization across the platform for administrative oversight and capacity planning. This functionality enables administrators to track active tenants, view aggregated CPU and GPU resource utilization by resource type, and export utilization data for reporting and analysis. Helps administrators understand platform-wide resource consumption, identify tenants with high utilization, plan capacity upgrades, and optimize resource allocation across the organization.. The feature is comprised of the following endpoints: - GET /api/v2/tenantUsageResources/activeTenants/ - GET /api/v2/tenants/utilizationResources/ - GET /api/v2/tenants/utilizationResources/export/ - GET /api/v2/tenants/utilizationResources/{resourceType}/ ### API changes - Parameter 'templateType' schema was updated forGET /api/v2/codeSnippets/. - Parameter 'eventGroup' schema was updated forGET /api/v2/entityNotificationPolicies/{relatedEntityType}/{relatedEntityId}/. - Parameter 'eventGroup' schema was updated forGET /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/. - Parameter 'eventGroup' schema was updated forGET /api/v2/notificationPolicies/. - The new optional parameterinvitedhas been added toGET /api/v2/users/. ## v2.39 changelog Reference for the changes introduced to version 2.39 of DataRobot's REST API. ### New features - Added an endpoint for custom metrics bulk upload:POST /api/v2/deployments/{deploymentId}/customMetrics/bulkUpload/ - Added new endpoints for prompt template management to manage LLMs and prompt templates for generative AI workflows. This functionality enables users to retrieve LLM information, create and manage prompt templates with versioning support, and organize prompts for consistent use across generative AI applications. Prompt templates can be versioned to track changes over time and maintain reproducibility. The endpoints support the full lifecycle of prompt management from creation to version tracking, enabling you to develop, test, and deploy prompts systematically. - Added new endpoints for OpenTelemetry metrics observability. Retrieve OpenTelemetry metrics data to monitor and observe DataRobot entities. This functionality provides access to automatically collected metrics, pod and container information, and metric values over time grouped by attributes. The endpoints enable users to monitor system performance, resource utilization, and application health through standardized OpenTelemetry metrics. They also help troubleshoot performance issues and understand resource consumption patterns across deployments and other entities. - Added new endpoints for Tenant Usage and Resource Management. Use the endpoints to monitor and export tenant resource usage and active user information for administrative oversight and capacity planning. This functionality enables system and organization administrators to track resource consumption across tenants, identify active users, retrieve available resource categories, and export usage data for reporting and analysis. The endpoints support both tenant-specific and cluster-wide usage reporting, helping administrators understand platform utilization, plan capacity, and ensure efficient resource allocation. ### API changes - ParameterentityTypeschema updated forGET /api/v2/comments/{entityType}/{entityId}/. - ParametereventGroupschema updated forGET /api/v2/entityNotificationPolicies/{relatedEntityType}/{relatedEntityId}/. - ParametereventGroupschema updated forGET /api/v2/entityNotificationPolicyTemplates/{relatedEntityType}/. - ParametereventGroupschema updated forGET /api/v2/notificationPolicies/. - The new optional parameterskip_consenthas been added toPOST /api/v2/externalOAuth/providers/{providerId}/authorize/. - Parameterlimitschema updated forGET /api/v2/genai/llms/. - The new optional parameterspanIdhas been added toGET /api/v2/otel/{entityType}/{entityId}/logs/. - Description updated forGET /api/v2/tenants/{tenantId}/activeUsers/. - Removed parametertenantId(path) forGET /api/v2/tenants/{tenantId}/resourceCategories/. - Description updated forGET /api/v2/tenants/{tenantId}/usage/. - Description updated forGET /api/v2/tenants/{tenantId}/usageExport/. ## v2.38 changelog ### Deprecation summary - Deprecated thecustomJobIdfield within the trigger property fromPOST /api/v2/deployments/(deploymentId)/retrainingPolicies/andPATCH /api/v2/deployments/(deploymentId)/retrainingPolicies/(retrainingPolicyId)/. This field will be removed in v2.40. - Thecustom_jobtrigger type is no longer supported forPOST /api/v2/deployments/(deploymentId)/retrainingPolicies/andPATCH /api/v2/deployments/(deploymentId)/retrainingPolicies/(retrainingPolicyId)/. To create a custom job retraining policy, usemodel_selection: custom_jobinstead. The option is no longer documented and will be removed in v2.40. ## v2.37 changelog v2.37 of the DataRobot REST API introduced no reported changes. ## v2.36 changelog ### New features - Added new endpoints for role based access management. Uses the following endpoints: - GET /api/v2/accessRoles/ - GET /api/v2/accessRoles/{role_id}/ - POST /api/v2/accessRoles/ - PUT /api/v2/accessRoles/{role_id}/ - DELETE /api/v2/accessRoles/{role_id}/ - GET /api/v2/accessRoles/users/ - Added an endpoint that initializes the incremental learning model and begins training using the chunking servicePOST /api/v2/projects/(projectId)/incrementalLearningModels/fromSampleModel/. To use it, enable the feature flag "Sample Data to Start Project." - Added an endpoint for retrieving themodel_historyof a deploymentGET /api/v2/deployments/(deploymentId)/modelHistory/. - Added new APIs for secure configuration management. Uses the following endpoints: - POST /api/v2/secureConfigs/ - GET /api/v2/secureConfigSchemas/ - GET /api/v2/secureConfigSchemas/(secureConfigSchemaId)/ - PATCH /api/v2/secureConfigs/(secureConfigId)/ - GET /api/v2/secureConfigs/(secureConfigId)/values/ - GET /api/v2/secureConfigs/(secureConfigId)/sharedRoles/ - PATCH /api/v2/secureConfigs/(secureConfigId)/sharedRoles/ - The start and end query parameters are no longer required for GET /api/v2/deployments/(deploymentId)/dataQualityView/ . Defaults will be provided for a one week span when not specified. - The start and end body parameters are no longer required for POST /api/v2/deployments/{deploymentId}/predictionDataExports/ . Defaults will be provided for a one week span when not specified. ### Deprecation summary - Removed the field capabilities from GET /api/v2/deployments/ and GET /api/v2/deployments/(deploymentId)/ , after being deprecated in 2.29. Instead, use GET /api/v2/deployments/(deploymentId)/capabilities/ . - Removed the query param targetClasses from GET /api/v2/deployments/(deploymentId)/accuracy/ and GET /api/v2/deployments/(deploymentId)/accuracyOverTime/ , after being deprecated in 2.31. Instead, use targetClass query param. - Deprecated metrics and modelId fields from GET /api/v2/deployments/(deploymentId)/accuracy/ , use data field in the same endpoint instead. The deprecated fields will be removed in 2.40. ## v2.35 changelog Reference the changes introduced to version 2.35 of DataRobot's REST API. ### New features - Added new endpoints for OCR Jobs. This is a feature for processing datasets with PDFs. It runs OCR on the PDFs in the dataset and replaces all images in those PDFs with text. The feature is comprised of the following endpoints: ### API changes - The new optional parameternumberOfIncrementalLearningIterationsBeforeBestModelSelectionfor projects with the auto-incremental learning option has been added toPATCH /api/v2/projects/(projectId)/aim/. It is used in automated incremental learning Autopilot mode. - Added endpoints to calculate and retrieve the SHAP distributions data: - POST /api/v2/insights/shapDistributions/ - GET /api/v2/insights/shapDistributions/models/(entityId)/ ### Deprecation summary - The API endpoint GET /api/v2/projects/(projectId)/models/(modelId)/primeInfo/ is deprecated, as creating Prime models is no longer supported. - The query params projectId and applicationId have been deprecated for GET /api/v2/useCases/ . ### Documentation changes - The correct expected HTTP response code has been documented for POST /api/v2/notifications/ . It should return a 201 , but 200 was previously documented. - The correct expected HTTP response code has been documented for PATCH /api/v2/ssoConfigurations/(configurationId)/ . It should return a 204 , but 200 was previously documented. in June 2025. ## v2.34 changelog Reference the changes introduced to version 2.34 of DataRobot's REST API. ### New features - Added an endpoint to create a hosted custom metric from a custom job: POST /api/v2/deployments/(deploymentId)/customMetrics/fromCustomJob/ . - Added an endpoint to list hosted custom metrics: GET /api/v2/customJobs/(customJobId)/customMetrics/ . - Added an endpoint to update the hosted custom metric PATCH /api/v2/deployments/(deploymentId)/customMetrics/(customMetricId)/ . - Added an endpoint to delete the hosted custom metric DELETE /api/v2/deployments/(deploymentId)/customMetrics/(customMetricId)/ . - Added an endpoint to create custom job from hosted custom metrics gallery template POST /api/v2/customJobs/fromHostedCustomMetricGalleryTemplate/ . - Added an endpoint to create a blueprint for the hosted custom metric POST /api/v2/customJobs/(customJobId)/hostedCustomMetricTemplate/ . - Added an endpoint to retrieve a blueprint for the hosted custom metric GET /api/v2/customJobs/(customJobId)/hostedCustomMetricTemplate/ . - Added an endpoint to update a blueprint for the hosted custom metric GET /api/v2/customJobs/(customJobId)/hostedCustomMetricTemplate/ . ### Enhancements - Requests for insights at the/api/v2/insights/*group of endpoints now accept theentityTypeparameter, which enables insight computation forcustomModelentities as well as nativedatarobotModelentities. To compute insights on a custom model, you must first initialize insights with a call toPOST /api/v2/modelComplianceDocsInitializations/(entityId)/. - The response from the Get SHAP impact API endpointGET /api/v2/insights/shapImpact/models/(entityId)/has been changed. Now theshapImpactsparameter is a list of key-pair values. Thecappingparameter has been removed from the response. ### Deprecation summary - The API endpoint GET /api/v2/projects/(projectId)/models/ is deprecated. Instead, use GET /api/v2/projects/(projectId)/modelRecords/ . - The API endpoint /projects/(projectId)/featureAssocationFeaturelists/ is removed, after being deprecated in release 2.19. Users should continue to use GET /api/v2/projects/(projectId)/featureAssociationFeaturelists/ . - The field modelsCount in GET /api/v2/useCases/ and GET /api/v2/useCases/(useCaseId)/ is deprecated and will be removed in June 2025. --- # API reference URL: https://docs.datarobot.com/en/docs/api/reference/index.html > Review the reference documentation available for DataRobot's APIs. The table below outlines the reference documentation available for DataRobot's API, SDKs, and code-first tools. | Resource | Description | | --- | --- | | REST API | The DataRobot REST API provides a programmatic alternative to the UI for creating and managing DataRobot assets. It allows you to automate processes and iterate more quickly, and lets you use DataRobot with scripted control. The API provides an intuitive modeling and prediction interface. | | Python API client | Installation, configuration, and usage guidelines for working with the Python client library. To access previous version of the Python API client documentation, access ReadTheDocs. | | Prediction API | DataRobot's Prediction API provides a mechanism for using your model for real-time predictions on a prediction server. | | Batch prediction API | The Batch Prediction API provides flexible options for scoring large datasets using the prediction servers you have already deployed. | | API changelogs | Changelogs contain curated, ordered lists of notable changes for each versioned release for DataRobot's SDKs and REST API. | | Self-managed resources | Details the resources available for self-managed DataRobot deployments. | | R client | Installation, configuration, and reference documentation for working with the R client library. | | OpenAPI specification | Reference the OpenAPI specification for the DataRobot REST API, which helps automate the generation of a client for languages that DataRobot doesn't directly support. It also assists with the design, implementation, and testing integration with DataRobot's REST API using a variety of automated OpenAPI-compatible tools. | --- # Make predictions with the API URL: https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html > This section describes how to use DataRobot's Prediction API to make predictions using serverless prediction environments. This section describes how to use DataRobot's Prediction API to make predictions using serverless prediction environments. If you need Prediction API reference documentation, it is available [here](https://docs.datarobot.com/en/docs/api/reference/predapi/pred-ref-serverless/index.html). You can use DataRobot's Prediction API for making predictions on a model deployment (by specifying the deployment ID). This provides access to advanced [model management](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/index.html) features like target or data drift detection. DataRobot's model management features are safely decoupled from the Prediction API so that you can gain their benefit without sacrificing prediction speed or reliability. See the [deployment](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/deploy-methods/index.html) section for details on creating a model deployment. Before generating predictions with the Prediction API, review the recommended [best practices](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#best-practices-for-the-fastest-predictions) to ensure the fastest predictions. ## Making predictions To generate predictions on new data using the Prediction API, you need: - The model's deployment ID. You can find the ID in the sample code output of the Deployments > Predictions > Prediction API tab (with Interface set to "API Client"). - Your API key . > [!WARNING] Warning > If your model is an open-source R script, it will run considerably slower. Prediction requests are submitted as POST requests to the REST API endpoint, for example: ``` curl -i -X POST "https://example.datarobot.com/api/v2/deployments//predictions" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` The order of the prediction response rows is the same as the order of the sent data. The Response returned is similar to: ``` HTTP/1.1 200 OK Content-Type: application/json X-DataRobot-Execution-Time: 38 X-DataRobot-Model-Cache-Hit: true {"data":[...]} ``` > [!NOTE] Note > The example above shows an arbitrary hostname ( `example.datarobot.com`) as the Prediction API URL; be sure to use the correct hostname of your DataRobot instance. The configured (predictions) URL is displayed in the sample code of the [Deployments > Predictions > Prediction API](https://docs.datarobot.com/en/docs/classic-ui/predictions/realtime/code-py.html) tab. See your system administrator for more assistance if needed. ## Using persistent HTTP connections All prediction requests are served over a secure connection (SSL/TLS), which can result in significant connection setup time. Depending on your network latency to the prediction instance, this can be anywhere from 30ms to upwards of 100-150ms. To address this, the Prediction API supports HTTP Keep-Alive, enabling your systems to keep a connection open for up to a minute after the last prediction request. Using the Python `requests` module, run your prediction requests from `requests.Session`: ``` import json import requests data = [ json.dumps({'Feature1': 42, 'Feature2': 'text value 1'}), json.dumps({'Feature1': 60, 'Feature2': 'text value 2'}), ] api_key = '...' api_endpoint = '...' session = requests.Session() session.headers = { 'Authorization': 'Bearer {}'.format(api_key), 'Content-Type': 'text/json', } for row in data: print(session.post(api_endpoint, data=row).json()) ``` Check the documentation of your favorite HTTP library for how to use persistent connections in your integration. ## Prediction inputs The API supports both JSON- and CSV-formatted input data (although JSON can be a safer choice if it is created with a good quality JSON parser). Data can either be posted in the [request body](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#request-schema) or via a [file upload](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#file-input) (multipart form). > [!NOTE] Note > When using the Prediction API, the only supported column separator in CSV files and request bodies is the comma ( `,`). ### JSON input The JSON input is formatted as an array of objects where the key is the feature name and the value is the value in the dataset. For example, a CSV file that looks like: ``` a,b,c 1,2,3 7,8,9 ``` Would be represented in JSON as: ``` [ { "a": 1, "b": 2, "c": 3 }, { "a": 7, "b": 8, "c": 9 } ] ``` Submit a JSON array to the Prediction API by sending the data to the `/api/v2/deployments//predictions` endpoint. For example: ``` curl -H "Content-Type: application/json" -X POST --data '[{"a": 4, "b": 5, "c": 6}\]' \ -H "Authorization: Bearer " \ https://example.datarobot.com/api/v2/deployments//predictions ``` ### File input This example assumes a CSV file, `dataset.csv`, that contains a header and the rows of data to predict on. cURL automatically sets the content type. ``` curl -i -X POST "https://example.datarobot.com/api/v2/deployments//predictions" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv HTTP/1.1 200 OK Date: Fri, 08 Feb 2019 10:00:00 GMT Content-Type: application/json Content-Length: 60624 Connection: keep-alive Server: nginx/1.12.2 X-DataRobot-Execution-Time: 39 X-DataRobot-Model-Cache-Hit: true Access-Control-Allow-Methods: OPTIONS, POST Access-Control-Allow-Credentials: true Access-Control-Expose-Headers: Content-Type,Content-Length,X-DataRobot-Execution-Time,X-DataRobot-Model-Cache-Hit,X-DataRobot-Model-Id,X-DataRobot-Request-Id Access-Control-Allow-Headers: Content-Type,Authorization X-DataRobot-Request-ID: 9e61f97bf07903b8c526f4eb47830a86 { "data": [ { "predictionValues": [ { "value": 0.2570950924, "label": 1 }, { "value": 0.7429049076, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 0, "rowId": 0 }, { "predictionValues": [ { "value": 0.7631880558, "label": 1 }, { "value": 0.2368119442, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 1, "rowId": 1 } ] } ``` ### In-body text input This example includes the CSV file content in the request body. With this format, you must set the Content-Type of the form data to `text/plain`. ``` curl -i -X POST "https://example.datarobot.com/api/v2/deployments//predictions" --data-binary $'a,b,c\n1,2,3\n7,8,9\n' -H "content-type: text/plain" \ -H "Authorization: Bearer " \ ``` ## Prediction outputs The Content-Type header value must be set appropriately for the type of data being sent ( `text/csv` or `application/json`); the raw API request responds with JSON by default. Reference the [output format](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html) for more information about the structure of the output schema. The output schema shares the same format for real-time and batch predictions. ### CSV output To return CSV in addition to JSON from the Prediction API for real-time predictions, use `-H "Accept: text/csv"`. ## Prediction objects The following sections describe the content of the various prediction objects. ### Request schema Note that Request schema are standard for any kind of predictions. The following are the accepted headers: | Name | Value(s) | | --- | --- | | Content-Type | text/csv;charset=utf8 | | | application/json | | | multipart/form-data | | Content-Encoding | gzip | | | bz2 | | Authorization | Bearer | Note the following: - If you are submitting predictions as a raw stream of data, you can specify an encoding by adding;charset=to theContent-Typeheader. See thePython standard encodingsfor a list of valid values. DataRobot usesutf8by default. - If you are sending an encoded stream of data, you should specify theContent-Encodingheader. - TheAuthorizationfield is a Bearer authentication HTTP authentication scheme that involves security tokens called bearer tokens. While it is possible to authenticate via pair username + API token (Basic auth) or just via API token, these authentication methods are deprecated and not recommended. You can parameterize a request using URI query parameters: | Parameter name | Type | Notes | | --- | --- | --- | | passthroughColumns | string | List of columns from a scoring dataset to return in the prediction response. | | passthroughColumnsSet | string | If passthroughColumnsSet=all is passed, all columns from the scoring dataset are returned in the prediction response. | Note the following: - The passthroughColumns and passthroughColumnsSet parameters cannot both be passed in the same request. - While there is no limit on the number of column names you can pass with the passthroughColumns query parameter, there is a limit on the HTTP request line (currently 8192 bytes). The following example illustrates the use of multiple passthrough columns: ``` curl -i -X POST "https://example.datarobot.com/api/v2/deployments//predictions?passthroughColumns=Latitude&passthroughColumns=Longitude" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` ### Response schema The following is a sample prediction response body (also see the additional example of a [time series response body](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#making-predictions-with-time-series)): ``` { "data": [ { "predictionValues": [ { "value": 0.6856798909, "label": 1 }, { "value": 0.3143201091, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 1, "rowId": 0, "passthroughValues": { "Latitude": -25.433508, "Longitude": 22.759397 } }, { "predictionValues": [ { "value": 0.765656753, "label": 1 }, { "value": 0.234343247, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 1, "rowId": 1, "passthroughValues": { "Latitude": 41.051128, "Longitude": 14.49598 } } ] } ``` The table below lists custom DataRobot headers: | Name | Value | Note | | --- | --- | --- | | X-DataRobot-Execution-Time | numeric | Time for compute predictions (ms). | | X-DataRobot-Model-Cache-Hit | true or false | Indication of in-memory presence of model (bool). | | X-DataRobot-Model-Id | ObjectId | ID of the model used to serve the prediction request (only returned for predictions made on model deployments). | | X-DataRobot-Request-Id | uuid | Unique identifier of a prediction request. | The following table describes the Response Prediction Rows of the JSON array: | Name | Type | Note | | --- | --- | --- | | predictionValues | array | An array of predictionValues (schema described below). | | predictionThreshold | float | The threshold used for predictions (applicable to binary classification projects only). | | prediction | float | The output of the model for this row. | | rowId | int | The row described. | | passthroughValues | object | A JSON object where key is a column name and value is a corresponding value for a predicted row from the scoring dataset. This JSON item is only returned if either passthroughColumns or passthroughColumnsSet is passed. | | adjustedPrediction | float | The exposure-adjusted output of the model for this row if the exposure was used during model building. The adjustedPrediction is included in responses if the request parameter excludeAdjustedPredictions is false. | | adjustedPredictionValues | array | An array of exposure-adjusted PredictionValue (schema described below). The adjustedPredictionValues is included in responses if the request parameter excludeAdjustedPredictions is false. | | predictionExplanations | array | An array of PredictionExplanations (schema described below). This JSON item is only returned with Prediction Explanations. | #### Prediction values schema The following table describes the `predictionValues` schema in the JSON Response array: | Name | Type | Note | | --- | --- | --- | | label | - | Describes what the model output corresponds to. For regression projects, it is the name of the target feature. For classification projects, it is a label from the target feature. | | value | float | The output of the prediction. For regression projects, it is the predicted value of the target. For classification projects, it is the probability associated with the label that is predicted to be most likely (implying a threshold of 0.5 for binary classification problems). | #### Extra custom model output schema > [!NOTE] Availability information > Additional output in prediction responses for custom models is off by default. Contact your DataRobot representative or administrator for information on enabling this feature. > > Feature flag: Enable Additional Custom Model Output in Prediction Responses In some cases, the prediction response from your model may contain extra model output. This is possible for [custom models with additional output columns](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#additional-output-columns) defined in the `score()` hook and for Generative AI (GenAI) models. The `score()` hook can return any number of extra columns, containing data of types `string`, `int`, `float`, `bool`, or `datetime`. When additional columns are returned through the `score()` method, the prediction response is as follows: - For a tabular response (CSV) , the additional columns are returned as part of the response table or dataframe. - For a JSON response , the extraModelOutput key is returned alongside each row. This key is a dictionary containing the values of each additional column in the row. As custom models, deployed GenAI models can return extra columns through the `extraModelOutput` key to provide information about the text generation model (citations, latency, confidence, LLM blueprint ID, token counts, etc.), as shown in the example below: > [!NOTE] Citations in prediction response > For citations to be included in a Gen AI model's prediction response, the [LLM deployed from the playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/deploy-llm.html) must have a [vector database (VDB)](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html) associated with it. ``` # JSON response for GenAI model predictions { "data": [ { "rowId": 0, "prediction": "In the field of biology, there have been some exciting new discoveries made through research conducted on the International Space Station (ISS). Here are three examples:\n\n1. Understanding Plant Root Orientation: Scientists have been studying the growth and development of plants in microgravity. They found that plants grown in space exhibit different root orientation compared to those grown on Earth. This discovery helps us understand how plants adapt and respond to the absence of gravity. This knowledge can be applied to improve agricultural practices and develop innovative techniques for growing plants in challenging environments on Earth.\n\n2. Tissue Damage and Repair: One fascinating area of research on the ISS involves studying how living organisms respond to injuries in space. Scientists have investigated tissue damage and repair mechanisms in various organisms, including humans. By studying the healing processes in microgravity, researchers gained insights into how wounds heal differently in space compared to on Earth. This knowledge has implications for developing new therapies and treatments for wound healing and tissue regeneration.\n\n3. Bubbles, Lightning, and Fire Dynamics: The ISS provides a unique laboratory environment for studying the behavior of bubbles, lightning, and fire in microgravity. Scientists have conducted experiments to understand how these phenomena behave differently without the influence of gravity. These studies have practical applications, such as improving combustion processes, enhancing fire safety measures, and developing more efficient cooling systems.\n\nThese are just a few examples of the exciting discoveries that have been made in the field of biology through research conducted on the ISS. The microgravity environment of space offers a unique perspective and enables researchers to uncover new insights into the workings of living organisms and their interactions with the environment.", "predictionValues": [ { "label": "resultText", "value": "In the field of biology, there have been some exciting new discoveries made through research conducted on the International Space Station (ISS). Here are three examples:\n\n1. Understanding Plant Root Orientation: Scientists have been studying the growth and development of plants in microgravity. They found that plants grown in space exhibit different root orientation compared to those grown on Earth. This discovery helps us understand how plants adapt and respond to the absence of gravity. This knowledge can be applied to improve agricultural practices and develop innovative techniques for growing plants in challenging environments on Earth.\n\n2. Tissue Damage and Repair: One fascinating area of research on the ISS involves studying how living organisms respond to injuries in space. Scientists have investigated tissue damage and repair mechanisms in various organisms, including humans. By studying the healing processes in microgravity, researchers gained insights into how wounds heal differently in space compared to on Earth. This knowledge has implications for developing new therapies and treatments for wound healing and tissue regeneration.\n\n3. Bubbles, Lightning, and Fire Dynamics: The ISS provides a unique laboratory environment for studying the behavior of bubbles, lightning, and fire in microgravity. Scientists have conducted experiments to understand how these phenomena behave differently without the influence of gravity. These studies have practical applications, such as improving combustion processes, enhancing fire safety measures, and developing more efficient cooling systems.\n\nThese are just a few examples of the exciting discoveries that have been made in the field of biology through research conducted on the ISS. The microgravity environment of space offers a unique perspective and enables researchers to uncover new insights into the workings of living organisms and their interactions with the environment." } ], "deploymentApprovalStatus": "APPROVED", "extraModelOutput": { "CITATION_CONTENT_8": "3\nthe research study is received by others and how the \nknowledge is disseminated through citations in other \njournals. For example, six ISS studies have been \npublished in Nature, represented as a small node in the \ngraph. Network analysis shows that findings published \nin Nature are likely to be cited by other similar leading \njournals such as Science and Astrophysical Journal \nLetters (represented in bright yellow links) as well as \nspecialized journals such as Physical Review D and New \nJournal of Physics (represented in a yellow-green link). \nSix publications in Nature led to 512 citations according \nto VOSviewer\u2019s network map (version 1.6.11), an \nincrease of over 8,000% from publication to citation. \nFor comparison purposes, 6 publications in a small \njournal like American Journal of Botany led to 185 \ncitations and 107 publications in Acta Astronautica, \na popular journal among ISS scientists, led to 1,050 \ncitations (Figure 3, panel B). This count of 1,050", "CITATION_CONTENT_9": "Introduction\n4\nFigure 3. Count of publications reported in journals ranked in the top 100 according to global standards of Clarivate. A total of 567 top-tier publications \nthrough the end of FY-23 are shown by year and research category.\nIn this year\u2019s edition of the Annual Highlights of Results, we report findings from a \nwide range of topics in biology and biotechnology, physics, human research, Earth and \nspace science, and technology development \u2013 including investigations about plant root \norientation, tissue damage and repair, bubbles, lightning, fire dynamics, neutron stars, \ncosmic ray nuclei, imaging technology improvements, brain and vascular health, solar \npanel materials, grain flow, as well as satellite and robot control. \nThe findings highlighted here are only a small sample representative of the research \nconducted by the participating space agencies \u2013 ASI (Agenzia Spaziale Italiana), CSA \n(Canadian Space Agency), ESA (European Space Agency), JAXA (Japanese Aerospace", "CITATION_PAGE_3": 4, "CITATION_PAGE_8": 6, "CITATION_CONTENT_5": "23\nPUBLICATION HIGHLIGHTS: \nEARTH AND SPACE SCIENCE\nThe ISS laboratories enable scientific experiments in the biological sciences \nthat explore the complex responses of living organisms to the microgravity \nenvironment. The lab facilities support the exploration of biological systems \nranging from microorganisms and cellular biology to integrated functions \nof multicellular plants and animals. Several recent biological sciences \nexperiments have facilitated new technology developments that allow \ngrowth and maintenance of living cells, tissues, and organisms.\nThe Alpha Magnetic \nSpectrometer-02 (AMS-02) is \na state-of-the-art particle \nphysics detector constructed, \ntested, and operated by an \ninternational team composed \nof 60 institutes from \n16 countries and organized \nunder the United States \nDepartment of Energy (DOE) sponsorship. \nThe AMS-02 uses the unique environment of \nspace to advance knowledge of the universe \nand lead to the understanding of the universe\u2019s", "CITATION_SOURCE_5": "Space_Station_Annual_Highlights/iss_2017_highlights.pdf", "CITATION_CONTENT_3": "Introduction\n2\nExtensive international collaboration in the \nunique environment of LEO as well as procedural \nimprovements to assist researchers in the collection \nof data from the ISS have produced promising \nresults in the areas of protein crystal growth, tissue \nregeneration, vaccine and drug development, 3D \nprinting, and fiber optics, among many others. In \nthis year\u2019s edition of the Annual Highlights of Results, \nwe report findings from a wide range of topics in \nbiotechnology, physics, human research, Earth and \nspace science, and technology development \n\u2013 including investigations about human retinal cells, \nbacterial resistance, black hole detection, space \nanemia, brain health, Bose-Einstein condensates, \nparticle self-assembly, RNA extraction technology, \nand more. The findings highlighted here represent \nonly a sample of the work ISS has contributed to \nsociety during the past 12 months.\nAs of Oct. 1, 2022, we have identified a total of 3,679", "CITATION_SOURCE_8": "Space_Station_Annual_Highlights/iss_2021_highlights.pdf", "CITATION_PAGE_7": 8, "CITATION_PAGE_6": 8, "CITATION_PAGE_2": 4, "CITATION_CONTENT_7": "Biology and Biotechnology Earth and Space Science Educational and Cultural Activities\nHuman Research Physical Science Technology Development and Demonstration", "CITATION_SOURCE_9": "Space_Station_Annual_Highlights/iss_2023_highlights.pdf", "datarobot_latency": 3.1466632366, "blocked_resultText": false, "CITATION_SOURCE_2": "Space_Station_Annual_Highlights/iss_2023_highlights.pdf", "CITATION_SOURCE_6": "Space_Station_Annual_Highlights/iss_2021_highlights.pdf", "CITATION_SOURCE_7": "Space_Station_Annual_Highlights/iss_2023_highlights.pdf", "datarobot_confidence_score": 0.6524822695, "CITATION_PAGE_9": 7, "CITATION_CONTENT_4": "Molecular Life Sciences. 2021 October 29; DOI: \n10.1007/s00018-021-03989-2.\nFigure 7. Immunoflourescent images of human retinal \ncells in different conditions. Image adopted from \nCialdai, Cellular and Molecular Life Sciences.\nThe ISS laboratory provides a platform for investigations in the biological sciences that \nexplores the complex responses of living organisms to the microgravity environment. Lab \nfacilities support the exploration of biological systems, from microorganisms and cellular \nbiology to the integrated functions of multicellular plants and animals.", "CITATION_SOURCE_1": "Space_Station_Annual_Highlights/iss_2023_highlights.pdf", "CITATION_SOURCE_0": "Space_Station_Annual_Highlights/iss_2018_highlights.pdf", "CITATION_SOURCE_3": "Space_Station_Annual_Highlights/iss_2022_highlights.pdf", "CITATION_PAGE_5": 26, "CITATION_PAGE_0": 7, "CITATION_PAGE_1": 11, "LLM_BLUEPRINT_ID": "662ba0062ade64c4fc4c1a1f", "CITATION_PAGE_4": 9, "datarobot_token_count": 320, "CITATION_CONTENT_0": "more effectively in space by addressing \nsuch topics as understanding radiation effects on \ncrew health, combating bone and muscle loss, \nimproving designs of systems that handle fluids \nin microgravity, and determining how to maintain \nenvironmental control efficiently. \nResults from the ISS provide new \ncontributions to the body of scientific \nknowledge in the physical sciences, life \nsciences, and Earth and space sciences \nto advance scientific discoveries in multi\u0002disciplinary ways. \nISS science results have Earth-based \napplications, including understanding our \nclimate, contributing to the treatment of \ndisease, improving existing materials, and inspiring \nthe future generation of scientists, clinicians, \ntechnologists, engineers, mathematicians, artists, \nand explorers.\nBENEFITS\nFOR HUMANITY\nDISCOVERY\nFigure 4. A heat map of all of the countries whose authors have cited scientific results publications from ISS Research through October 1, 2018.\nEXPLORATION", "CITATION_SOURCE_4": "Space_Station_Annual_Highlights/iss_2022_highlights.pdf", "CITATION_CONTENT_2": "capabilities (i.e., facilities), and data delivery are critical to the effective operation \nof scientific projects for accurate results to be shared with the scientific community, \nsponsors, legislators, and the public. \nOver 3,700 investigations have operated since Expedition 1, with more than 250 active \nresearch facilities, the participation of more than 100 countries, the work of more than \n5,000 researchers, and over 4,000 publications. The growth in research (Figure 1) and \ninternational collaboration (Figure 2) has prompted the publication of over 560 research \narticles in top-tier scientific journals with about 75 percent of those groundbreaking studies \noccurring since 2018 (Figure 3). \nBibliometric analyses conducted through VOSviewer1\n measure the impact of space station \nresearch by quantifying and visualizing networks of journals, citations, subject areas, and \ncollaboration between authors, countries, or organizations. Using bibliometrics, a broad", "CITATION_CONTENT_1": "technologists, engineers, mathematicians, artists, and explorers.\nEXPLORATION\nDISCOVERY\nBENEFITS\nFOR HUMANITY", "CITATION_CONTENT_6": "control efficiently. \nResults from the ISS provide new \ncontributions to the body of scientific \nknowledge in the physical sciences, life \nsciences, and Earth and space sciences \nto advance scientific discoveries in multi\u0002disciplinary ways. \nISS science results have Earth-based \napplications, including understanding our \nclimate, contributing to the treatment of \ndisease, improving existing materials, and \ninspiring the future generation of scientists, \nclinicians, technologists, engineers, \nmathematicians, artists and explorers.\nBENEFITS\nFOR HUMANITY\nDISCOVERY\nEXPLORATION" } }, ] } ``` ## Making predictions with time series > [!TIP] Tip > Time series predictions are specific to time series projects, not all time-aware modeling projects. Specifically, the CSV file must follow a specific format, described in the [predictions section](https://docs.datarobot.com/en/docs/classic-ui/modeling/time/ts-predictions.html#make-predictions-tab) of the time series modeling pages. If you are making predictions with the forecast point, you can skip the forecast window in your prediction data as DataRobot generates a forecast point automatically. This is called autoexpansion. Autoexpansion applies automatically if: - Predictions are made for a specific forecast point and not a forecast range. - The time series project has a regular time step and does not use Nowcasting. When using autoexpansion, note the following: - If you have Known in Advance features that are important for your model, it is recommended that you manually create a forecast window to increase prediction accuracy. - If you plan to use an association ID other than the primary date/time column in your deployment to track accuracy, create a forecast window manually. The URL for making predictions with time series deployments and regular non-time series deployments is the same. The only difference is that you can optionally specify forecast point, prediction start/end date, or some other time series specific URL parameters. Using the deployment ID, the server automatically detects the deployed model as a time series deployment and processes it accordingly: ``` curl -i -X POST "https://example.datarobot.com/api/v2/deployments//predictions" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` The following is a sample Response body for a multiseries project: ``` HTTP/1.1 200 OK Content-Type: application/json X-DataRobot-Execution-Time: 1405 X-DataRobot-Model-Cache-Hit: false { "data": [ { "seriesId": 1, "forecastPoint": "2018-01-09T00:00:00Z", "rowId": 365, "timestamp": "2018-01-10T00:00:00.000000Z", "predictionValues": [ { "value": 45180.4041874386, "label": "target (actual)" } ], "forecastDistance": 1, "prediction": 45180.4041874386 }, { "seriesId": 1, "forecastPoint": "2018-01-09T00:00:00Z", "rowId": 366, "timestamp": "2018-01-11T00:00:00.000000Z", "predictionValues": [ { "value": 47742.9432499386, "label": "target (actual)" } ], "forecastDistance": 2, "prediction": 47742.9432499386 }, { "seriesId": 1, "forecastPoint": "2018-01-09T00:00:00Z", "rowId": 367, "timestamp": "2018-01-12T00:00:00.000000Z", "predictionValues": [ { "value": 46394.5698978878, "label": "target (actual)" } ], "forecastDistance": 3, "prediction": 46394.5698978878 }, { "seriesId": 2, "forecastPoint": "2018-01-09T00:00:00Z", "rowId": 697, "timestamp": "2018-01-10T00:00:00.000000Z", "predictionValues": [ { "value": 39794.833199375, "label": "target (actual)" } ] } ] } ``` ### Request parameters You can parameterize the time series prediction request using URI query parameters. For example, overriding the default inferred forecast point can look like this: ``` curl -i -X POST "https://example.datarobot.com/api/v2/deployments//predictions?forecastPoint=1961-01-01T00:00:00?relaxKnownInAdvanceFeaturesCheck=true" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` For the full list of time series-specific parameters, see [Time series predictions for deployments](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/time-pred.html). ### Response schema The [Response schema](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#response-schema_1) is consistent with [standard predictions](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#response-schema_2) but adds a number of columns for each `PredictionRow` object: | Name | Type | Notes | | --- | --- | --- | | seriesId | string, int, or None | A multiseries identifier of a predicted row that identifies the series in a multiseries project. | | forecastPoint | string | An ISO 8601 formatted DateTime string corresponding to the forecast point for the prediction request, either user-configured or selected by DataRobot. | | timestamp | string | An ISO 8601 formatted DateTime string corresponding to the DateTime column of the predicted row. | | forecastDistance | int | A forecast distance identifier of the predicted row, or how far it is from forecastPoint in the scoring dataset. | | originalFormatTimestamp | string | A DateTime string corresponding to the DateTime column of the predicted row. Unlike the timestamp column, this column will keep the same DateTime formatting as the uploaded prediction dataset. (This column is shown if enabled by your administrator.) | ## Making Prediction Explanations The DataRobot [Prediction Explanations](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/index.html) feature gives insight into which attributes of a particular input cause it to have exceptionally high or exceptionally low predicted values. > [!TIP] Tip > You must run the following two critical dependencies before running Prediction Explanations: > > You must compute > Feature Impact > for the model. > You must generate predictions on the dataset using the selected model. To initialize Prediction Explanations, use the [Prediction Explanations](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/index.html) tab. Making Prediction Explanations is very similar to standard prediction requests. First, Prediction Explanations requests are submitted as POST requests to the resource: ``` curl -i -X POST "https://example.datarobot.com/api/v2/deployments//predictionExplanations" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` The following is a sample Response body: ``` HTTP/1.1 200 OK Content-Type: application/json X-DataRobot-Execution-Time: 841 X-DataRobot-Model-Cache-Hit: true { "data": [ { "predictionValues": [ { "value": 0.6634830442, "label": 1 }, { "value": 0.3365169558, "label": 0 } ], "prediction": 1, "rowId": 0, "predictionExplanations": [ { "featureValue": 49, "strength": 0.6194461777, "feature": "driver_age", "qualitativeStrength": "+++", "label": 1 }, { "featureValue": 1, "strength": 0.3501610895, "feature": "territory", "qualitativeStrength": "++", "label": 1 }, { "featureValue": "M", "strength": -0.171075409, "feature": "gender", "qualitativeStrength": "--", "label": 1 } ] }, { "predictionValues": [ { "value": 0.3565584672, "label": 1 }, { "value": 0.6434415328, "label": 0 } ], "prediction": 0, "rowId": 1, "predictionExplanations": [] } ] } ``` ### Request parameters You can parameterize the Prediction Explanations prediction request using URI query parameters: | Parameter name | Type | Notes | | --- | --- | --- | | maxExplanations | int | Maximum number of codes generated per prediction. Default is 3. Previously called maxCodes. | | thresholdLow | float | Prediction Explanation low threshold. Predictions must be below this value (or above the thresholdHigh value) for Prediction Explanations to compute. This value can be null. | | thresholdHigh | float | Prediction Explanation high threshold. Predictions must be above this value (or below the thresholdLow value) for Prediction Explanations to compute. This value can be null. | | excludeAdjustedPredictions | string | Includes or excludes exposure-adjusted predictions in prediction responses if exposure was used during model building. The default value is 'true' (exclude exposure-adjusted predictions). | The following is an example of a parameterized request: ``` curl -i -X POST "https://example.datarobot.com/api/v2/deployments//predictionExplanations?maxExplanations=2&thresholdLow=0.2&thresholdHigh=0.5" -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` DataRobot's headers schema is the same as that for prediction responses. The [Response schema](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#response-schema_2) is consistent with standard predictions, but adds "predictionExplanations", an array of `PredictionExplanations` for each `PredictionRow` object. #### PredictionExplanations schema Response JSON Array of Objects: | Name | Type | Notes | | --- | --- | --- | | label | – | Describes which output was driven by this Prediction Explanation. For regression projects, it is the name of the target feature. For classification projects, it is the class whose probability increasing would correspond to a positive strength of this Prediction Explanation. | | feature | string | Name of the feature contributing to the prediction. | | featureValue | - | Value the feature took on for this row. | | strength | float | Amount this feature's value affected the prediction. | | qualitativeStrength | string | Human-readable description of how strongly the feature affected the prediction (e.g., +++, –, +). | > [!TIP] Tip > The prediction explanation `strength` value is not bounded to the values `[-1, 1]`; its interpretation may change as the number of features in the model changes. For normalized values, use `qualitativeStrength` instead.`qualitativeStrength` expresses the `[-1, 1]` range with visuals, with `---` representing `-1` and `+++` representing `1`. For explanations with the same `qualitativeStrength`, you can then use the `strength` value for ranking. > > See the section on [interpreting Prediction Explanation output](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/xemp-pe.html#interpret-xemp-prediction-explanations) for more information. ## Making predictions with humility monitoring Predictions with [humility monitoring](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment-settings/humility-settings.html) allow you to monitor predictions using user-defined humility rules. When a prediction falls outside the thresholds provided for the "Uncertain Prediction" Trigger, it will default to the action assigned to the trigger. The humility key is added to the body of the prediction response when the trigger is activated. The following is a sample Response body for a Regression project with an `Uncertain Prediction Trigger` with `Action - No Operation`: ``` { "data": [ { "predictionValues": [ { "value": 122.8034057617, "label": "length" } ], "prediction": 122.8034057617, "rowId": 99, "humility": [ { "ruleId": "5ebad4735f11b33a38ff3e0d", "triggered": true, "ruleName": "Uncertain Prediction Trigger" } ] } ] } ``` The following is an example of a Response body for a regression model deployment. It uses the "Uncertain Prediction" trigger with the "Throw Error" action: ``` 480 Error: {"message":"Humility ReturnError action triggered."} ``` The following is an example of a Response body for a regression model deployment. It uses the "Uncertain Prediction" trigger with the "Override Prediction" action: ``` { "data": [ { "predictionValues": [ { "value": 122.8034057617, "label": "length" } ], "prediction": 5220, "rowId": 99, "humility": [ { "ruleId": "5ebad4735f11b33a38ff3e0d", "triggered": true, "ruleName": "Uncertain Prediction Trigger" } ] } ] } ``` ### Response schema The [response schema](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#response-schema_2) is consistent with [standard predictions](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html#response-schema) but adds a new humility column with a subset of columns for each `Humility` object: | Name | Type | Notes | | --- | --- | --- | | ruleId | string | The ID of the humility rule assigned to the deployment | | triggered | boolean | Returns "True" or "False" depending on if the rule was triggered or not | | ruleName | string | The name of the rule that is either defined by the user or auto-generated with a timestamp | ## Error responses Any error is indicated by a non-200 code attribute. Codes starting with 4XX indicate request errors (e.g., missing columns, wrong credentials, unknown model ID). The message attribute gives a detailed description of the error in the case of a 4XX code. For example: ``` curl -H "Content-Type: application/json" -X POST --data '' \ -H "Authorization: Bearer " \ https://example.datarobot.com/api/v2/deployments//predictions HTTP/1.1 400 BAD REQUEST Date: Fri, 08 Feb 2019 11:00:00 GMT Content-Type: application/json Content-Length: 53 Connection: keep-alive Server: nginx/1.12.2 X-DataRobot-Execution-Time: 332 X-DataRobot-Request-ID: fad6a0b62c1ff30db74c6359648d12fd { "message": "The requested URL was not found on the server. If you entered the URL manually, please check your spelling and try again." } ``` Codes starting with 5XX indicate server-side errors. Retry the request or contact your DataRobot representative. ## Knowing the limitations The following describes the size and timeout boundaries for real-time deployment predictions: - Maximum data submission size is 50MB. - There is no limit on the number of rows, but timeout limits are as follows: If your request exceeds the timeout, or you are trying to score a large file using serverless predictions, consider using thebatch scoring package. - There is a limit on the size of theHTTP request line(currently 8192 bytes). - For managed AI Platform deployments, serverless prediction environments automatically close persistent HTTP connections if they are idle for more than 600 seconds. To use persistent connections, the client side must be able to handle these disconnects correctly. The following example configures Python HTTP libraryrequeststo automatically retry HTTP requests on transport failure: ``` import requests import urllib3 # create a transport adapter that will automatically retry GET/POST/HEAD requests on failures up to 3 times adapter = requests.adapters.HTTPAdapter( max_retries=urllib3.Retry( total=3, method_whitelist=frozenset(['GET', 'POST', 'HEAD']) ) ) # create a Session (a pool of connections) and make it use the given adapter for HTTP and HTTPS requests session = requests.Session() session.mount('http://', adapter) session.mount('https://', adapter) # execute a prediction request that will be retried on transport failures, if needed api_token = '' response = session.post( 'https://example.datarobot.com/api/v2/deployments//predictions', headers={ 'Authorization': 'Bearer %s' % api_token, 'Content-Type': 'text/csv', }, data='', ) print(response.content) ``` ### Model caching Serverless prediction environments fetch models, as needed, from the DataRobot cluster. To speed up subsequent predictions that use the same model, DataRobot stores a certain number of models in memory (cache). When the cache fills, each new model request will require that one of the existing models in the cache be removed. DataRobot removes the least recently used model (which is not necessarily the model that has been in the cache the longest). For Self-Managed AI Platform installations, the default size for the cache is 16 models, but it can vary from installation to installation. Please contact DataRobot support if you have questions regarding the cache size of your specific installation. A serverless prediction environment runs multiple prediction processes, each of which has its own exclusive model cache. Prediction processes do not share between themselves. Because of this, it is possible that you send two consecutive requests to a serverless prediction environment, and each has to download the model data. Each response from the serverless prediction environment includes a header, `X-DataRobot-Model-Cache-Hit`, indicating whether the model used was in the cache. If the model was in the cache, the value of the header is true; if the value is false, the model was not in the cache. ## Best practices for the fastest predictions The following checklist summarizes the suggestions above to help deliver the fastest predictions possible: - Implementpersistent HTTP connections: This reduces network round-trips, and thus latency, to the Prediction API. - Use CSV data:Because JSON serialization of large amounts of data can take longer than using CSV, consider using CSV for yourprediction inputs. - Keep the number of requested models low:This allows the Prediction API to make use ofmodel caching. - Batch data together in chunks:Batch as many rows together as possible without going over the50MB real-time deployment prediction request limit. If scoring larger files, consider using theBatch Prediction APIwhich, in addition to scoring local files, also supports scoring to and from S3 and databases. --- # Prediction API URL: https://docs.datarobot.com/en/docs/api/reference/predapi/index.html > DataRobot's Prediction API provides a mechanism for using your model for real-time predictions via serverless prediction environments. DataRobot's Prediction API provides a mechanism for using your model for real-time predictions via serverless or dedicated prediction environments. Follow [the guidelines](https://docs.datarobot.com/en/docs/api/reference/predapi/dr-predapi-serverless.html) for making serverless predictions with the Prediction API. Serverless predictions use the REST API endpoint `/api/v2/deployments/:id/predictions` and do not require a `datarobot-key` header. To access documentation for the dedicated server Prediction API, navigate to the [Dedicated Prediction API page](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/index.html). | Topic | Description | | --- | --- | | Make predictions with the API (serverless) | Make predictions using serverless prediction environments. | | Make predictions with the Python API client | Use the DataRobot Prediction Library, a Python library for making predictions with various prediction methods. | | Get a prediction server ID | Retrieve a prediction server ID using cURL commands from the REST API or the DataRobot Python client. | | Serverless Prediction API reference | Review Prediction API methods, input and output parameters, and errors for serverless predictions. | | Make predictions with the API (dedicated server) | Legacy documentation for using the Prediction API with dedicated servers. | | Deprecated API routes | Review deprecated Prediction API routes in a reference document listing the deprecated requests and their replacements. | --- # Deprecated API routes URL: https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/deprecated-prediction-api.html > An overview of DataRobot's deprecated Prediction API routes, with a complete list of the specific deprecated POST and GET requests and their replacements. The Prediction API has changed significantly over time and accumulated a number of old routes. Even though these routes are already deprecated, they are still available in some installations since not all users have migrated to newer versions yet. This page describes: - all such deprecated routes - deadlines for their complete removal - new REST endpoints that should be used instead of old ones - how Prediction Admins can capture usages of deprecated routes within their organization to safely upgrade DataRobot Please refer to Prediction API [reference](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/index.html) documentation for details on each specific route. ## Deprecated Prediction API routes The Prediction API has moved from "project/model" routes to "deployment-aware" routes. To support this transition, the following routes have been deprecated: > [!WARNING] Warning > Availability of deprecated routes (those not using the deployment-aware model) is dependent on the initial DataRobot deployment version. > See the table below for complete details. Contact your DataRobot representative if you need help migrating to the new API routes. | Deployment type | Installation timeline | Status and notes | | --- | --- | --- | | Self-Managed AI Platform | New as of v6.0 or later | Disabled | | Self-Managed AI Platform | Upgraded to v6.0 or v.6.1 | Supported | | Self-Managed AI Platform | v6.2 upgrade (future) | All deprecated routes removed entirely | | AI Platform* | Migrated individually, contact your DataRobot representative | Migration is in progress | * Managed AI Platform accounts newer than May 2020 only have access to the new routes. ### The full list of deprecated routes #### Make AutoML predictions Deprecated route: `POST /predApi/v1.0///predict` New route: `POST /predApi/v1.0/deployments//predictions` #### Make time series predictions Deprecated routes: `POST /predApi/v1.0///timeSeriesPredict` `POST /predApi/v1.0/deployments//timeSeriesPredictions` New route: `POST /predApi/v1.0/deployments//predictions` #### Prediction Explanations Deprecated routes: `POST /predApi/v1.0///reasonCodesPredictions` `POST /predApi/v1.0///predictionExplanations` `POST /predApi/v1.0/deployments//predictionExplanations` New route: `POST /predApi/v1.0/deployments//predictions` #### Ping Deprecated route: `GET /api/v1/ping` New route: `GET /predApi/v1.0/ping` #### List models Deprecated route: `GET /api/v1//models` New route: Use Public V2 API to fetch the list of models in a project #### Using tokens Deprecated routes: `GET /api/v1/api_token` `POST /api/v1/api_token` `GET /predApi/v1.0/api_token` New route: API tokens are superseded by API Keys and are managed by the public V2 API only. See the [UI platform documentation](https://docs.datarobot.com/en/docs/platform/acct-settings/api-key-mgmt.html) or the Account > API keys and tools section of the public V2 API documentation. ## Request examples for legacy Prediction API routes This section provides examples showing how to make predictions using legacy and soon to be disabled Prediction API routes directly on a model by specifying the model’s project and model ID. See the table above for a [deprecation timeline](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/deprecated-prediction-api.html#tracking-deprecated-routes-usage) based on release status. Generating predictions for classification and regression projects: ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0///predict" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` Generating predictions for time series projects: ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0///timeSeriesPredict" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` Generating Prediction Explanations: ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0///predictionExplanations" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` If you are using he managed AI Platform (SaaS), include the `datarobot-key` in the cURL header: `-H "datarobot-key: xxxx`. ## Tracking deprecated routes usage > [!NOTE] Availability information > This feature is not available for managed AI Platform (SaaS) users. Contact your DataRobot representative for information on handling migrations. > [!NOTE] Note > This feature is available in v6.1 only. In v6.2 it will be deleted along with all deprecated routes. For prediction admins convenience, it is possible to track all deprecated routes usage from a single page. This feature is only available to users who have "Enable Predictions Admin" permission enabled: Prediction admins can access it via Manage Predictions page: In order to access deprecated routes statistics click on the button in the top right corner: The table with statistics will look like this: The table has the following columns: - Last Used : the last time this request was made (UTC) - Request : HTTP request that was made. Includes query parameters, if any - Username : name of the DR user who made this request - Total Use Count : total number of times request was made since v6.1. Notes: 1. The table is sorted by Last Used descending, so that the most recent requests are shown on top. 2. Two different users making the same request are counted separately. 3. The total number of the most recent requests shown is limited by 100. 4. This table only shows requests to deprecated routes. Requests using new routes are not shown.. --- # Make predictions with the API URL: https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html > This section describes how to use DataRobot's Prediction API to make predictions on a dedicated prediction server. This section describes how to use DataRobot's Prediction API to make predictions on a dedicated prediction server. If you need Prediction API reference documentation, it is available [here](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/index.html). You can use DataRobot's Prediction API for making predictions on a model deployment (by specifying the deployment ID). This provides access to advanced [model management](https://docs.datarobot.com/en/docs/api/dev-learning/python/mlops/index.html) features like target or data drift detection. DataRobot's model management features are safely decoupled from the Prediction API so that you can gain their benefit without sacrificing prediction speed or reliability. See the [deployment](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/deploy-methods/index.html) section for details on creating a model deployment. Before generating predictions with the Prediction API, review the recommended [best practices](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html#best-practices-for-the-fastest-predictions) to ensure the fastest predictions. ## Making predictions To generate predictions on new data using the Prediction API, you need: - The model's deployment ID. You can find the ID in the sample code output of the Deployments > Predictions > Prediction API tab (with Interface set to "API Client"). - Your API key . > [!WARNING] Warning > If your model is an open-source R script, it will run considerably slower. Prediction requests are submitted as POST requests to the resource, for example: ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0/deployments//predictions" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` > [!NOTE] Availability information > Managed AI Platform (SaaS) users must include the `datarobot-key` in the cURL header (for example, `curl -H "Content-Type: application/json", -H "datarobot-key: xxxx"`). Find the key by displaying secrets on the [Predictions > Prediction API](https://docs.datarobot.com/en/docs/classic-ui/predictions/realtime/code-py.html) tab or by contacting your DataRobot representative. The order of the prediction response rows is the same as the order of the sent data. The Response returned is similar to: ``` HTTP/1.1 200 OK Content-Type: application/json X-DataRobot-Execution-Time: 38 X-DataRobot-Model-Cache-Hit: true {"data":[...]} ``` > [!NOTE] Note > The example above shows an arbitrary hostname ( `example.datarobot.com`) as the Prediction API URL; be sure to use the correct hostname of your dedicated prediction server. The configured (predictions) URL is displayed in the sample code of the [Deployments > Predictions > Prediction API](https://docs.datarobot.com/en/docs/classic-ui/predictions/realtime/code-py.html) tab. See your system administrator for more assistance if needed. ## Using persistent HTTP connections All prediction requests are served over a secure connection (SSL/TLS), which can result in significant connection setup time. Depending on your network latency to the prediction instance, this can be anywhere from 30ms to upwards of 100-150ms. To address this, the Prediction API supports HTTP Keep-Alive, enabling your systems to keep a connection open for up to a minute after the last prediction request. Using the Python `requests` module, run your prediction requests from `requests.Session`: ``` import json import requests data = [ json.dumps({'Feature1': 42, 'Feature2': 'text value 1'}), json.dumps({'Feature1': 60, 'Feature2': 'text value 2'}), ] api_key = '...' api_endpoint = '...' session = requests.Session() session.headers = { 'Authorization': 'Bearer {}'.format(api_key), 'Content-Type': 'text/json', } for row in data: print(session.post(api_endpoint, data=row).json()) ``` Check the documentation of your favorite HTTP library for how to use persistent connections in your integration. ## Prediction inputs The API supports both JSON- and CSV-formatted input data (although JSON can be a safer choice if it is created with a good quality JSON parser). Data can either be posted in the [request body](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html#request-schema) or via a [file upload](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html#file-input) (multipart form). > [!NOTE] Note > When using the Prediction API, the only supported column separator in CSV files and request bodies is the comma ( `,`). ### JSON input The JSON input is formatted as an array of objects where the key is the feature name and the value is the value in the dataset. For example, a CSV file that looks like: ``` a,b,c 1,2,3 7,8,9 ``` Would be represented in JSON as: ``` [ { "a": 1, "b": 2, "c": 3 }, { "a": 7, "b": 8, "c": 9 } ] ``` Submit a JSON array to the Prediction API by sending the data to the `/predApi/v1.0/deployments//predictions` endpoint. For example: ``` curl -H "Content-Type: application/json" -X POST --data '[{"a": 4, "b": 5, "c": 6}\]' \ -H "Authorization: Bearer " \ https://example.datarobot.com/predApi/v1.0/deployments//predictions ``` ### File input This example assumes a CSV file, `dataset.csv`, that contains a header and the rows of data to predict on. cURL automatically sets the content type. ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0/deployments//predictions" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv HTTP/1.1 200 OK Date: Fri, 08 Feb 2019 10:00:00 GMT Content-Type: application/json Content-Length: 60624 Connection: keep-alive Server: nginx/1.12.2 X-DataRobot-Execution-Time: 39 X-DataRobot-Model-Cache-Hit: true Access-Control-Allow-Methods: OPTIONS, POST Access-Control-Allow-Credentials: true Access-Control-Expose-Headers: Content-Type,Content-Length,X-DataRobot-Execution-Time,X-DataRobot-Model-Cache-Hit,X-DataRobot-Model-Id,X-DataRobot-Request-Id Access-Control-Allow-Headers: Content-Type,Authorization,datarobot-key X-DataRobot-Request-ID: 9e61f97bf07903b8c526f4eb47830a86 { "data": [ { "predictionValues": [ { "value": 0.2570950924, "label": 1 }, { "value": 0.7429049076, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 0, "rowId": 0 }, { "predictionValues": [ { "value": 0.7631880558, "label": 1 }, { "value": 0.2368119442, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 1, "rowId": 1 } ] } ``` ### In-body text input This example includes the CSV file content in the request body. With this format, you must set the Content-Type of the form data to `text/plain`. ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0/deployments//predictions" --data-binary $'a,b,c\n1,2,3\n7,8,9\n' -H "content-type: text/plain" \ -H "Authorization: Bearer " \ ``` ## Prediction outputs The Content-Type header value must be set appropriately for the type of data being sent ( `text/csv` or `application/json`); the raw API request responds with JSON by default. Reference the [output format](https://docs.datarobot.com/en/docs/api/reference/batch-prediction-api/output-format.html) for more information about the structure of the output schema. The output schema shares the same format for real-time and batch predictions. ### CSV output To return CSV in addition to JSON from the Prediction API for real-time predictions, use `-H "Accept: text/csv"`. ## Prediction objects The following sections describe the content of the various prediction objects. ### Request schema Note that Request schema are standard for any kind of predictions. The following are the accepted headers: | Name | Value(s) | | --- | --- | | Content-Type | text/csv;charset=utf8 | | | application/json | | | multipart/form-data | | Content-Encoding | gzip | | | bz2 | | Authorization | Bearer | Note the following: - If you are submitting predictions as a raw stream of data, you can specify an encoding by adding;charset=to theContent-Typeheader. See thePython standard encodingsfor a list of valid values. DataRobot usesutf8by default. - If you are sending an encoded stream of data, you should specify theContent-Encodingheader. - TheAuthorizationfield is a Bearer authentication HTTP authentication scheme that involves security tokens called bearer tokens. While it is possible to authenticate via pair username + API token (Basic auth) or just via API token, these authentication methods are deprecated and not recommended. You can parameterize a request using URI query parameters: | Parameter name | Type | Notes | | --- | --- | --- | | passthroughColumns | string | List of columns from a scoring dataset to return in the prediction response. | | passthroughColumnsSet | string | If passthroughColumnsSet=all is passed, all columns from the scoring dataset are returned in the prediction response. | Note the following: - The passthroughColumns and passthroughColumnsSet parameters cannot both be passed in the same request. - While there is no limit on the number of column names you can pass with the passthroughColumns query parameter, there is a limit on the HTTP request line (currently 8192 bytes). The following example illustrates the use of multiple passthrough columns: ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0/deployments//predictions?passthroughColumns=Latitude&passthroughColumns=Longitude" \ -H "Authorization: Bearer " \ -H "datarobot-key: " -F \ file=@~/.home/path/to/dataset.csv ``` ### Response schema The following is a sample prediction response body (also see the additional example of a [time series response body](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html#making-predictions-with-time-series)): ``` { "data": [ { "predictionValues": [ { "value": 0.6856798909, "label": 1 }, { "value": 0.3143201091, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 1, "rowId": 0, "passthroughValues": { "Latitude": -25.433508, "Longitude": 22.759397 } }, { "predictionValues": [ { "value": 0.765656753, "label": 1 }, { "value": 0.234343247, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 1, "rowId": 1, "passthroughValues": { "Latitude": 41.051128, "Longitude": 14.49598 } } ] } ``` The table below lists custom DataRobot headers: | Name | Value | Note | | --- | --- | --- | | X-DataRobot-Execution-Time | numeric | Time for compute predictions (ms). | | X-DataRobot-Model-Cache-Hit | true or false | Indication of in-memory presence of model (bool). | | X-DataRobot-Model-Id | ObjectId | ID of the model used to serve the prediction request (only returned for predictions made on model deployments). | | X-DataRobot-Request-Id | uuid | Unique identifier of a prediction request. | The following table describes the Response Prediction Rows of the JSON array: | Name | Type | Note | | --- | --- | --- | | predictionValues | array | An array of predictionValues (schema described below). | | predictionThreshold | float | The threshold used for predictions (applicable to binary classification projects only). | | prediction | float | The output of the model for this row. | | rowId | int | The row described. | | passthroughValues | object | A JSON object where key is a column name and value is a corresponding value for a predicted row from the scoring dataset. This JSON item is only returned if either passthroughColumns or passthroughColumnsSet is passed. | | adjustedPrediction | float | The exposure-adjusted output of the model for this row if the exposure was used during model building. The adjustedPrediction is included in responses if the request parameter excludeAdjustedPredictions is false. | | adjustedPredictionValues | array | An array of exposure-adjusted PredictionValue (schema described below). The adjustedPredictionValues is included in responses if the request parameter excludeAdjustedPredictions is false. | | predictionExplanations | array | An array of PredictionExplanations (schema described below). This JSON item is only returned with Prediction Explanations. | #### Prediction values schema The following table describes the `predictionValues` schema in the JSON Response array: | Name | Type | Note | | --- | --- | --- | | label | - | Describes what the model output corresponds to. For regression projects, it is the name of the target feature. For classification projects, it is a label from the target feature. | | value | float | The output of the prediction. For regression projects, it is the predicted value of the target. For classification projects, it is the probability associated with the label that is predicted to be most likely (implying a threshold of 0.5 for binary classification problems). | #### Extra custom model output schema > [!NOTE] Availability information > Additional output in prediction responses for custom models is off by default. Contact your DataRobot representative or administrator for information on enabling this feature. > > Feature flag: Enable Additional Custom Model Output in Prediction Responses In some cases, the prediction response from your model may contain extra model output. This is possible for [custom models with additional output columns](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/structured-custom-models.html#additional-output-columns) defined in the `score()` hook and for Generative AI (GenAI) models. The `score()` hook can return any number of extra columns, containing data of types `string`, `int`, `float`, `bool`, or `datetime`. When additional columns are returned through the `score()` method, the prediction response is as follows: - For a tabular response (CSV) , the additional columns are returned as part of the response table or dataframe. - For a JSON response , the extraModelOutput key is returned alongside each row. This key is a dictionary containing the values of each additional column in the row. As custom models, deployed GenAI models can return extra columns through the `extraModelOutput` key to provide information about the text generation model (citations, latency, confidence, LLM blueprint ID, token counts, etc.), as shown in the example below: > [!NOTE] Citations in prediction response > For citations to be included in a Gen AI model's prediction response, the [LLM deployed from the playground](https://docs.datarobot.com/en/docs/agentic-ai/playground-tools/deploy-llm.html) must have a [vector database (VDB)](https://docs.datarobot.com/en/docs/agentic-ai/vector-database/vector-dbs.html) associated with it. ``` # JSON response for GenAI model predictions { "data": [ { "rowId": 0, "prediction": "In the field of biology, there have been some exciting new discoveries made through research conducted on the International Space Station (ISS). Here are three examples:\n\n1. Understanding Plant Root Orientation: Scientists have been studying the growth and development of plants in microgravity. They found that plants grown in space exhibit different root orientation compared to those grown on Earth. This discovery helps us understand how plants adapt and respond to the absence of gravity. This knowledge can be applied to improve agricultural practices and develop innovative techniques for growing plants in challenging environments on Earth.\n\n2. Tissue Damage and Repair: One fascinating area of research on the ISS involves studying how living organisms respond to injuries in space. Scientists have investigated tissue damage and repair mechanisms in various organisms, including humans. By studying the healing processes in microgravity, researchers gained insights into how wounds heal differently in space compared to on Earth. This knowledge has implications for developing new therapies and treatments for wound healing and tissue regeneration.\n\n3. Bubbles, Lightning, and Fire Dynamics: The ISS provides a unique laboratory environment for studying the behavior of bubbles, lightning, and fire in microgravity. Scientists have conducted experiments to understand how these phenomena behave differently without the influence of gravity. These studies have practical applications, such as improving combustion processes, enhancing fire safety measures, and developing more efficient cooling systems.\n\nThese are just a few examples of the exciting discoveries that have been made in the field of biology through research conducted on the ISS. The microgravity environment of space offers a unique perspective and enables researchers to uncover new insights into the workings of living organisms and their interactions with the environment.", "predictionValues": [ { "label": "resultText", "value": "In the field of biology, there have been some exciting new discoveries made through research conducted on the International Space Station (ISS). Here are three examples:\n\n1. Understanding Plant Root Orientation: Scientists have been studying the growth and development of plants in microgravity. They found that plants grown in space exhibit different root orientation compared to those grown on Earth. This discovery helps us understand how plants adapt and respond to the absence of gravity. This knowledge can be applied to improve agricultural practices and develop innovative techniques for growing plants in challenging environments on Earth.\n\n2. Tissue Damage and Repair: One fascinating area of research on the ISS involves studying how living organisms respond to injuries in space. Scientists have investigated tissue damage and repair mechanisms in various organisms, including humans. By studying the healing processes in microgravity, researchers gained insights into how wounds heal differently in space compared to on Earth. This knowledge has implications for developing new therapies and treatments for wound healing and tissue regeneration.\n\n3. Bubbles, Lightning, and Fire Dynamics: The ISS provides a unique laboratory environment for studying the behavior of bubbles, lightning, and fire in microgravity. Scientists have conducted experiments to understand how these phenomena behave differently without the influence of gravity. These studies have practical applications, such as improving combustion processes, enhancing fire safety measures, and developing more efficient cooling systems.\n\nThese are just a few examples of the exciting discoveries that have been made in the field of biology through research conducted on the ISS. The microgravity environment of space offers a unique perspective and enables researchers to uncover new insights into the workings of living organisms and their interactions with the environment." } ], "deploymentApprovalStatus": "APPROVED", "extraModelOutput": { "CITATION_CONTENT_8": "3\nthe research study is received by others and how the \nknowledge is disseminated through citations in other \njournals. For example, six ISS studies have been \npublished in Nature, represented as a small node in the \ngraph. Network analysis shows that findings published \nin Nature are likely to be cited by other similar leading \njournals such as Science and Astrophysical Journal \nLetters (represented in bright yellow links) as well as \nspecialized journals such as Physical Review D and New \nJournal of Physics (represented in a yellow-green link). \nSix publications in Nature led to 512 citations according \nto VOSviewer\u2019s network map (version 1.6.11), an \nincrease of over 8,000% from publication to citation. \nFor comparison purposes, 6 publications in a small \njournal like American Journal of Botany led to 185 \ncitations and 107 publications in Acta Astronautica, \na popular journal among ISS scientists, led to 1,050 \ncitations (Figure 3, panel B). This count of 1,050", "CITATION_CONTENT_9": "Introduction\n4\nFigure 3. Count of publications reported in journals ranked in the top 100 according to global standards of Clarivate. A total of 567 top-tier publications \nthrough the end of FY-23 are shown by year and research category.\nIn this year\u2019s edition of the Annual Highlights of Results, we report findings from a \nwide range of topics in biology and biotechnology, physics, human research, Earth and \nspace science, and technology development \u2013 including investigations about plant root \norientation, tissue damage and repair, bubbles, lightning, fire dynamics, neutron stars, \ncosmic ray nuclei, imaging technology improvements, brain and vascular health, solar \npanel materials, grain flow, as well as satellite and robot control. \nThe findings highlighted here are only a small sample representative of the research \nconducted by the participating space agencies \u2013 ASI (Agenzia Spaziale Italiana), CSA \n(Canadian Space Agency), ESA (European Space Agency), JAXA (Japanese Aerospace", "CITATION_PAGE_3": 4, "CITATION_PAGE_8": 6, "CITATION_CONTENT_5": "23\nPUBLICATION HIGHLIGHTS: \nEARTH AND SPACE SCIENCE\nThe ISS laboratories enable scientific experiments in the biological sciences \nthat explore the complex responses of living organisms to the microgravity \nenvironment. The lab facilities support the exploration of biological systems \nranging from microorganisms and cellular biology to integrated functions \nof multicellular plants and animals. Several recent biological sciences \nexperiments have facilitated new technology developments that allow \ngrowth and maintenance of living cells, tissues, and organisms.\nThe Alpha Magnetic \nSpectrometer-02 (AMS-02) is \na state-of-the-art particle \nphysics detector constructed, \ntested, and operated by an \ninternational team composed \nof 60 institutes from \n16 countries and organized \nunder the United States \nDepartment of Energy (DOE) sponsorship. \nThe AMS-02 uses the unique environment of \nspace to advance knowledge of the universe \nand lead to the understanding of the universe\u2019s", "CITATION_SOURCE_5": "Space_Station_Annual_Highlights/iss_2017_highlights.pdf", "CITATION_CONTENT_3": "Introduction\n2\nExtensive international collaboration in the \nunique environment of LEO as well as procedural \nimprovements to assist researchers in the collection \nof data from the ISS have produced promising \nresults in the areas of protein crystal growth, tissue \nregeneration, vaccine and drug development, 3D \nprinting, and fiber optics, among many others. In \nthis year\u2019s edition of the Annual Highlights of Results, \nwe report findings from a wide range of topics in \nbiotechnology, physics, human research, Earth \nand space science, and technology development \n\u2013 including investigations about human retinal cells, \nbacterial resistance, black hole detection, space \nanemia, brain health, Bose-Einstein condensates, \nparticle self-assembly, RNA extraction technology, \nand more. The findings highlighted here represent \nonly a sample of the work ISS has contributed to \nsociety during the past 12 months.\nAs of Oct. 1, 2022, we have identified a total of 3,679", "CITATION_SOURCE_8": "Space_Station_Annual_Highlights/iss_2021_highlights.pdf", "CITATION_PAGE_7": 8, "CITATION_PAGE_6": 8, "CITATION_PAGE_2": 4, "CITATION_CONTENT_7": "Biology and Biotechnology Earth and Space Science Educational and Cultural Activities\nHuman Research Physical Science Technology Development and Demonstration", "CITATION_SOURCE_9": "Space_Station_Annual_Highlights/iss_2023_highlights.pdf", "datarobot_latency": 3.1466632366, "blocked_resultText": false, "CITATION_SOURCE_2": "Space_Station_Annual_Highlights/iss_2023_highlights.pdf", "CITATION_SOURCE_6": "Space_Station_Annual_Highlights/iss_2021_highlights.pdf", "CITATION_SOURCE_7": "Space_Station_Annual_Highlights/iss_2023_highlights.pdf", "datarobot_confidence_score": 0.6524822695, "CITATION_PAGE_9": 7, "CITATION_CONTENT_4": "Molecular Life Sciences. 2021 October 29; DOI: \n10.1007/s00018-021-03989-2.\nFigure 7. Immunoflourescent images of human retinal \ncells in different conditions. Image adopted from \nCialdai, Cellular and Molecular Life Sciences.\nThe ISS laboratory provides a platform for investigations in the biological sciences that \nexplores the complex responses of living organisms to the microgravity environment. Lab \nfacilities support the exploration of biological systems, from microorganisms and cellular \nbiology to the integrated functions of multicellular plants and animals.", "CITATION_SOURCE_1": "Space_Station_Annual_Highlights/iss_2023_highlights.pdf", "CITATION_SOURCE_0": "Space_Station_Annual_Highlights/iss_2018_highlights.pdf", "CITATION_SOURCE_3": "Space_Station_Annual_Highlights/iss_2022_highlights.pdf", "CITATION_PAGE_5": 26, "CITATION_PAGE_0": 7, "CITATION_PAGE_1": 11, "LLM_BLUEPRINT_ID": "662ba0062ade64c4fc4c1a1f", "CITATION_PAGE_4": 9, "datarobot_token_count": 320, "CITATION_CONTENT_0": "more effectively in space by addressing \nsuch topics as understanding radiation effects on \ncrew health, combating bone and muscle loss, \nimproving designs of systems that handle fluids \nin microgravity, and determining how to maintain \nenvironmental control efficiently. \nResults from the ISS provide new \ncontributions to the body of scientific \nknowledge in the physical sciences, life \nsciences, and Earth and space sciences \nto advance scientific discoveries in multi\u0002disciplinary ways. \nISS science results have Earth-based \napplications, including understanding our \nclimate, contributing to the treatment of \ndisease, improving existing materials, and inspiring \nthe future generation of scientists, clinicians, \ntechnologists, engineers, mathematicians, artists, \nand explorers.\nBENEFITS\nFOR HUMANITY\nDISCOVERY\nFigure 4. A heat map of all of the countries whose authors have cited scientific results publications from ISS Research through October 1, 2018.\nEXPLORATION", "CITATION_SOURCE_4": "Space_Station_Annual_Highlights/iss_2022_highlights.pdf", "CITATION_CONTENT_2": "capabilities (i.e., facilities), and data delivery are critical to the effective operation \nof scientific projects for accurate results to be shared with the scientific community, \nsponsors, legislators, and the public. \nOver 3,700 investigations have operated since Expedition 1, with more than 250 active \nresearch facilities, the participation of more than 100 countries, the work of more than \n5,000 researchers, and over 4,000 publications. The growth in research (Figure 1) and \ninternational collaboration (Figure 2) has prompted the publication of over 560 research \narticles in top-tier scientific journals with about 75 percent of those groundbreaking studies \noccurring since 2018 (Figure 3). \nBibliometric analyses conducted through VOSviewer1\n measure the impact of space station \nresearch by quantifying and visualizing networks of journals, citations, subject areas, and \ncollaboration between authors, countries, or organizations. Using bibliometrics, a broad", "CITATION_CONTENT_1": "technologists, engineers, mathematicians, artists, and explorers.\nEXPLORATION\nDISCOVERY\nBENEFITS\nFOR HUMANITY", "CITATION_CONTENT_6": "control efficiently. \nResults from the ISS provide new \ncontributions to the body of scientific \nknowledge in the physical sciences, life \nsciences, and Earth and space sciences \nto advance scientific discoveries in multi\u0002disciplinary ways. \nISS science results have Earth-based \napplications, including understanding our \nclimate, contributing to the treatment of \ndisease, improving existing materials, and \ninspiring the future generation of scientists, \nclinicians, technologists, engineers, \nmathematicians, artists and explorers.\nBENEFITS\nFOR HUMANITY\nDISCOVERY\nEXPLORATION" } }, ] } ``` ## Making predictions with time series > [!TIP] Tip > Time series predictions are specific to time series projects, not all time-aware modeling projects. Specifically, the CSV file must follow a specific format, described in the [predictions section](https://docs.datarobot.com/en/docs/classic-ui/modeling/time/ts-predictions.html#make-predictions-tab) of the time series modeling pages. If you are making predictions with the forecast point, you can skip the forecast window in your prediction data as DataRobot generates a forecast point automatically. This is called autoexpansion. Autoexpansion applies automatically if: - Predictions are made for a specific forecast point and not a forecast range. - The time series project has a regular time step and does not use Nowcasting. When using autoexpansion, note the following: - If you have Known in Advance features that are important for your model, it is recommended that you manually create a forecast window to increase prediction accuracy. - If you plan to use an association ID other than the primary date/time column in your deployment to track accuracy, create a forecast window manually. The URL for making predictions with time series deployments and regular non-time series deployments is the same. The only difference is that you can optionally specify forecast point, prediction start/end date, or some other time series specific URL parameters. Using the deployment ID, the server automatically detects the deployed model as a time series deployment and processes it accordingly: ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0/deployments//predictions" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` The following is a sample Response body for a multiseries project: ``` HTTP/1.1 200 OK Content-Type: application/json X-DataRobot-Execution-Time: 1405 X-DataRobot-Model-Cache-Hit: false { "data": [ { "seriesId": 1, "forecastPoint": "2018-01-09T00:00:00Z", "rowId": 365, "timestamp": "2018-01-10T00:00:00.000000Z", "predictionValues": [ { "value": 45180.4041874386, "label": "target (actual)" } ], "forecastDistance": 1, "prediction": 45180.4041874386 }, { "seriesId": 1, "forecastPoint": "2018-01-09T00:00:00Z", "rowId": 366, "timestamp": "2018-01-11T00:00:00.000000Z", "predictionValues": [ { "value": 47742.9432499386, "label": "target (actual)" } ], "forecastDistance": 2, "prediction": 47742.9432499386 }, { "seriesId": 1, "forecastPoint": "2018-01-09T00:00:00Z", "rowId": 367, "timestamp": "2018-01-12T00:00:00.000000Z", "predictionValues": [ { "value": 46394.5698978878, "label": "target (actual)" } ], "forecastDistance": 3, "prediction": 46394.5698978878 }, { "seriesId": 2, "forecastPoint": "2018-01-09T00:00:00Z", "rowId": 697, "timestamp": "2018-01-10T00:00:00.000000Z", "predictionValues": [ { "value": 39794.833199375, "label": "target (actual)" } ] } ] } ``` ### Request parameters You can parameterize the time series prediction request using URI query parameters. For example, overriding the default inferred forecast point can look like this: ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0/deployments//predictions?forecastPoint=1961-01-01T00:00:00?relaxKnownInAdvanceFeaturesCheck=true" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` For the full list of time series-specific parameters, see [Time series predictions for deployments](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/time-pred.html). ### Response schema The [Response schema](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html#response-schema_1) is consistent with [standard predictions](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html#response-schema_2) but adds a number of columns for each `PredictionRow` object: | Name | Type | Notes | | --- | --- | --- | | seriesId | string, int, or None | A multiseries identifier of a predicted row that identifies the series in a multiseries project. | | forecastPoint | string | An ISO 8601 formatted DateTime string corresponding to the forecast point for the prediction request, either user-configured or selected by DataRobot. | | timestamp | string | An ISO 8601 formatted DateTime string corresponding to the DateTime column of the predicted row. | | forecastDistance | int | A forecast distance identifier of the predicted row, or how far it is from forecastPoint in the scoring dataset. | | originalFormatTimestamp | string | A DateTime string corresponding to the DateTime column of the predicted row. Unlike the timestamp column, this column will keep the same DateTime formatting as the uploaded prediction dataset. (This column is shown if enabled by your administrator.) | ## Making Prediction Explanations The DataRobot [Prediction Explanations](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/index.html) feature gives insight into which attributes of a particular input cause it to have exceptionally high or exceptionally low predicted values. > [!TIP] Tip > You must run the following two critical dependencies before running Prediction Explanations: > > You must compute > Feature Impact > for the model. > You must generate predictions on the dataset using the selected model. To initialize Prediction Explanations, use the [Prediction Explanations](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/index.html) tab. Making Prediction Explanations is very similar to standard prediction requests. First, Prediction Explanations requests are submitted as POST requests to the resource: ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0/deployments//predictionExplanations" \ -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` The following is a sample Response body: ``` HTTP/1.1 200 OK Content-Type: application/json X-DataRobot-Execution-Time: 841 X-DataRobot-Model-Cache-Hit: true { "data": [ { "predictionValues": [ { "value": 0.6634830442, "label": 1 }, { "value": 0.3365169558, "label": 0 } ], "prediction": 1, "rowId": 0, "predictionExplanations": [ { "featureValue": 49, "strength": 0.6194461777, "feature": "driver_age", "qualitativeStrength": "+++", "label": 1 }, { "featureValue": 1, "strength": 0.3501610895, "feature": "territory", "qualitativeStrength": "++", "label": 1 }, { "featureValue": "M", "strength": -0.171075409, "feature": "gender", "qualitativeStrength": "--", "label": 1 } ] }, { "predictionValues": [ { "value": 0.3565584672, "label": 1 }, { "value": 0.6434415328, "label": 0 } ], "prediction": 0, "rowId": 1, "predictionExplanations": [] } ] } ``` ### Request parameters You can parameterize the Prediction Explanations prediction request using URI query parameters: | Parameter name | Type | Notes | | --- | --- | --- | | maxExplanations | int | Maximum number of codes generated per prediction. Default is 3. Previously called maxCodes. | | thresholdLow | float | Prediction Explanation low threshold. Predictions must be below this value (or above the thresholdHigh value) for Prediction Explanations to compute. This value can be null. | | thresholdHigh | float | Prediction Explanation high threshold. Predictions must be above this value (or below the thresholdLow value) for Prediction Explanations to compute. This value can be null. | | excludeAdjustedPredictions | string | Includes or excludes exposure-adjusted predictions in prediction responses if exposure was used during model building. The default value is 'true' (exclude exposure-adjusted predictions). | The following is an example of a parameterized request: ``` curl -i -X POST "https://example.datarobot.com/predApi/v1.0/deployments//predictionExplanations?maxExplanations=2&thresholdLow=0.2&thresholdHigh=0.5" -H "Authorization: Bearer " -F \ file=@~/.home/path/to/dataset.csv ``` DataRobot's headers schema is the same as that for prediction responses. The [Response schema](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html#response-schema_2) is consistent with standard predictions, but adds "predictionExplanations", an array of `PredictionExplanations` for each `PredictionRow` object. #### PredictionExplanations schema Response JSON Array of Objects: | Name | Type | Notes | | --- | --- | --- | | label | – | Describes which output was driven by this Prediction Explanation. For regression projects, it is the name of the target feature. For classification projects, it is the class whose probability increasing would correspond to a positive strength of this Prediction Explanation. | | feature | string | Name of the feature contributing to the prediction. | | featureValue | - | Value the feature took on for this row. | | strength | float | Amount this feature’s value affected the prediction. | | qualitativeStrength | string | Human-readable description of how strongly the feature affected the prediction (e.g., +++, –, +). | > [!TIP] Tip > The prediction explanation `strength` value is not bounded to the values `[-1, 1]`; its interpretation may change as the number of features in the model changes. For normalized values, use `qualitativeStrength` instead.`qualitativeStrength` expresses the `[-1, 1]` range with visuals, with `---` representing `-1` and `+++` representing `1`. For explanations with the same `qualitativeStrength`, you can then use the `strength` value for ranking. > > See the section on [interpreting Prediction Explanation output](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/xemp-pe.html#interpret-xemp-prediction-explanations) for more information. ## Making predictions with humility monitoring Predictions with [humility monitoring](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment-settings/humility-settings.html) allow you to monitor predictions using user-defined humility rules. When a prediction falls outside the thresholds provided for the "Uncertain Prediction" Trigger, it will default to the action assigned to the trigger. The humility key is added to the body of the prediction response when the trigger is activated. The following is a sample Response body for a Regression project with an `Uncertain Prediction Trigger` with `Action - No Operation`: ``` { "data": [ { "predictionValues": [ { "value": 122.8034057617, "label": "length" } ], "prediction": 122.8034057617, "rowId": 99, "humility": [ { "ruleId": "5ebad4735f11b33a38ff3e0d", "triggered": true, "ruleName": "Uncertain Prediction Trigger" } ] } ] } ``` The following is an example of a Response body for a regression model deployment. It uses the "Uncertain Prediction" trigger with the "Throw Error" action: ``` 480 Error: {"message":"Humility ReturnError action triggered."} ``` The following is an example of a Response body for a regression model deployment. It uses the "Uncertain Prediction" trigger with the "Override Prediction" action: ``` { "data": [ { "predictionValues": [ { "value": 122.8034057617, "label": "length" } ], "prediction": 5220, "rowId": 99, "humility": [ { "ruleId": "5ebad4735f11b33a38ff3e0d", "triggered": true, "ruleName": "Uncertain Prediction Trigger" } ] } ] } ``` ### Response schema The [response schema](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html#response-schema_2) is consistent with [standard predictions](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html#response-schema) but adds a new humility column with a subset of columns for each `Humility` object: | Name | Type | Notes | | --- | --- | --- | | ruleId | string | The ID of the humility rule assigned to the deployment | | triggered | boolean | Returns "True" or "False" depending on if the rule was triggered or not | | ruleName | string | The name of the rule that is either defined by the user or auto-generated with a timestamp | ## Error responses Any error is indicated by a non-200 code attribute. Codes starting with 4XX indicate request errors (e.g., missing columns, wrong credentials, unknown model ID). The message attribute gives a detailed description of the error in the case of a 4XX code. For example: ``` curl -H "Content-Type: application/json" -X POST --data '' \ -H "Authorization: Bearer " \ https://example.datarobot.com/predApi/v1.0/deployments/ HTTP/1.1 400 BAD REQUEST Date: Fri, 08 Feb 2019 11:00:00 GMT Content-Type: application/json Content-Length: 53 Connection: keep-alive Server: nginx/1.12.2 X-DataRobot-Execution-Time: 332 X-DataRobot-Request-ID: fad6a0b62c1ff30db74c6359648d12fd { "message": "The requested URL was not found on the server. If you entered the URL manually, please check your spelling and try again." } ``` Codes starting with 5XX indicate server-side errors. Retry the request or contact your DataRobot representative. ## Knowing the limitations The following describes the size and timeout boundaries for real-time deployment predictions: - Maximum data submission size is 50MB. - There is no limit on the number of rows, but timeout limits are as follows: If your request exceeds the timeout, or you are trying to score a large file using dedicated predictions, consider using thebatch scoring package. - There is a limit on the size of theHTTP request line(currently 8192 bytes). - For managed AI Platform deployments, dedicated Prediction API servers automatically close persistent HTTP connections if they are idle for more than 600 seconds. To use persistent connections, the client side must be able to handle these disconnects correctly. The following example configures Python HTTP libraryrequeststo automatically retry HTTP requests on transport failure: ``` import requests import urllib3 # create a transport adapter that will automatically retry GET/POST/HEAD requests on failures up to 3 times adapter = requests.adapters.HTTPAdapter( max_retries=urllib3.Retry( total=3, method_whitelist=frozenset(['GET', 'POST', 'HEAD']) ) ) # create a Session (a pool of connections) and make it use the given adapter for HTTP and HTTPS requests session = requests.Session() session.mount('http://', adapter) session.mount('https://', adapter) # execute a prediction request that will be retried on transport failures, if needed api_token = '' dr_key = '' response = session.post( 'https://example.datarobot.com/predApi/v1.0/deployments//predictions', headers={ 'Authorization': 'Bearer %s' % api_token, 'DataRobot-Key': dr_key, 'Content-Type': 'text/csv', }, data='', ) print(response.content) ``` ### Model caching The dedicated prediction server fetches models, as needed, from the DataRobot cluster. To speed up subsequent predictions that use the same model, DataRobot stores a certain number of models in memory (cache). When the cache fills, each new model request will require that one of the existing models in the cache be removed. DataRobot removes the least recently used model (which is not necessarily the model that has been in the cache the longest). For Self-Managed AI Platform installations, the default size for the cache is 16 models, but it can vary from installation to installation. Please contact DataRobot support if you have questions regarding the cache size of your specific installation. A prediction server runs multiple prediction processes, each of which has its own exclusive model cache. Prediction processes do not share between themselves. Because of this, it is possible that you send two consecutive requests to a prediction server, and each has to download the model data. Each response from the prediction server includes a header, `X-DataRobot-Model-Cache-Hit`, indicating whether the model used was in the cache. If the model was in the cache, the value of the header is true; if the value is false, the model was not in the cache. ## Best practices for the fastest predictions The following checklist summarizes the suggestions above to help deliver the fastest predictions possible: - Implementpersistent HTTP connections: This reduces network round-trips, and thus latency, to the Prediction API. - Use CSV data:Because JSON serialization of large amounts of data can take longer than using CSV, consider using CSV for yourprediction inputs. - Keep the number of requested models low:This allows the Prediction API to make use ofmodel caching. - Batch data together in chunks:Batch as many rows together as possible without going over the50MB real-time deployment prediction request limit. If scoring larger files, consider using theBatch Prediction APIwhich, in addition to scoring local files, also supports scoring to and from S3 and databases. --- # Dedicated Prediction API URL: https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/index.html > DataRobot's Prediction API provides a mechanism for using your model for real-time predictions on a prediction server. DataRobot's Prediction API provides a mechanism for using your model for real-time predictions on a dedicated external application server. Follow [the guidelines](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html) for making predictions with the Prediction API. You can also review how to [retrieve a prediction server ID](https://docs.datarobot.com/en/docs/api/reference/predapi/pred-server-id.html) using cURL commands from the REST API or by using the DataRobot Python client to make predictions with a deployment. | Topic | Description | | --- | --- | | Make predictions with the API | Make predictions on a dedicated prediction server. | | Make predictions with the Python API client | Use the DataRobot Prediction Library, a Python library for making predictions with various prediction methods. | | Dedicated Prediction API reference | Review Prediction API methods, input and output parameters, and errors. | | Get a prediction server ID | Retrieve a prediction server ID using cURL commands from the REST API or the DataRobot Python client. | | Deprecated API routes | Review deprecated Prediction API routes in a reference document listing the deprecated requests and their replacements. | --- # Predictions for unstructured model deployments URL: https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/dep-pred-unstructured.html > Using a specified endpoint, calculates predictions based on user-provided data for a specific unstructured model deployment. Using the endpoint below, you can provide the data necessary to calculate predictions for a specific unstructured model deployment. If you need to make predictions for a standard model, see [Predictions for deployments](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/dep-pred.html). Endpoint: `/deployments//predictionsUnstructured` Calculates predictions based on user-provided data for a specific unstructured model deployment. This endpoint works only for deployed custom inference models with an unstructured target type. For more information, see [Assemble unstructured custom models](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/unstructured-custom-models.html). This endpoint does the following: - Calls the/predictUnstructuredroute on the target custom inference model, allowing you to use the custom request and response schema, which may go beyond the standard DataRobot prediction API interface. - Passes any payload and content type (MIME type and charset, if provided) to the model. - Passes any model-returned payload, along with the content type (MIME type and charset, if provided), back to the caller. In the [DRUM library](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-drum.html), this call is handled by the [score_unstructured()hook](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/unstructured-custom-models.html#score). > [!NOTE] Note > You can find the deployment ID in the sample code output of the [Deployments > Predictions > Prediction API](https://docs.datarobot.com/en/docs/classic-ui/predictions/realtime/code-py.html) tab (with Interface set to API Client). Request Method: `POST` Request URL: deployed URL, for example: `https://your-company.orm.datarobot.com/predApi/v1.0` ## Request parameters ### Headers | Key | Description | Example(s) | | --- | --- | --- | | Datarobot-key | Required for managed AI Platform users; string type Once a model is deployed, see the code snippet in the DataRobot UI, Predictions > Prediction API. | DR-key-12345abcdb-xyz6789 | | Authorization | Required; string Three methods are supported: Bearer authentication (deprecated) Basic authentication: User_email and API token (deprecated) API token | Example for Bearer authentication method: Bearer API_key-12345abcdb-xyz6789(deprecated) Example for User_email and API token method: Basic Auth_basic-12345abcdb-xyz6789(deprecated) Example for API token method: Token API_key-12345abcdb-xyz6789 | | Content-Type | Optional; string type Default: application/octet-stream Any provided content type is passed to the model; however, the DRUM library has a built-in decoding mechanism for text content-types using the specified charset. For more information, see Assemble unstructured custom models. | text/plaintext/csvtext/plain; charset=latin1application/json; charset=UTF-8custom/typeapplication/octet-stream | | Content-Encoding | Optional; string type Currently supports only gzip-encoding with the default data extension. | gzip | | Accept | Optional; string type | */* (default) The response is defined by the model output. | | | | | ### Query arguments Currently not supported for the `predictionsUnstructured` endpoint. ### Body | Data | Type | Example(s) | | --- | --- | --- | | Data to pass to the custom model | Bytes | PassengerId,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked 892,3,"Kelly, Mr. James",male,34.5,0,0,330911,7.8292,,Q 893,3,"Wilkes, Mrs. James (Ellen Needs)",female,47,1,0,363272,7,,S 894,2,"Myles, Mr. Thomas Francis",male,62,0,0,240276,9.6875,,Q{“data”: [{“some”: “json”}]}Custom payload 123 (for example, image data) | ## Response 200 The HTTP Response contains a payload returned by the custom model’s `/predictUnstructured` route and passed back as-is. The `Content-Type` header is passed to the caller. If the `Content-Type` header isn't provided, the `application/octet-stream` default is applied. In the case of a DataRobot-acknowledged error in a request, an `application/json` error message is returned. In the [DRUM library](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/custom-model-drum.html), the response payload and content type are generated by the [score_unstructured()hook](https://docs.datarobot.com/en/docs/api/code-first-tools/drum/unstructured-custom-models.html#score). ## Errors list | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 400 BAD REQUEST | {"message":"Query parameters not accepted on this endpoint"} | The request passed query parameters to the endpoint. | | 404 NOT FOUND | {"message": "Deployment :deploymentId cannot be found for user :userId"} | The request provided an invalid :deploymentId (a deleted or non-existent deployment). | | 422 UNPROCESSABLE CONTENT | {"message": "Only unstructured custom models can be used with this endpoint. Use /predictions instead.} | The request provided a :deploymentId for a deployment that isn't an unstructured custom inference model deployment. | --- # Predictions for deployments URL: https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/dep-pred.html > Using a specified endpoint, calculates predictions based on user-provided data for a specific deployment. Using the endpoint below, you can provide the data necessary to calculate predictions for a specific deployment. If you need to make predictions for an unstructured custom inference model, see [Predictions for unstructured model deployments](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/dep-pred-unstructured.html). Endpoint: `/deployments//predictions` Calculates predictions based on user-provided data for a specific deployment. Note that this endpoint works only for deployed models. > [!NOTE] Note > You can find the deployment ID in the sample code output of the [Deployments > Predictions > Prediction API](https://docs.datarobot.com/en/docs/classic-ui/predictions/realtime/code-py.html) tab (with Interface set to API Client). Request Method: `POST` Request URL: deployed URL, for example: `https://your-company.orm.datarobot.com/predApi/v1.0` ## Request parameters ### Headers | Key | Description | Example(s) | | --- | --- | --- | | Datarobot-key | A per-organization secret used as an additional authentication factor for prediction servers. Retrieve a datarobot-key programmatically by accessing the /api/v2/predictionServers/ endpoint. The endpoint returns a URL to a prediction server and a corresponding datarobot-key. Required for Self-Managed AI Platform users; string type Once a model is deployed, see the code snippet in the DataRobot UI, Predictions > Prediction API. | DR-key-12345abcdb-xyz6789 | | Authorization | Required; string Three methods are supported: Bearer authentication (deprecated) Basic authentication: User_email and API token (deprecated) API token | Example for Bearer authentication method: Bearer API_key-12345abcdb-xyz6789 (deprecated) Example for User_email and API token method: Basic Auth_basic-12345abcdb-xyz6789 (deprecated) Example for API token method: Token API_key-12345abcdb-xyz6789 | | Content-Type | Optional; string type | text/plain; charset=UTF-8text/csvapplication/jsonmultipart/form-data (for files with data, i.e., .csv, .txt files) | | Content-Encoding | Optional; string type Currently supports only gzip-encoding with the default data extension. | gzip | | Accept | Optional; string type Controls the shape of the response schema. Currently JSON(default) and CSV are supported. See examples. | application/json (default)text/csv (for CSV output) | Datarobot-key: This header is required only with the managed AI Platform. It is used as a precaution to secure user data from other verified DataRobot users. The key can also be retrieved with the following request to the DataRobot API: `GET /api/v2/modelDeployments/` ### Query arguments | Key | Type | Description | Example(s) | | --- | --- | --- | --- | | passthroughColumns | list of strings | (Optional) Controls which columns from a scoring dataset to expose (or copy over) in a prediction response. The request may contain zero, one, or more columns. (There’s no limit on how many column names you can pass.) Column names must be passed as UTF-8 bytes and be percent-encoded (see the HTTP standard for this requirement). Make sure to use the exact name of a column as a value. | /v1.0/deployments//predictions?passthroughColumns=colA&passthroughColumns=colB | | passthroughColumnsSet | string | (Optional) Controls which columns from a scoring dataset to expose (or to copy over) in a prediction response. The only possible option is all and, if passed, all columns from a scoring dataset are exposed. | /v1.0/deployments/deploymentId/predictions?passthroughColumnsSet=all | | predictionWarningEnabled | bool | (Optional) DataRobot monitors unusual or anomalous predictions in real-time and indicates when they are detected. If this argument is set to true, a new key is added to each prediction to specify the result of the Humility check. Otherwise, there are no changes in the prediction response. | /v1.0/deployments/deploymentId/predictions?predictionWarningEnabled=true Response: { "data": [ { "predictionValues": [ { "value": 18.6948852, "label": "y" } ], "isOutlierPrediction": false, "rowId": 0, "prediction": 18.6948852 } ] } | | decimalsNumber | integer | (Optional) Configures the float precision in prediction results by setting the number of digits after the decimal point. If there aren't any digits after the decimal point, rather than adding zeros, the float precision is less than the value set by decimalsNumber. | ?decimalsNumber=15 | > [!NOTE] Note > The `passthroughColumns` and `passthroughColumnsSet` parameters are mutually exclusive and cannot both be passed in the same request. Also, while there isn't a limit on the number of column names you can pass with the `passthroughColumns` query parameter, there is a limit on the size of the [HTTP request line](https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1) (currently 8192 bytes). ### Body | Data | Type | Example(s) | | --- | --- | --- | | Data to predict | raw text form-data | PassengerId,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked 892,3,"Kelly, Mr. James",male,34.5,0,0,330911,7.8292,,Q 893,3,"Wilkes, Mrs. James (Ellen Needs)",female,47,1,0,363272,7,,S 894,2,"Myles, Mr. Thomas Francis",male,62,0,0,240276,9.6875,,Q Key: file, value: file_with_data_to_predict.csv | ## Response 200 ### Binary prediction Label: For regression and binary classification tasks, DataRobot API always returns 1 for the positive class and 0 for the negative class. Although the actual values for the classes may be different depending on the data provided (like "yes"/"no"), the DataRobot API will always return 1/0. For multiclass classification, the DataRobot API returns the value itself. Value: Shows the probability of an event happening (where 0 and 1 are min and max probability, respectively). The user can adjust the threshold that links the value with the prediction label. PredictionThreshold ( Applicable to binary classification projects only): Threshold is the point that sets the class boundary for a predicted value. The model classifies an observation below the threshold as FALSE, and an observation above the threshold as TRUE. In other words, DataRobot automatically assigns the positive class label to any prediction exceeding the threshold. This can be configured manually through the UI (the [Deploytab](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/deploy-methods/deploy-model.html#deploy-from-the-leaderboard)), or through the DataRobot API (i.e., the `PATCH /api/v2/projects/(projectId)/models/(modelId)` route). The actual response is dependent on the classification task: binary classification, regression or multiclass task. ### Binary classification example ``` { "data": [ { "predictionValues": [ { "value": 0.2789450715, "label": 1 }, { "value": 0.7210549285, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 0, "rowId": 0 } ] } ``` ### Regression prediction example ``` { "data": [ { "predictionValues": [ { "value": 6754486.5, "label": "revenue" } ], "prediction": 6754486.5, "rowId": 0 } ] } ``` ### Multiclass classification prediction example ``` { "data": [ { "predictionValues": [ { "value": 0.9999997616, "label": "setosa" }, { "value": 2.433e-7, "label": "versicolor" }, { "value": 1.997631915e-16, "label": "virginica" } ], "prediction": "setosa", "rowId": 0 } ] } ``` ## Errors list | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 400 BAD REQUEST | {"message": "Bad request"} | Added external deployments that are unsupported. | | 404 NOT FOUND | {"message": "Deployment :deploymentId cannot be found for user :userId"} | Provided an invalid :deploymentId (deleted deployment). | --- # Prediction Explanations for deployment URL: https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/dep-predex.html > Using a specified endpoint, makes predictions on a given deployment and provides explanations. Endpoint: `/deployments//predictions?maxExplanations=` Prediction Explanations identify why a given model makes a certain prediction. To calculate Prediction Explanations, use the same endpoint used for calculating bare predictions with the `maxExplanations` URI parameter set to a positive integer value. For specific calculation information, review the main [Prediction Explanations](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/index.html) documentation. Prediction Explanations can be either: - XEMP-based (the default). To use XEMP-based explanations, first calculate feature impact and initialize Prediction Explanations to provide aqualitative indicator (qualitativeStrength)of the effect variables have on the predictions. Explanations are computed for the top 50 features, ranked by feature impact scores (not including features with zero feature impact). - SHAP-based. To use SHAP-based explanations, calculating feature impact is not required. ThequalitativeStrengthindicator is not available for SHAP. > [!NOTE] Prediction Explanation considerations > Neither XEMP or SHAP explanations are available for images (that is, no > Image Explanations > ). > SHAP-based Prediction Explanations cannot be generated for multiclass projects. Only XEMP is supported for multiclass projects. > > More information to consider while working with explanations can be found [here](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/index.html#feature-considerations). > [!WARNING] Performance considerations for XEMP-based explanations > XEMP-based explanations can be 100x slower than regular predictions. Avoid them for low-latency critical use cases. SHAP-based explanations are much faster but can add some latency too. Request Method: `POST` Request URL: deployed URL, for example: `https://your-company.orm.datarobot.com/predApi/v1.0` ## Request parameters ### Headers | Key | Description | Example(s) | | --- | --- | --- | | Datarobot-key | A per-organization secret used as an additional authentication factor for prediction servers. Retrieve a datarobot-key programmatically by accessing the /api/v2/predictionServers endpoint. The endpoint returns a URL to a prediction server and a corresponding datarobot-key. Required for Self-Managed AI Platform users; string type Once a model is deployed, see the code snippet in the DataRobot UI, Predictions > Prediction API. | DR-key-12345abcdb-xyz6789 | | Authorization | Required; string Three methods are supported: Bearer authentication (deprecated) Basic authentication: User_email and API token (deprecated) API token | Example for Bearer authentication method: Bearer API_key-12345abcdb-xyz6789 (deprecated) Example for User_email and API token method: Basic Auth_basic-12345abcdb-xyz6789 (deprecated) Example for API token method: Token API_key-12345abcdb-xyz6789 | | Content-Type | Optional; string type | text/plain; charset=UTF-8 text/csv application/JSON multipart/form-data (For files with data, i.e., .csv, .txt files) | | Content-Encoding | Optional; string type Currently supports only gzip-encoding with the default data extension. | gzip | Datarobot-key: This header is required only with the managed AI Platform. It is used as a precaution to secure user data from other verified DataRobot users. The key can also be retrieved with the following request to DataRobot API: `GET /api/v2/modelDeployments/` ### Query arguments (explanations specific) > [!NOTE] Note > To trigger prediction explanations, your request must send `maxExplanations=N` where N is greater than `0`. | Key | Type | Description | Example(s) | | --- | --- | --- | --- | | maxExplanations | int OR string | (Optional) Limits the number of explanations returned by server. Previously called maxCodes (deprecated). For SHAP explanations only a special constant all is also accepted. | ?maxExplanations=5?maxExplanations=all | | thresholdLow | float | (Optional) The lower threshold for requiring a Prediction Explanation. Predictions must be below this value (or above the thresholdHigh value) for Prediction Explanations to compute. | ?thresholdLow=0.678 | | thresholdHigh | float | (Optional) The upper threshold for requiring a Prediction Explanation. Predictions must be above this value (or below the thresholdLow value) for Prediction Explanations to compute. | ?thresholdHigh=0.345 | | excludeAdjustedPredictions | bool | (Optional) Includes or excludes exposure-adjusted predictions in prediction responses if exposure was used during model building. The default value is true (exclude exposure-adjusted predictions). | ?excludeAdjustedPredictions=true | | explanationNumTopClasses | int | (Optional) This argument is only for multiclass model explanations, and it is mutually exclusive with explanationClassNames. The number of top predicted classes to explain for each row. The default value is 1. | ?explanationNumTopClasses=5 | | explanationClassNames | list of string types | (Optional) This argument is only for multiclass model explanations, and it is mutually exclusive with explanationNumTopClasses. A list of class names to explain for each row. Class names must be passed as UTF-8 bytes and must be percent-encoded (see the HTTP standard for this requirement). By default, ?explanationNumTopClasses=1 is assumed. | ?explanationClassNames=classA&explanationClassNames=classB | | explanationAlgorithm | string | Defines the Prediction Explanation algorithm used, SHAP or XEMP. | ?explanationAlgorithm=shapexplanationAlgorithm=xemp | The rest of the parameters like `passthroughColumns`, `passthroughColumnsSet`, and `predictionWarningEnabled` can also be used with Prediction Explanations. ### Body | Data | Type | Example(s) | | --- | --- | --- | | Data to predict | raw text form-data | PassengerId,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked 892,3,"Kelly, Mr. James",male,34.5,0,0,330911,7.8292,,Q 893,3,"Wilkes, Mrs. James (Ellen Needs)",female,47,1,0,363272,7,,S 894,2,"Myles, Mr. Thomas Francis",male,62,0,0,240276,9.6875,,Q Key: file, value: file_with_data_to_predict.csv | ### Response 200 #### Binary XEMP-based explanation response example ``` { "data": [ { "predictionValues": [ { "value": 0.07836511, "label": 1 }, { "value": 0.92163489, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 0, "rowId": 0, "predictionExplanations": [ { "featureValue": "male", "strength": -0.6706725349, "feature": "Sex", "qualitativeStrength": "---", "label": 1 }, { "featureValue": 62, "strength": -0.6325465255, "feature": "Age", "qualitativeStrength": "---", "label": 1 }, { "featureValue": 9.6875, "strength": -0.353000328, "feature": "Fare", "qualitativeStrength": "--", "label": 1 } ] } ] } ``` #### Binary SHAP-based explanation response example ``` { "data":[ { "deploymentApprovalStatus": "APPROVED", "prediction": 0.0, "predictionExplanations": [ { "featureValue": "9", "strength": 0.0534648234, "qualitativeStrength": null, "feature": "number_diagnoses", "label": 1 }, { "featureValue": "0", "strength": -0.0490243586, "qualitativeStrength": null, "feature": "number_inpatient", "label": 1 } ], "rowId": 0, "predictionValues": [ { "value": 0.3111782477, "label": 1 }, { "value": 0.6888217523, "label": 0.0 } ], "predictionThreshold": 0.5, "shapExplanationsMetadata": { "warnings": null, "remainingTotal": -0.089668474, "baseValue": 0.3964062631 } } ] } ``` ### "qualitativeStrength" indicator The "qualitativeStrength" indicates the effect of the feature's value on predictions, based on XEMP calculations. The following table provides an example for a model with two features. See the [XEMP calculation reference](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/xemp-calc.html) for full calculation details. > [!NOTE] Note > This response is an XEMP-only feature. | Indicator... | Description | | --- | --- | | +++ | Absolute score is > 0.75 and feature has positive impact. | | --- | Absolute score is > 0.75 and feature has negative impact. | | ++ | Absolute score is between (0.25, 0.75) and feature has positive impact. | | -- | Absolute score is between (0.25, 0.75) and feature has negative impact. | | + | Absolute score is between (0.001, 0.25) and feature has positive impact. | | - | Absolute score is between (0.001, 0.25) and feature has negative impact. | | <+ | Absolute score is between (0, 0.001) and feature has positive impact. | | <- | Absolute score is between (0, 0.001) and feature has negative impact. | ## Errors List | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 404 NOT FOUND | {"message": "Not found"} | Provided an invalid :deploymentId (deleted deployment). | | 404 NOT FOUND | {"message": "Bad request"} | Provided the wrong format for :deploymentId. | | 422 UNPROCESSABLE ENTITY | {"message": "{'max_codes': DataError(value can't be converted to int)}"} | Provided maxCodes parameter in unsupported data type (i.e., non-integer values). | | 422 UNPROCESSABLE ENTITY | {"message": "{'threshold_high': DataError(value can't be converted to float)}"} | Provided threshold_high parameter in unsupported data type (i.e., non-integer values). | | 422 UNPROCESSABLE ENTITY | {"message": "{'threshold_low': DataError(value can't be converted to float)}"} | Provided threshold_low parameter in unsupported data type (i.e., non-integer values). | | 422 UNPROCESSABLE ENTITY | {"message": "Multiclass models cannot be used for Prediction Explanations"} | Provided a multiclass classification problem dataset, which is not supported for this endpoint. | | 422 UNPROCESSABLE ENTITY | {"message": "This endpoint does not support predictions on time series models. Please use the timeSeriesPredictions route instead."} | Provided the deploymentId of a time series project, which is not supported for this endpoint. | | 422 UNPROCESSABLE ENTITY | {"message": "{'exclude\_adjusted\_predictions': DataError(value can't be converted to Bool)}"} | Sent an empty or non-Boolean value with the excludeAdjustedPredictions parameter. | | 422 UNPROCESSABLE ENTITY | {"message": "'predictionWarningEnabled': value can't be converted to Bool"} | Provided an invalid (non-boolean) value for predictionWarningEnabled parameter. | --- # Dedicated Prediction API reference URL: https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/index.html > This reference provides additional documentation for the Prediction API. It lists the methods, input and output parameters, and errors that the API may return. This reference provides additional documentation for the Prediction API to help you successfully use the API. The following pages list the methods, input and output parameters, and errors that may be returned by the API. This reference supplements the information provided in the user guide's Prediction API pages. There, you can also find information about prerequisites and best practices and instructions for obtaining your configured predictions URL. When using these examples, be sure to replace `https://your-company.orm.datarobot.com` with the name of your dedicated prediction instance. If you do not know whether you have a dedicated prediction instance, or its address, contact your DataRobot representative. ## General errors for Prediction API These errors may be returned from any Prediction API calls, depending on the issue. They are common to all endpoints. ### Authorization | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 401 UNAUTHORIZED | {"message": "Invalid API token"} | Provided an invalid or no API token key (Basic Auth). Provided an invalid or no API token key (Bearer Auth). Provided an invalid username with a valid API token key (Basic Auth). | | 401 UNAUTHORIZED | {"message": "Invalid Authorization header. No credentials provided."} | Did not provide an API token key (Bearer Token Auth). | | 401 UNAUTHORIZED | {"message": "The datarobot-key header is missing"} | Did not provide a DataRobot key parameter for a project that requires one. Provided an empty DataRobot key parameter. Provided an invalid DataRobot key parameter. | ### Parameters | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 400 BAD REQUEST | {"message": "passthroughColumns do not match columns, columns expected but not found: [u'Name']"} | Provided the name of a column that does not exist. | | 422 UNPROCESSABLE ENTITY | {"message": "'wd': wd is not allowed key"} | Provided an unsupported parameter, e.g., wd. | | 422 UNPROCESSABLE ENTITY | {"message": "'passthroughColumns': blank value is not allowed"} | Provided an empty value for the passthroughColumns parameter. | | 422 UNPROCESSABLE ENTITY | {"message": "'passthroughColumnsSet': value is not exactly 'all'"} | Needed to provide the all value for the passthroughColumnsSet parameter, and provided some other value (or empty). | | 422 UNPROCESSABLE ENTITY | {"message": "'passthroughColumns' and 'passthroughColumnsSet' cannot be used together"} | Passed parameters for both passthroughColumns and passthroughColumnsSet in the same request. Need to pass parameters in separate requests. | | 422 UNPROCESSABLE ENTITY | {"message": "'predictionWarningEnabled': value can't be converted to Bool"} | Provided an invalid (non-boolean) value for predictionWarningEnabled parameter. | ### Payload | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 400 BAD REQUEST | {"message": "Submitted file '10k\_diabetes.xlsx' has unsupported extension"} | Provided a file with an unsupported extension, e.g., .xlsx. | | 400 BAD REQUEST | {"message": "Bad JSON format"} | Provided raw text with content type application/JSON. | | 400 BAD REQUEST | {"message": "Mimetype '' not supported"} | Provided an empty body with the Text mimetype. | | 400 BAD REQUEST | {"message": "No data was received"} | Provided an empty body with application/JSON content-type. | | 400 BAD REQUEST | {"message": "Mimetype 'application/xml' not supported"} | Provided a request with unsupported mimetype. | | 400 BAD REQUEST | {"message": "Requires non-empty JSON input"} | Provided empty JSON, {}. | | 400 BAD REQUEST | {"message": "JSON uploads must be formatted as an array of objects"} | Provided JSON was malformatted: {"0": {"PassengerId": 892, "Pclass": 3}} | | 400 BAD REQUEST | {"message": "Malformed CSV, please check schema and encoding.\nError tokenizing data. C error: Expected 11 fields in line 5, saw 12\n"} | Provided CSV has issues: 1 row has more fields than expected (in this instance). | | 413 Entity Too Large | {"message": "Request is too large. The request size is $content\_length bytes and the maximum message size allowed by the server is 50MB"} | Provided file is too large. DataRobot accepts files of up to 50MB in size for real-time deployment predictions; if the file size exceeds the limit, the batch-scoring tool should be used. The same limit is applied for archived datasets. | | 422 UNPROCESSABLE ENTITY | {"message": "No data to predict on"} | Provided an empty request payload. | | 422 UNPROCESSABLE ENTITY | {"message": "Missing column(s): Age, Cabin, Embarked, Fare, Name, Parch, PassengerId, Pclass, Sex & SibSp"} | Dataset is missing all required fields. Use a dataset from the project you try to predict on, with expected fields. | ## Prediction API infinity behavior [IEEE-754](https://en.wikipedia.org/wiki/IEEE_754), the standard for floating-point arithmetic, defines finite numbers, infinities, and a special NaN (not-a-number) value. According to [RFC-8259](https://datatracker.ietf.org/doc/html/rfc8259#section-6), infinities and NaN are not allowed in JSON. DataRobot tries to replace these values before they are returned in APIs using the following rules: - Infis replaced with1.7976931348623157e+308(double precision floating-point max). - -Infis replaced with-1.7976931348623157e+308(double precision floating-point min). - NaNis replaced with0.0. The Predictions API rounds floating-point numbers to 10 decimal places. However, the rounding logic changes when the floating point minimum and maximum are rounded below and above the limits, respectively: - 1.7976931348623157e+308(double precision floating-point max) is returned as1.797693135e+308(greater than the maximum limit). - -1.7976931348623157e+308(double precision floating-point min) is returned as-1.797693135e+308(lower than the minimum limit). - Note that CPython’s built-in JSON parser parses such values asinfand-infrespectively, but some other languages may crash. --- # Ping health check URL: https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/ping.html > Health check to determine if the service is Endpoint: `/ping` Health check to determine if the service is "alive". Request Method: `GET` Request URL: deployed URL, example: `https://your-company.orm.datarobot.com/predApi/v1.0` ### Request parameters None required. ### Response 200 | Data | Type | Example(s) | | --- | --- | --- | | response | string | pong | --- # Time series predictions for deployments URL: https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/time-pred.html > Make time series predictions for a deployed model. Endpoint: `/deployments//predictions` Makes time series predictions for a deployed model. Request Method: `POST` Request URL: deployed URL, for example: `https://your-company.orm.datarobot.com/predApi/v1.0` ## Request parameters ### Headers | Key | Type | Description | Example(s) | | --- | --- | --- | --- | | Datarobot-key | string | A per-organization secret used as an additional authentication factor for prediction servers. Retrieve a datarobot-key programmatically by accessing the /api/v2/predictionServers endpoint. The endpoint returns a URL to a prediction server and a corresponding datarobot-key. Required for Self-Managed AI Platform users; string type | 33257d41-fcc9-7c01-161c-3467df169a50 | | Authorization | string | Three methods are supported: Bearer authentication(deprecated) Basic authentication: User_email and API token(deprecated) API token | Example for Bearer authentication method: Bearer API_key-12345abcdb-xyz6789(deprecated) Example for User_email and API token method: Basic Auth_basic-12345abcdb-xyz6789(deprecated) Example for API token method: Token API_key-12345abcdb-xyz6789 | Datarobot-key: This header is required only with the managed AI Platform. It is used as a precaution to secure user data from other verified DataRobot users. The key can also be retrieved with the following request to the DataRobot API: `GET /api/v2/modelDeployments/` ### Query arguments (time series models only) | Key | Type | Description | Example(s) | | --- | --- | --- | --- | | forecastPoint | ISO-8601 string | An ISO 8601 formatted DateTime string, without timezone, representing the forecast point. This parameter cannot be used if predictionsStartDate and predictionsEndDate are passed. | ?forecastPoint=2013-12-20T01:30:00Z | | relaxKnownInAdvanceFeaturesCheck | bool | true or false. When true, missing values for known-in-advance features are allowed in the forecast window at prediction time. The default value is false. Note that the absence of known-in-advance values can negatively impact prediction quality. | ?relaxKnownInAdvanceFeaturesCheck=true | | predictionsStartDate | ISO-8601 string | The time in the dataset when bulk predictions begin generating. This parameter must be defined together with predictionsEndDate. The forecastPoint parameter cannot be used if predictionsStartDate and predictionsEndDate are passed. | ?predictionsStartDate=2013-12-20T01:30:00Z&predictionsEndDate=2013-12-20T01:40:00Z | | predictionsEndDate | ISO-8601 string | The time in the dataset when bulk predictions stop generating. This parameter must be defined together with predictionsStartDate. The forecastPoint parameter cannot be used if predictionsStartDate and predictionsEndDate are passed. | See above. | It is possible to use standard URI parameters, including `passthroughColumns`, `passthroughColumnsSet`, and `maxExplanations`. > [!NOTE] XEMP-based explanations support > Time series supports XEMP explanations. See [Prediction Explanations](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/dep-predex.html) for examples of the `maxExplanations` URI parameter. ### Body | Data | Type | Example(s) | | --- | --- | --- | | Historic and prediction data | JSON | Raw data shown in image below. | ### Response 200 #### Regression prediction example ``` { "data": [ { "seriesId": null, "forecastPoint": "2013-12-20T00:00:00Z", "rowId": 35, "timestamp": "2013-12-21T00:00:00.000000Z", "predictionValues": [ { "value": 2.3353628422, "label": "sales (actual)" } ], "forecastDistance": 1, "prediction": 2.3353628422 } ] } ``` #### Binary classification prediction example ``` { "data": [ { "rowId": 147, "prediction": "low", "predictionThreshold": 0.5, "predictionValues": [ {"label": "low", "value": 0.5158823954}, {"label": "high", "value": 0.4841176046}, ], "timestamp": "1961-04-01T00:00:00.000000Z", "forecastDistance": 2, "forecastPoint": "1961-02-01T00:00:00Z", "seriesId": null, } ] } ``` ## Errors List | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 400 BAD REQUEST | {"message": "Based on the forecast point (10/26/08), there are no rows to predict that fall inside of the forecast window (10/27/08 to 11/02/08). Try adjusting the forecast point to an earlier date or appending new future rows to the data."} | No empty rows were provided to predict on. | | 400 BAD REQUEST | {"message": "No valid output rows"} | No historic information was provided; there's only 1 row to predict on. | | 400 BAD REQUEST | {"message": "The \"Time\" feature contains the value 'OCT-27', which does not match the original format %m/%d/%y (e.g., '06/24/19'). To upload this data, first correct the format in your prediction dataset and then try the import again. Because some software automatically converts the format for display, it is best to check the actual format using a text editor."} | Prediction row has a different format than the rest of the data. | | 400 BAD REQUEST | {"message": "The following errors are found:\n • The prediction data must contain historical values spanning more than 35 day(s) into the past. In addition, the target cannot have missing values or missing rows which are used for differencing"} | Provided dataset has fewer than the required 35 rows of historical data. | | 400 BAD REQUEST | {"message": {"forecastPoint": "Invalid RFC 3339 datetime string: "}} | Provided an empty or non-valid forecastPoint. | | 404 NOT FOUND | {"message": "Deployment :deploymentId cannot be found for user :userId"} | Deployment was removed or doesn’t exist. | | 422 UNPROCESSABLE ENTITY | {"message": "Predictions on models that are not time series models are not supported on this endpoint. Please use the predict endpoint instead."} | Provided deploymentId that is not for a time series project. | | 422 UNPROCESSABLE ENTITY | {"message": {"relaxKnownInAdvanceFeaturesCheck": "value can't be converted to Bool"}} | Provided an empty or non-valid value for relaxKnownInAdvanceFeaturesCheck'. | --- # Predictions for deployments (Serverless) URL: https://docs.datarobot.com/en/docs/api/reference/predapi/pred-ref-serverless/dep-pred.html > Using a specified endpoint, calculates predictions based on user-provided data for a specific deployment. Using the endpoint below, you can provide the data necessary to calculate predictions for a specific deployment. If you need to make predictions for an unstructured custom inference model, see [Predictions for unstructured model deployments](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/dep-pred-unstructured.html). Endpoint: `/api/v2/deployments//predictions` Calculates predictions based on user-provided data for a specific deployment. Note that this endpoint works only for deployed models. > [!NOTE] Note > You can find the deployment ID in the sample code output of the [Deployments > Predictions > Prediction API](https://docs.datarobot.com/en/docs/classic-ui/predictions/realtime/code-py.html) tab (with Interface set to API Client). Request Method: `POST` ## Request parameters ### Headers | Key | Description | Example(s) | | --- | --- | --- | | Authorization | Required; string Three methods are supported: Bearer authentication (deprecated) Basic authentication: User_email and API token (deprecated) API token | Example for Bearer authentication method: Bearer API_key-12345abcdb-xyz6789 (deprecated) Example for User_email and API token method: Basic Auth_basic-12345abcdb-xyz6789 (deprecated) Example for API token method: Token API_key-12345abcdb-xyz6789 | | Content-Type | Optional; string type | text/plain; charset=UTF-8text/csvapplication/jsonmultipart/form-data (for files with data, i.e., .csv, .txt files) | | Content-Encoding | Optional; string type Currently supports only gzip-encoding with the default data extension. | gzip | | Accept | Optional; string type Controls the shape of the response schema. Currently JSON(default) and CSV are supported. See examples. | application/json (default)text/csv (for CSV output) | ### Query arguments | Key | Type | Description | Example(s) | | --- | --- | --- | --- | | passthroughColumns | list of strings | (Optional) Controls which columns from a scoring dataset to expose (or copy over) in a prediction response. The request may contain zero, one, or more columns. (There's no limit on how many column names you can pass.) Column names must be passed as UTF-8 bytes and be percent-encoded (see the HTTP standard for this requirement). Make sure to use the exact name of a column as a value. | /api/v2/deployments//predictions?passthroughColumns=colA&passthroughColumns=colB | | passthroughColumnsSet | string | (Optional) Controls which columns from a scoring dataset to expose (or to copy over) in a prediction response. The only possible option is all and, if passed, all columns from a scoring dataset are exposed. | /api/v2/deployments//predictions?passthroughColumnsSet=all | | predictionWarningEnabled | bool | (Optional) DataRobot monitors unusual or anomalous predictions in real-time and indicates when they are detected. If this argument is set to true, a new key is added to each prediction to specify the result of the Humility check. Otherwise, there are no changes in the prediction response. | /api/v2/deployments//predictions?predictionWarningEnabled=true Response: { "data": [ { "predictionValues": [ { "value": 18.6948852, "label": "y" } ], "isOutlierPrediction": false, "rowId": 0, "prediction": 18.6948852 } ] } | | decimalsNumber | integer | (Optional) Configures the float precision in prediction results by setting the number of digits after the decimal point. If there aren't any digits after the decimal point, rather than adding zeros, the float precision is less than the value set by decimalsNumber. | ?decimalsNumber=15 | > [!NOTE] Note > The `passthroughColumns` and `passthroughColumnsSet` parameters are mutually exclusive and cannot both be passed in the same request. Also, while there isn't a limit on the number of column names you can pass with the `passthroughColumns` query parameter, there is a limit on the size of the [HTTP request line](https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html#sec5.1) (currently 8192 bytes). ### Body | Data | Type | Example(s) | | --- | --- | --- | | Data to predict | raw text form-data | PassengerId,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked 892,3,"Kelly, Mr. James",male,34.5,0,0,330911,7.8292,,Q 893,3,"Wilkes, Mrs. James (Ellen Needs)",female,47,1,0,363272,7,,S 894,2,"Myles, Mr. Thomas Francis",male,62,0,0,240276,9.6875,,Q Key: file, value: file_with_data_to_predict.csv | ## Response 200 ### Binary prediction Label: For regression and binary classification tasks, DataRobot API always returns 1 for the positive class and 0 for the negative class. Although the actual values for the classes may be different depending on the data provided (like "yes"/"no"), the DataRobot API will always return 1/0. For multiclass classification, the DataRobot API returns the value itself. Value: Shows the probability of an event happening (where 0 and 1 are min and max probability, respectively). The user can adjust the threshold that links the value with the prediction label. PredictionThreshold ( Applicable to binary classification projects only): Threshold is the point that sets the class boundary for a predicted value. The model classifies an observation below the threshold as FALSE, and an observation above the threshold as TRUE. In other words, DataRobot automatically assigns the positive class label to any prediction exceeding the threshold. This can be configured manually through the UI (the [Deploytab](https://docs.datarobot.com/en/docs/classic-ui/mlops/deployment/deploy-methods/deploy-model.html#deploy-from-the-leaderboard)), or through the DataRobot API (i.e., the `PATCH /api/v2/projects/(projectId)/models/(modelId)` route). The actual response is dependent on the classification task: binary classification, regression or multiclass task. ### Binary classification example ``` { "data": [ { "predictionValues": [ { "value": 0.2789450715, "label": 1 }, { "value": 0.7210549285, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 0, "rowId": 0 } ] } ``` ### Regression prediction example ``` { "data": [ { "predictionValues": [ { "value": 6754486.5, "label": "revenue" } ], "prediction": 6754486.5, "rowId": 0 } ] } ``` ### Multiclass classification prediction example ``` { "data": [ { "predictionValues": [ { "value": 0.9999997616, "label": "setosa" }, { "value": 2.433e-7, "label": "versicolor" }, { "value": 1.997631915e-16, "label": "virginica" } ], "prediction": "setosa", "rowId": 0 } ] } ``` ## Errors list | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 400 BAD REQUEST | {"message": "Bad request"} | Added external deployments that are unsupported. | | 404 NOT FOUND | {"message": "Deployment :deploymentId cannot be found for user :userId"} | Provided an invalid :deploymentId (deleted deployment). | --- # Prediction Explanations for deployment (Serverless) URL: https://docs.datarobot.com/en/docs/api/reference/predapi/pred-ref-serverless/dep-predex.html > Using a specified endpoint, makes predictions on a given deployment and provides explanations. Endpoint: `/api/v2/deployments//predictions?maxExplanations=` Prediction Explanations identify why a given model makes a certain prediction. To calculate Prediction Explanations, use the same endpoint used for calculating bare predictions with the `maxExplanations` URI parameter set to a positive integer value. For specific calculation information, review the main [Prediction Explanations](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/index.html) documentation. Prediction Explanations can be either: - XEMP-based (the default). To use XEMP-based explanations, first calculate feature impact and initialize Prediction Explanations to provide aqualitative indicator (qualitativeStrength)of the effect variables have on the predictions. Explanations are computed for the top 50 features, ranked by feature impact scores (not including features with zero feature impact). - SHAP-based. To use SHAP-based explanations, calculating feature impact is not required. ThequalitativeStrengthindicator is not available for SHAP. > [!NOTE] Prediction Explanation considerations > Neither XEMP or SHAP explanations are available for images (that is, no > Image Explanations > ). > SHAP-based Prediction Explanations cannot be generated for multiclass projects. Only XEMP is supported for multiclass projects. > > More information to consider while working with explanations can be found [here](https://docs.datarobot.com/en/docs/classic-ui/modeling/analyze-models/understand/pred-explain/index.html#feature-considerations). > [!WARNING] Performance considerations for XEMP-based explanations > XEMP-based explanations can be 100x slower than regular predictions. Avoid them for low-latency critical use cases. SHAP-based explanations are much faster but can add some latency too. Request Method: `POST` Request URL: REST API URL, for example: `https://your-company.datarobot.com/api/v2` ## Request parameters ### Headers | Key | Description | Example(s) | | --- | --- | --- | | Authorization | Required; string Three methods are supported: Bearer authentication (deprecated) Basic authentication: User_email and API token (deprecated) API token | Example for Bearer authentication method: Bearer API_key-12345abcdb-xyz6789 (deprecated) Example for User_email and API token method: Basic Auth_basic-12345abcdb-xyz6789 (deprecated) Example for API token method: Token API_key-12345abcdb-xyz6789 | | Content-Type | Optional; string type | text/plain; charset=UTF-8 text/csv application/JSON multipart/form-data (For files with data, i.e., .csv, .txt files) | | Content-Encoding | Optional; string type Currently supports only gzip-encoding with the default data extension. | gzip | ### Query arguments (explanations specific) > [!NOTE] Note > To trigger prediction explanations, your request must send `maxExplanations=N` where N is greater than `0`. | Key | Type | Description | Example(s) | | --- | --- | --- | --- | | maxExplanations | int OR string | (Optional) Limits the number of explanations returned by server. Previously called maxCodes (deprecated). For SHAP explanations only a special constant all is also accepted. | ?maxExplanations=5?maxExplanations=all | | thresholdLow | float | (Optional) The lower threshold for requiring a Prediction Explanation. Predictions must be below this value (or above the thresholdHigh value) for Prediction Explanations to compute. | ?thresholdLow=0.678 | | thresholdHigh | float | (Optional) The upper threshold for requiring a Prediction Explanation. Predictions must be above this value (or below the thresholdLow value) for Prediction Explanations to compute. | ?thresholdHigh=0.345 | | excludeAdjustedPredictions | bool | (Optional) Includes or excludes exposure-adjusted predictions in prediction responses if exposure was used during model building. The default value is true (exclude exposure-adjusted predictions). | ?excludeAdjustedPredictions=true | | explanationNumTopClasses | int | (Optional) This argument is only for multiclass model explanations, and it is mutually exclusive with explanationClassNames. The number of top predicted classes to explain for each row. The default value is 1. | ?explanationNumTopClasses=5 | | explanationClassNames | list of string types | (Optional) This argument is only for multiclass model explanations, and it is mutually exclusive with explanationNumTopClasses. A list of class names to explain for each row. Class names must be passed as UTF-8 bytes and must be percent-encoded (see the HTTP standard for this requirement). By default, ?explanationNumTopClasses=1 is assumed. | ?explanationClassNames=classA&explanationClassNames=classB | | explanationAlgorithm | string | Defines the Prediction Explanation algorithm used, SHAP or XEMP. | ?explanationAlgorithm=shapexplanationAlgorithm=xemp | The rest of the parameters like `passthroughColumns`, `passthroughColumnsSet`, and `predictionWarningEnabled` can also be used with Prediction Explanations. ### Body | Data | Type | Example(s) | | --- | --- | --- | | Data to predict | raw text form-data | PassengerId,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked 892,3,"Kelly, Mr. James",male,34.5,0,0,330911,7.8292,,Q 893,3,"Wilkes, Mrs. James (Ellen Needs)",female,47,1,0,363272,7,,S 894,2,"Myles, Mr. Thomas Francis",male,62,0,0,240276,9.6875,,Q Key: file, value: file_with_data_to_predict.csv | ### Response 200 #### Binary XEMP-based explanation response example ``` { "data": [ { "predictionValues": [ { "value": 0.07836511, "label": 1 }, { "value": 0.92163489, "label": 0 } ], "predictionThreshold": 0.5, "prediction": 0, "rowId": 0, "predictionExplanations": [ { "featureValue": "male", "strength": -0.6706725349, "feature": "Sex", "qualitativeStrength": "---", "label": 1 }, { "featureValue": 62, "strength": -0.6325465255, "feature": "Age", "qualitativeStrength": "---", "label": 1 }, { "featureValue": 9.6875, "strength": -0.353000328, "feature": "Fare", "qualitativeStrength": "--", "label": 1 } ] } ] } ``` #### Binary SHAP-based explanation response example ``` { "data":[ { "deploymentApprovalStatus": "APPROVED", "prediction": 0.0, "predictionExplanations": [ { "featureValue": "9", "strength": 0.0534648234, "qualitativeStrength": null, "feature": "number_diagnoses", "label": 1 }, { "featureValue": "0", "strength": -0.0490243586, "qualitativeStrength": null, "feature": "number_inpatient", "label": 1 } ], "rowId": 0, "predictionValues": [ { "value": 0.3111782477, "label": 1 }, { "value": 0.6888217523, "label": 0.0 } ], "predictionThreshold": 0.5, "shapExplanationsMetadata": { "warnings": null, "remainingTotal": -0.089668474, "baseValue": 0.3964062631 } } ] } ``` ### "qualitativeStrength" indicator The "qualitativeStrength" indicates the effect of the feature's value on predictions, based on XEMP calculations. The following table provides an example for a model with two features. See the [XEMP calculation reference](https://docs.datarobot.com/en/docs/reference/pred-ai-ref/xemp-calc.html) for full calculation details. > [!NOTE] Note > This response is an XEMP-only feature. | Indicator... | Description | | --- | --- | | +++ | Absolute score is > 0.75 and feature has positive impact. | | --- | Absolute score is > 0.75 and feature has negative impact. | | ++ | Absolute score is between (0.25, 0.75) and feature has positive impact. | | -- | Absolute score is between (0.25, 0.75) and feature has negative impact. | | + | Absolute score is between (0.001, 0.25) and feature has positive impact. | | - | Absolute score is between (0.001, 0.25) and feature has negative impact. | | <+ | Absolute score is between (0, 0.001) and feature has positive impact. | | <- | Absolute score is between (0, 0.001) and feature has negative impact. | ## Errors List | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 404 NOT FOUND | {"message": "Not found"} | Provided an invalid :deploymentId (deleted deployment). | | 404 NOT FOUND | {"message": "Bad request"} | Provided the wrong format for :deploymentId. | | 422 UNPROCESSABLE ENTITY | {"message": "{'max_codes': DataError(value can't be converted to int)}"} | Provided maxCodes parameter in unsupported data type (i.e., non-integer values). | | 422 UNPROCESSABLE ENTITY | {"message": "{'threshold_high': DataError(value can't be converted to float)}"} | Provided threshold_high parameter in unsupported data type (i.e., non-integer values). | | 422 UNPROCESSABLE ENTITY | {"message": "{'threshold_low': DataError(value can't be converted to float)}"} | Provided threshold_low parameter in unsupported data type (i.e., non-integer values). | | 422 UNPROCESSABLE ENTITY | {"message": "Multiclass models cannot be used for Prediction Explanations"} | Provided a multiclass classification problem dataset, which is not supported for this endpoint. | | 422 UNPROCESSABLE ENTITY | {"message": "This endpoint does not support predictions on time series models. Please use the timeSeriesPredictions route instead."} | Provided the deploymentId of a time series project, which is not supported for this endpoint. | | 422 UNPROCESSABLE ENTITY | {"message": "{'exclude\_adjusted\_predictions': DataError(value can't be converted to Bool)}"} | Sent an empty or non-Boolean value with the excludeAdjustedPredictions parameter. | | 422 UNPROCESSABLE ENTITY | {"message": "'predictionWarningEnabled': value can't be converted to Bool"} | Provided an invalid (non-boolean) value for predictionWarningEnabled parameter. | --- # Serverless Prediction API reference URL: https://docs.datarobot.com/en/docs/api/reference/predapi/pred-ref-serverless/index.html > This reference provides additional documentation for the Serverless Prediction API. It lists the methods, input and output parameters, and errors that the API may return. This reference provides additional documentation for the Serverless Prediction API to help you successfully use the API. The following pages list the methods, input and output parameters, and errors that may be returned by the API. This reference supplements the information provided in the user guide's Prediction API pages. There, you can also find information about prerequisites and best practices and instructions for using the REST API endpoint. When using these examples, be sure to replace `https://your-company.datarobot.com` with the name of your DataRobot instance. Serverless predictions use the REST API endpoint `/api/v2/deployments/:id/predictions` and do not require a `datarobot-key` header. ## General errors for Prediction API These errors may be returned from any Prediction API calls, depending on the issue. They are common to all endpoints. ### Authorization | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 401 UNAUTHORIZED | {"message": "Invalid API token"} | Provided an invalid or no API token key (Basic Auth). Provided an invalid or no API token key (Bearer Auth). Provided an invalid username with a valid API token key (Basic Auth). | | 401 UNAUTHORIZED | {"message": "Invalid Authorization header. No credentials provided."} | Did not provide an API token key (Bearer Token Auth). | ### Parameters | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 400 BAD REQUEST | {"message": "passthroughColumns do not match columns, columns expected but not found: [u'Name']"} | Provided the name of a column that does not exist. | | 422 UNPROCESSABLE ENTITY | {"message": "'wd': wd is not allowed key"} | Provided an unsupported parameter, e.g., wd. | | 422 UNPROCESSABLE ENTITY | {"message": "'passthroughColumns': blank value is not allowed"} | Provided an empty value for the passthroughColumns parameter. | | 422 UNPROCESSABLE ENTITY | {"message": "'passthroughColumnsSet': value is not exactly 'all'"} | Needed to provide the all value for the passthroughColumnsSet parameter, and provided some other value (or empty). | | 422 UNPROCESSABLE ENTITY | {"message": "'passthroughColumns' and 'passthroughColumnsSet' cannot be used together"} | Passed parameters for both passthroughColumns and passthroughColumnsSet in the same request. Need to pass parameters in separate requests. | | 422 UNPROCESSABLE ENTITY | {"message": "'predictionWarningEnabled': value can't be converted to Bool"} | Provided an invalid (non-boolean) value for predictionWarningEnabled parameter. | ### Payload | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 400 BAD REQUEST | {"message": "Submitted file '10k\_diabetes.xlsx' has unsupported extension"} | Provided a file with an unsupported extension, e.g., .xlsx. | | 400 BAD REQUEST | {"message": "Bad JSON format"} | Provided raw text with content type application/JSON. | | 400 BAD REQUEST | {"message": "Mimetype '' not supported"} | Provided an empty body with the Text mimetype. | | 400 BAD REQUEST | {"message": "No data was received"} | Provided an empty body with application/JSON content-type. | | 400 BAD REQUEST | {"message": "Mimetype 'application/xml' not supported"} | Provided a request with unsupported mimetype. | | 400 BAD REQUEST | {"message": "Requires non-empty JSON input"} | Provided empty JSON, {}. | | 400 BAD REQUEST | {"message": "JSON uploads must be formatted as an array of objects"} | Provided JSON was malformatted: {"0": {"PassengerId": 892, "Pclass": 3}} | | 400 BAD REQUEST | {"message": "Malformed CSV, please check schema and encoding.\nError tokenizing data. C error: Expected 11 fields in line 5, saw 12\n"} | Provided CSV has issues: 1 row has more fields than expected (in this instance). | | 413 Entity Too Large | {"message": "Request is too large. The request size is $content\_length bytes and the maximum message size allowed by the server is 50MB"} | Provided file is too large. DataRobot accepts files of up to 50MB in size for real-time deployment predictions; if the file size exceeds the limit, the batch-scoring tool should be used. The same limit is applied for archived datasets. | | 422 UNPROCESSABLE ENTITY | {"message": "No data to predict on"} | Provided an empty request payload. | | 422 UNPROCESSABLE ENTITY | {"message": "Missing column(s): Age, Cabin, Embarked, Fare, Name, Parch, PassengerId, Pclass, Sex & SibSp"} | Dataset is missing all required fields. Use a dataset from the project you try to predict on, with expected fields. | ## Prediction API infinity behavior [IEEE-754](https://en.wikipedia.org/wiki/IEEE_754), the standard for floating-point arithmetic, defines finite numbers, infinities, and a special NaN (not-a-number) value. According to [RFC-8259](https://datatracker.ietf.org/doc/html/rfc8259#section-6), infinities and NaN are not allowed in JSON. DataRobot tries to replace these values before they are returned in APIs using the following rules: - Infis replaced with1.7976931348623157e+308(double precision floating-point max). - -Infis replaced with-1.7976931348623157e+308(double precision floating-point min). - NaNis replaced with0.0. The Predictions API rounds floating-point numbers to 10 decimal places. However, the rounding logic changes when the floating point minimum and maximum are rounded below and above the limits, respectively: - 1.7976931348623157e+308(double precision floating-point max) is returned as1.797693135e+308(greater than the maximum limit). - -1.7976931348623157e+308(double precision floating-point min) is returned as-1.797693135e+308(lower than the minimum limit). - Note that CPython's built-in JSON parser parses such values asinfand-infrespectively, but some other languages may crash. --- # Time series predictions for deployments (Serverless) URL: https://docs.datarobot.com/en/docs/api/reference/predapi/pred-ref-serverless/time-pred.html > Make time series predictions for a deployed model using serverless prediction environments. Endpoint: `/api/v2/deployments//predictions` Makes time series predictions for a deployed model using serverless prediction environments. Request Method: `POST` Request URL: REST API URL, for example: `https://your-company.datarobot.com/api/v2` ## Request parameters ### Headers | Key | Type | Description | Example(s) | | --- | --- | --- | --- | | Authorization | string | Three methods are supported: Bearer authentication(deprecated) Basic authentication: User_email and API token(deprecated) API token | Example for Bearer authentication method: Bearer API_key-12345abcdb-xyz6789(deprecated) Example for User_email and API token method: Basic Auth_basic-12345abcdb-xyz6789(deprecated) Example for API token method: Token API_key-12345abcdb-xyz6789 | ### Query arguments (time series models only) | Key | Type | Description | Example(s) | | --- | --- | --- | --- | | forecastPoint | ISO-8601 string | An ISO 8601 formatted DateTime string, without timezone, representing the forecast point. This parameter cannot be used if predictionsStartDate and predictionsEndDate are passed. | ?forecastPoint=2013-12-20T01:30:00Z | | relaxKnownInAdvanceFeaturesCheck | bool | true or false. When true, missing values for known-in-advance features are allowed in the forecast window at prediction time. The default value is false. Note that the absence of known-in-advance values can negatively impact prediction quality. | ?relaxKnownInAdvanceFeaturesCheck=true | | predictionsStartDate | ISO-8601 string | The time in the dataset when bulk predictions begin generating. This parameter must be defined together with predictionsEndDate. The forecastPoint parameter cannot be used if predictionsStartDate and predictionsEndDate are passed. | ?predictionsStartDate=2013-12-20T01:30:00Z&predictionsEndDate=2013-12-20T01:40:00Z | | predictionsEndDate | ISO-8601 string | The time in the dataset when bulk predictions stop generating. This parameter must be defined together with predictionsStartDate. The forecastPoint parameter cannot be used if predictionsStartDate and predictionsEndDate are passed. | See above. | It is possible to use standard URI parameters, including `passthroughColumns`, `passthroughColumnsSet`, and `maxExplanations`. > [!NOTE] XEMP-based explanations support > Time series supports XEMP explanations. See [Prediction Explanations](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/pred-ref/dep-predex.html) for examples of the `maxExplanations` URI parameter. ### Body | Data | Type | Example(s) | | --- | --- | --- | | Historic and prediction data | JSON | Raw data shown in image below. | ### Response 200 #### Regression prediction example ``` { "data": [ { "seriesId": null, "forecastPoint": "2013-12-20T00:00:00Z", "rowId": 35, "timestamp": "2013-12-21T00:00:00.000000Z", "predictionValues": [ { "value": 2.3353628422, "label": "sales (actual)" } ], "forecastDistance": 1, "prediction": 2.3353628422 } ] } ``` #### Binary classification prediction example ``` { "data": [ { "rowId": 147, "prediction": "low", "predictionThreshold": 0.5, "predictionValues": [ {"label": "low", "value": 0.5158823954}, {"label": "high", "value": 0.4841176046}, ], "timestamp": "1961-04-01T00:00:00.000000Z", "forecastDistance": 2, "forecastPoint": "1961-02-01T00:00:00Z", "seriesId": null, } ] } ``` ## Errors List | HTTP Code | Sample error message | Reason(s) | | --- | --- | --- | | 400 BAD REQUEST | {"message": "Based on the forecast point (10/26/08), there are no rows to predict that fall inside of the forecast window (10/27/08 to 11/02/08). Try adjusting the forecast point to an earlier date or appending new future rows to the data."} | No empty rows were provided to predict on. | | 400 BAD REQUEST | {"message": "No valid output rows"} | No historic information was provided; there's only 1 row to predict on. | | 400 BAD REQUEST | {"message": "The \"Time\" feature contains the value 'OCT-27', which does not match the original format %m/%d/%y (e.g., '06/24/19'). To upload this data, first correct the format in your prediction dataset and then try the import again. Because some software automatically converts the format for display, it is best to check the actual format using a text editor."} | Prediction row has a different format than the rest of the data. | | 400 BAD REQUEST | {"message": "The following errors are found:\n • The prediction data must contain historical values spanning more than 35 day(s) into the past. In addition, the target cannot have missing values or missing rows which are used for differencing"} | Provided dataset has fewer than the required 35 rows of historical data. | | 400 BAD REQUEST | {"message": {"forecastPoint": "Invalid RFC 3339 datetime string: "}} | Provided an empty or non-valid forecastPoint. | | 404 NOT FOUND | {"message": "Deployment :deploymentId cannot be found for user :userId"} | Deployment was removed or doesn't exist. | | 422 UNPROCESSABLE ENTITY | {"message": "Predictions on models that are not time series models are not supported on this endpoint. Please use the predict endpoint instead."} | Provided deploymentId that is not for a time series project. | | 422 UNPROCESSABLE ENTITY | {"message": {"relaxKnownInAdvanceFeaturesCheck": "value can't be converted to Bool"}} | Provided an empty or non-valid value for relaxKnownInAdvanceFeaturesCheck'. | --- # Get a prediction server ID URL: https://docs.datarobot.com/en/docs/api/reference/predapi/pred-server-id.html > Learn how to retrieve a prediction server ID using cURL commands from the REST API or by using the DataRobot Python client. In order to make predictions from a deployment via DataRobot's [Prediction API](https://docs.datarobot.com/en/docs/api/reference/predapi/legacy-predapi/dr-predapi.html), you need a prediction server ID. In this tutorial, you'll learn how to retrieve the ID using cURL commands from the REST API or by using the DataRobot Python client. Once obtained, you can use the prediction server ID to deploy a model and make predictions. > [!NOTE] Note > Before proceeding, note that an API key is required for this tutorial. Reference the [Create API keys](https://docs.datarobot.com/en/docs/platform/acct-settings/api-key-mgmt.html) tutorial for more information. **cURL:** ``` curl -v \ -H "Authorization: Bearer API_KEY" \ YOUR_DR_URL/api/v2/predictionServers/ Example API_KEY=YOUR_API_KEY ENDPOINT=YOUR_DR_URL/api/v2/predictionServers/ curl -v \ -H "Authorization: Bearer $API_KEY" \ $ENDPOINT ``` **Python:** Before continuing with Python, be sure you have installed the DataRobot Python client and configured your connection to DataRobot as outlined in the [Developer quickstart](https://docs.datarobot.com/en/docs/api/dev-learning/api-quickstart.html). ``` # Set up your environment import os import datarobot as dr API_KEY = os.environ["API_KEY"] YOUR_DR_URL = os.environ["YOUR_DR_URL"] FILE_PATH = os.environ["FILE_PATH"] ENDPOINT = YOUR_DR_URL+"/api/v2" # Instantiate DataRobot instance dr.Client( token=API_KEY, endpoint=ENDPOINT ) prediction_server_id = dr.PredictionServer.list()[0].id print(prediction_server_id) ``` ## Documentation The following provides additional documentation for features mentioned in this tutorial. - API key management - DataRobot Developers portal - DataRobot Prediction API --- # Agent cards URL: https://docs.datarobot.com/en/docs/api/reference/public-api/agent_cards.html > Use the endpoints described below to discover agents registered within the platform and manage agent cards. Use the endpoints described below to discover agents registered within the platform and manage agent cards. ## List registered agent cards Operation path: `GET /api/v2/agentCards/` Authentication requirements: `BearerAuth` Retrieve the list of agent cards within tenant. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | offset | query | integer | true | The number of agent cards to skip. Defaults to 0. | | limit | query | integer | true | The number of agent cards (greater than zero, max 100) to return. Defaults to 25. | | orderBy | query | string | false | The order to sort the agent cards. Defaults to order by creation time in ascending order. | | deploymentIds | query | array[string] | false | Filter registered agents based on deployment IDs. | | externalIds | query | string | false | Filter registered agents based on external IDs. | | workloadIds | query | array[string] | false | Filter registered agents based on workload IDs. This argument is mutually exclusive with deploymentIds. | ### Enumerated Values | Parameter | Value | | --- | --- | | orderBy | [deploymentId, -deploymentId, externalId, -externalId, workloadId, -workloadId, createdAt, -createdAt, updatedAt, -updatedAt] | ### Example responses > 200 Response ``` { "properties": { "count": { "description": "The number of items returned on this page.", "type": "integer" }, "data": { "description": "The list of formatted agent cards.", "items": { "properties": { "agentCard": { "description": "Plain agent card data.", "properties": { "additionalInterfaces": { "description": "Additional transport interfaces beyond the primary url.", "items": { "properties": { "transport": { "description": "The transport protocol (e.g. jsonrpc, grpc, http+json).", "type": "string" }, "url": { "description": "The url for this transport interface.", "type": "string" } }, "required": [ "transport", "url" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "capabilities": { "description": "Optional capabilities supported by the agent.", "properties": { "extensions": { "description": "Protocol extensions supported by the agent.", "items": { "properties": { "description": { "description": "A human-readable description of the extension.", "type": [ "string", "null" ] }, "params": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "required": { "description": "Whether this extension is required by the agent.", "type": [ "boolean", "null" ] }, "uri": { "description": "The uri identifying the extension.", "type": "string" } }, "required": [ "uri" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "pushNotifications": { "description": "Whether the agent supports push notifications.", "type": [ "boolean", "null" ] }, "stateTransitionHistory": { "description": "Whether the agent supports state transition history.", "type": [ "boolean", "null" ] }, "streaming": { "description": "Whether the agent supports streaming responses.", "type": [ "boolean", "null" ] } }, "type": "object", "x-versionadded": "v2.46" }, "defaultInputModes": { "description": "Default mime types or modes accepted as input.", "items": { "type": "string" }, "maxItems": 100, "type": "array" }, "defaultOutputModes": { "description": "Default mime types or modes produced as output.", "items": { "type": "string" }, "maxItems": 100, "type": "array" }, "description": { "description": "A human-readable description of the agent.", "type": "string" }, "documentationUrl": { "description": "Url to the agent's documentation.", "type": [ "string", "null" ] }, "iconUrl": { "description": "Url to an icon representing the agent.", "type": [ "string", "null" ] }, "name": { "description": "The display name of the agent.", "type": "string" }, "preferredTransport": { "default": "JSONRPC", "description": "Preferred transport protocol (e.g. jsonrpc, grpc, http+json). defaults to 'jsonrpc'.", "type": [ "string", "null" ] }, "protocolVersion": { "default": "0.3.0", "description": "The a2a protocol version. defaults to '0.3.0'.", "type": [ "string", "null" ] }, "provider": { "description": "The service provider of the agent.", "properties": { "organization": { "description": "The organization name of the agent provider.", "type": "string" }, "url": { "description": "The url of the agent provider.", "type": "string" } }, "required": [ "organization", "url" ], "type": "object", "x-versionadded": "v2.46" }, "security": { "description": "Global security requirements. each entry maps a scheme name to required scopes.", "items": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "maxItems": 100, "type": [ "array", "null" ] }, "securitySchemes": { "additionalProperties": { "oneOf": [ { "properties": { "description": { "description": "A description of the API key security scheme.", "type": [ "string", "null" ] }, "in": { "description": "The location of the API key.", "enum": [ "cookie", "header", "query" ], "type": "string" }, "name": { "description": "The name of the API key parameter.", "type": "string" }, "type": { "default": "apiKey", "description": "The security scheme type; must be 'apikey'.", "enum": [ "apiKey" ], "type": "string" } }, "required": [ "in", "name" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "bearerFormat": { "description": "A hint to the client about the bearer token format (e.g. 'jwt').", "type": [ "string", "null" ] }, "description": { "description": "A description of the HTTP auth security scheme.", "type": [ "string", "null" ] }, "scheme": { "description": "The HTTP authentication scheme (e.g. 'bearer', 'basic').", "type": "string" }, "type": { "default": "http", "description": "The security scheme type; must be 'http'.", "enum": [ "http" ], "type": "string" } }, "required": [ "scheme" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the oauth 2.0 security scheme.", "type": [ "string", "null" ] }, "flows": { "description": "The supported oauth 2.0 flows.", "properties": { "authorizationCode": { "description": "Configuration for the oauth 2.0 authorization code flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the authorization code flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for exchanging the authorization code.", "type": "string" } }, "required": [ "authorizationUrl", "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "clientCredentials": { "description": "Configuration for the oauth 2.0 client credentials flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the client credentials flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "implicit": { "description": "Configuration for the oauth 2.0 implicit flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the implicit flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" } }, "required": [ "authorizationUrl", "scopes" ], "type": "object", "x-versionadded": "v2.46" }, "password": { "description": "Configuration for the oauth 2.0 resource owner password flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the password flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" } }, "type": "object", "x-versionadded": "v2.46" }, "oauth2MetadataUrl": { "description": "Url of the oauth 2.0 server metadata document.", "type": [ "string", "null" ] }, "type": { "default": "oauth2", "description": "The security scheme type; must be 'oauth2'.", "enum": [ "oauth2" ], "type": "string" } }, "required": [ "flows" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the openid connect security scheme.", "type": [ "string", "null" ] }, "openIdConnectUrl": { "description": "The openid connect discovery document url.", "type": "string" }, "type": { "default": "openIdConnect", "description": "The security scheme type; must be 'openidconnect'.", "enum": [ "openIdConnect" ], "type": "string" } }, "required": [ "openIdConnectUrl" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the mtls security scheme.", "type": [ "string", "null" ] }, "type": { "default": "mutualTLS", "description": "The security scheme type; must be 'mutualtls'.", "enum": [ "mutualTLS" ], "type": "string" } }, "type": "object", "x-versionadded": "v2.46" } ] }, "description": "Map of security scheme names to their definitions.", "type": "object" }, "signatures": { "description": "Jws signatures over this agent card.", "items": { "properties": { "header": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "protected": { "description": "The base64url-encoded jws protected header.", "type": "string" }, "signature": { "description": "The base64url-encoded jws signature value.", "type": "string" } }, "required": [ "protected", "signature" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "skills": { "description": "The list of skills this agent can perform.", "items": { "properties": { "description": { "description": "A human-readable description of the skill.", "type": "string" }, "examples": { "description": "Example prompts or inputs for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "id": { "description": "Unique identifier for the skill.", "type": "string" }, "inputModes": { "description": "Mime types or modes accepted as input.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "name": { "description": "The display name of the skill.", "type": "string" }, "outputModes": { "description": "Mime types or modes produced as output.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "security": { "description": "Security requirements for the skill. each entry maps a scheme name to required scopes.", "items": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "maxItems": 100, "type": [ "array", "null" ] }, "tags": { "description": "Categorization tags for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": "array" } }, "required": [ "description", "id", "name", "tags" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": "array" }, "supportsAuthenticatedExtendedCard": { "description": "Whether the agent exposes an authenticated extended card.", "type": [ "boolean", "null" ] }, "url": { "description": "The primary endpoint url for the agent.", "type": "string" }, "version": { "description": "The version of the agent (e.g. '1.0.0').", "type": "string" } }, "required": [ "capabilities", "defaultInputModes", "defaultOutputModes", "description", "name", "skills", "url", "version" ], "type": "object", "x-versionadded": "v2.46" }, "createdAt": { "description": "The date and time of when agent has been deployed and it's agent card registered, in iso 8601 format.", "format": "date-time", "type": "string" }, "deploymentId": { "description": "The ID of the agent deployment. every agent card has exactly one of deploymentid or workloadid set.", "type": [ "string", "null" ] }, "externalId": { "description": "The external ID of the agent.", "type": [ "string", "null" ] }, "id": { "description": "The ID of the agent card.", "type": "string" }, "tenantId": { "description": "The ID of the tenant agent card is registered in.", "maxLength": 36, "minLength": 32, "type": [ "string", "null" ] }, "updatedAt": { "description": "The date and time indicating when the agent card was last updated.", "format": "date-time", "type": "string" }, "workloadId": { "description": "The ID of the workload API workload the agent runs as. every agent card has exactly one of deploymentid or workloadid set.", "type": [ "string", "null" ], "x-versionadded": "v2.48" } }, "required": [ "agentCard", "createdAt", "id", "updatedAt" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": "array" }, "next": { "description": "The url of the next page (if null, there is no next page).", "format": "uri", "type": [ "string", "null" ] }, "previous": { "description": "The url of the previous page (if null, there is no previous page).", "format": "uri", "type": [ "string", "null" ] }, "totalCount": { "description": "The total number of items across all pages.", "type": "integer" } }, "required": [ "data", "next", "previous", "totalCount" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | The list of agent cards registered within tenant. | AgentCardListResponse | ## Delete the agent card by deployment ID Operation path: `DELETE /api/v2/deployments/{deploymentId}/agentCard/` Authentication requirements: `BearerAuth` Delete the agent card associated with an external agent deployment. This operation is idempotent - returns 204 even if no agent card exists. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | deploymentId | path | string | true | The ID of the deployment. | ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | none | None | | 404 | Not Found | Deployment not found. | None | | 405 | Method Not Allowed | Agent card deletion is only allowed for external deployments. | None | ## Retrieve the agent card by deployment ID Operation path: `GET /api/v2/deployments/{deploymentId}/agentCard/` Authentication requirements: `BearerAuth` Retrieve the agent card associated with an agent deployment. The agent card is returned by this route as it was returned by the agent during deployment creation or as it was uploaded to DataRobot. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | deploymentId | path | string | true | The ID of the deployment. | ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | The agent card for the deployment. | None | | 404 | Not Found | No agent card found for this deployment. | None | ## Create by deployment ID Operation path: `PUT /api/v2/deployments/{deploymentId}/agentCard/` Authentication requirements: `BearerAuth` Upload the agent card for an agent deployment. The request body must be the agent card document as a JSON object. If an agent card already exists for this deployment it will be replaced. This operation is only supported for external deployments. For DataRobot hosted deployments, the agent card document is pulled automatically from the deployment. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | deploymentId | path | string | true | The ID of the deployment. | ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | The stored agent card for the deployment. | None | | 413 | Payload Too Large | The agent card exceeds the maximum allowed size. | None | | 422 | Unprocessable Entity | Agent card upload is only supported for external deployments. | None | # Schemas ## APIKeySecurityScheme ``` { "properties": { "description": { "description": "A description of the API key security scheme.", "type": [ "string", "null" ] }, "in": { "description": "The location of the API key.", "enum": [ "cookie", "header", "query" ], "type": "string" }, "name": { "description": "The name of the API key parameter.", "type": "string" }, "type": { "default": "apiKey", "description": "The security scheme type; must be 'apikey'.", "enum": [ "apiKey" ], "type": "string" } }, "required": [ "in", "name" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | string,null | false | | A description of the API key security scheme. | | in | string | true | | The location of the API key. | | name | string | true | | The name of the API key parameter. | | type | string | false | | The security scheme type; must be 'apikey'. | ### Enumerated Values | Property | Value | | --- | --- | | in | [cookie, header, query] | | type | apiKey | ## AgentCapabilities ``` { "description": "Optional capabilities supported by the agent.", "properties": { "extensions": { "description": "Protocol extensions supported by the agent.", "items": { "properties": { "description": { "description": "A human-readable description of the extension.", "type": [ "string", "null" ] }, "params": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "required": { "description": "Whether this extension is required by the agent.", "type": [ "boolean", "null" ] }, "uri": { "description": "The uri identifying the extension.", "type": "string" } }, "required": [ "uri" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "pushNotifications": { "description": "Whether the agent supports push notifications.", "type": [ "boolean", "null" ] }, "stateTransitionHistory": { "description": "Whether the agent supports state transition history.", "type": [ "boolean", "null" ] }, "streaming": { "description": "Whether the agent supports streaming responses.", "type": [ "boolean", "null" ] } }, "type": "object", "x-versionadded": "v2.46" } ``` Optional capabilities supported by the agent. ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | extensions | array,null | false | maxItems: 100 | Protocol extensions supported by the agent. | | pushNotifications | boolean,null | false | | Whether the agent supports push notifications. | | stateTransitionHistory | boolean,null | false | | Whether the agent supports state transition history. | | streaming | boolean,null | false | | Whether the agent supports streaming responses. | ## AgentCard ``` { "description": "Plain agent card data.", "properties": { "additionalInterfaces": { "description": "Additional transport interfaces beyond the primary url.", "items": { "properties": { "transport": { "description": "The transport protocol (e.g. jsonrpc, grpc, http+json).", "type": "string" }, "url": { "description": "The url for this transport interface.", "type": "string" } }, "required": [ "transport", "url" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "capabilities": { "description": "Optional capabilities supported by the agent.", "properties": { "extensions": { "description": "Protocol extensions supported by the agent.", "items": { "properties": { "description": { "description": "A human-readable description of the extension.", "type": [ "string", "null" ] }, "params": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "required": { "description": "Whether this extension is required by the agent.", "type": [ "boolean", "null" ] }, "uri": { "description": "The uri identifying the extension.", "type": "string" } }, "required": [ "uri" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "pushNotifications": { "description": "Whether the agent supports push notifications.", "type": [ "boolean", "null" ] }, "stateTransitionHistory": { "description": "Whether the agent supports state transition history.", "type": [ "boolean", "null" ] }, "streaming": { "description": "Whether the agent supports streaming responses.", "type": [ "boolean", "null" ] } }, "type": "object", "x-versionadded": "v2.46" }, "defaultInputModes": { "description": "Default mime types or modes accepted as input.", "items": { "type": "string" }, "maxItems": 100, "type": "array" }, "defaultOutputModes": { "description": "Default mime types or modes produced as output.", "items": { "type": "string" }, "maxItems": 100, "type": "array" }, "description": { "description": "A human-readable description of the agent.", "type": "string" }, "documentationUrl": { "description": "Url to the agent's documentation.", "type": [ "string", "null" ] }, "iconUrl": { "description": "Url to an icon representing the agent.", "type": [ "string", "null" ] }, "name": { "description": "The display name of the agent.", "type": "string" }, "preferredTransport": { "default": "JSONRPC", "description": "Preferred transport protocol (e.g. jsonrpc, grpc, http+json). defaults to 'jsonrpc'.", "type": [ "string", "null" ] }, "protocolVersion": { "default": "0.3.0", "description": "The a2a protocol version. defaults to '0.3.0'.", "type": [ "string", "null" ] }, "provider": { "description": "The service provider of the agent.", "properties": { "organization": { "description": "The organization name of the agent provider.", "type": "string" }, "url": { "description": "The url of the agent provider.", "type": "string" } }, "required": [ "organization", "url" ], "type": "object", "x-versionadded": "v2.46" }, "security": { "description": "Global security requirements. each entry maps a scheme name to required scopes.", "items": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "maxItems": 100, "type": [ "array", "null" ] }, "securitySchemes": { "additionalProperties": { "oneOf": [ { "properties": { "description": { "description": "A description of the API key security scheme.", "type": [ "string", "null" ] }, "in": { "description": "The location of the API key.", "enum": [ "cookie", "header", "query" ], "type": "string" }, "name": { "description": "The name of the API key parameter.", "type": "string" }, "type": { "default": "apiKey", "description": "The security scheme type; must be 'apikey'.", "enum": [ "apiKey" ], "type": "string" } }, "required": [ "in", "name" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "bearerFormat": { "description": "A hint to the client about the bearer token format (e.g. 'jwt').", "type": [ "string", "null" ] }, "description": { "description": "A description of the HTTP auth security scheme.", "type": [ "string", "null" ] }, "scheme": { "description": "The HTTP authentication scheme (e.g. 'bearer', 'basic').", "type": "string" }, "type": { "default": "http", "description": "The security scheme type; must be 'http'.", "enum": [ "http" ], "type": "string" } }, "required": [ "scheme" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the oauth 2.0 security scheme.", "type": [ "string", "null" ] }, "flows": { "description": "The supported oauth 2.0 flows.", "properties": { "authorizationCode": { "description": "Configuration for the oauth 2.0 authorization code flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the authorization code flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for exchanging the authorization code.", "type": "string" } }, "required": [ "authorizationUrl", "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "clientCredentials": { "description": "Configuration for the oauth 2.0 client credentials flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the client credentials flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "implicit": { "description": "Configuration for the oauth 2.0 implicit flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the implicit flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" } }, "required": [ "authorizationUrl", "scopes" ], "type": "object", "x-versionadded": "v2.46" }, "password": { "description": "Configuration for the oauth 2.0 resource owner password flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the password flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" } }, "type": "object", "x-versionadded": "v2.46" }, "oauth2MetadataUrl": { "description": "Url of the oauth 2.0 server metadata document.", "type": [ "string", "null" ] }, "type": { "default": "oauth2", "description": "The security scheme type; must be 'oauth2'.", "enum": [ "oauth2" ], "type": "string" } }, "required": [ "flows" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the openid connect security scheme.", "type": [ "string", "null" ] }, "openIdConnectUrl": { "description": "The openid connect discovery document url.", "type": "string" }, "type": { "default": "openIdConnect", "description": "The security scheme type; must be 'openidconnect'.", "enum": [ "openIdConnect" ], "type": "string" } }, "required": [ "openIdConnectUrl" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the mtls security scheme.", "type": [ "string", "null" ] }, "type": { "default": "mutualTLS", "description": "The security scheme type; must be 'mutualtls'.", "enum": [ "mutualTLS" ], "type": "string" } }, "type": "object", "x-versionadded": "v2.46" } ] }, "description": "Map of security scheme names to their definitions.", "type": "object" }, "signatures": { "description": "Jws signatures over this agent card.", "items": { "properties": { "header": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "protected": { "description": "The base64url-encoded jws protected header.", "type": "string" }, "signature": { "description": "The base64url-encoded jws signature value.", "type": "string" } }, "required": [ "protected", "signature" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "skills": { "description": "The list of skills this agent can perform.", "items": { "properties": { "description": { "description": "A human-readable description of the skill.", "type": "string" }, "examples": { "description": "Example prompts or inputs for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "id": { "description": "Unique identifier for the skill.", "type": "string" }, "inputModes": { "description": "Mime types or modes accepted as input.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "name": { "description": "The display name of the skill.", "type": "string" }, "outputModes": { "description": "Mime types or modes produced as output.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "security": { "description": "Security requirements for the skill. each entry maps a scheme name to required scopes.", "items": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "maxItems": 100, "type": [ "array", "null" ] }, "tags": { "description": "Categorization tags for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": "array" } }, "required": [ "description", "id", "name", "tags" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": "array" }, "supportsAuthenticatedExtendedCard": { "description": "Whether the agent exposes an authenticated extended card.", "type": [ "boolean", "null" ] }, "url": { "description": "The primary endpoint url for the agent.", "type": "string" }, "version": { "description": "The version of the agent (e.g. '1.0.0').", "type": "string" } }, "required": [ "capabilities", "defaultInputModes", "defaultOutputModes", "description", "name", "skills", "url", "version" ], "type": "object", "x-versionadded": "v2.46" } ``` Plain agent card data. ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | additionalInterfaces | array,null | false | maxItems: 100 | Additional transport interfaces beyond the primary url. | | capabilities | AgentCapabilities | true | | Optional capabilities supported by the agent. | | defaultInputModes | [string] | true | maxItems: 100 | Default mime types or modes accepted as input. | | defaultOutputModes | [string] | true | maxItems: 100 | Default mime types or modes produced as output. | | description | string | true | | A human-readable description of the agent. | | documentationUrl | string,null | false | | Url to the agent's documentation. | | iconUrl | string,null | false | | Url to an icon representing the agent. | | name | string | true | | The display name of the agent. | | preferredTransport | string,null | false | | Preferred transport protocol (e.g. jsonrpc, grpc, http+json). defaults to 'jsonrpc'. | | protocolVersion | string,null | false | | The a2a protocol version. defaults to '0.3.0'. | | provider | AgentProvider | false | | The service provider of the agent. | | security | array,null | false | maxItems: 100 | Global security requirements. each entry maps a scheme name to required scopes. | | securitySchemes | object | false | | Map of security scheme names to their definitions. | | » additionalProperties | any | false | | none | oneOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | »» anonymous | APIKeySecurityScheme | false | | none | xor | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | »» anonymous | HTTPAuthSecurityScheme | false | | none | xor | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | »» anonymous | OAuth2SecurityScheme | false | | none | xor | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | »» anonymous | OpenIdConnectSecurityScheme | false | | none | xor | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | »» anonymous | MutualTLSSecurityScheme | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | signatures | array,null | false | maxItems: 100 | Jws signatures over this agent card. | | skills | [AgentSkill] | true | maxItems: 100 | The list of skills this agent can perform. | | supportsAuthenticatedExtendedCard | boolean,null | false | | Whether the agent exposes an authenticated extended card. | | url | string | true | | The primary endpoint url for the agent. | | version | string | true | | The version of the agent (e.g. '1.0.0'). | ## AgentCardListResponse ``` { "properties": { "count": { "description": "The number of items returned on this page.", "type": "integer" }, "data": { "description": "The list of formatted agent cards.", "items": { "properties": { "agentCard": { "description": "Plain agent card data.", "properties": { "additionalInterfaces": { "description": "Additional transport interfaces beyond the primary url.", "items": { "properties": { "transport": { "description": "The transport protocol (e.g. jsonrpc, grpc, http+json).", "type": "string" }, "url": { "description": "The url for this transport interface.", "type": "string" } }, "required": [ "transport", "url" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "capabilities": { "description": "Optional capabilities supported by the agent.", "properties": { "extensions": { "description": "Protocol extensions supported by the agent.", "items": { "properties": { "description": { "description": "A human-readable description of the extension.", "type": [ "string", "null" ] }, "params": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "required": { "description": "Whether this extension is required by the agent.", "type": [ "boolean", "null" ] }, "uri": { "description": "The uri identifying the extension.", "type": "string" } }, "required": [ "uri" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "pushNotifications": { "description": "Whether the agent supports push notifications.", "type": [ "boolean", "null" ] }, "stateTransitionHistory": { "description": "Whether the agent supports state transition history.", "type": [ "boolean", "null" ] }, "streaming": { "description": "Whether the agent supports streaming responses.", "type": [ "boolean", "null" ] } }, "type": "object", "x-versionadded": "v2.46" }, "defaultInputModes": { "description": "Default mime types or modes accepted as input.", "items": { "type": "string" }, "maxItems": 100, "type": "array" }, "defaultOutputModes": { "description": "Default mime types or modes produced as output.", "items": { "type": "string" }, "maxItems": 100, "type": "array" }, "description": { "description": "A human-readable description of the agent.", "type": "string" }, "documentationUrl": { "description": "Url to the agent's documentation.", "type": [ "string", "null" ] }, "iconUrl": { "description": "Url to an icon representing the agent.", "type": [ "string", "null" ] }, "name": { "description": "The display name of the agent.", "type": "string" }, "preferredTransport": { "default": "JSONRPC", "description": "Preferred transport protocol (e.g. jsonrpc, grpc, http+json). defaults to 'jsonrpc'.", "type": [ "string", "null" ] }, "protocolVersion": { "default": "0.3.0", "description": "The a2a protocol version. defaults to '0.3.0'.", "type": [ "string", "null" ] }, "provider": { "description": "The service provider of the agent.", "properties": { "organization": { "description": "The organization name of the agent provider.", "type": "string" }, "url": { "description": "The url of the agent provider.", "type": "string" } }, "required": [ "organization", "url" ], "type": "object", "x-versionadded": "v2.46" }, "security": { "description": "Global security requirements. each entry maps a scheme name to required scopes.", "items": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "maxItems": 100, "type": [ "array", "null" ] }, "securitySchemes": { "additionalProperties": { "oneOf": [ { "properties": { "description": { "description": "A description of the API key security scheme.", "type": [ "string", "null" ] }, "in": { "description": "The location of the API key.", "enum": [ "cookie", "header", "query" ], "type": "string" }, "name": { "description": "The name of the API key parameter.", "type": "string" }, "type": { "default": "apiKey", "description": "The security scheme type; must be 'apikey'.", "enum": [ "apiKey" ], "type": "string" } }, "required": [ "in", "name" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "bearerFormat": { "description": "A hint to the client about the bearer token format (e.g. 'jwt').", "type": [ "string", "null" ] }, "description": { "description": "A description of the HTTP auth security scheme.", "type": [ "string", "null" ] }, "scheme": { "description": "The HTTP authentication scheme (e.g. 'bearer', 'basic').", "type": "string" }, "type": { "default": "http", "description": "The security scheme type; must be 'http'.", "enum": [ "http" ], "type": "string" } }, "required": [ "scheme" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the oauth 2.0 security scheme.", "type": [ "string", "null" ] }, "flows": { "description": "The supported oauth 2.0 flows.", "properties": { "authorizationCode": { "description": "Configuration for the oauth 2.0 authorization code flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the authorization code flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for exchanging the authorization code.", "type": "string" } }, "required": [ "authorizationUrl", "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "clientCredentials": { "description": "Configuration for the oauth 2.0 client credentials flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the client credentials flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "implicit": { "description": "Configuration for the oauth 2.0 implicit flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the implicit flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" } }, "required": [ "authorizationUrl", "scopes" ], "type": "object", "x-versionadded": "v2.46" }, "password": { "description": "Configuration for the oauth 2.0 resource owner password flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the password flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" } }, "type": "object", "x-versionadded": "v2.46" }, "oauth2MetadataUrl": { "description": "Url of the oauth 2.0 server metadata document.", "type": [ "string", "null" ] }, "type": { "default": "oauth2", "description": "The security scheme type; must be 'oauth2'.", "enum": [ "oauth2" ], "type": "string" } }, "required": [ "flows" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the openid connect security scheme.", "type": [ "string", "null" ] }, "openIdConnectUrl": { "description": "The openid connect discovery document url.", "type": "string" }, "type": { "default": "openIdConnect", "description": "The security scheme type; must be 'openidconnect'.", "enum": [ "openIdConnect" ], "type": "string" } }, "required": [ "openIdConnectUrl" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the mtls security scheme.", "type": [ "string", "null" ] }, "type": { "default": "mutualTLS", "description": "The security scheme type; must be 'mutualtls'.", "enum": [ "mutualTLS" ], "type": "string" } }, "type": "object", "x-versionadded": "v2.46" } ] }, "description": "Map of security scheme names to their definitions.", "type": "object" }, "signatures": { "description": "Jws signatures over this agent card.", "items": { "properties": { "header": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "protected": { "description": "The base64url-encoded jws protected header.", "type": "string" }, "signature": { "description": "The base64url-encoded jws signature value.", "type": "string" } }, "required": [ "protected", "signature" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "skills": { "description": "The list of skills this agent can perform.", "items": { "properties": { "description": { "description": "A human-readable description of the skill.", "type": "string" }, "examples": { "description": "Example prompts or inputs for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "id": { "description": "Unique identifier for the skill.", "type": "string" }, "inputModes": { "description": "Mime types or modes accepted as input.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "name": { "description": "The display name of the skill.", "type": "string" }, "outputModes": { "description": "Mime types or modes produced as output.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "security": { "description": "Security requirements for the skill. each entry maps a scheme name to required scopes.", "items": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "maxItems": 100, "type": [ "array", "null" ] }, "tags": { "description": "Categorization tags for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": "array" } }, "required": [ "description", "id", "name", "tags" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": "array" }, "supportsAuthenticatedExtendedCard": { "description": "Whether the agent exposes an authenticated extended card.", "type": [ "boolean", "null" ] }, "url": { "description": "The primary endpoint url for the agent.", "type": "string" }, "version": { "description": "The version of the agent (e.g. '1.0.0').", "type": "string" } }, "required": [ "capabilities", "defaultInputModes", "defaultOutputModes", "description", "name", "skills", "url", "version" ], "type": "object", "x-versionadded": "v2.46" }, "createdAt": { "description": "The date and time of when agent has been deployed and it's agent card registered, in iso 8601 format.", "format": "date-time", "type": "string" }, "deploymentId": { "description": "The ID of the agent deployment. every agent card has exactly one of deploymentid or workloadid set.", "type": [ "string", "null" ] }, "externalId": { "description": "The external ID of the agent.", "type": [ "string", "null" ] }, "id": { "description": "The ID of the agent card.", "type": "string" }, "tenantId": { "description": "The ID of the tenant agent card is registered in.", "maxLength": 36, "minLength": 32, "type": [ "string", "null" ] }, "updatedAt": { "description": "The date and time indicating when the agent card was last updated.", "format": "date-time", "type": "string" }, "workloadId": { "description": "The ID of the workload API workload the agent runs as. every agent card has exactly one of deploymentid or workloadid set.", "type": [ "string", "null" ], "x-versionadded": "v2.48" } }, "required": [ "agentCard", "createdAt", "id", "updatedAt" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": "array" }, "next": { "description": "The url of the next page (if null, there is no next page).", "format": "uri", "type": [ "string", "null" ] }, "previous": { "description": "The url of the previous page (if null, there is no previous page).", "format": "uri", "type": [ "string", "null" ] }, "totalCount": { "description": "The total number of items across all pages.", "type": "integer" } }, "required": [ "data", "next", "previous", "totalCount" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | count | integer | false | | The number of items returned on this page. | | data | [AgentCardResponse] | true | maxItems: 100 | The list of formatted agent cards. | | next | string,null(uri) | true | | The url of the next page (if null, there is no next page). | | previous | string,null(uri) | true | | The url of the previous page (if null, there is no previous page). | | totalCount | integer | true | | The total number of items across all pages. | ## AgentCardResponse ``` { "properties": { "agentCard": { "description": "Plain agent card data.", "properties": { "additionalInterfaces": { "description": "Additional transport interfaces beyond the primary url.", "items": { "properties": { "transport": { "description": "The transport protocol (e.g. jsonrpc, grpc, http+json).", "type": "string" }, "url": { "description": "The url for this transport interface.", "type": "string" } }, "required": [ "transport", "url" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "capabilities": { "description": "Optional capabilities supported by the agent.", "properties": { "extensions": { "description": "Protocol extensions supported by the agent.", "items": { "properties": { "description": { "description": "A human-readable description of the extension.", "type": [ "string", "null" ] }, "params": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "required": { "description": "Whether this extension is required by the agent.", "type": [ "boolean", "null" ] }, "uri": { "description": "The uri identifying the extension.", "type": "string" } }, "required": [ "uri" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "pushNotifications": { "description": "Whether the agent supports push notifications.", "type": [ "boolean", "null" ] }, "stateTransitionHistory": { "description": "Whether the agent supports state transition history.", "type": [ "boolean", "null" ] }, "streaming": { "description": "Whether the agent supports streaming responses.", "type": [ "boolean", "null" ] } }, "type": "object", "x-versionadded": "v2.46" }, "defaultInputModes": { "description": "Default mime types or modes accepted as input.", "items": { "type": "string" }, "maxItems": 100, "type": "array" }, "defaultOutputModes": { "description": "Default mime types or modes produced as output.", "items": { "type": "string" }, "maxItems": 100, "type": "array" }, "description": { "description": "A human-readable description of the agent.", "type": "string" }, "documentationUrl": { "description": "Url to the agent's documentation.", "type": [ "string", "null" ] }, "iconUrl": { "description": "Url to an icon representing the agent.", "type": [ "string", "null" ] }, "name": { "description": "The display name of the agent.", "type": "string" }, "preferredTransport": { "default": "JSONRPC", "description": "Preferred transport protocol (e.g. jsonrpc, grpc, http+json). defaults to 'jsonrpc'.", "type": [ "string", "null" ] }, "protocolVersion": { "default": "0.3.0", "description": "The a2a protocol version. defaults to '0.3.0'.", "type": [ "string", "null" ] }, "provider": { "description": "The service provider of the agent.", "properties": { "organization": { "description": "The organization name of the agent provider.", "type": "string" }, "url": { "description": "The url of the agent provider.", "type": "string" } }, "required": [ "organization", "url" ], "type": "object", "x-versionadded": "v2.46" }, "security": { "description": "Global security requirements. each entry maps a scheme name to required scopes.", "items": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "maxItems": 100, "type": [ "array", "null" ] }, "securitySchemes": { "additionalProperties": { "oneOf": [ { "properties": { "description": { "description": "A description of the API key security scheme.", "type": [ "string", "null" ] }, "in": { "description": "The location of the API key.", "enum": [ "cookie", "header", "query" ], "type": "string" }, "name": { "description": "The name of the API key parameter.", "type": "string" }, "type": { "default": "apiKey", "description": "The security scheme type; must be 'apikey'.", "enum": [ "apiKey" ], "type": "string" } }, "required": [ "in", "name" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "bearerFormat": { "description": "A hint to the client about the bearer token format (e.g. 'jwt').", "type": [ "string", "null" ] }, "description": { "description": "A description of the HTTP auth security scheme.", "type": [ "string", "null" ] }, "scheme": { "description": "The HTTP authentication scheme (e.g. 'bearer', 'basic').", "type": "string" }, "type": { "default": "http", "description": "The security scheme type; must be 'http'.", "enum": [ "http" ], "type": "string" } }, "required": [ "scheme" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the oauth 2.0 security scheme.", "type": [ "string", "null" ] }, "flows": { "description": "The supported oauth 2.0 flows.", "properties": { "authorizationCode": { "description": "Configuration for the oauth 2.0 authorization code flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the authorization code flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for exchanging the authorization code.", "type": "string" } }, "required": [ "authorizationUrl", "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "clientCredentials": { "description": "Configuration for the oauth 2.0 client credentials flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the client credentials flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "implicit": { "description": "Configuration for the oauth 2.0 implicit flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the implicit flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" } }, "required": [ "authorizationUrl", "scopes" ], "type": "object", "x-versionadded": "v2.46" }, "password": { "description": "Configuration for the oauth 2.0 resource owner password flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the password flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" } }, "type": "object", "x-versionadded": "v2.46" }, "oauth2MetadataUrl": { "description": "Url of the oauth 2.0 server metadata document.", "type": [ "string", "null" ] }, "type": { "default": "oauth2", "description": "The security scheme type; must be 'oauth2'.", "enum": [ "oauth2" ], "type": "string" } }, "required": [ "flows" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the openid connect security scheme.", "type": [ "string", "null" ] }, "openIdConnectUrl": { "description": "The openid connect discovery document url.", "type": "string" }, "type": { "default": "openIdConnect", "description": "The security scheme type; must be 'openidconnect'.", "enum": [ "openIdConnect" ], "type": "string" } }, "required": [ "openIdConnectUrl" ], "type": "object", "x-versionadded": "v2.46" }, { "properties": { "description": { "description": "A description of the mtls security scheme.", "type": [ "string", "null" ] }, "type": { "default": "mutualTLS", "description": "The security scheme type; must be 'mutualtls'.", "enum": [ "mutualTLS" ], "type": "string" } }, "type": "object", "x-versionadded": "v2.46" } ] }, "description": "Map of security scheme names to their definitions.", "type": "object" }, "signatures": { "description": "Jws signatures over this agent card.", "items": { "properties": { "header": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "protected": { "description": "The base64url-encoded jws protected header.", "type": "string" }, "signature": { "description": "The base64url-encoded jws signature value.", "type": "string" } }, "required": [ "protected", "signature" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": [ "array", "null" ] }, "skills": { "description": "The list of skills this agent can perform.", "items": { "properties": { "description": { "description": "A human-readable description of the skill.", "type": "string" }, "examples": { "description": "Example prompts or inputs for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "id": { "description": "Unique identifier for the skill.", "type": "string" }, "inputModes": { "description": "Mime types or modes accepted as input.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "name": { "description": "The display name of the skill.", "type": "string" }, "outputModes": { "description": "Mime types or modes produced as output.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "security": { "description": "Security requirements for the skill. each entry maps a scheme name to required scopes.", "items": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "maxItems": 100, "type": [ "array", "null" ] }, "tags": { "description": "Categorization tags for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": "array" } }, "required": [ "description", "id", "name", "tags" ], "type": "object", "x-versionadded": "v2.46" }, "maxItems": 100, "type": "array" }, "supportsAuthenticatedExtendedCard": { "description": "Whether the agent exposes an authenticated extended card.", "type": [ "boolean", "null" ] }, "url": { "description": "The primary endpoint url for the agent.", "type": "string" }, "version": { "description": "The version of the agent (e.g. '1.0.0').", "type": "string" } }, "required": [ "capabilities", "defaultInputModes", "defaultOutputModes", "description", "name", "skills", "url", "version" ], "type": "object", "x-versionadded": "v2.46" }, "createdAt": { "description": "The date and time of when agent has been deployed and it's agent card registered, in iso 8601 format.", "format": "date-time", "type": "string" }, "deploymentId": { "description": "The ID of the agent deployment. every agent card has exactly one of deploymentid or workloadid set.", "type": [ "string", "null" ] }, "externalId": { "description": "The external ID of the agent.", "type": [ "string", "null" ] }, "id": { "description": "The ID of the agent card.", "type": "string" }, "tenantId": { "description": "The ID of the tenant agent card is registered in.", "maxLength": 36, "minLength": 32, "type": [ "string", "null" ] }, "updatedAt": { "description": "The date and time indicating when the agent card was last updated.", "format": "date-time", "type": "string" }, "workloadId": { "description": "The ID of the workload API workload the agent runs as. every agent card has exactly one of deploymentid or workloadid set.", "type": [ "string", "null" ], "x-versionadded": "v2.48" } }, "required": [ "agentCard", "createdAt", "id", "updatedAt" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | agentCard | AgentCard | true | | Plain agent card data. | | createdAt | string(date-time) | true | | The date and time of when agent has been deployed and it's agent card registered, in iso 8601 format. | | deploymentId | string,null | false | | The ID of the agent deployment. every agent card has exactly one of deploymentid or workloadid set. | | externalId | string,null | false | | The external ID of the agent. | | id | string | true | | The ID of the agent card. | | tenantId | string,null | false | maxLength: 36minLength: 32minLength: 32 | The ID of the tenant agent card is registered in. | | updatedAt | string(date-time) | true | | The date and time indicating when the agent card was last updated. | | workloadId | string,null | false | | The ID of the workload API workload the agent runs as. every agent card has exactly one of deploymentid or workloadid set. | ## AgentCardSignature ``` { "properties": { "header": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "protected": { "description": "The base64url-encoded jws protected header.", "type": "string" }, "signature": { "description": "The base64url-encoded jws signature value.", "type": "string" } }, "required": [ "protected", "signature" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | header | AllowExtra | false | | The parameters submitted by the user to the failed job. | | protected | string | true | | The base64url-encoded jws protected header. | | signature | string | true | | The base64url-encoded jws signature value. | ## AgentExtension ``` { "properties": { "description": { "description": "A human-readable description of the extension.", "type": [ "string", "null" ] }, "params": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "required": { "description": "Whether this extension is required by the agent.", "type": [ "boolean", "null" ] }, "uri": { "description": "The uri identifying the extension.", "type": "string" } }, "required": [ "uri" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | string,null | false | | A human-readable description of the extension. | | params | AllowExtra | false | | The parameters submitted by the user to the failed job. | | required | boolean,null | false | | Whether this extension is required by the agent. | | uri | string | true | | The uri identifying the extension. | ## AgentInterface ``` { "properties": { "transport": { "description": "The transport protocol (e.g. jsonrpc, grpc, http+json).", "type": "string" }, "url": { "description": "The url for this transport interface.", "type": "string" } }, "required": [ "transport", "url" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | transport | string | true | | The transport protocol (e.g. jsonrpc, grpc, http+json). | | url | string | true | | The url for this transport interface. | ## AgentProvider ``` { "description": "The service provider of the agent.", "properties": { "organization": { "description": "The organization name of the agent provider.", "type": "string" }, "url": { "description": "The url of the agent provider.", "type": "string" } }, "required": [ "organization", "url" ], "type": "object", "x-versionadded": "v2.46" } ``` The service provider of the agent. ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | organization | string | true | | The organization name of the agent provider. | | url | string | true | | The url of the agent provider. | ## AgentSkill ``` { "properties": { "description": { "description": "A human-readable description of the skill.", "type": "string" }, "examples": { "description": "Example prompts or inputs for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "id": { "description": "Unique identifier for the skill.", "type": "string" }, "inputModes": { "description": "Mime types or modes accepted as input.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "name": { "description": "The display name of the skill.", "type": "string" }, "outputModes": { "description": "Mime types or modes produced as output.", "items": { "type": "string" }, "maxItems": 100, "type": [ "array", "null" ] }, "security": { "description": "Security requirements for the skill. each entry maps a scheme name to required scopes.", "items": { "description": "The parameters submitted by the user to the failed job.", "type": "object" }, "maxItems": 100, "type": [ "array", "null" ] }, "tags": { "description": "Categorization tags for the skill.", "items": { "type": "string" }, "maxItems": 100, "type": "array" } }, "required": [ "description", "id", "name", "tags" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | string | true | | A human-readable description of the skill. | | examples | array,null | false | maxItems: 100 | Example prompts or inputs for the skill. | | id | string | true | | Unique identifier for the skill. | | inputModes | array,null | false | maxItems: 100 | Mime types or modes accepted as input. | | name | string | true | | The display name of the skill. | | outputModes | array,null | false | maxItems: 100 | Mime types or modes produced as output. | | security | array,null | false | maxItems: 100 | Security requirements for the skill. each entry maps a scheme name to required scopes. | | tags | [string] | true | maxItems: 100 | Categorization tags for the skill. | ## AllowExtra ``` { "description": "The parameters submitted by the user to the failed job.", "type": "object" } ``` The parameters submitted by the user to the failed job. ### Properties None ## AuthorizationCodeOAuthFlow ``` { "description": "Configuration for the oauth 2.0 authorization code flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the authorization code flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for exchanging the authorization code.", "type": "string" } }, "required": [ "authorizationUrl", "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" } ``` Configuration for the oauth 2.0 authorization code flow. ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | authorizationUrl | string | true | | The authorization url for the authorization code flow. | | refreshUrl | string,null | false | | The url for refreshing tokens. | | scopes | object | true | | Available scopes mapping scope name to description. | | » additionalProperties | string | false | | none | | tokenUrl | string | true | | The token url for exchanging the authorization code. | ## ClientCredentialsOAuthFlow ``` { "description": "Configuration for the oauth 2.0 client credentials flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the client credentials flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" } ``` Configuration for the oauth 2.0 client credentials flow. ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | refreshUrl | string,null | false | | The url for refreshing tokens. | | scopes | object | true | | Available scopes mapping scope name to description. | | » additionalProperties | string | false | | none | | tokenUrl | string | true | | The token url for the client credentials flow. | ## HTTPAuthSecurityScheme ``` { "properties": { "bearerFormat": { "description": "A hint to the client about the bearer token format (e.g. 'jwt').", "type": [ "string", "null" ] }, "description": { "description": "A description of the HTTP auth security scheme.", "type": [ "string", "null" ] }, "scheme": { "description": "The HTTP authentication scheme (e.g. 'bearer', 'basic').", "type": "string" }, "type": { "default": "http", "description": "The security scheme type; must be 'http'.", "enum": [ "http" ], "type": "string" } }, "required": [ "scheme" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | bearerFormat | string,null | false | | A hint to the client about the bearer token format (e.g. 'jwt'). | | description | string,null | false | | A description of the HTTP auth security scheme. | | scheme | string | true | | The HTTP authentication scheme (e.g. 'bearer', 'basic'). | | type | string | false | | The security scheme type; must be 'http'. | ### Enumerated Values | Property | Value | | --- | --- | | type | http | ## ImplicitOAuthFlow ``` { "description": "Configuration for the oauth 2.0 implicit flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the implicit flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" } }, "required": [ "authorizationUrl", "scopes" ], "type": "object", "x-versionadded": "v2.46" } ``` Configuration for the oauth 2.0 implicit flow. ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | authorizationUrl | string | true | | The authorization url for the implicit flow. | | refreshUrl | string,null | false | | The url for refreshing tokens. | | scopes | object | true | | Available scopes mapping scope name to description. | | » additionalProperties | string | false | | none | ## MutualTLSSecurityScheme ``` { "properties": { "description": { "description": "A description of the mtls security scheme.", "type": [ "string", "null" ] }, "type": { "default": "mutualTLS", "description": "The security scheme type; must be 'mutualtls'.", "enum": [ "mutualTLS" ], "type": "string" } }, "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | string,null | false | | A description of the mtls security scheme. | | type | string | false | | The security scheme type; must be 'mutualtls'. | ### Enumerated Values | Property | Value | | --- | --- | | type | mutualTLS | ## OAuth2SecurityScheme ``` { "properties": { "description": { "description": "A description of the oauth 2.0 security scheme.", "type": [ "string", "null" ] }, "flows": { "description": "The supported oauth 2.0 flows.", "properties": { "authorizationCode": { "description": "Configuration for the oauth 2.0 authorization code flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the authorization code flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for exchanging the authorization code.", "type": "string" } }, "required": [ "authorizationUrl", "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "clientCredentials": { "description": "Configuration for the oauth 2.0 client credentials flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the client credentials flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "implicit": { "description": "Configuration for the oauth 2.0 implicit flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the implicit flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" } }, "required": [ "authorizationUrl", "scopes" ], "type": "object", "x-versionadded": "v2.46" }, "password": { "description": "Configuration for the oauth 2.0 resource owner password flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the password flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" } }, "type": "object", "x-versionadded": "v2.46" }, "oauth2MetadataUrl": { "description": "Url of the oauth 2.0 server metadata document.", "type": [ "string", "null" ] }, "type": { "default": "oauth2", "description": "The security scheme type; must be 'oauth2'.", "enum": [ "oauth2" ], "type": "string" } }, "required": [ "flows" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | string,null | false | | A description of the oauth 2.0 security scheme. | | flows | OAuthFlows | true | | The supported oauth 2.0 flows. | | oauth2MetadataUrl | string,null | false | | Url of the oauth 2.0 server metadata document. | | type | string | false | | The security scheme type; must be 'oauth2'. | ### Enumerated Values | Property | Value | | --- | --- | | type | oauth2 | ## OAuthFlows ``` { "description": "The supported oauth 2.0 flows.", "properties": { "authorizationCode": { "description": "Configuration for the oauth 2.0 authorization code flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the authorization code flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for exchanging the authorization code.", "type": "string" } }, "required": [ "authorizationUrl", "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "clientCredentials": { "description": "Configuration for the oauth 2.0 client credentials flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the client credentials flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" }, "implicit": { "description": "Configuration for the oauth 2.0 implicit flow.", "properties": { "authorizationUrl": { "description": "The authorization url for the implicit flow.", "type": "string" }, "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" } }, "required": [ "authorizationUrl", "scopes" ], "type": "object", "x-versionadded": "v2.46" }, "password": { "description": "Configuration for the oauth 2.0 resource owner password flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the password flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" } }, "type": "object", "x-versionadded": "v2.46" } ``` The supported oauth 2.0 flows. ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | authorizationCode | AuthorizationCodeOAuthFlow | false | | Configuration for the oauth 2.0 authorization code flow. | | clientCredentials | ClientCredentialsOAuthFlow | false | | Configuration for the oauth 2.0 client credentials flow. | | implicit | ImplicitOAuthFlow | false | | Configuration for the oauth 2.0 implicit flow. | | password | PasswordOAuthFlow | false | | Configuration for the oauth 2.0 resource owner password flow. | ## OpenIdConnectSecurityScheme ``` { "properties": { "description": { "description": "A description of the openid connect security scheme.", "type": [ "string", "null" ] }, "openIdConnectUrl": { "description": "The openid connect discovery document url.", "type": "string" }, "type": { "default": "openIdConnect", "description": "The security scheme type; must be 'openidconnect'.", "enum": [ "openIdConnect" ], "type": "string" } }, "required": [ "openIdConnectUrl" ], "type": "object", "x-versionadded": "v2.46" } ``` ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | string,null | false | | A description of the openid connect security scheme. | | openIdConnectUrl | string | true | | The openid connect discovery document url. | | type | string | false | | The security scheme type; must be 'openidconnect'. | ### Enumerated Values | Property | Value | | --- | --- | | type | openIdConnect | ## PasswordOAuthFlow ``` { "description": "Configuration for the oauth 2.0 resource owner password flow.", "properties": { "refreshUrl": { "description": "The url for refreshing tokens.", "type": [ "string", "null" ] }, "scopes": { "additionalProperties": { "type": "string" }, "description": "Available scopes mapping scope name to description.", "type": "object" }, "tokenUrl": { "description": "The token url for the password flow.", "type": "string" } }, "required": [ "scopes", "tokenUrl" ], "type": "object", "x-versionadded": "v2.46" } ``` Configuration for the oauth 2.0 resource owner password flow. ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | refreshUrl | string,null | false | | The url for refreshing tokens. | | scopes | object | true | | Available scopes mapping scope name to description. | | » additionalProperties | string | false | | none | | tokenUrl | string | true | | The token url for the password flow. | --- # Agentic memory service URL: https://docs.datarobot.com/en/docs/api/reference/public-api/agentic_memory.html > Endpoints for managing agentic memory spaces, sessions, and events. Endpoints for managing agentic memory spaces, sessions, and events. ## List memory spaces Operation path: `GET /` List all Memory Spaces accessible by the authenticated user - Covers spaces the user created and spaces shared with them, each carrying the permissions the caller holds on it. - Returns offset/limit pagination with a total count. - Soft-deleted spaces, and spaces belonging to another tenant, are excluded. - filter narrows that set to the spaces whose description or ID contains the given text. It is one search box, so the caller does not have to say which of the two they mean. - ownership narrows it to one half of that set: the spaces the caller created, or the ones shared with them. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | deduplicationKey | query | any | false | Filter memory spaces by deduplication key (exact match). Keys are unique per creator, so this filter matches only spaces you created. | | ownership | query | any | false | Return only the memory spaces you created (owned), or only the ones shared with you (shared). Omit it for both, which is everything you can reach. | | filter | query | any | false | Filter memory spaces by description or ID (case-insensitive substring match on either). Any length is accepted, and blank means no filter, so a caller can narrow the list from the first character they type. A partial ID matches, so the shortened ID shown to a user is enough to find their space. | ### Example responses > 200 Response ``` { "properties": { "items": { "description": "List of items for the current page.", "items": { "description": "Response schema for a single memory space.", "properties": { "createdAt": { "description": "Memory space creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "customInstructions": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Custom prompt instructions used for fact extraction. null means the default mem0 extraction prompt is used.", "title": "Custominstructions" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique per user, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Memory space description.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM model configured for memory extraction.", "title": "Llmmodelname" }, "memorySpaceId": { "description": "Unique memory space identifier.", "format": "uuid", "title": "Memoryspaceid", "type": "string" }, "permissions": { "description": "Actions the authenticated caller may perform on this memory space. the creator of a registered space holds all four; a space shared with the caller carries only what the share granted, and one still awaiting registration cannot be shared at all.", "items": { "enum": [ "CAN_VIEW", "CAN_UPDATE", "CAN_DELETE", "CAN_SHARE" ], "type": "string" }, "maxItems": 4, "title": "Permissions", "type": "array" }, "rbacResourceId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Identifier of this memory space in the platform's access-control system. sharing is managed there, and this is the ID its API expects. null while the space has no such record yet, which is the same state that withholds `can_share` from `permissions`.", "title": "RBACResourceID" }, "tenantId": { "description": "Tenant identifier.", "format": "uuid", "title": "Tenantid", "type": "string" }, "userId": { "description": "ID of the user who owns this memory space.", "title": "Userid", "type": "string" } }, "required": [ "memorySpaceId", "userId", "tenantId", "createdAt", "permissions" ], "title": "MemorySpaceResponse", "type": "object" }, "maxItems": 100, "title": "Items", "type": "array" }, "limit": { "description": "Maximum number of returned items.", "exclusiveMinimum": 0, "title": "Limit", "type": "integer" }, "offset": { "description": "Number of skipped items.", "minimum": 0, "title": "Offset", "type": "integer" }, "total": { "description": "Total number of matching items.", "title": "Total", "type": "integer" } }, "required": [ "items", "offset", "limit", "total" ], "title": "PaginatedResponse[MemorySpaceResponse]", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | PaginatedResponse_MemorySpaceResponse_ | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Create memory space Operation path: `POST /new/` Create a new Memory Space tenant_id is bound from the authenticated user's context; clients cannot set or override it. ### Body parameter ``` { "description": "Request body for creating a memory space.", "properties": { "customInstructions": { "anyOf": [ { "maxLength": 10000, "type": "string" }, { "type": "null" } ], "description": "Custom prompt instructions used for fact extraction null leaves mem0 on its default extraction prompt.", "title": "Custominstructions" }, "deduplicationKey": { "anyOf": [ { "maxLength": 72, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Optional key, unique per user, for idempotent memory space creation by coordinating ha agents. a second create with the same key returns 409.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "maxLength": 1000, "type": "string" }, { "type": "null" } ], "description": "Optional description for the memory space.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "maxLength": 200, "type": "string" }, { "type": "null" } ], "description": "Name of the LLM model to use for memory extraction. non-reasoning models such as ``gpt-4o`` are recommended. reasoning-capable models are significantly slower for fact extraction without producing meaningfully better results.", "title": "Llmmodelname" } }, "title": "CreateMemorySpaceRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | CreateMemorySpaceRequest | false | none | ### Example responses > 201 Response ``` { "description": "Response schema for a single memory space.", "properties": { "createdAt": { "description": "Memory space creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "customInstructions": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Custom prompt instructions used for fact extraction. null means the default mem0 extraction prompt is used.", "title": "Custominstructions" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique per user, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Memory space description.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM model configured for memory extraction.", "title": "Llmmodelname" }, "memorySpaceId": { "description": "Unique memory space identifier.", "format": "uuid", "title": "Memoryspaceid", "type": "string" }, "permissions": { "description": "Actions the authenticated caller may perform on this memory space. the creator of a registered space holds all four; a space shared with the caller carries only what the share granted, and one still awaiting registration cannot be shared at all.", "items": { "enum": [ "CAN_VIEW", "CAN_UPDATE", "CAN_DELETE", "CAN_SHARE" ], "type": "string" }, "maxItems": 4, "title": "Permissions", "type": "array" }, "rbacResourceId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Identifier of this memory space in the platform's access-control system. sharing is managed there, and this is the ID its API expects. null while the space has no such record yet, which is the same state that withholds `can_share` from `permissions`.", "title": "RBACResourceID" }, "tenantId": { "description": "Tenant identifier.", "format": "uuid", "title": "Tenantid", "type": "string" }, "userId": { "description": "ID of the user who owns this memory space.", "title": "Userid", "type": "string" } }, "required": [ "memorySpaceId", "userId", "tenantId", "createdAt", "permissions" ], "title": "MemorySpaceResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 201 | Created | Successful Response | MemorySpaceResponse | | 409 | Conflict | A memory space with this deduplicationKey already exists for the user. Body carries existingMemorySpaceId / existingMemorySpaceUrl when the live winner can be resolved; the response also sets a Location header to its absolute URL. | MemorySpaceDeduplicationErrorResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Delete memory space by memory_space_ ID Operation path: `DELETE /{memory_space_id}/` Delete a Memory Space Returns 404 if already deleted or not accessible by the caller, and 403 if the space is shared with them under a role that cannot delete it. Sharing grants are revoked asynchronously by the RBAC reconciliation worker. The persisted RBAC resource pointer is its durable cleanup marker. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | ### Example responses > 422 Response ``` { "properties": { "detail": { "description": "List of validation errors.", "items": { "properties": { "ctx": { "description": "Additional context about the validation error.", "title": "Context", "type": "object" }, "input": { "additionalProperties": true, "description": "The input value that caused the validation error.", "title": "Input", "type": "object" }, "loc": { "description": "The location in the request where the validation error occurred.", "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "maxItems": 100, "title": "Location", "type": "array" }, "msg": { "description": "A human-readable description of the validation error.", "title": "Message", "type": "string" }, "type": { "description": "A machine-readable error type identifier.", "title": "Error Type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "maxItems": 100, "title": "Detail", "type": "array" } }, "title": "HTTPValidationError", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Successful Response | None | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | ## Get memory space by memory_space_ ID Operation path: `GET /{memory_space_id}/` Get a Memory Space by ID Covers spaces the caller created and spaces shared with them, carrying the permissions the caller holds on it. Returns 404 if the space does not exist, is soft-deleted, belongs to another tenant, or is not shared with the caller. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | ### Example responses > 200 Response ``` { "description": "Response schema for a single memory space.", "properties": { "createdAt": { "description": "Memory space creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "customInstructions": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Custom prompt instructions used for fact extraction. null means the default mem0 extraction prompt is used.", "title": "Custominstructions" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique per user, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Memory space description.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM model configured for memory extraction.", "title": "Llmmodelname" }, "memorySpaceId": { "description": "Unique memory space identifier.", "format": "uuid", "title": "Memoryspaceid", "type": "string" }, "permissions": { "description": "Actions the authenticated caller may perform on this memory space. the creator of a registered space holds all four; a space shared with the caller carries only what the share granted, and one still awaiting registration cannot be shared at all.", "items": { "enum": [ "CAN_VIEW", "CAN_UPDATE", "CAN_DELETE", "CAN_SHARE" ], "type": "string" }, "maxItems": 4, "title": "Permissions", "type": "array" }, "rbacResourceId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Identifier of this memory space in the platform's access-control system. sharing is managed there, and this is the ID its API expects. null while the space has no such record yet, which is the same state that withholds `can_share` from `permissions`.", "title": "RBACResourceID" }, "tenantId": { "description": "Tenant identifier.", "format": "uuid", "title": "Tenantid", "type": "string" }, "userId": { "description": "ID of the user who owns this memory space.", "title": "Userid", "type": "string" } }, "required": [ "memorySpaceId", "userId", "tenantId", "createdAt", "permissions" ], "title": "MemorySpaceResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | MemorySpaceResponse | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Update memory space by memory_space_ ID Operation path: `PATCH /{memory_space_id}/` Partially update a Memory Space. ### Body parameter ``` { "additionalProperties": false, "description": "Request body for partially updating a memory space.", "properties": { "customInstructions": { "anyOf": [ { "maxLength": 10000, "type": "string" }, { "type": "null" } ], "description": "Updated custom prompt instructions used for fact extraction. pass null to clear and revert to mem0's default extraction prompt.", "title": "Custominstructions" }, "description": { "anyOf": [ { "maxLength": 1000, "type": "string" }, { "type": "null" } ], "description": "Updated description for the memory space.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "maxLength": 200, "type": "string" }, { "type": "null" } ], "description": "Updated LLM model name. non-reasoning models such as ``gpt-4o`` are recommended. reasoning-capable models are significantly slower for fact extraction without producing meaningfully better results.", "title": "Llmmodelname" } }, "title": "UpdateMemorySpaceRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | body | body | UpdateMemorySpaceRequest | true | none | ### Example responses > 200 Response ``` { "description": "Response schema for a single memory space.", "properties": { "createdAt": { "description": "Memory space creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "customInstructions": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Custom prompt instructions used for fact extraction. null means the default mem0 extraction prompt is used.", "title": "Custominstructions" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique per user, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Memory space description.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM model configured for memory extraction.", "title": "Llmmodelname" }, "memorySpaceId": { "description": "Unique memory space identifier.", "format": "uuid", "title": "Memoryspaceid", "type": "string" }, "permissions": { "description": "Actions the authenticated caller may perform on this memory space. the creator of a registered space holds all four; a space shared with the caller carries only what the share granted, and one still awaiting registration cannot be shared at all.", "items": { "enum": [ "CAN_VIEW", "CAN_UPDATE", "CAN_DELETE", "CAN_SHARE" ], "type": "string" }, "maxItems": 4, "title": "Permissions", "type": "array" }, "rbacResourceId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Identifier of this memory space in the platform's access-control system. sharing is managed there, and this is the ID its API expects. null while the space has no such record yet, which is the same state that withholds `can_share` from `permissions`.", "title": "RBACResourceID" }, "tenantId": { "description": "Tenant identifier.", "format": "uuid", "title": "Tenantid", "type": "string" }, "userId": { "description": "ID of the user who owns this memory space.", "title": "Userid", "type": "string" } }, "required": [ "memorySpaceId", "userId", "tenantId", "createdAt", "permissions" ], "title": "MemorySpaceResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | MemorySpaceResponse | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## List sessions by memory_space_ ID Operation path: `GET /{memory_space_id}/sessions/` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | participants | query | any | false | Filter sessions by participant IDs (ObjectId format). | | description | query | any | false | Filter sessions by description (case-insensitive substring match). Must be at least 3 characters. | | deduplicationKey | query | any | false | Filter sessions by deduplication key (exact match). | ### Example responses > 200 Response ``` { "properties": { "items": { "description": "List of items for the current page.", "items": { "description": "Schema for session response.", "properties": { "createdAt": { "description": "Session creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique within the memory space, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Session description.", "title": "Description" }, "id": { "description": "Session id.", "format": "uuid", "title": "Id", "type": "string" }, "lifecycleStrategies": { "description": "Lifecycle strategies associated with this session.", "items": { "description": "Schema for lifecycle strategy in response. execution status is tracked in the strategy_executions table and can be queried separately if needed.", "properties": { "opts": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Strategy-specific configuration parameters.", "title": "Opts" }, "trigger": { "anyOf": [ { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ], "description": "Trigger condition.", "title": "Trigger" }, "type": { "description": "Strategy type.", "enum": [ "soft_delete", "extract_memories" ], "title": "Type", "type": "string" } }, "required": [ "type", "trigger" ], "title": "LifecycleStrategyResponse", "type": "object" }, "maxItems": 5, "title": "Lifecyclestrategies", "type": "array" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Custom metadata as key-value pairs for storing application-specific data.", "title": "Metadata" }, "participants": { "description": "List of participant ids.", "items": { "type": "string" }, "maxItems": 50, "title": "Participants", "type": "array" }, "version": { "default": 1, "description": "Monotonic version for optimistic concurrency. echo back as `if-match: \"\"` on patch to detect lost updates.", "minimum": 1, "title": "Version", "type": "integer" } }, "required": [ "id", "participants", "createdAt" ], "title": "SessionResponse", "type": "object" }, "maxItems": 100, "title": "Items", "type": "array" }, "limit": { "description": "Maximum number of returned items.", "exclusiveMinimum": 0, "title": "Limit", "type": "integer" }, "offset": { "description": "Number of skipped items.", "minimum": 0, "title": "Offset", "type": "integer" }, "total": { "description": "Total number of matching items.", "title": "Total", "type": "integer" } }, "required": [ "items", "offset", "limit", "total" ], "title": "PaginatedResponse[SessionResponse]", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | PaginatedResponse_SessionResponse_ | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Create session by memory_space_ ID Operation path: `POST /{memory_space_id}/sessions/` ### Body parameter ``` { "description": "Customer support conversation", "lifecycleStrategies": [ { "trigger": { "ttl": 604800 }, "type": "soft_delete" } ], "metadata": { "department": "sales", "priority": "high", "region": "us-east" }, "participants": [ "507f1f77bcf86cd799439011" ] } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | body | body | CreateSessionRequest | true | none | ### Example responses > 201 Response ``` { "description": "Schema for session response.", "properties": { "createdAt": { "description": "Session creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique within the memory space, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Session description.", "title": "Description" }, "id": { "description": "Session id.", "format": "uuid", "title": "Id", "type": "string" }, "lifecycleStrategies": { "description": "Lifecycle strategies associated with this session.", "items": { "description": "Schema for lifecycle strategy in response. execution status is tracked in the strategy_executions table and can be queried separately if needed.", "properties": { "opts": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Strategy-specific configuration parameters.", "title": "Opts" }, "trigger": { "anyOf": [ { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ], "description": "Trigger condition.", "title": "Trigger" }, "type": { "description": "Strategy type.", "enum": [ "soft_delete", "extract_memories" ], "title": "Type", "type": "string" } }, "required": [ "type", "trigger" ], "title": "LifecycleStrategyResponse", "type": "object" }, "maxItems": 5, "title": "Lifecyclestrategies", "type": "array" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Custom metadata as key-value pairs for storing application-specific data.", "title": "Metadata" }, "participants": { "description": "List of participant ids.", "items": { "type": "string" }, "maxItems": 50, "title": "Participants", "type": "array" }, "version": { "default": 1, "description": "Monotonic version for optimistic concurrency. echo back as `if-match: \"\"` on patch to detect lost updates.", "minimum": 1, "title": "Version", "type": "integer" } }, "required": [ "id", "participants", "createdAt" ], "title": "SessionResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 201 | Created | Successful Response | SessionResponse | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 409 | Conflict | A session with this deduplicationKey already exists in the memory space. Body carries existingSessionId / existingSessionUrl when the live winner can be resolved; the response also sets a Location header to its absolute URL. | SessionDeduplicationErrorResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Delete session by memory_space_ ID Operation path: `DELETE /{memory_space_id}/sessions/{session_id}/` Delete a session. Performs a soft-delete by setting the deleted_at timestamp. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | session_id | path | string(uuid) | true | Session ID | | memory_space_id | path | string | true | Memory Space ID | ### Example responses > 422 Response ``` { "properties": { "detail": { "description": "List of validation errors.", "items": { "properties": { "ctx": { "description": "Additional context about the validation error.", "title": "Context", "type": "object" }, "input": { "additionalProperties": true, "description": "The input value that caused the validation error.", "title": "Input", "type": "object" }, "loc": { "description": "The location in the request where the validation error occurred.", "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "maxItems": 100, "title": "Location", "type": "array" }, "msg": { "description": "A human-readable description of the validation error.", "title": "Message", "type": "string" }, "type": { "description": "A machine-readable error type identifier.", "title": "Error Type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "maxItems": 100, "title": "Detail", "type": "array" } }, "title": "HTTPValidationError", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Successful Response | None | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space or Session not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | ## Get session by memory_space_ ID Operation path: `GET /{memory_space_id}/sessions/{session_id}/` Get a single session by ID. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | session_id | path | string(uuid) | true | Session ID | | memory_space_id | path | string | true | Memory Space ID | ### Example responses > 200 Response ``` { "description": "Schema for session response.", "properties": { "createdAt": { "description": "Session creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique within the memory space, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Session description.", "title": "Description" }, "id": { "description": "Session id.", "format": "uuid", "title": "Id", "type": "string" }, "lifecycleStrategies": { "description": "Lifecycle strategies associated with this session.", "items": { "description": "Schema for lifecycle strategy in response. execution status is tracked in the strategy_executions table and can be queried separately if needed.", "properties": { "opts": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Strategy-specific configuration parameters.", "title": "Opts" }, "trigger": { "anyOf": [ { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ], "description": "Trigger condition.", "title": "Trigger" }, "type": { "description": "Strategy type.", "enum": [ "soft_delete", "extract_memories" ], "title": "Type", "type": "string" } }, "required": [ "type", "trigger" ], "title": "LifecycleStrategyResponse", "type": "object" }, "maxItems": 5, "title": "Lifecyclestrategies", "type": "array" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Custom metadata as key-value pairs for storing application-specific data.", "title": "Metadata" }, "participants": { "description": "List of participant ids.", "items": { "type": "string" }, "maxItems": 50, "title": "Participants", "type": "array" }, "version": { "default": 1, "description": "Monotonic version for optimistic concurrency. echo back as `if-match: \"\"` on patch to detect lost updates.", "minimum": 1, "title": "Version", "type": "integer" } }, "required": [ "id", "participants", "createdAt" ], "title": "SessionResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | SessionResponse | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space or Session not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Update session by memory_space_ ID Operation path: `PATCH /{memory_space_id}/sessions/{session_id}/` Patch a session with new description, metadata and/or lifecycle strategies. Only fields explicitly included in the request body are modified others remain untouched.`lifecycleStrategies`, when present, replaces the whole list. A strategy type that already executed for this session can never run again, so changing its configuration is rejected with 409; re-sending it unchanged is accepted and stays inert. ### Body parameter ``` { "description": "Schema for session partial patch-updates.", "properties": { "description": { "anyOf": [ { "maxLength": 1000, "type": "string" }, { "type": "null" } ], "description": "New session description.", "title": "Description" }, "lifecycleStrategies": { "anyOf": [ { "items": { "description": "Lifecycle strategy configuration that defines when and how to manage session lifecycle.", "example": { "trigger": { "ttl": 86400 }, "type": "soft_delete" }, "properties": { "opts": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Optional strategy-specific configuration parameters.", "title": "Opts" }, "trigger": { "anyOf": [ { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ], "description": "Trigger condition. available: 'ttl' (time in seconds), 'eventcount', 'tokencount', 'idle', 'never' (retain indefinitely; only accepted in environments where the never-expire trigger is enabled).", "title": "Trigger" }, "type": { "description": "Strategy type. available: 'soft_delete'.", "enum": [ "soft_delete", "extract_memories" ], "title": "Type", "type": "string" } }, "required": [ "type", "trigger" ], "title": "LifecycleStrategiesDataModel", "type": "object" }, "maxItems": 5, "minItems": 1, "type": "array" }, { "type": "null" } ], "description": "When present, replaces the session's lifecycle strategies entirely. at least one strategy is required so every session stays controlled by one; null is rejected -- omit the field to leave the strategies unchanged. a strategy type that already executed for this session can never run again, so changing its configuration is rejected with 409; re-sending it unchanged is inert.", "title": "Lifecyclestrategies" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Custom metadata as key-value pairs for storing application-specific data.", "title": "Metadata" } }, "title": "UpdateSessionRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | session_id | path | string(uuid) | true | Session ID | | memory_space_id | path | string | true | Memory Space ID | | If-Match | header | any | false | Optional optimistic-concurrency precondition. Supply the version from the prior response (quoted or bare integer, e.g. "3" or 3). Returns 409 if the stored version no longer matches. | | body | body | UpdateSessionRequest | true | none | ### Example responses > 200 Response ``` { "description": "Schema for session response.", "properties": { "createdAt": { "description": "Session creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique within the memory space, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Session description.", "title": "Description" }, "id": { "description": "Session id.", "format": "uuid", "title": "Id", "type": "string" }, "lifecycleStrategies": { "description": "Lifecycle strategies associated with this session.", "items": { "description": "Schema for lifecycle strategy in response. execution status is tracked in the strategy_executions table and can be queried separately if needed.", "properties": { "opts": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Strategy-specific configuration parameters.", "title": "Opts" }, "trigger": { "anyOf": [ { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ], "description": "Trigger condition.", "title": "Trigger" }, "type": { "description": "Strategy type.", "enum": [ "soft_delete", "extract_memories" ], "title": "Type", "type": "string" } }, "required": [ "type", "trigger" ], "title": "LifecycleStrategyResponse", "type": "object" }, "maxItems": 5, "title": "Lifecyclestrategies", "type": "array" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Custom metadata as key-value pairs for storing application-specific data.", "title": "Metadata" }, "participants": { "description": "List of participant ids.", "items": { "type": "string" }, "maxItems": 50, "title": "Participants", "type": "array" }, "version": { "default": 1, "description": "Monotonic version for optimistic concurrency. echo back as `if-match: \"\"` on patch to detect lost updates.", "minimum": 1, "title": "Version", "type": "integer" } }, "required": [ "id", "participants", "createdAt" ], "title": "SessionResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | SessionResponse | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space or Session not found | None | | 409 | Conflict | Session version does not match the If-Match precondition, or the request changes a lifecycle strategy type that has already been executed | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## List events by memory_space_ ID Operation path: `GET /{memory_space_id}/sessions/{session_id}/events/` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | session_id | path | string(uuid) | true | Session ID | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | lastN | query | any | false | Return the last N events in chronological order. Mutually exclusive with offset | | eventType | query | any | false | Filter session events by type. Omit to return events of all types. | ### Example responses > 200 Response ``` { "properties": { "items": { "description": "List of items for the current page.", "items": { "description": "Schema for event response.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Event body content.", "title": "Body" }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Event creation timestamp.", "title": "Createdat" }, "emitterId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier.", "title": "Emitterid" }, "emitterType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter type.", "title": "Emittertype" }, "eventType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Event type.", "title": "Eventtype" }, "sequenceId": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Event sequence number within the session.", "title": "Sequenceid" } }, "title": "EventResponse", "type": "object" }, "maxItems": 100, "title": "Items", "type": "array" }, "limit": { "description": "Maximum number of returned items.", "exclusiveMinimum": 0, "title": "Limit", "type": "integer" }, "offset": { "description": "Number of skipped items.", "minimum": 0, "title": "Offset", "type": "integer" }, "total": { "description": "Total number of matching items.", "title": "Total", "type": "integer" } }, "required": [ "items", "offset", "limit", "total" ], "title": "PaginatedResponse[EventResponse]", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | PaginatedResponse_EventResponse_ | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space or Session not found | None | | 422 | Unprocessable Entity | Validation error (e.g. lastN used together with offset) | None | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Create event by memory_space_ ID Operation path: `POST /{memory_space_id}/sessions/{session_id}/events/` ### Body parameter ``` { "description": "Schema for create event request.", "properties": { "body": { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" }, "emitter": { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" }, "type": { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" } }, "required": [ "body", "emitter" ], "title": "CreateEventRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | session_id | path | string(uuid) | true | Session ID | | body | body | CreateEventRequest | true | none | ### Example responses > 201 Response ``` { "description": "Schema for event response.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Event body content.", "title": "Body" }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Event creation timestamp.", "title": "Createdat" }, "emitterId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier.", "title": "Emitterid" }, "emitterType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter type.", "title": "Emittertype" }, "eventType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Event type.", "title": "Eventtype" }, "sequenceId": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Event sequence number within the session.", "title": "Sequenceid" } }, "title": "EventResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 201 | Created | Successful Response | EventResponse | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space or Session not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Update events batch by memory_space_ ID Operation path: `PATCH /{memory_space_id}/sessions/{session_id}/events/batch/` Update up to MAX_BATCH_SIZE events atomically; any failure rolls back the batch. ### Body parameter ``` { "description": "Schema for batch event update.", "properties": { "events": { "description": "Events to update.", "items": { "description": "A single item of a batch update: identifies the target event by sequence_id. inherits the mutable body/type/emitter fields from updateeventrequest. the inherited \"at least one field present\" validator is overridden below because the always-present sequence_id would otherwise satisfy it vacuously.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" }, { "type": "null" } ], "description": "Updated event body." }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Optional per-item version check: must match the target event's timestamp.", "title": "Createdat" }, "emitter": { "anyOf": [ { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" }, { "type": "null" } ], "description": "Updated emitter information." }, "sequenceId": { "description": "Target event sequence ID within the session.", "minimum": 0, "title": "Sequenceid", "type": "integer" }, "type": { "anyOf": [ { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" }, { "type": "null" } ], "description": "Updated event type." } }, "required": [ "sequenceId" ], "title": "BatchUpdateEventItem", "type": "object" }, "maxItems": 200, "minItems": 1, "title": "Events", "type": "array" } }, "required": [ "events" ], "title": "BatchUpdateEventRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | session_id | path | string(uuid) | true | Session ID | | body | body | BatchUpdateEventRequest | true | none | ### Example responses > 200 Response ``` { "description": "Envelope wrapping a list of event responses for batch endpoints. per the datarobot API design policy enforced by adipose, every JSON response must be an object. the single-event endpoints already satisfy this; the batch endpoints wrap their list in this envelope so the same rule holds.", "properties": { "items": { "description": "Events processed by the batch operation.", "items": { "description": "Schema for event response.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Event body content.", "title": "Body" }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Event creation timestamp.", "title": "Createdat" }, "emitterId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier.", "title": "Emitterid" }, "emitterType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter type.", "title": "Emittertype" }, "eventType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Event type.", "title": "Eventtype" }, "sequenceId": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Event sequence number within the session.", "title": "Sequenceid" } }, "title": "EventResponse", "type": "object" }, "maxItems": 200, "title": "Items", "type": "array" } }, "required": [ "items" ], "title": "BatchEventResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | BatchEventResponse | | 400 | Bad Request | An event emitter is not a session participant | None | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space, Session or one of the events was not found | None | | 422 | Unprocessable Entity | Tried to update an outdated version of an event | None | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Create events batch by memory_space_ ID Operation path: `POST /{memory_space_id}/sessions/{session_id}/events/batch/` Append up to MAX_BATCH_SIZE events to a session atomically. ### Body parameter ``` { "description": "Schema for batch event creation. events are appended in list order.", "properties": { "events": { "description": "Events to append, in order.", "items": { "description": "Schema for create event request.", "properties": { "body": { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" }, "emitter": { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" }, "type": { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" } }, "required": [ "body", "emitter" ], "title": "CreateEventRequest", "type": "object" }, "maxItems": 200, "minItems": 1, "title": "Events", "type": "array" } }, "required": [ "events" ], "title": "BatchCreateEventRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | session_id | path | string(uuid) | true | Session ID | | body | body | BatchCreateEventRequest | true | none | ### Example responses > 201 Response ``` { "description": "Envelope wrapping a list of event responses for batch endpoints. per the datarobot API design policy enforced by adipose, every JSON response must be an object. the single-event endpoints already satisfy this; the batch endpoints wrap their list in this envelope so the same rule holds.", "properties": { "items": { "description": "Events processed by the batch operation.", "items": { "description": "Schema for event response.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Event body content.", "title": "Body" }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Event creation timestamp.", "title": "Createdat" }, "emitterId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier.", "title": "Emitterid" }, "emitterType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter type.", "title": "Emittertype" }, "eventType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Event type.", "title": "Eventtype" }, "sequenceId": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Event sequence number within the session.", "title": "Sequenceid" } }, "title": "EventResponse", "type": "object" }, "maxItems": 200, "title": "Items", "type": "array" } }, "required": [ "items" ], "title": "BatchEventResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 201 | Created | Successful Response | BatchEventResponse | | 400 | Bad Request | An event emitter is not a session participant | None | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space or Session not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Delete event by memory_space_ ID Operation path: `DELETE /{memory_space_id}/sessions/{session_id}/events/{sequence_id}/` Soft-delete one session event by sequence id. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | sequence_id | path | integer | true | Event sequence ID | | memory_space_id | path | string | true | Memory Space ID | | session_id | path | string(uuid) | true | Session ID | ### Example responses > 422 Response ``` { "properties": { "detail": { "description": "List of validation errors.", "items": { "properties": { "ctx": { "description": "Additional context about the validation error.", "title": "Context", "type": "object" }, "input": { "additionalProperties": true, "description": "The input value that caused the validation error.", "title": "Input", "type": "object" }, "loc": { "description": "The location in the request where the validation error occurred.", "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "maxItems": 100, "title": "Location", "type": "array" }, "msg": { "description": "A human-readable description of the validation error.", "title": "Message", "type": "string" }, "type": { "description": "A machine-readable error type identifier.", "title": "Error Type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "maxItems": 100, "title": "Detail", "type": "array" } }, "title": "HTTPValidationError", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Successful Response | None | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space, Session or event not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | ## Update event by memory_space_ ID Operation path: `PATCH /{memory_space_id}/sessions/{session_id}/events/{sequence_id}/` ### Body parameter ``` { "description": "Schema for create event request.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" }, { "type": "null" } ], "description": "Updated event body." }, "emitter": { "anyOf": [ { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" }, { "type": "null" } ], "description": "Updated emitter information." }, "type": { "anyOf": [ { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" }, { "type": "null" } ], "description": "Updated event type." } }, "title": "UpdateEventRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | sequence_id | path | integer | true | Event sequence ID | | memory_space_id | path | string | true | Memory Space ID | | session_id | path | string(uuid) | true | Session ID | | createdAt | query | any | false | Verification of target version of event | | body | body | UpdateEventRequest | true | none | ### Example responses > 200 Response ``` { "description": "Schema for event response.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Event body content.", "title": "Body" }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Event creation timestamp.", "title": "Createdat" }, "emitterId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier.", "title": "Emitterid" }, "emitterType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter type.", "title": "Emittertype" }, "eventType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Event type.", "title": "Eventtype" }, "sequenceId": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Event sequence number within the session.", "title": "Sequenceid" } }, "title": "EventResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | EventResponse | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space, Session or event are not found | None | | 422 | Unprocessable Entity | Try to update outdated version of event | None | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Delete all memories by memory_space_ ID Operation path: `DELETE /{memory_space_id}/v1/memories/` Delete all memories matching the given filter. At least one filter param is required to prevent accidental deletion of all memories across all identities. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | user_id | query | any | false | none | | agent_id | query | any | false | none | | run_id | query | any | false | none | | app_id | query | any | false | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | ### Response Schema ## Get all memories v1 by memory_space_ ID Operation path: `GET /{memory_space_id}/v1/memories/` Retrieve all memories matching the given identity filters (v1 query-param interface). ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | user_id | query | any | false | none | | agent_id | query | any | false | none | | run_id | query | any | false | none | | app_id | query | any | false | none | | top_k | query | any | false | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Post memories by memory_space_ ID Operation path: `POST /{memory_space_id}/v1/memories/` Extract facts from a conversation and persist them as memories. ### Body parameter ``` { "description": "Request body for post /v1/memories — extracts facts from a conversation.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "expirationDate": { "anyOf": [ { "maxLength": 40, "type": "string" }, { "type": "null" } ], "title": "Expirationdate" }, "infer": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "title": "Infer" }, "messages": { "items": { "description": "A single conversation message passed to the post /v1/memory endpoint.", "properties": { "content": { "maxLength": 50000, "minLength": 1, "title": "Content", "type": "string" }, "role": { "enum": [ "user", "assistant", "system" ], "title": "Role", "type": "string" } }, "required": [ "role", "content" ], "title": "Message", "type": "object" }, "maxItems": 100, "minItems": 1, "title": "Messages", "type": "array" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Metadata" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "userId": { "maxLength": 64, "minLength": 1, "title": "Userid", "type": "string" } }, "required": [ "userId", "messages" ], "title": "MemoryCreate", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | body | body | MemoryCreate | true | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 409 | Conflict | Immutable memory with the same content already exists. | None | | 422 | Unprocessable Entity | Validation error (request body) or domain conflict (content mismatch). | Inline | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | | 503 | Service Unavailable | Transient condition (insert conflict or upstream LLM rate limit); retry advised. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | | 503 | Retry-After | integer | | none | ## Search memories v1 by memory_space_ ID Operation path: `POST /{memory_space_id}/v1/memories/search/` Search memories by semantic similarity to a query string. ### Body parameter ``` { "description": "Request body for post /v2/memories/search/ extra=\"ignore\" is set because the mem0 client injects org_id/project_id fields that are not relevant for datarobot.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "filters": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Filters" }, "query": { "maxLength": 2000, "minLength": 1, "title": "Query", "type": "string" }, "rerank": { "default": true, "title": "Rerank", "type": "boolean" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "threshold": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "title": "Threshold" }, "topK": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Topk" }, "userId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Userid" } }, "required": [ "query" ], "title": "MemorySearch", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | body | body | MemorySearch | true | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Delete memory by memory_space_ ID Operation path: `DELETE /{memory_space_id}/v1/memories/{memory_id}/` Delete a single memory by its ID. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_id | path | string(uuid) | true | none | | memory_space_id | path | string | true | Memory Space ID | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | ### Response Schema ## Get memory by memory_space_ ID Operation path: `GET /{memory_space_id}/v1/memories/{memory_id}/` Retrieve a single memory by its ID. Returns null if the memory does not exist. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_id | path | string(uuid) | true | none | | memory_space_id | path | string | true | Memory Space ID | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Put memory by memory_space_ ID Operation path: `PUT /{memory_space_id}/v1/memories/{memory_id}/` Replace the text of an existing memory. Returns 404 if the memory does not exist. Returns 409 if the memory is immutable. ### Body parameter ``` { "description": "Request body for put /v1/memories/{memory_id}. metadata and timestamp are accepted for mem0 client compatibility but not forwarded.", "properties": { "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Metadata" }, "text": { "maxLength": 10000, "minLength": 1, "title": "Text", "type": "string" }, "timestamp": { "anyOf": [ { "maxLength": 40, "type": "string" }, { "type": "null" } ], "title": "Timestamp" } }, "required": [ "text" ], "title": "MemoryUpdate", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_id | path | string(uuid) | true | none | | memory_space_id | path | string | true | Memory Space ID | | body | body | MemoryUpdate | true | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory not found. | None | | 409 | Conflict | Memory is immutable and cannot be updated. | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Get memory history by memory_space_ ID Operation path: `GET /{memory_space_id}/v1/memories/{memory_id}/history/` Retrieve the mutation history (audit trail) for a single memory. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_id | path | string(uuid) | true | none | | memory_space_id | path | string | true | Memory Space ID | | limit | query | integer | false | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## ping by memory_space_ ID Operation path: `GET /{memory_space_id}/v1/ping/` Health check endpoint returning fake user context for mem0.MemoryClient compatibility. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | ### Response Schema ## Reset memories by memory_space_ ID Operation path: `POST /{memory_space_id}/v1/reset/` Delete all memories in the current memory space. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | ### Response Schema ## Get all memories v2 by memory_space_ ID Operation path: `POST /{memory_space_id}/v2/memories/` Retrieve all memories matching the given filters. At least one of user_id, agent_id, or run_id is required (as enforced by mem0's AsyncMemory). ### Body parameter ``` { "description": "Request body for post /v2/memories extra=\"ignore\" is set because the mem0 client injects org_id/project_id fields that are not relevant for datarobot.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "filters": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Filters" }, "pageSize": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Pagesize" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "topK": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Topk" }, "userId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Userid" } }, "title": "MemoryGetAll", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | page_size | query | any | false | none | | body | body | MemoryGetAll | true | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 400 | Bad Request | At least one of user_id, agent_id, or run_id must be provided. | None | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation error (request body) or domain conflict (content mismatch). | Inline | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Search memories v2 by memory_space_ ID Operation path: `POST /{memory_space_id}/v2/memories/search/` Search memories by semantic similarity to a query string. At least one of user_id, agent_id, or run_id is required to scope the search. ### Body parameter ``` { "description": "Request body for post /v2/memories/search/ extra=\"ignore\" is set because the mem0 client injects org_id/project_id fields that are not relevant for datarobot.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "filters": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Filters" }, "query": { "maxLength": 2000, "minLength": 1, "title": "Query", "type": "string" }, "rerank": { "default": true, "title": "Rerank", "type": "boolean" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "threshold": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "title": "Threshold" }, "topK": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Topk" }, "userId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Userid" } }, "required": [ "query" ], "title": "MemorySearch", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | body | body | MemorySearch | true | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 400 | Bad Request | At least one of user_id, agent_id, or run_id must be provided. | None | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation error (request body) or domain conflict (content mismatch). | Inline | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Create memories by ID by memory_space_ ID Operation path: `POST /{memory_space_id}/v3/memories/` Retrieve all memories matching the given filters. At least one of user_id, agent_id, or run_id is required (as enforced by mem0's AsyncMemory). ### Body parameter ``` { "description": "Request body for post /v2/memories extra=\"ignore\" is set because the mem0 client injects org_id/project_id fields that are not relevant for datarobot.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "filters": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Filters" }, "pageSize": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Pagesize" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "topK": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Topk" }, "userId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Userid" } }, "title": "MemoryGetAll", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | page_size | query | any | false | none | | body | body | MemoryGetAll | true | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Create add by ID by memory_space_ ID Operation path: `POST /{memory_space_id}/v3/memories/add/` Extract facts from a conversation and persist them as memories. ### Body parameter ``` { "description": "Request body for post /v1/memories — extracts facts from a conversation.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "expirationDate": { "anyOf": [ { "maxLength": 40, "type": "string" }, { "type": "null" } ], "title": "Expirationdate" }, "infer": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "title": "Infer" }, "messages": { "items": { "description": "A single conversation message passed to the post /v1/memory endpoint.", "properties": { "content": { "maxLength": 50000, "minLength": 1, "title": "Content", "type": "string" }, "role": { "enum": [ "user", "assistant", "system" ], "title": "Role", "type": "string" } }, "required": [ "role", "content" ], "title": "Message", "type": "object" }, "maxItems": 100, "minItems": 1, "title": "Messages", "type": "array" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Metadata" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "userId": { "maxLength": 64, "minLength": 1, "title": "Userid", "type": "string" } }, "required": [ "userId", "messages" ], "title": "MemoryCreate", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | body | body | MemoryCreate | true | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | ## Create search by ID by memory_space_ ID Operation path: `POST /{memory_space_id}/v3/memories/search/` Search memories by semantic similarity to a query string. At least one of user_id, agent_id, or run_id is required to scope the search. ### Body parameter ``` { "description": "Request body for post /v2/memories/search/ extra=\"ignore\" is set because the mem0 client injects org_id/project_id fields that are not relevant for datarobot.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "filters": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Filters" }, "query": { "maxLength": 2000, "minLength": 1, "title": "Query", "type": "string" }, "rerank": { "default": true, "title": "Rerank", "type": "boolean" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "threshold": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "title": "Threshold" }, "topK": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Topk" }, "userId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Userid" } }, "required": [ "query" ], "title": "MemorySearch", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | memory_space_id | path | string | true | Memory Space ID | | body | body | MemorySearch | true | none | ### Example responses > 200 Response ``` { "additionalProperties": true, "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 403 | Forbidden | Insufficient permissions on this Memory Space | None | | 404 | Not Found | Memory Space not found | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationError | | 429 | Too Many Requests | Trial usage limit reached: the tenant's monthly read/write request quota or its total storage cap is exhausted. Carries X-RateLimit-* headers; quota (not storage) responses also carry Retry-After. | None | ### Response Schema ### Response Headers | Status | Header | Type | Format | Description | | --- | --- | --- | --- | --- | | 429 | Retry-After | integer | | Seconds until the monthly quota resets. Quota responses only. | | 429 | X-RateLimit-Limit | integer | | The applicable limit (monthly request quota, or storage cap). | | 429 | X-RateLimit-Remaining | integer | | Units remaining before the limit is reached. | | 429 | X-RateLimit-Reset | integer | | Epoch seconds at which the quota resets. Quota responses only. | # Schemas ## BatchCreateEventRequest ``` { "description": "Schema for batch event creation. events are appended in list order.", "properties": { "events": { "description": "Events to append, in order.", "items": { "description": "Schema for create event request.", "properties": { "body": { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" }, "emitter": { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" }, "type": { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" } }, "required": [ "body", "emitter" ], "title": "CreateEventRequest", "type": "object" }, "maxItems": 200, "minItems": 1, "title": "Events", "type": "array" } }, "required": [ "events" ], "title": "BatchCreateEventRequest", "type": "object" } ``` BatchCreateEventRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | events | [CreateEventRequest] | true | maxItems: 200minItems: 1 | Events to append, in order. | ## BatchEventResponse ``` { "description": "Envelope wrapping a list of event responses for batch endpoints. per the datarobot API design policy enforced by adipose, every JSON response must be an object. the single-event endpoints already satisfy this; the batch endpoints wrap their list in this envelope so the same rule holds.", "properties": { "items": { "description": "Events processed by the batch operation.", "items": { "description": "Schema for event response.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Event body content.", "title": "Body" }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Event creation timestamp.", "title": "Createdat" }, "emitterId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier.", "title": "Emitterid" }, "emitterType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter type.", "title": "Emittertype" }, "eventType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Event type.", "title": "Eventtype" }, "sequenceId": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Event sequence number within the session.", "title": "Sequenceid" } }, "title": "EventResponse", "type": "object" }, "maxItems": 200, "title": "Items", "type": "array" } }, "required": [ "items" ], "title": "BatchEventResponse", "type": "object" } ``` BatchEventResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | items | [EventResponse] | true | maxItems: 200 | Events processed by the batch operation. | ## BatchUpdateEventItem ``` { "description": "A single item of a batch update: identifies the target event by sequence_id. inherits the mutable body/type/emitter fields from updateeventrequest. the inherited \"at least one field present\" validator is overridden below because the always-present sequence_id would otherwise satisfy it vacuously.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" }, { "type": "null" } ], "description": "Updated event body." }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Optional per-item version check: must match the target event's timestamp.", "title": "Createdat" }, "emitter": { "anyOf": [ { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" }, { "type": "null" } ], "description": "Updated emitter information." }, "sequenceId": { "description": "Target event sequence ID within the session.", "minimum": 0, "title": "Sequenceid", "type": "integer" }, "type": { "anyOf": [ { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" }, { "type": "null" } ], "description": "Updated event type." } }, "required": [ "sequenceId" ], "title": "BatchUpdateEventItem", "type": "object" } ``` BatchUpdateEventItem ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | body | any | false | | Updated event body. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | EventBody | false | | Body for all event types. content field is mandatory, any additional fields are allowed. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | createdAt | any | false | | Optional per-item version check: must match the target event's timestamp. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string(date-time) | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | emitter | any | false | | Updated emitter information. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | EmitterDataModel | false | | Schema for emitter data. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | sequenceId | integer | true | minimum: 0 | Target event sequence ID within the session. | | type | any | false | | Updated event type. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | EventType | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## BatchUpdateEventRequest ``` { "description": "Schema for batch event update.", "properties": { "events": { "description": "Events to update.", "items": { "description": "A single item of a batch update: identifies the target event by sequence_id. inherits the mutable body/type/emitter fields from updateeventrequest. the inherited \"at least one field present\" validator is overridden below because the always-present sequence_id would otherwise satisfy it vacuously.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" }, { "type": "null" } ], "description": "Updated event body." }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Optional per-item version check: must match the target event's timestamp.", "title": "Createdat" }, "emitter": { "anyOf": [ { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" }, { "type": "null" } ], "description": "Updated emitter information." }, "sequenceId": { "description": "Target event sequence ID within the session.", "minimum": 0, "title": "Sequenceid", "type": "integer" }, "type": { "anyOf": [ { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" }, { "type": "null" } ], "description": "Updated event type." } }, "required": [ "sequenceId" ], "title": "BatchUpdateEventItem", "type": "object" }, "maxItems": 200, "minItems": 1, "title": "Events", "type": "array" } }, "required": [ "events" ], "title": "BatchUpdateEventRequest", "type": "object" } ``` BatchUpdateEventRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | events | [BatchUpdateEventItem] | true | maxItems: 200minItems: 1 | Events to update. | ## CreateEventRequest ``` { "description": "Schema for create event request.", "properties": { "body": { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" }, "emitter": { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" }, "type": { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" } }, "required": [ "body", "emitter" ], "title": "CreateEventRequest", "type": "object" } ``` CreateEventRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | body | EventBody | true | | Event body. | | emitter | EmitterDataModel | true | | Emitter information. | | type | EventType | false | | Event type. | ## CreateMemorySpaceRequest ``` { "description": "Request body for creating a memory space.", "properties": { "customInstructions": { "anyOf": [ { "maxLength": 10000, "type": "string" }, { "type": "null" } ], "description": "Custom prompt instructions used for fact extraction null leaves mem0 on its default extraction prompt.", "title": "Custominstructions" }, "deduplicationKey": { "anyOf": [ { "maxLength": 72, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Optional key, unique per user, for idempotent memory space creation by coordinating ha agents. a second create with the same key returns 409.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "maxLength": 1000, "type": "string" }, { "type": "null" } ], "description": "Optional description for the memory space.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "maxLength": 200, "type": "string" }, { "type": "null" } ], "description": "Name of the LLM model to use for memory extraction. non-reasoning models such as ``gpt-4o`` are recommended. reasoning-capable models are significantly slower for fact extraction without producing meaningfully better results.", "title": "Llmmodelname" } }, "title": "CreateMemorySpaceRequest", "type": "object" } ``` CreateMemorySpaceRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | customInstructions | any | false | | Custom prompt instructions used for fact extraction null leaves mem0 on its default extraction prompt. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 10000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | deduplicationKey | any | false | | Optional key, unique per user, for idempotent memory space creation by coordinating ha agents. a second create with the same key returns 409. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 72minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | any | false | | Optional description for the memory space. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 1000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | llmBaseUrl | any | false | | Chat API url to use for memory extraction. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string(uri) | false | maxLength: 2083minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | llmModelName | any | false | | Name of the LLM model to use for memory extraction. non-reasoning models such as gpt-4o are recommended. reasoning-capable models are significantly slower for fact extraction without producing meaningfully better results. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 200 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## CreateSessionRequest ``` { "description": "Customer support conversation", "lifecycleStrategies": [ { "trigger": { "ttl": 604800 }, "type": "soft_delete" } ], "metadata": { "department": "sales", "priority": "high", "region": "us-east" }, "participants": [ "507f1f77bcf86cd799439011" ] } ``` CreateSessionRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | deduplicationKey | any | false | | Optional key, unique within the memory space, for idempotent session creation by coordinating ha agents. a second create with the same key returns 409. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 72minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | any | false | | Optional session description. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 1000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | lifecycleStrategies | [LifecycleStrategiesDataModel] | true | maxItems: 5 | Lifecycle strategies that automatically manage session lifecycle. | | metadata | any | false | | Custom metadata as key-value pairs for storing application-specific data. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | participants | [string] | true | maxItems: 50minItems: 1 | List of participant ids (objectid format). | ## EmitterDataModel ``` { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" } ``` EmitterDataModel ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | id | any | false | | Emitter identifier (objectid format). | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | type | string | true | | Emitter type. one of: 'user', 'agent'. | ### Enumerated Values | Property | Value | | --- | --- | | type | [user, agent] | ## EventBody ``` { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" } ``` EventBody ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | content | string | true | maxLength: 100000minLength: 1minLength: 1 | Event body content text. | ## EventCountTrigger ``` { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" } ``` EventCountTrigger ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | eventCount | integer | true | maximum: 1000000 | Event count threshold. | ## EventResponse ``` { "description": "Schema for event response.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Event body content.", "title": "Body" }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Event creation timestamp.", "title": "Createdat" }, "emitterId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier.", "title": "Emitterid" }, "emitterType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter type.", "title": "Emittertype" }, "eventType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Event type.", "title": "Eventtype" }, "sequenceId": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Event sequence number within the session.", "title": "Sequenceid" } }, "title": "EventResponse", "type": "object" } ``` EventResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | body | any | false | | Event body content. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | createdAt | any | false | | Event creation timestamp. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string(date-time) | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | emitterId | any | false | | Emitter identifier. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | emitterType | any | false | | Emitter type. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | eventType | any | false | | Event type. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | sequenceId | any | false | | Event sequence number within the session. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## EventType ``` { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" } ``` EventType ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | EventType | string | false | | none | ### Enumerated Values | Property | Value | | --- | --- | | EventType | [message, tool_output, status] | ## HTTPValidationError ``` { "properties": { "detail": { "description": "List of validation errors.", "items": { "properties": { "ctx": { "description": "Additional context about the validation error.", "title": "Context", "type": "object" }, "input": { "additionalProperties": true, "description": "The input value that caused the validation error.", "title": "Input", "type": "object" }, "loc": { "description": "The location in the request where the validation error occurred.", "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "maxItems": 100, "title": "Location", "type": "array" }, "msg": { "description": "A human-readable description of the validation error.", "title": "Message", "type": "string" }, "type": { "description": "A machine-readable error type identifier.", "title": "Error Type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "maxItems": 100, "title": "Detail", "type": "array" } }, "title": "HTTPValidationError", "type": "object" } ``` HTTPValidationError ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | detail | [ValidationError] | false | maxItems: 100 | List of validation errors. | ## IdleTimeoutTrigger ``` { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" } ``` IdleTimeoutTrigger ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | idle | integer | true | maximum: 315360000 | Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds. | ## LifecycleStrategiesDataModel ``` { "trigger": { "ttl": 86400 }, "type": "soft_delete" } ``` LifecycleStrategiesDataModel ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | opts | any | false | | Optional strategy-specific configuration parameters. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | trigger | any | true | | Trigger condition. available: 'ttl' (time in seconds), 'eventcount', 'tokencount', 'idle', 'never' (retain indefinitely; only accepted in environments where the never-expire trigger is enabled). | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | TTLTrigger | false | | Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | EventCountTrigger | false | | Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | TokenCountTrigger | false | | Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | IdleTimeoutTrigger | false | | Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | NeverTrigger | false | | Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap. | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | type | string | true | | Strategy type. available: 'soft_delete'. | ### Enumerated Values | Property | Value | | --- | --- | | type | [soft_delete, extract_memories] | ## LifecycleStrategyResponse ``` { "description": "Schema for lifecycle strategy in response. execution status is tracked in the strategy_executions table and can be queried separately if needed.", "properties": { "opts": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Strategy-specific configuration parameters.", "title": "Opts" }, "trigger": { "anyOf": [ { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ], "description": "Trigger condition.", "title": "Trigger" }, "type": { "description": "Strategy type.", "enum": [ "soft_delete", "extract_memories" ], "title": "Type", "type": "string" } }, "required": [ "type", "trigger" ], "title": "LifecycleStrategyResponse", "type": "object" } ``` LifecycleStrategyResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | opts | any | false | | Strategy-specific configuration parameters. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | trigger | any | true | | Trigger condition. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | TTLTrigger | false | | Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | EventCountTrigger | false | | Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | TokenCountTrigger | false | | Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | IdleTimeoutTrigger | false | | Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | NeverTrigger | false | | Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap. | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | type | string | true | | Strategy type. | ### Enumerated Values | Property | Value | | --- | --- | | type | [soft_delete, extract_memories] | ## MemoryCreate ``` { "description": "Request body for post /v1/memories — extracts facts from a conversation.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "expirationDate": { "anyOf": [ { "maxLength": 40, "type": "string" }, { "type": "null" } ], "title": "Expirationdate" }, "infer": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "title": "Infer" }, "messages": { "items": { "description": "A single conversation message passed to the post /v1/memory endpoint.", "properties": { "content": { "maxLength": 50000, "minLength": 1, "title": "Content", "type": "string" }, "role": { "enum": [ "user", "assistant", "system" ], "title": "Role", "type": "string" } }, "required": [ "role", "content" ], "title": "Message", "type": "object" }, "maxItems": 100, "minItems": 1, "title": "Messages", "type": "array" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Metadata" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "userId": { "maxLength": 64, "minLength": 1, "title": "Userid", "type": "string" } }, "required": [ "userId", "messages" ], "title": "MemoryCreate", "type": "object" } ``` MemoryCreate ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | agentId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | appId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | expirationDate | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 40 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | infer | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | boolean | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | messages | [Message] | true | maxItems: 100minItems: 1 | [A single conversation message passed to the post /v1/memory endpoint.] | | metadata | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | runId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | userId | string | true | maxLength: 64minLength: 1minLength: 1 | none | ## MemoryGetAll ``` { "description": "Request body for post /v2/memories extra=\"ignore\" is set because the mem0 client injects org_id/project_id fields that are not relevant for datarobot.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "filters": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Filters" }, "pageSize": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Pagesize" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "topK": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Topk" }, "userId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Userid" } }, "title": "MemoryGetAll", "type": "object" } ``` MemoryGetAll ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | agentId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | appId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | filters | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | pageSize | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | maximum: 50minimum: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | runId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | topK | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | maximum: 50minimum: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | userId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## MemorySearch ``` { "description": "Request body for post /v2/memories/search/ extra=\"ignore\" is set because the mem0 client injects org_id/project_id fields that are not relevant for datarobot.", "properties": { "agentId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Agentid" }, "appId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Appid" }, "filters": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Filters" }, "query": { "maxLength": 2000, "minLength": 1, "title": "Query", "type": "string" }, "rerank": { "default": true, "title": "Rerank", "type": "boolean" }, "runId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Runid" }, "threshold": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "title": "Threshold" }, "topK": { "anyOf": [ { "maximum": 50, "minimum": 1, "type": "integer" }, { "type": "null" } ], "title": "Topk" }, "userId": { "anyOf": [ { "maxLength": 64, "minLength": 1, "type": "string" }, { "type": "null" } ], "title": "Userid" } }, "required": [ "query" ], "title": "MemorySearch", "type": "object" } ``` MemorySearch ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | agentId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | appId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | filters | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | query | string | true | maxLength: 2000minLength: 1minLength: 1 | none | | rerank | boolean | false | | none | | runId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | threshold | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | number | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | topK | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | maximum: 50minimum: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | userId | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 64minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## MemorySpaceDeduplicationErrorResponse ``` { "description": "Body returned with HTTP 409 when a memory space create collides on `deduplicationkey`. carries enough context for an ha-replica caller to adopt the existing winner without a second round-trip: the live space's ID and a path to it.", "properties": { "deduplicationKey": { "description": "The deduplication key that triggered the conflict.", "title": "Deduplicationkey", "type": "string" }, "detail": { "description": "Human-readable explanation of the conflict.", "title": "Detail", "type": "string" }, "errorName": { "const": "MemorySpaceDeduplicationConflict", "default": "MemorySpaceDeduplicationConflict", "description": "Stable machine-readable error name. clients should switch on this, not detail.", "title": "Errorname", "type": "string" }, "existingMemorySpaceId": { "anyOf": [ { "format": "uuid", "type": "string" }, { "type": "null" } ], "description": "ID of the live memory space already holding the key, or null if it could not be resolved (e.g. concurrently soft-deleted).", "title": "Existingmemoryspaceid" }, "existingMemorySpaceUrl": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Path to get the existing memory space. null when `existingmemoryspaceid` is null. mirrors the `location` response header (which is absolute).", "title": "Existingmemoryspaceurl" } }, "required": [ "detail", "deduplicationKey" ], "title": "MemorySpaceDeduplicationErrorResponse", "type": "object" } ``` MemorySpaceDeduplicationErrorResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | deduplicationKey | string | true | | The deduplication key that triggered the conflict. | | detail | string | true | | Human-readable explanation of the conflict. | | errorName | string | false | | Stable machine-readable error name. clients should switch on this, not detail. | | existingMemorySpaceId | any | false | | ID of the live memory space already holding the key, or null if it could not be resolved (e.g. concurrently soft-deleted). | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string(uuid) | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | existingMemorySpaceUrl | any | false | | Path to get the existing memory space. null when existingmemoryspaceid is null. mirrors the location response header (which is absolute). | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## MemorySpaceOwnership ``` { "description": "Which half of the accessible set a listing asks for. a caller reaches a space either by creating it or by being given a share, and the two differ in what they may do with it, so the listing can return one without the other. there is no member for both: that is the absence of the filter, so it stays none rather than becoming a third value.", "enum": [ "owned", "shared" ], "title": "MemorySpaceOwnership", "type": "string" } ``` MemorySpaceOwnership ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | MemorySpaceOwnership | string | false | | Which half of the accessible set a listing asks for. a caller reaches a space either by creating it or by being given a share, and the two differ in what they may do with it, so the listing can return one without the other. there is no member for both: that is the absence of the filter, so it stays none rather than becoming a third value. | ### Enumerated Values | Property | Value | | --- | --- | | MemorySpaceOwnership | [owned, shared] | ## MemorySpaceResponse ``` { "description": "Response schema for a single memory space.", "properties": { "createdAt": { "description": "Memory space creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "customInstructions": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Custom prompt instructions used for fact extraction. null means the default mem0 extraction prompt is used.", "title": "Custominstructions" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique per user, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Memory space description.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM model configured for memory extraction.", "title": "Llmmodelname" }, "memorySpaceId": { "description": "Unique memory space identifier.", "format": "uuid", "title": "Memoryspaceid", "type": "string" }, "permissions": { "description": "Actions the authenticated caller may perform on this memory space. the creator of a registered space holds all four; a space shared with the caller carries only what the share granted, and one still awaiting registration cannot be shared at all.", "items": { "enum": [ "CAN_VIEW", "CAN_UPDATE", "CAN_DELETE", "CAN_SHARE" ], "type": "string" }, "maxItems": 4, "title": "Permissions", "type": "array" }, "rbacResourceId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Identifier of this memory space in the platform's access-control system. sharing is managed there, and this is the ID its API expects. null while the space has no such record yet, which is the same state that withholds `can_share` from `permissions`.", "title": "RBACResourceID" }, "tenantId": { "description": "Tenant identifier.", "format": "uuid", "title": "Tenantid", "type": "string" }, "userId": { "description": "ID of the user who owns this memory space.", "title": "Userid", "type": "string" } }, "required": [ "memorySpaceId", "userId", "tenantId", "createdAt", "permissions" ], "title": "MemorySpaceResponse", "type": "object" } ``` MemorySpaceResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | createdAt | string(date-time) | true | | Memory space creation timestamp. | | customInstructions | any | false | | Custom prompt instructions used for fact extraction. null means the default mem0 extraction prompt is used. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | deduplicationKey | any | false | | Deduplication key, unique per user, if one was set. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | any | false | | Memory space description. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | llmBaseUrl | any | false | | Chat API url to use for memory extraction. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string(uri) | false | maxLength: 2083minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | llmModelName | any | false | | LLM model configured for memory extraction. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | memorySpaceId | string(uuid) | true | | Unique memory space identifier. | | permissions | [string] | true | maxItems: 4 | Actions the authenticated caller may perform on this memory space. the creator of a registered space holds all four; a space shared with the caller carries only what the share granted, and one still awaiting registration cannot be shared at all. | | rbacResourceId | any | false | | Identifier of this memory space in the platform's access-control system. sharing is managed there, and this is the ID its API expects. null while the space has no such record yet, which is the same state that withholds can_share from permissions. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | tenantId | string(uuid) | true | | Tenant identifier. | | userId | string | true | | ID of the user who owns this memory space. | ## MemoryUpdate ``` { "description": "Request body for put /v1/memories/{memory_id}. metadata and timestamp are accepted for mem0 client compatibility but not forwarded.", "properties": { "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "title": "Metadata" }, "text": { "maxLength": 10000, "minLength": 1, "title": "Text", "type": "string" }, "timestamp": { "anyOf": [ { "maxLength": 40, "type": "string" }, { "type": "null" } ], "title": "Timestamp" } }, "required": [ "text" ], "title": "MemoryUpdate", "type": "object" } ``` MemoryUpdate ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | metadata | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | text | string | true | maxLength: 10000minLength: 1minLength: 1 | none | | timestamp | any | false | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 40 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## Message ``` { "description": "A single conversation message passed to the post /v1/memory endpoint.", "properties": { "content": { "maxLength": 50000, "minLength": 1, "title": "Content", "type": "string" }, "role": { "enum": [ "user", "assistant", "system" ], "title": "Role", "type": "string" } }, "required": [ "role", "content" ], "title": "Message", "type": "object" } ``` Message ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | content | string | true | maxLength: 50000minLength: 1minLength: 1 | none | | role | string | true | | none | ### Enumerated Values | Property | Value | | --- | --- | | role | [user, assistant, system] | ## NeverTrigger ``` { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ``` NeverTrigger ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | never | boolean | true | | Never fire; retain the session indefinitely. | ## PaginatedResponse_EventResponse_ ``` { "properties": { "items": { "description": "List of items for the current page.", "items": { "description": "Schema for event response.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Event body content.", "title": "Body" }, "createdAt": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Event creation timestamp.", "title": "Createdat" }, "emitterId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier.", "title": "Emitterid" }, "emitterType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter type.", "title": "Emittertype" }, "eventType": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Event type.", "title": "Eventtype" }, "sequenceId": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Event sequence number within the session.", "title": "Sequenceid" } }, "title": "EventResponse", "type": "object" }, "maxItems": 100, "title": "Items", "type": "array" }, "limit": { "description": "Maximum number of returned items.", "exclusiveMinimum": 0, "title": "Limit", "type": "integer" }, "offset": { "description": "Number of skipped items.", "minimum": 0, "title": "Offset", "type": "integer" }, "total": { "description": "Total number of matching items.", "title": "Total", "type": "integer" } }, "required": [ "items", "offset", "limit", "total" ], "title": "PaginatedResponse[EventResponse]", "type": "object" } ``` PaginatedResponse[EventResponse] ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | items | [EventResponse] | true | maxItems: 100 | List of items for the current page. | | limit | integer | true | | Maximum number of returned items. | | offset | integer | true | minimum: 0 | Number of skipped items. | | total | integer | true | | Total number of matching items. | ## PaginatedResponse_MemorySpaceResponse_ ``` { "properties": { "items": { "description": "List of items for the current page.", "items": { "description": "Response schema for a single memory space.", "properties": { "createdAt": { "description": "Memory space creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "customInstructions": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Custom prompt instructions used for fact extraction. null means the default mem0 extraction prompt is used.", "title": "Custominstructions" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique per user, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Memory space description.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM model configured for memory extraction.", "title": "Llmmodelname" }, "memorySpaceId": { "description": "Unique memory space identifier.", "format": "uuid", "title": "Memoryspaceid", "type": "string" }, "permissions": { "description": "Actions the authenticated caller may perform on this memory space. the creator of a registered space holds all four; a space shared with the caller carries only what the share granted, and one still awaiting registration cannot be shared at all.", "items": { "enum": [ "CAN_VIEW", "CAN_UPDATE", "CAN_DELETE", "CAN_SHARE" ], "type": "string" }, "maxItems": 4, "title": "Permissions", "type": "array" }, "rbacResourceId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Identifier of this memory space in the platform's access-control system. sharing is managed there, and this is the ID its API expects. null while the space has no such record yet, which is the same state that withholds `can_share` from `permissions`.", "title": "RBACResourceID" }, "tenantId": { "description": "Tenant identifier.", "format": "uuid", "title": "Tenantid", "type": "string" }, "userId": { "description": "ID of the user who owns this memory space.", "title": "Userid", "type": "string" } }, "required": [ "memorySpaceId", "userId", "tenantId", "createdAt", "permissions" ], "title": "MemorySpaceResponse", "type": "object" }, "maxItems": 100, "title": "Items", "type": "array" }, "limit": { "description": "Maximum number of returned items.", "exclusiveMinimum": 0, "title": "Limit", "type": "integer" }, "offset": { "description": "Number of skipped items.", "minimum": 0, "title": "Offset", "type": "integer" }, "total": { "description": "Total number of matching items.", "title": "Total", "type": "integer" } }, "required": [ "items", "offset", "limit", "total" ], "title": "PaginatedResponse[MemorySpaceResponse]", "type": "object" } ``` PaginatedResponse[MemorySpaceResponse] ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | items | [MemorySpaceResponse] | true | maxItems: 100 | List of items for the current page. | | limit | integer | true | | Maximum number of returned items. | | offset | integer | true | minimum: 0 | Number of skipped items. | | total | integer | true | | Total number of matching items. | ## PaginatedResponse_SessionResponse_ ``` { "properties": { "items": { "description": "List of items for the current page.", "items": { "description": "Schema for session response.", "properties": { "createdAt": { "description": "Session creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique within the memory space, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Session description.", "title": "Description" }, "id": { "description": "Session id.", "format": "uuid", "title": "Id", "type": "string" }, "lifecycleStrategies": { "description": "Lifecycle strategies associated with this session.", "items": { "description": "Schema for lifecycle strategy in response. execution status is tracked in the strategy_executions table and can be queried separately if needed.", "properties": { "opts": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Strategy-specific configuration parameters.", "title": "Opts" }, "trigger": { "anyOf": [ { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ], "description": "Trigger condition.", "title": "Trigger" }, "type": { "description": "Strategy type.", "enum": [ "soft_delete", "extract_memories" ], "title": "Type", "type": "string" } }, "required": [ "type", "trigger" ], "title": "LifecycleStrategyResponse", "type": "object" }, "maxItems": 5, "title": "Lifecyclestrategies", "type": "array" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Custom metadata as key-value pairs for storing application-specific data.", "title": "Metadata" }, "participants": { "description": "List of participant ids.", "items": { "type": "string" }, "maxItems": 50, "title": "Participants", "type": "array" }, "version": { "default": 1, "description": "Monotonic version for optimistic concurrency. echo back as `if-match: \"\"` on patch to detect lost updates.", "minimum": 1, "title": "Version", "type": "integer" } }, "required": [ "id", "participants", "createdAt" ], "title": "SessionResponse", "type": "object" }, "maxItems": 100, "title": "Items", "type": "array" }, "limit": { "description": "Maximum number of returned items.", "exclusiveMinimum": 0, "title": "Limit", "type": "integer" }, "offset": { "description": "Number of skipped items.", "minimum": 0, "title": "Offset", "type": "integer" }, "total": { "description": "Total number of matching items.", "title": "Total", "type": "integer" } }, "required": [ "items", "offset", "limit", "total" ], "title": "PaginatedResponse[SessionResponse]", "type": "object" } ``` PaginatedResponse[SessionResponse] ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | items | [SessionResponse] | true | maxItems: 100 | List of items for the current page. | | limit | integer | true | | Maximum number of returned items. | | offset | integer | true | minimum: 0 | Number of skipped items. | | total | integer | true | | Total number of matching items. | ## SessionDeduplicationErrorResponse ``` { "description": "Body returned with HTTP 409 when a session create collides on `deduplicationkey`. carries enough context for an ha-replica caller to adopt the existing winner without a second round-trip: the live session's ID and a path to it.", "properties": { "deduplicationKey": { "description": "The deduplication key that triggered the conflict.", "title": "Deduplicationkey", "type": "string" }, "detail": { "description": "Human-readable explanation of the conflict.", "title": "Detail", "type": "string" }, "errorName": { "const": "SessionDeduplicationConflict", "default": "SessionDeduplicationConflict", "description": "Stable machine-readable error name. clients should switch on this, not detail.", "title": "Errorname", "type": "string" }, "existingSessionId": { "anyOf": [ { "format": "uuid", "type": "string" }, { "type": "null" } ], "description": "ID of the live session already holding the key, or null if it could not be resolved (e.g. concurrently soft-deleted).", "title": "Existingsessionid" }, "existingSessionUrl": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Path to get the existing session. null when `existingsessionid` is null. mirrors the `location` response header (which is absolute).", "title": "Existingsessionurl" } }, "required": [ "detail", "deduplicationKey" ], "title": "SessionDeduplicationErrorResponse", "type": "object" } ``` SessionDeduplicationErrorResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | deduplicationKey | string | true | | The deduplication key that triggered the conflict. | | detail | string | true | | Human-readable explanation of the conflict. | | errorName | string | false | | Stable machine-readable error name. clients should switch on this, not detail. | | existingSessionId | any | false | | ID of the live session already holding the key, or null if it could not be resolved (e.g. concurrently soft-deleted). | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string(uuid) | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | existingSessionUrl | any | false | | Path to get the existing session. null when existingsessionid is null. mirrors the location response header (which is absolute). | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## SessionResponse ``` { "description": "Schema for session response.", "properties": { "createdAt": { "description": "Session creation timestamp.", "format": "date-time", "title": "Createdat", "type": "string" }, "deduplicationKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Deduplication key, unique within the memory space, if one was set.", "title": "Deduplicationkey" }, "description": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Session description.", "title": "Description" }, "id": { "description": "Session id.", "format": "uuid", "title": "Id", "type": "string" }, "lifecycleStrategies": { "description": "Lifecycle strategies associated with this session.", "items": { "description": "Schema for lifecycle strategy in response. execution status is tracked in the strategy_executions table and can be queried separately if needed.", "properties": { "opts": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Strategy-specific configuration parameters.", "title": "Opts" }, "trigger": { "anyOf": [ { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ], "description": "Trigger condition.", "title": "Trigger" }, "type": { "description": "Strategy type.", "enum": [ "soft_delete", "extract_memories" ], "title": "Type", "type": "string" } }, "required": [ "type", "trigger" ], "title": "LifecycleStrategyResponse", "type": "object" }, "maxItems": 5, "title": "Lifecyclestrategies", "type": "array" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Custom metadata as key-value pairs for storing application-specific data.", "title": "Metadata" }, "participants": { "description": "List of participant ids.", "items": { "type": "string" }, "maxItems": 50, "title": "Participants", "type": "array" }, "version": { "default": 1, "description": "Monotonic version for optimistic concurrency. echo back as `if-match: \"\"` on patch to detect lost updates.", "minimum": 1, "title": "Version", "type": "integer" } }, "required": [ "id", "participants", "createdAt" ], "title": "SessionResponse", "type": "object" } ``` SessionResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | createdAt | string(date-time) | true | | Session creation timestamp. | | deduplicationKey | any | false | | Deduplication key, unique within the memory space, if one was set. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | any | false | | Session description. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | id | string(uuid) | true | | Session id. | | lifecycleStrategies | [LifecycleStrategyResponse] | false | maxItems: 5 | Lifecycle strategies associated with this session. | | metadata | any | false | | Custom metadata as key-value pairs for storing application-specific data. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | participants | [string] | true | maxItems: 50 | List of participant ids. | | version | integer | false | minimum: 1 | Monotonic version for optimistic concurrency. echo back as if-match: "" on patch to detect lost updates. | ## TTLTrigger ``` { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" } ``` TTLTrigger ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | ttl | integer | true | maximum: 315360000 | Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds. | ## TokenCountTrigger ``` { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" } ``` TokenCountTrigger ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | tokenCount | integer | true | maximum: 100000000 | Token count threshold. | ## UpdateEventRequest ``` { "description": "Schema for create event request.", "properties": { "body": { "anyOf": [ { "additionalProperties": true, "description": "Body for all event types. content field is mandatory, any additional fields are allowed.", "properties": { "content": { "description": "Event body content text.", "maxLength": 100000, "minLength": 1, "title": "Content", "type": "string" } }, "required": [ "content" ], "title": "EventBody", "type": "object" }, { "type": "null" } ], "description": "Updated event body." }, "emitter": { "anyOf": [ { "description": "Schema for emitter data.", "properties": { "id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Emitter identifier (objectid format).", "title": "Id" }, "type": { "description": "Emitter type. one of: 'user', 'agent'.", "enum": [ "user", "agent" ], "title": "Type", "type": "string" } }, "required": [ "type" ], "title": "EmitterDataModel", "type": "object" }, { "type": "null" } ], "description": "Updated emitter information." }, "type": { "anyOf": [ { "enum": [ "message", "tool_output", "status" ], "title": "EventType", "type": "string" }, { "type": "null" } ], "description": "Updated event type." } }, "title": "UpdateEventRequest", "type": "object" } ``` UpdateEventRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | body | any | false | | Updated event body. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | EventBody | false | | Body for all event types. content field is mandatory, any additional fields are allowed. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | emitter | any | false | | Updated emitter information. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | EmitterDataModel | false | | Schema for emitter data. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | type | any | false | | Updated event type. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | EventType | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## UpdateMemorySpaceRequest ``` { "additionalProperties": false, "description": "Request body for partially updating a memory space.", "properties": { "customInstructions": { "anyOf": [ { "maxLength": 10000, "type": "string" }, { "type": "null" } ], "description": "Updated custom prompt instructions used for fact extraction. pass null to clear and revert to mem0's default extraction prompt.", "title": "Custominstructions" }, "description": { "anyOf": [ { "maxLength": 1000, "type": "string" }, { "type": "null" } ], "description": "Updated description for the memory space.", "title": "Description" }, "llmBaseUrl": { "anyOf": [ { "format": "uri", "maxLength": 2083, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Chat API url to use for memory extraction.", "title": "Llmbaseurl" }, "llmModelName": { "anyOf": [ { "maxLength": 200, "type": "string" }, { "type": "null" } ], "description": "Updated LLM model name. non-reasoning models such as ``gpt-4o`` are recommended. reasoning-capable models are significantly slower for fact extraction without producing meaningfully better results.", "title": "Llmmodelname" } }, "title": "UpdateMemorySpaceRequest", "type": "object" } ``` UpdateMemorySpaceRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | customInstructions | any | false | | Updated custom prompt instructions used for fact extraction. pass null to clear and revert to mem0's default extraction prompt. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 10000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | any | false | | Updated description for the memory space. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 1000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | llmBaseUrl | any | false | | Chat API url to use for memory extraction. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string(uri) | false | maxLength: 2083minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | llmModelName | any | false | | Updated LLM model name. non-reasoning models such as gpt-4o are recommended. reasoning-capable models are significantly slower for fact extraction without producing meaningfully better results. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 200 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## UpdateSessionRequest ``` { "description": "Schema for session partial patch-updates.", "properties": { "description": { "anyOf": [ { "maxLength": 1000, "type": "string" }, { "type": "null" } ], "description": "New session description.", "title": "Description" }, "lifecycleStrategies": { "anyOf": [ { "items": { "description": "Lifecycle strategy configuration that defines when and how to manage session lifecycle.", "example": { "trigger": { "ttl": 86400 }, "type": "soft_delete" }, "properties": { "opts": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Optional strategy-specific configuration parameters.", "title": "Opts" }, "trigger": { "anyOf": [ { "additionalProperties": false, "description": "Trigger based on time-to-live (session age). fires when the session age exceeds the configured ttl threshold.", "properties": { "ttl": { "description": "Time-to-live in seconds. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Ttl", "type": "integer" } }, "required": [ "ttl" ], "title": "TTLTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on event count. fires when the session's event count reaches or exceeds the configured threshold.", "properties": { "eventCount": { "description": "Event count threshold.", "exclusiveMinimum": 0, "maximum": 1000000, "title": "Eventcount", "type": "integer" } }, "required": [ "eventCount" ], "title": "EventCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on token count. fires when the session's total token count reaches or exceeds the configured threshold. tokens are calculated from event message content using tiktoken.", "properties": { "tokenCount": { "description": "Token count threshold.", "exclusiveMinimum": 0, "maximum": 100000000, "title": "Tokencount", "type": "integer" } }, "required": [ "tokenCount" ], "title": "TokenCountTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger based on time since the last event in the session. fires when no new events were added to the session within the configured ttl.", "properties": { "idle": { "description": "Time in seconds since last event. values above the environment's configured maximum session ttl are rejected; see max_session_ttl_seconds.", "exclusiveMinimum": 0, "maximum": 315360000, "title": "Idle", "type": "integer" } }, "required": [ "idle" ], "title": "IdleTimeoutTrigger", "type": "object" }, { "additionalProperties": false, "description": "Trigger that never fires. marks the session as exempt from automatic lifecycle execution: the data is retained until explicitly deleted. used for app system data and for compliance retention beyond the ttl/idle 2-year cap.", "properties": { "never": { "const": true, "description": "Never fire; retain the session indefinitely.", "title": "Never", "type": "boolean" } }, "required": [ "never" ], "title": "NeverTrigger", "type": "object" } ], "description": "Trigger condition. available: 'ttl' (time in seconds), 'eventcount', 'tokencount', 'idle', 'never' (retain indefinitely; only accepted in environments where the never-expire trigger is enabled).", "title": "Trigger" }, "type": { "description": "Strategy type. available: 'soft_delete'.", "enum": [ "soft_delete", "extract_memories" ], "title": "Type", "type": "string" } }, "required": [ "type", "trigger" ], "title": "LifecycleStrategiesDataModel", "type": "object" }, "maxItems": 5, "minItems": 1, "type": "array" }, { "type": "null" } ], "description": "When present, replaces the session's lifecycle strategies entirely. at least one strategy is required so every session stays controlled by one; null is rejected -- omit the field to leave the strategies unchanged. a strategy type that already executed for this session can never run again, so changing its configuration is rejected with 409; re-sending it unchanged is inert.", "title": "Lifecyclestrategies" }, "metadata": { "anyOf": [ { "additionalProperties": true, "type": "object" }, { "type": "null" } ], "description": "Custom metadata as key-value pairs for storing application-specific data.", "title": "Metadata" } }, "title": "UpdateSessionRequest", "type": "object" } ``` UpdateSessionRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | any | false | | New session description. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 1000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | lifecycleStrategies | any | false | | When present, replaces the session's lifecycle strategies entirely. at least one strategy is required so every session stays controlled by one; null is rejected -- omit the field to leave the strategies unchanged. a strategy type that already executed for this session can never run again, so changing its configuration is rejected with 409; re-sending it unchanged is inert. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | [LifecycleStrategiesDataModel] | false | maxItems: 5minItems: 1 | [Lifecycle strategy configuration that defines when and how to manage session lifecycle.] | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | metadata | any | false | | Custom metadata as key-value pairs for storing application-specific data. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## ValidationError ``` { "properties": { "ctx": { "description": "Additional context about the validation error.", "title": "Context", "type": "object" }, "input": { "additionalProperties": true, "description": "The input value that caused the validation error.", "title": "Input", "type": "object" }, "loc": { "description": "The location in the request where the validation error occurred.", "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "maxItems": 100, "title": "Location", "type": "array" }, "msg": { "description": "A human-readable description of the validation error.", "title": "Message", "type": "string" }, "type": { "description": "A machine-readable error type identifier.", "title": "Error Type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" } ``` ValidationError ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | ctx | object | false | | Additional context about the validation error. | | input | object | false | | The input value that caused the validation error. | | loc | [anyOf] | true | maxItems: 100 | The location in the request where the validation error occurred. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | msg | string | true | | A human-readable description of the validation error. | | type | string | true | | A machine-readable error type identifier. | --- # Agents URL: https://docs.datarobot.com/en/docs/api/reference/public-api/agents.html > The following endpoints outline how to manage agents. The following endpoints outline how to manage agents. ## Request chat completion by custom model ID Operation path: `POST /api/v2/genai/agents/fromCustomModel/{customModelId}/chat/` Authentication requirements: `BearerAuth` Create a chat completion request for an agent using a custom model. ### Body parameter ``` { "additionalProperties": true, "description": "Represents a chat completion request for an agent.", "properties": { "customModelVersionId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The version ID of the custom model to use for the chat completion.", "title": "customModelVersionId" }, "messages": { "description": "A list of messages comprising the conversation so far.", "items": { "additionalProperties": true, "description": "Represents a message in a chat conversation.", "properties": { "content": { "anyOf": [ { "maxLength": 50000, "type": "string" }, { "type": "null" } ], "description": "The contents of the message.", "title": "content" }, "role": { "description": "The role of the author of this message.", "title": "role", "type": "string" } }, "required": [ "role" ], "title": "AgentMessage", "type": "object" }, "title": "messages", "type": "array" }, "model": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The model identifier to use for completion following openai API notation.", "title": "model" }, "tracingContext": { "anyOf": [ { "description": "Represents a custom tracing context for a chat completion request.", "properties": { "attributes": { "anyOf": [ { "additionalProperties": { "type": "string" }, "type": "object" }, { "type": "null" } ], "description": "The attributes of the tracing context.", "title": "attributes" }, "entityId": { "description": "The ID of the entity in context of which the agent request is performed. should be an entity which user has access to.", "title": "entityId", "type": "string" }, "entityType": { "description": "Type of an entity in context of which the agent request is performed.", "enum": [ "deployment", "use_case" ], "title": "TracingContextEntityType", "type": "string" } }, "required": [ "entityId", "entityType", "attributes" ], "title": "AgentTracingContext", "type": "object" }, { "type": "null" } ], "description": "Optional tracing context for the chat completion request." } }, "required": [ "messages" ], "title": "AgentChatCompletionRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | customModelId | path | string | true | The ID of a custom model to use for the chat completion. | | body | body | AgentChatCompletionRequest | true | none | ### Example responses > 202 Response ``` {} ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 202 | Accepted | Successful Response | Inline | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ### Response Schema ## Obtain chat completion response by custom model ID Operation path: `GET /api/v2/genai/agents/fromCustomModel/{customModelId}/chat/{chatCompletionId}/` Authentication requirements: `BearerAuth` Obtain chat completion response for a given chat completion ID. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | customModelId | path | string | true | The ID of a custom model to use for the chat completion. | | chatCompletionId | path | string | true | The ID of a chat completion object. | ### Example responses > 200 Response ``` { "additionalProperties": true, "description": "Chat completion response from an agent.", "properties": { "choices": { "anyOf": [ { "items": { "additionalProperties": true, "description": "Represents a single choice in the chat completion response.", "properties": { "message": { "additionalProperties": true, "description": "Represents a message in a chat conversation.", "properties": { "content": { "anyOf": [ { "maxLength": 50000, "type": "string" }, { "type": "null" } ], "description": "The contents of the message.", "title": "content" }, "role": { "description": "The role of the author of this message.", "title": "role", "type": "string" } }, "required": [ "role" ], "title": "AgentMessage", "type": "object" } }, "required": [ "message" ], "title": "AgentChoice", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "A list of agent choices. can be more than one. none when failed.", "title": "choices" }, "errorDetails": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Detailed error information if the chat completion failed.", "title": "errorDetails" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Error message if the chat completion failed.", "title": "errorMessage" } }, "title": "AgentChatCompletionResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | AgentChatCompletionResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List search studies by use case ID Operation path: `GET /api/v2/genai/syftrSearch/` Authentication requirements: `BearerAuth` Return all search studies for the specified use case . ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | useCaseId | query | string | true | Use case ID to retrieve search studies from. | | offset | query | integer | false | Skip the specified number of search studies. | | limit | query | integer | false | Retrieve only the specified number of search studies. | | playgroundId | query | any | false | Playground ID associated with a search study. | | search | query | any | false | Only retrieve the search studies with names matching the search query. | | sort | query | any | false | Apply this sort order to the results.Valid options are 'name'. | ### Example responses > 200 Response ``` { "description": "Paginated list of search studies.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "API response object for search study retrieval.", "properties": { "allTrials": { "anyOf": [ { "items": { "description": "Represents a search trial from history.", "properties": { "llmBlueprintId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent blueprint id.", "title": "llmBlueprintId" }, "searchParameters": { "additionalProperties": true, "description": "Search parameters of the point.", "title": "searchParameters", "type": "object" }, "values": { "description": "The resulting values of optimization objectives.", "items": { "type": "number" }, "title": "values", "type": "array" }, "vectorDatabaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent vector database id.", "title": "vectorDatabaseId" } }, "required": [ "llmBlueprintId", "vectorDatabaseId", "values", "searchParameters" ], "title": "HistoryPoint", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "Trials history.", "title": "allTrials" }, "datetimeEnd": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Study end time.", "title": "datetimeEnd" }, "datetimeStart": { "description": "Study start time.", "format": "date-time", "title": "datetimeStart", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Error message if search study fails.", "title": "errorMessage" }, "evalDatasetId": { "description": "The ID of the evaluation dataset.", "title": "evalDatasetId", "type": "string" }, "evalDatasetName": { "description": "The name of evaluation dataset.", "title": "evalDatasetName", "type": "string" }, "evalResults": { "anyOf": [ { "items": {}, "type": "array" }, { "type": "null" } ], "description": "The results of the comparative evaluation of LLM blueprints.", "title": "evalResults" }, "existingBlueprintIds": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The ids of existing LLM blueprints for comparative evaluation.", "title": "existingBlueprintIds" }, "groundingDatasetId": { "description": "The ID of the dataset the vector databases will be built from.", "title": "groundingDatasetId", "type": "string" }, "groundingDatasetName": { "description": "The name of the grouding dataset.", "title": "groundingDatasetName", "type": "string" }, "jobId": { "anyOf": [ { "format": "uuid4", "type": "string" }, { "type": "null" } ], "description": "The ID of the worker job.", "title": "jobId" }, "name": { "description": "Name of the search study.", "title": "name", "type": "string" }, "numConcurrentTrials": { "description": "The number of simultaneously running trials.", "title": "numConcurrentTrials", "type": "integer" }, "numTrials": { "description": "The number of search trials to sample.", "title": "numTrials", "type": "integer" }, "optimizationObjectives": { "description": "Optimization objectives of a study.", "items": { "maxItems": 2, "minItems": 2, "prefixItems": [ { "description": "List of supported search objectives.", "enum": [ "correctness", "all_tokens" ], "title": "SearchObjective", "type": "string" }, { "description": "Whether to minimize or maximize search objective.", "enum": [ "maximize", "minimize" ], "title": "SearchDirection", "type": "string" } ], "type": "array" }, "title": "optimizationObjectives", "type": "array" }, "paretoFront": { "anyOf": [ { "items": {}, "type": "array" }, { "type": "null" } ], "description": "Pareto frontier of a study.", "title": "paretoFront" }, "playgroundId": { "description": "The ID of the existing playground that will be associated with the search.", "title": "playgroundId", "type": "string" }, "searchSpace": { "anyOf": [ { "description": "Represents full search space.", "properties": { "chunkingParameters": { "description": "Parameters of the text chunkers.", "properties": { "chunkOverlapPercentageMax": { "default": 50, "description": "Maximum value of chunk overlap.", "title": "chunkOverlapPercentageMax", "type": "number" }, "chunkOverlapPercentageMin": { "default": 0, "description": "Minimum value of chunk overlap.", "title": "chunkOverlapPercentageMin", "type": "number" }, "chunkOverlapPercentageStep": { "default": 10, "description": "Step value of chunk overlap.", "title": "chunkOverlapPercentageStep", "type": "number" }, "chunkSizeMaxExp": { "default": 8, "description": "Maximum exponent for chunk size (2^8 = 256).", "title": "chunkSizeMaxExp", "type": "integer" }, "chunkSizeMinExp": { "default": 7, "description": "Minimum exponent for chunk size (2^7 = 128).", "title": "chunkSizeMinExp", "type": "integer" }, "chunkingMethods": { "description": "List of chunking methods to use.", "items": { "type": "string" }, "title": "chunkingMethods", "type": "array" }, "embeddingModelNames": { "description": "List of embedding models to use.", "items": { "type": "string" }, "title": "embeddingModelNames", "type": "array" } }, "required": [ "embeddingModelNames" ], "title": "ChunkingParametersConfig", "type": "object" }, "llmConfig": { "description": "Configuration of llms in the search space.", "properties": { "llmNames": { "description": "List of LLM names to use.", "items": { "type": "string" }, "title": "llmNames", "type": "array" }, "temperatureMax": { "default": 1, "description": "Maximum temperature of an llm.", "title": "temperatureMax", "type": "number" }, "temperatureMin": { "default": 0, "description": "Minimum temperature of an llm.", "title": "temperatureMin", "type": "number" }, "temperatureStep": { "default": 0.05, "description": "Step size for LLM temperature.", "title": "temperatureStep", "type": "number" }, "topPMax": { "default": 1, "description": "Maximum top_p of an llm.", "title": "topPMax", "type": "number" }, "topPMin": { "default": 0, "description": "Minimum top_p of an llm.", "title": "topPMin", "type": "number" }, "topPStep": { "default": 0.05, "description": "Step size for LLM top_p.", "title": "topPStep", "type": "number" } }, "required": [ "llmNames" ], "title": "LLMConfig", "type": "object" }, "vectorDatabaseSettings": { "description": "Settings of the vector database.", "properties": { "addNeighborChunks": { "description": "Add neighboring chunks to those that the similarity search retrieves.", "items": { "type": "boolean" }, "title": "addNeighborChunks", "type": "array" }, "maxDocumentRetrievedPerPromptMax": { "default": 10, "description": "Max value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMax", "type": "integer" }, "maxDocumentRetrievedPerPromptMin": { "default": 1, "description": "Min value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMin", "type": "integer" }, "maxDocumentRetrievedPerPromptStep": { "default": 1, "description": "Step for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptStep", "type": "integer" }, "maxMmrLambdaMax": { "default": 1, "description": "Maximum value of mmr lambda.", "title": "maxMmrLambdaMax", "type": "number" }, "maxMmrLambdaMin": { "default": 0, "description": "Minimum value of mmr lambda.", "title": "maxMmrLambdaMin", "type": "number" }, "maxMmrLambdaStep": { "default": 0.1, "description": "Step value of mmr lambda.", "title": "maxMmrLambdaStep", "type": "number" }, "retrievalModes": { "description": "List of retriever modes to use.", "items": { "type": "string" }, "title": "retrievalModes", "type": "array" }, "retrievers": { "description": "List of retriever types to use.", "items": { "type": "string" }, "title": "retrievers", "type": "array" } }, "required": [ "retrievers", "retrievalModes" ], "title": "VectorDatabaseConfig", "type": "object" } }, "title": "SearchSpace", "type": "object" }, { "type": "null" } ], "description": "Search space for the search." }, "searchStudyId": { "description": "The ID of the search study.", "title": "searchStudyId", "type": "string" }, "studyStatus": { "description": "Represents a search study execution state.", "enum": [ "RUNNING", "COMPLETED", "STOPPED", "FAILED" ], "title": "JobStatus", "type": "string" }, "tempPlaygroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the temp playground.", "title": "tempPlaygroundId" }, "trialsFailed": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of failed trials.", "title": "trialsFailed" }, "trialsRunning": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of currently running trials.", "title": "trialsRunning" }, "trialsSuccess": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of completed trials.", "title": "trialsSuccess" }, "useCaseId": { "description": "The ID of the use case the search study is linked to.", "title": "useCaseId", "type": "string" }, "userId": { "description": "The ID of the user.", "title": "userId", "type": "string" }, "userName": { "description": "The user name of the user who ran the study.", "title": "userName", "type": "string" } }, "required": [ "searchSpace", "useCaseId", "groundingDatasetId", "evalDatasetId", "groundingDatasetName", "evalDatasetName", "userId", "userName", "numTrials", "numConcurrentTrials", "optimizationObjectives", "playgroundId", "tempPlaygroundId", "paretoFront", "datetimeStart", "datetimeEnd", "studyStatus", "searchStudyId", "name", "jobId", "trialsRunning", "trialsFailed", "trialsSuccess", "allTrials", "existingBlueprintIds", "evalResults", "errorMessage" ], "title": "SearchStudyResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListSearchStudyResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Search studies has been successfully retrieved. | ListSearchStudyResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Run agentic search Operation path: `POST /api/v2/genai/syftrSearch/` Authentication requirements: `BearerAuth` Run agentic search. ### Body parameter ``` { "description": "API request for run agentic search request.", "properties": { "evalDatasetId": { "description": "The ID of the evaluation dataset.", "title": "evalDatasetId", "type": "string" }, "groundingDatasetId": { "description": "The ID of the dataset the vector databases will be built from.", "title": "groundingDatasetId", "type": "string" }, "name": { "description": "Name of the search study.", "title": "name", "type": "string" }, "numConcurrentTrials": { "description": "The number of simultaneously running trials.", "title": "numConcurrentTrials", "type": "integer" }, "numTrials": { "description": "The number of search trials to sample.", "title": "numTrials", "type": "integer" }, "optimizationObjectives": { "description": "Optimization objectives of a study.", "items": { "maxItems": 2, "minItems": 2, "prefixItems": [ { "description": "List of supported search objectives.", "enum": [ "correctness", "all_tokens" ], "title": "SearchObjective", "type": "string" }, { "description": "Whether to minimize or maximize search objective.", "enum": [ "maximize", "minimize" ], "title": "SearchDirection", "type": "string" } ], "type": "array" }, "title": "optimizationObjectives", "type": "array" }, "playgroundId": { "description": "The ID of the existing playground that will be associated with the search.", "title": "playgroundId", "type": "string" }, "searchSpace": { "description": "Represents full search space.", "properties": { "chunkingParameters": { "description": "Parameters of the text chunkers.", "properties": { "chunkOverlapPercentageMax": { "default": 50, "description": "Maximum value of chunk overlap.", "title": "chunkOverlapPercentageMax", "type": "number" }, "chunkOverlapPercentageMin": { "default": 0, "description": "Minimum value of chunk overlap.", "title": "chunkOverlapPercentageMin", "type": "number" }, "chunkOverlapPercentageStep": { "default": 10, "description": "Step value of chunk overlap.", "title": "chunkOverlapPercentageStep", "type": "number" }, "chunkSizeMaxExp": { "default": 8, "description": "Maximum exponent for chunk size (2^8 = 256).", "title": "chunkSizeMaxExp", "type": "integer" }, "chunkSizeMinExp": { "default": 7, "description": "Minimum exponent for chunk size (2^7 = 128).", "title": "chunkSizeMinExp", "type": "integer" }, "chunkingMethods": { "description": "List of chunking methods to use.", "items": { "type": "string" }, "title": "chunkingMethods", "type": "array" }, "embeddingModelNames": { "description": "List of embedding models to use.", "items": { "type": "string" }, "title": "embeddingModelNames", "type": "array" } }, "required": [ "embeddingModelNames" ], "title": "ChunkingParametersConfig", "type": "object" }, "llmConfig": { "description": "Configuration of llms in the search space.", "properties": { "llmNames": { "description": "List of LLM names to use.", "items": { "type": "string" }, "title": "llmNames", "type": "array" }, "temperatureMax": { "default": 1, "description": "Maximum temperature of an llm.", "title": "temperatureMax", "type": "number" }, "temperatureMin": { "default": 0, "description": "Minimum temperature of an llm.", "title": "temperatureMin", "type": "number" }, "temperatureStep": { "default": 0.05, "description": "Step size for LLM temperature.", "title": "temperatureStep", "type": "number" }, "topPMax": { "default": 1, "description": "Maximum top_p of an llm.", "title": "topPMax", "type": "number" }, "topPMin": { "default": 0, "description": "Minimum top_p of an llm.", "title": "topPMin", "type": "number" }, "topPStep": { "default": 0.05, "description": "Step size for LLM top_p.", "title": "topPStep", "type": "number" } }, "required": [ "llmNames" ], "title": "LLMConfig", "type": "object" }, "vectorDatabaseSettings": { "description": "Settings of the vector database.", "properties": { "addNeighborChunks": { "description": "Add neighboring chunks to those that the similarity search retrieves.", "items": { "type": "boolean" }, "title": "addNeighborChunks", "type": "array" }, "maxDocumentRetrievedPerPromptMax": { "default": 10, "description": "Max value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMax", "type": "integer" }, "maxDocumentRetrievedPerPromptMin": { "default": 1, "description": "Min value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMin", "type": "integer" }, "maxDocumentRetrievedPerPromptStep": { "default": 1, "description": "Step for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptStep", "type": "integer" }, "maxMmrLambdaMax": { "default": 1, "description": "Maximum value of mmr lambda.", "title": "maxMmrLambdaMax", "type": "number" }, "maxMmrLambdaMin": { "default": 0, "description": "Minimum value of mmr lambda.", "title": "maxMmrLambdaMin", "type": "number" }, "maxMmrLambdaStep": { "default": 0.1, "description": "Step value of mmr lambda.", "title": "maxMmrLambdaStep", "type": "number" }, "retrievalModes": { "description": "List of retriever modes to use.", "items": { "type": "string" }, "title": "retrievalModes", "type": "array" }, "retrievers": { "description": "List of retriever types to use.", "items": { "type": "string" }, "title": "retrievers", "type": "array" } }, "required": [ "retrievers", "retrievalModes" ], "title": "VectorDatabaseConfig", "type": "object" } }, "title": "SearchSpace", "type": "object" }, "useCaseId": { "description": "The ID of the use case the search study is linked to.", "title": "useCaseId", "type": "string" } }, "required": [ "useCaseId", "playgroundId", "groundingDatasetId", "evalDatasetId", "numTrials", "numConcurrentTrials", "optimizationObjectives", "searchSpace", "name" ], "title": "RunAgenticSearchRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | RunAgenticSearchRequest | true | none | ### Example responses > 202 Response ``` { "description": "API response object for run agentic search request.", "properties": { "jobId": { "description": "The ID of the worker job.", "format": "uuid4", "title": "jobId", "type": "string" }, "searchStudyId": { "description": "The ID of the search study.", "title": "searchStudyId", "type": "string" } }, "required": [ "searchStudyId", "jobId" ], "title": "RunSearchApiResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 202 | Accepted | Agentic search job successfully accepted. Follow the Location header to poll for job execution status. | RunSearchApiResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Delete existing search study by ID by search study ID Operation path: `DELETE /api/v2/genai/syftrSearch/{searchStudyId}/` Authentication requirements: `BearerAuth` Delete existing search study object from the database. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | searchStudyId | path | string | true | The ID of the search study to be retrieved. | ### Example responses > 202 Response ``` { "description": "API response for the deletion of a search study.", "properties": { "jobId": { "description": "The ID of the worker job.", "format": "uuid4", "title": "jobId", "type": "string" }, "searchStudyId": { "description": "The ID of the search study.", "title": "searchStudyId", "type": "string" } }, "required": [ "searchStudyId", "jobId" ], "title": "DeleteSearchApiResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 202 | Accepted | Search study has been successfully deleted. | DeleteSearchApiResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Get existing search study by ID by search study ID Operation path: `GET /api/v2/genai/syftrSearch/{searchStudyId}/` Authentication requirements: `BearerAuth` Return existing search study object from the database. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | searchStudyId | path | string | true | The ID of the search study to be retrieved. | ### Example responses > 200 Response ``` { "description": "API response object for search study retrieval.", "properties": { "allTrials": { "anyOf": [ { "items": { "description": "Represents a search trial from history.", "properties": { "llmBlueprintId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent blueprint id.", "title": "llmBlueprintId" }, "searchParameters": { "additionalProperties": true, "description": "Search parameters of the point.", "title": "searchParameters", "type": "object" }, "values": { "description": "The resulting values of optimization objectives.", "items": { "type": "number" }, "title": "values", "type": "array" }, "vectorDatabaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent vector database id.", "title": "vectorDatabaseId" } }, "required": [ "llmBlueprintId", "vectorDatabaseId", "values", "searchParameters" ], "title": "HistoryPoint", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "Trials history.", "title": "allTrials" }, "datetimeEnd": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Study end time.", "title": "datetimeEnd" }, "datetimeStart": { "description": "Study start time.", "format": "date-time", "title": "datetimeStart", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Error message if search study fails.", "title": "errorMessage" }, "evalDatasetId": { "description": "The ID of the evaluation dataset.", "title": "evalDatasetId", "type": "string" }, "evalDatasetName": { "description": "The name of evaluation dataset.", "title": "evalDatasetName", "type": "string" }, "evalResults": { "anyOf": [ { "items": {}, "type": "array" }, { "type": "null" } ], "description": "The results of the comparative evaluation of LLM blueprints.", "title": "evalResults" }, "existingBlueprintIds": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The ids of existing LLM blueprints for comparative evaluation.", "title": "existingBlueprintIds" }, "groundingDatasetId": { "description": "The ID of the dataset the vector databases will be built from.", "title": "groundingDatasetId", "type": "string" }, "groundingDatasetName": { "description": "The name of the grouding dataset.", "title": "groundingDatasetName", "type": "string" }, "jobId": { "anyOf": [ { "format": "uuid4", "type": "string" }, { "type": "null" } ], "description": "The ID of the worker job.", "title": "jobId" }, "name": { "description": "Name of the search study.", "title": "name", "type": "string" }, "numConcurrentTrials": { "description": "The number of simultaneously running trials.", "title": "numConcurrentTrials", "type": "integer" }, "numTrials": { "description": "The number of search trials to sample.", "title": "numTrials", "type": "integer" }, "optimizationObjectives": { "description": "Optimization objectives of a study.", "items": { "maxItems": 2, "minItems": 2, "prefixItems": [ { "description": "List of supported search objectives.", "enum": [ "correctness", "all_tokens" ], "title": "SearchObjective", "type": "string" }, { "description": "Whether to minimize or maximize search objective.", "enum": [ "maximize", "minimize" ], "title": "SearchDirection", "type": "string" } ], "type": "array" }, "title": "optimizationObjectives", "type": "array" }, "paretoFront": { "anyOf": [ { "items": {}, "type": "array" }, { "type": "null" } ], "description": "Pareto frontier of a study.", "title": "paretoFront" }, "playgroundId": { "description": "The ID of the existing playground that will be associated with the search.", "title": "playgroundId", "type": "string" }, "searchSpace": { "anyOf": [ { "description": "Represents full search space.", "properties": { "chunkingParameters": { "description": "Parameters of the text chunkers.", "properties": { "chunkOverlapPercentageMax": { "default": 50, "description": "Maximum value of chunk overlap.", "title": "chunkOverlapPercentageMax", "type": "number" }, "chunkOverlapPercentageMin": { "default": 0, "description": "Minimum value of chunk overlap.", "title": "chunkOverlapPercentageMin", "type": "number" }, "chunkOverlapPercentageStep": { "default": 10, "description": "Step value of chunk overlap.", "title": "chunkOverlapPercentageStep", "type": "number" }, "chunkSizeMaxExp": { "default": 8, "description": "Maximum exponent for chunk size (2^8 = 256).", "title": "chunkSizeMaxExp", "type": "integer" }, "chunkSizeMinExp": { "default": 7, "description": "Minimum exponent for chunk size (2^7 = 128).", "title": "chunkSizeMinExp", "type": "integer" }, "chunkingMethods": { "description": "List of chunking methods to use.", "items": { "type": "string" }, "title": "chunkingMethods", "type": "array" }, "embeddingModelNames": { "description": "List of embedding models to use.", "items": { "type": "string" }, "title": "embeddingModelNames", "type": "array" } }, "required": [ "embeddingModelNames" ], "title": "ChunkingParametersConfig", "type": "object" }, "llmConfig": { "description": "Configuration of llms in the search space.", "properties": { "llmNames": { "description": "List of LLM names to use.", "items": { "type": "string" }, "title": "llmNames", "type": "array" }, "temperatureMax": { "default": 1, "description": "Maximum temperature of an llm.", "title": "temperatureMax", "type": "number" }, "temperatureMin": { "default": 0, "description": "Minimum temperature of an llm.", "title": "temperatureMin", "type": "number" }, "temperatureStep": { "default": 0.05, "description": "Step size for LLM temperature.", "title": "temperatureStep", "type": "number" }, "topPMax": { "default": 1, "description": "Maximum top_p of an llm.", "title": "topPMax", "type": "number" }, "topPMin": { "default": 0, "description": "Minimum top_p of an llm.", "title": "topPMin", "type": "number" }, "topPStep": { "default": 0.05, "description": "Step size for LLM top_p.", "title": "topPStep", "type": "number" } }, "required": [ "llmNames" ], "title": "LLMConfig", "type": "object" }, "vectorDatabaseSettings": { "description": "Settings of the vector database.", "properties": { "addNeighborChunks": { "description": "Add neighboring chunks to those that the similarity search retrieves.", "items": { "type": "boolean" }, "title": "addNeighborChunks", "type": "array" }, "maxDocumentRetrievedPerPromptMax": { "default": 10, "description": "Max value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMax", "type": "integer" }, "maxDocumentRetrievedPerPromptMin": { "default": 1, "description": "Min value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMin", "type": "integer" }, "maxDocumentRetrievedPerPromptStep": { "default": 1, "description": "Step for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptStep", "type": "integer" }, "maxMmrLambdaMax": { "default": 1, "description": "Maximum value of mmr lambda.", "title": "maxMmrLambdaMax", "type": "number" }, "maxMmrLambdaMin": { "default": 0, "description": "Minimum value of mmr lambda.", "title": "maxMmrLambdaMin", "type": "number" }, "maxMmrLambdaStep": { "default": 0.1, "description": "Step value of mmr lambda.", "title": "maxMmrLambdaStep", "type": "number" }, "retrievalModes": { "description": "List of retriever modes to use.", "items": { "type": "string" }, "title": "retrievalModes", "type": "array" }, "retrievers": { "description": "List of retriever types to use.", "items": { "type": "string" }, "title": "retrievers", "type": "array" } }, "required": [ "retrievers", "retrievalModes" ], "title": "VectorDatabaseConfig", "type": "object" } }, "title": "SearchSpace", "type": "object" }, { "type": "null" } ], "description": "Search space for the search." }, "searchStudyId": { "description": "The ID of the search study.", "title": "searchStudyId", "type": "string" }, "studyStatus": { "description": "Represents a search study execution state.", "enum": [ "RUNNING", "COMPLETED", "STOPPED", "FAILED" ], "title": "JobStatus", "type": "string" }, "tempPlaygroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the temp playground.", "title": "tempPlaygroundId" }, "trialsFailed": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of failed trials.", "title": "trialsFailed" }, "trialsRunning": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of currently running trials.", "title": "trialsRunning" }, "trialsSuccess": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of completed trials.", "title": "trialsSuccess" }, "useCaseId": { "description": "The ID of the use case the search study is linked to.", "title": "useCaseId", "type": "string" }, "userId": { "description": "The ID of the user.", "title": "userId", "type": "string" }, "userName": { "description": "The user name of the user who ran the study.", "title": "userName", "type": "string" } }, "required": [ "searchSpace", "useCaseId", "groundingDatasetId", "evalDatasetId", "groundingDatasetName", "evalDatasetName", "userId", "userName", "numTrials", "numConcurrentTrials", "optimizationObjectives", "playgroundId", "tempPlaygroundId", "paretoFront", "datetimeStart", "datetimeEnd", "studyStatus", "searchStudyId", "name", "jobId", "trialsRunning", "trialsFailed", "trialsSuccess", "allTrials", "existingBlueprintIds", "evalResults", "errorMessage" ], "title": "SearchStudyResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Search study has been successfully retrieved. | SearchStudyResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Edit search study by search study ID Operation path: `PATCH /api/v2/genai/syftrSearch/{searchStudyId}/` Authentication requirements: `BearerAuth` Edit an existing search study object. ### Body parameter ``` { "description": "The body of the \"edit search study\" request.", "properties": { "name": { "description": "The new name of the search study.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" } }, "required": [ "name" ], "title": "EditSearchStudyRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | searchStudyId | path | string | true | The ID of the search study to be edited. | | body | body | EditSearchStudyRequest | true | none | ### Example responses > 200 Response ``` {} ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | Inline | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ### Response Schema # Schemas ## AgentChatCompletionRequest ``` { "additionalProperties": true, "description": "Represents a chat completion request for an agent.", "properties": { "customModelVersionId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The version ID of the custom model to use for the chat completion.", "title": "customModelVersionId" }, "messages": { "description": "A list of messages comprising the conversation so far.", "items": { "additionalProperties": true, "description": "Represents a message in a chat conversation.", "properties": { "content": { "anyOf": [ { "maxLength": 50000, "type": "string" }, { "type": "null" } ], "description": "The contents of the message.", "title": "content" }, "role": { "description": "The role of the author of this message.", "title": "role", "type": "string" } }, "required": [ "role" ], "title": "AgentMessage", "type": "object" }, "title": "messages", "type": "array" }, "model": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The model identifier to use for completion following openai API notation.", "title": "model" }, "tracingContext": { "anyOf": [ { "description": "Represents a custom tracing context for a chat completion request.", "properties": { "attributes": { "anyOf": [ { "additionalProperties": { "type": "string" }, "type": "object" }, { "type": "null" } ], "description": "The attributes of the tracing context.", "title": "attributes" }, "entityId": { "description": "The ID of the entity in context of which the agent request is performed. should be an entity which user has access to.", "title": "entityId", "type": "string" }, "entityType": { "description": "Type of an entity in context of which the agent request is performed.", "enum": [ "deployment", "use_case" ], "title": "TracingContextEntityType", "type": "string" } }, "required": [ "entityId", "entityType", "attributes" ], "title": "AgentTracingContext", "type": "object" }, { "type": "null" } ], "description": "Optional tracing context for the chat completion request." } }, "required": [ "messages" ], "title": "AgentChatCompletionRequest", "type": "object" } ``` AgentChatCompletionRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | customModelVersionId | any | false | | The version ID of the custom model to use for the chat completion. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | messages | [AgentMessage] | true | | A list of messages comprising the conversation so far. | | model | any | false | | The model identifier to use for completion following openai API notation. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | tracingContext | any | false | | Optional tracing context for the chat completion request. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | AgentTracingContext | false | | Represents a custom tracing context for a chat completion request. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## AgentChatCompletionResponse ``` { "additionalProperties": true, "description": "Chat completion response from an agent.", "properties": { "choices": { "anyOf": [ { "items": { "additionalProperties": true, "description": "Represents a single choice in the chat completion response.", "properties": { "message": { "additionalProperties": true, "description": "Represents a message in a chat conversation.", "properties": { "content": { "anyOf": [ { "maxLength": 50000, "type": "string" }, { "type": "null" } ], "description": "The contents of the message.", "title": "content" }, "role": { "description": "The role of the author of this message.", "title": "role", "type": "string" } }, "required": [ "role" ], "title": "AgentMessage", "type": "object" } }, "required": [ "message" ], "title": "AgentChoice", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "A list of agent choices. can be more than one. none when failed.", "title": "choices" }, "errorDetails": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Detailed error information if the chat completion failed.", "title": "errorDetails" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Error message if the chat completion failed.", "title": "errorMessage" } }, "title": "AgentChatCompletionResponse", "type": "object" } ``` AgentChatCompletionResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | choices | any | false | | A list of agent choices. can be more than one. none when failed. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | [AgentChoice] | false | | [Represents a single choice in the chat completion response.] | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | errorDetails | any | false | | Detailed error information if the chat completion failed. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | errorMessage | any | false | | Error message if the chat completion failed. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## AgentChoice ``` { "additionalProperties": true, "description": "Represents a single choice in the chat completion response.", "properties": { "message": { "additionalProperties": true, "description": "Represents a message in a chat conversation.", "properties": { "content": { "anyOf": [ { "maxLength": 50000, "type": "string" }, { "type": "null" } ], "description": "The contents of the message.", "title": "content" }, "role": { "description": "The role of the author of this message.", "title": "role", "type": "string" } }, "required": [ "role" ], "title": "AgentMessage", "type": "object" } }, "required": [ "message" ], "title": "AgentChoice", "type": "object" } ``` AgentChoice ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | message | AgentMessage | true | | The message content of the choice. | ## AgentMessage ``` { "additionalProperties": true, "description": "Represents a message in a chat conversation.", "properties": { "content": { "anyOf": [ { "maxLength": 50000, "type": "string" }, { "type": "null" } ], "description": "The contents of the message.", "title": "content" }, "role": { "description": "The role of the author of this message.", "title": "role", "type": "string" } }, "required": [ "role" ], "title": "AgentMessage", "type": "object" } ``` AgentMessage ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | content | any | false | | The contents of the message. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 50000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | role | string | true | | The role of the author of this message. | ## AgentTracingContext ``` { "description": "Represents a custom tracing context for a chat completion request.", "properties": { "attributes": { "anyOf": [ { "additionalProperties": { "type": "string" }, "type": "object" }, { "type": "null" } ], "description": "The attributes of the tracing context.", "title": "attributes" }, "entityId": { "description": "The ID of the entity in context of which the agent request is performed. should be an entity which user has access to.", "title": "entityId", "type": "string" }, "entityType": { "description": "Type of an entity in context of which the agent request is performed.", "enum": [ "deployment", "use_case" ], "title": "TracingContextEntityType", "type": "string" } }, "required": [ "entityId", "entityType", "attributes" ], "title": "AgentTracingContext", "type": "object" } ``` AgentTracingContext ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | attributes | any | true | | The attributes of the tracing context. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | object | false | | none | | »» additionalProperties | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | entityId | string | true | | The ID of the entity in context of which the agent request is performed. should be an entity which user has access to. | | entityType | TracingContextEntityType | true | | The type of an entity in context of which the agent request is performed. should be an entity which user has access to. | ## ChunkingParametersConfig ``` { "description": "Parameters of the text chunkers.", "properties": { "chunkOverlapPercentageMax": { "default": 50, "description": "Maximum value of chunk overlap.", "title": "chunkOverlapPercentageMax", "type": "number" }, "chunkOverlapPercentageMin": { "default": 0, "description": "Minimum value of chunk overlap.", "title": "chunkOverlapPercentageMin", "type": "number" }, "chunkOverlapPercentageStep": { "default": 10, "description": "Step value of chunk overlap.", "title": "chunkOverlapPercentageStep", "type": "number" }, "chunkSizeMaxExp": { "default": 8, "description": "Maximum exponent for chunk size (2^8 = 256).", "title": "chunkSizeMaxExp", "type": "integer" }, "chunkSizeMinExp": { "default": 7, "description": "Minimum exponent for chunk size (2^7 = 128).", "title": "chunkSizeMinExp", "type": "integer" }, "chunkingMethods": { "description": "List of chunking methods to use.", "items": { "type": "string" }, "title": "chunkingMethods", "type": "array" }, "embeddingModelNames": { "description": "List of embedding models to use.", "items": { "type": "string" }, "title": "embeddingModelNames", "type": "array" } }, "required": [ "embeddingModelNames" ], "title": "ChunkingParametersConfig", "type": "object" } ``` ChunkingParametersConfig ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | chunkOverlapPercentageMax | number | false | | Maximum value of chunk overlap. | | chunkOverlapPercentageMin | number | false | | Minimum value of chunk overlap. | | chunkOverlapPercentageStep | number | false | | Step value of chunk overlap. | | chunkSizeMaxExp | integer | false | | Maximum exponent for chunk size (2^8 = 256). | | chunkSizeMinExp | integer | false | | Minimum exponent for chunk size (2^7 = 128). | | chunkingMethods | [string] | false | | List of chunking methods to use. | | embeddingModelNames | [string] | true | | List of embedding models to use. | ## DeleteSearchApiResponse ``` { "description": "API response for the deletion of a search study.", "properties": { "jobId": { "description": "The ID of the worker job.", "format": "uuid4", "title": "jobId", "type": "string" }, "searchStudyId": { "description": "The ID of the search study.", "title": "searchStudyId", "type": "string" } }, "required": [ "searchStudyId", "jobId" ], "title": "DeleteSearchApiResponse", "type": "object" } ``` DeleteSearchApiResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | jobId | string(uuid4) | true | | The ID of the worker job. | | searchStudyId | string | true | | The ID of the search study. | ## EditSearchStudyRequest ``` { "description": "The body of the \"edit search study\" request.", "properties": { "name": { "description": "The new name of the search study.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" } }, "required": [ "name" ], "title": "EditSearchStudyRequest", "type": "object" } ``` EditSearchStudyRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | name | string | true | maxLength: 5000minLength: 1minLength: 1 | The new name of the search study. | ## HTTPValidationErrorResponse ``` { "properties": { "detail": { "items": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "title": "detail", "type": "array" } }, "title": "HTTPValidationErrorResponse", "type": "object" } ``` HTTPValidationErrorResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | detail | [ValidationError] | false | | none | ## HistoryPoint ``` { "description": "Represents a search trial from history.", "properties": { "llmBlueprintId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent blueprint id.", "title": "llmBlueprintId" }, "searchParameters": { "additionalProperties": true, "description": "Search parameters of the point.", "title": "searchParameters", "type": "object" }, "values": { "description": "The resulting values of optimization objectives.", "items": { "type": "number" }, "title": "values", "type": "array" }, "vectorDatabaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent vector database id.", "title": "vectorDatabaseId" } }, "required": [ "llmBlueprintId", "vectorDatabaseId", "values", "searchParameters" ], "title": "HistoryPoint", "type": "object" } ``` HistoryPoint ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | llmBlueprintId | any | true | | The correspondent blueprint id. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | searchParameters | object | true | | Search parameters of the point. | | values | [number] | true | | The resulting values of optimization objectives. | | vectorDatabaseId | any | true | | The correspondent vector database id. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## JobStatus ``` { "description": "Represents a search study execution state.", "enum": [ "RUNNING", "COMPLETED", "STOPPED", "FAILED" ], "title": "JobStatus", "type": "string" } ``` JobStatus ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | JobStatus | string | false | | Represents a search study execution state. | ### Enumerated Values | Property | Value | | --- | --- | | JobStatus | [RUNNING, COMPLETED, STOPPED, FAILED] | ## LLMConfig ``` { "description": "Configuration of llms in the search space.", "properties": { "llmNames": { "description": "List of LLM names to use.", "items": { "type": "string" }, "title": "llmNames", "type": "array" }, "temperatureMax": { "default": 1, "description": "Maximum temperature of an llm.", "title": "temperatureMax", "type": "number" }, "temperatureMin": { "default": 0, "description": "Minimum temperature of an llm.", "title": "temperatureMin", "type": "number" }, "temperatureStep": { "default": 0.05, "description": "Step size for LLM temperature.", "title": "temperatureStep", "type": "number" }, "topPMax": { "default": 1, "description": "Maximum top_p of an llm.", "title": "topPMax", "type": "number" }, "topPMin": { "default": 0, "description": "Minimum top_p of an llm.", "title": "topPMin", "type": "number" }, "topPStep": { "default": 0.05, "description": "Step size for LLM top_p.", "title": "topPStep", "type": "number" } }, "required": [ "llmNames" ], "title": "LLMConfig", "type": "object" } ``` LLMConfig ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | llmNames | [string] | true | | List of LLM names to use. | | temperatureMax | number | false | | Maximum temperature of an llm. | | temperatureMin | number | false | | Minimum temperature of an llm. | | temperatureStep | number | false | | Step size for LLM temperature. | | topPMax | number | false | | Maximum top_p of an llm. | | topPMin | number | false | | Minimum top_p of an llm. | | topPStep | number | false | | Step size for LLM top_p. | ## ListSearchStudyResponse ``` { "description": "Paginated list of search studies.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "API response object for search study retrieval.", "properties": { "allTrials": { "anyOf": [ { "items": { "description": "Represents a search trial from history.", "properties": { "llmBlueprintId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent blueprint id.", "title": "llmBlueprintId" }, "searchParameters": { "additionalProperties": true, "description": "Search parameters of the point.", "title": "searchParameters", "type": "object" }, "values": { "description": "The resulting values of optimization objectives.", "items": { "type": "number" }, "title": "values", "type": "array" }, "vectorDatabaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent vector database id.", "title": "vectorDatabaseId" } }, "required": [ "llmBlueprintId", "vectorDatabaseId", "values", "searchParameters" ], "title": "HistoryPoint", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "Trials history.", "title": "allTrials" }, "datetimeEnd": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Study end time.", "title": "datetimeEnd" }, "datetimeStart": { "description": "Study start time.", "format": "date-time", "title": "datetimeStart", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Error message if search study fails.", "title": "errorMessage" }, "evalDatasetId": { "description": "The ID of the evaluation dataset.", "title": "evalDatasetId", "type": "string" }, "evalDatasetName": { "description": "The name of evaluation dataset.", "title": "evalDatasetName", "type": "string" }, "evalResults": { "anyOf": [ { "items": {}, "type": "array" }, { "type": "null" } ], "description": "The results of the comparative evaluation of LLM blueprints.", "title": "evalResults" }, "existingBlueprintIds": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The ids of existing LLM blueprints for comparative evaluation.", "title": "existingBlueprintIds" }, "groundingDatasetId": { "description": "The ID of the dataset the vector databases will be built from.", "title": "groundingDatasetId", "type": "string" }, "groundingDatasetName": { "description": "The name of the grouding dataset.", "title": "groundingDatasetName", "type": "string" }, "jobId": { "anyOf": [ { "format": "uuid4", "type": "string" }, { "type": "null" } ], "description": "The ID of the worker job.", "title": "jobId" }, "name": { "description": "Name of the search study.", "title": "name", "type": "string" }, "numConcurrentTrials": { "description": "The number of simultaneously running trials.", "title": "numConcurrentTrials", "type": "integer" }, "numTrials": { "description": "The number of search trials to sample.", "title": "numTrials", "type": "integer" }, "optimizationObjectives": { "description": "Optimization objectives of a study.", "items": { "maxItems": 2, "minItems": 2, "prefixItems": [ { "description": "List of supported search objectives.", "enum": [ "correctness", "all_tokens" ], "title": "SearchObjective", "type": "string" }, { "description": "Whether to minimize or maximize search objective.", "enum": [ "maximize", "minimize" ], "title": "SearchDirection", "type": "string" } ], "type": "array" }, "title": "optimizationObjectives", "type": "array" }, "paretoFront": { "anyOf": [ { "items": {}, "type": "array" }, { "type": "null" } ], "description": "Pareto frontier of a study.", "title": "paretoFront" }, "playgroundId": { "description": "The ID of the existing playground that will be associated with the search.", "title": "playgroundId", "type": "string" }, "searchSpace": { "anyOf": [ { "description": "Represents full search space.", "properties": { "chunkingParameters": { "description": "Parameters of the text chunkers.", "properties": { "chunkOverlapPercentageMax": { "default": 50, "description": "Maximum value of chunk overlap.", "title": "chunkOverlapPercentageMax", "type": "number" }, "chunkOverlapPercentageMin": { "default": 0, "description": "Minimum value of chunk overlap.", "title": "chunkOverlapPercentageMin", "type": "number" }, "chunkOverlapPercentageStep": { "default": 10, "description": "Step value of chunk overlap.", "title": "chunkOverlapPercentageStep", "type": "number" }, "chunkSizeMaxExp": { "default": 8, "description": "Maximum exponent for chunk size (2^8 = 256).", "title": "chunkSizeMaxExp", "type": "integer" }, "chunkSizeMinExp": { "default": 7, "description": "Minimum exponent for chunk size (2^7 = 128).", "title": "chunkSizeMinExp", "type": "integer" }, "chunkingMethods": { "description": "List of chunking methods to use.", "items": { "type": "string" }, "title": "chunkingMethods", "type": "array" }, "embeddingModelNames": { "description": "List of embedding models to use.", "items": { "type": "string" }, "title": "embeddingModelNames", "type": "array" } }, "required": [ "embeddingModelNames" ], "title": "ChunkingParametersConfig", "type": "object" }, "llmConfig": { "description": "Configuration of llms in the search space.", "properties": { "llmNames": { "description": "List of LLM names to use.", "items": { "type": "string" }, "title": "llmNames", "type": "array" }, "temperatureMax": { "default": 1, "description": "Maximum temperature of an llm.", "title": "temperatureMax", "type": "number" }, "temperatureMin": { "default": 0, "description": "Minimum temperature of an llm.", "title": "temperatureMin", "type": "number" }, "temperatureStep": { "default": 0.05, "description": "Step size for LLM temperature.", "title": "temperatureStep", "type": "number" }, "topPMax": { "default": 1, "description": "Maximum top_p of an llm.", "title": "topPMax", "type": "number" }, "topPMin": { "default": 0, "description": "Minimum top_p of an llm.", "title": "topPMin", "type": "number" }, "topPStep": { "default": 0.05, "description": "Step size for LLM top_p.", "title": "topPStep", "type": "number" } }, "required": [ "llmNames" ], "title": "LLMConfig", "type": "object" }, "vectorDatabaseSettings": { "description": "Settings of the vector database.", "properties": { "addNeighborChunks": { "description": "Add neighboring chunks to those that the similarity search retrieves.", "items": { "type": "boolean" }, "title": "addNeighborChunks", "type": "array" }, "maxDocumentRetrievedPerPromptMax": { "default": 10, "description": "Max value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMax", "type": "integer" }, "maxDocumentRetrievedPerPromptMin": { "default": 1, "description": "Min value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMin", "type": "integer" }, "maxDocumentRetrievedPerPromptStep": { "default": 1, "description": "Step for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptStep", "type": "integer" }, "maxMmrLambdaMax": { "default": 1, "description": "Maximum value of mmr lambda.", "title": "maxMmrLambdaMax", "type": "number" }, "maxMmrLambdaMin": { "default": 0, "description": "Minimum value of mmr lambda.", "title": "maxMmrLambdaMin", "type": "number" }, "maxMmrLambdaStep": { "default": 0.1, "description": "Step value of mmr lambda.", "title": "maxMmrLambdaStep", "type": "number" }, "retrievalModes": { "description": "List of retriever modes to use.", "items": { "type": "string" }, "title": "retrievalModes", "type": "array" }, "retrievers": { "description": "List of retriever types to use.", "items": { "type": "string" }, "title": "retrievers", "type": "array" } }, "required": [ "retrievers", "retrievalModes" ], "title": "VectorDatabaseConfig", "type": "object" } }, "title": "SearchSpace", "type": "object" }, { "type": "null" } ], "description": "Search space for the search." }, "searchStudyId": { "description": "The ID of the search study.", "title": "searchStudyId", "type": "string" }, "studyStatus": { "description": "Represents a search study execution state.", "enum": [ "RUNNING", "COMPLETED", "STOPPED", "FAILED" ], "title": "JobStatus", "type": "string" }, "tempPlaygroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the temp playground.", "title": "tempPlaygroundId" }, "trialsFailed": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of failed trials.", "title": "trialsFailed" }, "trialsRunning": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of currently running trials.", "title": "trialsRunning" }, "trialsSuccess": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of completed trials.", "title": "trialsSuccess" }, "useCaseId": { "description": "The ID of the use case the search study is linked to.", "title": "useCaseId", "type": "string" }, "userId": { "description": "The ID of the user.", "title": "userId", "type": "string" }, "userName": { "description": "The user name of the user who ran the study.", "title": "userName", "type": "string" } }, "required": [ "searchSpace", "useCaseId", "groundingDatasetId", "evalDatasetId", "groundingDatasetName", "evalDatasetName", "userId", "userName", "numTrials", "numConcurrentTrials", "optimizationObjectives", "playgroundId", "tempPlaygroundId", "paretoFront", "datetimeStart", "datetimeEnd", "studyStatus", "searchStudyId", "name", "jobId", "trialsRunning", "trialsFailed", "trialsSuccess", "allTrials", "existingBlueprintIds", "evalResults", "errorMessage" ], "title": "SearchStudyResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListSearchStudyResponse", "type": "object" } ``` ListSearchStudyResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | count | integer | true | | The number of records on this page. | | data | [SearchStudyResponse] | true | | The list of records. | | next | any | true | | The url to the next page, or null if there is no such page. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | previous | any | true | | The url to the previous page, or null if there is no such page. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | totalCount | integer | true | | The total number of records. | ## ListSearchStudySortQueryParam ``` { "description": "API object for sort order values for listiing search studies.", "enum": [ "name", "-name" ], "title": "ListSearchStudySortQueryParam", "type": "string" } ``` ListSearchStudySortQueryParam ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | ListSearchStudySortQueryParam | string | false | | API object for sort order values for listiing search studies. | ### Enumerated Values | Property | Value | | --- | --- | | ListSearchStudySortQueryParam | [name, -name] | ## RunAgenticSearchRequest ``` { "description": "API request for run agentic search request.", "properties": { "evalDatasetId": { "description": "The ID of the evaluation dataset.", "title": "evalDatasetId", "type": "string" }, "groundingDatasetId": { "description": "The ID of the dataset the vector databases will be built from.", "title": "groundingDatasetId", "type": "string" }, "name": { "description": "Name of the search study.", "title": "name", "type": "string" }, "numConcurrentTrials": { "description": "The number of simultaneously running trials.", "title": "numConcurrentTrials", "type": "integer" }, "numTrials": { "description": "The number of search trials to sample.", "title": "numTrials", "type": "integer" }, "optimizationObjectives": { "description": "Optimization objectives of a study.", "items": { "maxItems": 2, "minItems": 2, "prefixItems": [ { "description": "List of supported search objectives.", "enum": [ "correctness", "all_tokens" ], "title": "SearchObjective", "type": "string" }, { "description": "Whether to minimize or maximize search objective.", "enum": [ "maximize", "minimize" ], "title": "SearchDirection", "type": "string" } ], "type": "array" }, "title": "optimizationObjectives", "type": "array" }, "playgroundId": { "description": "The ID of the existing playground that will be associated with the search.", "title": "playgroundId", "type": "string" }, "searchSpace": { "description": "Represents full search space.", "properties": { "chunkingParameters": { "description": "Parameters of the text chunkers.", "properties": { "chunkOverlapPercentageMax": { "default": 50, "description": "Maximum value of chunk overlap.", "title": "chunkOverlapPercentageMax", "type": "number" }, "chunkOverlapPercentageMin": { "default": 0, "description": "Minimum value of chunk overlap.", "title": "chunkOverlapPercentageMin", "type": "number" }, "chunkOverlapPercentageStep": { "default": 10, "description": "Step value of chunk overlap.", "title": "chunkOverlapPercentageStep", "type": "number" }, "chunkSizeMaxExp": { "default": 8, "description": "Maximum exponent for chunk size (2^8 = 256).", "title": "chunkSizeMaxExp", "type": "integer" }, "chunkSizeMinExp": { "default": 7, "description": "Minimum exponent for chunk size (2^7 = 128).", "title": "chunkSizeMinExp", "type": "integer" }, "chunkingMethods": { "description": "List of chunking methods to use.", "items": { "type": "string" }, "title": "chunkingMethods", "type": "array" }, "embeddingModelNames": { "description": "List of embedding models to use.", "items": { "type": "string" }, "title": "embeddingModelNames", "type": "array" } }, "required": [ "embeddingModelNames" ], "title": "ChunkingParametersConfig", "type": "object" }, "llmConfig": { "description": "Configuration of llms in the search space.", "properties": { "llmNames": { "description": "List of LLM names to use.", "items": { "type": "string" }, "title": "llmNames", "type": "array" }, "temperatureMax": { "default": 1, "description": "Maximum temperature of an llm.", "title": "temperatureMax", "type": "number" }, "temperatureMin": { "default": 0, "description": "Minimum temperature of an llm.", "title": "temperatureMin", "type": "number" }, "temperatureStep": { "default": 0.05, "description": "Step size for LLM temperature.", "title": "temperatureStep", "type": "number" }, "topPMax": { "default": 1, "description": "Maximum top_p of an llm.", "title": "topPMax", "type": "number" }, "topPMin": { "default": 0, "description": "Minimum top_p of an llm.", "title": "topPMin", "type": "number" }, "topPStep": { "default": 0.05, "description": "Step size for LLM top_p.", "title": "topPStep", "type": "number" } }, "required": [ "llmNames" ], "title": "LLMConfig", "type": "object" }, "vectorDatabaseSettings": { "description": "Settings of the vector database.", "properties": { "addNeighborChunks": { "description": "Add neighboring chunks to those that the similarity search retrieves.", "items": { "type": "boolean" }, "title": "addNeighborChunks", "type": "array" }, "maxDocumentRetrievedPerPromptMax": { "default": 10, "description": "Max value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMax", "type": "integer" }, "maxDocumentRetrievedPerPromptMin": { "default": 1, "description": "Min value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMin", "type": "integer" }, "maxDocumentRetrievedPerPromptStep": { "default": 1, "description": "Step for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptStep", "type": "integer" }, "maxMmrLambdaMax": { "default": 1, "description": "Maximum value of mmr lambda.", "title": "maxMmrLambdaMax", "type": "number" }, "maxMmrLambdaMin": { "default": 0, "description": "Minimum value of mmr lambda.", "title": "maxMmrLambdaMin", "type": "number" }, "maxMmrLambdaStep": { "default": 0.1, "description": "Step value of mmr lambda.", "title": "maxMmrLambdaStep", "type": "number" }, "retrievalModes": { "description": "List of retriever modes to use.", "items": { "type": "string" }, "title": "retrievalModes", "type": "array" }, "retrievers": { "description": "List of retriever types to use.", "items": { "type": "string" }, "title": "retrievers", "type": "array" } }, "required": [ "retrievers", "retrievalModes" ], "title": "VectorDatabaseConfig", "type": "object" } }, "title": "SearchSpace", "type": "object" }, "useCaseId": { "description": "The ID of the use case the search study is linked to.", "title": "useCaseId", "type": "string" } }, "required": [ "useCaseId", "playgroundId", "groundingDatasetId", "evalDatasetId", "numTrials", "numConcurrentTrials", "optimizationObjectives", "searchSpace", "name" ], "title": "RunAgenticSearchRequest", "type": "object" } ``` RunAgenticSearchRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | evalDatasetId | string | true | | The ID of the evaluation dataset. | | groundingDatasetId | string | true | | The ID of the dataset the vector databases will be built from. | | name | string | true | | Name of the search study. | | numConcurrentTrials | integer | true | | The number of simultaneously running trials. | | numTrials | integer | true | | The number of search trials to sample. | | optimizationObjectives | [array] | true | | Optimization objectives of a study. | | playgroundId | string | true | | The ID of the existing playground that will be associated with the search. | | searchSpace | SearchSpace | true | | Search space for the search. | | useCaseId | string | true | | The ID of the use case the search study is linked to. | ## RunSearchApiResponse ``` { "description": "API response object for run agentic search request.", "properties": { "jobId": { "description": "The ID of the worker job.", "format": "uuid4", "title": "jobId", "type": "string" }, "searchStudyId": { "description": "The ID of the search study.", "title": "searchStudyId", "type": "string" } }, "required": [ "searchStudyId", "jobId" ], "title": "RunSearchApiResponse", "type": "object" } ``` RunSearchApiResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | jobId | string(uuid4) | true | | The ID of the worker job. | | searchStudyId | string | true | | The ID of the search study. | ## SearchDirection ``` { "description": "Whether to minimize or maximize search objective.", "enum": [ "maximize", "minimize" ], "title": "SearchDirection", "type": "string" } ``` SearchDirection ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | SearchDirection | string | false | | Whether to minimize or maximize search objective. | ### Enumerated Values | Property | Value | | --- | --- | | SearchDirection | [maximize, minimize] | ## SearchObjective ``` { "description": "List of supported search objectives.", "enum": [ "correctness", "all_tokens" ], "title": "SearchObjective", "type": "string" } ``` SearchObjective ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | SearchObjective | string | false | | List of supported search objectives. | ### Enumerated Values | Property | Value | | --- | --- | | SearchObjective | [correctness, all_tokens] | ## SearchSpace ``` { "description": "Represents full search space.", "properties": { "chunkingParameters": { "description": "Parameters of the text chunkers.", "properties": { "chunkOverlapPercentageMax": { "default": 50, "description": "Maximum value of chunk overlap.", "title": "chunkOverlapPercentageMax", "type": "number" }, "chunkOverlapPercentageMin": { "default": 0, "description": "Minimum value of chunk overlap.", "title": "chunkOverlapPercentageMin", "type": "number" }, "chunkOverlapPercentageStep": { "default": 10, "description": "Step value of chunk overlap.", "title": "chunkOverlapPercentageStep", "type": "number" }, "chunkSizeMaxExp": { "default": 8, "description": "Maximum exponent for chunk size (2^8 = 256).", "title": "chunkSizeMaxExp", "type": "integer" }, "chunkSizeMinExp": { "default": 7, "description": "Minimum exponent for chunk size (2^7 = 128).", "title": "chunkSizeMinExp", "type": "integer" }, "chunkingMethods": { "description": "List of chunking methods to use.", "items": { "type": "string" }, "title": "chunkingMethods", "type": "array" }, "embeddingModelNames": { "description": "List of embedding models to use.", "items": { "type": "string" }, "title": "embeddingModelNames", "type": "array" } }, "required": [ "embeddingModelNames" ], "title": "ChunkingParametersConfig", "type": "object" }, "llmConfig": { "description": "Configuration of llms in the search space.", "properties": { "llmNames": { "description": "List of LLM names to use.", "items": { "type": "string" }, "title": "llmNames", "type": "array" }, "temperatureMax": { "default": 1, "description": "Maximum temperature of an llm.", "title": "temperatureMax", "type": "number" }, "temperatureMin": { "default": 0, "description": "Minimum temperature of an llm.", "title": "temperatureMin", "type": "number" }, "temperatureStep": { "default": 0.05, "description": "Step size for LLM temperature.", "title": "temperatureStep", "type": "number" }, "topPMax": { "default": 1, "description": "Maximum top_p of an llm.", "title": "topPMax", "type": "number" }, "topPMin": { "default": 0, "description": "Minimum top_p of an llm.", "title": "topPMin", "type": "number" }, "topPStep": { "default": 0.05, "description": "Step size for LLM top_p.", "title": "topPStep", "type": "number" } }, "required": [ "llmNames" ], "title": "LLMConfig", "type": "object" }, "vectorDatabaseSettings": { "description": "Settings of the vector database.", "properties": { "addNeighborChunks": { "description": "Add neighboring chunks to those that the similarity search retrieves.", "items": { "type": "boolean" }, "title": "addNeighborChunks", "type": "array" }, "maxDocumentRetrievedPerPromptMax": { "default": 10, "description": "Max value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMax", "type": "integer" }, "maxDocumentRetrievedPerPromptMin": { "default": 1, "description": "Min value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMin", "type": "integer" }, "maxDocumentRetrievedPerPromptStep": { "default": 1, "description": "Step for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptStep", "type": "integer" }, "maxMmrLambdaMax": { "default": 1, "description": "Maximum value of mmr lambda.", "title": "maxMmrLambdaMax", "type": "number" }, "maxMmrLambdaMin": { "default": 0, "description": "Minimum value of mmr lambda.", "title": "maxMmrLambdaMin", "type": "number" }, "maxMmrLambdaStep": { "default": 0.1, "description": "Step value of mmr lambda.", "title": "maxMmrLambdaStep", "type": "number" }, "retrievalModes": { "description": "List of retriever modes to use.", "items": { "type": "string" }, "title": "retrievalModes", "type": "array" }, "retrievers": { "description": "List of retriever types to use.", "items": { "type": "string" }, "title": "retrievers", "type": "array" } }, "required": [ "retrievers", "retrievalModes" ], "title": "VectorDatabaseConfig", "type": "object" } }, "title": "SearchSpace", "type": "object" } ``` SearchSpace ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | chunkingParameters | ChunkingParametersConfig | false | | Chunking parameters for rag. | | llmConfig | LLMConfig | false | | LLM configuration. | | vectorDatabaseSettings | VectorDatabaseConfig | false | | Vector database settings. | ## SearchStudyResponse ``` { "description": "API response object for search study retrieval.", "properties": { "allTrials": { "anyOf": [ { "items": { "description": "Represents a search trial from history.", "properties": { "llmBlueprintId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent blueprint id.", "title": "llmBlueprintId" }, "searchParameters": { "additionalProperties": true, "description": "Search parameters of the point.", "title": "searchParameters", "type": "object" }, "values": { "description": "The resulting values of optimization objectives.", "items": { "type": "number" }, "title": "values", "type": "array" }, "vectorDatabaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The correspondent vector database id.", "title": "vectorDatabaseId" } }, "required": [ "llmBlueprintId", "vectorDatabaseId", "values", "searchParameters" ], "title": "HistoryPoint", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "Trials history.", "title": "allTrials" }, "datetimeEnd": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "Study end time.", "title": "datetimeEnd" }, "datetimeStart": { "description": "Study start time.", "format": "date-time", "title": "datetimeStart", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Error message if search study fails.", "title": "errorMessage" }, "evalDatasetId": { "description": "The ID of the evaluation dataset.", "title": "evalDatasetId", "type": "string" }, "evalDatasetName": { "description": "The name of evaluation dataset.", "title": "evalDatasetName", "type": "string" }, "evalResults": { "anyOf": [ { "items": {}, "type": "array" }, { "type": "null" } ], "description": "The results of the comparative evaluation of LLM blueprints.", "title": "evalResults" }, "existingBlueprintIds": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The ids of existing LLM blueprints for comparative evaluation.", "title": "existingBlueprintIds" }, "groundingDatasetId": { "description": "The ID of the dataset the vector databases will be built from.", "title": "groundingDatasetId", "type": "string" }, "groundingDatasetName": { "description": "The name of the grouding dataset.", "title": "groundingDatasetName", "type": "string" }, "jobId": { "anyOf": [ { "format": "uuid4", "type": "string" }, { "type": "null" } ], "description": "The ID of the worker job.", "title": "jobId" }, "name": { "description": "Name of the search study.", "title": "name", "type": "string" }, "numConcurrentTrials": { "description": "The number of simultaneously running trials.", "title": "numConcurrentTrials", "type": "integer" }, "numTrials": { "description": "The number of search trials to sample.", "title": "numTrials", "type": "integer" }, "optimizationObjectives": { "description": "Optimization objectives of a study.", "items": { "maxItems": 2, "minItems": 2, "prefixItems": [ { "description": "List of supported search objectives.", "enum": [ "correctness", "all_tokens" ], "title": "SearchObjective", "type": "string" }, { "description": "Whether to minimize or maximize search objective.", "enum": [ "maximize", "minimize" ], "title": "SearchDirection", "type": "string" } ], "type": "array" }, "title": "optimizationObjectives", "type": "array" }, "paretoFront": { "anyOf": [ { "items": {}, "type": "array" }, { "type": "null" } ], "description": "Pareto frontier of a study.", "title": "paretoFront" }, "playgroundId": { "description": "The ID of the existing playground that will be associated with the search.", "title": "playgroundId", "type": "string" }, "searchSpace": { "anyOf": [ { "description": "Represents full search space.", "properties": { "chunkingParameters": { "description": "Parameters of the text chunkers.", "properties": { "chunkOverlapPercentageMax": { "default": 50, "description": "Maximum value of chunk overlap.", "title": "chunkOverlapPercentageMax", "type": "number" }, "chunkOverlapPercentageMin": { "default": 0, "description": "Minimum value of chunk overlap.", "title": "chunkOverlapPercentageMin", "type": "number" }, "chunkOverlapPercentageStep": { "default": 10, "description": "Step value of chunk overlap.", "title": "chunkOverlapPercentageStep", "type": "number" }, "chunkSizeMaxExp": { "default": 8, "description": "Maximum exponent for chunk size (2^8 = 256).", "title": "chunkSizeMaxExp", "type": "integer" }, "chunkSizeMinExp": { "default": 7, "description": "Minimum exponent for chunk size (2^7 = 128).", "title": "chunkSizeMinExp", "type": "integer" }, "chunkingMethods": { "description": "List of chunking methods to use.", "items": { "type": "string" }, "title": "chunkingMethods", "type": "array" }, "embeddingModelNames": { "description": "List of embedding models to use.", "items": { "type": "string" }, "title": "embeddingModelNames", "type": "array" } }, "required": [ "embeddingModelNames" ], "title": "ChunkingParametersConfig", "type": "object" }, "llmConfig": { "description": "Configuration of llms in the search space.", "properties": { "llmNames": { "description": "List of LLM names to use.", "items": { "type": "string" }, "title": "llmNames", "type": "array" }, "temperatureMax": { "default": 1, "description": "Maximum temperature of an llm.", "title": "temperatureMax", "type": "number" }, "temperatureMin": { "default": 0, "description": "Minimum temperature of an llm.", "title": "temperatureMin", "type": "number" }, "temperatureStep": { "default": 0.05, "description": "Step size for LLM temperature.", "title": "temperatureStep", "type": "number" }, "topPMax": { "default": 1, "description": "Maximum top_p of an llm.", "title": "topPMax", "type": "number" }, "topPMin": { "default": 0, "description": "Minimum top_p of an llm.", "title": "topPMin", "type": "number" }, "topPStep": { "default": 0.05, "description": "Step size for LLM top_p.", "title": "topPStep", "type": "number" } }, "required": [ "llmNames" ], "title": "LLMConfig", "type": "object" }, "vectorDatabaseSettings": { "description": "Settings of the vector database.", "properties": { "addNeighborChunks": { "description": "Add neighboring chunks to those that the similarity search retrieves.", "items": { "type": "boolean" }, "title": "addNeighborChunks", "type": "array" }, "maxDocumentRetrievedPerPromptMax": { "default": 10, "description": "Max value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMax", "type": "integer" }, "maxDocumentRetrievedPerPromptMin": { "default": 1, "description": "Min value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMin", "type": "integer" }, "maxDocumentRetrievedPerPromptStep": { "default": 1, "description": "Step for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptStep", "type": "integer" }, "maxMmrLambdaMax": { "default": 1, "description": "Maximum value of mmr lambda.", "title": "maxMmrLambdaMax", "type": "number" }, "maxMmrLambdaMin": { "default": 0, "description": "Minimum value of mmr lambda.", "title": "maxMmrLambdaMin", "type": "number" }, "maxMmrLambdaStep": { "default": 0.1, "description": "Step value of mmr lambda.", "title": "maxMmrLambdaStep", "type": "number" }, "retrievalModes": { "description": "List of retriever modes to use.", "items": { "type": "string" }, "title": "retrievalModes", "type": "array" }, "retrievers": { "description": "List of retriever types to use.", "items": { "type": "string" }, "title": "retrievers", "type": "array" } }, "required": [ "retrievers", "retrievalModes" ], "title": "VectorDatabaseConfig", "type": "object" } }, "title": "SearchSpace", "type": "object" }, { "type": "null" } ], "description": "Search space for the search." }, "searchStudyId": { "description": "The ID of the search study.", "title": "searchStudyId", "type": "string" }, "studyStatus": { "description": "Represents a search study execution state.", "enum": [ "RUNNING", "COMPLETED", "STOPPED", "FAILED" ], "title": "JobStatus", "type": "string" }, "tempPlaygroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the temp playground.", "title": "tempPlaygroundId" }, "trialsFailed": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of failed trials.", "title": "trialsFailed" }, "trialsRunning": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of currently running trials.", "title": "trialsRunning" }, "trialsSuccess": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "The number of completed trials.", "title": "trialsSuccess" }, "useCaseId": { "description": "The ID of the use case the search study is linked to.", "title": "useCaseId", "type": "string" }, "userId": { "description": "The ID of the user.", "title": "userId", "type": "string" }, "userName": { "description": "The user name of the user who ran the study.", "title": "userName", "type": "string" } }, "required": [ "searchSpace", "useCaseId", "groundingDatasetId", "evalDatasetId", "groundingDatasetName", "evalDatasetName", "userId", "userName", "numTrials", "numConcurrentTrials", "optimizationObjectives", "playgroundId", "tempPlaygroundId", "paretoFront", "datetimeStart", "datetimeEnd", "studyStatus", "searchStudyId", "name", "jobId", "trialsRunning", "trialsFailed", "trialsSuccess", "allTrials", "existingBlueprintIds", "evalResults", "errorMessage" ], "title": "SearchStudyResponse", "type": "object" } ``` SearchStudyResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | allTrials | any | true | | Trials history. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | [HistoryPoint] | false | | [Represents a search trial from history.] | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | datetimeEnd | any | true | | Study end time. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string(date-time) | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | datetimeStart | string(date-time) | true | | Study start time. | | errorMessage | any | true | | Error message if search study fails. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | evalDatasetId | string | true | | The ID of the evaluation dataset. | | evalDatasetName | string | true | | The name of evaluation dataset. | | evalResults | any | true | | The results of the comparative evaluation of LLM blueprints. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | [any] | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | existingBlueprintIds | any | true | | The ids of existing LLM blueprints for comparative evaluation. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | [string] | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | groundingDatasetId | string | true | | The ID of the dataset the vector databases will be built from. | | groundingDatasetName | string | true | | The name of the grouding dataset. | | jobId | any | true | | The ID of the worker job. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string(uuid4) | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | name | string | true | | Name of the search study. | | numConcurrentTrials | integer | true | | The number of simultaneously running trials. | | numTrials | integer | true | | The number of search trials to sample. | | optimizationObjectives | [array] | true | | Optimization objectives of a study. | | paretoFront | any | true | | Pareto frontier of a study. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | [any] | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | playgroundId | string | true | | The ID of the existing playground that will be associated with the search. | | searchSpace | any | true | | Search space for the search. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | SearchSpace | false | | Represents full search space. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | searchStudyId | string | true | | The ID of the search study. | | studyStatus | JobStatus | true | | Status of a study (running, completed or failed). | | tempPlaygroundId | any | true | | The ID of the temp playground. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | trialsFailed | any | true | | The number of failed trials. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | trialsRunning | any | true | | The number of currently running trials. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | trialsSuccess | any | true | | The number of completed trials. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | useCaseId | string | true | | The ID of the use case the search study is linked to. | | userId | string | true | | The ID of the user. | | userName | string | true | | The user name of the user who ran the study. | ## TracingContextEntityType ``` { "description": "Type of an entity in context of which the agent request is performed.", "enum": [ "deployment", "use_case" ], "title": "TracingContextEntityType", "type": "string" } ``` TracingContextEntityType ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | TracingContextEntityType | string | false | | Type of an entity in context of which the agent request is performed. | ### Enumerated Values | Property | Value | | --- | --- | | TracingContextEntityType | [deployment, use_case] | ## ValidationError ``` { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" } ``` ValidationError ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | loc | [anyOf] | true | | none | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | msg | string | true | | none | | type | string | true | | none | ## VectorDatabaseConfig ``` { "description": "Settings of the vector database.", "properties": { "addNeighborChunks": { "description": "Add neighboring chunks to those that the similarity search retrieves.", "items": { "type": "boolean" }, "title": "addNeighborChunks", "type": "array" }, "maxDocumentRetrievedPerPromptMax": { "default": 10, "description": "Max value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMax", "type": "integer" }, "maxDocumentRetrievedPerPromptMin": { "default": 1, "description": "Min value for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptMin", "type": "integer" }, "maxDocumentRetrievedPerPromptStep": { "default": 1, "description": "Step for the max number of chunks to retrieve from the vector database.", "title": "maxDocumentRetrievedPerPromptStep", "type": "integer" }, "maxMmrLambdaMax": { "default": 1, "description": "Maximum value of mmr lambda.", "title": "maxMmrLambdaMax", "type": "number" }, "maxMmrLambdaMin": { "default": 0, "description": "Minimum value of mmr lambda.", "title": "maxMmrLambdaMin", "type": "number" }, "maxMmrLambdaStep": { "default": 0.1, "description": "Step value of mmr lambda.", "title": "maxMmrLambdaStep", "type": "number" }, "retrievalModes": { "description": "List of retriever modes to use.", "items": { "type": "string" }, "title": "retrievalModes", "type": "array" }, "retrievers": { "description": "List of retriever types to use.", "items": { "type": "string" }, "title": "retrievers", "type": "array" } }, "required": [ "retrievers", "retrievalModes" ], "title": "VectorDatabaseConfig", "type": "object" } ``` VectorDatabaseConfig ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | addNeighborChunks | [boolean] | false | | Add neighboring chunks to those that the similarity search retrieves. | | maxDocumentRetrievedPerPromptMax | integer | false | | Max value for the max number of chunks to retrieve from the vector database. | | maxDocumentRetrievedPerPromptMin | integer | false | | Min value for the max number of chunks to retrieve from the vector database. | | maxDocumentRetrievedPerPromptStep | integer | false | | Step for the max number of chunks to retrieve from the vector database. | | maxMmrLambdaMax | number | false | | Maximum value of mmr lambda. | | maxMmrLambdaMin | number | false | | Minimum value of mmr lambda. | | maxMmrLambdaStep | number | false | | Step value of mmr lambda. | | retrievalModes | [string] | true | | List of retriever modes to use. | | retrievers | [string] | true | | List of retriever types to use. | --- # LLM compliance tests URL: https://docs.datarobot.com/en/docs/api/reference/public-api/ai_robustness_tests.html > The following endpoints outline how to manage AI robustness tests. The following endpoints outline how to manage AI robustness tests. ## Create cost metric configuration Operation path: `POST /api/v2/genai/costMetricConfigurations/` Authentication requirements: `BearerAuth` Create a new cost metric configuration. ### Body parameter ``` { "description": "The body of the \"create cost metric configuration\" request.", "properties": { "costMetricConfigurations": { "description": "The list of cost metric configurations to use.", "items": { "description": "API request/response object for a cost configuration of a single llm.", "properties": { "currencyCode": { "default": "USD", "description": "The arbitrary code code of the currency of `inputtokenprice` and `outputtokenprice`.", "maxLength": 7, "title": "currencyCode", "type": "string" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation (if using a custom model llm).", "title": "customModelLLMValidationId" }, "inputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceinputtokencount` input tokens.", "minimum": 0, "title": "inputTokenPrice", "type": "number" }, "llmId": { "description": "The ID of the LLM associated with this cost configuration.", "title": "llmId", "type": "string" }, "outputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceoutputtokencount` output tokens.", "minimum": 0, "title": "outputTokenPrice", "type": "number" }, "referenceInputTokenCount": { "default": 1000, "description": "The number of input tokens corresponding to `inputtokenprice`.", "minimum": 0, "title": "referenceInputTokenCount", "type": "integer" }, "referenceOutputTokenCount": { "default": 1000, "description": "The number of output tokens corresponding to `outputtokenprice`.", "minimum": 0, "title": "referenceOutputTokenCount", "type": "integer" } }, "required": [ "llmId" ], "title": "LLMCostConfigurationResponse", "type": "object" }, "title": "costMetricConfigurations", "type": "array" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name to use for the cost configuration.", "title": "name" }, "playgroundId": { "description": "The ID of the playground to associate with the cost metric configuration.", "title": "playgroundId", "type": "string" }, "useCaseId": { "description": "The ID of the use case to associate with the cost metric configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "useCaseId", "playgroundId", "costMetricConfigurations" ], "title": "CreateCostMetricConfigurationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | CreateCostMetricConfigurationRequest | true | none | ### Example responses > 201 Response ``` { "description": "API response object for a single cost metric configuration.", "properties": { "costConfigurationId": { "description": "The ID of the cost metric configuration.", "title": "costConfigurationId", "type": "string" }, "costMetricConfigurations": { "description": "The list of individual LLM cost configurations that constitute this cost metric configuration.", "items": { "description": "API request/response object for a cost configuration of a single llm.", "properties": { "currencyCode": { "default": "USD", "description": "The arbitrary code code of the currency of `inputtokenprice` and `outputtokenprice`.", "maxLength": 7, "title": "currencyCode", "type": "string" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation (if using a custom model llm).", "title": "customModelLLMValidationId" }, "inputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceinputtokencount` input tokens.", "minimum": 0, "title": "inputTokenPrice", "type": "number" }, "llmId": { "description": "The ID of the LLM associated with this cost configuration.", "title": "llmId", "type": "string" }, "outputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceoutputtokencount` output tokens.", "minimum": 0, "title": "outputTokenPrice", "type": "number" }, "referenceInputTokenCount": { "default": 1000, "description": "The number of input tokens corresponding to `inputtokenprice`.", "minimum": 0, "title": "referenceInputTokenCount", "type": "integer" }, "referenceOutputTokenCount": { "default": 1000, "description": "The number of output tokens corresponding to `outputtokenprice`.", "minimum": 0, "title": "referenceOutputTokenCount", "type": "integer" } }, "required": [ "llmId" ], "title": "LLMCostConfigurationResponse", "type": "object" }, "title": "costMetricConfigurations", "type": "array" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name to use for the cost configuration.", "title": "name" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the cost metric configuration.", "title": "playgroundId" }, "useCaseId": { "description": "The ID of the use case associated with the cost metric configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "costConfigurationId", "useCaseId", "costMetricConfigurations" ], "title": "CostMetricConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 201 | Created | Cost configuration created successfully | CostMetricConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Delete cost metric configuration by cost metric configuration ID Operation path: `DELETE /api/v2/genai/costMetricConfigurations/{costMetricConfigurationId}/` Authentication requirements: `BearerAuth` Delete an existing cost metric configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | costMetricConfigurationId | path | string | true | The ID of the cost metric configuration to delete. | ### Example responses > 422 Response ``` { "properties": { "detail": { "items": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "title": "detail", "type": "array" } }, "title": "HTTPValidationErrorResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Cost metric configuration successfully deleted. | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Retrieve cost metric configuration by cost metric configuration ID Operation path: `GET /api/v2/genai/costMetricConfigurations/{costMetricConfigurationId}/` Authentication requirements: `BearerAuth` Retrieve an existing cost metric configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | costMetricConfigurationId | path | string | true | The ID of the cost metric configuration to retrieve. | ### Example responses > 200 Response ``` { "description": "API response object for a single cost metric configuration.", "properties": { "costConfigurationId": { "description": "The ID of the cost metric configuration.", "title": "costConfigurationId", "type": "string" }, "costMetricConfigurations": { "description": "The list of individual LLM cost configurations that constitute this cost metric configuration.", "items": { "description": "API request/response object for a cost configuration of a single llm.", "properties": { "currencyCode": { "default": "USD", "description": "The arbitrary code code of the currency of `inputtokenprice` and `outputtokenprice`.", "maxLength": 7, "title": "currencyCode", "type": "string" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation (if using a custom model llm).", "title": "customModelLLMValidationId" }, "inputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceinputtokencount` input tokens.", "minimum": 0, "title": "inputTokenPrice", "type": "number" }, "llmId": { "description": "The ID of the LLM associated with this cost configuration.", "title": "llmId", "type": "string" }, "outputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceoutputtokencount` output tokens.", "minimum": 0, "title": "outputTokenPrice", "type": "number" }, "referenceInputTokenCount": { "default": 1000, "description": "The number of input tokens corresponding to `inputtokenprice`.", "minimum": 0, "title": "referenceInputTokenCount", "type": "integer" }, "referenceOutputTokenCount": { "default": 1000, "description": "The number of output tokens corresponding to `outputtokenprice`.", "minimum": 0, "title": "referenceOutputTokenCount", "type": "integer" } }, "required": [ "llmId" ], "title": "LLMCostConfigurationResponse", "type": "object" }, "title": "costMetricConfigurations", "type": "array" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name to use for the cost configuration.", "title": "name" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the cost metric configuration.", "title": "playgroundId" }, "useCaseId": { "description": "The ID of the use case associated with the cost metric configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "costConfigurationId", "useCaseId", "costMetricConfigurations" ], "title": "CostMetricConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Cost metric configuration successfully retrieved. | CostMetricConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Edit cost metric configuration by cost metric configuration ID Operation path: `PATCH /api/v2/genai/costMetricConfigurations/{costMetricConfigurationId}/` Authentication requirements: `BearerAuth` Edit an existing cost metric configuration. ### Body parameter ``` { "description": "The body of the \"edit cost metric configuration\" request.", "properties": { "costMetricConfigurations": { "description": "The list of LLM cost configurations to apply to this cost metric configuration.", "items": { "description": "API request/response object for a cost configuration of a single llm.", "properties": { "currencyCode": { "default": "USD", "description": "The arbitrary code code of the currency of `inputtokenprice` and `outputtokenprice`.", "maxLength": 7, "title": "currencyCode", "type": "string" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation (if using a custom model llm).", "title": "customModelLLMValidationId" }, "inputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceinputtokencount` input tokens.", "minimum": 0, "title": "inputTokenPrice", "type": "number" }, "llmId": { "description": "The ID of the LLM associated with this cost configuration.", "title": "llmId", "type": "string" }, "outputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceoutputtokencount` output tokens.", "minimum": 0, "title": "outputTokenPrice", "type": "number" }, "referenceInputTokenCount": { "default": 1000, "description": "The number of input tokens corresponding to `inputtokenprice`.", "minimum": 0, "title": "referenceInputTokenCount", "type": "integer" }, "referenceOutputTokenCount": { "default": 1000, "description": "The number of output tokens corresponding to `outputtokenprice`.", "minimum": 0, "title": "referenceOutputTokenCount", "type": "integer" } }, "required": [ "llmId" ], "title": "LLMCostConfigurationResponse", "type": "object" }, "minItems": 1, "title": "costMetricConfigurations", "type": "array" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name to use for the cost configuration.", "title": "name" } }, "required": [ "costMetricConfigurations" ], "title": "EditCostMetricConfigurationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | costMetricConfigurationId | path | string | true | The ID of the cost metric configuration to edit. | | body | body | EditCostMetricConfigurationRequest | true | none | ### Example responses > 200 Response ``` { "description": "API response object for a single cost metric configuration.", "properties": { "costConfigurationId": { "description": "The ID of the cost metric configuration.", "title": "costConfigurationId", "type": "string" }, "costMetricConfigurations": { "description": "The list of individual LLM cost configurations that constitute this cost metric configuration.", "items": { "description": "API request/response object for a cost configuration of a single llm.", "properties": { "currencyCode": { "default": "USD", "description": "The arbitrary code code of the currency of `inputtokenprice` and `outputtokenprice`.", "maxLength": 7, "title": "currencyCode", "type": "string" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation (if using a custom model llm).", "title": "customModelLLMValidationId" }, "inputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceinputtokencount` input tokens.", "minimum": 0, "title": "inputTokenPrice", "type": "number" }, "llmId": { "description": "The ID of the LLM associated with this cost configuration.", "title": "llmId", "type": "string" }, "outputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceoutputtokencount` output tokens.", "minimum": 0, "title": "outputTokenPrice", "type": "number" }, "referenceInputTokenCount": { "default": 1000, "description": "The number of input tokens corresponding to `inputtokenprice`.", "minimum": 0, "title": "referenceInputTokenCount", "type": "integer" }, "referenceOutputTokenCount": { "default": 1000, "description": "The number of output tokens corresponding to `outputtokenprice`.", "minimum": 0, "title": "referenceOutputTokenCount", "type": "integer" } }, "required": [ "llmId" ], "title": "LLMCostConfigurationResponse", "type": "object" }, "title": "costMetricConfigurations", "type": "array" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name to use for the cost configuration.", "title": "name" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the cost metric configuration.", "title": "playgroundId" }, "useCaseId": { "description": "The ID of the use case associated with the cost metric configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "costConfigurationId", "useCaseId", "costMetricConfigurations" ], "title": "CostMetricConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Cost metric configuration successfully updated. | CostMetricConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List evaluation dataset configurations Operation path: `GET /api/v2/genai/evaluationDatasetConfigurations/` Authentication requirements: `BearerAuth` List evaluation dataset configurations. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | useCaseId | query | string | true | Only retrieve the evaluation dataset configurations associated with this use case ID. | | playgroundId | query | string | true | Only retrieve the evaluation dataset configuration associated with this playground ID. | | evaluationDatasetConfigurationId | query | any | false | Only retrieve the evaluation dataset configuration with this ID. | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | search | query | any | false | Only retrieve the evaluation dataset configurations matching the search query. | | sort | query | any | false | Apply this sort order to the results. Valid options are "name", "creationUserId", "creationDate", "datasetId", "userName", "datasetName", "promptColumnName", "responseColumnName". Prefix the attribute name with a dash to sort in descending order, e.g., sort=-creationDate. | | correctnessEnabledOnly | query | boolean | false | If true, only retrieve the evaluation dataset configurations with correctness enabled. The default is false. | | completedOnly | query | boolean | false | If true, only retrieve the evaluation dataset configurations where the evaluation dataset is in the completed status. The default is false. | ### Example responses > 200 Response ``` { "description": "Paginated list of evaludation dataset configurations.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "API response object for a single evaluation dataset configuration.", "properties": { "agentGoalsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected agent goals (for agentic workflows).", "title": "agentGoalsColumnName" }, "correctnessEnabled": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "deprecated": true, "description": "Whether correctness is enabled for the evaluation dataset configuration.", "title": "correctnessEnabled" }, "creationDate": { "description": "The creation date of the evaluation dataset configuration (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the evaluation dataset configuration.", "title": "creationUserId", "type": "string" }, "datasetId": { "description": "The ID of the evaluation dataset.", "title": "datasetId", "type": "string" }, "datasetName": { "description": "The name of the evaluation dataset.", "title": "datasetName", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration.", "title": "errorMessage" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "id": { "description": "The ID of the evaluation dataset configuration.", "title": "id", "type": "string" }, "name": { "description": "The name of the evaluation dataset configuration.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the evaluation dataset configuration.", "title": "playgroundId" }, "promptColumnName": { "description": "The name of the dataset column containing the prompt text.", "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing the response text.", "title": "responseColumnName" }, "rowsCount": { "description": "The rows count of the evaluation dataset.", "title": "rowsCount", "type": "integer" }, "size": { "description": "The size of the evaluation dataset (in bytes).", "title": "size", "type": "integer" }, "tenantId": { "description": "The ID of the datarobot tenant this evaluation dataset configuration belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "toolCallsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected tool calls (for agentic workflows).", "title": "toolCallsColumnName" }, "useCaseId": { "description": "The ID of the use case associated with the evaluation dataset configuration.", "title": "useCaseId", "type": "string" }, "userName": { "description": "The name of the user that created the evaluation dataset configuration.", "title": "userName", "type": "string" } }, "required": [ "id", "name", "size", "rowsCount", "useCaseId", "playgroundId", "datasetId", "datasetName", "promptColumnName", "responseColumnName", "toolCallsColumnName", "agentGoalsColumnName", "userName", "correctnessEnabled", "creationUserId", "creationDate", "tenantId", "executionStatus" ], "title": "EvaluationDatasetConfigurationResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListEvaluationDatasetConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Evaluation dataset configurations successfully retrieved. | ListEvaluationDatasetConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Create evaluation dataset configuration Operation path: `POST /api/v2/genai/evaluationDatasetConfigurations/` Authentication requirements: `BearerAuth` Create a new evaluation dataset configuration. ### Body parameter ``` { "description": "The body of the \"create evaluation dataset configuration\" request.", "properties": { "agentGoalsColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected agent goals. it is required to evaluate the agentgoalaccuracywithreference metric for agentic workflows.", "title": "agentGoalsColumnName" }, "correctnessEnabled": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "deprecated": true, "description": "Whether correctness is enabled for the evaluation dataset configuration.", "title": "correctnessEnabled" }, "datasetId": { "description": "The ID of the evaluation dataset.", "title": "datasetId", "type": "string" }, "isSyntheticDataset": { "default": false, "description": "Whether the evaluation dataset is synthetic.", "title": "isSyntheticDataset", "type": "boolean" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name of the evaluation dataset configuration.", "title": "name" }, "playgroundId": { "description": "The ID of the playground to associate with the evaluation dataset configuration.", "title": "playgroundId", "type": "string" }, "promptColumnName": { "description": "The name of the dataset column containing the prompt text.", "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing the response text.", "title": "responseColumnName" }, "toolCallsColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected tool calls. it is required to evaluate the toolcallaccuracy metric for agentic workflows.", "title": "toolCallsColumnName" }, "useCaseId": { "description": "The ID of the use case to associate with the evaluation dataset configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "useCaseId", "playgroundId", "datasetId", "promptColumnName" ], "title": "CreateEvaluationDatasetConfigurationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | CreateEvaluationDatasetConfigurationRequest | true | none | ### Example responses > 201 Response ``` { "description": "API response object for a single evaluation dataset configuration.", "properties": { "agentGoalsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected agent goals (for agentic workflows).", "title": "agentGoalsColumnName" }, "correctnessEnabled": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "deprecated": true, "description": "Whether correctness is enabled for the evaluation dataset configuration.", "title": "correctnessEnabled" }, "creationDate": { "description": "The creation date of the evaluation dataset configuration (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the evaluation dataset configuration.", "title": "creationUserId", "type": "string" }, "datasetId": { "description": "The ID of the evaluation dataset.", "title": "datasetId", "type": "string" }, "datasetName": { "description": "The name of the evaluation dataset.", "title": "datasetName", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration.", "title": "errorMessage" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "id": { "description": "The ID of the evaluation dataset configuration.", "title": "id", "type": "string" }, "name": { "description": "The name of the evaluation dataset configuration.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the evaluation dataset configuration.", "title": "playgroundId" }, "promptColumnName": { "description": "The name of the dataset column containing the prompt text.", "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing the response text.", "title": "responseColumnName" }, "rowsCount": { "description": "The rows count of the evaluation dataset.", "title": "rowsCount", "type": "integer" }, "size": { "description": "The size of the evaluation dataset (in bytes).", "title": "size", "type": "integer" }, "tenantId": { "description": "The ID of the datarobot tenant this evaluation dataset configuration belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "toolCallsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected tool calls (for agentic workflows).", "title": "toolCallsColumnName" }, "useCaseId": { "description": "The ID of the use case associated with the evaluation dataset configuration.", "title": "useCaseId", "type": "string" }, "userName": { "description": "The name of the user that created the evaluation dataset configuration.", "title": "userName", "type": "string" } }, "required": [ "id", "name", "size", "rowsCount", "useCaseId", "playgroundId", "datasetId", "datasetName", "promptColumnName", "responseColumnName", "toolCallsColumnName", "agentGoalsColumnName", "userName", "correctnessEnabled", "creationUserId", "creationDate", "tenantId", "executionStatus" ], "title": "EvaluationDatasetConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 201 | Created | Evaluation dataset configuration successfully created | EvaluationDatasetConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Delete evaluation dataset configuration by evaluation dataset configuration ID Operation path: `DELETE /api/v2/genai/evaluationDatasetConfigurations/{evaluationDatasetConfigurationId}/` Authentication requirements: `BearerAuth` Delete an existing evaluation dataset configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | evaluationDatasetConfigurationId | path | string | true | The ID of the evaluation dataset configuration to delete. | ### Example responses > 422 Response ``` { "properties": { "detail": { "items": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "title": "detail", "type": "array" } }, "title": "HTTPValidationErrorResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Evaluation dataset configuration successfully deleted. | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Retrieve evaluation dataset configuration by evaluation dataset configuration ID Operation path: `GET /api/v2/genai/evaluationDatasetConfigurations/{evaluationDatasetConfigurationId}/` Authentication requirements: `BearerAuth` Retrieve an existing evaluation dataset configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | evaluationDatasetConfigurationId | path | string | true | The ID of the evaluation dataset configuration to retrieve. | ### Example responses > 200 Response ``` { "description": "API response object for a single evaluation dataset configuration.", "properties": { "agentGoalsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected agent goals (for agentic workflows).", "title": "agentGoalsColumnName" }, "correctnessEnabled": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "deprecated": true, "description": "Whether correctness is enabled for the evaluation dataset configuration.", "title": "correctnessEnabled" }, "creationDate": { "description": "The creation date of the evaluation dataset configuration (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the evaluation dataset configuration.", "title": "creationUserId", "type": "string" }, "datasetId": { "description": "The ID of the evaluation dataset.", "title": "datasetId", "type": "string" }, "datasetName": { "description": "The name of the evaluation dataset.", "title": "datasetName", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration.", "title": "errorMessage" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "id": { "description": "The ID of the evaluation dataset configuration.", "title": "id", "type": "string" }, "name": { "description": "The name of the evaluation dataset configuration.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the evaluation dataset configuration.", "title": "playgroundId" }, "promptColumnName": { "description": "The name of the dataset column containing the prompt text.", "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing the response text.", "title": "responseColumnName" }, "rowsCount": { "description": "The rows count of the evaluation dataset.", "title": "rowsCount", "type": "integer" }, "size": { "description": "The size of the evaluation dataset (in bytes).", "title": "size", "type": "integer" }, "tenantId": { "description": "The ID of the datarobot tenant this evaluation dataset configuration belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "toolCallsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected tool calls (for agentic workflows).", "title": "toolCallsColumnName" }, "useCaseId": { "description": "The ID of the use case associated with the evaluation dataset configuration.", "title": "useCaseId", "type": "string" }, "userName": { "description": "The name of the user that created the evaluation dataset configuration.", "title": "userName", "type": "string" } }, "required": [ "id", "name", "size", "rowsCount", "useCaseId", "playgroundId", "datasetId", "datasetName", "promptColumnName", "responseColumnName", "toolCallsColumnName", "agentGoalsColumnName", "userName", "correctnessEnabled", "creationUserId", "creationDate", "tenantId", "executionStatus" ], "title": "EvaluationDatasetConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Evaluation dataset configuration successfully retrieved. | EvaluationDatasetConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Edit evaluation dataset configuration by evaluation dataset configuration ID Operation path: `PATCH /api/v2/genai/evaluationDatasetConfigurations/{evaluationDatasetConfigurationId}/` Authentication requirements: `BearerAuth` Edit an existing evaluation dataset configuration. ### Body parameter ``` { "description": "The body of the \"edit evaluation dataset configuration\" request.", "properties": { "agentGoalsColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "If specified, changes the expected name of the dataset column containing expected agent goals. it is required to evaluate the agentgoalaccuracywithreference metric for agentic workflows.", "title": "agentGoalsColumnName" }, "correctnessEnabled": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "deprecated": true, "description": "If specified, enables or disables correctness for the evaluation dataset configuration.", "title": "correctnessEnabled" }, "datasetId": { "default": "000000000000000000000000", "description": "If specified, updates the ID of the evaluation dataset.", "title": "datasetId", "type": "string" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "If specified, renames the evaluation dataset configuration to this value.", "title": "name" }, "promptColumnName": { "default": "None", "description": "If specified, changes the expected name of the dataset column containing the prompt text.", "maxLength": 5000, "minLength": 1, "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "If specified, changes the expected name of the dataset column containing the response text.", "title": "responseColumnName" }, "toolCallsColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "If specified, changes the expected name of the dataset column containing expected tool calls. it is required to evaluate the toolcallaccuracy metric for agentic workflows.", "title": "toolCallsColumnName" } }, "title": "EditEvaluationDatasetConfigurationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | evaluationDatasetConfigurationId | path | string | true | The ID of the evaluation dataset configuration to edit. | | body | body | EditEvaluationDatasetConfigurationRequest | true | none | ### Example responses > 200 Response ``` { "description": "API response object for a single evaluation dataset configuration.", "properties": { "agentGoalsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected agent goals (for agentic workflows).", "title": "agentGoalsColumnName" }, "correctnessEnabled": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "deprecated": true, "description": "Whether correctness is enabled for the evaluation dataset configuration.", "title": "correctnessEnabled" }, "creationDate": { "description": "The creation date of the evaluation dataset configuration (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the evaluation dataset configuration.", "title": "creationUserId", "type": "string" }, "datasetId": { "description": "The ID of the evaluation dataset.", "title": "datasetId", "type": "string" }, "datasetName": { "description": "The name of the evaluation dataset.", "title": "datasetName", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration.", "title": "errorMessage" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "id": { "description": "The ID of the evaluation dataset configuration.", "title": "id", "type": "string" }, "name": { "description": "The name of the evaluation dataset configuration.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the evaluation dataset configuration.", "title": "playgroundId" }, "promptColumnName": { "description": "The name of the dataset column containing the prompt text.", "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing the response text.", "title": "responseColumnName" }, "rowsCount": { "description": "The rows count of the evaluation dataset.", "title": "rowsCount", "type": "integer" }, "size": { "description": "The size of the evaluation dataset (in bytes).", "title": "size", "type": "integer" }, "tenantId": { "description": "The ID of the datarobot tenant this evaluation dataset configuration belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "toolCallsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected tool calls (for agentic workflows).", "title": "toolCallsColumnName" }, "useCaseId": { "description": "The ID of the use case associated with the evaluation dataset configuration.", "title": "useCaseId", "type": "string" }, "userName": { "description": "The name of the user that created the evaluation dataset configuration.", "title": "userName", "type": "string" } }, "required": [ "id", "name", "size", "rowsCount", "useCaseId", "playgroundId", "datasetId", "datasetName", "promptColumnName", "responseColumnName", "toolCallsColumnName", "agentGoalsColumnName", "userName", "correctnessEnabled", "creationUserId", "creationDate", "tenantId", "executionStatus" ], "title": "EvaluationDatasetConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Evaluation dataset configuration successfully updated. | EvaluationDatasetConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Delete evaluation dataset metric aggregation Operation path: `DELETE /api/v2/genai/evaluationDatasetMetricAggregations/` Authentication requirements: `BearerAuth` Delete the evaluation dataset metric aggregation associated with the specified LLM blueprint IDs and/or chat IDs. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmBlueprintIds | query | any | false | The IDs of the LLM blueprints to delete the associated evaluation dataset metric aggregation for. If both llmBlueprintIds and chatIds are specified, will delete the aggregation record only if it matches both criteria. | | chatIds | query | any | false | The IDs of the chats to delete the associated evaluation dataset metric aggregation for. If both llmBlueprintIds and chatIds are specified, will delete the aggregation record only if it matches both criteria. | ### Example responses > 422 Response ``` { "properties": { "detail": { "items": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "title": "detail", "type": "array" } }, "title": "HTTPValidationErrorResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Evaluation dataset metric aggregation successfully deleted. | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List evaluation dataset metric aggregations Operation path: `GET /api/v2/genai/evaluationDatasetMetricAggregations/` Authentication requirements: `BearerAuth` List evaluation dataset metric aggregations. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmBlueprintIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these LLM blueprint IDs. | | chatIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these chat IDs. | | evaluationDatasetConfigurationIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these evaluation dataset configuration IDs. | | metricNames | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these metric names. | | aggregationTypes | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these aggregation types. | | currentConfigurationOnly | query | boolean | false | Only retrieve the evaluation dataset metric aggregations associated with the current configuration of the llmblueprints. | | sort | query | any | false | Apply this sort order to the results. Valid options are "name", "creationUserId", "creationDate", "datasetId", "userName", "datasetName", "promptColumnName", "responseColumnName". Prefix the attribute name with a dash to sort in descending order, e.g., sort=-creationDate. | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | nonErroredOnly | query | boolean | false | If true, only retrieve the evaluation dataset metric aggregations that are in a non-errored status. The default is false. | ### Example responses > 200 Response ``` { "description": "Paginated list of evaluation dataset metric aggregations.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "API response object for a single evaluation dataset metric aggregation.", "properties": { "aggregationType": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "aggregationValue": { "anyOf": [ { "type": "number" }, { "items": { "description": "An individual record in an itemized metric aggregation.", "properties": { "item": { "description": "The name of the item.", "title": "item", "type": "string" }, "value": { "description": "The value associated with the item.", "title": "value", "type": "number" } }, "required": [ "item", "value" ], "title": "AggregationValue", "type": "object" }, "type": "array" }, { "items": { "description": "Aggregated record of multiple of the same item across different metric aggregation runs.", "properties": { "count": { "description": "The number of metric aggregation items aggregated.", "title": "count", "type": "integer" }, "item": { "description": "The name of the item.", "title": "item", "type": "string" }, "value": { "description": "The value associated with the item.", "title": "value", "type": "number" } }, "required": [ "item", "value", "count" ], "title": "AggregatedAggregationValue", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "The aggregated value of the metric.", "title": "aggregationValue" }, "chatId": { "description": "The ID of the chat associated with the metric aggregation.", "title": "chatId", "type": "string" }, "chatLink": { "description": "The link to the chat associated with the metric aggregation.", "title": "chatLink", "type": "string" }, "chatName": { "description": "The name of the chat associated with the metric aggregation.", "title": "chatName", "type": "string" }, "creationDate": { "description": "The creation date of the metric aggregation (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the metric aggregation.", "title": "creationUserId", "type": "string" }, "creationUserName": { "description": "The name of the user that created the metric aggregation.", "title": "creationUserName", "type": "string" }, "customModelGuardId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model's guard the metric aggregation belongs to.", "title": "customModelGuardId" }, "datasetId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The dataset ID of the evaluation dataset configuration.", "title": "datasetId" }, "datasetName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The data registry dataset name of the evaluation dataset configuration.", "title": "datasetName" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration associated with the metric aggregation.", "title": "evaluationDatasetConfigurationId" }, "llmBlueprintId": { "description": "The ID of the LLM blueprint associated with the metric aggregation.", "title": "llmBlueprintId", "type": "string" }, "metricName": { "description": "The name of the metric associated with the metric aggregation.", "title": "metricName", "type": "string" }, "ootbDatasetName": { "anyOf": [ { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, { "type": "null" } ], "description": "The name of the out-of-the-box dataset." }, "tenantId": { "description": "The ID of the tenant the metric aggregation belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" } }, "required": [ "chatId", "chatName", "chatLink", "creationDate", "creationUserId", "creationUserName", "llmBlueprintId", "evaluationDatasetConfigurationId", "ootbDatasetName", "datasetId", "datasetName", "metricName", "aggregationValue", "aggregationType", "tenantId", "customModelGuardId" ], "title": "EvaluationDatasetMetricAggregationResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListEvaluationDatasetMetricAggregationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Evaluation dataset metric aggregations successfully retrieved. | ListEvaluationDatasetMetricAggregationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Create evaluation dataset metric aggregation Operation path: `POST /api/v2/genai/evaluationDatasetMetricAggregations/` Authentication requirements: `BearerAuth` Create a new evaluation dataset metric aggregation. ### Body parameter ``` { "description": "The body of the \"create evaluation dataset metric aggregation\" request.", "properties": { "chatName": { "default": "Aggregated chat", "description": "The name for the new chat that will contain the associated prompts and responses.", "maxLength": 5000, "title": "chatName", "type": "string" }, "evaluationDatasetConfigurationId": { "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId", "type": "string" }, "insightsConfiguration": { "description": "The configuration of insights for the metric aggregation.", "items": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "minItems": 1, "title": "insightsConfiguration", "type": "array" }, "llmBlueprintIds": { "description": "The ids of the LLM blueprints to use for the metric aggregation.", "items": { "type": "string" }, "maxItems": 3, "minItems": 1, "title": "llmBlueprintIds", "type": "array" } }, "required": [ "llmBlueprintIds", "evaluationDatasetConfigurationId", "insightsConfiguration" ], "title": "CreateEvaluationDatasetMetricAggregationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | CreateEvaluationDatasetMetricAggregationRequest | true | none | ### Example responses > 202 Response ``` { "description": "The body of the \"create evaluation dataset metric aggregation\" response.", "properties": { "chatIds": { "description": "The ids of the chats associated with the metric aggregation.", "items": { "type": "string" }, "title": "chatIds", "type": "array" }, "jobId": { "description": "The ID of the evaluation dataset metric aggregation job.", "format": "uuid4", "title": "jobId", "type": "string" } }, "required": [ "jobId", "chatIds" ], "title": "CreateEvaluationDatasetMetricAggregationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 202 | Accepted | Evaluation dataset metric aggregation job successfully accepted. Follow the Location header to poll for job execution status. | CreateEvaluationDatasetMetricAggregationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List evaluation dataset metric aggregations aggregated by LLM blueprint Operation path: `GET /api/v2/genai/evaluationDatasetMetricAggregations/aggregateByLLMBlueprint/` Authentication requirements: `BearerAuth` List evaluation dataset metric aggregations aggregated by llm blueprint. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmBlueprintIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these LLM blueprint IDs. | | chatIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these chat IDs. | | evaluationDatasetConfigurationIds | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these evaluation dataset configuration IDs. | | metricNames | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these metric names. | | aggregationTypes | query | any | false | Only retrieve the evaluation dataset metric aggregations associated with these aggregation types. | | currentConfigurationOnly | query | boolean | false | Only retrieve the evaluation dataset metric aggregations associated with the current configuration of the llmblueprints. | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | nonErroredOnly | query | boolean | false | If true, only retrieve the evaluation dataset metric aggregations that are in a non-errored status. The default is false. | ### Example responses > 200 Response ``` { "description": "Paginated list of evaluation dataset metric aggregations, aggregated by LLM blueprint.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "API response object for multiple evaluation dataset metric aggregation aggregated by LLM blueprint.", "properties": { "aggregatedItemCount": { "description": "Number of items aggregated.", "title": "aggregatedItemCount", "type": "integer" }, "aggregatedItemDetails": { "description": "List of details for aggregated items.", "items": { "description": "Details for aggregated items.", "properties": { "chatId": { "description": "The ID of the chat associated with the metric aggregation.", "title": "chatId", "type": "string" }, "chatLink": { "description": "The link to the chat associated with the metric aggregation.", "title": "chatLink", "type": "string" }, "chatName": { "description": "The name of the chat associated with the metric aggregation.", "title": "chatName", "type": "string" }, "creationDate": { "description": "The creation date of the metric aggregation (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the metric aggregation.", "title": "creationUserId", "type": "string" }, "creationUserName": { "description": "The name of the user that created the metric aggregation.", "title": "creationUserName", "type": "string" } }, "required": [ "chatId", "chatName", "chatLink", "creationDate", "creationUserId", "creationUserName" ], "title": "EvaluationDatasetMetricAggregationChatDetails", "type": "object" }, "title": "aggregatedItemDetails", "type": "array" }, "aggregationType": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "aggregationValue": { "anyOf": [ { "type": "number" }, { "items": { "description": "An individual record in an itemized metric aggregation.", "properties": { "item": { "description": "The name of the item.", "title": "item", "type": "string" }, "value": { "description": "The value associated with the item.", "title": "value", "type": "number" } }, "required": [ "item", "value" ], "title": "AggregationValue", "type": "object" }, "type": "array" }, { "items": { "description": "Aggregated record of multiple of the same item across different metric aggregation runs.", "properties": { "count": { "description": "The number of metric aggregation items aggregated.", "title": "count", "type": "integer" }, "item": { "description": "The name of the item.", "title": "item", "type": "string" }, "value": { "description": "The value associated with the item.", "title": "value", "type": "number" } }, "required": [ "item", "value", "count" ], "title": "AggregatedAggregationValue", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "The aggregated value of the metric.", "title": "aggregationValue" }, "customModelGuardId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model's guard the metric aggregation belongs to.", "title": "customModelGuardId" }, "datasetId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The dataset ID of the evaluation dataset configuration.", "title": "datasetId" }, "datasetName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The data registry dataset name of the evaluation dataset configuration.", "title": "datasetName" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration associated with the metric aggregation.", "title": "evaluationDatasetConfigurationId" }, "llmBlueprintId": { "description": "The ID of the LLM blueprint associated with the metric aggregation.", "title": "llmBlueprintId", "type": "string" }, "metricName": { "description": "The name of the metric associated with the metric aggregation.", "title": "metricName", "type": "string" }, "ootbDatasetName": { "anyOf": [ { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, { "type": "null" } ], "description": "The name of the out-of-the-box dataset." }, "tenantId": { "description": "The ID of the tenant the metric aggregation belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" } }, "required": [ "llmBlueprintId", "evaluationDatasetConfigurationId", "ootbDatasetName", "datasetId", "datasetName", "metricName", "aggregationValue", "aggregationType", "tenantId", "customModelGuardId", "aggregatedItemDetails", "aggregatedItemCount" ], "title": "EvaluationDatasetMetricAggregationAggregatedByLLMBlueprintResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListEvaluationDatasetMetricAggregationAggregatedByLLMBlueprintResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Evaluation dataset metric aggregations aggregated by llm blueprint successfully retrieved. | ListEvaluationDatasetMetricAggregationAggregatedByLLMBlueprintResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List evaluation dataset metric aggregations unique computed metrics by uniquefield Operation path: `GET /api/v2/genai/evaluationDatasetMetricAggregations/uniqueFieldValues/{uniqueField}/` Authentication requirements: `BearerAuth` List evaluation dataset metric aggregations unique computed metrics. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | uniqueField | path | EvaluationDatasetMetricAggregationFieldQueryParam | true | Retrieve the list of this unique field. | | llmBlueprintIds | query | any | false | Only retrieve the list of the unique field associated with these LLM blueprint IDs. | | metricNames | query | any | false | Only retrieve the list of the unique field associated with these metric names. | | chatIds | query | any | false | Only retrieve the list of the unique field associated with these chat IDs. | | evaluationDatasetConfigurationIds | query | any | false | Only retrieve the list of the unique field associated with these evaluation dataset configuration IDs. | | aggregationTypes | query | any | false | Only retrieve the list of the unique field associated with these aggregation types. | | currentConfigurationOnly | query | boolean | false | Only retrieve the evaluation dataset metric aggregations associated with the current configuration of the llmblueprints. | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | nonErroredOnly | query | boolean | false | If true, only retrieve the list of the unique field for aggregation records that are in a non-errored status. The default is false. | ### Enumerated Values | Parameter | Value | | --- | --- | | uniqueField | [metricName, llmBlueprintId, aggregationType, evaluationDatasetConfigurationId] | ### Example responses > 200 Response ``` { "description": "Paginated list of evaluation dataset metric aggregations with unique computed metrics.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "API response object for a single unique computed metric.", "properties": { "uniqueFieldValue": { "description": "The unique value associated with the metric aggregation.", "title": "uniqueFieldValue", "type": "string" } }, "required": [ "uniqueFieldValue" ], "title": "EvaluationDatasetMetricAggregationUniqueFieldValuesResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListEvaluationDatasetMetricAggregationUniqueFieldValuesResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Evaluation dataset metric aggregations unique computed metrics successfully retrieved. | ListEvaluationDatasetMetricAggregationUniqueFieldValuesResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List LLM test configuration Operation path: `GET /api/v2/genai/llmTestConfigurations/` Authentication requirements: `BearerAuth` List LLM test configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | useCaseId | query | any | false | Use Case ID. | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | testConfigType | query | LLMTestConfigurationType | false | Whether to return out-of-the-box (ootb) or custom LLM test configurations in the response. | ### Enumerated Values | Parameter | Value | | --- | --- | | testConfigType | [ootb, custom] | ### Example responses > 200 Response ``` { "description": "Paginated list of LLM test configurations.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "API response object for a single llmtestconfiguration.", "properties": { "creationDate": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The creation date of the LLM test configuration. for ootb LLM test configurations this is null.", "title": "creationDate" }, "creationUserId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the user who created the LLM test configuration. for ootb LLM test configurations this is null.", "title": "creationUserId" }, "datasetEvaluations": { "description": "The LLM test dataset evaluations.", "items": { "description": "Dataset evaluation.", "properties": { "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the dataset evaluation.", "title": "errorMessage" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration for this dataset evaluation.", "title": "evaluationDatasetConfigurationId" }, "evaluationDatasetName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset name.", "title": "evaluationDatasetName" }, "evaluationName": { "description": "The name of the evaluation. this name should provide context regarding what is being evaluated.", "maxLength": 5000, "minLength": 1, "title": "evaluationName", "type": "string" }, "insightConfiguration": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "maxNumPrompts": { "default": 100, "description": "The max number of prompts to evaluate.", "exclusiveMinimum": 0, "maximum": 5000, "title": "maxNumPrompts", "type": "integer" }, "ootbDataset": { "anyOf": [ { "description": "Out-of-the-box dataset.", "properties": { "datasetName": { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, "datasetUrl": { "anyOf": [ { "description": "Out-of-the-box dataset url.", "enum": [ "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-age-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-gender-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-race-ethnicity-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-religion-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-disability-status-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-sexual-orientation-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-nationality-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-ses-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-parent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-great-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/pii-v1.csv" ], "title": "OOTBDatasetUrl", "type": "string" }, { "type": "null" } ], "description": "The public url of the evaluation dataset. this applies only to our predefined public evaluation datasets." }, "promptColumnName": { "description": "The name of the prompt column.", "maxLength": 5000, "minLength": 1, "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the response column, if present.", "title": "responseColumnName" }, "rowsCount": { "description": "The number rows in the dataset.", "title": "rowsCount", "type": "integer" }, "warning": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Warning about the content of the dataset.", "title": "warning" } }, "required": [ "datasetName", "datasetUrl", "promptColumnName", "responseColumnName", "rowsCount" ], "title": "OOTBDataset", "type": "object" }, { "type": "null" } ], "description": "Out-of-the-box evaluation dataset. this applies only to our predefined public evaluation datasets." }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "evaluationName", "insightConfiguration", "insightGradingCriteria", "evaluationDatasetName" ], "title": "DatasetEvaluationResponse", "type": "object" }, "title": "datasetEvaluations", "type": "array" }, "description": { "description": "The description of the LLM test configuration.", "title": "description", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the LLM test configuration.", "title": "errorMessage" }, "id": { "description": "The ID of the LLM test configuration.", "title": "id", "type": "string" }, "isOutOfTheBoxTestConfiguration": { "description": "Identifies the LLM test configuration as an out-of-the-box (ootb) test configuration.", "title": "isOutOfTheBoxTestConfiguration", "type": "boolean" }, "lastUpdateDate": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The last update date of the LLM test configuration. for ootb LLM test configurations this is null.", "title": "lastUpdateDate" }, "lastUpdateUserId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the user who last updated the LLM test configuration. for ootb LLM test configurations this is null.", "title": "lastUpdateUserId" }, "llmTestGradingCriteria": { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, "name": { "description": "The name of the LLM test configuration.", "title": "name", "type": "string" }, "useCaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "If specified, the use case ID associated with the LLM test configuration.", "title": "useCaseId" }, "warnings": { "description": "Warnings for this LLM test configuration.", "items": { "additionalProperties": { "type": "string" }, "propertyNames": { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, "type": "object" }, "title": "warnings", "type": "array" } }, "required": [ "id", "name", "description", "datasetEvaluations", "llmTestGradingCriteria", "isOutOfTheBoxTestConfiguration", "warnings" ], "title": "LLMTestConfigurationResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListLLMTestConfigurationsResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | ListLLMTestConfigurationsResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Create LLM test configuration Operation path: `POST /api/v2/genai/llmTestConfigurations/` Authentication requirements: `BearerAuth` Create a new LLM test configuration. ### Body parameter ``` { "description": "Request object for creating a llmtestconfiguration.", "properties": { "datasetEvaluations": { "description": "Dataset evaluations.", "items": { "description": "Dataset evaluation.", "properties": { "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration for this dataset evaluation.", "title": "evaluationDatasetConfigurationId" }, "evaluationName": { "description": "The name of the evaluation. this name should provide context regarding what is being evaluated.", "maxLength": 5000, "minLength": 1, "title": "evaluationName", "type": "string" }, "insightConfiguration": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "maxNumPrompts": { "default": 0, "description": "The max number of prompts to evaluate.", "maximum": 5000, "minimum": 0, "title": "maxNumPrompts", "type": "integer" }, "ootbDatasetName": { "anyOf": [ { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, { "type": "null" } ], "description": "Out-of-the-box evaluation dataset name. this applies only to our predefined public evaluation datasets." }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "evaluationName", "insightConfiguration", "insightGradingCriteria" ], "title": "DatasetEvaluationRequest", "type": "object" }, "maxItems": 10, "minItems": 1, "title": "datasetEvaluations", "type": "array" }, "description": { "default": "", "description": "LLM test configuration description.", "maxLength": 5000, "title": "description", "type": "string" }, "llmTestGradingCriteria": { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, "name": { "description": "LLM test configuration name.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "useCaseId": { "description": "The use case ID associated with the LLM test configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "name", "useCaseId", "datasetEvaluations", "llmTestGradingCriteria" ], "title": "CreateLLMTestConfigurationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | CreateLLMTestConfigurationRequest | true | none | ### Example responses > 201 Response ``` { "description": "API response object for a single llmtestconfiguration.", "properties": { "creationDate": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The creation date of the LLM test configuration. for ootb LLM test configurations this is null.", "title": "creationDate" }, "creationUserId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the user who created the LLM test configuration. for ootb LLM test configurations this is null.", "title": "creationUserId" }, "datasetEvaluations": { "description": "The LLM test dataset evaluations.", "items": { "description": "Dataset evaluation.", "properties": { "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the dataset evaluation.", "title": "errorMessage" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration for this dataset evaluation.", "title": "evaluationDatasetConfigurationId" }, "evaluationDatasetName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset name.", "title": "evaluationDatasetName" }, "evaluationName": { "description": "The name of the evaluation. this name should provide context regarding what is being evaluated.", "maxLength": 5000, "minLength": 1, "title": "evaluationName", "type": "string" }, "insightConfiguration": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "maxNumPrompts": { "default": 100, "description": "The max number of prompts to evaluate.", "exclusiveMinimum": 0, "maximum": 5000, "title": "maxNumPrompts", "type": "integer" }, "ootbDataset": { "anyOf": [ { "description": "Out-of-the-box dataset.", "properties": { "datasetName": { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, "datasetUrl": { "anyOf": [ { "description": "Out-of-the-box dataset url.", "enum": [ "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-age-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-gender-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-race-ethnicity-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-religion-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-disability-status-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-sexual-orientation-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-nationality-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-ses-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-parent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-great-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/pii-v1.csv" ], "title": "OOTBDatasetUrl", "type": "string" }, { "type": "null" } ], "description": "The public url of the evaluation dataset. this applies only to our predefined public evaluation datasets." }, "promptColumnName": { "description": "The name of the prompt column.", "maxLength": 5000, "minLength": 1, "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the response column, if present.", "title": "responseColumnName" }, "rowsCount": { "description": "The number rows in the dataset.", "title": "rowsCount", "type": "integer" }, "warning": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Warning about the content of the dataset.", "title": "warning" } }, "required": [ "datasetName", "datasetUrl", "promptColumnName", "responseColumnName", "rowsCount" ], "title": "OOTBDataset", "type": "object" }, { "type": "null" } ], "description": "Out-of-the-box evaluation dataset. this applies only to our predefined public evaluation datasets." }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "evaluationName", "insightConfiguration", "insightGradingCriteria", "evaluationDatasetName" ], "title": "DatasetEvaluationResponse", "type": "object" }, "title": "datasetEvaluations", "type": "array" }, "description": { "description": "The description of the LLM test configuration.", "title": "description", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the LLM test configuration.", "title": "errorMessage" }, "id": { "description": "The ID of the LLM test configuration.", "title": "id", "type": "string" }, "isOutOfTheBoxTestConfiguration": { "description": "Identifies the LLM test configuration as an out-of-the-box (ootb) test configuration.", "title": "isOutOfTheBoxTestConfiguration", "type": "boolean" }, "lastUpdateDate": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The last update date of the LLM test configuration. for ootb LLM test configurations this is null.", "title": "lastUpdateDate" }, "lastUpdateUserId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the user who last updated the LLM test configuration. for ootb LLM test configurations this is null.", "title": "lastUpdateUserId" }, "llmTestGradingCriteria": { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, "name": { "description": "The name of the LLM test configuration.", "title": "name", "type": "string" }, "useCaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "If specified, the use case ID associated with the LLM test configuration.", "title": "useCaseId" }, "warnings": { "description": "Warnings for this LLM test configuration.", "items": { "additionalProperties": { "type": "string" }, "propertyNames": { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, "type": "object" }, "title": "warnings", "type": "array" } }, "required": [ "id", "name", "description", "datasetEvaluations", "llmTestGradingCriteria", "isOutOfTheBoxTestConfiguration", "warnings" ], "title": "LLMTestConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 201 | Created | Successful Response | LLMTestConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List non out-of-the-box datasets Operation path: `GET /api/v2/genai/llmTestConfigurations/nonOotbDatasets/` Authentication requirements: `BearerAuth` List the supported non out-of-the-box datasets that can be used with an LLM test configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | useCaseId | query | string | true | Use Case ID. | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | search | query | any | false | Only retrieve the datasets with names matching the search query. | ### Example responses > 200 Response ``` { "description": "Paginated list of non-ootb datasets for use with LLM test configurations.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "Non out-of-the-box dataset used with an LLM test configuration.", "properties": { "agentGoalsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected agent goals (for agentic workflows).", "title": "agentGoalsColumnName" }, "correctnessEnabled": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "deprecated": true, "description": "Whether correctness is enabled for the evaluation dataset configuration.", "title": "correctnessEnabled" }, "creationDate": { "description": "The creation date of the evaluation dataset configuration (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the evaluation dataset configuration.", "title": "creationUserId", "type": "string" }, "datasetId": { "description": "The ID of the evaluation dataset.", "title": "datasetId", "type": "string" }, "datasetName": { "description": "The name of the evaluation dataset.", "title": "datasetName", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration.", "title": "errorMessage" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "id": { "description": "The ID of the evaluation dataset configuration.", "title": "id", "type": "string" }, "name": { "description": "The name of the evaluation dataset configuration.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the evaluation dataset configuration.", "title": "playgroundId" }, "promptColumnName": { "description": "The name of the dataset column containing the prompt text.", "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing the response text.", "title": "responseColumnName" }, "rowsCount": { "description": "The rows count of the evaluation dataset.", "title": "rowsCount", "type": "integer" }, "size": { "description": "The size of the evaluation dataset (in bytes).", "title": "size", "type": "integer" }, "tenantId": { "description": "The ID of the datarobot tenant this evaluation dataset configuration belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "toolCallsColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected tool calls (for agentic workflows).", "title": "toolCallsColumnName" }, "useCaseId": { "description": "The ID of the use case associated with the evaluation dataset configuration.", "title": "useCaseId", "type": "string" }, "userName": { "description": "The name of the user that created the evaluation dataset configuration.", "title": "userName", "type": "string" } }, "required": [ "id", "name", "size", "rowsCount", "useCaseId", "playgroundId", "datasetId", "datasetName", "promptColumnName", "responseColumnName", "toolCallsColumnName", "agentGoalsColumnName", "userName", "correctnessEnabled", "creationUserId", "creationDate", "tenantId", "executionStatus" ], "title": "LLMTestConfigurationNonOOTBDatasetResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListLLMTestConfigurationNonOOTBDatasetsResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Non out-of-the-box datasets successfully retrieved. | ListLLMTestConfigurationNonOOTBDatasetsResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List out-of-the-box datasets Operation path: `GET /api/v2/genai/llmTestConfigurations/ootbDatasets/` Authentication requirements: `BearerAuth` List the supported out-of-the-box datasets that can be used with an LLM test configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | search | query | any | false | Only retrieve the datasets with names matching the search query. | ### Example responses > 200 Response ``` { "description": "Paginated list of ootb datasets for use with LLM test configurations.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "Out-of-the-box dataset used with an LLM test configuration.", "properties": { "datasetName": { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, "datasetUrl": { "anyOf": [ { "description": "Out-of-the-box dataset url.", "enum": [ "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-age-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-gender-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-race-ethnicity-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-religion-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-disability-status-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-sexual-orientation-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-nationality-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-ses-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-parent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-great-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/pii-v1.csv" ], "title": "OOTBDatasetUrl", "type": "string" }, { "type": "null" } ], "description": "The public url of the evaluation dataset. this applies only to our predefined public evaluation datasets." }, "promptColumnName": { "description": "The name of the prompt column.", "maxLength": 5000, "minLength": 1, "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the response column, if present.", "title": "responseColumnName" }, "rowsCount": { "description": "The number rows in the dataset.", "title": "rowsCount", "type": "integer" }, "warning": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Warning about the content of the dataset.", "title": "warning" } }, "required": [ "datasetName", "datasetUrl", "promptColumnName", "responseColumnName", "rowsCount" ], "title": "LLMTestConfigurationOOTBDatasetResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListLLMTestConfigurationOOTBDatasetsResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Out-of-the-box datasets successfully retrieved. | ListLLMTestConfigurationOOTBDatasetsResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List supported insights Operation path: `GET /api/v2/genai/llmTestConfigurations/supportedInsights/` Authentication requirements: `BearerAuth` List the supported LLM test insight configurations for the specified use case. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | useCaseId | query | any | false | If specified, only retrieve the insights supported by this use case ID. | | playgroundId | query | any | false | If specified, only retrieve the insights supported by the use case for which the playgroundId belongs. | ### Example responses > 200 Response ``` { "description": "Response model for supported insights.", "properties": { "datasetsCompatibility": { "description": "The list of insight to evaluation datasets compatibility.", "items": { "description": "Insight to evaluation datasets compatibility.", "properties": { "incompatibleDatasets": { "description": "The list of incompatible datasets.", "items": { "description": "Dataset identifier.", "properties": { "datasetId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the dataset, if any.", "title": "datasetId" }, "datasetName": { "description": "The name of the dataset.", "title": "datasetName", "type": "string" } }, "required": [ "datasetName", "datasetId" ], "title": "DatasetIdentifier", "type": "object" }, "title": "incompatibleDatasets", "type": "array" }, "insightName": { "description": "The name of the insight.", "title": "insightName", "type": "string" } }, "required": [ "insightName", "incompatibleDatasets" ], "title": "InsightToEvalDatasetsCompatibility", "type": "object" }, "title": "datasetsCompatibility", "type": "array" }, "supportedInsightConfigurations": { "description": "The list of supported insight configurations for the LLM tests.", "items": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "title": "supportedInsightConfigurations", "type": "array" } }, "required": [ "supportedInsightConfigurations", "datasetsCompatibility" ], "title": "LLMTestConfigurationSupportedInsightsResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | LLM test supported insight configurations successfully retrieved. | LLMTestConfigurationSupportedInsightsResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Delete LLM test configuration by LLM test configuration ID Operation path: `DELETE /api/v2/genai/llmTestConfigurations/{llmTestConfigurationId}/` Authentication requirements: `BearerAuth` Delete an existing LLM test configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmTestConfigurationId | path | string | true | The ID of the LLM Test Configuration to delete. | ### Example responses > 422 Response ``` { "properties": { "detail": { "items": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "title": "detail", "type": "array" } }, "title": "HTTPValidationErrorResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Successful Response | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Retrieve LLM test configuration by LLM test configuration ID Operation path: `GET /api/v2/genai/llmTestConfigurations/{llmTestConfigurationId}/` Authentication requirements: `BearerAuth` Retrieve an existing LLM test configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmTestConfigurationId | path | string | true | The ID of the LLM Test Configuration to retrieve. | ### Example responses > 200 Response ``` { "description": "API response object for a single llmtestconfiguration.", "properties": { "creationDate": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The creation date of the LLM test configuration. for ootb LLM test configurations this is null.", "title": "creationDate" }, "creationUserId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the user who created the LLM test configuration. for ootb LLM test configurations this is null.", "title": "creationUserId" }, "datasetEvaluations": { "description": "The LLM test dataset evaluations.", "items": { "description": "Dataset evaluation.", "properties": { "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the dataset evaluation.", "title": "errorMessage" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration for this dataset evaluation.", "title": "evaluationDatasetConfigurationId" }, "evaluationDatasetName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset name.", "title": "evaluationDatasetName" }, "evaluationName": { "description": "The name of the evaluation. this name should provide context regarding what is being evaluated.", "maxLength": 5000, "minLength": 1, "title": "evaluationName", "type": "string" }, "insightConfiguration": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "maxNumPrompts": { "default": 100, "description": "The max number of prompts to evaluate.", "exclusiveMinimum": 0, "maximum": 5000, "title": "maxNumPrompts", "type": "integer" }, "ootbDataset": { "anyOf": [ { "description": "Out-of-the-box dataset.", "properties": { "datasetName": { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, "datasetUrl": { "anyOf": [ { "description": "Out-of-the-box dataset url.", "enum": [ "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-age-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-gender-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-race-ethnicity-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-religion-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-disability-status-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-sexual-orientation-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-nationality-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-ses-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-parent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-great-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/pii-v1.csv" ], "title": "OOTBDatasetUrl", "type": "string" }, { "type": "null" } ], "description": "The public url of the evaluation dataset. this applies only to our predefined public evaluation datasets." }, "promptColumnName": { "description": "The name of the prompt column.", "maxLength": 5000, "minLength": 1, "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the response column, if present.", "title": "responseColumnName" }, "rowsCount": { "description": "The number rows in the dataset.", "title": "rowsCount", "type": "integer" }, "warning": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Warning about the content of the dataset.", "title": "warning" } }, "required": [ "datasetName", "datasetUrl", "promptColumnName", "responseColumnName", "rowsCount" ], "title": "OOTBDataset", "type": "object" }, { "type": "null" } ], "description": "Out-of-the-box evaluation dataset. this applies only to our predefined public evaluation datasets." }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "evaluationName", "insightConfiguration", "insightGradingCriteria", "evaluationDatasetName" ], "title": "DatasetEvaluationResponse", "type": "object" }, "title": "datasetEvaluations", "type": "array" }, "description": { "description": "The description of the LLM test configuration.", "title": "description", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the LLM test configuration.", "title": "errorMessage" }, "id": { "description": "The ID of the LLM test configuration.", "title": "id", "type": "string" }, "isOutOfTheBoxTestConfiguration": { "description": "Identifies the LLM test configuration as an out-of-the-box (ootb) test configuration.", "title": "isOutOfTheBoxTestConfiguration", "type": "boolean" }, "lastUpdateDate": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The last update date of the LLM test configuration. for ootb LLM test configurations this is null.", "title": "lastUpdateDate" }, "lastUpdateUserId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the user who last updated the LLM test configuration. for ootb LLM test configurations this is null.", "title": "lastUpdateUserId" }, "llmTestGradingCriteria": { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, "name": { "description": "The name of the LLM test configuration.", "title": "name", "type": "string" }, "useCaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "If specified, the use case ID associated with the LLM test configuration.", "title": "useCaseId" }, "warnings": { "description": "Warnings for this LLM test configuration.", "items": { "additionalProperties": { "type": "string" }, "propertyNames": { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, "type": "object" }, "title": "warnings", "type": "array" } }, "required": [ "id", "name", "description", "datasetEvaluations", "llmTestGradingCriteria", "isOutOfTheBoxTestConfiguration", "warnings" ], "title": "LLMTestConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | LLMTestConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Edit LLM test configuration by LLM test configuration ID Operation path: `PATCH /api/v2/genai/llmTestConfigurations/{llmTestConfigurationId}/` Authentication requirements: `BearerAuth` Edit an existing LLM test configuration. ### Body parameter ``` { "description": "Request object for editing a llmtestconfiguration.", "properties": { "datasetEvaluations": { "anyOf": [ { "items": { "description": "Dataset evaluation.", "properties": { "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration for this dataset evaluation.", "title": "evaluationDatasetConfigurationId" }, "evaluationName": { "description": "The name of the evaluation. this name should provide context regarding what is being evaluated.", "maxLength": 5000, "minLength": 1, "title": "evaluationName", "type": "string" }, "insightConfiguration": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "maxNumPrompts": { "default": 0, "description": "The max number of prompts to evaluate.", "maximum": 5000, "minimum": 0, "title": "maxNumPrompts", "type": "integer" }, "ootbDatasetName": { "anyOf": [ { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, { "type": "null" } ], "description": "Out-of-the-box evaluation dataset name. this applies only to our predefined public evaluation datasets." }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "evaluationName", "insightConfiguration", "insightGradingCriteria" ], "title": "DatasetEvaluationRequest", "type": "object" }, "maxItems": 10, "minItems": 1, "type": "array" }, { "type": "null" } ], "description": "New dataset evaluations.", "title": "datasetEvaluations" }, "description": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "New LLM test configuration description.", "title": "description" }, "llmTestGradingCriteria": { "anyOf": [ { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, { "type": "null" } ], "description": "New LLM test grading criteria." }, "name": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "New LLM test configuration name.", "title": "name" } }, "title": "EditLLMTestConfigurationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmTestConfigurationId | path | string | true | The ID of the LLM Test Configuration to update. | | body | body | EditLLMTestConfigurationRequest | true | none | ### Example responses > 200 Response ``` { "description": "API response object for a single llmtestconfiguration.", "properties": { "creationDate": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The creation date of the LLM test configuration. for ootb LLM test configurations this is null.", "title": "creationDate" }, "creationUserId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the user who created the LLM test configuration. for ootb LLM test configurations this is null.", "title": "creationUserId" }, "datasetEvaluations": { "description": "The LLM test dataset evaluations.", "items": { "description": "Dataset evaluation.", "properties": { "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the dataset evaluation.", "title": "errorMessage" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration for this dataset evaluation.", "title": "evaluationDatasetConfigurationId" }, "evaluationDatasetName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset name.", "title": "evaluationDatasetName" }, "evaluationName": { "description": "The name of the evaluation. this name should provide context regarding what is being evaluated.", "maxLength": 5000, "minLength": 1, "title": "evaluationName", "type": "string" }, "insightConfiguration": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "maxNumPrompts": { "default": 100, "description": "The max number of prompts to evaluate.", "exclusiveMinimum": 0, "maximum": 5000, "title": "maxNumPrompts", "type": "integer" }, "ootbDataset": { "anyOf": [ { "description": "Out-of-the-box dataset.", "properties": { "datasetName": { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, "datasetUrl": { "anyOf": [ { "description": "Out-of-the-box dataset url.", "enum": [ "https://s3.amazonaws.com/datarobot_public_datasets/genai/jailbreak-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-age-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-gender-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-race-ethnicity-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-religion-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-disability-status-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-sexual-orientation-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-nationality-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/bbq-lite-ses-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-parent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/completeness-great-grandparent-v1.csv", "https://s3.amazonaws.com/datarobot_public_datasets/genai/pii-v1.csv" ], "title": "OOTBDatasetUrl", "type": "string" }, { "type": "null" } ], "description": "The public url of the evaluation dataset. this applies only to our predefined public evaluation datasets." }, "promptColumnName": { "description": "The name of the prompt column.", "maxLength": 5000, "minLength": 1, "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the response column, if present.", "title": "responseColumnName" }, "rowsCount": { "description": "The number rows in the dataset.", "title": "rowsCount", "type": "integer" }, "warning": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Warning about the content of the dataset.", "title": "warning" } }, "required": [ "datasetName", "datasetUrl", "promptColumnName", "responseColumnName", "rowsCount" ], "title": "OOTBDataset", "type": "object" }, { "type": "null" } ], "description": "Out-of-the-box evaluation dataset. this applies only to our predefined public evaluation datasets." }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "evaluationName", "insightConfiguration", "insightGradingCriteria", "evaluationDatasetName" ], "title": "DatasetEvaluationResponse", "type": "object" }, "title": "datasetEvaluations", "type": "array" }, "description": { "description": "The description of the LLM test configuration.", "title": "description", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the LLM test configuration.", "title": "errorMessage" }, "id": { "description": "The ID of the LLM test configuration.", "title": "id", "type": "string" }, "isOutOfTheBoxTestConfiguration": { "description": "Identifies the LLM test configuration as an out-of-the-box (ootb) test configuration.", "title": "isOutOfTheBoxTestConfiguration", "type": "boolean" }, "lastUpdateDate": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The last update date of the LLM test configuration. for ootb LLM test configurations this is null.", "title": "lastUpdateDate" }, "lastUpdateUserId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the user who last updated the LLM test configuration. for ootb LLM test configurations this is null.", "title": "lastUpdateUserId" }, "llmTestGradingCriteria": { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, "name": { "description": "The name of the LLM test configuration.", "title": "name", "type": "string" }, "useCaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "If specified, the use case ID associated with the LLM test configuration.", "title": "useCaseId" }, "warnings": { "description": "Warnings for this LLM test configuration.", "items": { "additionalProperties": { "type": "string" }, "propertyNames": { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, "type": "object" }, "title": "warnings", "type": "array" } }, "required": [ "id", "name", "description", "datasetEvaluations", "llmTestGradingCriteria", "isOutOfTheBoxTestConfiguration", "warnings" ], "title": "LLMTestConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | LLMTestConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List LLM test results Operation path: `GET /api/v2/genai/llmTestResults/` Authentication requirements: `BearerAuth` List LLM test results. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmTestConfigurationId | query | any | false | LLM Test Configuration ID. | | llmBlueprintId | query | any | false | LLM Blueprint ID. | | llmTestSuiteId | query | any | false | LLM Test Suite ID. | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | ### Example responses > 200 Response ``` { "description": "Paginated list of LLM test results.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "API response object for a single llmtestresult.", "properties": { "creationDate": { "description": "LLM test result creation date (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "ID of the user that created this LLM test result.", "title": "creationUserId", "type": "string" }, "creationUserName": { "description": "The name of the user who created this LLM result.", "title": "creationUserName", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message if the LLM test result failed.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error resolution message if the LLM test result failed.", "title": "errorResolution" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "gradingResult": { "anyOf": [ { "description": "Grading result.", "enum": [ "PASS", "FAIL" ], "title": "GradingResult", "type": "string" }, { "type": "null" } ], "description": "The grading result based on the LLM test grading criteria. if not specified, execution status is not completed." }, "id": { "description": "LLM test result id.", "title": "id", "type": "string" }, "insightEvaluationResults": { "description": "The insight evaluation results.", "items": { "description": "API response object for a single insightevaluationresult.", "properties": { "aggregationType": { "anyOf": [ { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, { "type": "null" } ], "description": "Aggregation type." }, "aggregationValue": { "anyOf": [ { "type": "number" }, { "items": { "description": "An individual record in an itemized metric aggregation.", "properties": { "item": { "description": "The name of the item.", "title": "item", "type": "string" }, "value": { "description": "The value associated with the item.", "title": "value", "type": "number" } }, "required": [ "item", "value" ], "title": "AggregationValue", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "Aggregation value. none indicates that the aggregation failed.", "title": "aggregationValue" }, "chatId": { "description": "Chat id.", "title": "chatId", "type": "string" }, "chatName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Chat name.", "title": "chatName" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Custom model LLM validation ID if using custom model llm.", "title": "customModelLLMValidationId" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset configuration id.", "title": "evaluationDatasetConfigurationId" }, "evaluationDatasetName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset name.", "title": "evaluationDatasetName" }, "evaluationName": { "description": "Evaluation name.", "maxLength": 5000, "title": "evaluationName", "type": "string" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "gradingResult": { "anyOf": [ { "description": "Grading result.", "enum": [ "PASS", "FAIL" ], "title": "GradingResult", "type": "string" }, { "type": "null" } ], "description": "The grading result for this insight evaluation result. if not specified, execution status is not completed." }, "id": { "description": "Insight evaluation result id.", "title": "id", "type": "string" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "lastUpdateDate": { "description": "Last update date of the insight evaluation result (iso 8601 formatted).", "format": "date-time", "title": "lastUpdateDate", "type": "string" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM ID used for this insight evaluation result.", "title": "llmId" }, "llmTestResultId": { "description": "LLM test result ID this insight evaluation result is associated to.", "title": "llmTestResultId", "type": "string" }, "maxNumPrompts": { "description": "Number of prompts used in evaluation.", "title": "maxNumPrompts", "type": "integer" }, "metricName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Name of the metric.", "title": "metricName" }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "id", "llmTestResultId", "maxNumPrompts", "promptSamplingStrategy", "chatId", "chatName", "evaluationName", "insightGradingCriteria", "lastUpdateDate" ], "title": "InsightEvaluationResultResponse", "type": "object" }, "title": "insightEvaluationResults", "type": "array" }, "isOutOfTheBoxTestConfiguration": { "description": "Identifies the LLM test configuration as an out-of-the-box (ootb) test configuration.", "title": "isOutOfTheBoxTestConfiguration", "type": "boolean" }, "llmBlueprintId": { "description": "LLM blueprint id.", "title": "llmBlueprintId", "type": "string" }, "llmBlueprintSnapshot": { "description": "A snapshot in time of a llmblueprint's functional parameters.", "properties": { "description": { "description": "The description of the llmblueprint at the time of snapshotting.", "title": "description", "type": "string" }, "id": { "description": "The ID of the llmblueprint for which the snapshot was produced.", "title": "id", "type": "string" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the LLM selected for this LLM blueprint.", "title": "llmId" }, "llmSettings": { "anyOf": [ { "additionalProperties": true, "description": "The settings that are available for all non-custom llms.", "properties": { "maxCompletionLength": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Maximum number of tokens allowed in the chat completion. use this value to, for example, control costs on token-based charges or manage response length for chat text limits.", "title": "maxCompletionLength" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "title": "CommonLLMSettings", "type": "object" }, { "additionalProperties": false, "description": "The settings that are available for custom model llms.", "properties": { "externalLlmContextSize": { "anyOf": [ { "maximum": 128000, "minimum": 128, "type": "integer" }, { "type": "null" } ], "default": 4096, "description": "The external llm's context size, in tokens. this value is only used for pruning documents supplied to the LLM when a vector database is associated with the LLM blueprint. it does not affect the external llm's actual context size in any way and is not supplied to the llm.", "title": "externalLlmContextSize" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" }, "validationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The validation ID of the custom model llm.", "title": "validationId" } }, "title": "CustomModelLLMSettings", "type": "object" }, { "additionalProperties": false, "description": "The settings that are available for custom model llms used via chat completion interface.", "properties": { "customModelId": { "description": "The ID of the custom model used via chat completion interface.", "title": "customModelId", "type": "string" }, "customModelVersionId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model version used via chat completion interface.", "title": "customModelVersionId" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "required": [ "customModelId" ], "title": "CustomModelChatLLMSettings", "type": "object" }, { "type": "null" } ], "description": "A key/value dictionary of LLM settings.", "title": "llmSettings" }, "name": { "description": "The name of the llmblueprint at the time of snapshotting.", "title": "name", "type": "string" }, "playgroundId": { "description": "The playground ID of the llmblueprint.", "title": "playgroundId", "type": "string" }, "promptType": { "description": "Determines whether chat history is submitted as context to the user prompt.", "enum": [ "CHAT_HISTORY_AWARE", "ONE_TIME_PROMPT" ], "title": "PromptType", "type": "string" }, "snapshotDate": { "description": "The date when the snapshot was produced.", "format": "date-time", "title": "snapshotDate", "type": "string" }, "vectorDatabaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the vector database linked to this LLM blueprint.", "title": "vectorDatabaseId" }, "vectorDatabaseSettings": { "anyOf": [ { "description": "Vector database retrieval settings.", "properties": { "addNeighborChunks": { "default": false, "description": "Add neighboring chunks to those that the similarity search retrieves, such that when selected, search returns i, i-1, and i+1.", "title": "addNeighborChunks", "type": "boolean" }, "maxDocumentsRetrievedPerPrompt": { "anyOf": [ { "maximum": 10, "minimum": 1, "type": "integer" }, { "type": "null" } ], "description": "The maximum number of chunks to retrieve from the vector database.", "title": "maxDocumentsRetrievedPerPrompt" }, "maxTokens": { "anyOf": [ { "maximum": 51200, "minimum": 1, "type": "integer" }, { "type": "null" } ], "description": "The maximum number of tokens to retrieve from the vector database.", "title": "maxTokens" }, "maximalMarginalRelevanceLambda": { "default": 0.5, "description": "Adjust the retrieval of chunks when using maximal marginal relevance to favor diversity (0.0) or similarity (1.0).", "maximum": 1, "minimum": 0, "title": "maximalMarginalRelevanceLambda", "type": "number" }, "retrievalMode": { "description": "Retrieval modes for vector databases.", "enum": [ "similarity", "maximal_marginal_relevance" ], "title": "RetrievalMode", "type": "string" }, "retriever": { "description": "The method used to retrieve relevant chunks from the vector database.", "enum": [ "SINGLE_LOOKUP_RETRIEVER", "CONVERSATIONAL_RETRIEVER", "MULTI_STEP_RETRIEVER" ], "title": "VectorDatabaseRetrievers", "type": "string" } }, "title": "VectorDatabaseSettings", "type": "object" }, { "type": "null" } ], "description": "A key/value dictionary of vector database settings." } }, "required": [ "id", "name", "description", "playgroundId", "promptType" ], "title": "LLMBlueprintSnapshot", "type": "object" }, "llmTestConfigurationId": { "description": "LLM test configuration ID this LLM result is associated to.", "title": "llmTestConfigurationId", "type": "string" }, "llmTestConfigurationName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Name of the LLM test configuration this LLM result is associated to.", "title": "llmTestConfigurationName" }, "llmTestGradingCriteria": { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, "llmTestSuiteId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM test suite ID to which the LLM test configuration is associated to.", "title": "llmTestSuiteId" }, "passPercentage": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "description": "The percentage of underlying insight evaluation results that have a pass grading result. if not specified, execution status is not completed.", "title": "passPercentage" }, "useCaseId": { "description": "Use case ID this LLM test result belongs to.", "title": "useCaseId", "type": "string" } }, "required": [ "id", "llmTestConfigurationId", "llmTestConfigurationName", "isOutOfTheBoxTestConfiguration", "useCaseId", "llmBlueprintId", "llmBlueprintSnapshot", "llmTestGradingCriteria", "executionStatus", "insightEvaluationResults", "creationDate", "creationUserId", "creationUserName" ], "title": "LLMTestResultResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListLLMTestResultResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | ListLLMTestResultResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Create LLM test result Operation path: `POST /api/v2/genai/llmTestResults/` Authentication requirements: `BearerAuth` Create a new LLM test result. ### Body parameter ``` { "description": "Request object for creating a llmtestresult.", "properties": { "llmBlueprintId": { "description": "The LLM blueprint ID associated with the LLM test result.", "title": "llmBlueprintId", "type": "string" }, "llmTestConfigurationId": { "description": "The use case ID associated with the LLM test result.", "title": "llmTestConfigurationId", "type": "string" } }, "required": [ "llmTestConfigurationId", "llmBlueprintId" ], "title": "CreateLLMTestResultRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | CreateLLMTestResultRequest | true | none | ### Example responses > 202 Response ``` { "description": "API response object for a single llmtestresult.", "properties": { "creationDate": { "description": "LLM test result creation date (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "ID of the user that created this LLM test result.", "title": "creationUserId", "type": "string" }, "creationUserName": { "description": "The name of the user who created this LLM result.", "title": "creationUserName", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message if the LLM test result failed.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error resolution message if the LLM test result failed.", "title": "errorResolution" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "gradingResult": { "anyOf": [ { "description": "Grading result.", "enum": [ "PASS", "FAIL" ], "title": "GradingResult", "type": "string" }, { "type": "null" } ], "description": "The grading result based on the LLM test grading criteria. if not specified, execution status is not completed." }, "id": { "description": "LLM test result id.", "title": "id", "type": "string" }, "insightEvaluationResults": { "description": "The insight evaluation results.", "items": { "description": "API response object for a single insightevaluationresult.", "properties": { "aggregationType": { "anyOf": [ { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, { "type": "null" } ], "description": "Aggregation type." }, "aggregationValue": { "anyOf": [ { "type": "number" }, { "items": { "description": "An individual record in an itemized metric aggregation.", "properties": { "item": { "description": "The name of the item.", "title": "item", "type": "string" }, "value": { "description": "The value associated with the item.", "title": "value", "type": "number" } }, "required": [ "item", "value" ], "title": "AggregationValue", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "Aggregation value. none indicates that the aggregation failed.", "title": "aggregationValue" }, "chatId": { "description": "Chat id.", "title": "chatId", "type": "string" }, "chatName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Chat name.", "title": "chatName" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Custom model LLM validation ID if using custom model llm.", "title": "customModelLLMValidationId" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset configuration id.", "title": "evaluationDatasetConfigurationId" }, "evaluationDatasetName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset name.", "title": "evaluationDatasetName" }, "evaluationName": { "description": "Evaluation name.", "maxLength": 5000, "title": "evaluationName", "type": "string" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "gradingResult": { "anyOf": [ { "description": "Grading result.", "enum": [ "PASS", "FAIL" ], "title": "GradingResult", "type": "string" }, { "type": "null" } ], "description": "The grading result for this insight evaluation result. if not specified, execution status is not completed." }, "id": { "description": "Insight evaluation result id.", "title": "id", "type": "string" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "lastUpdateDate": { "description": "Last update date of the insight evaluation result (iso 8601 formatted).", "format": "date-time", "title": "lastUpdateDate", "type": "string" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM ID used for this insight evaluation result.", "title": "llmId" }, "llmTestResultId": { "description": "LLM test result ID this insight evaluation result is associated to.", "title": "llmTestResultId", "type": "string" }, "maxNumPrompts": { "description": "Number of prompts used in evaluation.", "title": "maxNumPrompts", "type": "integer" }, "metricName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Name of the metric.", "title": "metricName" }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "id", "llmTestResultId", "maxNumPrompts", "promptSamplingStrategy", "chatId", "chatName", "evaluationName", "insightGradingCriteria", "lastUpdateDate" ], "title": "InsightEvaluationResultResponse", "type": "object" }, "title": "insightEvaluationResults", "type": "array" }, "isOutOfTheBoxTestConfiguration": { "description": "Identifies the LLM test configuration as an out-of-the-box (ootb) test configuration.", "title": "isOutOfTheBoxTestConfiguration", "type": "boolean" }, "llmBlueprintId": { "description": "LLM blueprint id.", "title": "llmBlueprintId", "type": "string" }, "llmBlueprintSnapshot": { "description": "A snapshot in time of a llmblueprint's functional parameters.", "properties": { "description": { "description": "The description of the llmblueprint at the time of snapshotting.", "title": "description", "type": "string" }, "id": { "description": "The ID of the llmblueprint for which the snapshot was produced.", "title": "id", "type": "string" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the LLM selected for this LLM blueprint.", "title": "llmId" }, "llmSettings": { "anyOf": [ { "additionalProperties": true, "description": "The settings that are available for all non-custom llms.", "properties": { "maxCompletionLength": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Maximum number of tokens allowed in the chat completion. use this value to, for example, control costs on token-based charges or manage response length for chat text limits.", "title": "maxCompletionLength" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "title": "CommonLLMSettings", "type": "object" }, { "additionalProperties": false, "description": "The settings that are available for custom model llms.", "properties": { "externalLlmContextSize": { "anyOf": [ { "maximum": 128000, "minimum": 128, "type": "integer" }, { "type": "null" } ], "default": 4096, "description": "The external llm's context size, in tokens. this value is only used for pruning documents supplied to the LLM when a vector database is associated with the LLM blueprint. it does not affect the external llm's actual context size in any way and is not supplied to the llm.", "title": "externalLlmContextSize" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" }, "validationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The validation ID of the custom model llm.", "title": "validationId" } }, "title": "CustomModelLLMSettings", "type": "object" }, { "additionalProperties": false, "description": "The settings that are available for custom model llms used via chat completion interface.", "properties": { "customModelId": { "description": "The ID of the custom model used via chat completion interface.", "title": "customModelId", "type": "string" }, "customModelVersionId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model version used via chat completion interface.", "title": "customModelVersionId" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "required": [ "customModelId" ], "title": "CustomModelChatLLMSettings", "type": "object" }, { "type": "null" } ], "description": "A key/value dictionary of LLM settings.", "title": "llmSettings" }, "name": { "description": "The name of the llmblueprint at the time of snapshotting.", "title": "name", "type": "string" }, "playgroundId": { "description": "The playground ID of the llmblueprint.", "title": "playgroundId", "type": "string" }, "promptType": { "description": "Determines whether chat history is submitted as context to the user prompt.", "enum": [ "CHAT_HISTORY_AWARE", "ONE_TIME_PROMPT" ], "title": "PromptType", "type": "string" }, "snapshotDate": { "description": "The date when the snapshot was produced.", "format": "date-time", "title": "snapshotDate", "type": "string" }, "vectorDatabaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the vector database linked to this LLM blueprint.", "title": "vectorDatabaseId" }, "vectorDatabaseSettings": { "anyOf": [ { "description": "Vector database retrieval settings.", "properties": { "addNeighborChunks": { "default": false, "description": "Add neighboring chunks to those that the similarity search retrieves, such that when selected, search returns i, i-1, and i+1.", "title": "addNeighborChunks", "type": "boolean" }, "maxDocumentsRetrievedPerPrompt": { "anyOf": [ { "maximum": 10, "minimum": 1, "type": "integer" }, { "type": "null" } ], "description": "The maximum number of chunks to retrieve from the vector database.", "title": "maxDocumentsRetrievedPerPrompt" }, "maxTokens": { "anyOf": [ { "maximum": 51200, "minimum": 1, "type": "integer" }, { "type": "null" } ], "description": "The maximum number of tokens to retrieve from the vector database.", "title": "maxTokens" }, "maximalMarginalRelevanceLambda": { "default": 0.5, "description": "Adjust the retrieval of chunks when using maximal marginal relevance to favor diversity (0.0) or similarity (1.0).", "maximum": 1, "minimum": 0, "title": "maximalMarginalRelevanceLambda", "type": "number" }, "retrievalMode": { "description": "Retrieval modes for vector databases.", "enum": [ "similarity", "maximal_marginal_relevance" ], "title": "RetrievalMode", "type": "string" }, "retriever": { "description": "The method used to retrieve relevant chunks from the vector database.", "enum": [ "SINGLE_LOOKUP_RETRIEVER", "CONVERSATIONAL_RETRIEVER", "MULTI_STEP_RETRIEVER" ], "title": "VectorDatabaseRetrievers", "type": "string" } }, "title": "VectorDatabaseSettings", "type": "object" }, { "type": "null" } ], "description": "A key/value dictionary of vector database settings." } }, "required": [ "id", "name", "description", "playgroundId", "promptType" ], "title": "LLMBlueprintSnapshot", "type": "object" }, "llmTestConfigurationId": { "description": "LLM test configuration ID this LLM result is associated to.", "title": "llmTestConfigurationId", "type": "string" }, "llmTestConfigurationName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Name of the LLM test configuration this LLM result is associated to.", "title": "llmTestConfigurationName" }, "llmTestGradingCriteria": { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, "llmTestSuiteId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM test suite ID to which the LLM test configuration is associated to.", "title": "llmTestSuiteId" }, "passPercentage": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "description": "The percentage of underlying insight evaluation results that have a pass grading result. if not specified, execution status is not completed.", "title": "passPercentage" }, "useCaseId": { "description": "Use case ID this LLM test result belongs to.", "title": "useCaseId", "type": "string" } }, "required": [ "id", "llmTestConfigurationId", "llmTestConfigurationName", "isOutOfTheBoxTestConfiguration", "useCaseId", "llmBlueprintId", "llmBlueprintSnapshot", "llmTestGradingCriteria", "executionStatus", "insightEvaluationResults", "creationDate", "creationUserId", "creationUserName" ], "title": "LLMTestResultResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 202 | Accepted | Successful Response | LLMTestResultResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Delete LLM test result by LLM test result ID Operation path: `DELETE /api/v2/genai/llmTestResults/{llmTestResultId}/` Authentication requirements: `BearerAuth` Delete an existing LLM test result. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmTestResultId | path | string | true | The ID of the LLM Test Result to delete. | ### Example responses > 422 Response ``` { "properties": { "detail": { "items": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "title": "detail", "type": "array" } }, "title": "HTTPValidationErrorResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Successful Response | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Retrieve LLM test result by LLM test result ID Operation path: `GET /api/v2/genai/llmTestResults/{llmTestResultId}/` Authentication requirements: `BearerAuth` Retrieve an existing LLM test result. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmTestResultId | path | string | true | The ID of the LLM Test Result to retrieve. | ### Example responses > 200 Response ``` { "description": "API response object for a single llmtestresult.", "properties": { "creationDate": { "description": "LLM test result creation date (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "ID of the user that created this LLM test result.", "title": "creationUserId", "type": "string" }, "creationUserName": { "description": "The name of the user who created this LLM result.", "title": "creationUserName", "type": "string" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message if the LLM test result failed.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error resolution message if the LLM test result failed.", "title": "errorResolution" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "gradingResult": { "anyOf": [ { "description": "Grading result.", "enum": [ "PASS", "FAIL" ], "title": "GradingResult", "type": "string" }, { "type": "null" } ], "description": "The grading result based on the LLM test grading criteria. if not specified, execution status is not completed." }, "id": { "description": "LLM test result id.", "title": "id", "type": "string" }, "insightEvaluationResults": { "description": "The insight evaluation results.", "items": { "description": "API response object for a single insightevaluationresult.", "properties": { "aggregationType": { "anyOf": [ { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, { "type": "null" } ], "description": "Aggregation type." }, "aggregationValue": { "anyOf": [ { "type": "number" }, { "items": { "description": "An individual record in an itemized metric aggregation.", "properties": { "item": { "description": "The name of the item.", "title": "item", "type": "string" }, "value": { "description": "The value associated with the item.", "title": "value", "type": "number" } }, "required": [ "item", "value" ], "title": "AggregationValue", "type": "object" }, "type": "array" }, { "type": "null" } ], "description": "Aggregation value. none indicates that the aggregation failed.", "title": "aggregationValue" }, "chatId": { "description": "Chat id.", "title": "chatId", "type": "string" }, "chatName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Chat name.", "title": "chatName" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Custom model LLM validation ID if using custom model llm.", "title": "customModelLLMValidationId" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset configuration id.", "title": "evaluationDatasetConfigurationId" }, "evaluationDatasetName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "Evaluation dataset name.", "title": "evaluationDatasetName" }, "evaluationName": { "description": "Evaluation name.", "maxLength": 5000, "title": "evaluationName", "type": "string" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "gradingResult": { "anyOf": [ { "description": "Grading result.", "enum": [ "PASS", "FAIL" ], "title": "GradingResult", "type": "string" }, { "type": "null" } ], "description": "The grading result for this insight evaluation result. if not specified, execution status is not completed." }, "id": { "description": "Insight evaluation result id.", "title": "id", "type": "string" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "lastUpdateDate": { "description": "Last update date of the insight evaluation result (iso 8601 formatted).", "format": "date-time", "title": "lastUpdateDate", "type": "string" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM ID used for this insight evaluation result.", "title": "llmId" }, "llmTestResultId": { "description": "LLM test result ID this insight evaluation result is associated to.", "title": "llmTestResultId", "type": "string" }, "maxNumPrompts": { "description": "Number of prompts used in evaluation.", "title": "maxNumPrompts", "type": "integer" }, "metricName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Name of the metric.", "title": "metricName" }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "id", "llmTestResultId", "maxNumPrompts", "promptSamplingStrategy", "chatId", "chatName", "evaluationName", "insightGradingCriteria", "lastUpdateDate" ], "title": "InsightEvaluationResultResponse", "type": "object" }, "title": "insightEvaluationResults", "type": "array" }, "isOutOfTheBoxTestConfiguration": { "description": "Identifies the LLM test configuration as an out-of-the-box (ootb) test configuration.", "title": "isOutOfTheBoxTestConfiguration", "type": "boolean" }, "llmBlueprintId": { "description": "LLM blueprint id.", "title": "llmBlueprintId", "type": "string" }, "llmBlueprintSnapshot": { "description": "A snapshot in time of a llmblueprint's functional parameters.", "properties": { "description": { "description": "The description of the llmblueprint at the time of snapshotting.", "title": "description", "type": "string" }, "id": { "description": "The ID of the llmblueprint for which the snapshot was produced.", "title": "id", "type": "string" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the LLM selected for this LLM blueprint.", "title": "llmId" }, "llmSettings": { "anyOf": [ { "additionalProperties": true, "description": "The settings that are available for all non-custom llms.", "properties": { "maxCompletionLength": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Maximum number of tokens allowed in the chat completion. use this value to, for example, control costs on token-based charges or manage response length for chat text limits.", "title": "maxCompletionLength" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "title": "CommonLLMSettings", "type": "object" }, { "additionalProperties": false, "description": "The settings that are available for custom model llms.", "properties": { "externalLlmContextSize": { "anyOf": [ { "maximum": 128000, "minimum": 128, "type": "integer" }, { "type": "null" } ], "default": 4096, "description": "The external llm's context size, in tokens. this value is only used for pruning documents supplied to the LLM when a vector database is associated with the LLM blueprint. it does not affect the external llm's actual context size in any way and is not supplied to the llm.", "title": "externalLlmContextSize" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" }, "validationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The validation ID of the custom model llm.", "title": "validationId" } }, "title": "CustomModelLLMSettings", "type": "object" }, { "additionalProperties": false, "description": "The settings that are available for custom model llms used via chat completion interface.", "properties": { "customModelId": { "description": "The ID of the custom model used via chat completion interface.", "title": "customModelId", "type": "string" }, "customModelVersionId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model version used via chat completion interface.", "title": "customModelVersionId" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "required": [ "customModelId" ], "title": "CustomModelChatLLMSettings", "type": "object" }, { "type": "null" } ], "description": "A key/value dictionary of LLM settings.", "title": "llmSettings" }, "name": { "description": "The name of the llmblueprint at the time of snapshotting.", "title": "name", "type": "string" }, "playgroundId": { "description": "The playground ID of the llmblueprint.", "title": "playgroundId", "type": "string" }, "promptType": { "description": "Determines whether chat history is submitted as context to the user prompt.", "enum": [ "CHAT_HISTORY_AWARE", "ONE_TIME_PROMPT" ], "title": "PromptType", "type": "string" }, "snapshotDate": { "description": "The date when the snapshot was produced.", "format": "date-time", "title": "snapshotDate", "type": "string" }, "vectorDatabaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the vector database linked to this LLM blueprint.", "title": "vectorDatabaseId" }, "vectorDatabaseSettings": { "anyOf": [ { "description": "Vector database retrieval settings.", "properties": { "addNeighborChunks": { "default": false, "description": "Add neighboring chunks to those that the similarity search retrieves, such that when selected, search returns i, i-1, and i+1.", "title": "addNeighborChunks", "type": "boolean" }, "maxDocumentsRetrievedPerPrompt": { "anyOf": [ { "maximum": 10, "minimum": 1, "type": "integer" }, { "type": "null" } ], "description": "The maximum number of chunks to retrieve from the vector database.", "title": "maxDocumentsRetrievedPerPrompt" }, "maxTokens": { "anyOf": [ { "maximum": 51200, "minimum": 1, "type": "integer" }, { "type": "null" } ], "description": "The maximum number of tokens to retrieve from the vector database.", "title": "maxTokens" }, "maximalMarginalRelevanceLambda": { "default": 0.5, "description": "Adjust the retrieval of chunks when using maximal marginal relevance to favor diversity (0.0) or similarity (1.0).", "maximum": 1, "minimum": 0, "title": "maximalMarginalRelevanceLambda", "type": "number" }, "retrievalMode": { "description": "Retrieval modes for vector databases.", "enum": [ "similarity", "maximal_marginal_relevance" ], "title": "RetrievalMode", "type": "string" }, "retriever": { "description": "The method used to retrieve relevant chunks from the vector database.", "enum": [ "SINGLE_LOOKUP_RETRIEVER", "CONVERSATIONAL_RETRIEVER", "MULTI_STEP_RETRIEVER" ], "title": "VectorDatabaseRetrievers", "type": "string" } }, "title": "VectorDatabaseSettings", "type": "object" }, { "type": "null" } ], "description": "A key/value dictionary of vector database settings." } }, "required": [ "id", "name", "description", "playgroundId", "promptType" ], "title": "LLMBlueprintSnapshot", "type": "object" }, "llmTestConfigurationId": { "description": "LLM test configuration ID this LLM result is associated to.", "title": "llmTestConfigurationId", "type": "string" }, "llmTestConfigurationName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "Name of the LLM test configuration this LLM result is associated to.", "title": "llmTestConfigurationName" }, "llmTestGradingCriteria": { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, "llmTestSuiteId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "LLM test suite ID to which the LLM test configuration is associated to.", "title": "llmTestSuiteId" }, "passPercentage": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "description": "The percentage of underlying insight evaluation results that have a pass grading result. if not specified, execution status is not completed.", "title": "passPercentage" }, "useCaseId": { "description": "Use case ID this LLM test result belongs to.", "title": "useCaseId", "type": "string" } }, "required": [ "id", "llmTestConfigurationId", "llmTestConfigurationName", "isOutOfTheBoxTestConfiguration", "useCaseId", "llmBlueprintId", "llmBlueprintSnapshot", "llmTestGradingCriteria", "executionStatus", "insightEvaluationResults", "creationDate", "creationUserId", "creationUserName" ], "title": "LLMTestResultResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | LLMTestResultResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List LLM test suites Operation path: `GET /api/v2/genai/llmTestSuites/` Authentication requirements: `BearerAuth` List LLM test suites. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | useCaseId | query | any | false | Only retrieve the LLM test suites associated with this use case ID. | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | sort | query | any | false | Apply this sort order to the results. Valid options are "name" and "creationDate". Prefix the attribute name with a dash to sort in descending order, e.g., sort=-creationDate. | ### Example responses > 200 Response ``` { "description": "Paginated list of LLM test suites.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "Llmtestsuite object formatted for API output.", "properties": { "creationDate": { "description": "The creation date of the chat (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the chat.", "title": "creationUserId", "type": "string" }, "description": { "description": "The description of the LLM test suite.", "title": "description", "type": "string" }, "id": { "description": "The ID of the LLM test suite.", "title": "id", "type": "string" }, "llmTestConfigurationIds": { "description": "The ids of the LLM test configurations in this LLM test suite.", "items": { "type": "string" }, "title": "llmTestConfigurationIds", "type": "array" }, "name": { "description": "The name of the LLM test suite.", "title": "name", "type": "string" }, "useCaseId": { "description": "The ID of the use case associated with the LLM test suite.", "title": "useCaseId", "type": "string" } }, "required": [ "id", "name", "description", "useCaseId", "llmTestConfigurationIds", "creationDate", "creationUserId" ], "title": "LLMTestSuiteResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListLLMTestSuitesResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | ListLLMTestSuitesResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Create LLM test suite Operation path: `POST /api/v2/genai/llmTestSuites/` Authentication requirements: `BearerAuth` Create a new LLM test suite. ### Body parameter ``` { "description": "The body of the \"create LLM test suite\" request.", "properties": { "description": { "default": "", "description": "The description of the LLM test suite.", "maxLength": 5000, "title": "description", "type": "string" }, "llmTestConfigurationIds": { "default": [], "description": "The ids of the LLM test configurations in the LLM test suite.", "items": { "type": "string" }, "maxItems": 100, "title": "llmTestConfigurationIds", "type": "array" }, "name": { "description": "The name of the LLM test suite.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "useCaseId": { "description": "The ID of the use case to associate with the LLM test suite.", "title": "useCaseId", "type": "string" } }, "required": [ "name", "useCaseId" ], "title": "CreateLLMTestSuiteRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | CreateLLMTestSuiteRequest | true | none | ### Example responses > 201 Response ``` { "description": "Llmtestsuite object formatted for API output.", "properties": { "creationDate": { "description": "The creation date of the chat (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the chat.", "title": "creationUserId", "type": "string" }, "description": { "description": "The description of the LLM test suite.", "title": "description", "type": "string" }, "id": { "description": "The ID of the LLM test suite.", "title": "id", "type": "string" }, "llmTestConfigurationIds": { "description": "The ids of the LLM test configurations in this LLM test suite.", "items": { "type": "string" }, "title": "llmTestConfigurationIds", "type": "array" }, "name": { "description": "The name of the LLM test suite.", "title": "name", "type": "string" }, "useCaseId": { "description": "The ID of the use case associated with the LLM test suite.", "title": "useCaseId", "type": "string" } }, "required": [ "id", "name", "description", "useCaseId", "llmTestConfigurationIds", "creationDate", "creationUserId" ], "title": "LLMTestSuiteResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 201 | Created | Successful Response | LLMTestSuiteResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Delete LLM test suite by LLM test suite ID Operation path: `DELETE /api/v2/genai/llmTestSuites/{llmTestSuiteId}/` Authentication requirements: `BearerAuth` Delete an existing LLM test suite. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmTestSuiteId | path | string | true | The ID of the LLM test suite to delete. | ### Example responses > 422 Response ``` { "properties": { "detail": { "items": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "title": "detail", "type": "array" } }, "title": "HTTPValidationErrorResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Successful Response | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Retrieve LLM test suite by LLM test suite ID Operation path: `GET /api/v2/genai/llmTestSuites/{llmTestSuiteId}/` Authentication requirements: `BearerAuth` Retrieve an existing LLM test suite. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmTestSuiteId | path | string | true | The ID of the LLM test suite to retrieve. | ### Example responses > 200 Response ``` { "description": "Llmtestsuite object formatted for API output.", "properties": { "creationDate": { "description": "The creation date of the chat (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the chat.", "title": "creationUserId", "type": "string" }, "description": { "description": "The description of the LLM test suite.", "title": "description", "type": "string" }, "id": { "description": "The ID of the LLM test suite.", "title": "id", "type": "string" }, "llmTestConfigurationIds": { "description": "The ids of the LLM test configurations in this LLM test suite.", "items": { "type": "string" }, "title": "llmTestConfigurationIds", "type": "array" }, "name": { "description": "The name of the LLM test suite.", "title": "name", "type": "string" }, "useCaseId": { "description": "The ID of the use case associated with the LLM test suite.", "title": "useCaseId", "type": "string" } }, "required": [ "id", "name", "description", "useCaseId", "llmTestConfigurationIds", "creationDate", "creationUserId" ], "title": "LLMTestSuiteResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | LLMTestSuiteResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Edit LLM test suite by LLM test suite ID Operation path: `PATCH /api/v2/genai/llmTestSuites/{llmTestSuiteId}/` Authentication requirements: `BearerAuth` Edit an existing LLM test suite. ### Body parameter ``` { "description": "The body of the \"edit LLM test suite\" request.", "properties": { "description": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The description of the LLM test suite.", "title": "description" }, "llmTestConfigurationIds": { "anyOf": [ { "items": { "type": "string" }, "maxItems": 100, "type": "array" }, { "type": "null" } ], "description": "The ids of the LLM test configurations in the LLM test suite.", "title": "llmTestConfigurationIds" }, "name": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the LLM test suite.", "title": "name" } }, "title": "EditLLMTestSuiteRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | llmTestSuiteId | path | string | true | The ID of the LLM test suite to edit. | | body | body | EditLLMTestSuiteRequest | true | none | ### Example responses > 200 Response ``` { "description": "Llmtestsuite object formatted for API output.", "properties": { "creationDate": { "description": "The creation date of the chat (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "creationUserId": { "description": "The ID of the user that created the chat.", "title": "creationUserId", "type": "string" }, "description": { "description": "The description of the LLM test suite.", "title": "description", "type": "string" }, "id": { "description": "The ID of the LLM test suite.", "title": "id", "type": "string" }, "llmTestConfigurationIds": { "description": "The ids of the LLM test configurations in this LLM test suite.", "items": { "type": "string" }, "title": "llmTestConfigurationIds", "type": "array" }, "name": { "description": "The name of the LLM test suite.", "title": "name", "type": "string" }, "useCaseId": { "description": "The ID of the use case associated with the LLM test suite.", "title": "useCaseId", "type": "string" } }, "required": [ "id", "name", "description", "useCaseId", "llmTestConfigurationIds", "creationDate", "creationUserId" ], "title": "LLMTestSuiteResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Successful Response | LLMTestSuiteResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Delete ootb metric configuration by ootb metric configuration ID Operation path: `DELETE /api/v2/genai/ootbMetricConfigurations/{ootbMetricConfigurationId}/` Authentication requirements: `BearerAuth` Delete single OOTB metric configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | ootbMetricConfigurationId | path | string | true | The ID of the metric configuration. | ### Example responses > 422 Response ``` { "properties": { "detail": { "items": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "title": "detail", "type": "array" } }, "title": "HTTPValidationErrorResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | OOTB metric configuration successfully deleted. | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Get ootb metric configuration by ootb metric configuration ID Operation path: `GET /api/v2/genai/ootbMetricConfigurations/{ootbMetricConfigurationId}/` Authentication requirements: `BearerAuth` Get OOTB metric configuration from the configuration. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | ootbMetricConfigurationId | path | string | true | The ID of the metric configuration. | ### Example responses > 200 Response ``` { "description": "API response object for a single ootb metric.", "properties": { "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation (if using a custom model llm).", "title": "customModelLLMValidationId" }, "customOotbMetricName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The custom ootb metric name to be associated with the ootb metric.", "title": "customOotbMetricName" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the ootb metric configuration.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "description": "Error type linking directly to the field name that is related to the error.", "enum": [ "ootbMetricName", "intervention", "guardCondition", "sidecarOverall", "sidecarRevalidate", "sidecarDeploymentId", "sidecarInputColumnName", "sidecarOutputColumnName", "promptPipelineFiles", "promptPipelineTemplateId", "responsePipelineFiles", "responsePipelineTemplateId" ], "title": "InsightErrorResolution", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "executionStatus": { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "isAgentic": { "default": false, "description": "Whether the ootb metric configuration is specific to agentic workflows.", "title": "isAgentic", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the LLM to use for `correctness` and `faithfulness` metrics.", "title": "llmId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration to be associated with the ootb metric." }, "ootbMetricConfigurationId": { "description": "The ID of ootb metric.", "title": "ootbMetricConfigurationId", "type": "string" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" } ], "description": "The name of the ootb metric.", "title": "ootbMetricName" } }, "required": [ "ootbMetricName", "ootbMetricConfigurationId", "executionStatus" ], "title": "OOTBMetricConfigurationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | OOTB metric configuration | OOTBMetricConfigurationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## List sidecar model metric validations Operation path: `GET /api/v2/genai/sidecarModelMetricValidations/` Authentication requirements: `BearerAuth` List sidecar model metric validations. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | useCaseId | query | any | false | Only retrieve the sidecar model metric validations associated with these use case IDs. | | offset | query | integer | false | Skip the specified number of values. | | limit | query | integer | false | Retrieve only the specified number of values. | | search | query | any | false | Only retrieve the sidecar model metric validations matching the search query. | | sort | query | any | false | Apply this sort order to the results. Valid options are "name", "deploymentName", "userName", "creationDate". Prefix the attribute name with a dash to sort in descending order, e.g., sort=-creationDate. | | completedOnly | query | boolean | false | If true, only retrieve the completed sidecar model metric validations. The default is false. | | deploymentId | query | any | false | Only retrieve the sidecar model metric validations associated with this deployment ID. | | modelId | query | any | false | Only retrieve the sidecar model metric validations associated with this model ID. | | promptColumnName | query | any | false | Only retrieve the sidecar model metric validations where the custom model uses this column name for prompt input. | | targetColumnName | query | any | false | Only retrieve the sidecar model metric validations where the custom model uses this column name for prediction output. | | citationsPrefixColumnName | query | any | false | Only retrieve the sidecar model metric validations where the custom model uses this column name prefix for citation inputs. | ### Example responses > 200 Response ``` { "description": "Paginated list of sidecar model metric validations.", "properties": { "count": { "description": "The number of records on this page.", "title": "count", "type": "integer" }, "data": { "description": "The list of records.", "items": { "description": "API response object for a single sidecar model metric validation.", "properties": { "citationsPrefixColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The column name prefix the custom model uses for citation inputs.", "title": "citationsPrefixColumnName" }, "creationDate": { "description": "The creation date of the custom model validation (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "deploymentAccessData": { "anyOf": [ { "description": "Add authorization_header to avoid breaking change to api.", "properties": { "authorizationHeader": { "default": "[REDACTED]", "description": "The `authorization` header to use for the deployment.", "title": "authorizationHeader", "type": "string" }, "chatApiUrl": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url of the deployment's chat api.", "title": "chatApiUrl" }, "datarobotKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The server key associated with the prediction api.", "title": "datarobotKey" }, "inputType": { "description": "The format of the input data submitted to a datarobot deployment.", "enum": [ "CSV", "JSON" ], "title": "DeploymentInputType", "type": "string" }, "modelType": { "description": "The type of the target output a datarobot deployment produces.", "enum": [ "TEXT_GENERATION", "VECTOR_DATABASE", "UNSTRUCTURED", "REGRESSION", "MULTICLASS", "BINARY", "NOT_SUPPORTED" ], "title": "SupportedDeploymentType", "type": "string" }, "predictionApiUrl": { "description": "The url of the deployment's prediction api.", "title": "predictionApiUrl", "type": "string" } }, "required": [ "predictionApiUrl", "datarobotKey", "inputType", "modelType" ], "title": "DeploymentAccessData", "type": "object" }, { "type": "null" } ], "description": "The parameters used for accessing the deployment." }, "deploymentId": { "description": "The ID of the custom model deployment.", "title": "deploymentId", "type": "string" }, "deploymentName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the custom model deployment.", "title": "deploymentName" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the validation error (if the validation failed).", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "description": "Error type linking directly to the field name that is related to the error.", "enum": [ "ootbMetricName", "intervention", "guardCondition", "sidecarOverall", "sidecarRevalidate", "sidecarDeploymentId", "sidecarInputColumnName", "sidecarOutputColumnName", "promptPipelineFiles", "promptPipelineTemplateId", "responsePipelineFiles", "responsePipelineTemplateId" ], "title": "InsightErrorResolution", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "id": { "description": "The ID of the custom model validation.", "title": "id", "type": "string" }, "modelId": { "description": "The ID of the model used in the deployment.", "title": "modelId", "type": "string" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the sidecar model metric." }, "name": { "description": "The name of the validated custom model.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the sidecar model metric validation.", "title": "playgroundId" }, "predictionTimeout": { "description": "The timeout in seconds for the prediction API used in this custom model validation.", "title": "predictionTimeout", "type": "integer" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName", "type": "string" }, "tenantId": { "description": "The ID of the tenant the custom model validation belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "useCaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the use case associated with the validated custom model.", "title": "useCaseId" }, "userId": { "description": "The ID of the user that created this custom model validation.", "title": "userId", "type": "string" }, "userName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the user that created this custom model validation.", "title": "userName" }, "validationStatus": { "description": "Status of custom model validation.", "enum": [ "TESTING", "PASSED", "FAILED" ], "title": "CustomModelValidationStatus", "type": "string" } }, "required": [ "id", "deploymentId", "targetColumnName", "validationStatus", "modelId", "deploymentAccessData", "tenantId", "name", "useCaseId", "creationDate", "userId", "predictionTimeout", "playgroundId", "citationsPrefixColumnName", "promptColumnName", "responseColumnName", "expectedResponseColumnName" ], "title": "SidecarModelMetricValidationResponse", "type": "object" }, "title": "data", "type": "array" }, "next": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the next page, or `null` if there is no such page.", "title": "next" }, "previous": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url to the previous page, or `null` if there is no such page.", "title": "previous" }, "totalCount": { "description": "The total number of records.", "title": "totalCount", "type": "integer" } }, "required": [ "totalCount", "count", "next", "previous", "data" ], "title": "ListSidecarModelMetricValidationnResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Sidecar model metric validations successfully retrieved. | ListSidecarModelMetricValidationnResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Validate sidecar model metric Operation path: `POST /api/v2/genai/sidecarModelMetricValidations/` Authentication requirements: `BearerAuth` Validate a metric hosted in a custom model deployment (also known as a sidecar model metric) for use in the playground. ### Body parameter ``` { "description": "The body of the \"validate sidecar model metric\" request.", "properties": { "citationsPrefixColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The column name prefix the custom model uses for citation inputs.", "title": "citationsPrefixColumnName" }, "deploymentId": { "description": "The ID of the custom model deployment.", "title": "deploymentId", "type": "string" }, "expectedResponseColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for the expected response text input.", "title": "expectedResponseColumnName" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model used in the deployment.", "title": "modelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration to be associated with the sidecar model metric." }, "name": { "default": "Untitled", "description": "The name to use for the validated custom model.", "maxLength": 5000, "title": "name", "type": "string" }, "playgroundId": { "description": "The ID of the playground to associate with the validated custom model.", "title": "playgroundId", "type": "string" }, "predictionTimeout": { "default": 300, "description": "The timeout in seconds for the prediction when validating a custom model. defaults to 300.", "maximum": 600, "minimum": 1, "title": "predictionTimeout", "type": "integer" }, "promptColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "description": "The name of the column the custom model uses for prediction output.", "maxLength": 5000, "title": "targetColumnName", "type": "string" }, "useCaseId": { "description": "The ID of the use case to associate with the validated custom model.", "title": "useCaseId", "type": "string" } }, "required": [ "deploymentId", "useCaseId", "playgroundId", "targetColumnName" ], "title": "CreateSidecarModelMetricValidationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | CreateSidecarModelMetricValidationRequest | true | none | ### Example responses > 202 Response ``` { "description": "API response object for a single sidecar model metric validation.", "properties": { "citationsPrefixColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The column name prefix the custom model uses for citation inputs.", "title": "citationsPrefixColumnName" }, "creationDate": { "description": "The creation date of the custom model validation (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "deploymentAccessData": { "anyOf": [ { "description": "Add authorization_header to avoid breaking change to api.", "properties": { "authorizationHeader": { "default": "[REDACTED]", "description": "The `authorization` header to use for the deployment.", "title": "authorizationHeader", "type": "string" }, "chatApiUrl": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url of the deployment's chat api.", "title": "chatApiUrl" }, "datarobotKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The server key associated with the prediction api.", "title": "datarobotKey" }, "inputType": { "description": "The format of the input data submitted to a datarobot deployment.", "enum": [ "CSV", "JSON" ], "title": "DeploymentInputType", "type": "string" }, "modelType": { "description": "The type of the target output a datarobot deployment produces.", "enum": [ "TEXT_GENERATION", "VECTOR_DATABASE", "UNSTRUCTURED", "REGRESSION", "MULTICLASS", "BINARY", "NOT_SUPPORTED" ], "title": "SupportedDeploymentType", "type": "string" }, "predictionApiUrl": { "description": "The url of the deployment's prediction api.", "title": "predictionApiUrl", "type": "string" } }, "required": [ "predictionApiUrl", "datarobotKey", "inputType", "modelType" ], "title": "DeploymentAccessData", "type": "object" }, { "type": "null" } ], "description": "The parameters used for accessing the deployment." }, "deploymentId": { "description": "The ID of the custom model deployment.", "title": "deploymentId", "type": "string" }, "deploymentName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the custom model deployment.", "title": "deploymentName" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the validation error (if the validation failed).", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "description": "Error type linking directly to the field name that is related to the error.", "enum": [ "ootbMetricName", "intervention", "guardCondition", "sidecarOverall", "sidecarRevalidate", "sidecarDeploymentId", "sidecarInputColumnName", "sidecarOutputColumnName", "promptPipelineFiles", "promptPipelineTemplateId", "responsePipelineFiles", "responsePipelineTemplateId" ], "title": "InsightErrorResolution", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "id": { "description": "The ID of the custom model validation.", "title": "id", "type": "string" }, "modelId": { "description": "The ID of the model used in the deployment.", "title": "modelId", "type": "string" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the sidecar model metric." }, "name": { "description": "The name of the validated custom model.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the sidecar model metric validation.", "title": "playgroundId" }, "predictionTimeout": { "description": "The timeout in seconds for the prediction API used in this custom model validation.", "title": "predictionTimeout", "type": "integer" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName", "type": "string" }, "tenantId": { "description": "The ID of the tenant the custom model validation belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "useCaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the use case associated with the validated custom model.", "title": "useCaseId" }, "userId": { "description": "The ID of the user that created this custom model validation.", "title": "userId", "type": "string" }, "userName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the user that created this custom model validation.", "title": "userName" }, "validationStatus": { "description": "Status of custom model validation.", "enum": [ "TESTING", "PASSED", "FAILED" ], "title": "CustomModelValidationStatus", "type": "string" } }, "required": [ "id", "deploymentId", "targetColumnName", "validationStatus", "modelId", "deploymentAccessData", "tenantId", "name", "useCaseId", "creationDate", "userId", "predictionTimeout", "playgroundId", "citationsPrefixColumnName", "promptColumnName", "responseColumnName", "expectedResponseColumnName" ], "title": "SidecarModelMetricValidationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 202 | Accepted | Sidecar model metric validation job successfully accepted. Follow the Location header to poll for job execution status. | SidecarModelMetricValidationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Delete sidecar model metric validation by validation ID Operation path: `DELETE /api/v2/genai/sidecarModelMetricValidations/{validationId}/` Authentication requirements: `BearerAuth` Delete an existing sidecar model metric validation. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | validationId | path | string | true | The ID of the sidecar model metric validation to delete. | ### Example responses > 422 Response ``` { "properties": { "detail": { "items": { "properties": { "loc": { "items": { "anyOf": [ { "type": "string" }, { "type": "integer" } ] }, "title": "loc", "type": "array" }, "msg": { "title": "msg", "type": "string" }, "type": { "title": "type", "type": "string" } }, "required": [ "loc", "msg", "type" ], "title": "ValidationError", "type": "object" }, "title": "detail", "type": "array" } }, "title": "HTTPValidationErrorResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 204 | No Content | Sidecar model metric validation successfully deleted. | None | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Retrieve sidecar model metric validation status by validation ID Operation path: `GET /api/v2/genai/sidecarModelMetricValidations/{validationId}/` Authentication requirements: `BearerAuth` Retrieve the status of validating a sidecar model metric. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | validationId | path | string | true | The ID of the sidecar model metric validation to retrieve. | ### Example responses > 200 Response ``` { "description": "API response object for a single sidecar model metric validation.", "properties": { "citationsPrefixColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The column name prefix the custom model uses for citation inputs.", "title": "citationsPrefixColumnName" }, "creationDate": { "description": "The creation date of the custom model validation (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "deploymentAccessData": { "anyOf": [ { "description": "Add authorization_header to avoid breaking change to api.", "properties": { "authorizationHeader": { "default": "[REDACTED]", "description": "The `authorization` header to use for the deployment.", "title": "authorizationHeader", "type": "string" }, "chatApiUrl": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url of the deployment's chat api.", "title": "chatApiUrl" }, "datarobotKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The server key associated with the prediction api.", "title": "datarobotKey" }, "inputType": { "description": "The format of the input data submitted to a datarobot deployment.", "enum": [ "CSV", "JSON" ], "title": "DeploymentInputType", "type": "string" }, "modelType": { "description": "The type of the target output a datarobot deployment produces.", "enum": [ "TEXT_GENERATION", "VECTOR_DATABASE", "UNSTRUCTURED", "REGRESSION", "MULTICLASS", "BINARY", "NOT_SUPPORTED" ], "title": "SupportedDeploymentType", "type": "string" }, "predictionApiUrl": { "description": "The url of the deployment's prediction api.", "title": "predictionApiUrl", "type": "string" } }, "required": [ "predictionApiUrl", "datarobotKey", "inputType", "modelType" ], "title": "DeploymentAccessData", "type": "object" }, { "type": "null" } ], "description": "The parameters used for accessing the deployment." }, "deploymentId": { "description": "The ID of the custom model deployment.", "title": "deploymentId", "type": "string" }, "deploymentName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the custom model deployment.", "title": "deploymentName" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the validation error (if the validation failed).", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "description": "Error type linking directly to the field name that is related to the error.", "enum": [ "ootbMetricName", "intervention", "guardCondition", "sidecarOverall", "sidecarRevalidate", "sidecarDeploymentId", "sidecarInputColumnName", "sidecarOutputColumnName", "promptPipelineFiles", "promptPipelineTemplateId", "responsePipelineFiles", "responsePipelineTemplateId" ], "title": "InsightErrorResolution", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "id": { "description": "The ID of the custom model validation.", "title": "id", "type": "string" }, "modelId": { "description": "The ID of the model used in the deployment.", "title": "modelId", "type": "string" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the sidecar model metric." }, "name": { "description": "The name of the validated custom model.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the sidecar model metric validation.", "title": "playgroundId" }, "predictionTimeout": { "description": "The timeout in seconds for the prediction API used in this custom model validation.", "title": "predictionTimeout", "type": "integer" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName", "type": "string" }, "tenantId": { "description": "The ID of the tenant the custom model validation belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "useCaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the use case associated with the validated custom model.", "title": "useCaseId" }, "userId": { "description": "The ID of the user that created this custom model validation.", "title": "userId", "type": "string" }, "userName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the user that created this custom model validation.", "title": "userName" }, "validationStatus": { "description": "Status of custom model validation.", "enum": [ "TESTING", "PASSED", "FAILED" ], "title": "CustomModelValidationStatus", "type": "string" } }, "required": [ "id", "deploymentId", "targetColumnName", "validationStatus", "modelId", "deploymentAccessData", "tenantId", "name", "useCaseId", "creationDate", "userId", "predictionTimeout", "playgroundId", "citationsPrefixColumnName", "promptColumnName", "responseColumnName", "expectedResponseColumnName" ], "title": "SidecarModelMetricValidationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Sidecar model metric validation status successfully retrieved. | SidecarModelMetricValidationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Edit sidecar model metric validation by validation ID Operation path: `PATCH /api/v2/genai/sidecarModelMetricValidations/{validationId}/` Authentication requirements: `BearerAuth` Edit an existing sidecar model metric validation. ### Body parameter ``` { "description": "The body of the \"edit sidecar model metric validation\" request.", "properties": { "chatModelId": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The model ID to specify when calling the openai chat completion API of the deployment. if this parameter is specified, the deployment must support the openai chat completion api.", "title": "chatModelId" }, "citationsPrefixColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "If specified, changes the column name prefix that will be used to submit the citation inputs to the sidecar model.", "title": "citationsPrefixColumnName" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "If specified, changes the ID of the deployment associated with this custom model validation.", "title": "deploymentId" }, "expectedResponseColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "If specified, changes the name of the column that will be used to submit the expected response text input to the sidecar model.", "title": "expectedResponseColumnName" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "If specified, changes the ID of the model associated with this custom model validation.", "title": "modelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration to be associated with the sidecar model metric." }, "name": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "If specified, renames the custom model validation to this value.", "title": "name" }, "predictionTimeout": { "anyOf": [ { "maximum": 600, "minimum": 1, "type": "integer" }, { "type": "null" } ], "description": "If specified, sets the timeout in seconds for the prediction when validating a custom model.", "title": "predictionTimeout" }, "promptColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "If specified, changes the name of the column that will be used to format the prompt text input for the custom model deployment.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "If specified, changes the name of the column that will be used to submit the response text input to the sidecar model.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "If specified, changes the name of the column that will be used to extract the prediction response from the custom model deployment.", "title": "targetColumnName" } }, "title": "EditSidecarModelMetricValidationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | validationId | path | string | true | The ID of the sidecar model metric validation to edit. | | body | body | EditSidecarModelMetricValidationRequest | true | none | ### Example responses > 200 Response ``` { "description": "API response object for a single sidecar model metric validation.", "properties": { "citationsPrefixColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The column name prefix the custom model uses for citation inputs.", "title": "citationsPrefixColumnName" }, "creationDate": { "description": "The creation date of the custom model validation (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "deploymentAccessData": { "anyOf": [ { "description": "Add authorization_header to avoid breaking change to api.", "properties": { "authorizationHeader": { "default": "[REDACTED]", "description": "The `authorization` header to use for the deployment.", "title": "authorizationHeader", "type": "string" }, "chatApiUrl": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url of the deployment's chat api.", "title": "chatApiUrl" }, "datarobotKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The server key associated with the prediction api.", "title": "datarobotKey" }, "inputType": { "description": "The format of the input data submitted to a datarobot deployment.", "enum": [ "CSV", "JSON" ], "title": "DeploymentInputType", "type": "string" }, "modelType": { "description": "The type of the target output a datarobot deployment produces.", "enum": [ "TEXT_GENERATION", "VECTOR_DATABASE", "UNSTRUCTURED", "REGRESSION", "MULTICLASS", "BINARY", "NOT_SUPPORTED" ], "title": "SupportedDeploymentType", "type": "string" }, "predictionApiUrl": { "description": "The url of the deployment's prediction api.", "title": "predictionApiUrl", "type": "string" } }, "required": [ "predictionApiUrl", "datarobotKey", "inputType", "modelType" ], "title": "DeploymentAccessData", "type": "object" }, { "type": "null" } ], "description": "The parameters used for accessing the deployment." }, "deploymentId": { "description": "The ID of the custom model deployment.", "title": "deploymentId", "type": "string" }, "deploymentName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the custom model deployment.", "title": "deploymentName" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the validation error (if the validation failed).", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "description": "Error type linking directly to the field name that is related to the error.", "enum": [ "ootbMetricName", "intervention", "guardCondition", "sidecarOverall", "sidecarRevalidate", "sidecarDeploymentId", "sidecarInputColumnName", "sidecarOutputColumnName", "promptPipelineFiles", "promptPipelineTemplateId", "responsePipelineFiles", "responsePipelineTemplateId" ], "title": "InsightErrorResolution", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "id": { "description": "The ID of the custom model validation.", "title": "id", "type": "string" }, "modelId": { "description": "The ID of the model used in the deployment.", "title": "modelId", "type": "string" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the sidecar model metric." }, "name": { "description": "The name of the validated custom model.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the sidecar model metric validation.", "title": "playgroundId" }, "predictionTimeout": { "description": "The timeout in seconds for the prediction API used in this custom model validation.", "title": "predictionTimeout", "type": "integer" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName", "type": "string" }, "tenantId": { "description": "The ID of the tenant the custom model validation belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "useCaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the use case associated with the validated custom model.", "title": "useCaseId" }, "userId": { "description": "The ID of the user that created this custom model validation.", "title": "userId", "type": "string" }, "userName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the user that created this custom model validation.", "title": "userName" }, "validationStatus": { "description": "Status of custom model validation.", "enum": [ "TESTING", "PASSED", "FAILED" ], "title": "CustomModelValidationStatus", "type": "string" } }, "required": [ "id", "deploymentId", "targetColumnName", "validationStatus", "modelId", "deploymentAccessData", "tenantId", "name", "useCaseId", "creationDate", "userId", "predictionTimeout", "playgroundId", "citationsPrefixColumnName", "promptColumnName", "responseColumnName", "expectedResponseColumnName" ], "title": "SidecarModelMetricValidationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Sidecar model metric validation successfully updated. | SidecarModelMetricValidationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Revalidate sidecar model metric by validation ID Operation path: `POST /api/v2/genai/sidecarModelMetricValidations/{validationId}/revalidate/` Authentication requirements: `BearerAuth` Revalidate an existing sidecar model metric validation. ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | validationId | path | string | true | The ID of the sidecar model metric validation to revalidate. | ### Example responses > 200 Response ``` { "description": "API response object for a single sidecar model metric validation.", "properties": { "citationsPrefixColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The column name prefix the custom model uses for citation inputs.", "title": "citationsPrefixColumnName" }, "creationDate": { "description": "The creation date of the custom model validation (iso 8601 formatted).", "format": "date-time", "title": "creationDate", "type": "string" }, "deploymentAccessData": { "anyOf": [ { "description": "Add authorization_header to avoid breaking change to api.", "properties": { "authorizationHeader": { "default": "[REDACTED]", "description": "The `authorization` header to use for the deployment.", "title": "authorizationHeader", "type": "string" }, "chatApiUrl": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The url of the deployment's chat api.", "title": "chatApiUrl" }, "datarobotKey": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The server key associated with the prediction api.", "title": "datarobotKey" }, "inputType": { "description": "The format of the input data submitted to a datarobot deployment.", "enum": [ "CSV", "JSON" ], "title": "DeploymentInputType", "type": "string" }, "modelType": { "description": "The type of the target output a datarobot deployment produces.", "enum": [ "TEXT_GENERATION", "VECTOR_DATABASE", "UNSTRUCTURED", "REGRESSION", "MULTICLASS", "BINARY", "NOT_SUPPORTED" ], "title": "SupportedDeploymentType", "type": "string" }, "predictionApiUrl": { "description": "The url of the deployment's prediction api.", "title": "predictionApiUrl", "type": "string" } }, "required": [ "predictionApiUrl", "datarobotKey", "inputType", "modelType" ], "title": "DeploymentAccessData", "type": "object" }, { "type": "null" } ], "description": "The parameters used for accessing the deployment." }, "deploymentId": { "description": "The ID of the custom model deployment.", "title": "deploymentId", "type": "string" }, "deploymentName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the custom model deployment.", "title": "deploymentName" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the validation error (if the validation failed).", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "description": "Error type linking directly to the field name that is related to the error.", "enum": [ "ootbMetricName", "intervention", "guardCondition", "sidecarOverall", "sidecarRevalidate", "sidecarDeploymentId", "sidecarInputColumnName", "sidecarOutputColumnName", "promptPipelineFiles", "promptPipelineTemplateId", "responsePipelineFiles", "responsePipelineTemplateId" ], "title": "InsightErrorResolution", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "id": { "description": "The ID of the custom model validation.", "title": "id", "type": "string" }, "modelId": { "description": "The ID of the model used in the deployment.", "title": "modelId", "type": "string" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the sidecar model metric." }, "name": { "description": "The name of the validated custom model.", "title": "name", "type": "string" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the sidecar model metric validation.", "title": "playgroundId" }, "predictionTimeout": { "description": "The timeout in seconds for the prediction API used in this custom model validation.", "title": "predictionTimeout", "type": "integer" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName", "type": "string" }, "tenantId": { "description": "The ID of the tenant the custom model validation belongs to.", "format": "uuid4", "title": "tenantId", "type": "string" }, "useCaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the use case associated with the validated custom model.", "title": "useCaseId" }, "userId": { "description": "The ID of the user that created this custom model validation.", "title": "userId", "type": "string" }, "userName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the user that created this custom model validation.", "title": "userName" }, "validationStatus": { "description": "Status of custom model validation.", "enum": [ "TESTING", "PASSED", "FAILED" ], "title": "CustomModelValidationStatus", "type": "string" } }, "required": [ "id", "deploymentId", "targetColumnName", "validationStatus", "modelId", "deploymentAccessData", "tenantId", "name", "useCaseId", "creationDate", "userId", "predictionTimeout", "playgroundId", "citationsPrefixColumnName", "promptColumnName", "responseColumnName", "expectedResponseColumnName" ], "title": "SidecarModelMetricValidationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 200 | OK | Sidecar model metric successfully revalidated. | SidecarModelMetricValidationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | ## Generate synthetic evaluation dataset Operation path: `POST /api/v2/genai/syntheticEvaluationDatasetGenerations/` Authentication requirements: `BearerAuth` Generate a synthetic evaluation dataset. ### Body parameter ``` { "description": "The body of the \"generate synthetic evaluation dataset\" request.", "properties": { "datasetName": { "anyOf": [ { "maxLength": 255, "type": "string" }, { "type": "null" } ], "description": "The name to use for the generated dataset.", "title": "datasetName" }, "language": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The language to use for the generated dataset.", "title": "language" }, "llmId": { "description": "The ID of the LLM to use for synthetic dataset generation.", "title": "llmId", "type": "string" }, "llmSettings": { "anyOf": [ { "additionalProperties": true, "description": "The settings that are available for all non-custom llms.", "properties": { "maxCompletionLength": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Maximum number of tokens allowed in the chat completion. use this value to, for example, control costs on token-based charges or manage response length for chat text limits.", "title": "maxCompletionLength" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "title": "CommonLLMSettings", "type": "object" }, { "additionalProperties": false, "description": "The settings that are available for custom model llms.", "properties": { "externalLlmContextSize": { "anyOf": [ { "maximum": 128000, "minimum": 128, "type": "integer" }, { "type": "null" } ], "default": 4096, "description": "The external llm's context size, in tokens. this value is only used for pruning documents supplied to the LLM when a vector database is associated with the LLM blueprint. it does not affect the external llm's actual context size in any way and is not supplied to the llm.", "title": "externalLlmContextSize" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" }, "validationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The validation ID of the custom model llm.", "title": "validationId" } }, "title": "CustomModelLLMSettings", "type": "object" }, { "additionalProperties": false, "description": "The settings that are available for custom model llms used via chat completion interface.", "properties": { "customModelId": { "description": "The ID of the custom model used via chat completion interface.", "title": "customModelId", "type": "string" }, "customModelVersionId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model version used via chat completion interface.", "title": "customModelVersionId" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "required": [ "customModelId" ], "title": "CustomModelChatLLMSettings", "type": "object" }, { "type": "null" } ], "description": "If specified, uses these LLM settings for the prompt and updates the settings of the corresponding chat or LLM blueprint to use these LLM settings.", "title": "llmSettings" }, "vectorDatabaseId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the vector database to use for synthetic dataset generation.", "title": "vectorDatabaseId" } }, "required": [ "llmId" ], "title": "SyntheticEvaluationDatasetGenerationRequest", "type": "object" } ``` ### Parameters | Name | In | Type | Required | Description | | --- | --- | --- | --- | --- | | body | body | SyntheticEvaluationDatasetGenerationRequest | true | none | ### Example responses > 202 Response ``` { "description": "The body of the \"create synthetic evaluation dataset\" response.", "properties": { "datasetId": { "description": "The ID of the created dataset.", "title": "datasetId", "type": "string" }, "promptColumnName": { "description": "The name of the dataset column containing the prompt text.", "title": "promptColumnName", "type": "string" }, "responseColumnName": { "description": "The name of the dataset column containing the response text.", "title": "responseColumnName", "type": "string" } }, "required": [ "datasetId", "promptColumnName", "responseColumnName" ], "title": "SyntheticEvaluationDatasetGenerationResponse", "type": "object" } ``` ### Responses | Status | Meaning | Description | Schema | | --- | --- | --- | --- | | 202 | Accepted | Synthetic evaluation data generation job successfully accepted. Follow the Location header to poll for job execution status. | SyntheticEvaluationDatasetGenerationResponse | | 422 | Unprocessable Entity | Validation Error | HTTPValidationErrorResponse | # Schemas ## AggregatedAggregationValue ``` { "description": "Aggregated record of multiple of the same item across different metric aggregation runs.", "properties": { "count": { "description": "The number of metric aggregation items aggregated.", "title": "count", "type": "integer" }, "item": { "description": "The name of the item.", "title": "item", "type": "string" }, "value": { "description": "The value associated with the item.", "title": "value", "type": "number" } }, "required": [ "item", "value", "count" ], "title": "AggregatedAggregationValue", "type": "object" } ``` AggregatedAggregationValue ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | count | integer | true | | The number of metric aggregation items aggregated. | | item | string | true | | The name of the item. | | value | number | true | | The value associated with the item. | ## AggregationType ``` { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" } ``` AggregationType ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | AggregationType | string | false | | The type of the metric aggregation. | ### Enumerated Values | Property | Value | | --- | --- | | AggregationType | [average, percentYes, classPercentCoverage, ngramImportance, guardConditionPercentYes] | ## AggregationValue ``` { "description": "An individual record in an itemized metric aggregation.", "properties": { "item": { "description": "The name of the item.", "title": "item", "type": "string" }, "value": { "description": "The value associated with the item.", "title": "value", "type": "number" } }, "required": [ "item", "value" ], "title": "AggregationValue", "type": "object" } ``` AggregationValue ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | item | string | true | | The name of the item. | | value | number | true | | The value associated with the item. | ## ArgumentMatchMode ``` { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } ``` ArgumentMatchMode ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | ArgumentMatchMode | string | false | | The different modes for comparing the arguments of tool calls. | ### Enumerated Values | Property | Value | | --- | --- | | ArgumentMatchMode | [exact_match, ignore_arguments] | ## CommonLLMSettings ``` { "additionalProperties": true, "description": "The settings that are available for all non-custom llms.", "properties": { "maxCompletionLength": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "description": "Maximum number of tokens allowed in the chat completion. use this value to, for example, control costs on token-based charges or manage response length for chat text limits.", "title": "maxCompletionLength" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "title": "CommonLLMSettings", "type": "object" } ``` CommonLLMSettings ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | maxCompletionLength | any | false | | Maximum number of tokens allowed in the chat completion. use this value to, for example, control costs on token-based charges or manage response length for chat text limits. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | systemPrompt | any | false | | System prompt guides the style of the LLM response. it is a "universal" prompt, prepended to all individual prompts. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## CostMetricConfigurationResponse ``` { "description": "API response object for a single cost metric configuration.", "properties": { "costConfigurationId": { "description": "The ID of the cost metric configuration.", "title": "costConfigurationId", "type": "string" }, "costMetricConfigurations": { "description": "The list of individual LLM cost configurations that constitute this cost metric configuration.", "items": { "description": "API request/response object for a cost configuration of a single llm.", "properties": { "currencyCode": { "default": "USD", "description": "The arbitrary code code of the currency of `inputtokenprice` and `outputtokenprice`.", "maxLength": 7, "title": "currencyCode", "type": "string" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation (if using a custom model llm).", "title": "customModelLLMValidationId" }, "inputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceinputtokencount` input tokens.", "minimum": 0, "title": "inputTokenPrice", "type": "number" }, "llmId": { "description": "The ID of the LLM associated with this cost configuration.", "title": "llmId", "type": "string" }, "outputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceoutputtokencount` output tokens.", "minimum": 0, "title": "outputTokenPrice", "type": "number" }, "referenceInputTokenCount": { "default": 1000, "description": "The number of input tokens corresponding to `inputtokenprice`.", "minimum": 0, "title": "referenceInputTokenCount", "type": "integer" }, "referenceOutputTokenCount": { "default": 1000, "description": "The number of output tokens corresponding to `outputtokenprice`.", "minimum": 0, "title": "referenceOutputTokenCount", "type": "integer" } }, "required": [ "llmId" ], "title": "LLMCostConfigurationResponse", "type": "object" }, "title": "costMetricConfigurations", "type": "array" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name to use for the cost configuration.", "title": "name" }, "playgroundId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the playground associated with the cost metric configuration.", "title": "playgroundId" }, "useCaseId": { "description": "The ID of the use case associated with the cost metric configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "costConfigurationId", "useCaseId", "costMetricConfigurations" ], "title": "CostMetricConfigurationResponse", "type": "object" } ``` CostMetricConfigurationResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | costConfigurationId | string | true | | The ID of the cost metric configuration. | | costMetricConfigurations | [LLMCostConfigurationResponse] | true | | The list of individual LLM cost configurations that constitute this cost metric configuration. | | name | any | false | | The name to use for the cost configuration. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | playgroundId | any | false | | The ID of the playground associated with the cost metric configuration. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | useCaseId | string | true | | The ID of the use case associated with the cost metric configuration. | ## CreateCostMetricConfigurationRequest ``` { "description": "The body of the \"create cost metric configuration\" request.", "properties": { "costMetricConfigurations": { "description": "The list of cost metric configurations to use.", "items": { "description": "API request/response object for a cost configuration of a single llm.", "properties": { "currencyCode": { "default": "USD", "description": "The arbitrary code code of the currency of `inputtokenprice` and `outputtokenprice`.", "maxLength": 7, "title": "currencyCode", "type": "string" }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation (if using a custom model llm).", "title": "customModelLLMValidationId" }, "inputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceinputtokencount` input tokens.", "minimum": 0, "title": "inputTokenPrice", "type": "number" }, "llmId": { "description": "The ID of the LLM associated with this cost configuration.", "title": "llmId", "type": "string" }, "outputTokenPrice": { "default": 0.01, "description": "The price of processing `referenceoutputtokencount` output tokens.", "minimum": 0, "title": "outputTokenPrice", "type": "number" }, "referenceInputTokenCount": { "default": 1000, "description": "The number of input tokens corresponding to `inputtokenprice`.", "minimum": 0, "title": "referenceInputTokenCount", "type": "integer" }, "referenceOutputTokenCount": { "default": 1000, "description": "The number of output tokens corresponding to `outputtokenprice`.", "minimum": 0, "title": "referenceOutputTokenCount", "type": "integer" } }, "required": [ "llmId" ], "title": "LLMCostConfigurationResponse", "type": "object" }, "title": "costMetricConfigurations", "type": "array" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name to use for the cost configuration.", "title": "name" }, "playgroundId": { "description": "The ID of the playground to associate with the cost metric configuration.", "title": "playgroundId", "type": "string" }, "useCaseId": { "description": "The ID of the use case to associate with the cost metric configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "useCaseId", "playgroundId", "costMetricConfigurations" ], "title": "CreateCostMetricConfigurationRequest", "type": "object" } ``` CreateCostMetricConfigurationRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | costMetricConfigurations | [LLMCostConfigurationResponse] | true | | The list of cost metric configurations to use. | | name | any | false | | The name to use for the cost configuration. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | playgroundId | string | true | | The ID of the playground to associate with the cost metric configuration. | | useCaseId | string | true | | The ID of the use case to associate with the cost metric configuration. | ## CreateEvaluationDatasetConfigurationRequest ``` { "description": "The body of the \"create evaluation dataset configuration\" request.", "properties": { "agentGoalsColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected agent goals. it is required to evaluate the agentgoalaccuracywithreference metric for agentic workflows.", "title": "agentGoalsColumnName" }, "correctnessEnabled": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "deprecated": true, "description": "Whether correctness is enabled for the evaluation dataset configuration.", "title": "correctnessEnabled" }, "datasetId": { "description": "The ID of the evaluation dataset.", "title": "datasetId", "type": "string" }, "isSyntheticDataset": { "default": false, "description": "Whether the evaluation dataset is synthetic.", "title": "isSyntheticDataset", "type": "boolean" }, "name": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name of the evaluation dataset configuration.", "title": "name" }, "playgroundId": { "description": "The ID of the playground to associate with the evaluation dataset configuration.", "title": "playgroundId", "type": "string" }, "promptColumnName": { "description": "The name of the dataset column containing the prompt text.", "title": "promptColumnName", "type": "string" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing the response text.", "title": "responseColumnName" }, "toolCallsColumnName": { "anyOf": [ { "maxLength": 5000, "minLength": 1, "type": "string" }, { "type": "null" } ], "description": "The name of the dataset column containing expected tool calls. it is required to evaluate the toolcallaccuracy metric for agentic workflows.", "title": "toolCallsColumnName" }, "useCaseId": { "description": "The ID of the use case to associate with the evaluation dataset configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "useCaseId", "playgroundId", "datasetId", "promptColumnName" ], "title": "CreateEvaluationDatasetConfigurationRequest", "type": "object" } ``` CreateEvaluationDatasetConfigurationRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | agentGoalsColumnName | any | false | | The name of the dataset column containing expected agent goals. it is required to evaluate the agentgoalaccuracywithreference metric for agentic workflows. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | correctnessEnabled | any | false | | Whether correctness is enabled for the evaluation dataset configuration. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | boolean | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | datasetId | string | true | | The ID of the evaluation dataset. | | isSyntheticDataset | boolean | false | | Whether the evaluation dataset is synthetic. | | name | any | false | | The name of the evaluation dataset configuration. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | playgroundId | string | true | | The ID of the playground to associate with the evaluation dataset configuration. | | promptColumnName | string | true | | The name of the dataset column containing the prompt text. | | responseColumnName | any | false | | The name of the dataset column containing the response text. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | toolCallsColumnName | any | false | | The name of the dataset column containing expected tool calls. it is required to evaluate the toolcallaccuracy metric for agentic workflows. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000minLength: 1minLength: 1 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | useCaseId | string | true | | The ID of the use case to associate with the evaluation dataset configuration. | ## CreateEvaluationDatasetMetricAggregationRequest ``` { "description": "The body of the \"create evaluation dataset metric aggregation\" request.", "properties": { "chatName": { "default": "Aggregated chat", "description": "The name for the new chat that will contain the associated prompts and responses.", "maxLength": 5000, "title": "chatName", "type": "string" }, "evaluationDatasetConfigurationId": { "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId", "type": "string" }, "insightsConfiguration": { "description": "The configuration of insights for the metric aggregation.", "items": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "minItems": 1, "title": "insightsConfiguration", "type": "array" }, "llmBlueprintIds": { "description": "The ids of the LLM blueprints to use for the metric aggregation.", "items": { "type": "string" }, "maxItems": 3, "minItems": 1, "title": "llmBlueprintIds", "type": "array" } }, "required": [ "llmBlueprintIds", "evaluationDatasetConfigurationId", "insightsConfiguration" ], "title": "CreateEvaluationDatasetMetricAggregationRequest", "type": "object" } ``` CreateEvaluationDatasetMetricAggregationRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | chatName | string | false | maxLength: 5000 | The name for the new chat that will contain the associated prompts and responses. | | evaluationDatasetConfigurationId | string | true | | The ID of the evaluation dataset configuration. | | insightsConfiguration | [InsightsConfigurationWithAdditionalData] | true | minItems: 1 | The configuration of insights for the metric aggregation. | | llmBlueprintIds | [string] | true | maxItems: 3minItems: 1 | The ids of the LLM blueprints to use for the metric aggregation. | ## CreateEvaluationDatasetMetricAggregationResponse ``` { "description": "The body of the \"create evaluation dataset metric aggregation\" response.", "properties": { "chatIds": { "description": "The ids of the chats associated with the metric aggregation.", "items": { "type": "string" }, "title": "chatIds", "type": "array" }, "jobId": { "description": "The ID of the evaluation dataset metric aggregation job.", "format": "uuid4", "title": "jobId", "type": "string" } }, "required": [ "jobId", "chatIds" ], "title": "CreateEvaluationDatasetMetricAggregationResponse", "type": "object" } ``` CreateEvaluationDatasetMetricAggregationResponse ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | chatIds | [string] | true | | The ids of the chats associated with the metric aggregation. | | jobId | string(uuid4) | true | | The ID of the evaluation dataset metric aggregation job. | ## CreateLLMTestConfigurationRequest ``` { "description": "Request object for creating a llmtestconfiguration.", "properties": { "datasetEvaluations": { "description": "Dataset evaluations.", "items": { "description": "Dataset evaluation.", "properties": { "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration for this dataset evaluation.", "title": "evaluationDatasetConfigurationId" }, "evaluationName": { "description": "The name of the evaluation. this name should provide context regarding what is being evaluated.", "maxLength": 5000, "minLength": 1, "title": "evaluationName", "type": "string" }, "insightConfiguration": { "description": "The configuration of insights with extra data.", "properties": { "aggregationTypes": { "anyOf": [ { "items": { "description": "The type of the metric aggregation.", "enum": [ "average", "percentYes", "classPercentCoverage", "ngramImportance", "guardConditionPercentYes" ], "title": "AggregationType", "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The aggregation types used in the insights configuration.", "title": "aggregationTypes" }, "costConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the cost configuration.", "title": "costConfigurationId" }, "customMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom metric (if using a custom metric).", "title": "customMetricId" }, "customModelGuard": { "anyOf": [ { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" }, { "type": "null" } ], "description": "Guard as configured in the custom model." }, "customModelLLMValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model LLM validation if using a custom model LLM for ootb metrics.", "title": "customModelLLMValidationId" }, "deploymentId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model deployment associated with the insight.", "title": "deploymentId" }, "errorMessage": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The error message associated with the evaluation dataset configuration or sidecar model metric validation or ootb metric.", "title": "errorMessage" }, "errorResolution": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "description": "The error type associated with the insight error status and error message as an indicator of what fields needs to be edited if any.", "title": "errorResolution" }, "evaluationDatasetConfigurationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the evaluation dataset configuration.", "title": "evaluationDatasetConfigurationId" }, "executionStatus": { "anyOf": [ { "description": "Job and entity execution status.", "enum": [ "NEW", "RUNNING", "COMPLETED", "REQUIRES_USER_INPUT", "SKIPPED", "ERROR" ], "title": "ExecutionStatus", "type": "string" }, { "type": "null" } ], "description": "The execution status of the evaluation dataset configuration." }, "extraMetricSettings": { "anyOf": [ { "description": "Extra settings for the metric that do not reference other entities.", "properties": { "toolCallAccuracy": { "anyOf": [ { "description": "Additional arguments for the tool call accuracy metric.", "properties": { "argumentComparison": { "description": "The different modes for comparing the arguments of tool calls.", "enum": [ "exact_match", "ignore_arguments" ], "title": "ArgumentMatchMode", "type": "string" } }, "required": [ "argumentComparison" ], "title": "ToolCallAccuracySettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the tool call accuracy metric." } }, "title": "ExtraMetricSettings", "type": "object" }, { "type": "null" } ], "description": "Extra settings for the metric that do not reference other entities." }, "insightName": { "description": "The name of the insight.", "maxLength": 5000, "minLength": 1, "title": "insightName", "type": "string" }, "insightType": { "anyOf": [ { "description": "The type of insight.", "enum": [ "Reference", "Quality metric", "Operational metric", "Evaluation deployment", "Custom metric", "Nemo" ], "title": "InsightTypes", "type": "string" }, { "type": "null" } ], "description": "The type of the insight." }, "isTransferable": { "default": false, "description": "Indicates if insight can be transferred to production.", "title": "isTransferable", "type": "boolean" }, "llmId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The LLM ID for ootb metrics that use llms.", "title": "llmId" }, "llmIsActive": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is active.", "title": "llmIsActive" }, "llmIsDeprecated": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "description": "Whether the LLM is deprecated and will be removed in a future release.", "title": "llmIsDeprecated" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model associated with `deploymentid`.", "title": "modelId" }, "modelPackageRegisteredModelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the registered model package associated with `deploymentid`.", "title": "modelPackageRegisteredModelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithID", "type": "object" }, { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration associated with the insight configuration.", "title": "moderationConfiguration" }, "nemoMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the nemo configuration.", "title": "nemoMetricId" }, "ootbMetricId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the ootb metric (if using an ootb metric).", "title": "ootbMetricId" }, "ootbMetricName": { "anyOf": [ { "description": "The out-of-the-box metric name that can be used in the playground.", "enum": [ "latency", "citations", "rouge_1", "faithfulness", "correctness", "prompt_tokens", "response_tokens", "document_tokens", "all_tokens", "jailbreak_violation", "toxicity_violation", "pii_violation", "exact_match", "starts_with", "contains" ], "title": "OOTBMetricInsightNames", "type": "string" }, { "description": "The out-of-the-box metric name that can be used in an agentic playground.", "enum": [ "tool_call_accuracy", "agent_goal_accuracy_with_reference" ], "title": "OOTBAgenticMetricInsightNames", "type": "string" }, { "description": "Metrics that can only be calculated using otel trace/metric data.", "enum": [ "agent_latency", "agent_tokens", "agent_cost" ], "title": "OTELMetricInsightNames", "type": "string" }, { "type": "null" } ], "description": "The ootb metric name.", "title": "ootbMetricName" }, "resultUnit": { "anyOf": [ { "description": "The unit of measurement associated with a metric.", "enum": [ "s", "ms", "%" ], "title": "MetricUnit", "type": "string" }, { "type": "null" } ], "description": "The unit of measurement associated with the insight result." }, "sidecarModelMetricMetadata": { "anyOf": [ { "description": "The metadata of a sidecar model metric.", "properties": { "expectedResponseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for expected response text input.", "title": "expectedResponseColumnName" }, "promptColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prediction output.", "title": "targetColumnName" } }, "required": [ "targetColumnName" ], "title": "SidecarModelMetricMetadata", "type": "object" }, { "type": "null" } ], "description": "The metadata of the sidecar model metric (if using a sidecar model metric)." }, "sidecarModelMetricValidationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the sidecar model metric validation (if using a sidecar model metric).", "title": "sidecarModelMetricValidationId" }, "stage": { "anyOf": [ { "description": "Enum that describes at which stage the metric may be calculated.", "enum": [ "prompt_pipeline", "response_pipeline" ], "title": "PipelineStage", "type": "string" }, { "type": "null" } ], "description": "The stage (prompt or response) where insight is calculated at." } }, "required": [ "insightName", "aggregationTypes" ], "title": "InsightsConfigurationWithAdditionalData", "type": "object" }, "insightGradingCriteria": { "description": "Grading criteria for an insight.", "properties": { "passThreshold": { "description": "The percentage threshold for pass result. greater than or equal to this threshold indicates a pass.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "InsightGradingCriteria", "type": "object" }, "maxNumPrompts": { "default": 0, "description": "The max number of prompts to evaluate.", "maximum": 5000, "minimum": 0, "title": "maxNumPrompts", "type": "integer" }, "ootbDatasetName": { "anyOf": [ { "description": "Out-of-the-box dataset name.", "enum": [ "jailbreak-v1.csv", "bbq-lite-age-v1.csv", "bbq-lite-gender-v1.csv", "bbq-lite-race-ethnicity-v1.csv", "bbq-lite-religion-v1.csv", "bbq-lite-disability-status-v1.csv", "bbq-lite-sexual-orientation-v1.csv", "bbq-lite-nationality-v1.csv", "bbq-lite-ses-v1.csv", "completeness-parent-v1.csv", "completeness-grandparent-v1.csv", "completeness-great-grandparent-v1.csv", "pii-v1.csv", "toxicity-v2.csv", "jbbq-age-v1.csv", "jbbq-gender-identity-v1.csv", "jbbq-physical-appearance-v1.csv", "jbbq-disability-status-v1.csv", "jbbq-sexual-orientation-v1.csv" ], "title": "OOTBDatasetName", "type": "string" }, { "type": "null" } ], "description": "Out-of-the-box evaluation dataset name. this applies only to our predefined public evaluation datasets." }, "promptSamplingStrategy": { "description": "The prompt sampling strategy for the evaluation dataset configuration.", "enum": [ "random_without_replacement", "first_n_rows" ], "title": "PromptSamplingStrategy", "type": "string" } }, "required": [ "evaluationName", "insightConfiguration", "insightGradingCriteria" ], "title": "DatasetEvaluationRequest", "type": "object" }, "maxItems": 10, "minItems": 1, "title": "datasetEvaluations", "type": "array" }, "description": { "default": "", "description": "LLM test configuration description.", "maxLength": 5000, "title": "description", "type": "string" }, "llmTestGradingCriteria": { "description": "Grading criteria for the LLM test configuration.", "properties": { "passThreshold": { "description": "The percentage threshold for pass results across dataset-insight pairs.", "maximum": 100, "minimum": 0, "title": "passThreshold", "type": "integer" } }, "required": [ "passThreshold" ], "title": "LLMTestGradingCriteria", "type": "object" }, "name": { "description": "LLM test configuration name.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "useCaseId": { "description": "The use case ID associated with the LLM test configuration.", "title": "useCaseId", "type": "string" } }, "required": [ "name", "useCaseId", "datasetEvaluations", "llmTestGradingCriteria" ], "title": "CreateLLMTestConfigurationRequest", "type": "object" } ``` CreateLLMTestConfigurationRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | datasetEvaluations | [DatasetEvaluationRequest] | true | maxItems: 10minItems: 1 | Dataset evaluations. | | description | string | false | maxLength: 5000 | LLM test configuration description. | | llmTestGradingCriteria | LLMTestGradingCriteria | true | | LLM test grading criteria. | | name | string | true | maxLength: 5000minLength: 1minLength: 1 | LLM test configuration name. | | useCaseId | string | true | | The use case ID associated with the LLM test configuration. | ## CreateLLMTestResultRequest ``` { "description": "Request object for creating a llmtestresult.", "properties": { "llmBlueprintId": { "description": "The LLM blueprint ID associated with the LLM test result.", "title": "llmBlueprintId", "type": "string" }, "llmTestConfigurationId": { "description": "The use case ID associated with the LLM test result.", "title": "llmTestConfigurationId", "type": "string" } }, "required": [ "llmTestConfigurationId", "llmBlueprintId" ], "title": "CreateLLMTestResultRequest", "type": "object" } ``` CreateLLMTestResultRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | llmBlueprintId | string | true | | The LLM blueprint ID associated with the LLM test result. | | llmTestConfigurationId | string | true | | The use case ID associated with the LLM test result. | ## CreateLLMTestSuiteRequest ``` { "description": "The body of the \"create LLM test suite\" request.", "properties": { "description": { "default": "", "description": "The description of the LLM test suite.", "maxLength": 5000, "title": "description", "type": "string" }, "llmTestConfigurationIds": { "default": [], "description": "The ids of the LLM test configurations in the LLM test suite.", "items": { "type": "string" }, "maxItems": 100, "title": "llmTestConfigurationIds", "type": "array" }, "name": { "description": "The name of the LLM test suite.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "useCaseId": { "description": "The ID of the use case to associate with the LLM test suite.", "title": "useCaseId", "type": "string" } }, "required": [ "name", "useCaseId" ], "title": "CreateLLMTestSuiteRequest", "type": "object" } ``` CreateLLMTestSuiteRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | description | string | false | maxLength: 5000 | The description of the LLM test suite. | | llmTestConfigurationIds | [string] | false | maxItems: 100 | The ids of the LLM test configurations in the LLM test suite. | | name | string | true | maxLength: 5000minLength: 1minLength: 1 | The name of the LLM test suite. | | useCaseId | string | true | | The ID of the use case to associate with the LLM test suite. | ## CreateSidecarModelMetricValidationRequest ``` { "description": "The body of the \"validate sidecar model metric\" request.", "properties": { "citationsPrefixColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The column name prefix the custom model uses for citation inputs.", "title": "citationsPrefixColumnName" }, "deploymentId": { "description": "The ID of the custom model deployment.", "title": "deploymentId", "type": "string" }, "expectedResponseColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for the expected response text input.", "title": "expectedResponseColumnName" }, "modelId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the model used in the deployment.", "title": "modelId" }, "moderationConfiguration": { "anyOf": [ { "description": "Moderation configuration associated with an insight.", "properties": { "guardConditions": { "description": "The guard conditions associated with a metric.", "items": { "description": "The guard condition for a metric.", "properties": { "comparand": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "items": { "type": "string" }, "type": "array" } ], "description": "The comparand(s) used in the guard condition.", "title": "comparand" }, "comparator": { "description": "The comparator used in a guard condition.", "enum": [ "greaterThan", "lessThan", "equals", "notEquals", "is", "isNot", "matches", "doesNotMatch", "contains", "doesNotContain" ], "title": "GuardConditionComparator", "type": "string" } }, "required": [ "comparator", "comparand" ], "title": "GuardCondition", "type": "object" }, "maxItems": 1, "minItems": 1, "title": "guardConditions", "type": "array" }, "intervention": { "description": "The intervention configuration for a metric.", "properties": { "action": { "description": "The moderation strategy.", "enum": [ "block", "report", "reportAndBlock" ], "title": "ModerationAction", "type": "string" }, "message": { "description": "The intervention message to replace the prediction when a guard condition is satisfied.", "minLength": 1, "title": "message", "type": "string" } }, "required": [ "action", "message" ], "title": "Intervention", "type": "object" } }, "required": [ "guardConditions", "intervention" ], "title": "ModerationConfigurationWithoutID", "type": "object" }, { "type": "null" } ], "description": "The moderation configuration to be associated with the sidecar model metric." }, "name": { "default": "Untitled", "description": "The name to use for the validated custom model.", "maxLength": 5000, "title": "name", "type": "string" }, "playgroundId": { "description": "The ID of the playground to associate with the validated custom model.", "title": "playgroundId", "type": "string" }, "predictionTimeout": { "default": 300, "description": "The timeout in seconds for the prediction when validating a custom model. defaults to 300.", "maximum": 600, "minimum": 1, "title": "predictionTimeout", "type": "integer" }, "promptColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for prompt text input.", "title": "promptColumnName" }, "responseColumnName": { "anyOf": [ { "maxLength": 5000, "type": "string" }, { "type": "null" } ], "description": "The name of the column the custom model uses for response text input.", "title": "responseColumnName" }, "targetColumnName": { "description": "The name of the column the custom model uses for prediction output.", "maxLength": 5000, "title": "targetColumnName", "type": "string" }, "useCaseId": { "description": "The ID of the use case to associate with the validated custom model.", "title": "useCaseId", "type": "string" } }, "required": [ "deploymentId", "useCaseId", "playgroundId", "targetColumnName" ], "title": "CreateSidecarModelMetricValidationRequest", "type": "object" } ``` CreateSidecarModelMetricValidationRequest ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | citationsPrefixColumnName | any | false | | The column name prefix the custom model uses for citation inputs. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | deploymentId | string | true | | The ID of the custom model deployment. | | expectedResponseColumnName | any | false | | The name of the column the custom model uses for the expected response text input. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | modelId | any | false | | The ID of the model used in the deployment. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | moderationConfiguration | any | false | | The moderation configuration to be associated with the sidecar model metric. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | ModerationConfigurationWithoutID | false | | Moderation configuration associated with an insight. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | name | string | false | maxLength: 5000 | The name to use for the validated custom model. | | playgroundId | string | true | | The ID of the playground to associate with the validated custom model. | | predictionTimeout | integer | false | maximum: 600minimum: 1 | The timeout in seconds for the prediction when validating a custom model. defaults to 300. | | promptColumnName | any | false | | The name of the column the custom model uses for prompt text input. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | responseColumnName | any | false | | The name of the column the custom model uses for response text input. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | targetColumnName | string | true | maxLength: 5000 | The name of the column the custom model uses for prediction output. | | useCaseId | string | true | | The ID of the use case to associate with the validated custom model. | ## CustomModelChatLLMSettings ``` { "additionalProperties": false, "description": "The settings that are available for custom model llms used via chat completion interface.", "properties": { "customModelId": { "description": "The ID of the custom model used via chat completion interface.", "title": "customModelId", "type": "string" }, "customModelVersionId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the custom model version used via chat completion interface.", "title": "customModelVersionId" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" } }, "required": [ "customModelId" ], "title": "CustomModelChatLLMSettings", "type": "object" } ``` CustomModelChatLLMSettings ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | customModelId | string | true | | The ID of the custom model used via chat completion interface. | | customModelVersionId | any | false | | The ID of the custom model version used via chat completion interface. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | systemPrompt | any | false | | System prompt guides the style of the LLM response. it is a "universal" prompt, prepended to all individual prompts. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | string | false | maxLength: 5000000 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | ## CustomModelGuard ``` { "description": "Details of a guard as defined for the custom model.", "properties": { "name": { "description": "The name of the guard.", "maxLength": 5000, "minLength": 1, "title": "name", "type": "string" }, "nemoEvaluatorType": { "anyOf": [ { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" }, { "type": "null" } ], "description": "Nemo evaluator type of the guard." }, "ootbType": { "anyOf": [ { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" }, { "type": "null" } ], "description": "Out of the box type of the guard." }, "stage": { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" }, "type": { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } }, "required": [ "type", "stage", "name" ], "title": "CustomModelGuard", "type": "object" } ``` CustomModelGuard ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | name | string | true | maxLength: 5000minLength: 1minLength: 1 | The name of the guard. | | nemoEvaluatorType | any | false | | Nemo evaluator type of the guard. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | CustomModelGuardNemoEvaluatorType | false | | Nemo evaluator type as used in the moderation_config.yaml file of the custom model. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | ootbType | any | false | | Out of the box type of the guard. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | CustomModelGuardOOTBType | false | | Ootb type as used in the moderation_config.yaml file of the custom model. | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | stage | CustomModelGuardStage | true | | Stage on which the guard gets applied. | | type | CustomModelGuardType | true | | Type of the guard. | ## CustomModelGuardNemoEvaluatorType ``` { "description": "Nemo evaluator type as used in the moderation_config.yaml file of the custom model.", "enum": [ "llm_judge", "context_relevance", "response_groundedness", "topic_adherence", "agent_goal_accuracy", "response_relevancy", "faithfulness" ], "title": "CustomModelGuardNemoEvaluatorType", "type": "string" } ``` CustomModelGuardNemoEvaluatorType ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | CustomModelGuardNemoEvaluatorType | string | false | | Nemo evaluator type as used in the moderation_config.yaml file of the custom model. | ### Enumerated Values | Property | Value | | --- | --- | | CustomModelGuardNemoEvaluatorType | [llm_judge, context_relevance, response_groundedness, topic_adherence, agent_goal_accuracy, response_relevancy, faithfulness] | ## CustomModelGuardOOTBType ``` { "description": "Ootb type as used in the moderation_config.yaml file of the custom model.", "enum": [ "token_count", "rouge_1", "faithfulness", "agent_goal_accuracy", "custom_metric", "cost", "task_adherence" ], "title": "CustomModelGuardOOTBType", "type": "string" } ``` CustomModelGuardOOTBType ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | CustomModelGuardOOTBType | string | false | | Ootb type as used in the moderation_config.yaml file of the custom model. | ### Enumerated Values | Property | Value | | --- | --- | | CustomModelGuardOOTBType | [token_count, rouge_1, faithfulness, agent_goal_accuracy, custom_metric, cost, task_adherence] | ## CustomModelGuardStage ``` { "description": "Guard stage as used in the moderation_config.yaml file of the custom model.", "enum": [ "prompt", "response" ], "title": "CustomModelGuardStage", "type": "string" } ``` CustomModelGuardStage ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | CustomModelGuardStage | string | false | | Guard stage as used in the moderation_config.yaml file of the custom model. | ### Enumerated Values | Property | Value | | --- | --- | | CustomModelGuardStage | [prompt, response] | ## CustomModelGuardType ``` { "description": "Guard type as used in the moderation_config.yaml file of the custom model.", "enum": [ "ootb", "model", "nemo_guardrails", "nemo_evaluator" ], "title": "CustomModelGuardType", "type": "string" } ``` CustomModelGuardType ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | CustomModelGuardType | string | false | | Guard type as used in the moderation_config.yaml file of the custom model. | ### Enumerated Values | Property | Value | | --- | --- | | CustomModelGuardType | [ootb, model, nemo_guardrails, nemo_evaluator] | ## CustomModelLLMSettings ``` { "additionalProperties": false, "description": "The settings that are available for custom model llms.", "properties": { "externalLlmContextSize": { "anyOf": [ { "maximum": 128000, "minimum": 128, "type": "integer" }, { "type": "null" } ], "default": 4096, "description": "The external llm's context size, in tokens. this value is only used for pruning documents supplied to the LLM when a vector database is associated with the LLM blueprint. it does not affect the external llm's actual context size in any way and is not supplied to the llm.", "title": "externalLlmContextSize" }, "systemPrompt": { "anyOf": [ { "maxLength": 5000000, "type": "string" }, { "type": "null" } ], "description": "System prompt guides the style of the LLM response. it is a \"universal\" prompt, prepended to all individual prompts.", "title": "systemPrompt" }, "validationId": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The validation ID of the custom model llm.", "title": "validationId" } }, "title": "CustomModelLLMSettings", "type": "object" } ``` CustomModelLLMSettings ### Properties | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | externalLlmContextSize | any | false | | The external llm's context size, in tokens. this value is only used for pruning documents supplied to the LLM when a vector database is associated with the LLM blueprint. it does not affect the external llm's actual context size in any way and is not supplied to the llm. | anyOf | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | integer | false | maximum: 128000minimum: 128 | none | or | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | » anonymous | null | false | | none | continued | Name | Type | Required | Restrictions | Description | | --- | --- | --- | --- | --- | | systemPrompt | an