開発者向けドキュメント > 開発者向け学習 > Python API client user guide > MLOps > チャレンジャーモデル

チャレンジャーモデル¶

Challenger models are alternative models that you can compare against your currently deployed model (the champion model). This allows you to test new models in production, compare their performance, and make data-driven decisions about whether to replace the champion with a better-performing challenger. This page describes how to create, manage, and work with challenger models.

ベストプラクティス¶

When to use challengers¶

A/B testing: Test new models against the current champion in production.
Model validation: Validate that a new model performs well before replacing the champion.
Performance comparison: Compare multiple model candidates simultaneously.
Risk mitigation: Test models with different characteristics (e.g., different algorithms, feature sets).

Challenger management¶

Naming convention: Use descriptive names that indicate the model type or purpose (e.g., "XGBoost_v2_Challenger").
Limit active challenger models: Too many challengers can impact prediction performance; typically 2-3 challengers is sufficient.
Monitor performance: Regularly review challenger performance metrics before deciding to promote one to champion.
Clean up: Remove challengers that are no longer being evaluated to keep your deployment clean.
Document: Keep track of why each challenger was created and what makes it different from the champion.

前提条件¶

Before creating a challenger model, ensure you have:

A registered model version (model package) ready to use.
An appropriate prediction environment configured.
Challenger models enabled for the deployment.
An understanding of what you want to test or compare.

Create a challenger model¶

To create a challenger model, you need a deployment, a model package (registered model version), and a prediction environment. The challenger will use the specified model package and prediction environment to make predictions alongside the champion model.

Basic challenger model creation¶

import datarobot as dr

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')

# Get a model package (registered model version) to use as challenger
project = dr.Project.get('6527eb38b9e5dead5fc12491')
model = project.get_models()[0]
registered_model_version = dr.RegisteredModelVersion.create_for_leaderboard_item(
    model_id=model.id,
    name="Challenger Model Version",
    registered_model_name='My Registered Model'
)

# Get a prediction environment
prediction_environments = dr.PredictionEnvironment.list()
prediction_environment = prediction_environments[0]

# Create the challenger
challenger = dr.Challenger.create(
    deployment_id=deployment.id,
    model_package_id=registered_model_version.id,
    prediction_environment_id=prediction_environment.id,
    name='Elastic-Net Classifier Challenger'
)

Create a challenger that waits for completion¶

By default, challenger creation is asynchronous. You can specify a maximum wait time for the creation to complete:

# Wait up to 600 seconds for creation to complete
challenger = dr.Challenger.create(
    deployment_id=deployment.id,
    model_package_id=registered_model_version.id,
    prediction_environment_id=prediction_environment.id,
    name='Random Forest Challenger',
    max_wait=600
)

List challengers¶

You can retrieve all challengers associated with a deployment.

List all challengers for a deployment¶

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
challengers = dr.Challenger.list(deployment_id=deployment.id)

You can also use the list_challengers() method:

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
challengers = deployment.list_challengers()

for challenger in challengers:
    print(f"{challenger.name}: {challenger.id}")

Get a specific challenger¶

To retrieve a single challenger by its ID:

challenger = dr.Challenger.get(
    deployment_id='5c939e08962d741e34f609f0',
    challenger_id='5c939e08962d741e34f609f1'
)

Access challenger properties¶

challenger = dr.Challenger.get(
    deployment_id=deployment.id,
    challenger_id='5c939e08962d741e34f609f1'
)

Update a challenger model¶

You can update a challenger model's name and prediction environment.

Update a challenger model name¶

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
challenger = deployment.list_challengers()[0]
challenger.update(name='Updated Challenger Name')

Update the prediction environment¶

# Get a different prediction environment
prediction_environments = dr.PredictionEnvironment.list()
new_environment = prediction_environments[1]

challenger.update(prediction_environment_id=new_environment.id)

To update both the name and prediction environment:

challenger.update(
    name='Final Challenger Name',
    prediction_environment_id=new_environment.id
)

Delete a challenger¶

To remove a challenger from a deployment:

challenger = dr.Challenger.get(
    deployment_id=deployment.id,
    challenger_id='5c939e08962d741e34f609f1'
)
challenger.delete()

# Verify deletion
challengers = deployment.list_challengers()
challenger_ids = [c.id for c in challengers]

Manage challenger settings¶

You can enable or disable challenger models for a deployment and configure challenger-related settings.

Get challenger settings¶

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
settings = deployment.get_challenger_models_settings()

Update challenger settings¶

To enable or disable challenger models for a deployment:

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
deployment.update_challenger_models_settings(challenger_models_enabled=True)

To disable challenger models:

deployment.update_challenger_models_settings(challenger_models_enabled=False)

Work with challenger predictions¶

Challengers make predictions alongside the champion model, allowing you to compare their performance.

Understanding challenger predictions¶

When challengers are enabled, predictions made to the deployment will also be scored by the challenger models. This allows you to:

Compare prediction outputs between champion and challengers.
Monitor challenger performance metrics.
Make informed decisions about model replacement.

Score challenger models¶

You can trigger challenger scoring for existing prediction requests:

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
# Score challengers for predictions with a specific timestamp
deployment.score_challenger_predictions(timestamp='2024-01-15T10:00:00Z')

一般的なワークフロー¶

Create multiple challengers from different models¶

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
project = dr.Deployment.get(deployment_id=deployment.id).model['project_id']
project = dr.Project.get(project)

# Get top 3 models from the project
models = project.get_models()[:3]
prediction_environment = dr.PredictionEnvironment.list()[0]

challengers = []
for i, model in enumerate(models):
    registered_model_version = dr.RegisteredModelVersion.create_for_leaderboard_item(
        model_id=model.id,
        name=f"Challenger {i+1}",
        registered_model_name=f'Challenger Model {i+1}'
    )
    challenger = dr.Challenger.create(
        deployment_id=deployment.id,
        model_package_id=registered_model_version.id,
        prediction_environment_id=prediction_environment.id,
        name=f'{model.model_type} Challenger'
    )
    challengers.append(challenger)

print(f"Created {len(challengers)} challengers")

Compare challenger information¶

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
challengers = deployment.list_challengers()

print("Challenger Comparison:")
print(f"{'Name':<40} {'Model Type':<30} {'Model Package ID':<20}")
print("-" * 90)
for challenger in challengers:
    model_type = challenger.model.get('type', 'Unknown')
    model_package_id = challenger.model_package.get('id', 'Unknown')
    print(f"{challenger.name:<40} {model_type:<30} {model_package_id:<20}")

Clean up old challenger models¶

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')
challengers = deployment.list_challengers()

# Delete challengers older than a certain date or based on criteria
# Example: delete challengers with specific naming pattern
for challenger in challengers:
    if 'Old' in challenger.name:
        print(f"Deleting challenger: {challenger.name}")
        challenger.delete()

Replace a challenger with a new model¶

deployment = dr.Deployment.get(deployment_id='5c939e08962d741e34f609f0')

# Get existing challenger
old_challenger = deployment.list_challengers()[0]

# Create new model version
# Get a different model
new_model = project.get_models()[1]
new_registered_model_version = dr.RegisteredModelVersion.create_for_leaderboard_item(
    model_id=new_model.id,
    name="Updated Challenger Version",
    registered_model_name='Challenger Model'
)

# Delete old challenger
old_challenger.delete()

# Create new challenger with same name but new model
new_challenger = dr.Challenger.create(
    deployment_id=deployment.id,
    model_package_id=new_registered_model_version.id,
    prediction_environment_id=old_challenger.prediction_environment['id'],
    name=old_challenger.name
)

注意事項¶

Challenger creation is an asynchronous process. The max_wait parameter controls how long to wait for creation to complete.
A deployment can have multiple challenger models active simultaneously.
Challengers use the same prediction requests as the champion, allowing for direct comparison.
The champion model (the currently deployed model) cannot be deleted while challengers exist that reference it.
Challenger models must use compatible prediction environments with the deployment.
Model packages used for challengers must have the same target type and compatible settings as the champion model.