LRS topology spread¶

The LRS operator places the replicas of every deployment-backed Long Running Service (LRS), such as custom model deployments, custom applications, and NIM deployments, using Kubernetes pod topology spread constraints. The operator writes two constraints, both configured cluster-wide in the lrs-operator chart:

• Zone spread (zoneTopologySpread): Spreads replicas across availability zones.
• Host spread (hostTopologySpread): Spreads replicas across nodes.

Starting with DataRobot 11.14, the spread constraint settings replace the two per-LRS settings: ha_topology_key pod anti-affinity and topology_spread_constraints. Both of these fields have been replaced and are ignored for deployment-backed LRS. Job-backed LRS keep the ha_topology_key anti-affinity.

デフォルト値¶

On-premise and other self-managed installations use the lrs-operator chart defaults unless you override them:

lrs-operator:
  operator:
    config:
      zoneTopologySpread:
        enabled: true
        topologyKey: topology.kubernetes.io/zone
        maxSkew: 1
        matchLabelKeys: pod-template-hash
      hostTopologySpread:
        enabled: true
        whenUnsatisfiable: ScheduleAnyway
        topologyKey: kubernetes.io/hostname
        maxSkew: 1
        minDomains: 3
        hardMinReplicas: 3
        enableReplicaDerivedMaxSkewRule: true
        matchLabelKeys: pod-template-hash
        reconcileDrift: false 

With these defaults, both spreads are scheduler preferences (whenUnsatisfiable: ScheduleAnyway). The scheduler places replicas in different zones and on different nodes when it can, and co-locates them when it can't. A pod never stays Pending because of the spread. Existing deployments keep their placement rules when the settings change; see Existing deployments.

Settings reference¶

This page uses the following terms:

• Effective replica count: auto_scale.min_replicas for an autoscaled LRS, and the configured replica count for any other LRS.
• Eligible node: A node that matches the LRS node selector or affinity, for example a dedicated GPU node group, and whose taints the LRS pods tolerate.

Both spreads honor node taints (nodeTaintsPolicy: Honor). Nodes with taints the LRS pods don’t tolerate are left out of the skew calculation, and so are zones that contain only such nodes.

For how a preference differs from a requirement, see Preference or requirement.

Zone spread¶

The zone spread is always a preference (ScheduleAnyway).

Host spread¶

The host spread is a preference by default. Set whenUnsatisfiable: DoNotSchedule to enforce it for deployments with at least hardMinReplicas effective replicas.

How the settings affect workloads¶

Preference or requirement¶

The main choice is hostTopologySpread.whenUnsatisfiable:

• ScheduleAnyway (default): Spreads replicas when the cluster has room, and never blocks scheduling. Replicas can still end up on the same node, for example when only one node in a dedicated node group has free capacity. If that node fails, every replica of the deployment goes down.
• DoNotSchedule: Makes the spread a requirement for deployments with at least hardMinReplicas effective replicas. No node can hold more than maxSkew replicas above the eligible node with the fewest, so the replicas run on at least ceil(effective replicas / maxSkew) nodes. If the cluster doesn't have enough eligible nodes with free capacity, the extra replicas stay Pending until capacity is added. Smaller deployments still treat the spread as a preference.

For example, with DoNotSchedule and the default minDomains: 3, a 3-replica deployment needs 3 eligible nodes, each with room for one replica. With only 2 eligible nodes, one replica stays Pending.

The replica-derived max skew rule (enableReplicaDerivedMaxSkewRule) keeps an enforced spread from needing one node per replica. It raises each deployment's maxSkew to max(maxSkew, ceil(effective replicas / minDomains)). With the defaults (maxSkew: 1 and minDomains: 3), the rule produces the following values::

A 4-replica deployment needs only 2 nodes because its maxSkew rises to 2, so each node can hold 2 replicas.

Without the rule, maxSkew stays at 1, so 12 replicas need 12 nodes, and a cluster autoscaler adds a node for each replica that doesn't fit. With ScheduleAnyway, the rule has no effect.

Pod rollouts¶

The spread is part of the pod template. When the operator writes it to an existing deployment, the deployment's pods roll. The operator writes the spread:

• When it creates a deployment.
• When a deployment's effective replica count changes, for example when a user changes the replica count or the autoscaling minimum.
• When it reconciles an LRS, if reconcileDrift is true. See Existing deployments.

Changing the operator settings doesn't update existing deployments on its own. Each deployment picks up the new settings the next time its effective replica count changes, or at the next reconcile if reconcileDrift is true.

既存のデプロイ¶

Existing deployments keep their previous placement rules until their effective replica count changes. This applies in two cases:

• Deployments created before DataRobot 11.14 keep the ha_topology_key pod anti-affinity.
• Deployments created before a change to the spread settings keep the spread constraints of the earlier settings.

If reconcileDrift is true, the operator checks deployments each time it reconciles an LRS, including every LRS when the operator starts. It rewrites any deployment that:

• Has at least hardMinReplicas effective replicas.
• Still has pod anti-affinity, or has topology spread constraints that differ from the current settings.

For each matching deployment, the operator removes any pod anti-affinity and writes the current spread. This rolls the deployment's pods once. Deployments below hardMinReplicas keep their rules until their replica count changes.

Before you turn on reconcileDrift on a cluster with existing deployments, plan for the following:

• The next time the operator starts, every matching deployment rolls its pods.
• During the rollout, the old pods' anti-affinity still keeps new pods off the nodes the old pods run on. The cluster needs spare capacity, or a cluster autoscaler, to place the new pods.
• While reconcileDrift is true, any later change to a spread setting, such as maxSkew, rolls all matching deployments again.

Choose settings for an installation or upgrade¶

Decide on the following before you install DataRobot 11.14 or later, or upgrade to it:

1. Choose whether to enforce the host spread. Keep ScheduleAnyway if the cluster has a fixed number of nodes, no cluster autoscaler, or small dedicated node groups (for example GPU nodes). Set DoNotSchedule if deployments with 3 or more replicas must survive the loss of a node, and the cluster can always provide at least minDomains eligible nodes for every node group that runs LRS, through spare capacity or an autoscaler.

2. If you enforce the host spread, set minDomains and hardMinReplicas.

   • minDomains sets how many eligible nodes the scheduler expects and, through the replica-derived max skew rule, how many nodes each enforced deployment is spread across. Don't set it higher than the number of nodes in your smallest LRS node group.
   • hardMinReplicas sets the smallest deployment the spread is enforced for. Smaller deployments still treat the spread as a preference. Set it to at least minDomains.

   With ScheduleAnyway, only hardMinReplicas has an effect, as the reconcileDrift threshold.

3. Choose whether to keep the zone spread. The zone spread never blocks scheduling and has no effect on single-zone clusters, so it's safe to leave on. If your nodes use a label other than topology.kubernetes.io/zone to identify zones, set zoneTopologySpread.topologyKey.

4. Choose the host spread's failure domain. Keep kubernetes.io/hostname to spread replicas across nodes. To spread them across a larger failure domain, such as racks, set hostTopologySpread.topologyKey to the node label that identifies it.

5. Choose whether to reconcile existing deployments. A new installation has no existing deployments, so reconcileDrift has no immediate effect there. On a cluster with existing deployments, either created before DataRobot 11.14 or before a change to the spread settings:

   • Keep reconcileDrift: false to leave those deployments as they are.
   • Set reconcileDrift: true to move all deployments with at least hardMinReplicas effective replicas to the current settings in a single rollout.

6. Check the matchLabelKeys feature gate. matchLabelKeys limits the skew calculation to pods from the same revision, so during a rollout, old pods don't count against the new pods' spread. It requires the MatchLabelKeysInPodTopologySpread feature gate, which is on by default in supported Kubernetes versions. Set matchLabelKeys: none for both spreads only if the gate is disabled in your cluster.

Example: Enforce the host spread¶

This example makes the host spread a requirement for deployments with 3 or more effective replicas. minDomains and hardMinReplicas are set to their defaults, shown here for clarity.

lrs-operator:
  operator:
    config:
      hostTopologySpread:
        whenUnsatisfiable: DoNotSchedule
        minDomains: 3
        hardMinReplicas: 3 

Example: Turn off both spreads¶

With both spreads off, the operator adds no placement rules to new deployments, and the scheduler places replicas using only its default scoring.

lrs-operator:
  operator:
    config:
      zoneTopologySpread:
        enabled: false
      hostTopologySpread:
        enabled: false