Rollout state machine

The rollout state machine is in the release controller. Each reconcile reads the persisted status.rollout, makes at most one transition, applies the render that the new state selects, and writes the status. This page describes the implementation. For the user-level behavior, see Progressive rollout.

Two decision points#

Only two functions decide a rollout transition:

planRollout performs no side effect. It returns a plan: the next rollout status and an action. The action is actionApplySource or actionApplyLastHealthy. reconcile executes the action.

Order in a reconcile#

reconcile does these steps:

1. stageRender fetches the source, resolves the Cluster and renders. If it fails, reconcile returns. The outcome has no rollout.
2. If the Release has no pin, planRollout makes the plan. With a pin, the plan is actionApplySource with no rollout.
3. For actionApplyLastHealthy, applyLastHealthy applies the last healthy render, and reconcile returns.
4. stageGates checks the manual sync mode and the cluster wave gate. If a gate stops the reconcile, the outcome takes the rollout of the plan.
5. stageSync runs the dry run, the persist, the hooks, the waves and the prune. If waveHealthGate set a rollout, that rollout replaces the plan. Otherwise the outcome takes the rollout of the plan.

Transition table#

R is the render of this reconcile. prior is the persisted status.rollout. The first matching row applies:

rollbackEligible is true when spec.failurePolicy.rollback is not Never and status.lastHealthy is set. It does not check that a render store exists.

A new render always gives Progressing first. The verdict for R applies only at a later reconcile. Thus a verdict of an earlier render cannot give an early Succeeded or Failed.

Verdict#

verdict reads the per-resource health from the ReleaseInventory. The observer writes this health. The function returns one of three values:

The controller owns the deadline. The clock starts at status.rollout.startedAt, which the controller writes when it opens the rollout. Thus each render has its own deadline. The observer does not decide the rollout failure.

If the observer stops, the inventory receives no new health. The verdict stays Pending, so convoy waits and does not roll back.

The wave gate#

waveHealthGate runs when a sync wave with a health strategy waits for the previous wave. It is the only transition at apply time. A wave timeout is visible only during the wave loop, so planRollout cannot decide it before the apply.

When the time since status.waves[].startedAt of the previous wave is more than the wave timeout:

priorRolloutStartedAt keeps the startedAt of prior when prior.renderRef is R. Otherwise it uses the current time. When the wave did not time out, the gate holds with reason WaitingForWaveHealth and sets no rollout.

The wave gate runs only when stageSync runs. With prior in RolledBack, reconcile returns at step 3, so the wave gate cannot change a RolledBack rollout.

Apply of the last healthy render#

applyLastHealthy is one code path for three cases: the first rollback, the hold at each later reconcile, and the rollback after a wave timeout. It does these steps:

1. It returns without work when status.lastHealthy is empty or there is no render store.
2. It reads the blob of status.lastHealthy.renderRef from the render store.
3. It applies all objects with the deployer in one pass. There are no sync waves, no dry run and no hooks.
4. It writes status.lastHealthy.renderRef as the render of each applied object.
5. It prunes with spec.syncPolicy.prune.
6. It writes the applied objects to the ReleaseInventory.

On success, the outcome has applied true and reason RolledBack. status.lastAttempted and status.lastSuccessful then contain the last healthy render. applyLastHealthy does not add a history entry and does not run the garbage collection.

If a step fails, applyLastHealthy logs the error and returns false. In the hold path, reconcile then returns an outcome with reason RolledBack and the planned rollout. It does not apply R. It returns no error, so the requeue interval starts the next attempt. The error backoff does not apply.

Status assembly#

syncHandler writes the status after reconcile:

• status.lastAttempted is the attempt of the outcome. status.lastSuccessful changes only when the outcome has applied true.
• status.lastHealthy is the render of the attempt when verdict for that render returns Healthy. Otherwise it keeps its value. A failure does not clear it.
• When the outcome has a rollout, convoy writes it to status.rollout.
• When the outcome has no rollout and spec.failurePolicy is set, status.rollout keeps its value. This keeps a RolledBack or Succeeded state after a transient source, render or cluster error.
• When spec.failurePolicy is not set, convoy removes status.rollout.
• The RolloutSucceeded condition follows status.rollout.state: True for Succeeded, False for Failed and RolledBack, Unknown for Progressing. Without a rollout, convoy does not change the condition.
• When a forced sync is pending, convoy writes its time to status.lastHandledForceSyncAt. This happens also when the reconcile fails, so a forced sync has an effect one time.

The lastHealthy check runs for every Release, also without a failure policy.

Wake-ups#

A Progressing rollout needs a new reconcile to read a new verdict. Three triggers start it:

inventory.HealthChanged compares the state and assessedRenderRef of each entry. A change of the entry count also counts. Without the health trigger, a healthy render could pass the deadline before the next poll, and convoy could roll it back.

A Failed or RolledBack rollout uses only the normal interval. With an interval of 0, only the informer resync starts the next reconcile.