Reconcile flow

This page follows one Release key through the Release controller of the controller-manager. It names the function of each step. Then it shows how the two observer loops wake the controller, and how the controller wakes them. For the behavior of each step, follow the links to the concept pages.

Entry: syncHandler#

The work queue calls (*Controller).syncHandler with a namespace/name key. The function does these steps in this order:

1. It reads the Release from the informer cache. If the Release does not exist, it returns.
2. Shard gate. If this replica does not own the shard of the target Cluster, it returns. It does not finalize the Release either. See Sharding.
3. If the Release has a deletion timestamp, it calls finalize and returns.
4. Else it calls ensureFinalizer. This adds release.core.convoy.kumodo.io/prune-workloads with an Update if it is missing.
5. It calls reconcile, which returns a reconcileOutcome and an error.
6. It builds the new status and writes it. The section “Status assembly” below shows the steps.
7. It schedules the next reconcile. The section “Requeue rules” below shows the rules.

finalize#

finalize runs only when the finalizer is present. It does these steps:

1. It resolves the target Cluster with clusterForFinalize, if it must prune or run delete hooks. A missing or unreachable cluster disables both steps. With blockOnClusterUnreachable, an unreachable cluster returns an error and the deletion waits.
2. It runs the preDelete hooks with runHookPhase. A hook that runs or fails requeues the key after 10 seconds.
3. It deletes the inventoried objects with pruneRefs, unless deletionPolicy.keepWorkloads is true.
4. It runs the postDelete hooks. A running hook requeues after 10 seconds. A failed hook is only logged.
5. It deletes the ReleaseInventory.
6. It deletes the render store prefix of the Release, if a render store is configured.
7. It clears the finalizer.

See Deletion.

reconcile#

reconcile first calls newReconcileState. This records the attempt time and the sync mode. It also reads the pending /sync and /sync?force=true requests from the annotations. Then it runs the stages below. Each stage can stop the pipeline with an outcome.

1. stageRender#

stageRender does these steps in this order:

1. Fetch. coordinator.Ensure resolves and fetches the source. An error is transient. A result that is not ready stops the reconcile with reason Unsupported and no error.
2. Cluster. It reads the target Cluster from the cache. A missing or unreachable cluster stops the reconcile with reason ClusterNotReady and an error.
3. Values. resolveValueLayers reads the Helm value sources. Only the helm content type has values.
4. Render. render.Render renders the content. See Rendering.
5. Mutations. render.ApplyMutations runs the post-render mutations. See Mutations.
6. Render reference. renderRef calculates the SHA-256 hash of the mutated objects. The result goes to lastAttempted.renderRef.
7. Pin. resolvePinnedRender replaces the objects with the pinned render from the render store, if the pin points to a different render. A missing store or a missing render stops the reconcile with reason Pinned. See Pin and history.

2. planRollout#

If the Release has no pin, reconcile calls planRollout. This function makes at most one transition of the rollout state machine. It returns the new rollout state and an action: actionApplySource or actionApplyLastHealthy. A pin skips the state machine and applies the pinned or current render. See Rollout state machine and Progressive rollout.

3. applyLastHealthy#

For actionApplyLastHealthy, reconcile calls applyLastHealthy and returns. This function reads the last healthy render from the render store, applies it, prunes and writes the inventory. It skips the gates, the dry run and the hooks. If a step fails, it logs the error. Then reconcile returns an outcome with reason RolledBack and does not apply the source render.

4. stageGates#

stageGates checks two holds. Neither hold applies anything.

1. Manual gate. In manual sync mode without a pending /sync, the stage stops. The reason is Applied when lastSuccessful.renderRef equals the new render. Else the reason is WaitingForSync. See Sync policy.
2. Cluster wave gate. clusterWaveGate reads the other Releases on the same cluster from the releaseByCluster index. A Release with a lower cluster wave that is not Applied or not Healthy holds this Release with reason WaitingForClusterWave. A bounded wait that times out gives ClusterWaveFailed. See Sync waves.

5. stageSync#

stageSync does these steps in this order. See Apply and prune.

1. Dry run. dryRunGate sends all objects as a server-side apply dry run. A rejection stops the sync with reason Unappliable and no error. A forced sync skips this step.
2. Persist. persistRender writes the render to the render store. Without a render store, it does nothing. See Render storage.
3. preSync hooks. runHookPhase for preSync. A running hook holds with reason WaitingForHook. A failed hook with onFailure: Fail stops with reason HookFailed. See Lifecycle hooks.
4. Waves. waves.Group groups the objects by sync wave. For each wave in ascending order:
   • If the wave needs health, and the previous wave is not healthy in the ReleaseInventory, waveHealthGate holds or fails the sync.
   • deployer.Deploy applies the objects of the wave.
   • inventory.Put writes the union of the previous inventory and the objects applied so far.
5. Prune. prune deletes the objects of the previous inventory that are not in the applied set.
6. Inventory. inventory.Put writes the exact applied set.
7. postSync hooks. runHookPhase for postSync. A failed hook does not change the result.
8. History. recordHistory appends the render to the history log of the render store, if it is not there yet.
9. GC. runGC deletes old history entries and render blobs. It keeps the renders that lastSuccessful, lastAttempted, the pin and lastHealthy refer to.

Step 8 and step 9 never stop the sync. They only log their errors.

6. syncFailed hooks#

After stageSync, reconcile checks the outcome with isSyncingFailure. If the attempt failed at the Syncing stage, runSyncFailedHooks arms the syncFailed hooks. It does not wait for them. A dry-run rejection, a failed wave and a failed prune are Syncing failures.

Status assembly#

syncHandler builds the new status from the cached status and the outcome:

1. computeStatus sets the Applied condition from the outcome and sets observedGeneration.
2. pinStatus sets status.pin.
3. The attempt goes to lastAttempted. If the outcome applied a render, it also goes to lastSuccessful.
4. In manual mode, a handled /sync goes to lastHandledSyncAt.
5. The outcome replaces status.waves.
6. resolvedRetentionStatus sets resolvedRetentionPolicy.
7. lastHealthy advances when the verdict for the attempted render is healthy.
8. A pending forced sync goes to lastHandledForceSyncAt.
9. The rollout of the outcome goes to status.rollout. If the outcome has no rollout, the previous rollout stays. Without a failure policy, convoy clears the rollout. rolloutCondition sets RolloutSucceeded.
10. applyLifecycleRollup reads the hook entries of the ReleaseInventory from the API server and sets status.lifecycle and LifecycleSucceeded.
11. If the status changed, the controller sets reconciledAt and writes the status with SSA under the field manager convoy-release.

See Status ownership.

Requeue rules#

After the status write, syncHandler decides the next reconcile:

backoffDelay is initial × factor^(n−1). The default factor is 2. The default maximum is 1000 seconds. requeueAfter is syncPolicy.interval. While the rollout is Progressing, it is at most 10 seconds. If the result is 0, the controller does not requeue the key. Then only events and the informer resync start the next reconcile.

An error from the status write, from ensureFinalizer or from applyLifecycleRollup is returned directly. The work queue retries it.

Each hold and each terminal failure returns no error. Examples are WaitingForSync, WaitingForHook, Unappliable and HookFailed. These outcomes wait for the interval or for an event.

Observer side#

The observer runs two loops on the same Release informer. Each loop has its own shard gate. The drift loop ignores a Release that has a deletion timestamp. The health loop then only polls the delete-phase hooks.

Drift loop#

(*observer.Controller).syncHandler does these steps:

1. renderDesired selects the desired objects. A pinned render in the store comes first. Then the last healthy render, if the rollout is RolledBack. Else it renders the source with renderReleaseObjects, which also runs the mutations.
2. diffObjects reads each live object and sends a server-side apply dry run.
3. writeInSync writes InSync with SSA under convoy-observe. If the drift is not known, the condition is Unknown.
4. In automatic mode with drift, requestSync sends POST /sync.
5. scheduleRecheck requeues after syncPolicy.interval, if it is set.

An undetermined drift does not cause a retry. Only a failed status write or a failed /sync request returns an error to the work queue. See Drift.

Health loop#

The health loop assesses the ReleaseInventory and writes Healthy. See Health loop.

How the loops wake each other#

The loops do not share memory. They communicate through API objects:

A POST /sync changes only an annotation. The two observer loops do not react to this annotation. A status write wakes no loop for the same Release. See Status ownership.