Drift

Drift is a difference between the live objects on the target cluster and the desired render of a Release. The observer looks for drift in a loop and writes the result to the InSync condition. In automatic sync mode, the observer then requests a /sync, and the controller applies the render again.

The desired render#

The observer uses the same render that the controller keeps on the cluster. It selects the first item that applies:

1. The pinned render, if the Release has a pin and the render store contains that render.
2. The last healthy render, if the rollout is RolledBack and the render store contains that render.
3. A fresh render of the current source, content and mutations.

Without this order, a pinned or rolled-back Release would always show drift. The /preview subresource returns the same desired render. See Pin and history.

In manual mode, the fresh render can be newer than the applied render. Then InSync is False until a /sync request applies the new render.

How the observer compares#

For each object of the desired render, the observer:

1. Reads the live object. If the object does not exist, its status is Missing.
2. Sends the desired object as a server-side apply dry run, with field manager convoy and force. This is the same request as the real apply, without a change on the cluster.
3. Clears these fields in the live object and in the dry-run result: metadata.resourceVersion, metadata.uid, metadata.generation, metadata.creationTimestamp, metadata.managedFields and status.
4. Compares the two results. Each difference is a field change. With one or more changes, the status is Modified. With no change, the status is InSync.

Both sides of the compare come from the API server. So server-side defaults and admission changes are on both sides and do not show as drift.

A field that another field manager owns, and that the render does not set, does not show as drift. A field that the render sets, and that another manager changed, shows as drift.

The InSync condition#

The observer writes InSync with field manager convoy-observe. It writes only when the condition or the generation of the Release changes.

• InSync (True): all objects are InSync.
• Drifted (False): at least one object is Modified or Missing.
• Undetermined (Unknown): the observer could not complete the compare.

The compare stops at the first error, and the result is Undetermined. Examples of errors:

• The source or the render failed.
• The Cluster does not exist or is not reachable.
• The cluster does not know the kind of an object. For example, its CRD is not installed yet.
• The cluster rejected the dry run of an object.

The observer does not retry an Undetermined result with backoff. It checks again at the next interval.

Self-heal#

The mode in spec.syncPolicy.automatic controls what occurs after the compare:

• Automatic mode. On Drifted, the observer sends POST /sync. The controller reconciles and applies the full render again, which corrects the drift.
• Manual mode. The observer only writes InSync. convoy changes nothing until you send a /sync request.

In automatic mode, the controller also applies the full render at each periodic reconcile. So the controller corrects drift also without the observer. See Sync policy.

When the observer checks#

The observer runs a drift pass for a Release when one of these events occurs:

• The Release is created, or its spec changes.
• The controller applies a different render, which changes the ReleaseInventory.
• The time of spec.syncPolicy.interval passes after the previous pass.
• The informer resync of the observer occurs. The default period is 2 minutes (flag --resync).

A /sync request does not start a drift pass. The observer does not watch the target cluster. A change on the cluster shows at the next pass. The observer skips a Release that is being deleted.

The /diff subresource#

GET releases/<name>/diff uses the same compare engine as the drift loop. So /diff and InSync always agree. /diff returns the status and the field changes of each object. With render=<renderRef>, it compares a stored render with the live objects. See Release diff.

Only declared objects#

The observer checks only the objects of the desired render. It does not report a live object that convoy applied before but the render no longer contains. It also does not report other objects in the target namespaces. Detection of these extra objects is planned. Not built

The prune deletes objects that the render no longer contains. See Apply and prune.