Shard-scoped caches

Sharding divides the reconcile work between replicas. It does not divide the memory. Without a filter, each replica caches all objects and skips the objects of other shards at reconcile time. With --shard-scoped-cache, a replica caches only the objects of the shards that it owns. Then the cache memory of a replica decreases when you add replicas.

Filtered kinds#

A replica that owns a shard has the Releases of that shard, their inventories and their target Clusters. A Release in namespace a can target a Cluster in namespace b. Both have the shard of b/<cluster>, so they are in the same cache.

Label selector#

The filtered informers send a label selector to the apiserver in each LIST and WATCH:

convoy.kumodo.io/shard in (0,3,6)

The apiserver filters the objects, so the replica does not receive the objects of other shards. When the replica owns no shard, the selector is convoy.kumodo.io/shard in (none). This selector matches no object. Without --shard-scoped-cache, the informers use no selector.

Rebuild on ownership change#

scopedinformers.Set holds one informer factory for the three filtered kinds. The controllers receive stable informer and lister facades from the Set. When the owned shards change, the OnChange callback of the Coordinator calls Rebuild:

1. Rebuild computes the new selector. If the selector did not change, it returns.
2. It creates a new factory with the new selector.
3. It adds the recorded indexers and event handlers to the new informers.
4. It starts the new factory and waits until the three caches are synced.
5. It replaces the informers behind the facades.
6. It stops the previous factory.

The previous cache serves reads until step 5, so reads never see an empty cache. For a short time, the replica holds two caches. The managers also call Rebuild once after Prime, before the controllers start. This call covers a replica that owns no shard, because then Prime does not call OnChange.

Rebuild runs on the tick goroutine of the Coordinator. It blocks the next lease renewal until the new caches are synced. If Rebuild fails, the replica logs the error and keeps the previous caches.

Label requirement#

A filtered informer receives only objects with a matching shard label. An object without the label is in no cache, so no replica reconciles it.

• The apiserver writes the label on each create and update of a Release and a Cluster. An object written by convoy 0.9.0 has the label.
• The controller-manager copies the label of the Release to the ReleaseInventory at each inventory write. If the Release has no label, the ReleaseInventory has no label.
• An object from a version without the shard label has no label until its next write.

Before you set --shard-scoped-cache, check that all Release and Cluster objects have the label. An update of each object makes the apiserver write it.

Memory#

With a full cache, each replica holds all Release, ReleaseInventory and Cluster objects. The memory of a replica does not decrease when you add replicas. With shard-scoped caches, a replica holds about 1/N of these objects for N replicas, up to N = S. Source and rule set objects are on each replica in both modes.

After a replica stops, the other replicas take its shards. Each replica then holds about 1/(N-1) of the objects. Set the memory limit for this case. See Scaling and high availability.

Default and deployment#

--shard-scoped-cache is a flag of convoy-controller-manager and convoy-observer. In the binary, the default is off. The development manifests in dev/deploy/base set the flag on both components. The overlays single-replica and multi-replica use this base. The multi-replica overlay runs two observer replicas with the flag.

Observer subresources#

The apiserver sends /preview, /diff, /health and /history to one observer address. The observer reads the Release from its own cache. With --shard-scoped-cache and more than one observer replica, the replica that receives the request does not always have the Release.