Scaling and high availability

convoy has three processes: the apiserver, the controller-manager and the observer. You can run more than one replica of the controller-manager and of the observer. These two components divide their work into shards. Each shard has a Kubernetes Lease. Only the replica that holds the lease of a shard does the work of that shard. The same mechanism gives high availability and scale. convoy has no separate leader election. See Sharding for the design.

Shards#

convoy divides the work by target cluster:

• A Release goes to the shard of its target Cluster (spec.clusterRef). All Releases for one cluster go to the same replica.
• A Cluster, GitRepository, HelmRegistry and OCIRegistry go to the shard of their own namespace and name.

The shard is the FNV-1a hash of <namespace>/<name>, modulo the shard count. The flag --shards sets the shard count. The default is 32.

The shard count is also the maximum parallelism. A replica gets no shards when there are more replicas than shards.

What must match#

The controller-manager and the observer shard independently. Each has its own set of leases, so they can run a different number of replicas. A Release can have its controller-manager work on one pod and its observer work on a different pod.

The apiserver uses --shards only to write the shard label convoy.kumodo.io/shard on objects. Only --shard-scoped-cache reads this label. Use the same value on all three components. The defaults are the same.

How replicas divide the shards#

Each replica runs a lease loop. At each --lease-renew-interval (default 10s), the replica does these steps:

1. It renews its member lease. The member lease shows that the replica is alive.
2. It reads the live member leases and sorts the replica IDs.
3. It calculates its shards. Replica k of N gets each shard i where i mod N = k.
4. It acquires or renews the leases of its shards.
5. It releases the leases of the shards that it no longer gets.

When the set of shards of a replica changes, the replica queues all objects of its new shards for a reconcile. Each reconcile starts with an ownership check, so a replica skips an object that it does not own.

Failover#

A lease is valid for --lease-duration (default 30s) after its last renewal. When a replica stops, the other replicas take its shards after its member lease expires. Then they acquire the expired shard leases at their next renewal.

• The time without an owner is up to about --lease-duration plus --lease-renew-interval. With the defaults, this is about 40 seconds.
• A replica that stops normally releases its shard leases. But its member lease stays live until it expires, so the other replicas do not take the shards earlier.
• After 10 times --lease-duration, a replica deletes the member lease of a dead replica.

A shorter --lease-duration gives a faster failover. Keep --lease-duration much longer than --lease-renew-interval and longer than one reconcile. Use the same values on all replicas of a component.

Replica identity and lease namespace#

Each replica must have a unique --replica-id. The dev Deployments set POD_NAME and POD_NAMESPACE from the Downward API. Use the same lease namespace on all replicas of a component.

The lease names are:

• convoy-cm-member-<replica-id> and convoy-cm-shard-<i> for the controller-manager.
• convoy-obs-member-<replica-id> and convoy-obs-shard-<i> for the observer.

Lease RBAC#

Each component needs these permissions on leases (coordination.k8s.io) in the lease namespace:

• get, list, create and update, for the member lease and the shard leases.
• delete, for the member leases of dead replicas.

The dev ClusterRoles also give watch. The lease loop does not watch.

Run more replicas#

The multi-replica overlay sets 2 replicas of the controller-manager and 2 replicas of the observer. The single-replica overlay sets 1 replica of each. To use more replicas, set a larger count in the overlay. Do not use more replicas than --shards.

Make sure that these conditions are true:

• Each replica has a unique --replica-id.
• All replicas of a component use the same --shards, --lease-namespace, --lease-duration and --lease-renew-interval.
• All replicas of the controller-manager and of the observer use the same --render-store-url. The store must be shared. See Render store backends.

Shard-scoped caches#

By default, each replica caches all Release, ReleaseInventory and Cluster objects. With --shard-scoped-cache, a replica caches only the objects of its own shards. Its memory then grows with the number of objects per replica, not with the total number of objects. The source kinds and the health rule sets are always fully cached.

The trade-offs are:

• The cache uses the shard label. An object without the label, or with a label from a different --shards value, is not in the cache.
• When the shards of a replica change, the replica builds new informers and waits for their cache sync. This delays its next lease renewal.
• On the observer, /preview, /diff, /health and /history read the local cache. The apiserver sends these requests to one Service. A replica that does not own the Release returns 404.

The base overlay sets --shard-scoped-cache on the controller-manager and on the observer. The multi-replica overlay thus runs 2 observer replicas with shard-scoped caches. Run the observer without the flag, or run one observer replica. See Shard-scoped caches.

The apiserver and etcd#

The apiserver does not use shards or leases. It keeps all convoy objects in etcd. The dev overlays run one apiserver replica. To run more replicas, give each replica a serving certificate that the CA in the caBundle of the APIService objects signed. The dev APIService objects use insecureSkipTLSVerify and a self-signed certificate for each pod.

The apiserver connects to etcd with the standard --etcd-* flags, for example --etcd-servers. The dev overlays run one etcd pod with an emptyDir volume. All convoy objects are lost when this pod is created again. For production, use an etcd cluster with persistent storage and backups.

The flags of each component are on Binaries and flags.