add rolling update / partial update support

[JuanJo Ciarlante]

For the discussion below , "update" refers to upgrade-charm or config-change.

To add reliability to mission critical services, we should be able to do rolling updates ("update ok -> next"), and/or being able to partially update a subset of units.
E.g. use-case: ~1000 units cassandra restarting at the same time (think also: shared storage layer thundering effect).

E.g suggested usage:
$ juju upgrade-charm --max-in-flight=2 ...

Then to properly support resuming above (if interrupted)
$ juju upgrade-charm --num-units-to-update=10

Then to complete either rollback:
$ juju upgrade-charm --roll-back ...
$ juju upgrade-charm --roll-forward ...

FYI webops currently has workarounded this at some charms by:
juju set units-to-update=0,2,5
juju set units-to-update=all

[JuanJo Ciarlante (jjo)]

ERRATA: at 1st post, it should obviously for 'upgrade-charm' (and likely 'set'), instead of 'deploy'.

[John A Meinel (jameinel)]

I'm not sure that this should be classified as High. Certainly it is feature work that isn't on the immediate roadmap. Definitely a nice to have, and might get higher priority from stakeholder escalation (we must have this for site X where we are rolling out 100s of units of Y).

jjo: you can edit a description after the fact, which I'll do now.

[John A Meinel (jameinel)]

Note that you can simulate this a little bit by doing multiple service deploys.

So you do something like:

 juju deploy cs:cassandra-10 cassandra-A
 juju add-unit -n 100 cassandra-A

Then when you want to upgrade
 juju deploy cs:cassandra-11 cassandra-B
 juju add-unit -n 10 cassandra-B
 juju remove-unit -n 10 cassandra-A

See that things keep working:
 juju add-unit -n 50 cassandra-B
 juju remove-unit -n 50 cassandra-A
etc

However, you'd need a way to configure the charm so that its configuration can cross services. (Peer relations only work inside a service ATM.)

The nice thing about this method is that it gives you very fine control over the process, and it lets you do the upgrade as clearly bringing up *more* units of work, rather than shutting down the units before adding more capacity. (though you can always remove-unit before you add-unit if you want that.)

It is a workaround, though, and being able to roll out upgrades in place would be very nice to have.

[Anastasia (anastasia-macmood)]

Re-targeting for Juju 2.x

[Peter Sabaini (peter-sabaini)]

This would be extremely useful for doing critical applications.

Eg. have a workload which is able to failover between AZs. Being able to do charm upgrades per AZ and then fail over the application would greatly reduce the risk of user impact. Lacking this feature means we have to have a maintenance window for the whole cloud.

[Xav Paice (xavpaice)]

https://launchpad.net/layer-coordinator provides a facility which may be what we need to allow this to happen in the charm code, rather than expecting Juju to know how to go about restarting the applications.

[Xav Paice (xavpaice)]

Added charm-helpers to this bug report since it is likely that a change to charm-helpers, then the multitude of charms that need rolling restarts/upgrades/etc is going to be what's needed.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.