rabbitmq-server ssl=off <-> ssl=only transition leave units w/o amqp server until all related units reconfigure

[Andrew Woodward]

I went to enable ssl on the rabbitmq-server charm, however during the transition, the unsecure port is removed from the rabbitmq-server before all of the relations are updated to use the new configuration. This results in the relations experiencing an outage and due to the time it takes to update[1], this can easily be too long of a process. The problem is present in either direction of the ssl flag

starting config:
ssl = "off"
ssl_ca = <unset>
ssl_cert = <unset>
ssl_key = <unset>

enable config
ssl = "only"
ssl_ca = <set>
ssl_cert = <set>
ssl_key = <set>

as soon as the first rabbitmq-server is done the new port is up, and the old is down, the relations start updating, however its somewhat OK, because the second rabbitmq-server is still running the old port. Shortly there after however the second rabbitmq-server is updated, since both hosts are on the new port, units that haven't updated yet are left with no usable connection

[1] https://bugs.launchpad.net/charm-rabbitmq-server/+bug/1717972

[James Page (james-page)]

Fixes are in-flight for bug 1717972; that should minimise the outage window as the change propagates across related applications.

Without a complex system of acks from related applications, which would in turn generate a large about of hook executions, I don't think there is a huge amount we can actually do about this.

Marking this as 'Opinion' so this is discoverable.