Liberty to Mitaka upgrade takes down rabbit too long

[Bjoern]

It seems that after the hostname changes, the rabbit cluster is down for as long as the playbooks need to execute in between

https://github.com/openstack/openstack-ansible/blob/stable/mitaka/scripts/run-upgrade.sh#L156
  and
https://github.com/openstack/openstack-ansible/blob/stable/mitaka/scripts/run-upgrade.sh#L169

and this can easily be 15-30 minutes or more.
To reduce the time we either should move the restart of the rabbit containers (restart-rabbitmq-containers.yml) before the rabbitmq-install.yml play or move the rabbitmq-install.yml up.
Otherwise the API is unnecessarily down for 15-30 minutes,

[Bjoern (bjoern-t)]

Cluster status after reboot:

root@infra01-rabbit-mq-container-9f6838e1:~# rabbitmqctl cluster_status
Cluster status of node 'rabbit@infra01-rabbit-mq-container-9f6838e1' ...
[{nodes,[{disc,['rabbit@infra01-rabbit-mq-container-9f6838e1']}]},
 {running_nodes,['rabbit@infra01-rabbit-mq-container-9f6838e1']},
 {cluster_name,<<"<email address hidden>">>},
 {partitions,[]}]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/384643

[Bjoern (bjoern-t)]

Seems like even with https://review.openstack.org/384643 I still see some rabbit issues until we actually run the nova plays, not sure yet if that is not related to olso/nova needing a restart:

root@Compute02:~# egrep 'MQP server.*:5671 closed the connection. Check login credentials: Socket closed' /var/log/nova/nova-compute.log |head -n1
2016-10-14 16:40:19.982 3912 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.29.239.123:5671 closed the connection. Check login credentials: Socket closed

root@Compute02:~# egrep 'MQP server.*:5671 closed the connection. Check login credentials: Socket closed' /var/log/nova/nova-compute.log |tail -n1
2016-10-14 16:53:31.206 3912 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.29.239.123:5671 closed the connection. Check login credentials: Socket closed

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kevin Carter (cloudnull) (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/384643
Reason: this was found to be undesirable

[Mohammed Naser (mnaser)]

Bug filed against an unsupported old release