Liberty to Mitaka upgrade takes down rabbit too long

Bug #1632040 reported by Bjoern Teipel on 2016-10-10
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
openstack-ansible
Low
Kevin Carter

Bug Description

It seems that after the hostname changes, the rabbit cluster is down for as long as the playbooks need to execute in between

https://github.com/openstack/openstack-ansible/blob/stable/mitaka/scripts/run-upgrade.sh#L156
  and
https://github.com/openstack/openstack-ansible/blob/stable/mitaka/scripts/run-upgrade.sh#L169

and this can easily be 15-30 minutes or more.
To reduce the time we either should move the restart of the rabbit containers (restart-rabbitmq-containers.yml) before the rabbitmq-install.yml play or move the rabbitmq-install.yml up.
Otherwise the API is unnecessarily down for 15-30 minutes,

Bjoern Teipel (bjoern-teipel) wrote :

Cluster status after reboot:

root@infra01-rabbit-mq-container-9f6838e1:~# rabbitmqctl cluster_status
Cluster status of node 'rabbit@infra01-rabbit-mq-container-9f6838e1' ...
[{nodes,[{disc,['rabbit@infra01-rabbit-mq-container-9f6838e1']}]},
 {running_nodes,['rabbit@infra01-rabbit-mq-container-9f6838e1']},
 {cluster_name,<<"<email address hidden>">>},
 {partitions,[]}]

Changed in openstack-ansible:
status: New → In Progress
importance: Undecided → Low
assignee: nobody → Kevin Carter (kevin-carter)
Bjoern Teipel (bjoern-teipel) wrote :

Seems like even with https://review.openstack.org/384643 I still see some rabbit issues until we actually run the nova plays, not sure yet if that is not related to olso/nova needing a restart:

root@Compute02:~# egrep 'MQP server.*:5671 closed the connection. Check login credentials: Socket closed' /var/log/nova/nova-compute.log |head -n1
2016-10-14 16:40:19.982 3912 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.29.239.123:5671 closed the connection. Check login credentials: Socket closed

root@Compute02:~# egrep 'MQP server.*:5671 closed the connection. Check login credentials: Socket closed' /var/log/nova/nova-compute.log |tail -n1
2016-10-14 16:53:31.206 3912 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server 172.29.239.123:5671 closed the connection. Check login credentials: Socket closed

Change abandoned by Kevin Carter (cloudnull) (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/384643
Reason: this was found to be undesirable

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers