Comment 14 for bug 1793269

Moving this to 5.0.3 as we do not want to upgrade rabbitmq in 5.0.2.

Hi Venky,

To recover this, we will need to do the following:
Login to the RabbitMQ docker on all the controllers, backup the directory “/var/lib/rabbitmq/mnesia/contrail@<Node_Name>” and delete it.
Once this folder is deleted in all the RabbitMQ dockers, restart the rabbitmq docker in one of the controller.
Wait for about 10 secs, and restart the rabbitmq docker on other two controller.
To verify that the rabbitmq cluster is correct, execute the following command and verify that all three nodes are present in the field “running_nodes” :
root@Config2:~/mnesia/contrail@Config2# rabbitmqctl cluster_status
Cluster status of node contrail@Config2
[{nodes,[{disc,[contrail@Config1,contrail@Config2,contrail@Config3]}]},
{running_nodes,[contrail@Config3,contrail@Config1,contrail@Config2]},
{cluster_name,<<"<email address hidden>">>},
{partitions,[]},
{alarms,[{contrail@Config3,[]},{contrail@Config1,[]},{contrail@Config2,[]}]}]

Thanks,
Sathish

From: Venkatesh Velpula <email address hidden>
Date: Tuesday, October 9, 2018 at 9:33 PM
To: Jeba Paulaiyan <email address hidden>, Abhay Joshi <email address hidden>, Shivayogi Ugaji <email address hidden>
Cc: Sathish Holla <email address hidden>, Sudheendra Rao <email address hidden>
Subject: Re: https://bugs.launchpad.net/juniperopenstack/r5.0/+bug/1793269

Hi Jeba,
       This is not happening always …but when it happens the impact is catastrophic..

Satish ,
       Could you please help us with recovery mechanism , we can release not the same for 5.0.2

Thanks
-Venky

From: Jeba Paulaiyan <email address hidden>
Date: Wednesday, October 10, 2018 at 5:33 AM
To: Abhay Joshi <email address hidden>, Shivayogi Ugaji <email address hidden>, Venkatesh Velpula <email address hidden>
Cc: Sathish Holla <email address hidden>, Madhava Rao Sudheendra Rao <email address hidden>
Subject: Re: https://bugs.launchpad.net/juniperopenstack/r5.0/+bug/1793269

Venky,

        This decision is based on assumption that this is not happening always and a race condition in RabbitMQ. Please feel free to disagree.

Thanks,
Jeba

From: Abhay Joshi <email address hidden>
Date: Tuesday, October 9, 2018 at 16:46
To: Shivayogi Ugaji <email address hidden>
Cc: Sathish Holla <email address hidden>, Jeba Paulaiyan <email address hidden>
Subject: Re: https://bugs.launchpad.net/juniperopenstack/r5.0/+bug/1793269

+ Jeba.

As discussed in bug scrub today, we will push this out to 5.1.0. Please update series accordingly.

Thanks,

Abhay

From: Shivayogi Ugaji <email address hidden>
Date: Tuesday, October 9, 2018 at 1:17 PM
To: Abhay Joshi <email address hidden>
Cc: Sathish Holla <email address hidden>
Subject: https://bugs.launchpad.net/juniperopenstack/r5.0/+bug/1793269

Hi Abhay,

This is due to a bug in rabbitMq implementation and the latest version of rabbitMq has the fix.
We need to update the rabbitMq version from 3.6 to 3.7. Any idea who can help with this.

Thanks
Shivayogi