RabbitMQ lost connection after 10 minutes.

[Narinder Gupta]

After deploy the openstack bundle at this https://gerrit.opnfv.org/gerrit/gitweb?p=joid.git;a=tree;f=ci/odl/juju-deployer;h=27714c87363ef27dd2c36f23bf69f9a196ea1415;hb=f09798d479a06366ba96fea0591c1f0d52cde2e0

https://gerrit.opnfv.org/gerrit/gitweb?p=joid.git;a=blob;f=ci/odl/juju-deployer/ovs-odl.yaml;h=b188a587ae8132e4ea996b84550ce1b3e3915778;hb=f09798d479a06366ba96fea0591c1f0d52cde2e0

After 10 minutes of successful deployment i am getting nov-compute and nova-conducter status disabled. Restart the nova-compute and nova-conductor services enable it again but after 15-20 minutes it again disables and rabbitmq has the following bug.

Within a day I got the rabbitmq log grew to 1.6 GB and nova-conductor logs grew to 600 MB.

Attaching the below rabbitmq logs snapshot where it started the error.

=INFO REPORT==== 14-Oct-2015::00:59:03 ===
accepting AMQP connection <0.12550.0> (10.4.1.140:42144 -> 10.4.1.143:5672)

=INFO REPORT==== 14-Oct-2015::00:59:03 ===
accepting AMQP connection <0.12553.0> (10.4.1.140:42145 -> 10.4.1.143:5672)

=INFO REPORT==== 14-Oct-2015::00:59:05 ===
accepting AMQP connection <0.12561.0> (10.4.1.139:57137 -> 10.4.1.143:5672)

=ERROR REPORT==== 14-Oct-2015::00:59:13 ===
closing AMQP connection <0.12550.0> (10.4.1.140:42144 -> 10.4.1.143:5672):
{handshake_timeout,frame_header}

=ERROR REPORT==== 14-Oct-2015::00:59:13 ===
closing AMQP connection <0.12553.0> (10.4.1.140:42145 -> 10.4.1.143:5672):
{handshake_timeout,frame_header}

=INFO REPORT==== 14-Oct-2015::00:59:21 ===
accepting AMQP connection <0.12592.0> (10.

nova-conductor logs are here:

2015-10-15 06:43:52.739 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-0e019e98-3954-464b-adf2-922d835fa12d - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.989 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-22fda8e7-cb2c-4b8b-b509-f40b71468683 - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.990 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-53edf620-34cd-468e-ba32-6745bf6452d1 - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.991 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-946d69d7-1963-4e3c-9d20-95dd6cf4d566 - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.992 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-53e267a1-6383-4b63-9edc-942285dc8b11 - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.992 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-4db2e186-21af-46c8-be1f-cad81f759fff - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.993 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-dc43813d-15ad-4c8f-9110-52a9892e8c3a - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.994 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-523eeeaf-3501-444f-86cd-2fc74bd44c48 - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.995 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-de7216c4-b024-4c4a-a162-932baab50718 - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.996 33443 ERROR oslo_messaging._drivers.impl_rabbit [req-5b87d527-21e7-4a65-a16d-f65e31e342ed - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:52.999 33443 WARNING nova.openstack.common.loopingcall [req-45c08f43-de34-4319-9315-9562dfb768e3 - - - - -] task <bound method DbDriver._report_state of <nova.servicegroup.drivers.db.DbDriver object at 0x7f2810fc6950>> run outlasted interval by 20.87 sec
2015-10-15 06:43:53.033 33488 ERROR oslo_messaging._drivers.impl_rabbit [req-bb30cfe5-f9fd-4b67-b66a-f47847b3b192 - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.
2015-10-15 06:43:53.035 33488 ERROR oslo_messaging._drivers.impl_rabbit [req-750cfb9c-8ced-4b4e-b86f-4c951a0fdc38 - - - - -] AMQP server on 10.4.1.143:5672 is unreachable: [Errno 104] ECONNRESET. Trying again in 2 seconds.

[David Ames (thedac)]

Narinder,

We are making a concerted push to get through our bug queue. This bug is old and there have been considerable improvements to the rabbitmq charm. Marking this as invalid. If you ever see this problem again please change it to new.