config rabbitmq queue not cleaned up upon restart

[Senthilnathan Murugappan]

Version: 3.2.5.0-48~mitaka
OS: ubuntu1404

On a three node cluster observed that there were rabbitmq queues for backup nodes too
schema_transformer.5b4s2 164950
schema_transformer.5b4s4 0
schema_transformer.5b4s6 164950

Did restart of the backup schema and was expecting the queue to be cleaned up upon start but observed that the messages are cleaned up but not the queue itself.

schema_transformer.5b4s2 0
schema_transformer.5b4s4 0
schema_transformer.5b4s6 164959

Cluster: 10.87.64.129 /opt/contrail/utils/fabfile/testbeds/testbed.py has the cluster info

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35090
Submitter: ?douard Thuleau (<email address hidden>)

[Édouard Thuleau (ethuleau)]

vnc_kombu common lib fails to close the consumer connection, raise an exception and leave the queue:

Traceback (most recent call last):
  File "/usr/bin/contrail-schema", line 9, in <module>
    load_entry_point('schema-transformer==0.1dev', 'console_scripts', 'contrail-schema')()
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 658, in server_main
    main()
  File "/usr/lib/python2.7/dist-packages/schema_transformer/to_bgp.py", line 638, in main
    vnc_amqp.close()
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_amqp.py", line 202, in close
    self._vnc_kombu.shutdown()
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_kombu.py", line 236, in shutdown
    self._consumer.close()
  File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 488, in cancel
    cancel(tag)
  File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1454, in basic_cancel
    wait=None if nowait else spec.Basic.CancelOk,
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 73, in send_method
    return self.wait(wait, returns_tuple=returns_tuple)
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in wait
    self.connection.drain_events(timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 485, in drain_events
    while not self.blocking_read(timeout):
  File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 491, in blocking_read
    return self.on_inbound_frame(frame)
  File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 69, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 495, in on_inbound_method
    method_sig, payload, content,
  File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 142, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 617, in _on_close
    (class_id, method_id), ConnectionError)
InternalError: (0, 0): (541) INTERNAL_ERROR

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35090
Committed: http://github.com/Juniper/contrail-controller/commit/51f6487961003faa9f77835a3b7bcf02e761d3b6
Submitter: Zuul (<email address hidden>)
Branch: master

commit 51f6487961003faa9f77835a3b7bcf02e761d3b6
Author: Édouard Thuleau <email address hidden>
Date: Wed Aug 30 12:42:34 2017 +0200

[VNC config] Don't close AMQP consumer

Don't close the AMQP producers or conusmers, just delete queue and
then close the AMQP connection.

Change-Id: I0efe52f889f0892b4a5e4336b39fa30062b08034
Closes-Bug: #1713899

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35242
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35243
Submitter: ?douard Thuleau (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35242
Committed: http://github.com/Juniper/contrail-controller/commit/d676ae1e9464e952618619d3616b9ad29b07de2a
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit d676ae1e9464e952618619d3616b9ad29b07de2a
Author: Édouard Thuleau <email address hidden>
Date: Wed Aug 30 12:42:34 2017 +0200

[VNC config] Don't close AMQP consumer

Don't close the AMQP producers or conusmers, just delete queue and
then close the AMQP connection.

Change-Id: I0efe52f889f0892b4a5e4336b39fa30062b08034
Closes-Bug: #1713899
(cherry picked from commit 51f6487961003faa9f77835a3b7bcf02e761d3b6)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35243
Committed: http://github.com/Juniper/contrail-controller/commit/586903f6afe60772195883018635d0569ab02041
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 586903f6afe60772195883018635d0569ab02041
Author: Édouard Thuleau <email address hidden>
Date: Wed Aug 30 12:42:34 2017 +0200

[VNC config] Don't close AMQP consumer

Don't close the AMQP producers or conusmers, just delete queue and
then close the AMQP connection.

Change-Id: I0efe52f889f0892b4a5e4336b39fa30062b08034
Closes-Bug: #1713899
(cherry picked from commit 51f6487961003faa9f77835a3b7bcf02e761d3b6)