provisioning: On centos7.2 HA cluster rabbitmq fails after provisioning

[Senthilnathan Murugappan]

Unable to create VN/delete VN after provisioning using Fab on a centos7.2 HA cluster.
The issue seems to be with rabbitmq which crashes.
Bootstrapping rabbitmq fixed the issue.

[Ignatious Johnson Christopher (ijohnson-x)]

Workaround:
==============
Re-cluster the rabbitmq after provisioning, if the issue is seen using following fab command.

fab setup_rabbitmq_cluster:force=yes

[Ignatious Johnson Christopher (ijohnson-x)]

[root@b7s32 ~]# neutron --debug --insecure net-delete cec382aa-8a52-4fbf-9f6b-1c60536ccdc4
DEBUG: stevedore.extension found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG: stevedore.extension found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG: stevedore.extension found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG: neutronclient.neutron.v2_0.network.DeleteNetwork run(Namespace(id=u'cec382aa-8a52-4fbf-9f6b-1c60536ccdc4', request_format='json'))
DEBUG: keystoneauth.session REQ: curl -g -i --insecure -X GET https://172.16.80.1:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.1 python-requests/2.9.1 CPython/2.7.5"
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:797: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: keystoneauth.session RESP: [200] Content-Length: 338 Vary: X-Auth-Token Connection: keep-alive Date: Fri, 03 Mar 2017 05:27:54 GMT Content-Type: application/json X-Openstack-Request-Id: req-8973515b-3ea2-4509-8aa1-606ea0447095
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://172.16.80.1:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG: keystoneauth.identity.v2 Making authentication request to https://172.16.80.1:5000/v2.0/tokens
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:797: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: keystoneauth.session REQ: curl -g -i --insecure -X GET https://172.16.80.50:9696/v2.0/networks.json?fields=id&id=cec382aa-8a52-4fbf-9f6b-1c60536ccdc4 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}4497b4d018d4a8ce9a8865ced161a744b5cc13eb"
/usr/lib/python2.7/site-packages/reque...

[Ignatious Johnson Christopher (ijohnson-x)]

[root@b7s32 ~]# rabbitmqctl list_queues
Listing queues ...

^C
Session terminated, killing shell...^C ...killed.
[root@b7s32 ~]#

[Ignatious Johnson Christopher (ijohnson-x)]

[root@b7s32 ~]# neutron --debug --insecure net-create VN1
DEBUG: stevedore.extension found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG: stevedore.extension found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG: stevedore.extension found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('table = cliff.formatters.table:TableFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('json = cliff.formatters.json_format:JSONFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('shell = cliff.formatters.shell:ShellFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('value = cliff.formatters.value:ValueFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('yaml = cliff.formatters.yaml_format:YAMLFormatter')
DEBUG: neutronclient.neutron.v2_0.network.CreateNetwork run(Namespace(admin_state=True, availability_zone_hints=None, columns=[], dns_domain=None, formatter='table', max_width=0, name=u'VN1', noindent=False, prefix='', provider:network_type=None, provider:physical_network=None, provider:segmentation_id=None, qos_policy=None, request_format='json', tenant_id=None, variables=[]))
DEBUG: keystoneauth.session REQ: curl -g -i --insecure -X GET https://172.16.80.1:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.1 python-requests/2.9.1 CPython/2.7.5"
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:797: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: keystoneauth.session RESP: [200] Content-Length: 338 Vary: X-Auth-Token Connection: keep-alive Date: Fri, 03 Mar 2017 05:36:39 GMT Content-Type: application/json X-Openstack-Request-Id: req-b0ce99a9-5071-438c-9c29-68d5d362d04e
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://172.16.80.1:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "...

[Jeba Paulaiyan (jebap)]

Ajay: I don’t know the root cause unfortunately but the contents of /var/log/rabbitmq/*-ctrl-sasl.log seem to have a lot of entries reporting crashes. If someone can attach that to the bug maybe we can look at it later.

Copied /var/log/rabbitmq/rabbit*-ctrl-sasl.log to /cs-shared/bugs/1651258