provisioning: On centos7.2 HA cluster rabbitmq fails after provisioning

Bug #1651258 reported by Senthilnathan Murugappan
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.2
Won't Fix
High
Hampapur Ajay
R4.0
Won't Fix
High
Hampapur Ajay
Trunk
Invalid
High
Hampapur Ajay

Bug Description

Unable to create VN/delete VN after provisioning using Fab on a centos7.2 HA cluster.
The issue seems to be with rabbitmq which crashes.
Bootstrapping rabbitmq fixed the issue.

Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

Workaround:
==============
Re-cluster the rabbitmq after provisioning, if the issue is seen using following fab command.

fab setup_rabbitmq_cluster:force=yes

Jeba Paulaiyan (jebap)
tags: added: releasenote
Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :
Download full text (6.8 KiB)

[root@b7s32 ~]# neutron --debug --insecure net-delete cec382aa-8a52-4fbf-9f6b-1c60536ccdc4
DEBUG: stevedore.extension found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG: stevedore.extension found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG: stevedore.extension found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG: neutronclient.neutron.v2_0.network.DeleteNetwork run(Namespace(id=u'cec382aa-8a52-4fbf-9f6b-1c60536ccdc4', request_format='json'))
DEBUG: keystoneauth.session REQ: curl -g -i --insecure -X GET https://172.16.80.1:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.1 python-requests/2.9.1 CPython/2.7.5"
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:797: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: keystoneauth.session RESP: [200] Content-Length: 338 Vary: X-Auth-Token Connection: keep-alive Date: Fri, 03 Mar 2017 05:27:54 GMT Content-Type: application/json X-Openstack-Request-Id: req-8973515b-3ea2-4509-8aa1-606ea0447095
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://172.16.80.1:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

DEBUG: keystoneauth.identity.v2 Making authentication request to https://172.16.80.1:5000/v2.0/tokens
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:797: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: keystoneauth.session REQ: curl -g -i --insecure -X GET https://172.16.80.50:9696/v2.0/networks.json?fields=id&id=cec382aa-8a52-4fbf-9f6b-1c60536ccdc4 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}4497b4d018d4a8ce9a8865ced161a744b5cc13eb"
/usr/lib/python2.7/site-packages/reque...

Read more...

Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :

[root@b7s32 ~]# rabbitmqctl list_queues
Listing queues ...

^C
Session terminated, killing shell...^C ...killed.
[root@b7s32 ~]#

Revision history for this message
Ignatious Johnson Christopher (ijohnson-x) wrote :
Download full text (6.3 KiB)

[root@b7s32 ~]# neutron --debug --insecure net-create VN1
DEBUG: stevedore.extension found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG: stevedore.extension found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG: stevedore.extension found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG: stevedore.extension found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG: stevedore.extension found extension EntryPoint.parse('table = cliff.formatters.table:TableFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('json = cliff.formatters.json_format:JSONFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('shell = cliff.formatters.shell:ShellFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('value = cliff.formatters.value:ValueFormatter')
DEBUG: stevedore.extension found extension EntryPoint.parse('yaml = cliff.formatters.yaml_format:YAMLFormatter')
DEBUG: neutronclient.neutron.v2_0.network.CreateNetwork run(Namespace(admin_state=True, availability_zone_hints=None, columns=[], dns_domain=None, formatter='table', max_width=0, name=u'VN1', noindent=False, prefix='', provider:network_type=None, provider:physical_network=None, provider:segmentation_id=None, qos_policy=None, request_format='json', tenant_id=None, variables=[]))
DEBUG: keystoneauth.session REQ: curl -g -i --insecure -X GET https://172.16.80.1:5000/v2.0/ -H "Accept: application/json" -H "User-Agent: keystoneauth1/2.4.1 python-requests/2.9.1 CPython/2.7.5"
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:797: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
DEBUG: keystoneauth.session RESP: [200] Content-Length: 338 Vary: X-Auth-Token Connection: keep-alive Date: Fri, 03 Mar 2017 05:36:39 GMT Content-Type: application/json X-Openstack-Request-Id: req-b0ce99a9-5071-438c-9c29-68d5d362d04e
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "https://172.16.80.1:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "...

Read more...

Revision history for this message
Jeba Paulaiyan (jebap) wrote :

Ajay: I don’t know the root cause unfortunately but the contents of /var/log/rabbitmq/*-ctrl-sasl.log seem to have a lot of entries reporting crashes. If someone can attach that to the bug maybe we can look at it later.

Copied /var/log/rabbitmq/rabbit*-ctrl-sasl.log to /cs-shared/bugs/1651258

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.