Juniper Openstack

During provisioning, network service setup in keystone was attempted(and failed) before keystone was up

Bug #1395518 reported by Vedamurthy Joshi on 2014-11-23

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Juniper Openstack	Fix Committed	High	Ignatious Johnson Christopher

Bug Description

http://anamika.englab.juniper.net:8080/job/centos64_os_icehouse_Multi_Node_Sanity/52/console

Build 2.0-2472, Centos 6.4 Multinode setup

All logs : /cs-shared/test_runs/nodec21/2014_11_23_03_47_33 on any blr shell server (anamika/noded1 etc)

testbed :
=========
env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6],
    'cfgm': [host1, host3],
    'openstack': [host2],
    'webui': [host3],
    'control': [host1, host3],
    'compute': [host4, host5, host6],
    'collector': [host1, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}

env.hostnames = {
'all': ['nodec21', 'nodec19', 'nodec20', 'nodec61', 'nodec60','nodec18']
}

bond= {
    host4 : { 'name': 'bond0', 'member': ['p2p0p0','p2p0p1','p2p0p2','p2p0p3'],'mode':'802.3ad' },
    host5 : { 'name': 'bond0', 'member': ['p2p0p0','p2p0p1','p2p0p2','p2p0p3'],'mode':'802.3ad' },
    host6 : { 'name': 'bond0', 'member': ['p2p0p0','p2p0p1'],'mode':'balance-xor' },
}

control_data= {

    host1 : { 'ip': '192.168.10.4/24', 'gw' : '192.168.10.254', 'device':'eth1' },
    host2 : { 'ip': '192.168.10.5/24', 'gw' : '192.168.10.254', 'device':'eth1' },
    host3 : { 'ip': '192.168.10.6/24', 'gw' : '192.168.10.254', 'device':'eth1' },
    host4 : { 'ip': '192.168.10.1/24', 'gw' : '192.168.10.254', 'device':'bond0' },
    host5 : { 'ip': '192.168.10.2/24', 'gw' : '192.168.10.254', 'device':'bond0' },
    host6 : { 'ip': '192.168.10.3/24', 'gw' : '192.168.10.254', 'device':'bond0' },
}

Below, setup-quantum-in-keystone failed due to ConnectionRefused on keystone port. It is seen that keystone roughly got into Running state just a bit late in time (after supervisor-openstack restart cmd)

2014-11-23 04:23:56:392420: [root@10.204.217.6] out: [localhost] local: setup-quantum-in-keystone --ks_server_ip 192.168.10.5 --quant_server_ip 192.168.10.4 --tenant admin --user admin --password contrail123 --svc_password 2f12ae8a6f8749740b65 --root_password None --region_name RegionOne
2014-11-23 04:23:56:399786: [root@10.204.217.6] out: Traceback (most recent call last):
2014-11-23 04:23:56:631652: [root@10.204.217.6] out: File "/usr/bin/setup-quantum-in-keystone", line 9, in <module>
2014-11-23 04:23:56:631788: [root@10.204.217.6] out: load_entry_point('ContrailProvisioning==0.1dev', 'console_scripts', 'setup-quantum-in-keystone')()
2014-11-23 04:23:56:631856: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/contrail_provisioning/config/quantum_in_keystone_setup.py", line 308, in main
2014-11-23 04:23:56:631920: [root@10.204.217.6] out: quant_obj.do_quant_setup()
2014-11-23 04:23:56:631982: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/contrail_provisioning/config/quantum_in_keystone_setup.py", line 274, in do_quant_setup
2014-11-23 04:23:56:632045: [root@10.204.217.6] out: self.quant_tenant_id = self.quant_set_tenant_id()
2014-11-23 04:23:56:632105: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/contrail_provisioning/config/quantum_in_keystone_setup.py", line 135, in quant_set_tenant_id
2014-11-23 04:23:56:632172: [root@10.204.217.6] out: quant_tenant = self.kshandle.tenants.find(name=self._quant_tenant_name)
2014-11-23 04:23:56:632233: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/base.py", line 229, in find
2014-11-23 04:23:56:632293: [root@10.204.217.6] out: rl = self.findall(**kwargs)
2014-11-23 04:23:56:632352: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/base.py", line 249, in findall
2014-11-23 04:23:56:632412: [root@10.204.217.6] out: for obj in self.list():
2014-11-23 04:23:56:632471: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/v2_0/tenants.py", line 118, in list
2014-11-23 04:23:56:632529: [root@10.204.217.6] out: tenant_list = self._list("/tenants%s" % query, "tenants")
2014-11-23 04:23:56:632589: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/base.py", line 106, in _list
2014-11-23 04:23:56:632648: [root@10.204.217.6] out: resp, body = self.client.get(url)
2014-11-23 04:23:56:632706: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 589, in get
2014-11-23 04:23:56:632765: [root@10.204.217.6] out: return self._cs_request(url, 'GET', **kwargs)
2014-11-23 04:23:56:632824: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 581, in _cs_request
2014-11-23 04:23:56:632882: [root@10.204.217.6] out: return self.request(url, method, **kwargs)
2014-11-23 04:23:56:632941: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/httpclient.py", line 563, in request
2014-11-23 04:23:56:632999: [root@10.204.217.6] out: resp = super(HTTPClient, self).request(url, method, **kwargs)
2014-11-23 04:23:56:633059: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/baseclient.py", line 21, in request
2014-11-23 04:23:56:633117: [root@10.204.217.6] out: return self.session.request(url, method, **kwargs)
2014-11-23 04:23:56:633180: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/utils.py", line 324, in inner
2014-11-23 04:23:56:633238: [root@10.204.217.6] out: return func(*args, **kwargs)
2014-11-23 04:23:56:633296: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/session.py", line 260, in request
2014-11-23 04:23:56:633353: [root@10.204.217.6] out: resp = self._send_request(url, method, redirect, **kwargs)
2014-11-23 04:23:56:633422: [root@10.204.217.6] out: File "/usr/lib/python2.6/site-packages/keystoneclient/session.py", line 294, in _send_request
2014-11-23 04:23:56:633481: [root@10.204.217.6] out: raise exceptions.ConnectionRefused(msg)
2014-11-23 04:23:56:633539: [root@10.204.217.6] out: keystoneclient.openstack.common.apiclient.exceptions.ConnectionRefused: Unable to establish connection to http://192.168.10.5:35357/v2.0/tenants
2014-11-23 04:23:56:633598: [root@10.204.217.6] out:
2014-11-23 04:23:56:650600: [root@10.204.217.6] out: Fatal error: local() encountered an error (return code 1) while executing 'setup-quantum-in-keystone --ks_server_ip 192.168.10.5 --quant_server_ip 192.168.10.4 --tenant admin --user admin --password contrail123 --svc_password 2f12ae8a6f8749740b65 --root_password None --region_name RegionOne'
2014-11-23 04:23:56:650722: [root@10.204.217.6] out:
2014-11-23 04:23:56:650793: [root@10.204.217.6] out: Aborting.
2014-11-23 04:23:56:650851: [root@10.204.217.6] out:

supervisord-openstack.log :
==========================

2014-11-23 04:23:54,668 INFO supervisord started with pid 10782
2014-11-23 04:23:55,670 INFO spawned: 'nova-novncproxy' with pid 10819
2014-11-23 04:23:55,671 INFO spawned: 'nova-conductor' with pid 10820
2014-11-23 04:23:55,672 INFO spawned: 'glance-api' with pid 10821
2014-11-23 04:23:55,673 INFO spawned: 'cinder-api' with pid 10822
2014-11-23 04:23:55,674 INFO spawned: 'nova-consoleauth' with pid 10823
2014-11-23 04:23:55,674 INFO spawned: 'keystone' with pid 10824
2014-11-23 04:23:55,675 INFO spawned: 'nova-api' with pid 10826
2014-11-23 04:23:55,677 INFO spawned: 'nova-scheduler' with pid 10827
2014-11-23 04:23:55,678 INFO spawned: 'heat-engine' with pid 10828
2014-11-23 04:23:55,679 INFO spawned: 'glance-registry' with pid 10829
2014-11-23 04:23:55,679 INFO spawned: 'heat-api' with pid 10830
2014-11-23 04:23:55,681 INFO spawned: 'cinder-scheduler' with pid 10831
2014-11-23 04:23:56,792 INFO success: nova-novncproxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,792 INFO success: nova-conductor entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,792 INFO success: glance-api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,792 INFO success: cinder-api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,792 INFO success: nova-consoleauth entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,792 INFO success: keystone entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,792 INFO success: nova-api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,792 INFO success: nova-scheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,792 INFO success: heat-engine entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,792 INFO success: glance-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,793 INFO success: heat-api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-11-23 04:23:56,793 INFO success: cinder-scheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Later, when we try the same setup-quantum-in-keystone cmd, it works fine..

Tags:

Vedamurthy Joshi (vedujoshi) on 2014-11-23

summary:

- During provisioning, network service setup in keystone happened(and
+ During provisioning, network service setup in keystone was attempted(and
failed) before keystone was up

Sandip Dey (sandipd) on 2014-11-24

tags:

added: sanity

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2014-11-26: A change has been merged

Reviewed: https://review.opencontrail.org/4978
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/e3bcc49b4e65d1fbddb01c2aafb6968dd8cd6bf2
Submitter: Zuul
Branch: master

commit e3bcc49b4e65d1fbddb01c2aafb6968dd8cd6bf2
Author: Ignatious Johnson <email address hidden>
Date: Tue Nov 25 16:04:59 2014 -0800

Make sure the opensack services are up

executing verify_openstack task to make sure the openstack services are up
before continuing to provisionig contrail, else exiting provisioning.

Change-Id: I8c7f59ccbfcf08d874fa2586c139601f4a394cab
Closes-Bug: 1395518

Changed in juniperopenstack:
status:	New → Fix Committed

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2014-11-26:

Reviewed: https://review.opencontrail.org/4979
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/2b389c8e77dbc5247f68ddeb92187ba17f3b8749
Submitter: Zuul
Branch: R2.0

commit 2b389c8e77dbc5247f68ddeb92187ba17f3b8749
Author: Ignatious Johnson <email address hidden>
Date: Tue Nov 25 16:10:31 2014 -0800

Make sure the opensack services are up

executing verify_openstack task to make sure the openstack services are up
before continuing to provisionig contrail, else exiting provisioning.

Change-Id: Ia0534d4372e466525e2617d448d7c05f967909a6
Closes-Bug: 1395518

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.