Juniper Openstack

Unable to boot vms, nova-compute.conf has missing neutron and glance sections

Bug #1595112 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
Trunk
Fix Committed
Critical
amudhar
Juniper Openstack r3.1.0.0-fcs

Bug Description

Build 3.1.0.0-2736 Kilo

On the compute-only node 10.204.216.52 , neutron and glance sections were missing in nova.conf
So VM boots were failing on that node. Looks like nova provisioning in broken on mainline.

We had to add the below config to make it work :
===========

[neutron]
admin_auth_url = http://192.168.10.1:35357/v2.0/
admin_username = neutron
admin_password = contrail123
admin_tenant_name = service
url = http://192.168.10.1:9696/
url_timeout = 300
service_metadata_proxy = True

[glance]
host = 192.168.10.1
============

testbed.py :

from fabric.api import env

#Management ip addresses of hosts in the cluster
host1 = 'root@10.204.216.57'
host2 = 'root@10.204.216.52'

#External routers if any
#for eg.
#ext_routers = [('mx1', '10.204.216.253')]
ext_routers = []

#Autonomous system number
router_asn = 64512

#Host from which the fab commands are triggered to install and provision
host_build = 'root@1.1.1.1'

#Role definition of the hosts.
env.roledefs = {
    'all': [host1, host2],
    'cfgm': [host1],
    'openstack': [host1],
    'control': [host1],
    'compute': [host1, host2],
    'collector': [host1],
    'webui': [host1],
    'database': [host1],
    'build': [host_build],
}

env.hostnames = {
    'all': ['noded2', 'nodef1']
}

# Passwords of each host
# for passwordless login's no need to set env.passwords,
# instead populate env.key_filename in testbed.py with public key.
env.password = 'c0ntrail123'
env.passwords = {
    host1: 'c0ntrail123',
    host2: 'c0ntrail123',
}

# SSH Public key file path for passwordless logins
# if env.passwords is not specified.
#env.key_filename = '/root/.ssh/id_rsa.pub'

#For reimage purpose
env.ostypes = {
    host1: 'ubuntu',
    host2: 'ubuntu',
}

#OPTIONAL SEPARATION OF MANAGEMENT AND CONTROL + DATA and OPTIONAL VLAN INFORMATION
#==================================================================================
control_data = {
    host1 : { 'ip': '192.168.10.1/24', 'gw' : '192.168.10.254', 'device':'p6p2' },
    host2 : { 'ip': '192.168.10.2/24', 'gw' : '192.168.10.254', 'device':'p6p2'},
}
===================================

nova-compute.log had the below exceptions

2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] block_device_info=block_device_info)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2385, in spawn
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] admin_pass=admin_password)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2798, in _create_image
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] instance, size, fallback_from_host)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5996, in _try_fetch_image_cache
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] size=size)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 252, in cache
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] *args, **kwargs)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 501, in create_image
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] prepare_template(target=base, max_size=size, *args, **kwargs)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] return f(*args, **kwargs)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 237, in fetch_func_sync
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] fetch_func(target=target, *args, **kwargs)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 507, in fetch_image
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] max_size=max_size)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in fetch_to_raw
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] max_size=max_size)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 77, in fetch
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] IMAGE_API.download(context, image_href, dest_path=path)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 182, in download
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] dst_path=dest_path)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 351, in download
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] _reraise_translated_image_exception(image_id)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 349, in download
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] image_chunks = self._client.call(context, 1, 'data', image_id)
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 236, in call
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] host=host, port=port, reason=six.text_type(e))
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6] GlanceConnectionFailed: Connection to glance host 10.204.216.52:9292 failed: Error finding address for http://10.204.216.52:9292/v1/images/ecae27b6-3cd7-4a1c-8a6f-3f6556311a5a: HTTPConnectionPool(host='10.204.216.52', port=9292): Max retries exceeded with url: /v1/images/ecae27b6-3cd7-4a1c-8a6f-3f6556311a5a (Caused by ProtocolError('Connection aborted.', error(111, 'ECONNREFUSED')))
2016-06-22 14:46:14.122 17366 TRACE nova.compute.manager [instance: e52d7528-8b27-40ce-a652-e107c26956d6]
2016-06-22 14:46:14.125 17366 INFO nova.compute.manager [req-db484f57-8e27-4b87-87b4-663002443fb7 2a288c289f13429383f0db2221af395f 71a3e23ab380459b85b2d24911640e49 - - -] [instance: e52d7528-8b27-40ce-a652-e107c26956d6] Terminating instance

-----------------------------

2016-06-22 14:46:13.707 17366 DEBUG keystoneclient.session [-] REQ: curl -g -i -X GET http://127.0.0.1:9696/v2.0/extensions.json -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}760c771207aefa7a5fb782cbcf0a4c24e4038f71" _http_log_request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:195
2016-06-22 14:46:13.709 17366 DEBUG urllib3.util.retry [-] Converted retries value: 0 -> Retry(total=0, connect=None, read=None, redirect=0) from_int /usr/lib/python2.7/dist-packages/urllib3/util/retry.py:155
2016-06-22 14:46:13.711 17366 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager Traceback (most recent call last):
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1798, in _allocate_network_async
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager dhcp_options=dhcp_options)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 410, in allocate_for_instance
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager refresh_cache=True, neutron=neutron) else
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 607, in _has_port_binding_extension
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager self._refresh_neutron_extensions_cache(context, neutron=neutron)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 599, in _refresh_neutron_extensions_cache
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager extensions_list = neutron.list_extensions()['extensions']
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager ret = self.function(instance, *args, **kwargs)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 493, in list_extensions
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager return self.get(self.extensions_path, params=_params)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 290, in get
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager headers=headers, params=params)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 267, in retry_request
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager headers=headers, params=params)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 197, in do_request
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager content_type=self.content_type())
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 301, in do_request
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager return self.request(url, method, **kwargs)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 296, in request
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager resp = super(SessionClient, self).request(*args, **kwargs)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 89, in request
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager return self.session.request(url, method, **kwargs)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager return func(*args, **kwargs)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 374, in request
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager resp = send(**kwargs)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 417, in _send_request
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager raise exceptions.ConnectionRefused(msg)
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager ConnectionRefused: Unable to establish connection to http://127.0.0.1:9696/v2.0/extensions.json
2016-06-22 14:46:13.711 17366 TRACE nova.compute.manager

Revision history for this message
Vedamurthy Joshi (vedujoshi) wrote :

Setup_all log attached

Nagabhushana R (bhushana)
tags: added: blocker
Revision history for this message
Ritam Gangopadhyay (ritam) wrote :

May be a issue only with kilo, on HA setup with liberty build 2738 I see these expected sections in nova.conf

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/21559
Submitter: amudha ramachandran (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/21559
Committed: http://github.org/Juniper/contrail-provisioning/commit/decd384429e7720c0ffe2c3329175acafa8e1aec
Submitter: Zuul
Branch: master

commit decd384429e7720c0ffe2c3329175acafa8e1aec
Author: Amudha <email address hidden>
Date: Wed Jun 29 16:18:32 2016 -0700

Correcting the check for nova-compute version in kilo

Change-Id: I240fe3d5a1938ded98b04d5f1ef9e2b0a41f7e51
Closes-Bug: #1595112

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.