Juniper Openstack

server-manager: nova-compute dead after provisioning

Bug #1638321 reported by tom murray
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.1
Fix Committed
High
Dheeraj Gautam
Juniper Openstack r3.1.1.0
R3.2
Fix Committed
High
Dheeraj Gautam
Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"
Trunk
Fix Committed
High
Dheeraj Gautam
Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

This is with Contrail OpenStack 3.1.1.0-36 Mitaka on ubuntu 14.04.4

After a successful provisioning of my cluster with 3 controllers set up in HA topology i see that the nova-compute service is dead when i do openstack-status:

root@gngbnm038d:~# openstack-status
== Nova services ==
openstack-nova-api: active
openstack-nova-compute: dead
openstack-nova-network: inactive (disabled on boot)
openstack-nova-scheduler: active
openstack-nova-volume: inactive (disabled on boot)
openstack-nova-conductor: active
== Glance services ==
openstack-glance-api: active
openstack-glance-registry: active
== Keystone service ==
openstack-keystone: active
== Cinder services ==
openstack-cinder-api: active
openstack-cinder-scheduler: active
openstack-cinder-volume: active
== Heat services ==
heat-api: active
heat-api-cfn: active
heat-api-cloudwatch: inactive (disabled on boot)
heat-engine: active
== Support services ==
mysql: inactive (disabled on boot)
libvirt-bin: active
rabbitmq-server: active
memcached: inactive (disabled on boot)
== Keystone users ==
Warning keystonerc not sourced

The nova-compute log shows:

2016-10-30 13:14:42.445 28228 WARNING oslo_config.cfg [req-c92614ea-1de8-4599-ae86-b18c0be553d2 - - - - -] Option "rabbit_max_retries" from group "DEFAULT" is deprecated. Use option "rabbit_max_retries" from group "oslo_messaging_rabbit".
2016-10-30 13:14:42.446 28228 WARNING oslo_config.cfg [req-c92614ea-1de8-4599-ae86-b18c0be553d2 - - - - -] Option "rabbit_retry_backoff" from group "DEFAULT" is deprecated. Use option "rabbit_retry_backoff" from group "oslo_messaging_rabbit".
2016-10-30 13:14:42.447 28228 WARNING oslo_config.cfg [req-c92614ea-1de8-4599-ae86-b18c0be553d2 - - - - -] Option "amqp_durable_queues" from group "DEFAULT" is deprecated. Use option "amqp_durable_queues" from group "oslo_messaging_rabbit".
2016-10-30 13:14:42.450 28228 DEBUG oslo.messaging._drivers.impl_rabbit [req-c92614ea-1de8-4599-ae86-b18c0be553d2 - - - - -] Connecting to AMQP server on 10.224.12.13:5672 __init__ /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:539
2016-10-30 13:14:42.460 28228 CRITICAL nova [req-c92614ea-1de8-4599-ae86-b18c0be553d2 - - - - -] AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2016-10-30 13:14:42.460 28228 ERROR nova Traceback (most recent call last):
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/bin/nova-compute", line 10, in <module>
2016-10-30 13:14:42.460 28228 ERROR nova sys.exit(main())
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/compute.py", line 74, in main
2016-10-30 13:14:42.460 28228 ERROR nova db_allowed=CONF.conductor.use_local)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 296, in create
2016-10-30 13:14:42.460 28228 ERROR nova db_allowed=db_allowed)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 191, in __init__
2016-10-30 13:14:42.460 28228 ERROR nova self.conductor_api.wait_until_ready(context.get_admin_context())
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 157, in wait_until_ready
2016-10-30 13:14:42.460 28228 ERROR nova timeout=timeout)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/baserpc.py", line 62, in ping
2016-10-30 13:14:42.460 28228 ERROR nova return cctxt.call(context, 'ping', arg=arg_p)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-10-30 13:14:42.460 28228 ERROR nova retry=self.retry)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2016-10-30 13:14:42.460 28228 ERROR nova timeout=timeout, retry=retry)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-10-30 13:14:42.460 28228 ERROR nova retry=retry)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 414, in _send
2016-10-30 13:14:42.460 28228 ERROR nova msg.update({'_reply_q': self._get_reply_q()})
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 386, in _get_reply_q
2016-10-30 13:14:42.460 28228 ERROR nova conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 377, in _get_connection
2016-10-30 13:14:42.460 28228 ERROR nova purpose=purpose)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/common.py", line 396, in __init__
2016-10-30 13:14:42.460 28228 ERROR nova self.connection = connection_pool.create(purpose)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/pool.py", line 110, in create
2016-10-30 13:14:42.460 28228 ERROR nova return self.connection_cls(self.conf, self.url, purpose)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 555, in __init__
2016-10-30 13:14:42.460 28228 ERROR nova self.ensure_connection()
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 653, in ensure_connection
2016-10-30 13:14:42.460 28228 ERROR nova self.ensure(method=lambda: self.connection.connection)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 758, in ensure
2016-10-30 13:14:42.460 28228 ERROR nova ret, channel = autoretry_method()
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2016-10-30 13:14:42.460 28228 ERROR nova return fun(*args, **kwargs)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 507, in __call__
2016-10-30 13:14:42.460 28228 ERROR nova self.revive(create_channel())
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 242, in channel
2016-10-30 13:14:42.460 28228 ERROR nova chan = self.transport.create_channel(self.connection)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection
2016-10-30 13:14:42.460 28228 ERROR nova self._connection = self._establish_connection()
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
2016-10-30 13:14:42.460 28228 ERROR nova conn = self.transport.establish_connection()
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2016-10-30 13:14:42.460 28228 ERROR nova conn = self.Connection(**opts)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 180, in __init__
2016-10-30 13:14:42.460 28228 ERROR nova (10, 30), # tune
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 67, in wait
2016-10-30 13:14:42.460 28228 ERROR nova self.channel_id, allowed_methods, timeout)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 274, in _wait_method
2016-10-30 13:14:42.460 28228 ERROR nova wait()
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 69, in wait
2016-10-30 13:14:42.460 28228 ERROR nova return self.dispatch_method(method_sig, args, content)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2016-10-30 13:14:42.460 28228 ERROR nova return amqp_method(self, args)
2016-10-30 13:14:42.460 28228 ERROR nova File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 530, in _close
2016-10-30 13:14:42.460 28228 ERROR nova (class_id, method_id), ConnectionError)
2016-10-30 13:14:42.460 28228 ERROR nova AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2016-10-30 13:14:42.460 28228 ERROR nova

Restarting the nova-compute service is successful.
Probably the nova-compute provisioning is occurring before the message queue is fully available?

sundarkh (sundar-kh)
information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/25570
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/25570
Committed: http://github.org/Juniper/contrail-puppet/commit/49e2d748f48773f250857a5f0d168859d1267ae5
Submitter: Zuul
Branch: R3.1

commit 49e2d748f48773f250857a5f0d168859d1267ae5
Author: Dheeraj Gautam <email address hidden>
Date: Tue Nov 1 13:59:13 2016 -0700

SM-Openstack: don't install nova-compute on openstack nodes

Closes-Bug: #1638321

nova::compute and nova::compute::neutron are not needed for openstack roles.
only needed for compute.

Change-Id: If36e794c246dccb3ea44446d3815517809c60627

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/26068
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/26069
Submitter: Dheeraj Gautam (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/26069
Committed: http://github.org/Juniper/contrail-puppet/commit/094399dcbd11aea07a5deecac2808a1146ecd450
Submitter: Zuul
Branch: R3.2

commit 094399dcbd11aea07a5deecac2808a1146ecd450
Author: Dheeraj Gautam <email address hidden>
Date: Tue Nov 1 13:59:13 2016 -0700

SM-Openstack: don't install nova-compute on openstack nodes

Closes-Bug: #1638321

nova::compute and nova::compute::neutron are not needed for openstack roles.
only needed for compute.

Change-Id: If36e794c246dccb3ea44446d3815517809c60627

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/26068
Committed: http://github.org/Juniper/contrail-puppet/commit/444c9ecc14e0eb667df0e8363d8d8e142d1f7880
Submitter: Zuul
Branch: master

commit 444c9ecc14e0eb667df0e8363d8d8e142d1f7880
Author: Dheeraj Gautam <email address hidden>
Date: Tue Nov 1 13:59:13 2016 -0700

SM-Openstack: don't install nova-compute on openstack nodes

Closes-Bug: #1638321

nova::compute and nova::compute::neutron are not needed for openstack roles.
only needed for compute.

Change-Id: If36e794c246dccb3ea44446d3815517809c60627

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.