Juniper Openstack

Contrail Networking :: ubuntu 14.04 build 29 mitaka :: nova compute fails to connect with rabbit.

Bug #1716090 reported by Ritam Gangopadhyay
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Fix Released
Critical
Siva Gurumurthy
Juniper Openstack r4.0.1.0 "r4.0.1.0"
Trunk
Fix Released
Critical
Siva Gurumurthy
Juniper Openstack r4.1.0.0-fcs "r4.1"

Bug Description

In a contrail networking setup, where opensatck is brought up using contrail-cloud image and contrail-services are brought up using networking image. I see nova compute services are not registered on the openstack node. On looking at the nova compute logs I see error while connecting to rabbitmq with user "guest" and password "contrail123". VM spawning fails due to this.

Setup:- R4.0.1.0 build 29 Mitaka multi node setup.

nodei19 10.204.217.131 controller, analytics, analyticsdb
nodei17 10.204.217.129 controller, analytics, analyticsdb
nodec28 10.204.217.13 controller, analytics, analyticsdb
nodec10 10.204.217.176 openstack
nodeg37 10.204.217.77 lb
nodei20 10.204.217.132 compute
nodec33 10.204.217.168 compute

***************************************
***************************************

root@nodec10:~# nova service-list
+----+------------------+---------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | nodec10 | internal | enabled | up | 2017-09-09T04:12:02.000000 | - |
| 5 | nova-scheduler | nodec10 | internal | enabled | up | 2017-09-09T04:12:03.000000 | - |
| 6 | nova-consoleauth | nodec10 | internal | enabled | up | 2017-09-09T04:11:55.000000 | - |
| 7 | nova-console | nodec10 | internal | enabled | up | 2017-09-09T04:12:02.000000 | - |
+----+------------------+---------+----------+---------+-------+----------------------------+-----------------+
root@nodec10:~#

root@nodec10:~# nova availability-zone-list
+-----------------------+----------------------------------------+
| Name | Status |
+-----------------------+----------------------------------------+
| internal | available |
| |- nodec10 | |
| | |- nova-conductor | enabled :-) 2017-09-09T04:12:42.000000 |
| | |- nova-scheduler | enabled :-) 2017-09-09T04:12:43.000000 |
| | |- nova-consoleauth | enabled :-) 2017-09-09T04:12:45.000000 |
| | |- nova-console | enabled :-) 2017-09-09T04:12:42.000000 |
+-----------------------+----------------------------------------+
root@nodec10:~#

**** Note no availability zone or nova-compute listed for i20 and c33

********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************

root@nodec10:~# rabbitmqctl list_users
Listing users ...
guest [administrator]
root@nodec10:~# rabbitmqctl list_consumers | grep i20
root@nodec10:~#

********************************************************************************
********************************************************************************
********************************************************************************
********************************************************************************

root@nodei20:~# grep -rn rabbit /etc/nova/nova.conf | grep user
92:rabbit_userid = guest
root@nodei20:~# grep -rn rabbit /etc/nova/nova.conf | grep password
93:rabbit_password = contrail123
root@nodei20:~#

root@nodei20:~# tail -f /var/log/nova/nova-compute.log
2017-09-09 10:20:05.051 30969 ERROR nova.openstack.common.threadgroup [req-13a14dcf-8a6e-4cfe-8ee7-26f5d61b2abb - - - - -] Timed out waiting for a reply to message ID 12035a882e464cabae200d90afe96b32
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 145, in wait
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup x.wait()
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 47, in wait
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 502, in run_service
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup service.start()
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/service.py", line 164, in start
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1290, in init_host
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup context, self.host, expected_attrs=['info_cache', 'metadata'])
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 161, in wrapper
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup args, kwargs)
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup objver=objver, args=args, kwargs=kwargs)
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup retry=self.retry)
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup timeout=timeout, retry=retry)
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 380, in send
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup retry=retry)
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 369, in _send
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup result = self._waiter.wait(msg_id, timeout)
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 272, in wait
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup message = self.waiters.get(msg_id, timeout=timeout)
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 178, in get
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup 'to message ID %s' % msg_id)
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup MessagingTimeout: Timed out waiting for a reply to message ID 12035a882e464cabae200d90afe96b32
2017-09-09 10:20:05.051 30969 TRACE nova.openstack.common.threadgroup
2017-09-09 10:20:07.150 31087 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2017-09-09 10:20:07.177 31087 INFO nova.virt.driver [-] Loading compute driver 'libvirt.LibvirtDriver'
2017-09-09 10:20:07.217 31087 WARNING oslo_config.cfg [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Option "rabbit_max_retries" from group "DEFAULT" is deprecated. Use option "rabbit_max_retries" from group "oslo_messaging_rabbit".
2017-09-09 10:20:07.217 31087 WARNING oslo_config.cfg [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Option "rabbit_retry_backoff" from group "DEFAULT" is deprecated. Use option "rabbit_retry_backoff" from group "oslo_messaging_rabbit".
2017-09-09 10:20:07.218 31087 WARNING oslo_config.cfg [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Option "rabbit_port" from group "DEFAULT" is deprecated. Use option "rabbit_port" from group "oslo_messaging_rabbit".
2017-09-09 10:20:07.222 31087 INFO oslo_messaging._drivers.impl_rabbit [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Connecting to AMQP server on 192.168.100.13:5672
2017-09-09 10:20:07.234 31087 INFO oslo_messaging._drivers.impl_rabbit [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Connected to AMQP server on 192.168.100.13:5672
2017-09-09 10:20:07.234 31087 WARNING oslo_config.cfg [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Option "rabbit_ha_queues" from group "DEFAULT" is deprecated. Use option "rabbit_ha_queues" from group "oslo_messaging_rabbit".
2017-09-09 10:20:07.241 31087 INFO oslo_messaging._drivers.impl_rabbit [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Connecting to AMQP server on 192.168.100.13:5672
2017-09-09 10:20:07.257 31087 INFO oslo_messaging._drivers.impl_rabbit [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Connected to AMQP server on 192.168.100.13:5672
2017-09-09 10:20:07.278 31087 WARNING oslo_config.cfg [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Option "kombu_reconnect_delay" from group "DEFAULT" is deprecated. Use option "kombu_reconnect_delay" from group "oslo_messaging_rabbit".
2017-09-09 10:20:07.279 31087 WARNING oslo_config.cfg [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Option "rabbit_host" from group "DEFAULT" is deprecated. Use option "rabbit_host" from group "oslo_messaging_rabbit".
2017-09-09 10:20:07.286 31087 WARNING oslo_config.cfg [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Option "disable_process_locking" from group "DEFAULT" is deprecated. Use option "disable_process_locking" from group "oslo_concurrency".
2017-09-09 10:20:07.287 31087 WARNING oslo_config.cfg [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Option "lock_path" from group "DEFAULT" is deprecated. Use option "lock_path" from group "oslo_concurrency".
2017-09-09 10:20:07.289 31087 WARNING oslo_config.cfg [req-03581aad-7078-460d-abb7-3d689971552c - - - - -] Option "url_timeout" from group "neutron" is deprecated. Use option "timeout" from group "neutron".
2017-09-09 10:20:07.290 31087 INFO nova.service [-] Starting compute node (version 2015.1.2)
2017-09-09 10:20:07.302 31087 INFO nova.virt.libvirt.driver [-] Connection event '1' reason 'None'
2017-09-09 10:21:07.308 31087 WARNING nova.virt.libvirt.driver [req-1573c19b-9a21-4efe-a0c5-1ead73154443 - - - - -] Cannot update service status on host "nodei20" due to an unexpected exception.
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver Traceback (most recent call last):
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3059, in _set_host_enabled
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver service = objects.Service.get_by_compute_host(ctx, CONF.host)
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 161, in wrapper
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver args, kwargs)
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 335, in object_class_action
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver objver=objver, args=args, kwargs=kwargs)
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver retry=self.retry)
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver timeout=timeout, retry=retry)
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 380, in send
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver retry=retry)
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 369, in _send
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver result = self._waiter.wait(msg_id, timeout)
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 272, in wait
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver message = self.waiters.get(msg_id, timeout=timeout)
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 178, in get
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver 'to message ID %s' % msg_id)
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver MessagingTimeout: Timed out waiting for a reply to message ID 05ca3ee7e0f64ce59c5ba500d0cfb604
2017-09-09 10:21:07.308 31087 TRACE nova.virt.libvirt.driver
2017-09-09 10:21:07.327 31087 INFO nova.virt.libvirt.host [req-1573c19b-9a21-4efe-a0c5-1ead73154443 - - - - -] Libvirt host capabilities <capabilities>

Revision history for this message
Ritam Gangopadhyay (ritam) wrote :

I see I am able to connect to rabbit from the compute nodes using guest/contrail123 credentials.

root@nodei20:~# cat check_rabbit.py
#!/usr/bin/env python
import socket
from kombu import Connection
host = "192.168.100.13"
port = 5672
user = "guest"
password = "contrail123"
vhost = "/"
url = 'amqp://{0}:{1}@{2}:{3}/{4}'.format(user, password, host, port, vhost)
with Connection(url) as c:
    try:
        c.connect()
    except socket.error:
        raise ValueError("Received socket.error, rabbitmq server probably isn't running")
    except IOError:
        raise ValueError("Received IOError, probably bad credentials")
    else:
        print "Credentials are valid"

root@nodei20:~#

root@nodei20:~# ./check_rabbit.py
Credentials are valid
root@nodei20:~#

Rudra Rugge (rrugge)
Changed in juniperopenstack:
assignee: nobody → Abhay Joshi (abhayj)
Sudheendra Rao (sudheendra-k)
tags: added: sanityblocker
Revision history for this message
Siva Gurumurthy (sgurumurthy) wrote :

You need to put the following following field as part of the contrail cluster as well
openstack_manage_amqp: true

Otherwise by default it will be false and that cluster will have it own rabbitmq and will not talk to Openstack rabbitmq. That's probably the reason it is failing.

Can you please set this option, try again and see if it goes through?

Revision history for this message
Ritam Gangopadhyay (ritam) wrote :

Sure I would give it a try, Did it work on your setup?

Since in the configuration as part of contrail cluster
"global_config": {
        "external_rabbitmq_servers" : "192.168.100.13"
 },

and

"openstack":{
        "external_openstack_ip": "192.168.100.13",

are set, It should be picking up rabbit connections from here like all other contrail services are doing.

Revision history for this message
Siva Gurumurthy (sgurumurthy) wrote :

It works for me with the latest CB build 74.
Able to launch the VM's and ping each other

Revision history for this message
Ritam Gangopadhyay (ritam) wrote :

Verified on latest CB build 74. closing.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.