Juniper Openstack

RHOSP13- R5.0-162: nova timeout while trying to connect to neutron

Bug #1785598 reported by alok kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R5.0
Invalid
High
alok kumar
Juniper Openstack r5.0.1
Trunk
Invalid
High
alok kumar
Juniper Openstack r5.1.0

Bug Description

setup: RHOSP13- rhel-queens-5.0-162

while running sanity, lots of VM launch failed with timeout errors "ConnectTimeout: Request to http://10.1.0.12:9696/v2.0/ports timed out"

single execution of a test passes sometime and fails intermittently with below timeout error in nova-compute.log:

2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] Traceback (most recent call last):
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2248, in _build_resources
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] yield resources
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2031, in _build_and_run_instance
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] block_device_info=block_device_info)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3087, in spawn
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] mdevs=mdevs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5352, in _get_guest_xml
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] network_info_str = str(network_info)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 568, in __str__
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self._sync_wrapper(fn, *args, **kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 551, in _sync_wrapper
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] self.wait()
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 583, in wait
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] self[:] = self._gt.wait()
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self._exit_event.wait()
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] current.throw(*self._exc)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] result = function(*args, **kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 906, in context_wrapper
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return func(*args, **kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1416, in _allocate_network_async
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] six.reraise(*exc_info)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1399, in _allocate_network_async
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] bind_host_id=bind_host_id)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 938, in allocate_for_instance
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] security_group_ids)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 871, in _create_ports_for_instance
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] neutron, instance, created_port_ids)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] self.force_reraise()
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] six.reraise(self.type_, self.value, self.tb)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] six.reraise(self.type_, self.value, self.tb)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 860, in _create_ports_for_instance
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] request.address, security_group_ids)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 424, in _create_port_minimal
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] port_response = port_client.create_port(port_req_body)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] ret = obj(*args, **kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 794, in create_port
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self.post(self.ports_path, body=body)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] ret = obj(*args, **kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 359, in post
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] headers=headers, params=params)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] ret = obj(*args, **kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 282, in do_request
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] headers=headers)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 343, in do_request
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self.request(url, method, **kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 331, in request
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] resp = super(SessionClient, self).request(*args, **kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 189, in request
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self.session.request(url, method, **kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 698, in request
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] resp = send(**kwargs)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 763, in _send_request
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] raise exceptions.ConnectTimeout(msg)
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] ConnectTimeout: Request to http://10.1.0.12:9696/v2.0/ports timed out
2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f]
2018-08-06 09:32:18.200 1 INFO nova.compute.manager [req-831d7a94-eb14-4529-a59f-2de3011f81da c234f5c6e16f45bd89bab9e7f78e5f05 bc3fab5836a24e6a8e614557b560bf67 - default default] [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] Terminating instance

setup info:
This virtualized setup with all the bms(VMs) running on below nodes:
Login for all hypervisors: root
Undercloud: 192.168.122.179 on 10.204.217.133
Controllers hypervisor: 10.204.217.134
Computes hypervisors: 10.204.217.135, 10.204.217.137, 10.204.217.138

target bms(VMs):

(undercloud) [stack@queensa ~]$ openstack server list
+--------------------------------------+--------------------------------+--------+------------------------+----------------+---------------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+--------------------------------+--------+------------------------+----------------+---------------------+
| 58f85277-04ca-4aec-91ce-d5a59ba9e609 | overcloud-contrailcontroller-2 | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | contrail-controller |
| 20d626ff-d15e-48b0-ad06-fba82fa1e5fa | overcloud-contrailcontroller-0 | ACTIVE | ctlplane=192.168.24.19 | overcloud-full | contrail-controller |
| c79d6bfd-4c73-452c-aaf8-03fe08beca1e | overcloud-contrailcontroller-1 | ACTIVE | ctlplane=192.168.24.24 | overcloud-full | contrail-controller |
| 9e65dd37-4e32-466d-900c-014cbed49ee2 | overcloud-novacompute-1 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | compute |
| 2e2c2b82-c296-4c43-9b36-c1d30859e794 | overcloud-controller-0 | ACTIVE | ctlplane=192.168.24.23 | overcloud-full | control |
| 3106a491-420c-4830-a7c0-a4668305ea16 | overcloud-novacompute-0 | ACTIVE | ctlplane=192.168.24.6 | overcloud-full | compute |
| 541c1f09-31b5-421f-a176-aa3ea137ba90 | overcloud-controller-1 | ACTIVE | ctlplane=192.168.24.13 | overcloud-full | control |
| 077c293a-320f-4ec3-9678-ee774d2dfb92 | overcloud-controller-2 | ACTIVE | ctlplane=192.168.24.18 | overcloud-full | control |
| 29615d3b-c9ca-4375-8113-d8339151321a | overcloud-novacompute-2 | ACTIVE | ctlplane=192.168.24.15 | overcloud-full | compute |
+--------------------------------------+--------------------------------+--------+------------------------+----------------+---------------------+

to connect to any bms: ssh root@10.204.217.133-> ssh root@192.168.122.179 -> su - stack-> source stackrc-> ssh heat-admin@192.168.24.19, this will connect to cfgm0

See original description
alok kumar (kalok)
tags: added: rhosp13
tags: added: config neutron nova
Sudheendra Rao (sudheendra-k)
information type: Proprietary → Public
Revision history for this message
Abhay Joshi (abhayj) wrote :

This looks like RHOSP13 specific. Andrey would you be able to look into this or should it be Michael Henkel? Please reassign if so.

Revision history for this message
alexey-mr (alexey-morlang) wrote :

The rabbit cluster is broken - one of the nodes has not joined to the cluster and created own one as it is described here: https://bugs.launchpad.net/juniperopenstack/+bug/1784924

Two nodes:
(undercloud) [stack@queensa ~]$ ssh heat-admin@192.168.24.24 sudo docker exec 7d1829bea8b6 rabbitmqctl cluster_status
Cluster status of node 'contrail@overcloud-contrailcontroller-1'
[{nodes,[{disc,['contrail@overcloud-contrailcontroller-0',
                'contrail@overcloud-contrailcontroller-1']}]},
 {running_nodes,['contrail@overcloud-contrailcontroller-0',
                 'contrail@overcloud-contrailcontroller-1']},
 {cluster_name,<<"<email address hidden>">>},
 {partitions,[]},
 {alarms,[{'contrail@overcloud-contrailcontroller-0',[]},
          {'contrail@overcloud-contrailcontroller-1',[]}]}]

Third node that created own cluster:
(undercloud) [stack@queensa ~]$ ssh heat-admin@192.168.24.14 sudo docker exec fd2081d241f5 rabbitmqctl cluster_status
Cluster status of node 'contrail@overcloud-contrailcontroller-2'
[{nodes,[{disc,['contrail@overcloud-contrailcontroller-2']}]},
 {running_nodes,['contrail@overcloud-contrailcontroller-2']},
 {cluster_name,<<"<email address hidden>">>},
 {partitions,[]},
 {alarms,[{'contrail@overcloud-contrailcontroller-2',[]}]}]

Could you try build with the reviews from that bug.

alok kumar (kalok)
description: updated
Revision history for this message
alok kumar (kalok) wrote :

after resolving rabbitmq cluster issue too, hitting the timeout issue while launching the VMs.
As of now we don't know exact reason, Alexey suspect this might be becasue of all the controllers(total 6 VMs. 3 for contrail controller and 3 for openstack) are running on same hypervisor.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.