setup: RHOSP13- rhel-queens-5.0-162 while running sanity, lots of VM launch failed with timeout errors "ConnectTimeout: Request to http://10.1.0.12:9696/v2.0/ports timed out" single execution of a test passes sometime and fails intermittently with below timeout error in nova-compute.log: 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] Traceback (most recent call last): 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2248, in _build_resources 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] yield resources 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2031, in _build_and_run_instance 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] block_device_info=block_device_info) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3087, in spawn 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] mdevs=mdevs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5352, in _get_guest_xml 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] network_info_str = str(network_info) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 568, in __str__ 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self._sync_wrapper(fn, *args, **kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 551, in _sync_wrapper 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] self.wait() 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 583, in wait 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] self[:] = self._gt.wait() 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self._exit_event.wait() 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] current.throw(*self._exc) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] result = function(*args, **kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/utils.py", line 906, in context_wrapper 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return func(*args, **kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1416, in _allocate_network_async 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] six.reraise(*exc_info) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1399, in _allocate_network_async 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] bind_host_id=bind_host_id) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 938, in allocate_for_instance 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] security_group_ids) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 871, in _create_ports_for_instance 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] neutron, instance, created_port_ids) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] self.force_reraise() 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] six.reraise(self.type_, self.value, self.tb) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] six.reraise(self.type_, self.value, self.tb) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 860, in _create_ports_for_instance 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] request.address, security_group_ids) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 424, in _create_port_minimal 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] port_response = port_client.create_port(port_req_body) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] ret = obj(*args, **kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 794, in create_port 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self.post(self.ports_path, body=body) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] ret = obj(*args, **kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 359, in post 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] headers=headers, params=params) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in wrapper 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] ret = obj(*args, **kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 282, in do_request 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] headers=headers) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 343, in do_request 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self.request(url, method, **kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 331, in request 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] resp = super(SessionClient, self).request(*args, **kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 189, in request 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] return self.session.request(url, method, **kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 698, in request 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] resp = send(**kwargs) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 763, in _send_request 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] raise exceptions.ConnectTimeout(msg) 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] ConnectTimeout: Request to http://10.1.0.12:9696/v2.0/ports timed out 2018-08-06 09:32:18.197 1 ERROR nova.compute.manager [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] 2018-08-06 09:32:18.200 1 INFO nova.compute.manager [req-831d7a94-eb14-4529-a59f-2de3011f81da c234f5c6e16f45bd89bab9e7f78e5f05 bc3fab5836a24e6a8e614557b560bf67 - default default] [instance: ca1ad565-dfc9-4a51-aaa6-eea075f8746f] Terminating instance setup info: This virtualized setup with all the bms(VMs) running on below nodes: Login for all hypervisors: root/c0ntrail123 Undercloud: 192.168.122.179(root/contrail123) on 10.204.217.133 Controllers hypervisor: 10.204.217.134 Computes hypervisors: 10.204.217.135, 10.204.217.137, 10.204.217.138 target bms(VMs): (undercloud) [stack@queensa ~]$ openstack server list +--------------------------------------+--------------------------------+--------+------------------------+----------------+---------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+--------------------------------+--------+------------------------+----------------+---------------------+ | 58f85277-04ca-4aec-91ce-d5a59ba9e609 | overcloud-contrailcontroller-2 | ACTIVE | ctlplane=192.168.24.14 | overcloud-full | contrail-controller | | 20d626ff-d15e-48b0-ad06-fba82fa1e5fa | overcloud-contrailcontroller-0 | ACTIVE | ctlplane=192.168.24.19 | overcloud-full | contrail-controller | | c79d6bfd-4c73-452c-aaf8-03fe08beca1e | overcloud-contrailcontroller-1 | ACTIVE | ctlplane=192.168.24.24 | overcloud-full | contrail-controller | | 9e65dd37-4e32-466d-900c-014cbed49ee2 | overcloud-novacompute-1 | ACTIVE | ctlplane=192.168.24.20 | overcloud-full | compute | | 2e2c2b82-c296-4c43-9b36-c1d30859e794 | overcloud-controller-0 | ACTIVE | ctlplane=192.168.24.23 | overcloud-full | control | | 3106a491-420c-4830-a7c0-a4668305ea16 | overcloud-novacompute-0 | ACTIVE | ctlplane=192.168.24.6 | overcloud-full | compute | | 541c1f09-31b5-421f-a176-aa3ea137ba90 | overcloud-controller-1 | ACTIVE | ctlplane=192.168.24.13 | overcloud-full | control | | 077c293a-320f-4ec3-9678-ee774d2dfb92 | overcloud-controller-2 | ACTIVE | ctlplane=192.168.24.18 | overcloud-full | control | | 29615d3b-c9ca-4375-8113-d8339151321a | overcloud-novacompute-2 | ACTIVE | ctlplane=192.168.24.15 | overcloud-full | compute | +--------------------------------------+--------------------------------+--------+------------------------+----------------+---------------------+ to connect to any bms: ssh root@10.204.217.133-> ssh root@192.168.122.179 (psswd: contrail123)-> su - stack-> source stackrc-> ssh heat-admin@192.168.24.19, this will connect to cfgm0