Hit this issue in [Build "Ubuntu 14.04 LTS" 2.0-22~juno] MultiNode Single Intf Sanity
While trying to spawn vm in testcase 'test_mx_gateway '
Topology : Config Nodes : [u'nodei11', u'nodei12'] Control Nodes : [u'nodei12', u'nodei13'] Compute Nodes : [u'nodei14', u'nodei15'] Openstack Node : nodei11 WebUI Node : nodei11 Analytics Nodes : [u'nodei11']
/var/log/nova/nova-compute.log :
2014-12-24 07:35:29.582 2180 AUDIT nova.compute.claims [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] Claim successful 2014-12-24 07:35:29.754 2180 INFO nova.scheduler.client.report [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] Compute_service record updated for ('nodei14', 'nodei14.englab.juniper.net') 2014-12-24 07:35:29.938 2180 INFO nova.scheduler.client.report [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] Compute_service record updated for ('nodei14', 'nodei14.englab.juniper.net') 2014-12-24 07:35:30.524 2180 INFO nova.virt.libvirt.driver [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] Creating image 2014-12-24 07:35:31.093 2180 INFO nova.scheduler.client.report [-] Compute_service record updated for ('nodei14', 'nodei14.englab.juniper.net') 2014-12-24 07:35:31.303 2180 INFO nova.virt.disk.vfs.api [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] Unable to import guestfsfalling back to VFSLocalFS 2014-12-24 07:35:31.861 2180 ERROR root [-] Second simultaneous read on fileno 21 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False) 2014-12-24 07:35:31.861 2180 TRACE root Traceback (most recent call last): 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/contrail_vrouter_api/vrouter_api.py", line 171, in periodic_connection_check 2014-12-24 07:35:31.861 2180 TRACE root self._client.KeepAliveCheck() 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/contrail_vrouter_api/gen_py/instance_service/InstanceService.py", line 176, in KeepAliveCheck 2014-12-24 07:35:31.861 2180 TRACE root return self.recv_KeepAliveCheck() 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/contrail_vrouter_api/gen_py/instance_service/InstanceService.py", line 186, in recv_KeepAliveCheck 2014-12-24 07:35:31.861 2180 TRACE root (fname, mtype, rseqid) = self._iprot.readMessageBegin() 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin 2014-12-24 07:35:31.861 2180 TRACE root sz = self.readI32() 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32 2014-12-24 07:35:31.861 2180 TRACE root buff = self.trans.readAll(4) 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/thrift/transport/TTransport.py", line 58, in readAll 2014-12-24 07:35:31.861 2180 TRACE root chunk = self.read(sz - have) 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/thrift/transport/TTransport.py", line 271, in read 2014-12-24 07:35:31.861 2180 TRACE root self.readFrame() 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/thrift/transport/TTransport.py", line 275, in readFrame 2014-12-24 07:35:31.861 2180 TRACE root buff = self.__trans.readAll(4) 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/thrift/transport/TTransport.py", line 58, in readAll 2014-12-24 07:35:31.861 2180 TRACE root chunk = self.read(sz - have) 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/thrift/transport/TSocket.py", line 103, in read 2014-12-24 07:35:31.861 2180 TRACE root buff = self.handle.recv(sz) 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/eventlet/greenio.py", line 262, in recv 2014-12-24 07:35:31.861 2180 TRACE root timeout_exc=socket.timeout("timed out")) 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/eventlet/hubs/__init__.py", line 151, in trampoline 2014-12-24 07:35:31.861 2180 TRACE root listener = hub.add(hub.READ, fileno, current.switch) 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/eventlet/hubs/epolls.py", line 48, in add 2014-12-24 07:35:31.861 2180 TRACE root listener = BaseHub.add(self, evtype, fileno, cb) 2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 126, in add 2014-12-24 07:35:31.861 2180 TRACE root evtype, fileno, evtype)) 2014-12-24 07:35:31.861 2180 TRACE root RuntimeError: Second simultaneous read on fileno 21 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False) 2014-12-24 07:35:31.861 2180 TRACE root 2014-12-24 07:35:32.372 2180 ERROR nova.compute.manager [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] Instance failed to spawn 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] Traceback (most recent call last): 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2234, in _build_resources 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] yield resources 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2104, in _build_and_run_instance 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] block_device_info=block_device_info) 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2622, in spawn 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] block_device_info, disk_info=disk_info) 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4424, in _create_domain_and_network 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] self.plug_vifs(instance, network_info) 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 947, in plug_vifs 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] self.vif_driver.plug(instance, vif) 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] File "/usr/lib/python2.7/dist-packages/nova_contrail_vif/contrailvif.py", line 207, in plug 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] **kwargs) 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] File "/usr/lib/python2.7/dist-packages/contrail_vrouter_api/vrouter_api.py", line 132, in add_port 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] result = self._client.AddPort([data]) 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] AttributeError: 'NoneType' object has no attribute 'AddPort' 2014-12-24 07:35:32.372 2180 TRACE nova.compute.manager [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] 2014-12-24 07:35:32.444 2180 AUDIT nova.compute.manager [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] Terminating instance 2014-12-24 07:35:32.460 2180 WARNING nova.virt.libvirt.driver [-] [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] During wait destroy, instance disappeared. 2014-12-24 07:35:32.657 2180 INFO nova.virt.libvirt.driver [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] Deleting instance files /var/lib/nova/instances/c13062f0-1fbc-4425-ae07-b4e8babe785b_del 2014-12-24 07:35:32.658 2180 INFO nova.virt.libvirt.driver [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] Deletion of /var/lib/nova/instances/c13062f0-1fbc-4425-ae07-b4e8babe785b_del complete 2014-12-24 07:35:32.876 2180 INFO nova.scheduler.client.report [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] Compute_service record updated for ('nodei14', 'nodei14.englab.juniper.net') 2014-12-24 07:35:32.950 2180 INFO nova.network.neutronv2.api [req-f5ae398e-014f-4ab6-821e-deae98200d10 None] [instance: c13062f0-1fbc-4425-ae07-b4e8babe785b] Unable to reset device ID for port None
Hit this issue in [Build "Ubuntu 14.04 LTS" 2.0-22~juno] MultiNode Single Intf Sanity
While trying to spawn vm in testcase 'test_mx_gateway '
Topology :
Config Nodes : [u'nodei11', u'nodei12']
Control Nodes : [u'nodei12', u'nodei13']
Compute Nodes : [u'nodei14', u'nodei15']
Openstack Node : nodei11
WebUI Node : nodei11
Analytics Nodes : [u'nodei11']
/var/log/ nova/nova- compute. log :
2014-12-24 07:35:29.582 2180 AUDIT nova.compute.claims [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] Claim successful client. report [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] Compute_service record updated for ('nodei14', 'nodei14. englab. juniper. net') client. report [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] Compute_service record updated for ('nodei14', 'nodei14. englab. juniper. net') libvirt. driver [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] Creating image client. report [-] Compute_service record updated for ('nodei14', 'nodei14. englab. juniper. net') disk.vfs. api [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] Unable to import guestfsfalling back to VFSLocalFS debug.hub_ prevent_ multiple_ readers( False) python2. 7/dist- packages/ contrail_ vrouter_ api/vrouter_ api.py" , line 171, in periodic_ connection_ check KeepAliveCheck( ) python2. 7/dist- packages/ contrail_ vrouter_ api/gen_ py/instance_ service/ InstanceService .py", line 176, in KeepAliveCheck KeepAliveCheck( ) python2. 7/dist- packages/ contrail_ vrouter_ api/gen_ py/instance_ service/ InstanceService .py", line 186, in recv_KeepAliveCheck readMessageBegi n() python2. 7/dist- packages/ thrift/ protocol/ TBinaryProtocol .py", line 126, in readMessageBegin python2. 7/dist- packages/ thrift/ protocol/ TBinaryProtocol .py", line 206, in readI32 readAll( 4) python2. 7/dist- packages/ thrift/ transport/ TTransport. py", line 58, in readAll python2. 7/dist- packages/ thrift/ transport/ TTransport. py", line 271, in read python2. 7/dist- packages/ thrift/ transport/ TTransport. py", line 275, in readFrame trans.readAll( 4) python2. 7/dist- packages/ thrift/ transport/ TTransport. py", line 58, in readAll python2. 7/dist- packages/ thrift/ transport/ TSocket. py", line 103, in read recv(sz) python2. 7/dist- packages/ eventlet/ greenio. py", line 262, in recv exc=socket. timeout( "timed out")) python2. 7/dist- packages/ eventlet/ hubs/__ init__. py", line 151, in trampoline python2. 7/dist- packages/ eventlet/ hubs/epolls. py", line 48, in add python2. 7/dist- packages/ eventlet/ hubs/hub. py", line 126, in add debug.hub_ prevent_ multiple_ readers( False) manager [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] Instance failed to spawn manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] Traceback (most recent call last): manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 2234, in _build_resources manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] yield resources manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] File "/usr/lib/ python2. 7/dist- packages/ nova/compute/ manager. py", line 2104, in _build_ and_run_ instance manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] block_device_ info=block_ device_ info) manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ driver. py", line 2622, in spawn manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] block_device_info, disk_info= disk_info) manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ driver. py", line 4424, in _create_ domain_ and_network manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] self.plug_ vifs(instance, network_info) manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] File "/usr/lib/ python2. 7/dist- packages/ nova/virt/ libvirt/ driver. py", line 947, in plug_vifs manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] self.vif_ driver. plug(instance, vif) manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] File "/usr/lib/ python2. 7/dist- packages/ nova_contrail_ vif/contrailvif .py", line 207, in plug manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] **kwargs) manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] File "/usr/lib/ python2. 7/dist- packages/ contrail_ vrouter_ api/vrouter_ api.py" , line 132, in add_port manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] result = self._client. AddPort( [data]) manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] AttributeError: 'NoneType' object has no attribute 'AddPort' manager [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] manager [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] Terminating instance libvirt. driver [-] [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] During wait destroy, instance disappeared. libvirt. driver [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] Deleting instance files /var/lib/ nova/instances/ c13062f0- 1fbc-4425- ae07-b4e8babe78 5b_del libvirt. driver [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] Deletion of /var/lib/ nova/instances/ c13062f0- 1fbc-4425- ae07-b4e8babe78 5b_del complete client. report [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] Compute_service record updated for ('nodei14', 'nodei14. englab. juniper. net') neutronv2. api [req-f5ae398e- 014f-4ab6- 821e-deae98200d 10 None] [instance: c13062f0- 1fbc-4425- ae07-b4e8babe78 5b] Unable to reset device ID for port None
2014-12-24 07:35:29.754 2180 INFO nova.scheduler.
2014-12-24 07:35:29.938 2180 INFO nova.scheduler.
2014-12-24 07:35:30.524 2180 INFO nova.virt.
2014-12-24 07:35:31.093 2180 INFO nova.scheduler.
2014-12-24 07:35:31.303 2180 INFO nova.virt.
2014-12-24 07:35:31.861 2180 ERROR root [-] Second simultaneous read on fileno 21 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.
2014-12-24 07:35:31.861 2180 TRACE root Traceback (most recent call last):
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root self._client.
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root return self.recv_
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root (fname, mtype, rseqid) = self._iprot.
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root sz = self.readI32()
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root buff = self.trans.
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root chunk = self.read(sz - have)
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root self.readFrame()
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root buff = self.__
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root chunk = self.read(sz - have)
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root buff = self.handle.
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root timeout_
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root listener = hub.add(hub.READ, fileno, current.switch)
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root listener = BaseHub.add(self, evtype, fileno, cb)
2014-12-24 07:35:31.861 2180 TRACE root File "/usr/lib/
2014-12-24 07:35:31.861 2180 TRACE root evtype, fileno, evtype))
2014-12-24 07:35:31.861 2180 TRACE root RuntimeError: Second simultaneous read on fileno 21 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.
2014-12-24 07:35:31.861 2180 TRACE root
2014-12-24 07:35:32.372 2180 ERROR nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.372 2180 TRACE nova.compute.
2014-12-24 07:35:32.444 2180 AUDIT nova.compute.
2014-12-24 07:35:32.460 2180 WARNING nova.virt.
2014-12-24 07:35:32.657 2180 INFO nova.virt.
2014-12-24 07:35:32.658 2180 INFO nova.virt.
2014-12-24 07:35:32.876 2180 INFO nova.scheduler.
2014-12-24 07:35:32.950 2180 INFO nova.network.