Someone should look into this bug.VM spawn failing due to this
2014-09-23 05:15:09.974 5897 ERROR nova.compute.manager [req-5a9a7595-e37b-44fb-aa67-85a16fb90b57 4aa1c9b2265f427fa3a56dbd25706bcf fe88547b4886406aac0330e855a72a15] [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] Instance failed to spawn 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] Traceback (most recent call last): 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1410, in _spawn 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] block_device_info) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2071, in spawn 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] block_device_info, context=context) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3222, in _create_domain_and_network 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] self.plug_vifs(instance, network_info) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 748, in plug_vifs 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] self.vif_driver.plug(instance, vif) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/nova_contrail_vif/contrailvif.py", line 128, in plug 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] **kwargs) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/contrail_vrouter_api/vrouter_api.py", line 123, in add_port 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] result = self._client.AddPort([data]) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/contrail_vrouter_api/gen_py/instance_service/InstanceService.py", line 150, in AddPort 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] result = self._client.AddPort([data]) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/contrail_vrouter_api/gen_py/instance_service/InstanceService.py", line 150, in AddPort 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] return self.recv_AddPort() 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/contrail_vrouter_api/gen_py/instance_service/InstanceService.py", line 161, in recv_AddPort 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] (fname, mtype, rseqid) = self._iprot.readMessageBegin() 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib64/python2.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 126, in readMessageBegin 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] sz = self.readI32() 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib64/python2.6/site-packages/thrift/protocol/TBinaryProtocol.py", line 206, in readI32 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] buff = self.trans.readAll(4) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib64/python2.6/site-packages/thrift/transport/TTransport.py", line 58, in readAll 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] chunk = self.read(sz - have) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib64/python2.6/site-packages/thrift/transport/TTransport.py", line 271, in read 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] self.readFrame() 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib64/python2.6/site-packages/thrift/transport/TTransport.py", line 275, in readFrame 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] buff = self.__trans.readAll(4) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib64/python2.6/site-packages/thrift/transport/TTransport.py", line 58, in readAll 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] chunk = self.read(sz - have) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib64/python2.6/site-packages/thrift/transport/TSocket.py", line 103, in read 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] buff = self.handle.recv(sz) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 238, in recv 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] timeout_exc=socket.timeout("timed out")) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/eventlet/hubs/__init__.py", line 117, in trampoline 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] listener = hub.add(hub.READ, fileno, current.switch) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/eventlet/hubs/epolls.py", line 48, in add 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] listener = BaseHub.add(self, evtype, fileno, cb) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 126, in add 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] evtype, fileno, evtype)) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] RuntimeError: Second simultaneous read on fileno 9 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_multiple_reader_prevention(False) 2014-09-23 05:15:09.974 5897 TRACE nova.compute.manager [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] 2014-09-23 05:15:14.321 5897 ERROR nova.virt.libvirt.driver [-] [instance: 3d9284a7-4144-4ad6-b1cb-afd5c38bd44e] During wait destroy, instance disappeared.
Someone should look into this bug.VM spawn failing due to this
2014-09-23 05:15:09.974 5897 ERROR nova.compute. manager [req-5a9a7595- e37b-44fb- aa67-85a16fb90b 57 4aa1c9b2265f427 fa3a56dbd25706b cf fe88547b4886406 aac0330e855a72a 15] [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] Instance failed to spawn manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] Traceback (most recent call last): manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ nova/compute/ manager. py", line 1410, in _spawn manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] block_device_info) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 2071, in spawn manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] block_device_info, context=context) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 3222, in _create_ domain_ and_network manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] self.plug_ vifs(instance, network_info) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ nova/virt/ libvirt/ driver. py", line 748, in plug_vifs manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] self.vif_ driver. plug(instance, vif) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ nova_contrail_ vif/contrailvif .py", line 128, in plug manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] **kwargs) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ contrail_ vrouter_ api/vrouter_ api.py" , line 123, in add_port manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] result = self._client. AddPort( [data]) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ contrail_ vrouter_ api/gen_ py/instance_ service/ InstanceService .py", line 150, in AddPort manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] result = self._client. AddPort( [data]) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ contrail_ vrouter_ api/gen_ py/instance_ service/ InstanceService .py", line 150, in AddPort manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] return self.recv_AddPort() manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ contrail_ vrouter_ api/gen_ py/instance_ service/ InstanceService .py", line 161, in recv_AddPort manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] (fname, mtype, rseqid) = self._iprot. readMessageBegi n() manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib64/ python2. 6/site- packages/ thrift/ protocol/ TBinaryProtocol .py", line 126, in readMessageBegin manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] sz = self.readI32() manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib64/ python2. 6/site- packages/ thrift/ protocol/ TBinaryProtocol .py", line 206, in readI32 manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] buff = self.trans. readAll( 4) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib64/ python2. 6/site- packages/ thrift/ transport/ TTransport. py", line 58, in readAll manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] chunk = self.read(sz - have) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib64/ python2. 6/site- packages/ thrift/ transport/ TTransport. py", line 271, in read manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] self.readFrame() manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib64/ python2. 6/site- packages/ thrift/ transport/ TTransport. py", line 275, in readFrame manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] buff = self.__ trans.readAll( 4) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib64/ python2. 6/site- packages/ thrift/ transport/ TTransport. py", line 58, in readAll manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] chunk = self.read(sz - have) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib64/ python2. 6/site- packages/ thrift/ transport/ TSocket. py", line 103, in read manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] buff = self.handle. recv(sz) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ eventlet/ greenio. py", line 238, in recv manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] timeout_ exc=socket. timeout( "timed out")) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ eventlet/ hubs/__ init__. py", line 117, in trampoline manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] listener = hub.add(hub.READ, fileno, current.switch) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ eventlet/ hubs/epolls. py", line 48, in add manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] listener = BaseHub.add(self, evtype, fileno, cb) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] File "/usr/lib/ python2. 6/site- packages/ eventlet/ hubs/hub. py", line 126, in add manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] evtype, fileno, evtype)) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] RuntimeError: Second simultaneous read on fileno 9 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet. debug.hub_ multiple_ reader_ prevention( False) manager [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] libvirt. driver [-] [instance: 3d9284a7- 4144-4ad6- b1cb-afd5c38bd4 4e] During wait destroy, instance disappeared.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:09.974 5897 TRACE nova.compute.
2014-09-23 05:15:14.321 5897 ERROR nova.virt.