OpenStack Compute (nova)

  1. Bug #1252947
  2. Comment #3

Comment 3 for bug 1252947

Revision history for this message
Matt Riedemann (mriedem) wrote : Re: tempest.api.compute.servers.test_servers_negative.ServersNegativeTestJSON fails sporadically

Going back to the original logs, the instance does show up with the build failure in the nova scheduler logs here:

http://logs.openstack.org/66/54966/2/check/check-tempest-devstack-vm-full/d611ed0/logs/screen-n-sch.txt.gz#_2013-11-19_22_06_38_278

Which then shows up in the n-cpu logs here:

http://logs.openstack.org/66/54966/2/check/check-tempest-devstack-vm-full/d611ed0/logs/screen-n-cpu.txt.gz#_2013-11-19_22_06_33_235

2013-11-19 22:06:33.235 ERROR nova.compute.manager [req-873bc199-a51e-4f4c-a085-3ae084c0d4dc ServersNegativeTestJSON-tempest-616305902-user ServersNegativeTestJSON-tempest-616305902-tenant] [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] Instance failed to spawn
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] Traceback (most recent call last):
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/opt/stack/new/nova/nova/compute/manager.py", line 1436, in _spawn
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] block_device_info)
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2107, in spawn
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] block_device_info, context=context)
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 3296, in _create_domain_and_network
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] self.firewall_driver.setup_basic_filtering(instance, network_info)
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/opt/stack/new/nova/nova/virt/libvirt/firewall.py", line 305, in setup_basic_filtering
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] self.nwfilter.setup_basic_filtering(instance, network_info)
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/opt/stack/new/nova/nova/virt/libvirt/firewall.py", line 134, in setup_basic_filtering
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] vif))
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/opt/stack/new/nova/nova/virt/libvirt/firewall.py", line 249, in _define_filter
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] self._conn.nwfilterDefineXML(xml)
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] rv = execute(f,*args,**kwargs)
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] rv = meth(*args,**kwargs)
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2651, in nwfilterDefineXML
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] if ret is None:raise libvirtError('virNWFilterDefineXML() failed', conn=self)
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26] libvirtError: Cannot recv data: Connection reset by peer
2013-11-19 22:06:33.235 24775 TRACE nova.compute.manager [instance: 62bfeebd-8878-477f-9eac-a8b21ec5ac26]

This may be similar to bug 1255624, but the error is a bit different.

Here is the query:

message:"libvirtError: Cannot recv data: Connection reset by peer" AND filename:"logs/screen-n-cpu.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwibGlidmlydEVycm9yOiBDYW5ub3QgcmVjdiBkYXRhOiBDb25uZWN0aW9uIHJlc2V0IGJ5IHBlZXJcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1uLWNwdS50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiYWxsIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM5MDg0MTk1MjExNH0=

That has 3 hits in the last 2 weeks, so looks like it's still a problem. I'm surprised there isn't an existing bug/query for this yet.