tempest

scheduling fails in tempest tests, host communication issues

Bug #1328338 reported by Joshua Harlow
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
Confirmed
Undecided
Unassigned

Bug Description

Seeing a new failure (or maybe this is an existing one), seems to be due to building ERROR?

Logs @ http://logs.openstack.org/64/98564/3/check/check-tempest-dsvm-neutron/9941781/console.html

2014-06-10 00:42:12.757 | Captured traceback:
2014-06-10 00:42:12.757 | ~~~~~~~~~~~~~~~~~~~
2014-06-10 00:42:12.757 | Traceback (most recent call last):
2014-06-10 00:42:12.757 | File "tempest/api/compute/servers/test_attach_interfaces.py", line 133, in test_add_remove_fixed_ip
2014-06-10 00:42:12.757 | server, ifs = self._create_server_get_interfaces()
2014-06-10 00:42:12.758 | File "tempest/api/compute/servers/test_attach_interfaces.py", line 48, in _create_server_get_interfaces
2014-06-10 00:42:12.758 | resp, server = self.create_test_server(wait_until='ACTIVE')
2014-06-10 00:42:12.758 | File "tempest/api/compute/base.py", line 247, in create_test_server
2014-06-10 00:42:12.758 | raise ex
2014-06-10 00:42:12.758 | BuildErrorException: Server e7c227f7-5eb2-4ec4-ba6a-1dcb833aa85e failed to build and is in ERROR status
2014-06-10 00:42:12.758 |

melanie witt (melwitt)
summary: - AttachInterfacesTestXML.test_add_remove_fixed_ip failed to build and is
- in ERROR status
+ various tests fail with BuildErrorException
Changed in tempest:
status: New → Confirmed
melanie witt (melwitt)
summary: - various tests fail with BuildErrorException
+ AttachInterfacesTestXML.test_add_remove_fixed_ip failed to build and is
+ in ERROR status
Revision history for this message
melanie witt (melwitt) wrote : Re: AttachInterfacesTestXML.test_add_remove_fixed_ip failed to build and is in ERROR status

This one failed at the scheduling step

2014-06-10 00:42:10.963 DEBUG nova.filters [req-247b18f2-2163-482b-8fd0-4f04193521b8 AttachInterfacesTestXML-245370658 AttachInterfacesTestXML-1095742173] Starting with 1 host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:70
2014-06-10 00:42:10.963 DEBUG nova.filters [req-247b18f2-2163-482b-8fd0-4f04193521b8 AttachInterfacesTestXML-245370658 AttachInterfacesTestXML-1095742173] Filter RetryFilter returned 1 host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:88
2014-06-10 00:42:10.964 DEBUG nova.filters [req-247b18f2-2163-482b-8fd0-4f04193521b8 AttachInterfacesTestXML-245370658 AttachInterfacesTestXML-1095742173] Filter AvailabilityZoneFilter returned 1 host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:88
2014-06-10 00:42:10.964 DEBUG nova.filters [req-247b18f2-2163-482b-8fd0-4f04193521b8 AttachInterfacesTestXML-245370658 AttachInterfacesTestXML-1095742173] Filter RamFilter returned 1 host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:88
2014-06-10 00:42:10.964 DEBUG nova.scheduler.filters.compute_filter [req-247b18f2-2163-482b-8fd0-4f04193521b8 AttachInterfacesTestXML-245370658 AttachInterfacesTestXML-1095742173] (devstack-precise-hpcloud-b5-256107, devstack-precise-hpcloud-b5-256107) ram:6778 disk:437248 io_ops:0 instances:3 is disabled, reason: AUTO: Failed to connect to libvirt host_passes /opt/stack/new/nova/nova/scheduler/filters/compute_filter.py:43
2014-06-10 00:42:10.964 INFO nova.filters [req-247b18f2-2163-482b-8fd0-4f04193521b8 AttachInterfacesTestXML-245370658 AttachInterfacesTestXML-1095742173] Filter ComputeFilter returned 0 hosts

http://logs.openstack.org/64/98564/3/check/check-tempest-dsvm-neutron/9941781/logs/screen-n-sch.txt.gz

Revision history for this message
melanie witt (melwitt) wrote :

logstash query: message:"Filter ComputeFilter returned 0 hosts" AND tags:"screen-n-sch.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmlsdGVyIENvbXB1dGVGaWx0ZXIgcmV0dXJuZWQgMCBob3N0c1wiIEFORCB0YWdzOlwic2NyZWVuLW4tc2NoLnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI5MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA1NDQ4MDIyMjYzfQ==

Revision history for this message
melanie witt (melwitt) wrote :

I spent some time poking around and I find the NoValidHost exception is no longer exposed, so the failures to schedule stay silent. The only log of the error shows as a warning in the conductor log as Exception is caught and logged in this way:

2014-06-10 00:42:10.968 WARNING nova.scheduler.driver [req-247b18f2-2163-482b-8fd0-4f04193521b8 AttachInterfacesTestXML-245370658 AttachInterfacesTestXML-1095742173] [instance: e7c227f7-5eb2-4ec4-ba6a-1dcb833aa85e] Setting instance to ERROR state.

http://logs.openstack.org/64/98564/3/check/check-tempest-dsvm-neutron/9941781/logs/screen-n-cond.txt.gz?level=TRACE

summary: - AttachInterfacesTestXML.test_add_remove_fixed_ip failed to build and is
- in ERROR status
+ scheduling fails in tempest tests, host communication issues
Revision history for this message
melanie witt (melwitt) wrote :

So far I see two different types of the error in the logstash query:

1. 2014-06-10 00:42:10.964 DEBUG nova.scheduler.filters.compute_filter [req-247b18f2-2163-482b-8fd0-4f04193521b8 AttachInterfacesTestXML-245370658 AttachInterfacesTestXML-1095742173] (devstack-precise-hpcloud-b5-256107, devstack-precise-hpcloud-b5-256107) ram:6778 disk:437248 io_ops:0 instances:3 is disabled, reason: AUTO: Failed to connect to libvirt host_passes

2. 2014-07-15 12:25:23.731 WARNING nova.scheduler.filters.compute_filter [req-f7f2bbfb-5439-4113-b9c0-15aad99266a0 ServersAdminNegativeTestXML-2129689411 ServersAdminNegativeTestXML-610854333] (devstack-trusty-rax-dfw-896533.slave.openstack.org, devstack-trusty-rax-dfw-896533.slave.openstack.org) ram:6835 disk:63488 io_ops:3 instances:6 has not been heard from in a while

Revision history for this message
melanie witt (melwitt) wrote :

I think this is a duplicate of bug 1331537

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.