test_resize_server_revert_deleted_flavor fails due to "No valid host was found"

[Ken'ichi Ohmichi]

test_resize_server_revert_deleted_flavor failed like the following on the gate.

http://logs.openstack.org/48/426448/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/a9e5c36/logs/

Traceback (most recent call last):
  File "tempest/api/compute/admin/test_migrations.py", line 90, in test_resize_server_revert_deleted_flavor
    self.servers_client.resize_server(server['id'], self.flavor_ref_alt)
  File "tempest/lib/services/compute/servers_client.py", line 274, in resize_server
    return self.action(server_id, 'resize', **kwargs)
  File "tempest/lib/services/compute/servers_client.py", line 186, in action
    post_body)
  File "tempest/lib/common/rest_client.py", line 276, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
    method, url, extra_headers, headers, body, chunked)
  File "tempest/lib/common/rest_client.py", line 664, in request
    self._error_checker(resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 766, in _error_checker
    raise exceptions.BadRequest(resp_body, resp=resp)
tempest.lib.exceptions.BadRequest: Bad request
Details: {u'code': 400, u'message': u'No valid host was found. No valid host found for resize'}

[Ken'ichi Ohmichi (oomichi)]

According to nova-sched.log nova-sched could not find valid host on ComputeFilter:

Filter results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'RamFilter: (start: 1, end: 1)', 'DiskFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 0)']

because the host is disabled due to "Connection to libvirt lost" like

(ubuntu-xenial-bluebox-sjc1-7106549, ubuntu-xenial-bluebox-sjc1-7106549) ram: 7086MB disk: 49152MB io_ops: 4 instances: 5 is disabled, reason: AUTO: Connection to libvirt lost: 0 host_passes

from nova-sched.log

[Ken'ichi Ohmichi (oomichi)]

libvirt.log contains the following error messages before the error happened:

2017-02-03 18:50:35.818+0000: 16812: error : qemuMonitorIO:689 : internal error: End of file from monitor
2017-02-03 18:50:35.818+0000: 16812: debug : qemuMonitorIO:738 : Error on monitor internal error: End of file from monitor
2017-02-03 18:50:35.818+0000: 16812: debug : qemuMonitorIO:759 : Triggering EOF callback
[..]
2017-02-03 18:50:42.147+0000: 16816: error : qemuDomainAgentAvailable:3599 : argument unsupported: QEMU guest agent is not configured
[..]
2017-02-03 18:50:53.306+0000: 16814: error : virNetClientProgramDispatchError:177 : Cannot open log file: '/var/log/libvirt/qemu/instance-0000000b.log': Device or resource busy
[..]
2017-02-03 18:50:54.630+0000: 16812: error : qemuMonitorIORead:580 : Unable to read from monitor: Connection reset by peer

[Ken'ichi Ohmichi (oomichi)]

http://logs.openstack.org/48/426448/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/a9e5c36/logs/syslog.txt.gz#_Feb_03_18_50_54

Feb 03 18:50:54 ubuntu-xenial-bluebox-sjc1-7106549 kernel: qbr03d589a9-1f: port 2(tap03d589a9-1f) entered disabled state
Feb 03 18:50:54 ubuntu-xenial-bluebox-sjc1-7106549 kernel: device tap03d589a9-1f left promiscuous mode
Feb 03 18:50:54 ubuntu-xenial-bluebox-sjc1-7106549 kernel: qbr03d589a9-1f: port 2(tap03d589a9-1f) entered disabled state
Feb 03 18:50:54 ubuntu-xenial-bluebox-sjc1-7106549 virtlogd[16840]: End of file while reading data: Input/output error
Feb 03 18:50:54 ubuntu-xenial-bluebox-sjc1-7106549 kernel: qbr03d589a9-1f: port 1(qvb03d589a9-1f) entered disabled state
Feb 03 18:50:54 ubuntu-xenial-bluebox-sjc1-7106549 ovs-vsctl[17486]: ovs|00001|vsctl|INFO|Called as ovs-vsctl --timeout=120 -- --if-exists del-port br-int qvo03d589a9-1f
Feb 03 18:50:54 ubuntu-xenial-bluebox-sjc1-7106549 libvirtd[16812]: *** Error in `/usr/sbin/libvirtd': malloc(): memory corruption: 0x000055fe7d977ca0 ***