SRIOVServersTest.test_create_server_with_VF intermittently fails due to "FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test'"

[Matt Riedemann]

Seen here:

http://logs.openstack.org/71/594571/2/gate/nova-tox-functional-py35/fd2d9ac/testr_results.html.gz

2018-08-24 16:36:47,192 ERROR [nova.compute.manager] Instance failed to spawn
Traceback (most recent call last):
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 2354, in _build_resources
    yield resources
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/compute/manager.py", line 2118, in _build_and_run_instance
    block_device_info=block_device_info)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/virt/libvirt/driver.py", line 3075, in spawn
    mdevs=mdevs)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/virt/libvirt/driver.py", line 5430, in _get_guest_xml
    context, mdevs)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/virt/libvirt/driver.py", line 5216, in _get_guest_config
    flavor, guest.os_type)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/virt/libvirt/driver.py", line 3995, in _get_guest_storage_config
    inst_type)
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/virt/libvirt/driver.py", line 3903, in _get_guest_disk_config
    self.disk_cachemode,
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/virt/libvirt/driver.py", line 416, in disk_cachemode
    if not nova.privsep.utils.supports_direct_io(CONF.instances_path):
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/privsep/utils.py", line 62, in supports_direct_io
    {'path': dirpath, 'ex': e})
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py35/lib/python3.5/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py35/lib/python3.5/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/zuul/src/git.openstack.org/openstack/nova/nova/privsep/utils.py", line 45, in supports_direct_io
    fd = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
FileNotFoundError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test'

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22FileNotFoundError%3A%20%5BErrno%202%5D%20No%20such%20file%20or%20directory%3A%20'%2Fhome%2Fzuul%2Fsrc%2Fgit.openstack.org%2Fopenstack%2Fnova%2Finstances%2F.directio.test'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

Just started, so it's likely related to these changes:

https://review.openstack.org/#/c/595802/ https://review.openstack.org/#/c/407055/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/596815

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/596815
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ecfcf8653815f658c0399b000e9386adc121312b
Submitter: Zuul
Branch: master

commit ecfcf8653815f658c0399b000e9386adc121312b
Author: Stephen Finucane <email address hidden>
Date: Mon Aug 27 16:28:32 2018 +0100

    privsep: Handle ENOENT when checking for direct IO support

    We've seen a recent issue that suggest direct IO support checks can fail
    in other valid ways than EINVAL, namely, failures with ENOENT or the
    FileNotFoundError exception, which is a Python 3-only exception type,
    can occur. While we can't test for this without breaking Python 2.7
    support, we can mimic this by looking for checking for the errno
    attribute of the OSError exception. Do this.

    Change-Id: I8aab86bb62cbc8ad538c706af037a30437c7964d
    Closes-Bug: #1788922

[Matt Riedemann (mriedem)]

FileNotFoundError is the py3 version of this failure, this is the py27 version:

OSError: [Errno 2] No such file or directory: '/home/zuul/src/git.openstack.org/openstack/nova/instances/.directio.test'

http://logs.openstack.org/33/526833/11/gate/nova-tox-functional/8bbb8e3/testr_results.html.gz

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.