Ubuntu
python-os-brick package

"FileNotFoundError: [Errno 2] No such file or directory" is raised when dmidecode is not installed

Bug #1994083 reported by Felipe Reyes
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned
Ubuntu Cloud Archive
New
Undecided
Unassigned
Zed
New
Undecided
Unassigned
os-brick
Fix Released
Low
Unassigned
python-os-brick (Ubuntu)
New
Undecided
Unassigned
Kinetic
Won't Fix
Undecided
Unassigned

Bug Description

On s390x there is no "dmidecode" package[0], with the inclusion of this change to get system uuid in privsep[1] nova regressed since privsep raises FileNotFoundError when there is an attempt to execute a program that doesn't exist, while subprocess would raise a higher level exception as subprocess.CalledProcessError.

Steps to reproduce:

- Deploy an OpenStack cloud with Zed release on s390x
- Launch an instance that boots from a volume, like tempest test tempest.api.compute.servers.test_create_server.ServersTestBootFromVolume

Expected result:

- The launched instance reaches active/running state

Actual result:

- The instance goes in error state with the folloing fault:

Details: Fault: {'code': 500, 'created': '2022-10-24T22:11:44Z', 'message': 'Build of instance 25879943-2e64-456c-b500-96a8a8735736 aborted: [Errno 2] No such file or directory'}. Server boot request ID: req-bb83a7cd-7cb2-4627-b4c8-4b6c05ba6068.

The /var/log/nova/nova-compute.log contains the following stacktrace:

49a 443c1aae740a4332bf85249b516d149a] [instance: 25879943-2e64-456c-b500-96a8a8735736] Instance failed block device setup: FileNotFoundError: [Errno 2] No such file or directory
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] Traceback (most recent call last):
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2087, in _prep_block_device
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] driver_block_device.attach_block_devices(
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 936, in attach_block_devi
ces
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] _log_and_attach(device)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 933, in _log_and_attach
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] bdm.attach(*attach_args, **attach_kwargs)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 46, in wrapped
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] ret_val = method(obj, context, *args, **kwargs)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 734, in attach
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] self._do_attach(context, instance, volume, volume_api,
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 713, in _do_attach
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] connector = virt_driver.get_volume_connector(instance)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 1783, in get_volume_con
nector
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] return connector.get_connector_properties(
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 171, in trace_logging_wrapper
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] return f(*args, **kwargs)
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/os_brick/initiator/connector.py", line 232, in get_connect
or_properties
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] connector.get_connector_properties(
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/nvmeof.py", line 772, in get
_connector_properties
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] suuid = priv_nvmeof.get_system_uuid()
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] return self.channel.remote_call(name, args, kwargs,
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 215, in remote_call
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] raise exc_type(*result[2])
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736] FileNotFoundError: [Errno 2] No such file or directory
2022-10-24 18:11:42.276 1115327 ERROR nova.compute.manager [instance: 25879943-2e64-456c-b500-96a8a8735736]
2022-10-24 18:11:43.134 1115327 ERROR nova.compute.manager [None req-bb83a7cd-7cb2-4627-b4c8-4b6c05ba6068 c1ae2fad28a24171badea4c539bb28a7 94686174ec6441559786f037d98e7159 - - 443c1aae740a4332bf85249b516d1
49a 443c1aae740a4332bf85249b516d149a] [instance: 25879943-2e64-456c-b500-96a8a8735736] Build of instance 25879943-2e64-456c-b500-96a8a8735736 aborted: [Errno 2] No such file or directory: nova.exception.Bu
ildAbortException: Build of instance 25879943-2e64-456c-b500-96a8a8735736 aborted: [Errno 2] No such file or directory

Environment:

- OpenStak Zed on Ubuntu Jammy

# dpkg -l | grep nova
ii nova-common 3:26.0.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 3:26.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 3:26.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 3:26.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii python3-nova 3:26.0.0-0ubuntu1~cloud0 all OpenStack Compute Python 3 libraries
ii python3-novaclient 2:18.1.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - 3.x

- The storage type is Ceph
- The networking is OpenVSwitch

[0] https://packages.ubuntu.com/jammy-updates/dmidecode
[1] https://review.opendev.org/c/openstack/os-brick/+/836057

Revision history for this message
Felipe Reyes (freyes) wrote :

Marking nova as invalid since the issue comes from os-brick code.

Changed in nova:
status: New → Invalid
no longer affects: nova (Ubuntu)
no longer affects: nova (Ubuntu Kinetic)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to os-brick (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/os-brick/+/862542

Changed in os-brick:
status: New → In Progress
Revision history for this message
Felipe Reyes (freyes) wrote :

The workaround I used is to create a symlink from /bin/false to /usr/local/bin/dmidecode in the nodes.

In the case of a juju driven cloud, you can run this command:

juju run -a nova-compute "sudo ln -s /bin/false /usr/local/bin/dmidecode"

Sofia Enriquez (lsofia-enriquez)
Changed in os-brick:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to os-brick (master)

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/862542
Committed: https://opendev.org/openstack/os-brick/commit/bc96120b7e709f345b110921954f44a5dc6c51c3
Submitter: "Zuul (22348)"
Branch: master

commit bc96120b7e709f345b110921954f44a5dc6c51c3
Author: Felipe Reyes <email address hidden>
Date: Mon Oct 24 20:31:11 2022 -0300

    Handle FileNotFoundError on get_system_uuid()

    On some platforms there is no 'dmidecode' program (e.g. s390x), when
    privsep attempts to run a program that's not found, it raises a
    FileNotFoundError instead of ProcessExecutionError.

    Closes-Bug: #1994083
    Change-Id: I6fc43ab7e243d50b74036d1af5f9e8f880401cc6

Changed in os-brick:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/os-brick 6.2.0

This issue was fixed in the openstack/os-brick 6.2.0 release.

Revision history for this message
Utkarsh Gupta (utkarsh) wrote :

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.

Changed in python-os-brick (Ubuntu Kinetic):
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.