Zun

OSError on executing 'numactl'

Bug #1808643 reported by hongbin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zun
Fix Released
Critical
hongbin

Bug Description

Here is the failure in Kolla's gate:

2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task [-] Error during Manager.inventory_host: OSError: [Errno 2] No such file or directory
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task Traceback (most recent call last):
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_service/periodic_task.py", line 224, in run_periodic_tasks
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task task(self, context)
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/compute/manager.py", line 1107, in inventory_host
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task rt.update_available_resources(context)
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/compute/compute_node_tracker.py", line 57, in update_available_resources
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task numa_obj = self.container_driver.get_host_numa_topology()
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/container/driver.py", line 183, in get_host_numa_topology
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task os_capability_linux.LinuxHost().get_host_numa_topology(numa_topo_obj)
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/container/os_capability/host_capability.py", line 41, in get_host_numa_topology
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task mem_info = self.get_mem_numa_info()
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/container/os_capability/linux/os_capability_linux.py", line 58, in get_mem_numa_info
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task output = utils.execute('numactl', '-H')
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/common/utils.py", line 356, in execute
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task return custom_execute(*cmd, **kwargs)
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/zun/common/utils.py", line 327, in custom_execute
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task return processutils.execute(*cmd, **kwargs)
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 391, in execute
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task env=env_variables)
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 58, in __init__
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task errread, errwrite)
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task raise child_exception
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task OSError: [Errno 2] No such file or directory
2018-12-13 19:20:24.664 6 ERROR oslo_service.periodic_task

Example: http://logs.openstack.org/41/617341/14/check/kolla-ansible-ubuntu-source-zun/540bfa3/primary/logs/kolla/zun/zun-compute.txt.gz

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to zun (master)

Fix proposed to branch: master
Review: https://review.openstack.org/625400

Changed in zun:
assignee: nobody → hongbin (hongbin034)
status: New → In Progress
hongbin (hongbin034)
Changed in zun:
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to zun (master)

Reviewed: https://review.openstack.org/625400
Committed: https://git.openstack.org/cgit/openstack/zun/commit/?id=c0a17036cba7016ea30304704cc19588227978ee
Submitter: Zuul
Branch: master

commit c0a17036cba7016ea30304704cc19588227978ee
Author: Hongbin Lu <email address hidden>
Date: Sat Dec 15 17:12:27 2018 +0000

    Handle the case that 'numactl' not installed

    If the program 'numactl' is not installed, the Zun compute process
    fails and the compute node is not up. This commit makes Zun to
    be robust in this case.

    Change-Id: I0969d52896cc89fe443f312debe2b4f477069952
    Closes-Bug: #1808643

Changed in zun:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/zun 3.0.0.0rc1

This issue was fixed in the openstack/zun 3.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.