cpu_power_management_strategy=cpu_state failes if the host supports no governors

Bug #2045966 reported by Balazs Gibizer
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Balazs Gibizer

Bug Description

The nova-compute service with libvirt driver fails to start with power management enabled with cpu_startegy if the host OS only support offlining CPUs but does not support any cpufreq governor.

The cpu_state strategy should not depend on any cpufreq governor so nova-compute should start.

[root@edpm-compute-0 ~]# egrep -e 'cpu_dedicated|power|strategy' /var/lib/openstack/config/nova/ -R
/var/lib/openstack/config/nova/01-nova.conf:cpu_power_management=true
/var/lib/openstack/config/nova/05-nova-extra.conf:cpu_dedicated_set = 1

2023-12-08 09:54:43.865 2 ERROR oslo_service.service [None req-7109fd97-f101-452e-abcd-37c0b52b649e - - - - - -] Error starting thread.: nova.exception.FileNotFound: File /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor could not be found.
2023-12-08 09:54:43.865 2 ERROR oslo_service.service Traceback (most recent call last):
2023-12-08 09:54:43.865 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/filesystem.py", line 37, in read_sys
2023-12-08 09:54:43.865 2 ERROR oslo_service.service with open(os.path.join(SYS, path), mode='r') as data:
2023-12-08 09:54:43.865 2 ERROR oslo_service.service FileNotFoundError: [Errno 2] No such file or directory: '/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor'
2023-12-08 09:54:43.865 2 ERROR oslo_service.service
2023-12-08 09:54:43.865 2 ERROR oslo_service.service The above exception was the direct cause of the following exception:
2023-12-08 09:54:43.865 2 ERROR oslo_service.service
2023-12-08 09:54:43.865 2 ERROR oslo_service.service Traceback (most recent call last):
2023-12-08 09:54:43.865 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/oslo_service/service.py", line 806, in run_service
2023-12-08 09:54:43.865 2 ERROR oslo_service.service service.start()
2023-12-08 09:54:43.865 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/service.py", line 162, in start
2023-12-08 09:54:43.865 2 ERROR oslo_service.service self.manager.init_host(self.service_ref)
2023-12-08 09:54:43.865 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 1608, in init_host
2023-12-08 09:54:43.865 2 ERROR oslo_service.service self.driver.init_host(host=self.host)
2023-12-08 09:54:43.865 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 825, in init_host
2023-12-08 09:54:43.865 2 ERROR oslo_service.service libvirt_cpu.validate_all_dedicated_cpus()
2023-12-08 09:54:43.865 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/cpu/api.py", line 140, in validate_all_dedicated_cpus
2023-12-08 09:54:43.865 2 ERROR oslo_service.service governors.add(pcpu.governor)
2023-12-08 09:54:43.865 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/cpu/api.py", line 63, in governor
2023-12-08 09:54:43.865 2 ERROR oslo_service.service return core.get_governor(self.ident)
2023-12-08 09:54:43.865 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/cpu/core.py", line 69, in get_governor
2023-12-08 09:54:43.865 2 ERROR oslo_service.service return filesystem.read_sys(
2023-12-08 09:54:43.865 2 ERROR oslo_service.service File "/usr/lib/python3.9/site-packages/nova/filesystem.py", line 40, in read_sys
2023-12-08 09:54:43.865 2 ERROR oslo_service.service raise exception.FileNotFound(file_path=path) from exc
2023-12-08 09:54:43.865 2 ERROR oslo_service.service nova.exception.FileNotFound: File /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor could not be found.

[root@edpm-compute-0 ~]# ll /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
ls: cannot access '/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor': No such file or directory
[root@edpm-compute-0 ~]# ll /sys/devices/system/cpu/cpu1/cpufreq/
ls: cannot access '/sys/devices/system/cpu/cpu1/cpufreq/': No such file or directory
[root@edpm-compute-0 ~]# ll /sys/devices/system/cpu/cpu1/
total 0
drwxr-xr-x. 6 root root 0 Dec 8 10:01 cache
-r--------. 1 root root 4096 Dec 8 09:21 crash_notes
-r--------. 1 root root 4096 Dec 8 09:21 crash_notes_size
lrwxrwxrwx. 1 root root 0 Dec 8 09:21 driver -> ../../../../bus/cpu/drivers/processor
lrwxrwxrwx. 1 root root 0 Dec 8 09:21 firmware_node -> ../../../LNXSYSTM:00/LNXSYBUS:00/ACPI0010:00/LNXCPU:01
drwxr-xr-x. 2 root root 0 Dec 8 09:21 hotplug
lrwxrwxrwx. 1 root root 0 Dec 8 09:21 node0 -> ../../node/node0
-rw-r--r--. 1 root root 4096 Dec 8 10:01 online
drwxr-xr-x. 2 root root 0 Dec 8 09:21 power
lrwxrwxrwx. 1 root root 0 Dec 8 09:21 subsystem -> ../../../../bus/cpu
drwxr-xr-x. 2 root root 0 Dec 8 10:01 topology
-rw-r--r--. 1 root root 4096 Dec 8 09:21 uevent

[root@edpm-compute-0 ~]# cat /etc/*release*
CentOS Stream release 9
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
CentOS Stream release 9
CentOS Stream release 9
cpe:/o:centos:centos:9

[root@edpm-compute-0 ~]# uname -a
Linux edpm-compute-0 5.14.0-390.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 24 10:44:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Tags: libvirt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/903169

Changed in nova:
status: New → In Progress
tags: added: libvirt
Changed in nova:
importance: Undecided → Medium
assignee: nobody → Balazs Gibizer (balazs-gibizer)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/903169
Committed: https://opendev.org/openstack/nova/commit/ed2ac71a46b6caf620ddc08219ebae31f5af197d
Submitter: "Zuul (22348)"
Branch: master

commit ed2ac71a46b6caf620ddc08219ebae31f5af197d
Author: Balazs Gibizer <email address hidden>
Date: Fri Dec 8 12:10:11 2023 +0100

    [pwmgmt]ignore missin governor when cpu_state used

    When cpu_state power management strategy is requested nova-compute
    should not fail to start if there is no cpufreq governor is supported by
    the host.

    Closes-Bug: #2045966
    Change-Id: Ice2fa47bdab320a7e472fbb4767761448d176bad

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/nova/+/903151

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/nova/+/903151
Committed: https://opendev.org/openstack/nova/commit/3c054dc6c5beaba5187a52d86ebdacb8e5f052d9
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 3c054dc6c5beaba5187a52d86ebdacb8e5f052d9
Author: Balazs Gibizer <email address hidden>
Date: Fri Dec 8 12:10:11 2023 +0100

    [pwmgmt]ignore missin governor when cpu_state used

    When cpu_state power management strategy is requested nova-compute
    should not fail to start if there is no cpufreq governor is supported by
    the host.

    Closes-Bug: #2045966
    Change-Id: Ice2fa47bdab320a7e472fbb4767761448d176bad
    (cherry picked from commit ed2ac71a46b6caf620ddc08219ebae31f5af197d)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/903553

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/nova/+/903553
Committed: https://opendev.org/openstack/nova/commit/41096f83426bc21bd35b8a1dce0e103ba48c3140
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 41096f83426bc21bd35b8a1dce0e103ba48c3140
Author: Balazs Gibizer <email address hidden>
Date: Fri Dec 8 12:10:11 2023 +0100

    [pwmgmt]ignore missin governor when cpu_state used

    When cpu_state power management strategy is requested nova-compute
    should not fail to start if there is no cpufreq governor is supported by
    the host.

    Closes-Bug: #2045966
    Change-Id: Ice2fa47bdab320a7e472fbb4767761448d176bad
    (cherry picked from commit ed2ac71a46b6caf620ddc08219ebae31f5af197d)
    (cherry picked from commit 3c054dc6c5beaba5187a52d86ebdacb8e5f052d9)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 29.0.0.0rc1

This issue was fixed in the openstack/nova 29.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 27.3.0

This issue was fixed in the openstack/nova 27.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 28.1.0

This issue was fixed in the openstack/nova 28.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.