Problems with HVCS and hotplugging
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
The Ubuntu-power-systems project |
Fix Committed
|
High
|
Ubuntu on IBM Power Systems Bug Triage | |||
linux (Ubuntu) | Status tracked in Noble | |||||
Focal |
Fix Released
|
High
|
Canonical Kernel Team | |||
Jammy |
Fix Committed
|
High
|
Canonical Kernel Team | |||
Mantic |
Invalid
|
Undecided
|
Unassigned | |||
Noble |
Invalid
|
Undecided
|
Unassigned |
Bug Description
SRU Justification:
==================
[Impact]
* HVCS (Hypervisor Virtual Console Server) is broken because the
virtual terminal mkvterm fails, caused by pvmutil failing.
* When mkvterm is ran, it ultimately fails because it calls pvmutil
which fails.
pvmutil calls drmgr, and drmgr is adding a slot correctly.
However, when drmgr writes the slot information to ?/add_slot,
the return is -ENODEV.
* This leads to HVCS never having probe() called.
* In addition, HVCS is missing patches/fixes, and is broken without them.
[Fix]
* Fix one and two is required for focal only, all other for focal and jammy:
* 57409d4fb12c 57409d4fb12c185
"powerpc/
* c5e76fa05b2d c5e76fa05b2df51
"powerpc/
* 6a9a733edd46 6a9a733edd46732
"hvcs: Fix hvcs port reference counting"
* 760aa5e81f33 760aa5e81f33e0d
"hvcs: Use dev_groups to manage hvcs device attributes"
* 503a90dd619d 503a90dd619d52d
"hvcs: Use driver groups to manage driver attributes"
* 3a8d3b366ce4 3a8d3b366ce4702
"hvcs: Get reference to tty in remove"
* d432228bc7b1 d432228bc7b1b3f
"hvcs: Use vhangup in hotplug remove"
* 28d49f8cbe9c 28d49f8cbe9c796
"hvcs: Synchronize hotplug remove with port free"
[Test Plan]
* The high level test plan is to run mkvterm with an id.
* mkvterm will fail because /dev/hvcs* device nodes are missing.
* Details see https:/
Especially the script provided by IBM
(see original bug description: `---Steps to Reproduce---`).
* IBM will (stress) test the updated kernel(s) provided in -proposed.
[Where problems could occur]
* The first two commits affect arch/powerpc/
and are needed to fix the hotplugging issue seen when drmgr goes to write
the slot information to /sys/bus/
In case of issues here hotplugging with drmgr might break.
* The issue lies in rpadlpar_io and rpaphp calling an of helper function
of_read_
drc_index_start is incorrect.
This ultimately results in the entire SLOT string being incorrect,
and rpaphp never finding the newly added slot by drmgr.
rpadlpar then returns -ENODEV.
Therefore, HVCS is never probed, and the device nodes are never created.
* HVCS, rpadlpar_io, and rpaphp should ideally not even need to be loaded
prior to drmgr adding a vio slot.
If rpadlpar_io and rpaphp are not loaded, drmgr will load them.
In addition, if rpadlpar_io and rpaphp register the new slot correctly,
rpadlpar_io will call dlpar_add_
which calls vio_register_
This is what tells the driver core to init and probe HVCS
(which is needed to create the device nodes).
* The remaning 6 commits are needed for HVCS, that is essentially
broken without them.
Overall, issues they fix are race conditions, hotplug remove issues,
as well as memory leaks.
* Please notice that this is entirely ppc64el architecture-
[Other Info]
* All the commits listed above are included in mantic and noble.
Hence these are set to Invalid.
* Meanwhile these requested commits have been added to other
kernels and distros.
__________
---Problem Description---
Issues with HVCS and hotplugging issues.
When working on Canonical bug 2023243, it was discovered that mkvterm was not working for multiple reasons. This bug will cover the issues found in HVCS, and hotplugging issues found when drmgr writes the slot information to .../add_slot.
When mkvterm is ran, it ultimately fails because it calls pvmutil which fails. pvmutil calls drmgr, and drmgr is adding a slot correctly. However, when drmgr writes the slot information to ?/add_slot, the return is -ENODEV. This leads to HVCS never having probe() called. In addition, HVCS is missing patches, and is broken without them. 8 kernel patches have been identified to fix these issues.
---uname output---
Linux neop91.
---Steps to Reproduce---
Run mkvterm with an id. mkvterm will fail because /dev/hvcs* device nodes are missing. See https:/
-------
2 commits made to arch/powerpc/
The issue lies in rpadlpar_io and rpaphp calling an of helper function of_read_
Ideally - HVCS, rpadlpar_io, and rpaphp should not even need to be loaded prior to drmgr adding a vio slot. If rpadlpar_io and rpaphp are not loaded, drmgr will load them. In addition, if rpadlpar_io and rpaphp register the new slot correctly, rpadlpar_io will call dlpar_add_
In addition to the 2 commits mentioned above, 6 HVCS commits are needed. HVCS is essentially broken without them. Issues include race conditions, hotplug remove issues, as well as memory leaks. These commits have been added to other distros after multiple issues were seen. Without these commits, 20.04 will experience the same issues. IBM plans on stress testing these changes after an updated kernel is provided in focal-proposed.
-------
2 commits that make changes to arch/powerpc/
powerpc/pseries: Fix bad drc_index_start value parsing of drc-info entry
https:/
powerpc/pseries: Fix of_read_
https:/
HVCS commits:
hvcs: Fix hvcs port reference counting
https:/
hvcs: Use dev_groups to manage hvcs device attributes
https:/
hvcs: Use driver groups to manage driver attributes
https:/
hvcs: Get reference to tty in remove
https:/
hvcs: Use vhangup in hotplug remove
https:/
hvcs: Synchronize hotplug remove with port free
https:/
CVE References
tags: | added: architecture-ppc64le bugnameltc-205798 severity-high targetmilestone-inin2004 |
Changed in ubuntu: | |
assignee: | nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) |
affects: | ubuntu → linux (Ubuntu) |
Changed in ubuntu-power-systems: | |
assignee: | nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) |
Changed in linux (Ubuntu): | |
assignee: | Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → nobody |
Changed in ubuntu-power-systems: | |
importance: | Undecided → High |
summary: |
- Multiple issues found on Ubuntu 20.04 against HVCS + Problems with HVCS and hotplugging |
description: | updated |
Changed in linux (Ubuntu Focal): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu Jammy): | |
status: | In Progress → Fix Committed |
Changed in ubuntu-power-systems: | |
status: | In Progress → Fix Committed |
tags: |
added: verification-done-focal-linux-iot verification-done-focal-linux-kvm verification-done-focal-linux-oracle verification-done-focal-linux-raspi verification-done-jammy-linux removed: verification-needed-focal-linux-iot verification-needed-focal-linux-kvm verification-needed-focal-linux-oracle verification-needed-focal-linux-raspi verification-needed-jammy-linux |
tags: |
added: verification-done-focal-linux-bluefield verification-done-focal-linux-fips verification-done-focal-linux-gcp verification-done-focal-linux-gkeop verification-done-focal-linux-ibm removed: verification-needed-focal-linux-bluefield verification-needed-focal-linux-fips verification-needed-focal-linux-gcp verification-needed-focal-linux-gkeop verification-needed-focal-linux-ibm |
tags: |
added: verification-done-bionic-linux-hwe-5.4 verification-done-focal-linux-aws verification-done-focal-linux-azure removed: verification-needed-bionic-linux-hwe-5.4 verification-needed-focal-linux-aws verification-needed-focal-linux-azure |
Hello and many thanks for raising these HVCS issue.
I first of all analyzed the commits upstream and in the specific Ubuntu kernel trees and found that focal and to some extend jammy are affected. Mantic and noble are not affected:
commit 57409d4fb12c got upstream accepted with v5.5(-rc1)
commit c5e76fa05b2d got upstream accepted with v5.7(-rc1)
commit 6a9a733edd46 got upstream accepted with v6.3(-rc1)
commit 760aa5e81f33 got upstream accepted with v6.3(-rc1)
commit 503a90dd619d got upstream accepted with v6.3(-rc1)
commit 3a8d3b366ce4 got upstream accepted with v6.3(-rc1)
commit d432228bc7b1 got upstream accepted with v6.3(-rc1)
commit 28d49f8cbe9c got upstream accepted with v6.3(-rc1)
commits/release focal jammy mantic noble
57409d4fb12c n y y y
c5e76fa05b2d n y y y
6a9a733edd46 n n y y
760aa5e81f33 n n y y
503a90dd619d n n y y
3a8d3b366ce4 n n y y
d432228bc7b1 n n y y
28d49f8cbe9c n n y y
So I'm adding focal and jammy as targets.