drmgr command fails during the scale-up test on Novalink System (Brazos)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
The Ubuntu-power-systems project |
Fix Released
|
High
|
Unassigned | ||
powerpc-utils (Ubuntu) |
Fix Released
|
High
|
Ubuntu on IBM Power Systems Bug Triage | ||
Xenial |
Fix Released
|
High
|
Steve Langasek | ||
Yakkety |
Fix Committed
|
High
|
Steve Langasek | ||
Zesty |
Fix Released
|
High
|
Steve Langasek |
Bug Description
[SRU Justification]
drmgr fails intermittently when adding devices to the system.
[Test case]
To be completed by IBM, who have access to the hardware.
1. Run a scale test of launching 1000 VMs on a Novalink system.
2. Observe that some of the deployments fail with the following error:
kernel I/O op failed, rc = 26 len = 26.
3. Install powerpc-utils from -proposed
4. Run the scale test again.
5. Observe that all the deployments succeed.
[Regression potential]
This change cherry-picked from upstream corrects faulty handling of a 0 return code from syscalls. Regression potential appears to be minimal.
Problem:
During the scale-up test to 1000 VMs I could see 20 deploys failed due to following command failure..
Command /usr/sbin/pvmdrmgr drmgr -c slot -s 'U9119.
Validating I/O DLPAR capability...yes.
kernel I/O op failed, rc = 26 len = 26.
I have been looking through the logs on this system to piece together what is happening when the dlpar add failures occur. From what I am seeing we are trying to dlpar add a virtual network device and getting a error when trying to add the device to the system.
> ########## May 17 05:18:00 2017 ##########
> drmgr: -c slot -s U9119.MHE.
> Validating I/O DLPAR capability...yes.
> Getting node types 0x00000003
> Could not find DRC property group in path: /proc/device-
> Acquiring drc index 0x30000406
> get-sensor for 30000406: 0, 2
> Setting allocation state to 'alloc usable'
> Setting indicator state to 'unisolate'
> Configuring connector for drc index 30000406
> Adding device-tree node /proc/device-
> ofdt update: add_node /vdevice/
> Getting node types 0x00000003
> performing kernel op for U9119.MHE.
> kernel I/O op failed, rc = 26 len = 26.
> No such device
> Releasing drc index 0x30000406
> get-sensor for 30000406: 0, 1
> Setting isolation state to 'isolate'
> Setting allocation state to 'alloc unusable'
> get-sensor for 30000406: 0, 2
> drc_index 30000406 sensor-state: 2
> Resource is not available to the partition.
> Removing device-tree node /proc/device-
> ########## May 17 05:20:11 2017 ##########
From the drmgr log, you can see that we get a ENODEV return code when performing the kernel operation to add the device to the system.
> performing kernel op for U9119.MHE.
> kernel I/O op failed, rc = 26 len = 26.
> No such device
This indicates that the rpadlpar_io kernel modules was unable to find the device in the device tree. This doesn not seem right because earlier in the drmgr logs we add the device to the device tree. Additionally, the drmgr code validates that the add succeeds by retrieveing the newly added device node from the device tree as a sanity check. There are no failures reported for this.
> Adding device-tree node /proc/device-
> ofdt update: add_node /vdevice/
> Getting node types 0x00000003
I started scale-up testing and I could see deploys are going fine. Will post a comment here if I see further drmgr failures.
Patches have been submitted upstream.
https:/
and
https:/
tags: | added: architecture-ppc64le bugnameltc-154853 severity-high targetmilestone-inin16043 |
Changed in ubuntu: | |
assignee: | nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) |
affects: | ubuntu → powerpc-ibm-utils (Ubuntu) |
Changed in ubuntu-power-systems: | |
status: | New → Confirmed |
affects: | powerpc-ibm-utils (Ubuntu) → powerpc-utils (Ubuntu) |
description: | updated |
Changed in powerpc-utils (Ubuntu Xenial): | |
milestone: | none → ubuntu-16.04.3 |
assignee: | nobody → Steve Langasek (vorlon) |
Changed in powerpc-utils (Ubuntu Yakkety): | |
assignee: | nobody → Steve Langasek (vorlon) |
Changed in powerpc-utils (Ubuntu Zesty): | |
assignee: | nobody → Steve Langasek (vorlon) |
Changed in powerpc-utils (Ubuntu Xenial): | |
status: | New → In Progress |
Changed in powerpc-utils (Ubuntu Yakkety): | |
status: | New → In Progress |
Changed in powerpc-utils (Ubuntu Zesty): | |
status: | New → In Progress |
Changed in ubuntu-power-systems: | |
status: | Confirmed → Fix Committed |
tags: | added: verification-done-xenial |
Changed in powerpc-utils (Ubuntu): | |
importance: | Undecided → High |
Changed in powerpc-utils (Ubuntu Xenial): | |
importance: | Undecided → High |
Changed in powerpc-utils (Ubuntu Yakkety): | |
importance: | Undecided → High |
Changed in powerpc-utils (Ubuntu Zesty): | |
importance: | Undecided → High |
Changed in ubuntu-power-systems: | |
importance: | Undecided → High |
tags: | added: triage-g |
tags: |
added: verification-done-zesty removed: verification-needed |
Changed in ubuntu-power-systems: | |
status: | Fix Committed → Fix Released |
Status changed to 'Confirmed' because the bug affects multiple users.