udev race condition with qeth device and bridge_role

Bug #1618463 reported by Andrew McLeod on 2016-08-30
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Undecided
Dimitri John Ledkov
s390-tools (Ubuntu)
Undecided
Dimitri John Ledkov
Xenial
Undecided
Dimitri John Ledkov
Yakkety
Undecided
Dimitri John Ledkov
Zesty
Undecided
Dimitri John Ledkov

Bug Description

[Impact]

 * bridge_role property setting is racy on boot

 * This results in incorrect bridge mode set on the devices, sometimes, which leads to lack of desired connectivity (e.g. bridging internet to containers)

 * The fix for this issue is to set bridge_role, only after the device is online

 * Unfortunately the udev rules are not regenerated, therefore affected systemd must manually remove and recreate chzdev rules

[Test Case]

 * Remove qeth udev rules from /etc/udev/rules.d/
 * Enable qeth device using chzdev with a non-default bridge_role setting, e.g.:
   chzdev --no-root-update -pVe c003 bridge_role=primary;
 * Reboot and check that bridge_role setting is correctly set in the sysfs, e.g.:
   /sys/devices/qeth/0.0.c003/bridge_role

[Regression Potential]

 * Minimal, the generated udev rules remain the same; the only difference in the generated udev rules is the ordering in setting the bridge_role attribute

[Other Info]

 * Original bug report:

Attempting to set bridge_role = primary with the following command in preseed:

in-target chzdev --no-root-update -pVe c003 bridge_role=primary;

...works, and generates the following udev rule for this device:

https://pastebin.canonical.com/164271/

However, after reboot:

systemd-udevd[2634]: error opening ATTR{/sys/devices/qeth/0.0.c003/bridge_role} for writing: Permission denied

More logging:

https://pastebin.canonical.com/164272/

after the system has booted, we are able to write to the file and set bridge_role to primary:

root@10-13-3-10:/var/log# cat /sys/devices/qeth/0.0.c003/bridge_role
none
root@10-13-3-10:/var/log# echo primary > /sys/devices/qeth/0.0.c003/bridge_role
root@10-13-3-10:/var/log# cat /sys/devices/qeth/0.0.c003/bridge_role
primary

tags: added: s390x
Andrew McLeod (admcleod) on 2016-09-09
tags: added: uosci
Changed in s390-tools (Ubuntu):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in s390-tools (Ubuntu):
status: New → In Progress
Dimitri John Ledkov (xnox) wrote :

sudo add-apt-repository ppa:ci-train-ppa-service/2038
sudo apt update
sudo apt dist-upgrade

After that please regenerate persistent configuration for your qeth device, e.g.:

$ sudo chzdev -p -d 600
$ sudo chzdev -p -e 600 bridge_role=primary
$ sudo update-initramfs -u

undo any other manual changes to set bridge_role to primary, and finally reboot to test the packages with a proposed fix for your issue.

Please note packages are pending publication, however should be available shortly.

Changed in s390-tools (Ubuntu):
status: In Progress → Fix Committed
description: updated
Changed in s390-tools (Ubuntu Yakkety):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in s390-tools (Ubuntu Xenial):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in s390-tools (Ubuntu Zesty):
milestone: none → ubuntu-17.03
Changed in s390-tools (Ubuntu Yakkety):
milestone: none → yakkety-updates
Changed in s390-tools (Ubuntu Xenial):
milestone: none → xenial-updates
Changed in s390-tools (Ubuntu Yakkety):
status: New → In Progress
Changed in s390-tools (Ubuntu Xenial):
status: New → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 1.37.0-0ubuntu2

---------------
s390-tools (1.37.0-0ubuntu2) zesty; urgency=medium

  * chzdev/qeth: set bridge_role after the device is online to avoid a
    race condition when setting qeth attributes on boot. LP: #1618463.

 -- Dimitri John Ledkov <email address hidden> Tue, 14 Mar 2017 09:26:12 +0000

Changed in s390-tools (Ubuntu Zesty):
status: Fix Committed → Fix Released

Hello Andrew, or anyone else affected,

Accepted s390-tools into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/1.36.1-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in s390-tools (Ubuntu Yakkety):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in s390-tools (Ubuntu Xenial):
status: In Progress → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Andrew, or anyone else affected,

Accepted s390-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/1.34.0-0ubuntu8.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ubuntu-z-systems:
status: New → Fix Committed
assignee: nobody → Dimitri John Ledkov (xnox)
Andrew McLeod (admcleod) wrote :

Hi Brian,

I have tested "s390-tools 1.34.0-0ubuntu8.3" in Xenial and can confirm it fixes the bug in Xenial. I am not currently able to test in Yakkety or Zesty

Thanks!

Dimitri John Ledkov (xnox) wrote :

@admcleod

Based on feedback from IBM i'm going to withdraw this update. The ordering of the flags is correct, as long as one activates the flag needed to have bridge_role attribute available before onlining the device.

Specifically one should use:

in-target chzdev --no-root-update -pVe c003 layer2=1 bridge_role=primary;

meaning... enable layer2 networking, which creates a bunch of attributes specific to layer2 mode, including the bridge_role. Then set bridge_role. Then online the device.

Testing this out here locally results in race free boots, without any changes to the code.

Could you please try out above command in your preseeds without using packages from proposed?

Regards,

Dimitri.

tags: added: verification-failed
removed: verification-needed
Changed in s390-tools (Ubuntu Xenial):
status: Fix Committed → Incomplete
Changed in s390-tools (Ubuntu Yakkety):
status: Fix Committed → Incomplete
Changed in s390-tools (Ubuntu):
status: Fix Released → Confirmed
Changed in s390-tools (Ubuntu Zesty):
status: Fix Released → Confirmed
Dimitri John Ledkov (xnox) wrote :

artful & zesty will need a revert.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 1.37.0-0ubuntu5

---------------
s390-tools (1.37.0-0ubuntu5) artful; urgency=medium

  * Revert 1.37.0-0ubuntu2 "chzdev/qeth: set bridge_role after the device
    is online", as one must specify layer2 parameter when specifying layer
    specific parameters. LP: #1618463.

 -- Dimitri John Ledkov <email address hidden> Wed, 24 May 2017 17:03:44 +0100

Changed in s390-tools (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers