udev race condition with qeth device and bridge_role

[Andrew McLeod]

[Impact]

 * bridge_role property setting is racy on boot

 * This results in incorrect bridge mode set on the devices, sometimes, which leads to lack of desired connectivity (e.g. bridging internet to containers)

 * The fix for this issue is to set bridge_role, only after the device is online

 * Unfortunately the udev rules are not regenerated, therefore affected systemd must manually remove and recreate chzdev rules

[Test Case]

 * Remove qeth udev rules from /etc/udev/rules.d/
 * Enable qeth device using chzdev with a non-default bridge_role setting, e.g.:
   chzdev --no-root-update -pVe c003 bridge_role=primary;
 * Reboot and check that bridge_role setting is correctly set in the sysfs, e.g.:
   /sys/devices/qeth/0.0.c003/bridge_role

[Regression Potential]

 * Minimal, the generated udev rules remain the same; the only difference in the generated udev rules is the ordering in setting the bridge_role attribute

[Other Info]

 * Original bug report:

Attempting to set bridge_role = primary with the following command in preseed:

in-target chzdev --no-root-update -pVe c003 bridge_role=primary;

...works, and generates the following udev rule for this device:

https://pastebin.canonical.com/164271/

However, after reboot:

systemd-udevd[2634]: error opening ATTR{/sys/devices/qeth/0.0.c003/bridge_role} for writing: Permission denied

More logging:

https://pastebin.canonical.com/164272/

after the system has booted, we are able to write to the file and set bridge_role to primary:

root@10-13-3-10:/var/log# cat /sys/devices/qeth/0.0.c003/bridge_role
none
root@10-13-3-10:/var/log# echo primary > /sys/devices/qeth/0.0.c003/bridge_role
root@10-13-3-10:/var/log# cat /sys/devices/qeth/0.0.c003/bridge_role
primary

[Dimitri John Ledkov (xnox)]

sudo add-apt-repository ppa:ci-train-ppa-service/2038
sudo apt update
sudo apt dist-upgrade

After that please regenerate persistent configuration for your qeth device, e.g.:

$ sudo chzdev -p -d 600
$ sudo chzdev -p -e 600 bridge_role=primary
$ sudo update-initramfs -u

undo any other manual changes to set bridge_role to primary, and finally reboot to test the packages with a proposed fix for your issue.

Please note packages are pending publication, however should be available shortly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools - 1.37.0-0ubuntu2

---------------
s390-tools (1.37.0-0ubuntu2) zesty; urgency=medium

  * chzdev/qeth: set bridge_role after the device is online to avoid a
    race condition when setting qeth attributes on boot. LP: #1618463.

 -- Dimitri John Ledkov <email address hidden> Tue, 14 Mar 2017 09:26:12 +0000

[Brian Murray (brian-murray)]

Hello Andrew, or anyone else affected,

Accepted s390-tools into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/1.36.1-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Andrew, or anyone else affected,

Accepted s390-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/1.34.0-0ubuntu8.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andrew McLeod (admcleod)]

Hi Brian,

I have tested "s390-tools 1.34.0-0ubuntu8.3" in Xenial and can confirm it fixes the bug in Xenial. I am not currently able to test in Yakkety or Zesty

Thanks!

[Dimitri John Ledkov (xnox)]

@admcleod

Based on feedback from IBM i'm going to withdraw this update. The ordering of the flags is correct, as long as one activates the flag needed to have bridge_role attribute available before onlining the device.

Specifically one should use:

in-target chzdev --no-root-update -pVe c003 layer2=1 bridge_role=primary;

meaning... enable layer2 networking, which creates a bunch of attributes specific to layer2 mode, including the bridge_role. Then set bridge_role. Then online the device.

Testing this out here locally results in race free boots, without any changes to the code.

Could you please try out above command in your preseeds without using packages from proposed?

Regards,

Dimitri.

[Dimitri John Ledkov (xnox)]

artful & zesty will need a revert.

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools - 1.37.0-0ubuntu5

---------------
s390-tools (1.37.0-0ubuntu5) artful; urgency=medium

  * Revert 1.37.0-0ubuntu2 "chzdev/qeth: set bridge_role after the device
    is online", as one must specify layer2 parameter when specifying layer
    specific parameters. LP: #1618463.

 -- Dimitri John Ledkov <email address hidden> Wed, 24 May 2017 17:03:44 +0100