[UBUNTU 20.04] udev rule change did not get applied

[bugproxy]

During the ubuntu installation in tessia, we do chzdev for both dasd and qeth devices, as below.

2020-08-20 09:54:45 | INFO | START subiquity/Early/run/command_1 : chzdev -e dasd 385c
2020-08-20 09:54:45 | INFO | SUCCESS subiquity/Early/run/command_1 : chzdev -e dasd 385c
2020-08-20 09:54:45 | INFO | START subiquity/Early/run/command_2 : chzdev -e qeth 0.0.bdf0
2020-08-20 09:54:47 | INFO | SUCCESS subiquity/Early/run/command_2 : chzdev -e qeth 0.0.bdf0

and we can see the below files in the /etc/udev/rules.d/
oot@m8360024:~# ls -l /etc/udev/rules.d/
total 76
-rw-r--r-- 1 root root 154 Aug 20 09:08 41-cio-ignore.rules
-rw-r--r-- 1 root root 430 Aug 20 09:08 41-dasd-eckd-0.0.385c.rules
-rw-r--r-- 1 root root 357 Aug 20 09:08 41-generic-ccw-0.0.0009.rules
-rw-r--r-- 1 root root 1049 Aug 20 09:08 41-qeth-0.0.bdf0.rules
-rw-r--r-- 1 root root 58549 Aug 20 09:10 70-snap.snapd.rules

Now, lsinitramfs shows files as below,

root@m8360024:~# lsinitramfs /boot/initrd.img-5.4.0-42-generic | grep 41
etc/udev/rules.d/41-cio-ignore-root.rules
etc/udev/rules.d/41-dasd-eckd-0.0.385c.rules
usr/lib/udev/rules.d/41-cio-ignore.rules
usr/lib/udev/rules.d/41-dasd-eckd-0.0.385c.rules
usr/lib/udev/rules.d/41-generic-ccw-0.0.0009.rules
usr/lib/udev/rules.d/41-qeth-0.0.bdf0.rules

Even though lsinitramfs shows the below files, they are overruled by the filesystem files.

Next thing we did was to modify the 41-qeth-0.0.bdf0.rules and modified the buffer_count to 128 (As in the attached file). In ideal scenario, the value should we modified as mentioned in the bug. But, in our case, if we are not doing a zipl or update-initramfs -u, the value is not getting modified.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Andrew Cloke (andrew-cloke)]

Triaged to the subiquity project, as this issue appears to be related to a subiquity 20.04 installation. Please feel free to reclassify if that's inappropriate.

[Dimitri John Ledkov (xnox)]

> But, in our case, if we are not doing a zipl or update-initramfs -u, the value is not getting modified.

Ubuntu initramfs copies all udev rules that are present in /etc/udev/rules.d into the initramfs. It cannot possibly know which ones are hand-edited, which ones are generated by zdev, and which ones must be in the initramfs or not. For example, we do support bringing up networking interactively inside the initrd, thus even the qeth ones are useful in the initrd.

Separately, we do ship zdev & initramfs integration hook (/lib/s390-tools/zdev-root-update) and thus zdev itself knows how to update the initramfs.

We have discussed this before that `chzdev` persistent enable|disable commands should always invoke zdev-root-update, but that has not been implemented upstream. I too find it confusing that chzdev enable|disable doesn't call zdev-root-update hook (which in turn calls update-initramfs -u to update udev rules in the initrd).

Please open issue against upstream chzdev https://github.com/ibm-s390-tools/s390-tools to call zdev-root-update hook upon any persistent changes to zdev devices.

[Heinz-Werner Seeck (heinz-werner-seeck)]

Test comment: Mirror was not initiated till today.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-09-01 10:18 EDT-------
> Ubuntu initramfs copies all udev rules that are present in /etc/udev/rules.d
> into the initramfs. It cannot possibly know which ones are hand-edited,
> which ones are generated by zdev, and which ones must be in the initramfs or
> not. For example, we do support bringing up networking interactively inside
> the initrd, thus even the qeth ones are useful in the initrd.
>

Basically, this means the custom 'hand-edited' value in the /etc/udev/rules.d/41-qeth-0.0.e200.rules should be executed and should over-rule the value of "buffer_count" attribute, which was set from /usr/lib/udev/rules.d/41*.

But, what we see is, this is not happening. The buffer_count attribute value was always the one which is part of initramfs.

[Frank Heimes (fheimes)]

Currently discussed IBM internally - hence setting project entry to Incomplete ...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-10-26 07:55 EDT-------
Addl Information for a possible solution.

zdev provides an option to specify that a specific device should be configured as part of the initial ram disk configuration steps:

zdev: Implement support for early device configuration
Enable user to specify that a device should be configured early, that is
during the initial RAM-disk boot phase. This may be necessary, e.g. to
override auto-configuration for a device which is also applied during
that boot phase. It may also be used to manually specify devices that
are required to access the root file system, such as networking devices.
Users can mark devices as requiring early configuration by specifying
a value of 1 for the newly added internal attribute zdev:early:
# chzdev dasd-eckd 0.0.1234 -p zdev:early=1
This can be changed back by removing the attribute setting, or by
setting the attribute value to 0:
# chzdev dasd-eckd 0.0.1234 -p -r zdev:early
or

# chzdev dasd-eckd 0.0.1234 -p zdev:early=0

This support was added with s390-tools v2.4.0. It can be used to work around the issue described in this bugzilla.

At the core of this problem lies the fact that unlike other linux vendors, Canonical decided to always want to copy all udev rules to the initial RAM-disk. As this is different between distributions, we cannot just change zdev to always update the RAM-disk as this would potentially introduce problems with other linux vendor distributions.

A potential solution could be done easily by providing a build-time switch that allows Canonical to specify that whenever a persistent device configuration changed, the RAM-disk is updated (this could be implemented via a modified root_check() function in s390-tools/zdev/root.c that only returns if a persistent device configuration was changed).

Could that be a potenital solution? Please provide your feedback. Many thanks in advance

[Frank Heimes (fheimes)]

Using the 'zdev:early' device attribute (and setting it to '1') could be indeed an alternative (at least for the most cases), since with this set one is automatically asked if the initramfs should be rebuild - and in combination with '--yes' this question is automatically confirmed.

I've added here a 2nd disk 0400 and lsinitramfs shows that it's not only listed under
'usr/lib/udev/rules.d', but also under 'etc/udev/rules.d'.

$ sudo chzdev dasd-eckd 0.0.0400 -p zdev:early=1 --yes
Configuring devices in the persistent configuration only
ECKD DASD 0.0.0400 configured
Note: The initial RAM-disk must be updated for these changes to take effect:
       - ECKD DASD 0.0.0400
update-initramfs: Generating /boot/initrd.img-5.4.0-52-generic
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: dasda (0200).
Done.

$ lsinitramfs /boot/initrd.img-5.4.0-52-generic | grep 41
etc/udev/rules.d/41-cio-ignore-root.rules
etc/udev/rules.d/41-dasd-eckd-0.0.0200.rules
etc/udev/rules.d/41-dasd-eckd-0.0.0400.rules <==
usr/lib/udev/rules.d/41-cio-ignore.rules
usr/lib/udev/rules.d/41-dasd-eckd-0.0.0200.rules
usr/lib/udev/rules.d/41-dasd-eckd-0.0.0400.rules <==
usr/lib/udev/rules.d/41-generic-ccw-0.0.0009.rules
usr/lib/udev/rules.d/41-qeth-0.0.0600.rules

However, like Dimitri already pointed out in comment #3, if someone manually modifies udev rules, a manual mkinitramfs is needed, too.

However (moving from the above DASD example to a qeth example) if you specify any further attributes, that you want to get activated, immediately with the chzdev command (like in your case 'buffer_count=128' and probably layer2=1), this will then land in the initramfs as well - and in case 'zdev:early=1' was specified, it will even land in a rule under 'etc/udev/rules.d', too. See:

$ sudo chzdev qeth 0603 -p zdev:early=1 layer2=1 buffer_count=128 --yes
Configuring devices in the persistent configuration only
QETH device 0.0.0603:0.0.0604:0.0.0605 configured
Note: The initial RAM-disk must be updated for these changes to take effect:
       - QETH device 0.0.0603:0.0.0604:0.0.0605
update-initramfs: Generating /boot/initrd.img-5.4.0-52-generic
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: dasda (0200).
Done.
$ cat /etc/udev/rules.d/41-qeth-0.0.0603.rules | egrep '(layer2|buffer_count)'
ATTR{[ccwgroup/0.0.0603]layer2}="1"
ATTR{[ccwgroup/0.0.0603]buffer_count}="128"
$ lsinitramfs -l /boot/initrd.img-5.4.0-52-generic | grep 41-qeth-0.0.0603.rules
-rw-r--r-- 1 root root 1023 Oct 27 09:07 etc/udev/rules.d/41-qeth-0.0.0603.rules
-rw-r--r-- 1 root root 1023 Oct 27 09:07 usr/lib/udev/rules.d/41-qeth-0.0.0603.rules

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-11-10 10:24 EDT-------
The zdev:early parameter in the chzdev command is a work-around for this issue.

The Actual issue is,the fact that unlike SUSE or Red Hat, Canonical always want to copy all udev rules to the initial RAM-disk. As this is different between distributions, we cannot just change zdev to always update the RAM-disk as this would potentially introduce problems with SUSE and Red Hat distributions.

As discussed internally with the team, what could be done is, to provide a build-time switch, a MACRO that allows Canonical to specify that whenever a persistent device configuration changed, the RAM-disk is updated (this could be implemented via a modified root_check() function in s390-tools/zdev/root.c that only returns if a persistent device configuration was changed).
The functionality mentioned above is same as having "zdev:early=1" on this usecase. But, we would like to avoid this extra parameter and make it work like other distros.

We would like to get a go from Canonical on this.

[Dimitri John Ledkov (xnox)]

Hi Vineeth,

that is correct that Debian/Ubuntu always copy all the udev rules from host, to the initramfs by default. And for us, stopping to do that will cause change of behavior which may trigger bug. Same as it would on RHEL/SUSE side if those would start doing this.

If I understand you correctly, an Ubuntu-only change to internal_attr_early to have .defval="1" is the sort of thing you are proposing right? Which should lead to any persistent commands that affect udev rules to trigger initramfs rebuild?

https://github.com/ibm-s390-tools/s390-tools/blob/58ecf1f363c35f452f5d00697620e520029bef83/zdev/src/internal.c#L23

That should fix the originally reported UX issue.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-10 08:39 EDT-------
Unfortunately .defval="1" functionality doesnt work for this case.
Instead, what we do is, we modify the root_check() function in root.c,such a way that, the required functionality of zdev:early=1 is enabled by default, based on a MACRO.

The new MACRO will be build-time configurable, which will then be used by Canonical while building the chzdev.

The patch is being developed and will push to s390-tools upstream once it is ready. Which can then be used by Canonical. So, we dont need a Canonical-only change in the code; But the change is only the MACRO-definition passed during the build-time.

[Dimitri John Ledkov (xnox)]

Sounds good! Looking forward to that patch. And then we will most likely try to pick it up for stable release updates too. Cause yes, this behaviour needs to be fixed properly.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-01-20 09:02 EDT-------
Upstream patch will be pushed soon to the next upcoming s390-tools...
-> in progresss

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-01-25 07:44 EDT-------
The patch mentioned in comment#39 is available in the s390-tools upstream now.

https://github.com/ibm-s390-tools/s390-tools/commit/7dd03eaeecdd0e2674f145aca34be1275d291bd8

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-02-01 09:15 EDT-------
@Can, when can we expect the integration of the fix ?

[Frank Heimes (fheimes)]

The idea is to pick this up once 2.16 is out (LP#1892367), since with 2.16 the udev topic is solved for hirsute, hence just the SRU to G and F is remaining.

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools - 2.16.0-0ubuntu1

---------------
s390-tools (2.16.0-0ubuntu1) hirsute; urgency=medium

  * New upstream release. LP: #1914574
  * Drop patches applied upstream.
  * Compile with ZDEV_ALWAYS_UPDATE_INITRD=1 option. This fixes calls to
    chzdev to trigger update-initramfs -u. LP: #1892367
  * Set install path via make variables, instead of patching source
    Makefiles.
  * Update homepage to github.com.

s390-tools (2.15.1-1ubuntu3) hirsute; urgency=medium

  * No-change rebuild to drop the udeb package.

s390-tools (2.15.1-1ubuntu2) hirsute; urgency=medium

  * No change rebuild with fixed ownership.

s390-tools (2.15.1-1ubuntu1) hirsute; urgency=low

  * Update debian/watch for new ibm-s390-linux repo
  * Update debian/control for [XC-]Package-Type
  * Merge from Debian unstable. Remaining changes:
    - keep providing zkey
    - split some tools into individuall packages
    - Drop udevadm-path.patch
    - Drop ziomon package
    - Linux on z Systems naming
    - 0001-ziomon-Use-exit-code-0-for-version-and-help.patch
    - 0001-zkey-add-initramfs-hook.patch
    - 0001-zkey-on-Ubuntu-use-default-benchmarked-Argon2i-with-.patch
    - 0001-dumpconf-Don-t-run-the-service-in-LXC.patch
    - update-install-paths.patch
    - 0010-no-pie-is-not-a-valid-option-for-ld.patch
    - s390-tools-lp1903984-hirsute.patch

s390-tools (2.15.1-2) unstable; urgency=medium

  * Remove myself as a maintainer.

s390-tools (2.15.1-1) unstable; urgency=medium

  * New upstream release
    - Install new tool lsstp.
    - Continue to skip out on zkey for now.

 -- Dimitri John Ledkov <email address hidden> Wed, 24 Feb 2021 05:49:54 +0000