[UBUNTU 20.04] chzdev -e is rebuilding initramfs even with zdev:early=0 set

[bugproxy]

Versions:
Ubuntu 20.04.5 s390-tools version 2.12.0-0ubuntu3.7.s390x
Ubuntu 22.04.2 s390-tools version 2.20.0-0ubuntu3.2.s390x

When I configure a zfcp LUN persistently via chzdev, the initrd is being rebuilt even with parameter zdev:early=0

root@a8315003:~# chzdev -e zfcp-lun 0.0.1803:0x500507630910d430:0x4019409200000000 zdev:early=0
zFCP LUN 0.0.1803:0x500507630910d430:0x4019409200000000 configured
Note: The initial RAM-disk must be updated for these changes to take effect:
       - zFCP LUN 0.0.1803:0x500507630910d430:0x4019409200000000
update-initramfs: Generating /boot/initrd.img-5.15.0-60-generic
I: The initramfs will attempt to resume from /dev/dasdb1
I: (UUID=e70ecb80-4d1e-4074-9cda-ce231ad6e698)
I: Set the RESUME variable to override this.
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: dasda (c00a).
Done.
root@a8315003:~#

== Comment: - Thorsten Diehl <email address hidden> - 2023-03-01 06:55:47 ==
@BOE-dev
This behaviour is unexpected.
https://www.ibm.com/docs/en/linux-on-systems?topic=commands-chzdev says:
Activating a device early during the boot process

Use the zdev:early device attribute to activate a device early during the boot process and to override any existing auto-configuration with a persistent device configuration.

zdev:early=1
    The device is activated during the initial RAM disc phase according to the persistent configuration.

zdev:early=0
    The device is activated as usual during the boot process. This is the default. If auto-configuration data is present, the device is activated during the initial RAM disc phase according to the auto-configuration.

I can't interprete a SCSI LUN as a device with auto configuration data. (At least, if the zfcp device hasn't NPIV enabled)

== Comment: #5 - Peter Oberparleiter <email address hidden> - 2023-03-01 11:18:28 ==
(In reply to comment #2)
> @BOE-dev
> This behaviour is unexpected.
> https://www.ibm.com/docs/en/linux-on-systems?topic=commands-chzdev says:
> Activating a device early during the boot process
>
> Use the zdev:early device attribute to activate a device early during the
> boot process and to override any existing auto-configuration with a
> persistent device configuration.
>
> zdev:early=1
> The device is activated during the initial RAM disc phase according to
> the persistent configuration.
>
> zdev:early=0
> The device is activated as usual during the boot process. This is the
> default. If auto-configuration data is present, the device is activated
> during the initial RAM disc phase according to the auto-configuration.

The documentation is incorrect for Ubuntu. Canonical specifically builds zdev in a way that every change to persistent device configuration causes an update to the initial RAM-disk. See also:

https://bugzilla.linux.ibm.com/show_bug.cgi?id=187578#c35
https://github.com/ibm-s390-linux/s390-tools/commit/7dd03eaeecdd0e2674f145aca34be1275d291bd8

> I can't interprete a SCSI LUN as a device with auto configuration data. (At
> least, if the zfcp device hasn't NPIV enabled)

This is related to auto-configuration as implemented for DPM.

== Comment: #6 - Thorsten Diehl <email address hidden> - 2023-03-03 12:41:44 ==
So, IIUC, chzdev is built for Ubuntu with ZDEV_ALWAYS_UPDATE_INITRD=1, which make the parameter zdev:early=0 ineffective. Correct?
If you confirm, you may also close this bug.

Not nice - then we have to find an alternate solution.

== Comment: #7 - Peter Oberparleiter <email address hidden> - 2023-03-07 06:48:07 ==
(In reply to comment #6)
> So, IIUC, chzdev is built for Ubuntu with ZDEV_ALWAYS_UPDATE_INITRD=1, which
> make the parameter zdev:early=0 ineffective. Correct?
> If you confirm, you may also close this bug.
>
> Not nice - then we have to find an alternate solution.

chzdev -p on Ubuntu will by default rebuild the initrd. This is intended
behavior by Canonical and controlled by the ZDEV_ALWAYS_UPDATE_INITRD build-time
switch. You can suppress it by adding option --no-root-update to the command
line.

Specifying zdev:early=0 to chzdev has exactly the effect that it is supposed to
have: it tells zdev not to enable that device during initrd processing,
resulting in the corresponding udev rule not being copied to the initrd [1].

Unfortunately there is another Ubuntu-initrd script [2] that simply copies ALL
udev rules, including those created by zdev, into the initrd. As a result,
zdev's early-attribute handling is rendered useless and all devices are enabled,
even if a user specified zdev:early=0.

Since this bug report indicates that there is a use-case for this function in
Ubuntu, it might be worth asking Canonical if current processing could be
changed to provide a way for users to specify that a device should specifically
NOT be enabled within initrd processing.

Technically this could easily be done:

1) Have the generic udev initramfs script not copy zdev-generated Udev rules,
   OR
   have the zdev initramfs script remove those rules (somewhat of a hack)

2) Change the zdev initramfs script logic from the current:

   - enable devices required for the root file system, AND
   - enable devices for which zdev:early=1 was specified

   to

   - enable all persistently configured devices EXCEPT those for which
     zdev:early=0 was specified

   This change would be needed to maintain Canonical's policy of enabling
   all devices in the initrd by default

I'm open to adding the change in 2) to our s390-tools package, but someone at
Canonical would need to work out a way to implement 1).

[1] https://github.com/ibm-s390-linux/s390-tools/blob/master/zdev/initramfs/hooks/s390-tools-zdev#L47
[2] https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/tree/debian/extra/initramfs-tools/hooks/udev#n42

[Frank Heimes (fheimes)]

This was implemented around 2020, mainly to catch any ccw config changes in a user friendly way:
https://bugs.launchpad.net/bugs/1892367
https://i527439087.restricted.launchpadlibrarian.net/527439087/20f6c87a-829f-11eb-9412-002481e91f22.txt?token=zC4n7TlCmffBDHm9Dl002PsfchDXC3Dk

Regarding:
"1) Have the generic udev initramfs script not copy zdev-generated Udev rules,"
we need to be super careful here to not break stuff ...

How to best identify the zdev-generated Udev rules,
since these are not all the rules generated by chzdev (indicated by first line in rule), are they?

I'm not very sure if initramfs(-tools) maintainer(s) are very happy ...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-11-22 09:17 EDT-------
(In reply to comment #19)
> This was implemented around 2020, mainly to catch any ccw config changes in
> a user friendly way:
> https://bugs.launchpad.net/bugs/1892367
> https://i527439087.restricted.launchpadlibrarian.net/527439087/20f6c87a-829f-
> 11eb-9412-002481e91f22.txt?token=zC4n7TlCmffBDHm9Dl002PsfchDXC3Dk
>
> Regarding:
> "1) Have the generic udev initramfs script not copy zdev-generated Udev
> rules,"
> we need to be super careful here to not break stuff ...
>
> How to best identify the zdev-generated Udev rules,
> since these are not all the rules generated by chzdev (indicated by first
> line in rule), are they?

chzdev creates udev rules that can most easily be recognized via their name:
- 41-<device-type>-<device-id>.rules
- 41-cio-ignore.rules
- 40-zdev-id.rules

So a simple filter would be to skip rules named 41-*.rules on s390x in the initramfs udev hook.

As an alternative, the s390-tools zdev initramfs hook could try to delete specifically those rules with zdev:early=0 that the upstream udev hook created. While this should work correctly, it is also a kind of a workaround, where one hook adds files that another hook removes again...

If this approach is more likely to be acceptable, it could be achieved by adding something like the following to /usr/share/initramfs-tools/hooks/s390-tools-zdev:

# Remove zdev rules not intended for early boot that were copied by udev hook
chzdev --disable --persistent --by-attr zdev:early=0 \
--base "/etc/udev/rules.d/=$DESTDIR/lib/udev/rules.d/" \
--yes --quiet --no-root-update --force >/dev/null 2>&1

[Simon Chopin (schopin)]

Tagging this to discuss this during the Foundations weekly meeting

[Benjamin Drung (bdrung)]

With my initramfs-tools maintainer hat on, I suggest/support following solution:

Modify /usr/share/initramfs-tools/hooks/udev (shipped by udev, source: system) to not copy the udev rules generated by chzdev. So changing:

```
for rules in /etc/udev/rules.d/*.rules; do
  if [ -e "$rules" ] && [ ! -e "/lib/${rules#/etc/}" ]; then
    cp -p "$rules" "$DESTDIR/lib/udev/rules.d/"
  fi
done
```

to something like:

```
for rules in /etc/udev/rules.d/*.rules; do
  if [ -e /sbin/chzdev ] && /sbin/chzdev --is-author-of-udev-rule "$rules"; then
    continue
  fi
  if [ -e "$rules" ] && [ ! -e "/lib/${rules#/etc/}" ]; then
    cp -p "$rules" "$DESTDIR/lib/udev/rules.d/"
  fi
done
```

Then `/sbin/chzdev --is-author-of-udev-rule "$rules"` should produce the correct exit code. The tool that generates the udev rules should be queried (is that chzdev or something else?) or a separate helper should be used.

[Benjamin Drung (bdrung)]

Re-assinging from initramfs-tools to systemd, because /usr/share/initramfs-tools/hooks/udev is shipped by udev.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-01-31 04:28 EDT-------
(In reply to comment #22)
> Then `/sbin/chzdev --is-author-of-udev-rule "$rules"` should produce the
> correct exit code. The tool that generates the udev rules should be queried
> (is that chzdev or something else?) or a separate helper should be used.
>
> Re-assinging from initramfs-tools to systemd, because
> /usr/share/initramfs-tools/hooks/udev is shipped by udev.

I agree with this approach.

Yes, the udev rules are generated by chzdev, so we'll work on adding a chzdev command line option that can be used to query ownership of a udev rule/configuration file similar to what is as outlined above.

I'll update this bug report once the option as available in upstream s390-tools.