[UBUNTU 20.04] chzdev -e is rebuilding initramfs even with zdev:early=0 set

[bugproxy]

Versions:
Ubuntu 20.04.5 s390-tools version 2.12.0-0ubuntu3.7.s390x
Ubuntu 22.04.2 s390-tools version 2.20.0-0ubuntu3.2.s390x

When I configure a zfcp LUN persistently via chzdev, the initrd is being rebuilt even with parameter zdev:early=0

root@a8315003:~# chzdev -e zfcp-lun 0.0.1803:0x500507630910d430:0x4019409200000000 zdev:early=0
zFCP LUN 0.0.1803:0x500507630910d430:0x4019409200000000 configured
Note: The initial RAM-disk must be updated for these changes to take effect:
       - zFCP LUN 0.0.1803:0x500507630910d430:0x4019409200000000
update-initramfs: Generating /boot/initrd.img-5.15.0-60-generic
I: The initramfs will attempt to resume from /dev/dasdb1
I: (UUID=e70ecb80-4d1e-4074-9cda-ce231ad6e698)
I: Set the RESUME variable to override this.
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: dasda (c00a).
Done.
root@a8315003:~#

== Comment: - Thorsten Diehl <email address hidden> - 2023-03-01 06:55:47 ==
@BOE-dev
This behaviour is unexpected.
https://www.ibm.com/docs/en/linux-on-systems?topic=commands-chzdev says:
Activating a device early during the boot process

Use the zdev:early device attribute to activate a device early during the boot process and to override any existing auto-configuration with a persistent device configuration.

zdev:early=1
    The device is activated during the initial RAM disc phase according to the persistent configuration.

zdev:early=0
    The device is activated as usual during the boot process. This is the default. If auto-configuration data is present, the device is activated during the initial RAM disc phase according to the auto-configuration.

I can't interprete a SCSI LUN as a device with auto configuration data. (At least, if the zfcp device hasn't NPIV enabled)

== Comment: #5 - Peter Oberparleiter <email address hidden> - 2023-03-01 11:18:28 ==
(In reply to comment #2)
> @BOE-dev
> This behaviour is unexpected.
> https://www.ibm.com/docs/en/linux-on-systems?topic=commands-chzdev says:
> Activating a device early during the boot process
>
> Use the zdev:early device attribute to activate a device early during the
> boot process and to override any existing auto-configuration with a
> persistent device configuration.
>
> zdev:early=1
> The device is activated during the initial RAM disc phase according to
> the persistent configuration.
>
> zdev:early=0
> The device is activated as usual during the boot process. This is the
> default. If auto-configuration data is present, the device is activated
> during the initial RAM disc phase according to the auto-configuration.

The documentation is incorrect for Ubuntu. Canonical specifically builds zdev in a way that every change to persistent device configuration causes an update to the initial RAM-disk. See also:

https://bugzilla.linux.ibm.com/show_bug.cgi?id=187578#c35
https://github.com/ibm-s390-linux/s390-tools/commit/7dd03eaeecdd0e2674f145aca34be1275d291bd8

> I can't interprete a SCSI LUN as a device with auto configuration data. (At
> least, if the zfcp device hasn't NPIV enabled)

This is related to auto-configuration as implemented for DPM.

== Comment: #6 - Thorsten Diehl <email address hidden> - 2023-03-03 12:41:44 ==
So, IIUC, chzdev is built for Ubuntu with ZDEV_ALWAYS_UPDATE_INITRD=1, which make the parameter zdev:early=0 ineffective. Correct?
If you confirm, you may also close this bug.

Not nice - then we have to find an alternate solution.

== Comment: #7 - Peter Oberparleiter <email address hidden> - 2023-03-07 06:48:07 ==
(In reply to comment #6)
> So, IIUC, chzdev is built for Ubuntu with ZDEV_ALWAYS_UPDATE_INITRD=1, which
> make the parameter zdev:early=0 ineffective. Correct?
> If you confirm, you may also close this bug.
>
> Not nice - then we have to find an alternate solution.

chzdev -p on Ubuntu will by default rebuild the initrd. This is intended
behavior by Canonical and controlled by the ZDEV_ALWAYS_UPDATE_INITRD build-time
switch. You can suppress it by adding option --no-root-update to the command
line.

Specifying zdev:early=0 to chzdev has exactly the effect that it is supposed to
have: it tells zdev not to enable that device during initrd processing,
resulting in the corresponding udev rule not being copied to the initrd [1].

Unfortunately there is another Ubuntu-initrd script [2] that simply copies ALL
udev rules, including those created by zdev, into the initrd. As a result,
zdev's early-attribute handling is rendered useless and all devices are enabled,
even if a user specified zdev:early=0.

Since this bug report indicates that there is a use-case for this function in
Ubuntu, it might be worth asking Canonical if current processing could be
changed to provide a way for users to specify that a device should specifically
NOT be enabled within initrd processing.

Technically this could easily be done:

1) Have the generic udev initramfs script not copy zdev-generated Udev rules,
   OR
   have the zdev initramfs script remove those rules (somewhat of a hack)

2) Change the zdev initramfs script logic from the current:

   - enable devices required for the root file system, AND
   - enable devices for which zdev:early=1 was specified

   to

   - enable all persistently configured devices EXCEPT those for which
     zdev:early=0 was specified

   This change would be needed to maintain Canonical's policy of enabling
   all devices in the initrd by default

I'm open to adding the change in 2) to our s390-tools package, but someone at
Canonical would need to work out a way to implement 1).

[1] https://github.com/ibm-s390-linux/s390-tools/blob/master/zdev/initramfs/hooks/s390-tools-zdev#L47
[2] https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/tree/debian/extra/initramfs-tools/hooks/udev#n42

[Frank Heimes (fheimes)]

This was implemented around 2020, mainly to catch any ccw config changes in a user friendly way:
https://bugs.launchpad.net/bugs/1892367
https://i527439087.restricted.launchpadlibrarian.net/527439087/20f6c87a-829f-11eb-9412-002481e91f22.txt?token=zC4n7TlCmffBDHm9Dl002PsfchDXC3Dk

Regarding:
"1) Have the generic udev initramfs script not copy zdev-generated Udev rules,"
we need to be super careful here to not break stuff ...

How to best identify the zdev-generated Udev rules,
since these are not all the rules generated by chzdev (indicated by first line in rule), are they?

I'm not very sure if initramfs(-tools) maintainer(s) are very happy ...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-11-22 09:17 EDT-------
(In reply to comment #19)
> This was implemented around 2020, mainly to catch any ccw config changes in
> a user friendly way:
> https://bugs.launchpad.net/bugs/1892367
> https://i527439087.restricted.launchpadlibrarian.net/527439087/20f6c87a-829f-
> 11eb-9412-002481e91f22.txt?token=zC4n7TlCmffBDHm9Dl002PsfchDXC3Dk
>
> Regarding:
> "1) Have the generic udev initramfs script not copy zdev-generated Udev
> rules,"
> we need to be super careful here to not break stuff ...
>
> How to best identify the zdev-generated Udev rules,
> since these are not all the rules generated by chzdev (indicated by first
> line in rule), are they?

chzdev creates udev rules that can most easily be recognized via their name:
- 41-<device-type>-<device-id>.rules
- 41-cio-ignore.rules
- 40-zdev-id.rules

So a simple filter would be to skip rules named 41-*.rules on s390x in the initramfs udev hook.

As an alternative, the s390-tools zdev initramfs hook could try to delete specifically those rules with zdev:early=0 that the upstream udev hook created. While this should work correctly, it is also a kind of a workaround, where one hook adds files that another hook removes again...

If this approach is more likely to be acceptable, it could be achieved by adding something like the following to /usr/share/initramfs-tools/hooks/s390-tools-zdev:

# Remove zdev rules not intended for early boot that were copied by udev hook
chzdev --disable --persistent --by-attr zdev:early=0 \
--base "/etc/udev/rules.d/=$DESTDIR/lib/udev/rules.d/" \
--yes --quiet --no-root-update --force >/dev/null 2>&1

[Simon Chopin (schopin)]

Tagging this to discuss this during the Foundations weekly meeting

[Benjamin Drung (bdrung)]

With my initramfs-tools maintainer hat on, I suggest/support following solution:

Modify /usr/share/initramfs-tools/hooks/udev (shipped by udev, source: systemd) to not copy the udev rules generated by chzdev. So changing:

```
for rules in /etc/udev/rules.d/*.rules; do
  if [ -e "$rules" ] && [ ! -e "/lib/${rules#/etc/}" ]; then
    cp -p "$rules" "$DESTDIR/lib/udev/rules.d/"
  fi
done
```

to something like:

```
for rules in /etc/udev/rules.d/*.rules; do
  if [ -e /sbin/chzdev ] && /sbin/chzdev --is-author-of-udev-rule "$rules"; then
    continue
  fi
  if [ -e "$rules" ] && [ ! -e "/lib/${rules#/etc/}" ]; then
    cp -p "$rules" "$DESTDIR/lib/udev/rules.d/"
  fi
done
```

Then `/sbin/chzdev --is-author-of-udev-rule "$rules"` should produce the correct exit code. The tool that generates the udev rules should be queried (is that chzdev or something else?) or a separate helper should be used.

[Benjamin Drung (bdrung)]

Re-assinging from initramfs-tools to systemd, because /usr/share/initramfs-tools/hooks/udev is shipped by udev.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-01-31 04:28 EDT-------
(In reply to comment #22)
> Then `/sbin/chzdev --is-author-of-udev-rule "$rules"` should produce the
> correct exit code. The tool that generates the udev rules should be queried
> (is that chzdev or something else?) or a separate helper should be used.
>
> Re-assinging from initramfs-tools to systemd, because
> /usr/share/initramfs-tools/hooks/udev is shipped by udev.

I agree with this approach.

Yes, the udev rules are generated by chzdev, so we'll work on adding a chzdev command line option that can be used to query ownership of a udev rule/configuration file similar to what is as outlined above.

I'll update this bug report once the option as available in upstream s390-tools.

[Frank Heimes (fheimes)]

Hi Peter,
is this now incl. in s390-tools v2.33.0
https://github.com/ibm-s390-linux/s390-tools/releases/tag/v2.33.0
and what I read as:
"chzdev: Add --is-owner to identify files created by zdev"
?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-05-28 04:26 EDT-------
(In reply to comment #26)
> Hi Peter,
> is this now incl. in s390-tools v2.33.0
> https://github.com/ibm-s390-linux/s390-tools/releases/tag/v2.33.0
> and what I read as:
> "chzdev: Add --is-owner to identify files created by zdev"
> ?

Peter is currently on vacation, but I can confirm that this feature/commit is now
included as part of the 2.33.0 release.

[Frank Heimes (fheimes)]

Okay, thanks Vineeth for jumping in!

In other words this will become unblocked once we have s390-tools >= 2.33.0 in, which I am currently working on (v2.33.1 - based on LP#2067355).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-05-28 05:00 EDT-------
(In reply to comment #28)
> Okay, thanks Vineeth for jumping in!
>
> In other words this will become unblocked once we have s390-tools >= 2.33.0
> in, which I am currently working on (v2.33.1 - based on LP#2067355).

Yes. That's right. Thank you.

[Steve Langasek (vorlon)]

oracular has s390-tools 2.33.1-0ubuntu3. Is there any further change needed on that package, or should that task be marked resolved?

[Frank Heimes (fheimes)]

@vorlon yes, that's correct. (This bug became a bit abandoned ...)
I just changed the target release to oracular and set s390-tools to Fix Released.

[Frank Heimes (fheimes)]

Where are the initramfs-tools gone? Added it again ... (sorry)

[Benjamin Drung (bdrung)]

See comment #4 and #5: /usr/share/initramfs-tools/hooks/udev is shipped by the udev package (not by initramfs-tools)

[bugproxy (bugproxy)]

Modify /usr/share/initramfs-tools/hooks/udev (shipped by udev, source: system) to not copy the udev rules generated by chzdev. So changing:

Where are the initramfs-tools gone? Added them again ...

[Frank Heimes (fheimes)]

I see Benjamin - thought that I had accidentally removed it by updating the bug (since it was still there in my old browser tab), removed it again.

(Btw. looks like comment #10 is an iisue with the bugzilla-launchpad bridge ...)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-09-05 04:23 EDT-------
The problem seems fixed meanwhile for Oracular (24.10) see here https://launchpad.net/ubuntu/+source/s390-tools
Are you thinking of backports for the LTS releases 22.04 and 20.04 ?

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 256.5-2ubuntu2

---------------
systemd (256.5-2ubuntu2) oracular; urgency=medium

  * initramfs-tools: ensure rules file exists before invoking chzdev (LP: #2079993)

systemd (256.5-2ubuntu1) oracular; urgency=medium

  * Merge with Debian unstable. Remaining changes:
    - debian/tests/tests-in-lxd: run some autopkgtests in LXD too
    - debian/tests/boot-and-services:
      + skip apparmor tests on armhf
      + consume stderr in systemctl status call in test_service
      + drop test_no_failed
    - debian/systemd.postinst:
      + skip daemon-reexec and try-restarts during shutdown
      + manually call systemd-tmpfiles --create in postinst
    - debian/systemd-resolved.postinst: copy existing /etc/resolv.conf to
      /run/systemd/resolve/stub-resolv.conf
    - debian/rules:
      + Remove unneeded efi artifacts on i386 to avoid debugedit errors
    - debian/rules,debian/control,debian/tests/control:
      + Do not build with tpm libraries on i386
      + Do not build with libqrencode on i386
    - debian/gbp.conf,debian/extra/wrap_cl.py:
      Use a customization script to add LP commit links to changelog
    - debian/control:
      + Add Recommends: networkd-dispatcher systemd-resolved to systemd package
      + Give systemd-resolved Priority: important
      + Add Recommends: systemd-hwe-hwdb to udev package
      + Add Breaks: systemd (<< ${binary:Version}) to udev package so that
        systemd is upgraded as well when upgrading udev
      + Make systemd-sysv Depends: on matching version of systemd
      + Drop Recommends: libnss-myhostname libnss-resolve from systemd-resolved
      + Build-Depends: linux-tools-generic
    - debian/tests/upstream: export QEMU_MEM="1024M" for all tests
    - debian/systemd.links: mask systemd-gpt-auto-generator by default
    - debian/systemd.install: exclude files that are not built for i386
    - debian/systemd.manpages: do not ship un-built manpages on i386
    - debian/tests/control: only install systemd-boot-efi for supported arches
    - switch-root: use MS_MOVE for /run when switchig from initrd
    - debian/systemd.postinst: do not create /etc/tmpfiles.d/tmp.conf on upgrades.
      We want the upgrades on Ubuntu to be aligned with what a new install
      would look like.
    - d/control: do not build systemd-boot-efi-{amd64,arm64}-signed-template
  * New changes:
    - Keep utmp support for this release, since we are passed Feature Freeze
      in Ubuntu.
    - Filter out zdev rules in the initramfs hook (LP: #2044104)
    - d/t/upstream: honor /etc/apt configured by autopkgtest

systemd (256.5-2) unstable; urgency=medium

  [ Helmut Grohne ]
  * Fix stage1 build (Closes: #1078821)

  [ Luca Boccassi ]
  * Disable utmp support, replaced by wtmpdb. utmp is not y2038-safe, util-
    linux has now turned it off and relies on logind, so disable utmp
    support in logind too, as it is no longer necessary. wtmpdb replaces
    the functionality.

systemd (256.5-1) unstable; urgency=medium

  * New upstream version 256.5
  * Drop patch merged upstream
  * autopkgtest: skip TEST-64-UDEV-STORAGE due to qemu crash. This tests
    randomly causes qemu to cr...