[UBUNTU 20.04] chzdev -e is rebuilding initramfs even with zdev:early=0 set

[bugproxy]

SRU Justification:

[ Impact ]

 * The CCW (or zdev) devices that are special for s390x always need
   to be explicitly enabled before they can be used.
   And this is usually done with the help of the 'chzdev -e' command
   (part of the s390-tools), that also creates underlying udev rules
   for the device activation.

 * When for example a qeth network device is persistently configured by
   'chzdev -e' the initramfs is usually rebuild, since the corresponding
   udev rule might be needed in initramfs (early at boot time).

 * But chzdev also has the parameter 'zdev:early' that allows to explicity
   direct if the initramfs should be rebuild and the udev rule integrated
   (zdev:early=1) or not (zdev:early=0).

 * Right now the initramfs is erroneously rebuild every time and
   includes all (zdev) udev rules, just ignoring the zdev:early parameter.

 * This can have a significant impact especially on systems with hundreds
   (or even thousands) of devices and can lead to space constraints.
   (Note that also larger ranges of devices can easily be enables with one cmd.)

 * For example supplemental devices (like disks), that are not relevant
   at early boot time (and for example may only be used in backup or take-over
   cases) must not be always activated at boot time.

 * On system in DPM mode this can also be (to a certain degree) controlled
   by the HMC (but only in DPM mode).

 * To fix this situation, two things are needed:
   - s390-tools: have an option in chzdev that allows to identify if an udev rule is
     zdev related or not.
     (since a udev rule could also be generic, and not specific to s390x).
   - systemd: handle zdev related rules in extra/initramfs-tools/hooks/udev
     properly according to the zdev:early parameter or use a proper default

[ Test Plan ]

 * Have an s390x LPAR or z/VM system with several ccw/zdev devices.
   Some might be in use (e.g. for the underlying disk of the main
   network device), but some spares to test with are needed.
   Easiest is to use qeth network devices.

 * List the available devices:
   $ lszdev | grep qeth | head -3
   qeth 0.0.c000:0.0.c001:0.0.c002 yes yes encc000
   qeth 0.0.c003:0.0.c004:0.0.c005 no no
   qeth 0.0.c006:0.0.c007:0.0.c008 no no
   Notice that using only a short form of the device triples is sufficient.
   Here c000 is already active, but c003 and c006 are not.

 * Check which qeth devices are in the current initramfs:
   lsinitramfs /boot/initrd.img-$(uname -r) | grep usr/lib/udev/rules.d/41-qeth-0.0.c00
   usr/lib/udev/rules.d/41-qeth-0.0.c000.rules
   Like expected, only c000 is listed.

 * Now add a second device and explicity direct to not include it
   into the initramfs (using parameter 'zdev:early=0'):
   $ sudo chzdev -e qeth 0.0.c003 zdev:early=0
   and check what's in the initramfs:
   $ lsinitramfs /boot/initrd.img-$(uname -r) | grep usr/lib/udev/rules.d/41-qeth-0.0.c00
   usr/lib/udev/rules.d/41-qeth-0.0.c000.rules
   Still c000 only.

 * Now add another device, but this time explicitly directing
   to incl. the corresponding udev rule into initramfs:
   $ sudo chzdev -e qeth 0.0.c006 zdev:early=1
   and check again the content of the initramfs:
   $ lsinitramfs /boot/initrd.img-(uname -r) | grep usr/lib/udev/rules.d/41-qeth-0.0.c00
   usr/lib/udev/rules.d/41-qeth-0.0.c000.rules
   usr/lib/udev/rules.d/41-qeth-0.0.c006.rules
   Now both are included.

 * More for regression testing disable/remove the devices again
   ('zdev:early' parameter is irrelevant in this case):
      sudo chzdev -d qeth c003
      sude chzdev -d qeth c006
      check:
      $ lsinitramfs /boot/initrd.img-(uname -r) | grep -i usr/lib/udev/rules.d/41-qeth-0.0.c00
      usr/lib/udev/rules.d/41-qeth-0.0.c000.rules

 * Add a device without parameter 'zdev:early' specified at all,
   which needs to default to 'zdev:early=1':
   sudo chzdev -e qeth c003
   and check:
   $ lsinitramfs /boot/initrd.img-(uname -r) | grep -i usr/lib/udev/rules.d/41-qeth-0.0.c00
   usr/lib/udev/rules.d/41-qeth-0.0.c000.rules
   usr/lib/udev/rules.d/41-qeth-0.0.c003.rules

 * The primary use case for the new chzdev option '--is-owner'
   was for the scripted udev rule handling, however,
   the option can also be more directly tested, but the result
   needs to be checked based on the the given return code:

   - this is a standard udev rule of a ccw qeth network device
     (zdev) and is with that always created by chzdev:
     $ ls /etc/udev/rules.d/41-qeth-0.0.0600.rules
     /etc/udev/rules.d/41-qeth-0.0.0600.rules
   - hence 'chzdev --is-owner' succeeds and returns exit code '0'
     (EXIT_OK in exit_code.h):
     $ chzdev --is-owner /etc/udev/rules.d/41-qeth-0.0.0600.rules
     $ echo $?
     0
   - however, the udev rule for snapd was obviously not added by chzdev:,
     $ ls /etc/udev/rules.d/70-snap.snapd.rules
     /etc/udev/rules.d/70-snap.snapd.rules
   - hence the return code here is the expected '33'
     (the newly introduced exit code 'EXIT_UNKNOWN_FILE' in exit_code.h)
     $ chzdev --is-owner /etc/udev/rules.d/70-snap.snapd.rules
     $ echo $?
     33

 * chzdev is especially used at install time, hence another test
   would be to start an installation, and at the initial subiquity
   screen immediately navigate the to installer shell and update
   the s390-tools to the updated version, leave the installer shell
   and proceed with the installation.
   The installation will then run through the usual zDev activation
   screen (using the updated s390-tools), which makes use of chzdev.

[ Where problems could occur ]

 * The modification in the s390-tools are to expand the chzdev command with
   the option '--is-owner <rule>' that allows to identify a zdev rule.

 * Since it's added (no existing code line was removed or modified) the impact
   is moderate, because it is obviously not in use yet by anyone using noble.

 * However, the code for this option got inserted into existing, hence in case
   the new lines are not properly closed/terminated problem can occur
   that can even have an impact on other chzdev arguments and paramaters
   (e.g. the ones that are in the case stmt before and after 'is-owner').

 * The exit code EXIT_UNKNOWN_FILE of 'is-owner' is 33, whereas the defined
   number could be wrong, used accidentially multiple times or a different
   exit code is expected, which may lead to wrong states.

 * The upstream commit needed to be modifed in one aspect (to backport it to 2.31).
   Between version 2.33, where 'is-owner' was added and the version in noble
   (2.31) a refactoring happend (commit 4c2bfb1d47e7), that led (amongst other,
   not relevant changes) to a the renaming of the file zdev/include/site.h to
   zdev/include/zdev.h.
   Fortunately the content of the file stayed the same, so that no add.
   commit needed to be applied, but only the file name in the quilt patch adjusted.

 * Some modification are for the man page and usage.txt file only.

 * For systemd / extra/initramfs-tools/hooks/udev the modification of this
   LP#2044104 are not sofficient, since after all this was introduced into oracular
   two more cases occured that needed to be handled on top, that are:
   - ensure rules file exists before invoking chzdev (LP: #2079993)
   - udev rules are copied in case zdev_early is not specified (LP: #2102236)

 * To simplyfy the systemd modifications (and with that reduce risk) the
   version check of the initial modfification that checks for the s390-tools
   version (2.33, to ensure that chzdev '--is-owner' is only used if the right
   s390-tools package is available) got removed, since this is now backported
   to previous version 2.31 (hence would fail).
   And because noble will never get a new version anymore, the check is obsolete.

 * All this affects the s390x architecture only.

 * (We may think of removing it from the current development release as well
    that comes today with v2.38.0, since we will never go back to an older
    s390-tools version.)

[ Other Info ]

 * The systemd / udev changes will be piggy-baged on a bigger systemd update
   (to avoid too many updates, since this affects s390x-only but would trigger
    updates for other architectures too.)

 * The s390-tools and systemd modifications can be done separately,
   in case s390-tools has landed in the archive before the systemd
   modifications, since systemd will be the first exploiter of the
   s390-tools modification.
   Hence a grouped upload is not needed, if s390-tools is handled first.

 * A test build in PPA is available here:
   https://launchpad.net/~fheimes/+archive/ubuntu/lp2103414+lp2078347+lp2044104
   and the test packages were tested:
   https://pastebin.canonical.com/p/nfGDnHVYWd/
__________

Versions:
Ubuntu 20.04.5 s390-tools version 2.12.0-0ubuntu3.7.s390x
Ubuntu 22.04.2 s390-tools version 2.20.0-0ubuntu3.2.s390x

When I configure a zfcp LUN persistently via chzdev, the initrd is being rebuilt even with parameter zdev:early=0

root@a8315003:~# chzdev -e zfcp-lun 0.0.1803:0x500507630910d430:0x4019409200000000 zdev:early=0
zFCP LUN 0.0.1803:0x500507630910d430:0x4019409200000000 configured
Note: The initial RAM-disk must be updated for these changes to take effect:
       - zFCP LUN 0.0.1803:0x500507630910d430:0x4019409200000000
update-initramfs: Generating /boot/initrd.img-5.15.0-60-generic
I: The initramfs will attempt to resume from /dev/dasdb1
I: (UUID=e70ecb80-4d1e-4074-9cda-ce231ad6e698)
I: Set the RESUME variable to override this.
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: dasda (c00a).
Done.
root@a8315003:~#

== Comment: - Thorsten Diehl <email address hidden> - 2023-03-01 06:55:47 ==
@BOE-dev
This behaviour is unexpected.
https://www.ibm.com/docs/en/linux-on-systems?topic=commands-chzdev says:
Activating a device early during the boot process

Use the zdev:early device attribute to activate a device early during the boot process and to override any existing auto-configuration with a persistent device configuration.

zdev:early=1
    The device is activated during the initial RAM disc phase according to the persistent configuration.

zdev:early=0
    The device is activated as usual during the boot process. This is the default. If auto-configuration data is present, the device is activated during the initial RAM disc phase according to the auto-configuration.

I can't interprete a SCSI LUN as a device with auto configuration data. (At least, if the zfcp device hasn't NPIV enabled)

== Comment: #5 - Peter Oberparleiter <email address hidden> - 2023-03-01 11:18:28 ==
(In reply to comment #2)
> @BOE-dev
> This behaviour is unexpected.
> https://www.ibm.com/docs/en/linux-on-systems?topic=commands-chzdev says:
> Activating a device early during the boot process
>
> Use the zdev:early device attribute to activate a device early during the
> boot process and to override any existing auto-configuration with a
> persistent device configuration.
>
> zdev:early=1
> The device is activated during the initial RAM disc phase according to
> the persistent configuration.
>
> zdev:early=0
> The device is activated as usual during the boot process. This is the
> default. If auto-configuration data is present, the device is activated
> during the initial RAM disc phase according to the auto-configuration.

The documentation is incorrect for Ubuntu. Canonical specifically builds zdev in a way that every change to persistent device configuration causes an update to the initial RAM-disk. See also:

https://bugzilla.linux.ibm.com/show_bug.cgi?id=187578#c35
https://github.com/ibm-s390-linux/s390-tools/commit/7dd03eaeecdd0e2674f145aca34be1275d291bd8

> I can't interprete a SCSI LUN as a device with auto configuration data. (At
> least, if the zfcp device hasn't NPIV enabled)

This is related to auto-configuration as implemented for DPM.

== Comment: #6 - Thorsten Diehl <email address hidden> - 2023-03-03 12:41:44 ==
So, IIUC, chzdev is built for Ubuntu with ZDEV_ALWAYS_UPDATE_INITRD=1, which make the parameter zdev:early=0 ineffective. Correct?
If you confirm, you may also close this bug.

Not nice - then we have to find an alternate solution.

== Comment: #7 - Peter Oberparleiter <email address hidden> - 2023-03-07 06:48:07 ==
(In reply to comment #6)
> So, IIUC, chzdev is built for Ubuntu with ZDEV_ALWAYS_UPDATE_INITRD=1, which
> make the parameter zdev:early=0 ineffective. Correct?
> If you confirm, you may also close this bug.
>
> Not nice - then we have to find an alternate solution.

chzdev -p on Ubuntu will by default rebuild the initrd. This is intended
behavior by Canonical and controlled by the ZDEV_ALWAYS_UPDATE_INITRD build-time
switch. You can suppress it by adding option --no-root-update to the command
line.

Specifying zdev:early=0 to chzdev has exactly the effect that it is supposed to
have: it tells zdev not to enable that device during initrd processing,
resulting in the corresponding udev rule not being copied to the initrd [1].

Unfortunately there is another Ubuntu-initrd script [2] that simply copies ALL
udev rules, including those created by zdev, into the initrd. As a result,
zdev's early-attribute handling is rendered useless and all devices are enabled,
even if a user specified zdev:early=0.

Since this bug report indicates that there is a use-case for this function in
Ubuntu, it might be worth asking Canonical if current processing could be
changed to provide a way for users to specify that a device should specifically
NOT be enabled within initrd processing.

Technically this could easily be done:

1) Have the generic udev initramfs script not copy zdev-generated Udev rules,
   OR
   have the zdev initramfs script remove those rules (somewhat of a hack)

2) Change the zdev initramfs script logic from the current:

   - enable devices required for the root file system, AND
   - enable devices for which zdev:early=1 was specified

   to

   - enable all persistently configured devices EXCEPT those for which
     zdev:early=0 was specified

   This change would be needed to maintain Canonical's policy of enabling
   all devices in the initrd by default

I'm open to adding the change in 2) to our s390-tools package, but someone at
Canonical would need to work out a way to implement 1).

[1] https://github.com/ibm-s390-linux/s390-tools/blob/master/zdev/initramfs/hooks/s390-tools-zdev#L47
[2] https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/tree/debian/extra/initramfs-tools/hooks/udev#n42

[Frank Heimes (fheimes)]

This was implemented around 2020, mainly to catch any ccw config changes in a user friendly way:
https://bugs.launchpad.net/bugs/1892367
https://i527439087.restricted.launchpadlibrarian.net/527439087/20f6c87a-829f-11eb-9412-002481e91f22.txt?token=zC4n7TlCmffBDHm9Dl002PsfchDXC3Dk

Regarding:
"1) Have the generic udev initramfs script not copy zdev-generated Udev rules,"
we need to be super careful here to not break stuff ...

How to best identify the zdev-generated Udev rules,
since these are not all the rules generated by chzdev (indicated by first line in rule), are they?

I'm not very sure if initramfs(-tools) maintainer(s) are very happy ...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-11-22 09:17 EDT-------
(In reply to comment #19)
> This was implemented around 2020, mainly to catch any ccw config changes in
> a user friendly way:
> https://bugs.launchpad.net/bugs/1892367
> https://i527439087.restricted.launchpadlibrarian.net/527439087/20f6c87a-829f-
> 11eb-9412-002481e91f22.txt?token=zC4n7TlCmffBDHm9Dl002PsfchDXC3Dk
>
> Regarding:
> "1) Have the generic udev initramfs script not copy zdev-generated Udev
> rules,"
> we need to be super careful here to not break stuff ...
>
> How to best identify the zdev-generated Udev rules,
> since these are not all the rules generated by chzdev (indicated by first
> line in rule), are they?

chzdev creates udev rules that can most easily be recognized via their name:
- 41-<device-type>-<device-id>.rules
- 41-cio-ignore.rules
- 40-zdev-id.rules

So a simple filter would be to skip rules named 41-*.rules on s390x in the initramfs udev hook.

As an alternative, the s390-tools zdev initramfs hook could try to delete specifically those rules with zdev:early=0 that the upstream udev hook created. While this should work correctly, it is also a kind of a workaround, where one hook adds files that another hook removes again...

If this approach is more likely to be acceptable, it could be achieved by adding something like the following to /usr/share/initramfs-tools/hooks/s390-tools-zdev:

# Remove zdev rules not intended for early boot that were copied by udev hook
chzdev --disable --persistent --by-attr zdev:early=0 \
--base "/etc/udev/rules.d/=$DESTDIR/lib/udev/rules.d/" \
--yes --quiet --no-root-update --force >/dev/null 2>&1

[Simon Chopin (schopin)]

Tagging this to discuss this during the Foundations weekly meeting

[Benjamin Drung (bdrung)]

With my initramfs-tools maintainer hat on, I suggest/support following solution:

Modify /usr/share/initramfs-tools/hooks/udev (shipped by udev, source: systemd) to not copy the udev rules generated by chzdev. So changing:

```
for rules in /etc/udev/rules.d/*.rules; do
  if [ -e "$rules" ] && [ ! -e "/lib/${rules#/etc/}" ]; then
    cp -p "$rules" "$DESTDIR/lib/udev/rules.d/"
  fi
done
```

to something like:

```
for rules in /etc/udev/rules.d/*.rules; do
  if [ -e /sbin/chzdev ] && /sbin/chzdev --is-author-of-udev-rule "$rules"; then
    continue
  fi
  if [ -e "$rules" ] && [ ! -e "/lib/${rules#/etc/}" ]; then
    cp -p "$rules" "$DESTDIR/lib/udev/rules.d/"
  fi
done
```

Then `/sbin/chzdev --is-author-of-udev-rule "$rules"` should produce the correct exit code. The tool that generates the udev rules should be queried (is that chzdev or something else?) or a separate helper should be used.

[Benjamin Drung (bdrung)]

Re-assinging from initramfs-tools to systemd, because /usr/share/initramfs-tools/hooks/udev is shipped by udev.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-01-31 04:28 EDT-------
(In reply to comment #22)
> Then `/sbin/chzdev --is-author-of-udev-rule "$rules"` should produce the
> correct exit code. The tool that generates the udev rules should be queried
> (is that chzdev or something else?) or a separate helper should be used.
>
> Re-assinging from initramfs-tools to systemd, because
> /usr/share/initramfs-tools/hooks/udev is shipped by udev.

I agree with this approach.

Yes, the udev rules are generated by chzdev, so we'll work on adding a chzdev command line option that can be used to query ownership of a udev rule/configuration file similar to what is as outlined above.

I'll update this bug report once the option as available in upstream s390-tools.

[Frank Heimes (fheimes)]

Hi Peter,
is this now incl. in s390-tools v2.33.0
https://github.com/ibm-s390-linux/s390-tools/releases/tag/v2.33.0
and what I read as:
"chzdev: Add --is-owner to identify files created by zdev"
?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-05-28 04:26 EDT-------
(In reply to comment #26)
> Hi Peter,
> is this now incl. in s390-tools v2.33.0
> https://github.com/ibm-s390-linux/s390-tools/releases/tag/v2.33.0
> and what I read as:
> "chzdev: Add --is-owner to identify files created by zdev"
> ?

Peter is currently on vacation, but I can confirm that this feature/commit is now
included as part of the 2.33.0 release.

[Frank Heimes (fheimes)]

Okay, thanks Vineeth for jumping in!

In other words this will become unblocked once we have s390-tools >= 2.33.0 in, which I am currently working on (v2.33.1 - based on LP#2067355).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-05-28 05:00 EDT-------
(In reply to comment #28)
> Okay, thanks Vineeth for jumping in!
>
> In other words this will become unblocked once we have s390-tools >= 2.33.0
> in, which I am currently working on (v2.33.1 - based on LP#2067355).

Yes. That's right. Thank you.

[Steve Langasek (vorlon)]

oracular has s390-tools 2.33.1-0ubuntu3. Is there any further change needed on that package, or should that task be marked resolved?

[Frank Heimes (fheimes)]

@vorlon yes, that's correct. (This bug became a bit abandoned ...)
I just changed the target release to oracular and set s390-tools to Fix Released.

[Frank Heimes (fheimes)]

Where are the initramfs-tools gone? Added it again ... (sorry)

[Benjamin Drung (bdrung)]

See comment #4 and #5: /usr/share/initramfs-tools/hooks/udev is shipped by the udev package (not by initramfs-tools)

[bugproxy (bugproxy)]

Modify /usr/share/initramfs-tools/hooks/udev (shipped by udev, source: system) to not copy the udev rules generated by chzdev. So changing:

Where are the initramfs-tools gone? Added them again ...

[Frank Heimes (fheimes)]

I see Benjamin - thought that I had accidentally removed it by updating the bug (since it was still there in my old browser tab), removed it again.

(Btw. looks like comment #10 is an iisue with the bugzilla-launchpad bridge ...)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-09-05 04:23 EDT-------
The problem seems fixed meanwhile for Oracular (24.10) see here https://launchpad.net/ubuntu/+source/s390-tools
Are you thinking of backports for the LTS releases 22.04 and 20.04 ?

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 256.5-2ubuntu2

---------------
systemd (256.5-2ubuntu2) oracular; urgency=medium

  * initramfs-tools: ensure rules file exists before invoking chzdev (LP: #2079993)

systemd (256.5-2ubuntu1) oracular; urgency=medium

  * Merge with Debian unstable. Remaining changes:
    - debian/tests/tests-in-lxd: run some autopkgtests in LXD too
    - debian/tests/boot-and-services:
      + skip apparmor tests on armhf
      + consume stderr in systemctl status call in test_service
      + drop test_no_failed
    - debian/systemd.postinst:
      + skip daemon-reexec and try-restarts during shutdown
      + manually call systemd-tmpfiles --create in postinst
    - debian/systemd-resolved.postinst: copy existing /etc/resolv.conf to
      /run/systemd/resolve/stub-resolv.conf
    - debian/rules:
      + Remove unneeded efi artifacts on i386 to avoid debugedit errors
    - debian/rules,debian/control,debian/tests/control:
      + Do not build with tpm libraries on i386
      + Do not build with libqrencode on i386
    - debian/gbp.conf,debian/extra/wrap_cl.py:
      Use a customization script to add LP commit links to changelog
    - debian/control:
      + Add Recommends: networkd-dispatcher systemd-resolved to systemd package
      + Give systemd-resolved Priority: important
      + Add Recommends: systemd-hwe-hwdb to udev package
      + Add Breaks: systemd (<< ${binary:Version}) to udev package so that
        systemd is upgraded as well when upgrading udev
      + Make systemd-sysv Depends: on matching version of systemd
      + Drop Recommends: libnss-myhostname libnss-resolve from systemd-resolved
      + Build-Depends: linux-tools-generic
    - debian/tests/upstream: export QEMU_MEM="1024M" for all tests
    - debian/systemd.links: mask systemd-gpt-auto-generator by default
    - debian/systemd.install: exclude files that are not built for i386
    - debian/systemd.manpages: do not ship un-built manpages on i386
    - debian/tests/control: only install systemd-boot-efi for supported arches
    - switch-root: use MS_MOVE for /run when switchig from initrd
    - debian/systemd.postinst: do not create /etc/tmpfiles.d/tmp.conf on upgrades.
      We want the upgrades on Ubuntu to be aligned with what a new install
      would look like.
    - d/control: do not build systemd-boot-efi-{amd64,arm64}-signed-template
  * New changes:
    - Keep utmp support for this release, since we are passed Feature Freeze
      in Ubuntu.
    - Filter out zdev rules in the initramfs hook (LP: #2044104)
    - d/t/upstream: honor /etc/apt configured by autopkgtest

systemd (256.5-2) unstable; urgency=medium

  [ Helmut Grohne ]
  * Fix stage1 build (Closes: #1078821)

  [ Luca Boccassi ]
  * Disable utmp support, replaced by wtmpdb. utmp is not y2038-safe, util-
    linux has now turned it off and relies on logind, so disable utmp
    support in logind too, as it is no longer necessary. wtmpdb replaces
    the functionality.

systemd (256.5-1) unstable; urgency=medium

  * New upstream version 256.5
  * Drop patch merged upstream
  * autopkgtest: skip TEST-64-UDEV-STORAGE due to qemu crash. This tests
    randomly causes qemu to cr...

[bugproxy (bugproxy)]

Uh, good catch Nick (with that it looks like it's since oracular ?!)

And yes, the names may indeed look a bit cryptic,
since these are for s390x specific (so called CCW) devices (here in this case disk devices, but there are more types; cio-ignore is to mask out a range of CCW devices).
One difference between these CCW devices and devices on s390x, and let's say devices in the PC world is, that even if CCW devices are visible and available (means they got detected), they need to be explicitly enabled before usage - and that is what these rules are supposed to do (e.g. "chzdev -e 271d" enables DASD disk device #271d).

I vaguely remember LP#2044104, but need to re-read it to get the full details again ...
(Looks to me like this is a case on non-DPM system (that do not support auto configuration), that was not properly considered. Guess we need to somehow identify if it's an autoconfigure rule - then skip/continue - or not - then copy.)

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2025-04-29 07:49 EDT-------
Hi @Frank,
could we try to get this SRUed to noble? Thanks for your help.

[Frank Heimes (fheimes)]

debdiff for noble s390-tools and systemd

[bugproxy (bugproxy)]

Default Comment by Bridge

[Frank Heimes (fheimes)]

well, due to the confusing two versions with the debdiffs.tgz for noble (one came from the bridge), I'm uploading better a v2 for clarity.

[Frank Heimes (fheimes)]

Packages s390-tools and s390-tools-signed uploaded
(with fixes not only for LP#2044104, but also for LP#2103414 ad LP#2078347).

[Nick Rosbrook (enr0n)]

I think s390-tools can indeed be accepted before systemd, but your test plan currently only tests the integration.

Please add a test plan for s390-tools only, e.g. exercising the new --is-owner flag directly, along with some other typical usage of that tool.

[Frank Heimes (fheimes)]

Hi Nick, ok - I've now added a more direct test that uses the --is-owner directly; a positive and a negative test, based on real world udev rules.
Other typical usage scenarios are mainly enabling and disabling devices, what is done in the more end2end test at the beginning of the test plan.
Since this is mainly used at install time, another test could be to start an installation, at the initial subiquity screen navigate the to installer shell and update the s390-tools there to the latest version, leave the installer shell and proceed with the installation, that then runs through the zDev activation screen.
I'm adding this to the Tet Plan on top ...

[Nick Rosbrook (enr0n)]

Hello bugproxy, or anyone else affected,

Accepted s390-tools into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/2.31.0-0ubuntu5.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

Hello bugproxy, or anyone else affected,

Accepted s390-tools-signed into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools-signed/2.31.0-0ubuntu5.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Frank Heimes (fheimes)]

I've successfully verified the s390-tools part:
https://pastebin.canonical.com/p/Ff5dBn7CRb/

[Frank Heimes (fheimes)]

... and I've updated the tags.

[Frank Heimes (fheimes)]

Successfully verified on plucky too: https://pastebin.canonical.com/p/BNZt4pvWnQ/

[Nick Rosbrook (enr0n)]

Hello bugproxy, or anyone else affected,

Accepted s390-tools into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/2.31.0-0ubuntu5.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Nick Rosbrook (enr0n)]

Hello bugproxy, or anyone else affected,

Accepted s390-tools-signed into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools-signed/2.31.0-0ubuntu5.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Frank Heimes (fheimes)]

s390-tools successfully verified on noble.
Updating the tags accordingly.

[Julian Andres Klode (juliank)]

Hello bugproxy, or anyone else affected,

Accepted systemd into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/255.4-1ubuntu8.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (255.4-1ubuntu8.11) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

casper/1.498 (amd64)
collectd/5.12.0-17.1build2 (s390x)
exim4/4.97-4ubuntu4.3 (ppc64el)
haproxy/2.8.5-1ubuntu3.3 (s390x)
linux-azure-6.11/6.11.0-1018.18~24.04.1 (arm64)
linux-gke/6.8.0-1033.37 (arm64)
linux-hwe-6.11/6.11.0-29.29~24.04.1 (ppc64el)
linux-hwe-6.14/unknown (arm64)
linux-nvidia/6.8.0-1036.39 (amd64)
linux-nvidia-6.11/6.11.0-1013.13 (arm64)
linux-nvidia-6.14/6.14.0-1009.9 (amd64)
linux-nvidia-lowlatency/unknown (arm64)
linux-raspi-realtime/6.8.0-2019.20 (arm64)
linux-realtime/6.8.1-1015.16 (arm64)
linux-xilinx/6.8.0-1017.18 (arm64)
mariadb/1:10.11.13-0ubuntu0.24.04.1 (arm64)
ovn/24.03.2-0ubuntu0.24.04.2 (amd64)
sssd/2.9.4-1.1ubuntu6.3 (s390x)
udisks2/2.10.1-6ubuntu1.3 (ppc64el)
upower/1.90.3-1 (ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Frank Heimes (fheimes)]

Now again successfully verified, with s390-tools and systemd.
Adjusting the tags accordingly ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools - 2.31.0-0ubuntu5.3

---------------
s390-tools (2.31.0-0ubuntu5.3) noble; urgency=medium

  * Add missing commit/patch
    d/p/lp-2103414-cpumf-lscpumf-Add-support-for-IBM-z17-counter-sets.patch
    that caused a failed verification of 2.31.0-0ubuntu5.2 (LP: #2103414).

s390-tools (2.31.0-0ubuntu5.2) noble; urgency=medium

  * Add d/p/lp-2103414-s390-tools-libutil-Add-machine-type-3932.patch and
    d/p/lp-2103414-libutil-Add-machine-type-definition-for-z17.patch to
    achieve support for IBM z17 (and z16 GA2) hw specific CPU-MF counters
    (LP: #2103414).
  * Add d/p/lp-2078347-udev-Introduce-rule-to-set-newly-hotplugged-CPUs.patch
    that fixes CPUs from staying offline (LP: #2078347).
  * Add d/p/lp-2044104-zdev-option-to-identify-files-created-by-zdev.patch to
    avoid rebuild initramfs by chzdev -e if zdev:early=0 set (LP: #2044104).

 -- Frank Heimes <email address hidden> Fri, 08 Aug 2025 11:32:43 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package s390-tools-signed - 2.31.0-0ubuntu5.3

---------------
s390-tools-signed (2.31.0-0ubuntu5.3) noble; urgency=medium

  * * Rebuild against 2.31.0-0ubuntu5.3 (LP: #2103414)

s390-tools-signed (2.31.0-0ubuntu5.2) noble; urgency=medium

  * Rebuild against 2.31.0-0ubuntu5.2 (LP: #2103414, LP: #2078347, LP: #2044104)

 -- Frank Heimes <email address hidden> Fri, 08 Aug 2025 11:37:21 +0200

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for s390-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2025-09-10 06:40 EDT-------
With the fix being released, we can close this bug.

Thanks everyone for all your work!

==> Changing the status to "CLOSED"

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 255.4-1ubuntu8.11

---------------
systemd (255.4-1ubuntu8.11) noble; urgency=medium

  [ Nick Rosbrook ]
  * initramfs-tools: copy hwdb.bin to initramfs (LP: #2112237)
  * d/t/tests-in-lxd: drop patching workaround (LP: #2115263)
    - d/t/control: add Depends: dnsmasq-base
      (Revealed by test progressing past previous failure)
  * initramfs-tools: filter out zdev rules in the initramfs hook (LP: #2044104)
    Backport the logic from plucky onward, but adjust the version string for
    noble.
  * test: fall back to SYSLOG_IDENTIFIER= matching in TEST-75-RESOLVED
    Partially backport the test fix from 49a954b08654dd06bab71224a2398a65c2555549,
    only targeting TEST-75-RESOLVED.

  [ Matthew Ruffell ]
  * pcrlock: handle measurement logs where hash algs in header.
    Fix pcrlock log to function correctly reading the TPM eventlog on hyper-v VMs
    (LP: #2115391)

  [ Chengen Du ]
  * network/dhcp6: consider the DHCPv6 protocol as finished when conflict addresses exist
    (LP: #2115418)

  [ Mario Limonciello ]
  * Drop support for using actual brightness (LP: #2110585)

 -- Nick Rosbrook <email address hidden> Fri, 11 Jul 2025 14:52:59 -0400