Differents test cases and attepmts 1. to 6. 1. okay - LPAR install with 162F snapd seeding stil takes a while ... 2. okay - LPAR install with 162F (second time, which is usually the critial case) once having hit "enable" it took a little bit longer than usual, but came back w/o problems This step: "[[0;32m OK [0m] Finished [0;1;39msnapd.seeded.serv Wait until snapd is fully seeded." takes (repeatingly with each installation) about 1:50secs, where a login is not yet possible. 2.1. - checking what happened with udev ubuntu@s1lp15:~$ grep -c udev /var/log/syslog 12 ubuntu@s1lp15:~$ grep udevadm /var/log/syslog 2023-09-27T08:12:20.445780+00:00 s1lp15 udevadm[591]: systemd-udev-settle.service is deprecated. Please fix zfs-load-module.service, zfs-import-cache.service not to pull it in. ubuntu@s1lp15:~$ [ ubuntu@s1lp15:~$ grep -c udev /var/log/installer/subiquity-server-debug.log 194 ubuntu@s1lp15:~$ grep -c udevadm /var/log/installer/subiquity-server-debug.log 132 ] 2.2 - manually enabling another (arbitrary) diskm that took ages: ubuntu@s1lp15:~$ sudo chzdev -e 260b Timed out for waiting the udev queue being empty. Timed out for waiting the udev queue being empty. ECKD DASD 0.0.260b configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.260b update-initramfs: Generating /boot/initrd.img-6.5.0-5-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device for CCW- and LD-IPL: dasda (162f). Done. ubuntu@s1lp15:~$ And top shows a lot of udev/systemd activity: 533256 root 20 0 26440 7224 2560 S 34.9 0.0 0:02.24 (udev-worker) 509 root 20 0 26436 8800 4576 R 31.2 0.0 4:28.72 /lib/systemd/systemd-udevd 533257 root 20 0 26440 7216 2560 R 30.9 0.0 0:01.98 (udev-worker) 912 root 20 0 468136 14592 10624 R 28.2 0.0 3:45.84 /usr/libexec/udisks2/udisksd 1 root 20 0 168664 12492 8268 S 27.9 0.0 3:40.41 /sbin/init 883 message+ 20 0 9420 4608 3968 R 22.6 0.0 3:02.79 @dbus-daemon --system --address=systemd: --nofork --nopidfil+ 81639 ubuntu 20 0 17992 9984 8448 S 16.9 0.0 1:51.68 /lib/systemd/systemd --user 533255 root 20 0 26440 7188 2560 S 10.0 0.0 0:00.64 (udev-worker) ubuntu@s1lp15:~$ date && sudo chzdev -e 2609 && date Wed Sep 27 08:30:46 AM UTC 2023 ECKD DASD 0.0.2609 configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.2609 update-initramfs: Generating /boot/initrd.img-6.5.0-5-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device for CCW- and LD-IPL: dasda (162f). Done. Wed Sep 27 08:30:54 AM UTC 2023 ubuntu@s1lp15:~$ After having enabled two more disks: ubuntu@s1lp15:~$ grep -c udev /var/log/syslog 21 Do we know what is happening at these "auto-import" lines ?! " 2023-09-27T08:12:20.445761+00:00 s1lp15 (udev-worker)[561]: dasda: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasda' failed with exit code 1. " or " 2023-09-27T08:30:48.734832+00:00 s1lp15 (udev-worker)[364360]: dasdc2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdc2' failed with exit code 1. " I don't think that I've noticed such messages before. And imho it does not make sense to try to auto-import anything from a dasd once it just enabled, since one does not know what's on it?! Looking now at /var/log/syslog I get repeating msgs like this: " 2023-09-27T08:56:23.428746+00:00 s1lp15 udisksd[912]: Couldn't find existing drive object for device /sys/devices/css0/0.0.028f/0.0.162f/block/dasda (uevent action 'change', VPD '0X162F') 2023-09-27T08:56:23.437296+00:00 s1lp15 udisksd[912]: Couldn't find existing drive object for device /sys/devices/css0/0.0.06cb/0.0.260b/block/dasdb (uevent action 'change', VPD '0X260B') " That's an issue ... and causes the syslog file to grow significantly: -rw-r----- 1 syslog adm 35346664 Sep 27 09:02 /var/log/syslog Bus-ID Status Name Device Type BlkSz Size Blocks ================================================================================ 0.0.162f active dasda 94:0 ECKD 4096 21129MB 5409180 0.0.260b active dasdb 94:4 ECKD 4096 7043MB 1803060 0.0.2609 active dasdc 94:8 ECKD 4096 7043MB 1803060 TODO: check chzdev upstream for any recent but suspicious changes ... 3. re-trying LPAR with 2609 (since the above msgs do not occur for 2609) top right after the system is up: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1803 root 20 0 24948 5732 2560 S 40.9 0.0 1:35.13 (udev-worker) 509 root 20 0 24944 7508 4600 S 21.9 0.0 0:48.52 /lib/systemd/systemd-udevd 1 root 20 0 168660 12876 8396 R 16.9 0.0 0:39.05 /sbin/init 905 root 20 0 467912 13384 10496 S 16.9 0.0 0:39.39 /usr/libexec/udisks2/udisksd 880 message+ 20 0 9308 4480 3840 S 14.6 0.0 0:31.19 @dbus-daemon --system --address=systemd: --nofork --+ again several udev related processes that eat up quite some resources ... Does not seem to be normal ... Right after a reboot: 561 root 20 0 23928 5076 2944 S 40.2 0.0 1:30.28 (udev-worker) 498 root 20 0 24992 7296 4608 S 21.3 0.0 0:47.36 systemd-udevd 784 root 20 0 467912 12792 10624 S 17.3 0.0 0:38.47 udisksd 1 root 20 0 167292 11468 8396 R 16.6 0.0 0:37.77 systemd 743 message+ 20 0 9296 4224 3840 S 13.6 0.0 0:30.26 dbus-daemon 83408 ubuntu 20 0 17980 9856 8448 S 10.6 0.0 0:02.16 systemd 483 root rt 0 288412 26368 7936 S 6.3 0.1 0:13.41 multipathd 3. install with FCP/SCSI LUN (instead of DASD) top right after reboot completed: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 290 root 20 0 0 0 0 I 0.3 0.0 0:00.04 kworker/u564:4-mlx4_en 1 root 20 0 168796 12748 8396 S 0.0 0.0 0:02.78 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd So it looks to me that the udev issues do not seem to happen with FCP/SCSI LUNs (even on mantic), for whatever reason... [ ubuntu@s1lp15:~$ grep -c udev /var/log/installer/subiquity-server-debug.log 19 ubuntu@s1lp15:~$ grep -c udev /var/log/syslog 22 ] Now enabling a DASD on top of SCSI: ubuntu@s1lp15:~$ sudo chzdev -e 162f [sudo] password for ubuntu: ECKD DASD 0.0.162f configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.162f update-initramfs: Generating /boot/initrd.img-6.5.0-5-generic Using config file '/etc/zipl.conf' Run /lib/s390-tools//zipl_helper.device-mapper /boot Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device for LD-IPL: dm-0. Done. ubuntu@s1lp15:~$ 3624 root 20 0 25964 6372 2560 D 32.6 0.0 0:01.59 (udev-worker) 982 root 20 0 25456 7820 4588 R 13.6 0.0 0:01.52 /lib/systemd/systemd-udevd 1583 root 20 0 468072 13040 10624 S 12.0 0.0 0:01.01 /usr/libexec/udisks2/udisksd but that worked relatively smooth ... (and udev stopped after just some seconds - looked reasonable) [ ubuntu@s1lp15:~$ grep -c udev /var/log/syslog 25 ubuntu@s1lp15:~$ ] Again another DASD: ubuntu@s1lp15:~$ sudo chzdev -e 260b ECKD DASD 0.0.260b configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.260b update-initramfs: Generating /boot/initrd.img-6.5.0-5-generic Using config file '/etc/zipl.conf' Run /lib/s390-tools//zipl_helper.device-mapper /boot Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device for LD-IPL: dm-0. Done. ubuntu@s1lp15:~$ top: 13296 root 20 0 25436 6300 2688 S 28.2 0.0 0:01.51 (udev-worker) 1 root 20 0 168796 12748 8396 S 15.9 0.0 0:04.97 /sbin/init 1583 root 20 0 468152 13160 10624 S 15.3 0.0 0:02.32 /usr/libexec/udisks2/udisksd 982 root 20 0 25448 7820 4588 S 15.0 0.0 0:02.89 /lib/systemd/systemd-udevd 1551 message+ 20 0 9268 4608 3968 S 12.6 0.0 0:01.81 @dbus-daemon --system --address=systemd: --n+ 13727 root 20 0 25452 6500 2944 S 10.0 0.0 0:00.38 (udev-worker) Was again quick and smooth. Another one: ubuntu@s1lp15:~$ sudo chzdev -e 2609 ECKD DASD 0.0.2609 configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.2609 update-initramfs: Generating /boot/initrd.img-6.5.0-5-generic Using config file '/etc/zipl.conf' Run /lib/s390-tools//zipl_helper.device-mapper /boot Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device for LD-IPL: dm-0. Done. ubuntu@s1lp15:~$ top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23375 root 20 0 25436 6348 2688 S 30.6 0.0 0:02.06 (udev-worker) 982 root 20 0 25432 7820 4588 S 14.6 0.0 0:04.38 /lib/systemd/systemd-udevd 1583 root 20 0 468192 13228 10624 S 12.6 0.0 0:03.66 /usr/libexec/udisks2/udisksd 1551 message+ 20 0 9268 4608 3968 S 9.3 0.0 0:02.93 @dbus-daemon --system --address=systemd: --n+ 2350 ubuntu 20 0 18012 10112 8448 S 6.3 0.0 0:02.11 /lib/systemd/systemd --user 1573 root 20 0 2731960 35552 20736 S 3.7 0.1 0:02.58 /usr/lib/snapd/snapd ubuntu@s1lp15:~$ grep -c udev /var/log/syslog 33 ubuntu@s1lp15:~$ ubuntu@s1lp15:~$ grep udev /var/log/syslog 2023-09-27T09:41:57.713406+00:00 s1lp15 systemd[1]: Starting systemd-udevd.service - Rule-based Manager for Device Events and Files... 2023-09-27T09:41:57.713409+00:00 s1lp15 systemd-udevd[982]: Using default interface naming scheme 'v253'. 2023-09-27T09:41:57.713415+00:00 s1lp15 systemd[1]: Started systemd-udevd.service - Rule-based Manager for Device Events and Files. 2023-09-27T09:41:57.713436+00:00 s1lp15 (udev-worker)[984]: dm-3: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-3' failed with exit code 1. 2023-09-27T09:41:57.713447+00:00 s1lp15 (udev-worker)[992]: dm-0: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-0' failed with exit code 1. 2023-09-27T09:41:57.713451+00:00 s1lp15 (udev-worker)[989]: dm-1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-1' failed with exit code 1. 2023-09-27T09:41:57.713482+00:00 s1lp15 systemd[1]: Finished systemd-udev-trigger.service - Coldplug All udev Devices. 2023-09-27T09:41:57.713500+00:00 s1lp15 systemd[1]: Starting systemd-udev-settle.service - Wait for udev To Complete Device Initialization... 2023-09-27T09:41:57.713503+00:00 s1lp15 udevadm[1157]: systemd-udev-settle.service is deprecated. Please fix zfs-load-module.service, zfs-import-cache.service not to pull it in. 2023-09-27T09:41:57.713516+00:00 s1lp15 (udev-worker)[1096]: sdd: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdd' failed with exit code 1. 2023-09-27T09:41:57.713548+00:00 s1lp15 (udev-worker)[1047]: sda: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda' failed with exit code 1. 2023-09-27T09:41:57.713554+00:00 s1lp15 (udev-worker)[1103]: sdc: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdc' failed with exit code 1. 2023-09-27T09:41:57.713559+00:00 s1lp15 (udev-worker)[1011]: sdb: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdb' failed with exit code 1. 2023-09-27T09:41:57.713575+00:00 s1lp15 (udev-worker)[1010]: dm-2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-2' failed with exit code 1. 2023-09-27T09:41:57.713585+00:00 s1lp15 (udev-worker)[1032]: rport-0:0-0: /etc/udev/rules.d/41-zfcp-lun-0.0.e000:0x50050763061b16b6:0x4026400300000000.rules:10 Failed to write ATTR{/sys/devices/css0/0.0.0d4d/0.0.e000/0x50050763061b16b6/unit_add}, ignoring: File exists 2023-09-27T09:41:57.713589+00:00 s1lp15 (udev-worker)[1096]: rport-0:0-1: /etc/udev/rules.d/41-zfcp-lun-0.0.e000:0x50050763060b16b6:0x4026400300000000.rules:10 Failed to write ATTR{/sys/devices/css0/0.0.0d4d/0.0.e000/0x50050763060b16b6/unit_add}, ignoring: File exists 2023-09-27T09:41:57.713593+00:00 s1lp15 (udev-worker)[1011]: rport-1:0-0: /etc/udev/rules.d/41-zfcp-lun-0.0.e100:0x50050763061b16b6:0x4026400300000000.rules:10 Failed to write ATTR{/sys/devices/css0/0.0.0d5d/0.0.e100/0x50050763061b16b6/unit_add}, ignoring: File exists 2023-09-27T09:41:57.713605+00:00 s1lp15 (udev-worker)[1098]: rport-1:0-1: /etc/udev/rules.d/41-zfcp-lun-0.0.e100:0x50050763060b16b6:0x4026400300000000.rules:10 Failed to write ATTR{/sys/devices/css0/0.0.0d5d/0.0.e100/0x50050763060b16b6/unit_add}, ignoring: File exists 2023-09-27T09:41:57.713721+00:00 s1lp15 systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization. 2023-09-27T09:41:57.714115+00:00 s1lp15 kernel: [ 17.266284] systemd[1]: Listening on systemd-udevd-control.socket - udev Control Socket. 2023-09-27T09:41:57.714115+00:00 s1lp15 kernel: [ 17.266500] systemd[1]: Listening on systemd-udevd-kernel.socket - udev Kernel Socket. 2023-09-27T09:41:57.714123+00:00 s1lp15 kernel: [ 17.292672] systemd[1]: Starting systemd-udev-trigger.service - Coldplug All udev Devices... 2023-09-27T10:06:38.948331+00:00 s1lp15 (udev-worker)[2868]: dasda: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasda' failed with exit code 1. 2023-09-27T10:06:39.006126+00:00 s1lp15 (udev-worker)[2864]: dasda1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasda1' failed with exit code 1. 2023-09-27T10:06:50.880703+00:00 s1lp15 (udev-worker)[3624]: dasda1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasda1' failed with exit code 1. 2023-09-27T10:09:53.743091+00:00 s1lp15 (udev-worker)[13296]: dasdb: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdb' failed with exit code 1. 2023-09-27T10:09:53.857514+00:00 s1lp15 (udev-worker)[13296]: dasdb1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdb1' failed with exit code 1. 2023-09-27T10:09:53.986111+00:00 s1lp15 (udev-worker)[13295]: dasdb2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdb2' failed with exit code 1. 2023-09-27T10:10:02.712525+00:00 s1lp15 (udev-worker)[13296]: dasdb1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdb1' failed with exit code 1. 2023-09-27T10:10:02.834632+00:00 s1lp15 (udev-worker)[13727]: dasdb2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdb2' failed with exit code 1. 2023-09-27T10:12:32.187906+00:00 s1lp15 (udev-worker)[23376]: dasdc: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdc' failed with exit code 1. 2023-09-27T10:12:32.226086+00:00 s1lp15 (udev-worker)[23375]: dasdc1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdc1' failed with exit code 1. 2023-09-27T10:12:50.224412+00:00 s1lp15 (udev-worker)[23375]: dasdc1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdc1' failed with exit code 1. But here no repeating messages (in syslog) ... 4. re-trying DASD install on 162F, BUT this time having the DASDs low level formatted first sudo dasdfmt -y -b 4096 /dev/dasda # and dasdb and dasdc (such a dasd is pristine and does not have any partitions at all, hence that's why I'm trying this ...) [ $ lsdasd Bus-ID Status Name Device Type BlkSz Size Blocks ================================================================================ 0.0.162f active dasda 94:0 ECKD 4096 21129MB 5409180 0.0.260b active dasdb 94:4 ECKD 4096 7043MB 1803060 0.0.2609 active dasdc 94:8 ECKD 4096 7043MB 1803060 ] Now doing the next LPAR install on 162F ... The enabling of disk 162F in the zdev screen was quick (~1 sec). This: "[[0;32m OK [0m] Finished [0;1;39msnapd.seeded.serv Wait until snapd is fully seeded." took again ~1:50. top shows udev related things: 508 root 20 0 28072 10728 4584 S 20.7 0.0 0:34.83 systemd-udevd 1 root 20 0 168664 12400 8304 S 16.7 0.0 0:28.18 systemd 891 root 20 0 467916 12884 10496 S 16.7 0.0 0:27.68 udisksd 878 message+ 20 0 9308 4352 3840 S 13.7 0.0 0:22.05 dbus-daemon 62383 ubuntu 20 0 17980 9728 8320 S 9.7 0.0 0:01.56 systemd 493 root rt 0 288412 26496 8064 S 6.0 0.1 0:09.80 multipathd 889 root 20 0 2657972 36672 21632 S 4.7 0.1 0:08.58 snapd syslog shows again msgs like: " 2023-09-27T10:49:49.092897+00:00 s1lp15 udisksd[891]: Couldn't find existing drive object for device /sys/devices/css0/0.0.028f/0.0.162f/block/dasda (uevent action 'change', VPD '0X162F') " but this time not repeatingly, hence syslog is reasonably small: " -rw-r----- 1 syslog adm 140474 Sep 27 10:49 /var/log/syslog " Now enabling more dasds: ubuntu@s1lp15:~$ sudo chzdev -e 260b takes a long time ... top: 163921 root 20 0 28076 8612 2432 S 38.3 0.0 0:07.27 (udev-worker) 508 root 20 0 28072 10728 4584 S 22.0 0.0 1:26.81 /lib/systemd/systemd-udevd 891 root 20 0 467976 13596 10496 S 17.0 0.0 1:09.08 /usr/libexec/udisks2/udisksd 1 root 20 0 168664 12400 8304 S 16.7 0.0 1:07.72 /sbin/init 878 message+ 20 0 9308 4352 3840 S 14.3 0.0 0:55.39 @dbus-daemon --system --address=systemd:+ ubuntu@s1lp15:~$ sudo chzdev -e 260b [sudo] password for ubuntu: ECKD DASD 0.0.260b configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.260b update-initramfs: Generating /boot/initrd.img-6.5.0-5-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device for CCW- and LD-IPL: dasda (162f). Done. the processes are not gone after chzdev is back: 208388 root 20 0 28076 8740 2560 S 40.3 0.0 0:56.82 (udev-worker) 508 root 20 0 28072 10728 4584 S 20.3 0.0 2:12.70 /lib/systemd/systemd-udevd 891 root 20 0 468016 14204 10496 D 17.0 0.0 1:46.48 /usr/libexec/udisks2/udisksd 1 root 20 0 168664 12400 8304 S 15.7 0.0 1:42.80 /sbin/init 878 message+ 20 0 9308 4352 3840 R 13.3 0.0 1:24.92 @dbus-daemon --system --address=systemd:+ 62383 ubuntu 20 0 17980 9728 8320 S 9.3 0.0 0:46.33 /lib/systemd/systemd --user $ lsdasd Bus-ID Status Name Device Type BlkSz Size Blocks ================================================================================ 0.0.162f active dasda 94:0 ECKD 4096 21129MB 5409180 0.0.260b active dasdb 94:4 ECKD 4096 7043MB 1803060 again with 2609: $ sudo chzdev -e 2609 ECKD DASD 0.0.2609 configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.2609 update-initramfs: Generating /boot/initrd.img-6.5.0-5-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device for CCW- and LD-IPL: dasda (162f). Done. This was quicker - hmm. ubuntu@s1lp15:~$ grep -c udev /var/log/syslog 14 s Name Device Type BlkSz Size Blocks ================================================================================ 0.0.162f active dasda 94:0 ECKD 4096 21129MB 5409180 0.0.260b active dasdb 94:4 ECKD 4096 7043MB 1803060 0.0.2609 active dasdc 94:8 ECKD 4096 7043MB 1803060 Now enabling SCSI/FCP hosts: $ sudo chzdev -e e000,e100 FCP device 0.0.e000 configured FCP device 0.0.e100 configured Timed out for waiting the udev queue being empty. Note: The initial RAM-disk must be updated for these changes to take effect: - FCP device 0.0.e000 - FCP device 0.0.e100 update-initramfs: Generating /boot/initrd.img-6.5.0-5-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device for CCW- and LD-IPL: dasda (162f). Done. Woow, that took more than 2 minutes (that's long). ____ 5. Now doing similar tests with 22.04.3 (and updating installer): updated from default: subiquity 23.08.1 5006 latest/stable/… canonical** classic to: subiquity 22.02.2+git1762.1b1ee6f4 5164 Enabling 162F was very quick (<= 1sec). The "[[0;32m OK [0m] Finished [0;1;39msnapd.seeded.serv Wait until snapd is fully seeded." messages does not come up at all (on jammy, with updated installer!) Right after the system is up, top shows no udev activities. ubuntu@s1lp15:~$ grep -c udev /var/log/installer/subiquity-server-info.log 0 Hmm - no entries at all?! --^ ubuntu@s1lp15:~$ grep -c udev /var/log/syslog 11 Enabling more DASDs: ubuntu@s1lp15:~$ sudo chzdev -e 260b [sudo] password for ubuntu: ECKD DASD 0.0.260b configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.260b update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device: dasda (162f). Done. ubuntu@s1lp15:~$ Very quick, time was mainly update-initramfs only, nothing in top about udev. Again - super quick, 5 secs for update-initramfs only: $ sudo chzdev -e 2609 ECKD DASD 0.0.2609 configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.2609 update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device: dasda (162f). Done. Enabling SCSI hosts: $ sudo chzdev -e e000,e100 FCP device 0.0.e000 configured FCP device 0.0.e100 configured Note: The initial RAM-disk must be updated for these changes to take effect: - FCP device 0.0.e000 - FCP device 0.0.e100 update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device: dasda (162f). Done. Again super quick, maybe 1 sec for the device enablement, and 5 secs for update-initramfs. And could not see anything about udev in top! 6. Re-doing the same with a 22.04.3 install (things seemed to fail in case DASDs are no longer pristine, now 162F is used ...) (again with updated installer) Enablement of 162F in zdev screen was super quick (<= 1sec). Now I've found some snapd/seeded msg: " "Starting [0;1;39mWait until snapd is fully seeded[0m..." ... "[[0;32m OK [0m] Finished [0;1;39mWait until snapd is fully seeded[0m." " but without any delay... Now manually enabling again some DASD and observe: ubuntu@s1lp15:~$ lsdasd Bus-ID Status Name Device Type BlkSz Size Blocks ================================================================================ 0.0.162f active dasda 94:0 ECKD 4096 21129MB 5409180 ubuntu@s1lp15:~$ sudo chzdev -e 260b [sudo] password for ubuntu: ECKD DASD 0.0.260b configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.260b update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device: dasda (162f). Done. ubuntu@s1lp15:~$ sudo chzdev -e 2609 ECKD DASD 0.0.2609 configured Note: The initial RAM-disk must be updated for these changes to take effect: - ECKD DASD 0.0.2609 update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device: dasda (162f). Done. ubuntu@s1lp15:~$ lsdasd Bus-ID Status Name Device Type BlkSz Size Blocks ================================================================================ 0.0.162f active dasda 94:0 ECKD 4096 21129MB 5409180 0.0.260b active dasdb 94:4 ECKD 4096 7043MB 1803060 0.0.2609 active dasdc 94:8 ECKD 4096 7043MB 1803060 ubuntu@s1lp15:~$ Zero delay in getting the devices enabled. Similar with FCP hosts: $ sudo chzdev -e e000,e100 FCP device 0.0.e000 configured FCP device 0.0.e100 configured Note: The initial RAM-disk must be updated for these changes to take effect: - FCP device 0.0.e000 - FCP device 0.0.e100 update-initramfs: Generating /boot/initrd.img-5.15.0-84-generic Using config file '/etc/zipl.conf' Building bootmap in '/boot' Adding IPL section 'ubuntu' (default) Preparing boot device: dasda (162f). Done. Some messages in the logs: ubuntu@s1lp15:~$ grep -c udev /var/log/syslog 30 ubuntu@s1lp15:~$ grep udev /var/log/syslog Sep 27 11:45:49 s1lp15 systemd[1]: Finished Coldplug All udev Devices. Sep 27 11:45:49 s1lp15 systemd-udevd[630]: dasda: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasda' failed with exit code 1. Sep 27 11:45:49 s1lp15 systemd-udevd[630]: dasda1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasda1' failed with exit code 1. Sep 27 11:45:49 s1lp15 systemd-udevd[643]: Using default interface naming scheme 'v249'. Sep 27 11:45:49 s1lp15 systemd-udevd[645]: Using default interface naming scheme 'v249'. Sep 27 11:45:49 s1lp15 systemd-udevd[636]: Using default interface naming scheme 'v249'. Sep 27 11:45:49 s1lp15 systemd-udevd[639]: Using default interface naming scheme 'v249'. Sep 27 11:45:49 s1lp15 systemd-udevd[642]: Using default interface naming scheme 'v249'. Sep 27 11:45:49 s1lp15 systemd-udevd[630]: Using default interface naming scheme 'v249'. Sep 27 11:45:49 s1lp15 kernel: [ 14.851119] systemd[1]: Listening on udev Control Socket. Sep 27 11:45:49 s1lp15 kernel: [ 14.851207] systemd[1]: Listening on udev Kernel Socket. Sep 27 11:45:49 s1lp15 kernel: [ 14.882905] systemd[1]: Starting Coldplug All udev Devices... Sep 27 11:49:24 s1lp15 systemd-udevd[1653]: dasdb: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdb' failed with exit code 1. Sep 27 11:49:36 s1lp15 systemd-udevd[6051]: dasdc: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dasdc' failed with exit code 1. Sep 27 11:50:15 s1lp15 systemd-udevd[10456]: sdb: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdb' failed with exit code 1. Sep 27 11:50:15 s1lp15 systemd-udevd[10455]: sda: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda' failed with exit code 1. Sep 27 11:50:15 s1lp15 systemd-udevd[10460]: sdb2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdb2' failed with exit code 1. Sep 27 11:50:15 s1lp15 systemd-udevd[10459]: sda2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda2' failed with exit code 1. Sep 27 11:50:15 s1lp15 systemd-udevd[10456]: sdb1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdb1' failed with exit code 1. Sep 27 11:50:15 s1lp15 systemd-udevd[10455]: sda1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sda1' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10459]: dm-0: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-0' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10455]: sdc: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdc' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10456]: sdd: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdd' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10455]: sdc1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdc1' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10460]: sdc2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdc2' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10459]: sdd2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdd2' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10456]: sdd1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/sdd1' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10460]: dm-1: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-1' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10459]: dm-2: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-2' failed with exit code 1. Sep 27 11:50:16 s1lp15 systemd-udevd[10456]: dm-3: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/dm-3' failed with exit code 1. ubuntu@s1lp15:~$ That means the snap auto-import things happened also before, hence it's probably not the cause. But here they don't seem to be repeating here, with jammy.