System can not shutdown if system has multiple VROC RAID arrays

[Cyrus Lien]

[ Impact ]

The system can not shutdown if the system has multiple VROC RAID arrays.
Intel has fixed it in systemd v251 [1].
Need to cherry-pick the commit to ubuntu-jammy systemd 249.11-0ubuntu3.9.

[1] The commit fixes the issue:
commit 3a3b022d2cc112803ea7b9beea98bbcad110368a
Author: Mariusz Tkaczyk <email address hidden>
Date: Tue Mar 29 12:49:54 2022 +0200

    shutdown: get only active md arrays.

    Current md_list_get() implementation filters all block devices, started from
    "md*". This is ambiguous because list could contain:
    - partitions created upon md device (mdXpY)
    - external metadata container- specific type of md array.

    For partitions there is no issue, because they aren't handle STOP_ARRAY
    ioctl sent later. It generates misleading errors only.

    Second case is more problematic because containers are not locked in kernel.
    They are stopped even if container member array is active. For that reason
    reboot or shutdown flow could be blocked because metadata manager cannot be
    restarted after switch root on shutdown.

    Add filters to remove partitions and containers from md_list. Partitions
    can be excluded by DEVTYPE. Containers are determined by MD_LEVEL
    property, we are excluding all with "container" value.

    Signed-off-by: Mariusz Tkaczyk <email address hidden>

In the journal, we can see systemd-shutdown looping repeatedly as it tries and fails to detach all md devices:

...
[ 513.416293] systemd-shutdown[1]: Stopping MD /dev/md124p2 (259:5).
[ 513.422953] systemd-shutdown[1]: Could not stop MD /dev/md124p2: Device or resource busy
[ 513.431227] systemd-shutdown[1]: Stopping MD /dev/md124p1 (259:4).
[ 513.437952] systemd-shutdown[1]: Could not stop MD /dev/md124p1: Device or resource busy
[ 513.449298] systemd-shutdown[1]: Stopping MD /dev/md124 (9:124).
[ 513.456278] systemd-shutdown[1]: Could not stop MD /dev/md124: Device or resource busy
[ 513.465323] systemd-shutdown[1]: Not all MD devices stopped, 4 left.
[ 513.472564] systemd-shutdown[1]: Couldn't finalize remaining MD devices, trying again.
[ 513.485302] systemd-shutdown[1]: Failed to open watchdog device /dev/watchdog: No such file or directory
[ 513.496195] systemd-shutdown[1]: Stopping MD devices.
[ 513.502176] systemd-shutdown[1]: sd-device-enumerator: Scan all dirs
[ 513.513382] systemd-shutdown[1]: sd-device-enumerator: Scanning /sys/bus
[ 513.521436] systemd-shutdown[1]: sd-device-enumerator: Scanning /sys/class
[ 513.534810] systemd-shutdown[1]: Stopping MD /dev/md126 (9:126).
[ 513.545384] systemd-shutdown[1]: Failed to sync MD block device /dev/md126, ignoring: Input/output error
[ 513.557265] md: md126 stopped.
[ 513.561451] systemd-shutdown[1]: Stopping MD /dev/md124p2 (259:5).
[ 513.576673] systemd-shutdown[1]: Could not stop MD /dev/md124p2: Device or resource busy
[ 513.589274] systemd-shutdown[1]: Stopping MD /dev/md124p1 (259:4).
[ 513.597976] systemd-shutdown[1]: Could not stop MD /dev/md124p1: Device or resource busy
[ 513.607263] systemd-shutdown[1]: Stopping MD /dev/md124 (9:124).
[ 513.615067] systemd-shutdown[1]: Could not stop MD /dev/md124: Device or resource busy
[ 513.625157] systemd-shutdown[1]: Not all MD devices stopped, 4 left.
[ 513.632209] systemd-shutdown[1]: Couldn't finalize remaining MD devices, trying again.
[ 513.641474] systemd-shutdown[1]: Failed to open watchdog device /dev/watchdog: No such file or directory
[ 513.653660] systemd-shutdown[1]: Stopping MD devices.
[ 513.661257] systemd-shutdown[1]: sd-device-enumerator: Scan all dirs
[ 513.668833] systemd-shutdown[1]: sd-device-enumerator: Scanning /sys/bus
[ 513.677347] systemd-shutdown[1]: sd-device-enumerator: Scanning /sys/class
[ 513.687047] systemd-shutdown[1]: Stopping MD /dev/md126 (9:126).
[ 513.697206] systemd-shutdown[1]: Failed to sync MD block device /dev/md126, ignoring: Input/output error
[ 513.707193] md: md126 stopped.
...

[ Test Plan ]

1. Build two VROC RAID. One RAID 0 for System volume, another RAID 10 for Data volume.
2. Install system on System volume.
3. Update systemd.
4. Reboot the system.
5. Verify if the system can reboot.

[ Where problems could occur ]

The patch confirmed fixed the reboot issue on the system with two VROC RAIDs but more than two VROC RAIDs and the combinations of RAID levels are not all tested. The patch itself adds logic to skip partitions and containers from the list of md devices to try and stop. Therefore any regressions would also be related to stopping md devices in systemd-shutdown.

[ Scope ]

Jammy

[Athos Ribeiro (athos-ribeiro)]

Hi Cyrus, thanks for the patch!

As per the SRU guidelines in https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template, regarding the "[ Where problems could occur ]" section,

> * This must '''never''' be "None" or "Low", or entirely an argument as to why
   your upload is low risk.

Describing where problems could occur and how they would manifest is a nice way to show to the SRU team that we did consider possible unwanted outcomes for the SRU and that we'd be ready to address them in case they do occur. It also helps people testing the SRU, guiding them on what to look for when checking for regressions.

[Cyrus Lien (cyruslien)]

Hi Athos,

Thanks for the review.
Please pause this SRU. There are currently issues that need clarification from Intel. I will reopen it and update the "[ Where problems could occur ]" section once the issue is clarified. Thank you.

[Simon Chopin (schopin)]

Hi,

Thanks for the patch. I took care of the LP metadata for the bug (targetting the proper series), and unsubscribe ubuntu-sponsors since it's not ready for upload yet. Please resubscribe ubuntu-sponsors once you feel it's ready for upload.

@enr0n can you chime in on this, do we have things pending for a Jammy SRU on systemd?

[Nick Rosbrook (enr0n)]

Yes, there are a couple low priority things pending for SRU on Jammy, so I would like to bundle those things eventually.

[Cyrus Lien (cyruslien)]

Hi,

The issue that pausing this SRU is going to be fixed in the kernel and the section "where problems could occur" updated.
Please resume this SRU, Thanks!

[Nick Rosbrook (enr0n)]

I am planning on doing an SRU for Jammy soon that will fix multiple bugs. I will make sure this is included.

[Lukas Märdian (slyon)]

Unsubscribing "ubuntu-sponsors", as this will be handled by the owning team.

[Nick Rosbrook (enr0n)]

Hi Cyrus,

Do you have an example of a system actually failing to shut down due to this? I don't see a problem with the patch per se, but I also am not convinced that without it a system just would not shutdown...

That code is basically a safety net. If I understand the code correctly, it should just ignore the failure of stopping some md devices, and continue on anyways. Certainly there would be log messages about this, but again, I don't see where this would prevent shutdown.

[Cyrus Lien (cyruslien)]

Without the patch, the systemd will forever keep trying to detach all md block devices but it always failed on "container" md devices.

[Cyrus Lien (cyruslien)]

With the patch, the partitions and container devices are filtered out, the system can successfully reboot.

[Cyrus Lien (cyruslien)]

The systemd PR: https://github.com/systemd/systemd/pull/22973/commits/50504f2e4ba17bbd58dd5cb707c70fe7c6f9d1f1

[Nick Rosbrook (enr0n)]

Thanks for the logs! That helps clarify things.

[Łukasz Zemczak (sil2100)]

Hello Cyrus, or anyone else affected,

Accepted systemd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/249.11-0ubuntu3.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Cyrus Lien (cyruslien)]

SUT information:

$ cat /proc/mdstat
Personalities : [raid0] [raid10] [linear] [multipath] [raid1] [raid6] [raid5] [raid4]
md124 : active raid10 nvme2n1[3] nvme3n1[2] nvme4n1[1] nvme5n1[0]
      500107264 blocks super external:/md127/0 64K chunks 2 near-copies [4/4] [UUUU]
      [>....................] resync = 2.6% (13288960/500107264) finish=39.3min speed=205974K/sec

md125 : active raid0 nvme0n1[1] nvme1n1[0]
      950198272 blocks super external:/md126/0 128k chunks

md126 : inactive nvme1n1[1](S) nvme0n1[0](S)
      10402 blocks super external:imsm

md127 : inactive nvme3n1[3](S) nvme5n1[2](S) nvme2n1[1](S) nvme4n1[0](S)
      20804 blocks super external:imsm

unused devices: <none>

$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
loop0 7:0 0 4K 1 loop /snap/bare/5
loop1 7:1 0 63.3M 1 loop /snap/core20/1879
loop2 7:2 0 73.1M 1 loop /snap/core22/634
loop3 7:3 0 242M 1 loop /snap/firefox/2667
loop4 7:4 0 349.7M 1 loop /snap/gnome-3-38-2004/140
loop5 7:5 0 460.6M 1 loop /snap/gnome-42-2204/102
loop6 7:6 0 91.7M 1 loop /snap/gtk-common-themes/1535
loop7 7:7 0 12.3M 1 loop /snap/snap-store/959
loop8 7:8 0 53.2M 1 loop /snap/snapd/19122
loop9 7:9 0 452K 1 loop /snap/snapd-desktop-integration/83
sr0 11:0 1 1024M 0 rom
nvme5n1 259:0 0 238.5G 0 disk
├─md124 9:124 0 476.9G 0 raid10
└─md127 9:127 0 0B 0 md
nvme4n1 259:1 0 238.5G 0 disk
├─md124 9:124 0 476.9G 0 raid10
└─md127 9:127 0 0B 0 md
nvme3n1 259:2 0 238.5G 0 disk
├─md124 9:124 0 476.9G 0 raid10
└─md127 9:127 0 0B 0 md
nvme2n1 259:3 0 238.5G 0 disk
├─md124 9:124 0 476.9G 0 raid10
└─md127 9:127 0 0B 0 md
nvme0n1 259:4 0 476.9G 0 disk
├─md125 9:125 0 906.2G 0 raid0
│ ├─md125p1 259:6 0 238.4M 0 part /boot/efi
│ ├─md125p2 259:7 0 7.6G 0 part
│ └─md125p3 259:8 0 898.3G 0 part /var/snap/firefox/common/host-hunspell
│ /
└─md126 9:126 0 0B 0 md
nvme1n1 259:5 0 476.9G 0 disk
├─md125 9:125 0 906.2G 0 raid0
│ ├─md125p1 259:6 0 238.4M 0 part /boot/efi
│ ├─md125p2 259:7 0 7.6G 0 part
│ └─md125p3 259:8 0 898.3G 0 part /var/snap/firefox/common/host-hunspell
│ /
└─md126 9:126 0 0B 0 md

[Cyrus Lien (cyruslien)]

With systemd-249.11-0ubuntu3.9, system failed to reboot.

[Cyrus Lien (cyruslien)]

With systemd-249.11-0ubuntu3.10, system reboot successfully.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (249.11-0ubuntu3.10) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

apt/2.4.10 (armhf)
casync/2+20201210-1build1 (ppc64el)
comitup/1.15-1 (armhf)
dbus/1.12.20-2ubuntu4.1 (armhf)
initramfs-tools/0.140ubuntu13.4 (s390x)
linux-azure-5.19/5.19.0-1027.30~22.04.2 (arm64)
linux-gcp-6.2/6.2.0-1011.11~22.04.3 (arm64)
linux-lowlatency/5.15.0-83.92 (arm64)
linux-lowlatency-hwe-5.19/5.19.0-1030.30 (arm64)
linux-nvidia-tegra/5.15.0-1016.16 (arm64)
linux-oracle-5.19/5.19.0-1027.30 (arm64)
mkosi/unknown (s390x)
munin/2.0.57-1ubuntu2 (armhf)
netplan.io/0.105-0ubuntu2~22.04.3 (arm64)
prometheus-postfix-exporter/unknown (s390x)
samba/2:4.15.13+dfsg-0ubuntu1.4 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 249.11-0ubuntu3.10

---------------
systemd (249.11-0ubuntu3.10) jammy; urgency=medium

  [ Nick Rosbrook ]
  * debian/tests/tests-in-lxd: use --reuse flag in lxc publish (LP: #2023229)
    File: debian/tests/tests-in-lxd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=85b2ceddff1a6cc1ddbca8a1b7e5381d146e6313
  * pstore: only try to load efi_pstore module (LP: #2023462)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7600bbfb1e8a399e5aeb1010a20deda3e5a06c89
  * shutdown: get only active md arrays. (LP: #2025563)
    File: debian/patches/lp2025563-shutdown-get-only-active-md-arrays.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=416a9245c8f0efbedcc4395cada23cb09c685ec3
  * udev-rules: fix nvme symlink creation on namespace changes (LP: #2028180)
    File: debian/patches/lp2028180-udev-rules-fix-nvme-symlink-creation-on-namespace-changes.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=26e85b944da9098e66fc0c39f64ee40254c0c278
  * core: reorder systemd arguments on reexec (LP: #2013543)
    File: debian/patches/lp2013543-core-reorder-systemd-arguments-on-reexec.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=19ba0f20d311642596dc65fa5d6eb96a2d4be280
  * network/dhcp4: do not ignore the gateway even if the destination is in same network (LP: #2009743)
    Files:
    - debian/patches/lp2009743/network-dhcp4-do-not-ignore-the-gateway-even-if-the-desti.patch
    - debian/patches/lp2009743/test-network-add-one-more-testcase-for-DHCPv4-classless-r.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=73e774a2fb99c82df6d0edd770bb84ab735ec2f0
  * Drop debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch (LP: #1982218)
    File: debian/patches/debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cf82f08feea456e1c65895b34bffa8c33d421588
  * debian/systemd.postint: do not daemon re-exec if we could hit LP: #2013543
    File: debian/systemd.postinst
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=be484dab06d590b1792a8f016f4292373d0174b7
  * binfmt: fix systemd-binfmt in LXD containers (LP: #1999275)
    Files:
    - debian/patches/lp1999275/binfmt-check-if-binfmt-is-mounted-before-applying-rules.patch
    - debian/patches/lp1999275/binfmt-util-also-check-if-binfmt-is-mounted-in-read-write.patch
    - debian/patches/lp1999275/binfmt-util-split-out-binfmt_mounted.patch
    - debian/patches/lp1999275/unit-check-more-specific-path-to-be-written-by-systemd-bi.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0b9eadf05ee6db0e009dacbc71521480095880b3

  [ Dan Streetman ]
  * Fix machinectl pull-tar and import-tar (LP: #1977630)
    Author: Dan Streetman
    File: debian/patches/lp1977630-fix_machinectl_pull_tar.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+sourc...

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Steve Langasek (vorlon)]

The release of this SRU has been rolled back in jammy because of bug #2035406 which appears to be a regression introduced in the new version of systemd.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 249.11-0ubuntu3.10

---------------
systemd (249.11-0ubuntu3.10) jammy; urgency=medium

  [ Nick Rosbrook ]
  * debian/tests/tests-in-lxd: use --reuse flag in lxc publish (LP: #2023229)
    File: debian/tests/tests-in-lxd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=85b2ceddff1a6cc1ddbca8a1b7e5381d146e6313
  * pstore: only try to load efi_pstore module (LP: #2023462)
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7600bbfb1e8a399e5aeb1010a20deda3e5a06c89
  * shutdown: get only active md arrays. (LP: #2025563)
    File: debian/patches/lp2025563-shutdown-get-only-active-md-arrays.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=416a9245c8f0efbedcc4395cada23cb09c685ec3
  * udev-rules: fix nvme symlink creation on namespace changes (LP: #2028180)
    File: debian/patches/lp2028180-udev-rules-fix-nvme-symlink-creation-on-namespace-changes.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=26e85b944da9098e66fc0c39f64ee40254c0c278
  * core: reorder systemd arguments on reexec (LP: #2013543)
    File: debian/patches/lp2013543-core-reorder-systemd-arguments-on-reexec.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=19ba0f20d311642596dc65fa5d6eb96a2d4be280
  * network/dhcp4: do not ignore the gateway even if the destination is in same network (LP: #2009743)
    Files:
    - debian/patches/lp2009743/network-dhcp4-do-not-ignore-the-gateway-even-if-the-desti.patch
    - debian/patches/lp2009743/test-network-add-one-more-testcase-for-DHCPv4-classless-r.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=73e774a2fb99c82df6d0edd770bb84ab735ec2f0
  * Drop debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch (LP: #1982218)
    File: debian/patches/debian/UBUNTU-wait-online-exit-if-no-links-are-managed.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cf82f08feea456e1c65895b34bffa8c33d421588
  * debian/systemd.postint: do not daemon re-exec if we could hit LP: #2013543
    File: debian/systemd.postinst
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=be484dab06d590b1792a8f016f4292373d0174b7
  * binfmt: fix systemd-binfmt in LXD containers (LP: #1999275)
    Files:
    - debian/patches/lp1999275/binfmt-check-if-binfmt-is-mounted-before-applying-rules.patch
    - debian/patches/lp1999275/binfmt-util-also-check-if-binfmt-is-mounted-in-read-write.patch
    - debian/patches/lp1999275/binfmt-util-split-out-binfmt_mounted.patch
    - debian/patches/lp1999275/unit-check-more-specific-path-to-be-written-by-systemd-bi.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0b9eadf05ee6db0e009dacbc71521480095880b3

  [ Dan Streetman ]
  * Fix machinectl pull-tar and import-tar (LP: #1977630)
    Author: Dan Streetman
    File: debian/patches/lp1977630-fix_machinectl_pull_tar.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+sourc...