storage / luks / dmsetup regressed (or got better) on ppc64le

[Dimitri John Ledkov]

in disco proposed with new systemd and v4.19 kernel it appears that dmsetup / cryptsetup storage either got better or worse.

Devices take very long to activate, and sometimes remain in use during test clean up.

This leads to udisks autopkgtest failing on ppc64le and systemd's "storage" autopkgtest is also failing.

I've tried to make ppc64le test more resilient, but it's still odd that it became unstable in disco, and used to be rock solid on ppc64le.

--
sru template for systemd upload:

[impact]
buffer overflow can cause memory corruption; this is seen in failed autopkgtests

[test case]
see comment 6

[regression potential]
the patch is minimal and clearly correct; however the regression potential is around invalid/corrupted keys read from the keyring.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1814373

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Sebastien Bacher (seb128)]

Laney got an udisks fix proposed/accepted upstream on https://github.com/storaged-project/udisks/pull/630

[Dimitri John Ledkov (xnox)]

got a few more tweaks to the storage test, and it's now more consistently passing on all other architectures but ppc64le.

[Thadeu Lima de Souza Cascardo (cascardo)]

With systemd storage test, I got two different situations:

Also on amd64, the test is still flaky, it needs to wait for the unit to be active before stopping it, otherwise it's canceled. I will send a comment with a preview of a fix.

On ppc64el, however, even after that fix, the patch will fail once in a while, with /lib/systemd/systemd-cryptsetup crashing on an invalid free. I started investigating that, but still can't figure what the real problem is. Maybe some corruption going on, but really hard to catch when, as running under valgrind doesn't allow the bug to be reproduced.

Cascardo.

[Thadeu Lima de Souza Cascardo (cascardo)]

Still need to fix it so it will exit when unit is failed, and maybe after some timeout.

---
 debian/tests/storage | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/debian/tests/storage b/debian/tests/storage
index 04d11c8..47fbe81 100755
--- a/debian/tests/storage
+++ b/debian/tests/storage
@@ -73,6 +73,8 @@ class CryptsetupTest(FakeDriveTestBase):
         subprocess.call(['umount', self.plaintext_dev], stderr=subprocess.DEVNULL)
         subprocess.call(['systemctl', 'start', '--no-ask-password', 'systemd-cryptsetup@%s.service' % self.plaintext_name],
                         stderr=subprocess.STDOUT)
+ while subprocess.call(['systemctl', 'is-active', 'systemd-cryptsetup@%s.service' % self.plaintext_name], stderr=subprocess.STDOUT) != 0:
+ pass
         subprocess.call(['systemctl', 'stop', 'systemd-cryptsetup@%s.service' % self.plaintext_name],
                         stderr=subprocess.STDOUT)
         if os.path.exists('/etc/crypttab'):
--

[Thadeu Lima de Souza Cascardo (cascardo)]

So, I tested with a 4.18 kernel and I could still reproduce the issue on eoan. Then, I tested on a bionic system, 4.15 kernel, systemd 237, glibc 2.27, and I see a failure on malloc, also memory corruption.

This appears on the backtrace.
3738 malloc_printerr ("malloc(): memory corruption");

I am going back to xenial.

Cascardo.

[Thadeu Lima de Souza Cascardo (cascardo)]

It goes back to xenial. I think I found the cause on systemd, will test a fix and report back soon.

[Thadeu Lima de Souza Cascardo (cascardo)]

After applying the two fixes, I managed to get the test running on a loop for more than 24 hours on disco. Will review the case with someone on the team before attaching the fix.

[Thadeu Lima de Souza Cascardo (cascardo)]

https://lists.freedesktop.org/archives/systemd-devel/2019-May/042570.html

[Dan Streetman (ddstreet)]

https://github.com/systemd/systemd/pull/12566

thnx @cascardo!

[Timo Aaltonen (tjaalton)]

Hello Dimitri, or anyone else affected,

Accepted systemd into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/240-6ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Dimitri, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Dimitri, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dan Streetman (ddstreet)]

autopkgtest failures for this upload analyzed in bug 1825997

[Dan Streetman (ddstreet)]

systemd 'storage' test case pass on all releases for ppc64el except disco due to bug 1831459, which i'll queue up to fix in the next upload.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 240-6ubuntu9

---------------
systemd (240-6ubuntu9) eoan; urgency=medium

  * Fix typpo in storage test.
    File: debian/tests/storage
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f28aa5fe4ab175b99b6ea702559c59ca473b4ca8

  * Fix bashism
    File: debian/extra/dhclient-enter-resolved-hook
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0725c1169ddde4f41cacba7af3e546704e2206be

systemd (240-6ubuntu8) eoan; urgency=medium

  * Only restart resolved on changes in dhclient enter hook.
    This prevents spurious restarts of resolved on rebounds when
    the addresses did not change. (LP: #1805183)
    Author: Julian Andres Klode
    File: debian/extra/dhclient-enter-resolved-hook
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=258893bae8cbb12670e4807636fe8f7e9fb5407a

  * Wait for cryptsetup unit to start, before stopping.
    Patch from cascardo. Plus small refactor for readability. (LP: #1814373)
    File: debian/tests/storage
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b65aa350be7e61c65927fbc0921a750fcfaa51cd

  * Wait for systemctl is-system-running state.
    File: debian/tests/boot-smoke
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=776998f1f55c445b6e385cab69a4219c42d00838

systemd (240-6ubuntu7) eoan; urgency=medium

  * Revert "Add check to switch VTs only between K_XLATE or K_UNICODE"
    This reverts commit 60407728a1a453104e3975ecfdf25a254dd7cc44.
    Files:
    - debian/patches/Add-check-to-switch-VTs-only-between-K_XLATE-or-K_UNICODE.patch
    - debian/patches/Move-verify_vc_kbmode-to-terminal-util.c-as-vt_verify_kbm.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=18029ab5ff436bfb3b401f24cd1e3a4cf2a1579c

  * Cherrypick missing systemd-stable patches to unbreak wireguard peer endpoints.
    Signed-off-by: Dimitri John Ledkov <email address hidden> (LP: #1825378)
    Author: Dan Streetman
    Files:
    - debian/patches/network-wireguard-fixes-sending-wireguard-peer-setti.patch
    - debian/patches/network-wireguard-use-sd_netlink_message_append_sock.patch
    - debian/patches/sd-netlink-introduce-sd_netlink_message_append_socka.patch
    - debian/patches/test-network-add-more-checks-in-NetworkdNetDevTests..patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4046f515e40c4dc80d18d2303466737f1f451f11

  * Remove expected failure from passing test.
    Signed-off-by: Dimitri John Ledkov <email address hidden> (LP: #1829450)
    Author: Dan Streetman
    File: debian/tests/systemd-fsckd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=c43b12037d08555dc1d26593307726d7c7992df0

  * Fix false negative checking for running jobs after boot.
    Signed-off-by: Dimitri John Ledkov <email address hidden> (LP: #1825997)
    Author: Dan Streetman
    File: debian/tests/boot-smoke
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=aeb01631efbaf3fe851dee15d496e0b66b5c347f

  * Cherrypick ask-password: prevent buffer ...

[Dimitri John Ledkov (xnox)]

Well, it's "better" but still failing.

http://autopkgtest.ubuntu.com/packages/systemd/eoan/ppc64el

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 240-6ubuntu5.1

---------------
systemd (240-6ubuntu5.1) disco; urgency=medium

  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)
  * d/p/network-wireguard-fixes-sending-wireguard-peer-setti.patch,
    d/p/test-network-add-more-checks-in-NetworkdNetDevTests..patch,
    d/p/sd-netlink-introduce-sd_netlink_message_append_socka.patch,
    d/p/network-wireguard-use-sd_netlink_message_append_sock.patch:
    - systemd doesn't set wireguard peer endpoint (LP: #1825378)
  * d/t/boot-smoke:
    - Fix false negative checking for running jobs after boot
      (LP: #1825997)

 -- Dan Streetman <email address hidden> Thu, 16 May 2019 06:07:49 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 239-7ubuntu10.14

---------------
systemd (239-7ubuntu10.14) cosmic; urgency=medium

  * d/p/resolved-rework-how-we-determine-which-scope-to-send.patch
    - fix DNS leakage (LP: 1754671)
  * d/t/boot-and-services:
    - skip test_no_failed if gdm failed to start (LP: #1830484)
  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)
  * d/t/boot-smoke:
    - Fix false negative checking for running jobs after boot
      (LP: #1825997)

 -- Dan Streetman <email address hidden> Wed, 24 Apr 2019 17:08:26 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.22

---------------
systemd (237-3ubuntu10.22) bionic; urgency=medium

  * d/p/resolved-rework-how-we-determine-which-scope-to-send.patch
    - fix DNS leakage (LP: 1754671)
  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)
  * d/t/boot-smoke:
    - Fix false negative checking for running jobs after boot
      (LP: #1825997)

 -- Dan Streetman <email address hidden> Wed, 24 Apr 2019 17:15:36 -0400

[Łukasz Zemczak (sil2100)]

Hello Dimitri, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

All autopkgtests for the newly accepted systemd (229-4ubuntu21.22) for xenial have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#xenial

[Dan Streetman (ddstreet)]

re: autopkgtests, a few failed, but I re-ran them and they passed on the re-run.

[Dan Streetman (ddstreet)]

storage test passed for all archs

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu21.22

---------------
systemd (229-4ubuntu21.22) xenial; urgency=medium

  [ Dan Streetman ]
  * d/t/systemd-fsckd, d/t/cmdline-upstart-boot:
    - skip on s390x; requires grub (LP: #1830477)
  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)

  [ Dimitri John Ledkov ]
  * Specify Ubuntu's Vcs-Git

  [ Balint Reczey ]
  * Append /snap/bin to default PATH.
    Snapd ships snapd-env-generator, but systemd does not not support
    environment generators. Hard-coding /snap/bin is less risky than
    backporting environment generator support and since snaps are considered
    to be first class packages on Ubuntu /snap/bin can safely added to
    the default PATH. (LP: #1771858)

  [ Ioanna Alifieraki ]
  * d/p/systemctl-Replace-check_one_unit-by-get_state_one_un.patch
    - Backport upstream PR#2768 needed for next patch
  * d/p/systemctl-load-unit-if-needed-in-systemctl-is-active.patch
    - Backport upstream PR#7997 to fix alias service reports inactive while
      aliased is active (LP: #1828892)

 -- Dan Streetman <email address hidden> Wed, 24 Apr 2019 17:15:36 -0400

[Brian Murray (brian-murray)]

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release