storage / luks / dmsetup regressed (or got better) on ppc64le

Bug #1814373 reported by Dimitri John Ledkov on 2019-02-02
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Status tracked in Eoan
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned
Disco
Undecided
Unassigned
Eoan
Undecided
Unassigned
systemd (Debian)
Fix Released
Unknown
systemd (Ubuntu)
Status tracked in Eoan
Xenial
Undecided
Unassigned
Bionic
Medium
Dan Streetman
Cosmic
Medium
Dan Streetman
Disco
Medium
Dan Streetman
Eoan
Medium
Unassigned

Bug Description

in disco proposed with new systemd and v4.19 kernel it appears that dmsetup / cryptsetup storage either got better or worse.

Devices take very long to activate, and sometimes remain in use during test clean up.

This leads to udisks autopkgtest failing on ppc64le and systemd's "storage" autopkgtest is also failing.

I've tried to make ppc64le test more resilient, but it's still odd that it became unstable in disco, and used to be rock solid on ppc64le.

--
sru template for systemd upload:

[impact]
buffer overflow can cause memory corruption; this is seen in failed autopkgtests

[test case]
see comment 6

[regression potential]
the patch is minimal and clearly correct; however the regression potential is around invalid/corrupted keys read from the keyring.

Related branches

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1814373

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
tags: added: bot-stop-nagging
affects: udisks (Ubuntu) → udisks2 (Ubuntu)
Sebastien Bacher (seb128) wrote :

Laney got an udisks fix proposed/accepted upstream on https://github.com/storaged-project/udisks/pull/630

Dimitri John Ledkov (xnox) wrote :

got a few more tweaks to the storage test, and it's now more consistently passing on all other architectures but ppc64le.

With systemd storage test, I got two different situations:

Also on amd64, the test is still flaky, it needs to wait for the unit to be active before stopping it, otherwise it's canceled. I will send a comment with a preview of a fix.

On ppc64el, however, even after that fix, the patch will fail once in a while, with /lib/systemd/systemd-cryptsetup crashing on an invalid free. I started investigating that, but still can't figure what the real problem is. Maybe some corruption going on, but really hard to catch when, as running under valgrind doesn't allow the bug to be reproduced.

Cascardo.

Still need to fix it so it will exit when unit is failed, and maybe after some timeout.

---
 debian/tests/storage | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/debian/tests/storage b/debian/tests/storage
index 04d11c8..47fbe81 100755
--- a/debian/tests/storage
+++ b/debian/tests/storage
@@ -73,6 +73,8 @@ class CryptsetupTest(FakeDriveTestBase):
         subprocess.call(['umount', self.plaintext_dev], stderr=subprocess.DEVNULL)
         subprocess.call(['systemctl', 'start', '--no-ask-password', 'systemd-cryptsetup@%s.service' % self.plaintext_name],
                         stderr=subprocess.STDOUT)
+ while subprocess.call(['systemctl', 'is-active', 'systemd-cryptsetup@%s.service' % self.plaintext_name], stderr=subprocess.STDOUT) != 0:
+ pass
         subprocess.call(['systemctl', 'stop', 'systemd-cryptsetup@%s.service' % self.plaintext_name],
                         stderr=subprocess.STDOUT)
         if os.path.exists('/etc/crypttab'):
--

So, I tested with a 4.18 kernel and I could still reproduce the issue on eoan. Then, I tested on a bionic system, 4.15 kernel, systemd 237, glibc 2.27, and I see a failure on malloc, also memory corruption.

This appears on the backtrace.
3738 malloc_printerr ("malloc(): memory corruption");

I am going back to xenial.

Cascardo.

It goes back to xenial. I think I found the cause on systemd, will test a fix and report back soon.

After applying the two fixes, I managed to get the test running on a loop for more than 24 hours on disco. Will review the case with someone on the team before attaching the fix.

Changed in udisks2 (Ubuntu Eoan):
importance: Undecided → High
Changed in udisks2 (Ubuntu Bionic):
status: New → Invalid
Changed in udisks2 (Ubuntu Disco):
status: New → Invalid
Changed in udisks2 (Ubuntu Eoan):
status: New → Invalid
Dan Streetman (ddstreet) on 2019-05-16
Changed in systemd (Ubuntu Eoan):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Medium
status: New → In Progress
Changed in systemd (Ubuntu Disco):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Medium
status: New → In Progress
Changed in systemd (Ubuntu Cosmic):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Medium
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → Medium
status: New → In Progress
tags: added: ddstreet-next
Changed in udisks2 (Ubuntu Cosmic):
status: New → Invalid
Changed in systemd (Ubuntu Eoan):
status: In Progress → Fix Committed
Dan Streetman (ddstreet) on 2019-05-30
tags: removed: ddstreet-next
description: updated

Hello Dimitri, or anyone else affected,

Accepted systemd into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/240-6ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco
Changed in systemd (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed-cosmic
Timo Aaltonen (tjaalton) wrote :

Hello Dimitri, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Timo Aaltonen (tjaalton) wrote :

Hello Dimitri, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Changed in systemd (Debian):
status: Unknown → New
no longer affects: udisks2 (Ubuntu)
no longer affects: udisks2 (Ubuntu Bionic)
no longer affects: udisks2 (Ubuntu Cosmic)
no longer affects: udisks2 (Ubuntu Disco)
no longer affects: udisks2 (Ubuntu Eoan)
Dan Streetman (ddstreet) wrote :

autopkgtest failures for this upload analyzed in bug 1825997

Dan Streetman (ddstreet) wrote :

systemd 'storage' test case pass on all releases for ppc64el except disco due to bug 1831459, which i'll queue up to fix in the next upload.

tags: added: verification-done verification-done-bionic verification-done-cosmic verification-done-disco
removed: verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco
Launchpad Janitor (janitor) wrote :
Download full text (3.5 KiB)

This bug was fixed in the package systemd - 240-6ubuntu9

---------------
systemd (240-6ubuntu9) eoan; urgency=medium

  * Fix typpo in storage test.
    File: debian/tests/storage
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f28aa5fe4ab175b99b6ea702559c59ca473b4ca8

  * Fix bashism
    File: debian/extra/dhclient-enter-resolved-hook
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0725c1169ddde4f41cacba7af3e546704e2206be

systemd (240-6ubuntu8) eoan; urgency=medium

  * Only restart resolved on changes in dhclient enter hook.
    This prevents spurious restarts of resolved on rebounds when
    the addresses did not change. (LP: #1805183)
    Author: Julian Andres Klode
    File: debian/extra/dhclient-enter-resolved-hook
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=258893bae8cbb12670e4807636fe8f7e9fb5407a

  * Wait for cryptsetup unit to start, before stopping.
    Patch from cascardo. Plus small refactor for readability. (LP: #1814373)
    File: debian/tests/storage
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b65aa350be7e61c65927fbc0921a750fcfaa51cd

  * Wait for systemctl is-system-running state.
    File: debian/tests/boot-smoke
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=776998f1f55c445b6e385cab69a4219c42d00838

systemd (240-6ubuntu7) eoan; urgency=medium

  * Revert "Add check to switch VTs only between K_XLATE or K_UNICODE"
    This reverts commit 60407728a1a453104e3975ecfdf25a254dd7cc44.
    Files:
    - debian/patches/Add-check-to-switch-VTs-only-between-K_XLATE-or-K_UNICODE.patch
    - debian/patches/Move-verify_vc_kbmode-to-terminal-util.c-as-vt_verify_kbm.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=18029ab5ff436bfb3b401f24cd1e3a4cf2a1579c

  * Cherrypick missing systemd-stable patches to unbreak wireguard peer endpoints.
    Signed-off-by: Dimitri John Ledkov <email address hidden> (LP: #1825378)
    Author: Dan Streetman
    Files:
    - debian/patches/network-wireguard-fixes-sending-wireguard-peer-setti.patch
    - debian/patches/network-wireguard-use-sd_netlink_message_append_sock.patch
    - debian/patches/sd-netlink-introduce-sd_netlink_message_append_socka.patch
    - debian/patches/test-network-add-more-checks-in-NetworkdNetDevTests..patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4046f515e40c4dc80d18d2303466737f1f451f11

  * Remove expected failure from passing test.
    Signed-off-by: Dimitri John Ledkov <email address hidden> (LP: #1829450)
    Author: Dan Streetman
    File: debian/tests/systemd-fsckd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=c43b12037d08555dc1d26593307726d7c7992df0

  * Fix false negative checking for running jobs after boot.
    Signed-off-by: Dimitri John Ledkov <email address hidden> (LP: #1825997)
    Author: Dan Streetman
    File: debian/tests/boot-smoke
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=aeb01631efbaf3fe851dee15d496e0b66b5c347f

  * Cherrypick ask-password: prevent buffer ...

Read more...

Changed in systemd (Ubuntu Eoan):
status: Fix Committed → Fix Released
Dimitri John Ledkov (xnox) wrote :
Changed in systemd (Ubuntu Eoan):
status: Fix Released → Confirmed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 240-6ubuntu5.1

---------------
systemd (240-6ubuntu5.1) disco; urgency=medium

  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)
  * d/p/network-wireguard-fixes-sending-wireguard-peer-setti.patch,
    d/p/test-network-add-more-checks-in-NetworkdNetDevTests..patch,
    d/p/sd-netlink-introduce-sd_netlink_message_append_socka.patch,
    d/p/network-wireguard-use-sd_netlink_message_append_sock.patch:
    - systemd doesn't set wireguard peer endpoint (LP: #1825378)
  * d/t/boot-smoke:
    - Fix false negative checking for running jobs after boot
      (LP: #1825997)

 -- Dan Streetman <email address hidden> Thu, 16 May 2019 06:07:49 -0400

Changed in systemd (Ubuntu Disco):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 239-7ubuntu10.14

---------------
systemd (239-7ubuntu10.14) cosmic; urgency=medium

  * d/p/resolved-rework-how-we-determine-which-scope-to-send.patch
    - fix DNS leakage (LP: 1754671)
  * d/t/boot-and-services:
    - skip test_no_failed if gdm failed to start (LP: #1830484)
  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)
  * d/t/boot-smoke:
    - Fix false negative checking for running jobs after boot
      (LP: #1825997)

 -- Dan Streetman <email address hidden> Wed, 24 Apr 2019 17:08:26 -0400

Changed in systemd (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 237-3ubuntu10.22

---------------
systemd (237-3ubuntu10.22) bionic; urgency=medium

  * d/p/resolved-rework-how-we-determine-which-scope-to-send.patch
    - fix DNS leakage (LP: 1754671)
  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)
  * d/t/boot-smoke:
    - Fix false negative checking for running jobs after boot
      (LP: #1825997)

 -- Dan Streetman <email address hidden> Wed, 24 Apr 2019 17:15:36 -0400

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in systemd (Debian):
status: New → Confirmed

Hello Dimitri, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial
removed: verification-done

All autopkgtests for the newly accepted systemd (229-4ubuntu21.22) for xenial have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#xenial

Changed in systemd (Debian):
status: Confirmed → Fix Committed
Dan Streetman (ddstreet) wrote :

re: autopkgtests, a few failed, but I re-ran them and they passed on the re-run.

Dan Streetman (ddstreet) wrote :

storage test passed for all archs

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 229-4ubuntu21.22

---------------
systemd (229-4ubuntu21.22) xenial; urgency=medium

  [ Dan Streetman ]
  * d/t/systemd-fsckd, d/t/cmdline-upstart-boot:
    - skip on s390x; requires grub (LP: #1830477)
  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)

  [ Dimitri John Ledkov ]
  * Specify Ubuntu's Vcs-Git

  [ Balint Reczey ]
  * Append /snap/bin to default PATH.
    Snapd ships snapd-env-generator, but systemd does not not support
    environment generators. Hard-coding /snap/bin is less risky than
    backporting environment generator support and since snaps are considered
    to be first class packages on Ubuntu /snap/bin can safely added to
    the default PATH. (LP: #1771858)

  [ Ioanna Alifieraki ]
  * d/p/systemctl-Replace-check_one_unit-by-get_state_one_un.patch
    - Backport upstream PR#2768 needed for next patch
  * d/p/systemctl-load-unit-if-needed-in-systemctl-is-active.patch
    - Backport upstream PR#7997 to fix alias service reports inactive while
      aliased is active (LP: #1828892)

 -- Dan Streetman <email address hidden> Wed, 24 Apr 2019 17:15:36 -0400

Changed in systemd (Ubuntu Xenial):
status: Fix Committed → Fix Released
Dan Streetman (ddstreet) on 2019-08-30
Changed in systemd (Ubuntu Eoan):
assignee: Dan Streetman (ddstreet) → nobody
Changed in systemd (Debian):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.