boot-smoke fails due to running jobs

Bug #1825997 reported by Dan Streetman on 2019-04-23
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Status tracked in Eoan
Xenial
Undecided
Unassigned
Bionic
Low
Dan Streetman
Cosmic
Low
Dan Streetman
Disco
Low
Dan Streetman
Eoan
Low
Dan Streetman

Bug Description

[impact]

boot-smoke test reboots 5 times and verifies systemd is fully started up after each boot, including checking if there are any running jobs (with list-jobs). However, this test makes the assumption that no further jobs will be started after systemd reaches 'running' (or 'degraded') state, which is a false assumption.

[test case]

see various boot-smoke failures in autopkgtest.ubuntu.com

[regression potential]

possible false-positive or false-negative autopkgtest results.

[other info]

Note: This patch is not required for debian, because debian's boot-smoke does not include the wait for systemctl is-system-running.

The problem appears to be that systemd reaches 'running' (or 'degraded') state, and then other systemd services are started. This confuses the boot-smoke test, because it sees that 'is-system-running' is done, but then it sees running jobs, which fails the test.

What is starting jobs after systemd reaches running state appears to be X inside the test system. There are various services started by gnome-session and dbus-daemon. Additionally, from the artifacts of one example:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/i386/s/systemd/20190416_171327_478f6@/artifacts.tar.gz

the artifacts/journal.txt shows that after the boot-smoke test causes the reboot and then re-ssh into the system after the reboot, it only gives the test system 9 seconds before deciding it has failed, and only 4 seconds after ssh'ing into the rebooted test system.

Another wait is needed when checking for remaining running jobs. Or, the running jobs check could be removed entirely, and we can just trust that systemd correctly knows when it has reached running|degraded state.

Related branches

Dan Streetman (ddstreet) on 2019-04-23
Changed in systemd (Ubuntu Eoan):
status: New → In Progress
Changed in systemd (Ubuntu Disco):
status: New → In Progress
Changed in systemd (Ubuntu Cosmic):
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
Changed in systemd (Ubuntu Eoan):
importance: Undecided → Low
Changed in systemd (Ubuntu Disco):
importance: Undecided → Low
Changed in systemd (Ubuntu Cosmic):
importance: Undecided → Low
Changed in systemd (Ubuntu Bionic):
importance: Undecided → Low
Changed in systemd (Ubuntu Eoan):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Disco):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Cosmic):
assignee: nobody → Dan Streetman (ddstreet)
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
Dan Streetman (ddstreet) on 2019-04-23
summary: - boot-smoke test timeout too short for overloaded autopkgtest.ubuntu.com
+ boot-smoke running jobs test based on bad assumption
summary: - boot-smoke running jobs test based on bad assumption
+ boot-smoke fails due to running jobs
Dan Streetman (ddstreet) on 2019-04-23
description: updated
Dan Streetman (ddstreet) on 2019-04-23
description: updated
Dan Streetman (ddstreet) on 2019-04-23
description: updated
Dan Streetman (ddstreet) on 2019-04-23
tags: added: ddstreet-next
Dan Streetman (ddstreet) on 2019-05-02
Changed in systemd (Ubuntu Eoan):
assignee: Dan Streetman (ddstreet) → nobody
Dan Streetman (ddstreet) on 2019-05-14
Changed in systemd (Ubuntu Eoan):
assignee: nobody → Dan Streetman (ddstreet)
Dan Streetman (ddstreet) on 2019-05-17
Changed in systemd (Ubuntu Xenial):
status: New → Invalid
Dan Streetman (ddstreet) on 2019-05-29
description: updated
Changed in systemd (Ubuntu Eoan):
status: In Progress → Fix Committed
Dan Streetman (ddstreet) on 2019-05-30
tags: removed: ddstreet-next

Hello Dan, or anyone else affected,

Accepted systemd into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/240-6ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco
Changed in systemd (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed-cosmic
Timo Aaltonen (tjaalton) wrote :

Hello Dan, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Timo Aaltonen (tjaalton) wrote :

Hello Dan, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Dan Streetman (ddstreet) wrote :

boot-smoke passed for all releases/archs from this upload

tags: added: verification-done verification-done-bionic verification-done-cosmic verification-done-disco
removed: verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco
Dan Streetman (ddstreet) wrote :

autopkgtest failures:

bionic:

linux-snapdragon, linux-gke-4.15, linux-aws-edge: these have never passed; ignore.

gvfs (ppc64el and i386): bug 1713098 (failure matches report from that bug)

systemd (ppc64el): only test failure is due to bug 1821625 (which requires kernel patch)

systemd (i386): test failed twice due to bug 1829829, re-running test, but impossible to tell when it will pass or fail due to this bug, so may need to just ignore this failure

Dan Streetman (ddstreet) wrote :

autopkgtest failures:

cosmic:

linux-oracle: tests never passed; ignore.

umockdev: I opened bug 1831467, and i'm re-running the test; it's an intermittent failure so hopefully it will pass the next time, but even if not i believe we can ignore its failure as it seems to be a test case problem of expecting very-accurate ms-level usleep() timing, as explained in the bug.

Dan Streetman (ddstreet) wrote :

autopkgtest failures:

disco:

linux-oracle, linux-snapdragon: tests never passed; ignore

dbus: this appears to be the same issue as is seen for systemd in bug 1829829; i'm re-running the test (again), but if it still doesn't pass I think we need to ignore it as a failure due to that bug.

systemd (amd64): only failure is due to bug 1831459

systemd (ppc64el): also fails due to bug 1831459, and also has some upstream failed tests:
FAILED TESTS: test/TEST-18-FAILUREACTION test/TEST-19-DELEGATE test/TEST-25-IMPORT test/TEST-27-STDOUTFILE
I opened bug 1831468 to track these failures; they do not appear related/caused-by this upload.

corosync (all archs): this has failed since before this upload, and looks like problem with the corosync source not being available in disco, possibly due to using disco-proposed. in any case, not caused or related to this upload and can be ignored.

Dan Streetman (ddstreet) wrote :

note on the corosync autopkgtest failure in disco; i opened bug 1831492 and the failure can be ignored for this upload.

Launchpad Janitor (janitor) wrote :
Download full text (3.5 KiB)

This bug was fixed in the package systemd - 240-6ubuntu9

---------------
systemd (240-6ubuntu9) eoan; urgency=medium

  * Fix typpo in storage test.
    File: debian/tests/storage
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f28aa5fe4ab175b99b6ea702559c59ca473b4ca8

  * Fix bashism
    File: debian/extra/dhclient-enter-resolved-hook
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=0725c1169ddde4f41cacba7af3e546704e2206be

systemd (240-6ubuntu8) eoan; urgency=medium

  * Only restart resolved on changes in dhclient enter hook.
    This prevents spurious restarts of resolved on rebounds when
    the addresses did not change. (LP: #1805183)
    Author: Julian Andres Klode
    File: debian/extra/dhclient-enter-resolved-hook
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=258893bae8cbb12670e4807636fe8f7e9fb5407a

  * Wait for cryptsetup unit to start, before stopping.
    Patch from cascardo. Plus small refactor for readability. (LP: #1814373)
    File: debian/tests/storage
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b65aa350be7e61c65927fbc0921a750fcfaa51cd

  * Wait for systemctl is-system-running state.
    File: debian/tests/boot-smoke
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=776998f1f55c445b6e385cab69a4219c42d00838

systemd (240-6ubuntu7) eoan; urgency=medium

  * Revert "Add check to switch VTs only between K_XLATE or K_UNICODE"
    This reverts commit 60407728a1a453104e3975ecfdf25a254dd7cc44.
    Files:
    - debian/patches/Add-check-to-switch-VTs-only-between-K_XLATE-or-K_UNICODE.patch
    - debian/patches/Move-verify_vc_kbmode-to-terminal-util.c-as-vt_verify_kbm.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=18029ab5ff436bfb3b401f24cd1e3a4cf2a1579c

  * Cherrypick missing systemd-stable patches to unbreak wireguard peer endpoints.
    Signed-off-by: Dimitri John Ledkov <email address hidden> (LP: #1825378)
    Author: Dan Streetman
    Files:
    - debian/patches/network-wireguard-fixes-sending-wireguard-peer-setti.patch
    - debian/patches/network-wireguard-use-sd_netlink_message_append_sock.patch
    - debian/patches/sd-netlink-introduce-sd_netlink_message_append_socka.patch
    - debian/patches/test-network-add-more-checks-in-NetworkdNetDevTests..patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4046f515e40c4dc80d18d2303466737f1f451f11

  * Remove expected failure from passing test.
    Signed-off-by: Dimitri John Ledkov <email address hidden> (LP: #1829450)
    Author: Dan Streetman
    File: debian/tests/systemd-fsckd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=c43b12037d08555dc1d26593307726d7c7992df0

  * Fix false negative checking for running jobs after boot.
    Signed-off-by: Dimitri John Ledkov <email address hidden> (LP: #1825997)
    Author: Dan Streetman
    File: debian/tests/boot-smoke
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=aeb01631efbaf3fe851dee15d496e0b66b5c347f

  * Cherrypick ask-password: prevent buffer ...

Read more...

Changed in systemd (Ubuntu Eoan):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 240-6ubuntu5.1

---------------
systemd (240-6ubuntu5.1) disco; urgency=medium

  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)
  * d/p/network-wireguard-fixes-sending-wireguard-peer-setti.patch,
    d/p/test-network-add-more-checks-in-NetworkdNetDevTests..patch,
    d/p/sd-netlink-introduce-sd_netlink_message_append_socka.patch,
    d/p/network-wireguard-use-sd_netlink_message_append_sock.patch:
    - systemd doesn't set wireguard peer endpoint (LP: #1825378)
  * d/t/boot-smoke:
    - Fix false negative checking for running jobs after boot
      (LP: #1825997)

 -- Dan Streetman <email address hidden> Thu, 16 May 2019 06:07:49 -0400

Changed in systemd (Ubuntu Disco):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 239-7ubuntu10.14

---------------
systemd (239-7ubuntu10.14) cosmic; urgency=medium

  * d/p/resolved-rework-how-we-determine-which-scope-to-send.patch
    - fix DNS leakage (LP: 1754671)
  * d/t/boot-and-services:
    - skip test_no_failed if gdm failed to start (LP: #1830484)
  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)
  * d/t/boot-smoke:
    - Fix false negative checking for running jobs after boot
      (LP: #1825997)

 -- Dan Streetman <email address hidden> Wed, 24 Apr 2019 17:08:26 -0400

Changed in systemd (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 237-3ubuntu10.22

---------------
systemd (237-3ubuntu10.22) bionic; urgency=medium

  * d/p/resolved-rework-how-we-determine-which-scope-to-send.patch
    - fix DNS leakage (LP: 1754671)
  * d/p/ask-password-prevent-buffer-overrow-when-reading-fro.patch:
    - prevent buffer overflow when reading keyring (LP: #1814373)
  * d/t/boot-smoke:
    - Fix false negative checking for running jobs after boot
      (LP: #1825997)

 -- Dan Streetman <email address hidden> Wed, 24 Apr 2019 17:15:36 -0400

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers