kernel panic on booting kernel 4.4 based d-i from 16.04.5 RC image

Bug #1784454 reported by Frank Heimes on 2018-07-30
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Critical
Unassigned
debian-installer (Ubuntu)
Critical
Canonical Foundations Team
linux (Ubuntu)
Critical
Canonical Kernel Team
systemd (Ubuntu)
Critical
Canonical Foundations Team
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned

Bug Description

Booting d-i based on kernel 4.4 (from ./boot folder) from the 16.04.5 RC image on a z/VM guest
leads to a kernel panic:

/lib/debian-installer/start-udev: line 15: can't create /sys/module/scsi_mod/par
ameters/scan: Permission denied
¬ 0.845071| Kernel panic - not syncing: Attempted to kill initÜ exitcode=0x00
000100
¬ 0.845071|
¬ 0.845075| CPU: 0 PID: 1 Comm: /init Not tainted 4.4.0-131-generic #157-Ubun
tu
¬ 0.845077| 000000007d107c20 000000007d107cb0 0000000000000002 0000000
000000000
       000000007d107d50 000000007d107cc8 000000007d107cc8 0000000000114732
       000000000000008b 00000000009a304c 00000000009f11d0 000000000000000b
       000000007d107d10 000000007d107cb0 0000000000000000 0000000000000000
       0400000001cdf800 0000000000114732 000000007d107cb0 000000007d107d10
¬ 0.845089| Call Trace:
¬ 0.845093| (¬<000000000011461a>| show_trace+0xfa/0x150)
¬ 0.845095| ¬<00000000001146e2>| show_stack+0x72/0xf8
¬ 0.845099| ¬<0000000000555752>| dump_stack+0x82/0xb8
¬ 0.845101| ¬<0000000000286390>| panic+0x108/0x250
¬ 0.845104| ¬<0000000000166228>| do_exit+0xac0/0xba0
¬ 0.845106| ¬<00000000001663c8>| do_group_exit+0x50/0xe0
¬ 0.845108| ¬<0000000000166488>| __wake_up_parent+0x0/0x28
¬ 0.845111| ¬<00000000007eadda>| system_call+0xee/0x28c
¬ 0.845113| ¬<000003ff8953983a>| 0x3ff8953983a
00: HCPGIR450W CP entered; disabled wait PSW 00020001 80000000 00000000 0010F8CA

Dimitri John Ledkov (xnox) wrote :

13: if [ -f /sys/module/scsi_mod/parameters/scan ]; then
14: echo sync > /sys/module/scsi_mod/parameters/scan
15: fi

Not sure how come the file is there, yet "cannot be created" when we try to echo into it... maybe it's not writable or disappears.

I think the quickest botch that i can think of is:

   echo sync > /sys/module/scsi_mod/parameters/scan 2>/dev/null || :

to redirect errors to oblivion, and to ignore errors from this action....

Changed in debian-installer (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in systemd (Ubuntu):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Changed in debian-installer (Ubuntu):
importance: Undecided → Critical
Changed in systemd (Ubuntu):
importance: Undecided → Critical
Frank Heimes (frank-heimes) wrote :

btw. d-i based on hwe kernel 4.15 (boot-hwe) works fine ...

Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1784454

tags: added: iso-testing
tags: added: kernel-key
Changed in linux (Ubuntu):
status: New → Triaged
importance: Undecided → Critical
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Steve Langasek (vorlon) on 2018-07-30
Changed in debian-installer (Ubuntu):
milestone: none → ubuntu-16.04.5
Carla Sella (carla-sella) wrote :

Hello Frank, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in systemd (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial
Changed in linux (Ubuntu):
status: Triaged → Invalid
no longer affects: ubuntu-z-systems
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 229-4ubuntu21.4

---------------
systemd (229-4ubuntu21.4) xenial; urgency=medium

  * debian/extra/start-udev: test that sync parameters file is writable, before
    writing to it. On old kernels (e.g. v4.4) the file is available but is
    read-only, thus writing to it causes init to fail and thus raise a kernel
    panic. As an extra caution ignore failures to write into that file anyway,
    since synchronous scsi probing is not critical enough to prevent starting
    d-i. LP: #1784454

 -- Dimitri John Ledkov <email address hidden> Mon, 30 Jul 2018 19:58:15 +0100

Changed in systemd (Ubuntu Xenial):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package debian-installer - 20101020ubuntu451.26

---------------
debian-installer (20101020ubuntu451.26) xenial; urgency=medium

  * Rebuild against new systemd for the 4.4 kernel regression fix (LP: #1784454)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Mon, 30 Jul 2018 22:26:39 +0200

Changed in debian-installer (Ubuntu):
status: New → Fix Released
Frank Heimes (frank-heimes) wrote :

I can confirm that the updated installer taken from here:
http://ports.ubuntu.com/ubuntu-ports/dists/xenial-proposed/main/installer-s390x/20101020ubuntu451.26/
now properly works and boots.

Setting the tags accordingly.

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Frank Heimes (frank-heimes) wrote :

Looks like the fixed installer was already included into the todays ISO:
http://cdimage.ubuntu.com/ubuntu-server/xenial/daily/20180731/xenial-server-s390x.iso
I can confirm that this works, too (adding a 'pass' to the QA tracker)

Changed in ubuntu-z-systems:
status: New → Fix Released
importance: Undecided → Critical
Łukasz Zemczak (sil2100) wrote :

Thanks for verifying!

Łukasz Zemczak (sil2100) wrote :

Hello Frank, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-done
Łukasz Zemczak (sil2100) wrote :

Hello Frank, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Frank Heimes (frank-heimes) wrote :

I can confirm that I don't see the above issue with systemd-237-3ubuntu10.10 from bionic-proposed.
Adjusting the tags accordingly.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Brad Figg (brad-figg) on 2019-07-24
tags: added: cscc
Steve Langasek (vorlon) wrote :

this bug lacks a clear SRU test case for bionic (there is obviously no 4.4 kernel in 18.04).

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Incomplete
Steve Langasek (vorlon) wrote :

Hello Frank, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.30 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: Incomplete → Fix Committed
tags: added: verification-needed verification-needed-bionic
removed: verification-done verification-done-bionic
Steve Langasek (vorlon) wrote :

a version of systemd with this change was accepted into bionic in error and has now been withdrawn. a test case is needed here before it's accepted properly.

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Incomplete
Frank Heimes (frank-heimes) wrote :

Since it was me who initially raised that bug, I feel like I should comment on this.

My test case was and is to install Ubuntu on z/VM using (at least one) zFCP/SCSI devices for disk storage.
That should be a proper end-2-end test since it includes the entire zFCP/SCSI stack (even incl. multipath) and with that covers the initially reported problem mentioned in the bug description.

Please notice that all this happened to me on 16.04(.5) only with GA kernel 4.4 (hence the '4.4' still in the headline and even 16.04's HWE kernel 4.15 worked for me in the past - see comment above), but now it looks like we talk about 18.04 too, but I never saw that problem on 18.04.

Nevertheless, I (re-)tested today on 18.04(.3) - so it's more a regression test than anything else.

I used 18.04.3 default (means proposed disabled) as well as with proposed enabled.
W/o proposed I see systemd 237-3ubuntu10.29 with propose I see 237-3ubuntu10.31.
Means I wasn't able to use 237-3ubuntu10.30 - like mentioned in comment #30.
But both tests worked fine for me and I didn't saw any anomalies (details see attachment).

Balint Reczey (rbalint) wrote :

@frank-heimes, thank you for the testing and also for your additional comments.
This fix was once accepted to bionic-proposed, while it does not have a test case in the bug description nor the description follows the SRU template: https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template .
Ensuring that is the job of the SRU uploader, and the first time is was skipped and I also missed that.
Since the bug did not exist in 18.04 I'm marking it invalid instead of performing the SRU again.

Changed in systemd (Ubuntu Bionic):
status: Incomplete → Invalid
tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Balint Reczey (rbalint) wrote :

I also set verification-done, since it is verified that the bug does not occur with .31 .

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers