Activity log for bug #1994601

Date Who What changed Old value New value Message
2022-10-26 14:09:31 bugproxy bug added bug
2022-10-26 14:09:34 bugproxy tags architecture-s39064 bugnameltc-200228 severity-critical targetmilestone-inin---
2022-10-26 14:09:36 bugproxy attachment added crash log output https://bugs.launchpad.net/bugs/1994601/+attachment/5627021/+files/log.out
2022-10-26 14:09:38 bugproxy ubuntu: assignee Skipper Bug Screeners (skipper-screen-team)
2022-10-26 14:09:43 bugproxy affects ubuntu linux (Ubuntu)
2022-10-26 14:12:19 Launchpad Janitor linux (Ubuntu): status New Confirmed
2022-10-26 14:18:18 Frank Heimes bug task added ubuntu-z-systems
2022-10-26 14:18:57 Frank Heimes ubuntu-z-systems: assignee Skipper Bug Screeners (skipper-screen-team)
2022-10-26 14:19:09 Frank Heimes ubuntu-z-systems: importance Undecided Critical
2022-10-26 15:20:32 Frank Heimes ubuntu-z-systems: status New Incomplete
2022-10-26 19:23:12 Frank Heimes ubuntu-z-systems: status Incomplete Confirmed
2022-10-27 05:54:09 Frank Heimes attachment added 0001-Revert-s390-archrandom-simplify-back-to-earlier-desi.patch https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1994601/+attachment/5627151/+files/0001-Revert-s390-archrandom-simplify-back-to-earlier-desi.patch
2022-10-27 08:37:10 Ubuntu Foundations Team Bug Bot tags architecture-s39064 bugnameltc-200228 severity-critical targetmilestone-inin--- architecture-s39064 bugnameltc-200228 patch severity-critical targetmilestone-inin---
2022-10-27 08:37:13 Ubuntu Foundations Team Bug Bot bug added subscriber Terry Rudd
2022-10-27 11:43:47 Frank Heimes description ---Problem Description--- Ubuntu 18.04 crashes during IPL with no output on the console. Contact Information = Viktor Mihajlovski <mihajlov@de.ibm.com> ---uname output--- n/a Machine Type = 3096 ---Debugger--- A debugger is not configured ---Steps to Reproduce--- Install Ubuntu 18.04 as a KVM guest using the following command: virt-install -n bionic --cdrom /var/lib/libvirt/images/ubuntu-18.04.5-server-s390x.iso --memory 2048 --disk size=8 then reboot. Stack trace output: no Oops output: no == Comment: #1 - Viktor Mihajlovski <MIHAJLOV@de.ibm.com> - 2022-10-25 10:48:30 == Installing under z/VM leads to the same failure. == Comment: #2 - Viktor Mihajlovski <MIHAJLOV@de.ibm.com> - 2022-10-25 10:55:10 == I have captured a dump using virsh dump --memory-only. The output of crash log is uploaded == Comment: #7 - Harald Freudenberger <freude@de.ibm.com> - 2022-10-26 07:33:52 == Looks like all ubuntu 18.04 installations on s390 are not working any more. It is not an issue with z14 but z17 also fails to run a fresh installed ubuntu 18.04. == Comment: #8 - Harald Freudenberger <freude@de.ibm.com> - 2022-10-26 08:25:52 == when I use the 'advanced installation' where I am able to choose the kernel package and then choose the 4.15.0-50 generic the installed Ubuntu 18.04 comes up fine. So this issue is somewhere between kernel 4.15.0-50 and 4.15.0-194. SRU Justification: ================== [ Impact ] * Ubuntu 18.04 / bionic installations with latest kernel 4.15.0-194 are no longer able to IPL (boot) on IBM z14 or newer hardware. * This issue got introduced by upstream commit e4f74400308c "s390/archrandom: simplify back to earlier design and initialize earlier" that was SRUed to 18.04/bionic based on LP#1989625, which made changes in the s390s IPL/boot area of kernel/arch/random. * The reason seems to be that the bad patch moves the decision about if arch randomness is available to setup.c function setup_randomness(). This code uses a static_key s390_arch_random_available. But in the Canonical kernel the initialization function for the jump labels (where the static keys are based on) jump_label_init() is called in generic start_kernel() wheres in the upstream kernel the init function is called early in setup_arch(). * Reverting this commit from bionic master-next makes bionic systems again bootable. (https://launchpad.net/~fheimes/+archive/ubuntu/test/) [ Test Plan ] * An IBM z14 or LinuxONE II or newer system is needed. * Now install latest bionic on that system - doesn't if it's on LPAR, z/VM or KVM. * After the installation (an the trigger of the post install reboot), the system will not come up. * To test a patched kernel with e4f74400308c can be tested in the following way: * Install 18.04 GA and prevent it from doing any kernel updates. * Means, install in 'island' mode or select in d-i 'Advanced Installation' and explicitly choose '4.15.0-50 generic' to install. * That allows the system to come up and to update the kernel to a modified one. * Then reboot and verify if the system comes up properly. [ Where problems could occur ] * Problems could occur due to the fact that the commit was not cleanly reversible because of minor context changes. * Adjustments that were needed might break other things if not done carefully. * Further commits (applied after e4f74400308c) may still rely on the bad e4f74400308c commit - or even further patches (from upstream stable). * In worst case IPL / boot might get broken, even on hardware older than z14. * If the revert works fine can be easily tested and was tested based on https://launchpad.net/~fheimes/+archive/ubuntu/test/ and the above test plan. [ Other Info ] * Ubuntu 20.04 (focal, using legacy image with virt-install) was tested as well, but is not affected by this issue. __________ ---Problem Description--- Ubuntu 18.04 crashes during IPL with no output on the console. Contact Information = Viktor Mihajlovski <mihajlov@de.ibm.com> ---uname output--- n/a Machine Type = 3096 ---Debugger--- A debugger is not configured ---Steps to Reproduce---  Install Ubuntu 18.04 as a KVM guest using the following command: virt-install -n bionic --cdrom /var/lib/libvirt/images/ubuntu-18.04.5-server-s390x.iso --memory 2048 --disk size=8 then reboot. Stack trace output:  no Oops output:  no == Comment: #1 - Viktor Mihajlovski <MIHAJLOV@de.ibm.com> - 2022-10-25 10:48:30 == Installing under z/VM leads to the same failure. == Comment: #2 - Viktor Mihajlovski <MIHAJLOV@de.ibm.com> - 2022-10-25 10:55:10 == I have captured a dump using virsh dump --memory-only. The output of crash log is uploaded == Comment: #7 - Harald Freudenberger <freude@de.ibm.com> - 2022-10-26 07:33:52 == Looks like all ubuntu 18.04 installations on s390 are not working any more. It is not an issue with z14 but z17 also fails to run a fresh installed ubuntu 18.04. == Comment: #8 - Harald Freudenberger <freude@de.ibm.com> - 2022-10-26 08:25:52 == when I use the 'advanced installation' where I am able to choose the kernel package and then choose the 4.15.0-50 generic the installed Ubuntu 18.04 comes up fine. So this issue is somewhere between kernel 4.15.0-50 and 4.15.0-194.
2022-10-27 11:49:04 Frank Heimes summary [UBUNTU 18.04] Ubuntu 18.04 crashes during IPL [UBUNTU 18.04] Ubuntu 18.04 kernel 4.15.0-194 crashes on IPL
2022-10-27 15:32:12 Frank Heimes linux (Ubuntu): status Confirmed In Progress
2022-10-27 15:32:15 Frank Heimes ubuntu-z-systems: status Confirmed In Progress
2022-10-27 15:32:27 Frank Heimes linux (Ubuntu): assignee Skipper Bug Screeners (skipper-screen-team) Canonical Kernel Team (canonical-kernel-team)
2022-10-27 21:28:28 Luke Nowakowski-Krijger nominated for series Ubuntu Bionic
2022-10-27 21:28:28 Luke Nowakowski-Krijger bug task added linux (Ubuntu Bionic)
2022-10-27 21:28:41 Luke Nowakowski-Krijger linux (Ubuntu Bionic): assignee Luke Nowakowski-Krijger (lukenow)
2022-10-27 21:28:47 Luke Nowakowski-Krijger linux (Ubuntu Bionic): status New Fix Committed
2022-10-28 06:45:04 Frank Heimes ubuntu-z-systems: status In Progress Fix Committed
2022-11-02 15:41:01 Launchpad Janitor linux (Ubuntu Bionic): status Fix Committed Fix Released
2022-11-02 15:59:49 bugproxy tags architecture-s39064 bugnameltc-200228 patch severity-critical targetmilestone-inin--- architecture-s39064 bugnameltc-200228 patch severity-critical targetmilestone-inin1804
2022-11-10 10:59:47 Frank Heimes linux (Ubuntu): status In Progress Invalid
2022-11-10 10:59:50 Frank Heimes ubuntu-z-systems: status Fix Committed Fix Released
2023-02-25 15:06:19 Ubuntu Kernel Bot tags architecture-s39064 bugnameltc-200228 patch severity-critical targetmilestone-inin1804 architecture-s39064 bugnameltc-200228 kernel-spammed-bionic-linux-azure-4.15 patch severity-critical targetmilestone-inin1804 verification-needed-bionic
2023-02-25 16:56:31 Frank Heimes tags architecture-s39064 bugnameltc-200228 kernel-spammed-bionic-linux-azure-4.15 patch severity-critical targetmilestone-inin1804 verification-needed-bionic architecture-s39064 bugnameltc-200228 kernel-spammed-bionic-linux-azure-4.15 patch severity-critical targetmilestone-inin1804 verification-done-bionic