| 2021-06-11 16:09:27 |
Iain Lane |
bug |
|
|
added bug |
| 2021-06-11 16:09:57 |
Iain Lane |
description |
This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack.
When we boot bionic Ubuntu cloud images in "bos01", they always fail like this:
...
[ 1.585611] Key type dns_resolver registered
[ 1.587408] registered taskstats version 1
[ 1.588913] Loading compiled-in X.509 certificates
[ 1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd0178c9ab722f3ae7b392f7714ac4'
[ 1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
[ 1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
[ 1.610861] Couldn't get size: 0x800000000000000e
[ 1.613413] MODSIGN: Couldn't get UEFI db list
[ 1.615920] Couldn't get size: 0x800000000000000e
[ 1.618256] MODSIGN: Couldn't get UEFI MokListRT
[ 1.620315] Couldn't get size: 0x800000000000000e
[ 1.622317] MODSIGN: Couldn't get UEFI dbx list
[ 1.624446] zswap: loaded using pool lzo/zbud
[ 1.628185] Key type big_key registered
[ 1.629937] Key type trusted registered
[ 1.632012] Key type encrypted registered
[ 1.633668] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[ 1.638009] ima: Allocated hash algorithm: sha1
[ 1.639272] evm: HMAC attrs: 0x1
[ 1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323)
[ 1.646247] Freeing unused kernel memory: 5824K
[ 1.657870] Checked W+X mappings: passed, no W+X pages found
[ 1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.660250]
[ 1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu
[ 1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 1.668204] Call trace:
[ 1.668981] dump_backtrace+0x0/0x198
[ 1.670212] show_stack+0x24/0x30
[ 1.671282] dump_stack+0x98/0xc8
[ 1.672433] panic+0x128/0x2b0
[ 1.673502] do_exit+0x75c/0xa80
[ 1.674749] do_group_exit+0x40/0xb0
[ 1.676191] get_signal+0x114/0x6e8
[ 1.677867] do_signal+0x18c/0x240
[ 1.679498] do_notify_resume+0xd0/0x328
[ 1.681302] work_pending+0x8/0x10
[ 1.682771] SMP: stopping secondary CPUs
[ 1.684624] Kernel Offset: disabled
[ 1.686119] CPU features: 0x04802008
[ 1.687353] Memory Limit: none
[ 1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.688453]
Full example log: https://autopkgtest.ubuntu.com/results/autopkgtest-bionic/bionic/arm64/c/chromium-browser/20210511_093739_ea3f2@/log.gz
We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible.
The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5)
The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11)
That's about all we have right now. |
This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack.
When we boot bionic Ubuntu cloud images in "bos01", they always fail like this:
...
[ 1.585611] Key type dns_resolver registered
[ 1.587408] registered taskstats version 1
[ 1.588913] Loading compiled-in X.509 certificates
[ 1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd0178c9ab722f3ae7b392f7714ac4'
[ 1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
[ 1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
[ 1.610861] Couldn't get size: 0x800000000000000e
[ 1.613413] MODSIGN: Couldn't get UEFI db list
[ 1.615920] Couldn't get size: 0x800000000000000e
[ 1.618256] MODSIGN: Couldn't get UEFI MokListRT
[ 1.620315] Couldn't get size: 0x800000000000000e
[ 1.622317] MODSIGN: Couldn't get UEFI dbx list
[ 1.624446] zswap: loaded using pool lzo/zbud
[ 1.628185] Key type big_key registered
[ 1.629937] Key type trusted registered
[ 1.632012] Key type encrypted registered
[ 1.633668] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[ 1.638009] ima: Allocated hash algorithm: sha1
[ 1.639272] evm: HMAC attrs: 0x1
[ 1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323)
[ 1.646247] Freeing unused kernel memory: 5824K
[ 1.657870] Checked W+X mappings: passed, no W+X pages found
[ 1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.660250]
[ 1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu
[ 1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 1.668204] Call trace:
[ 1.668981] dump_backtrace+0x0/0x198
[ 1.670212] show_stack+0x24/0x30
[ 1.671282] dump_stack+0x98/0xc8
[ 1.672433] panic+0x128/0x2b0
[ 1.673502] do_exit+0x75c/0xa80
[ 1.674749] do_group_exit+0x40/0xb0
[ 1.676191] get_signal+0x114/0x6e8
[ 1.677867] do_signal+0x18c/0x240
[ 1.679498] do_notify_resume+0xd0/0x328
[ 1.681302] work_pending+0x8/0x10
[ 1.682771] SMP: stopping secondary CPUs
[ 1.684624] Kernel Offset: disabled
[ 1.686119] CPU features: 0x04802008
[ 1.687353] Memory Limit: none
[ 1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.688453]
Full example log: https://autopkgtest.ubuntu.com/results/autopkgtest-bionic/bionic/arm64/c/chromium-browser/20210511_093739_ea3f2@/log.gz
We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible.
The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5)
The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11)
The other thing someone suggested we try is that MDS mitigation is enabled in the broken cloud, so we could disable it. No idea if that makes sense tbh.
That's about all we have right now. |
|
| 2021-06-11 16:18:22 |
Iain Lane |
description |
This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack.
When we boot bionic Ubuntu cloud images in "bos01", they always fail like this:
...
[ 1.585611] Key type dns_resolver registered
[ 1.587408] registered taskstats version 1
[ 1.588913] Loading compiled-in X.509 certificates
[ 1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd0178c9ab722f3ae7b392f7714ac4'
[ 1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
[ 1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
[ 1.610861] Couldn't get size: 0x800000000000000e
[ 1.613413] MODSIGN: Couldn't get UEFI db list
[ 1.615920] Couldn't get size: 0x800000000000000e
[ 1.618256] MODSIGN: Couldn't get UEFI MokListRT
[ 1.620315] Couldn't get size: 0x800000000000000e
[ 1.622317] MODSIGN: Couldn't get UEFI dbx list
[ 1.624446] zswap: loaded using pool lzo/zbud
[ 1.628185] Key type big_key registered
[ 1.629937] Key type trusted registered
[ 1.632012] Key type encrypted registered
[ 1.633668] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[ 1.638009] ima: Allocated hash algorithm: sha1
[ 1.639272] evm: HMAC attrs: 0x1
[ 1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323)
[ 1.646247] Freeing unused kernel memory: 5824K
[ 1.657870] Checked W+X mappings: passed, no W+X pages found
[ 1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.660250]
[ 1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu
[ 1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 1.668204] Call trace:
[ 1.668981] dump_backtrace+0x0/0x198
[ 1.670212] show_stack+0x24/0x30
[ 1.671282] dump_stack+0x98/0xc8
[ 1.672433] panic+0x128/0x2b0
[ 1.673502] do_exit+0x75c/0xa80
[ 1.674749] do_group_exit+0x40/0xb0
[ 1.676191] get_signal+0x114/0x6e8
[ 1.677867] do_signal+0x18c/0x240
[ 1.679498] do_notify_resume+0xd0/0x328
[ 1.681302] work_pending+0x8/0x10
[ 1.682771] SMP: stopping secondary CPUs
[ 1.684624] Kernel Offset: disabled
[ 1.686119] CPU features: 0x04802008
[ 1.687353] Memory Limit: none
[ 1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.688453]
Full example log: https://autopkgtest.ubuntu.com/results/autopkgtest-bionic/bionic/arm64/c/chromium-browser/20210511_093739_ea3f2@/log.gz
We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible.
The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5)
The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11)
The other thing someone suggested we try is that MDS mitigation is enabled in the broken cloud, so we could disable it. No idea if that makes sense tbh.
That's about all we have right now. |
This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack.
When we boot bionic arm64 Ubuntu cloud images in "bos01" and they land on the eMAG systems, they always fail like this:
...
[ 1.585611] Key type dns_resolver registered
[ 1.587408] registered taskstats version 1
[ 1.588913] Loading compiled-in X.509 certificates
[ 1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd0178c9ab722f3ae7b392f7714ac4'
[ 1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
[ 1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
[ 1.610861] Couldn't get size: 0x800000000000000e
[ 1.613413] MODSIGN: Couldn't get UEFI db list
[ 1.615920] Couldn't get size: 0x800000000000000e
[ 1.618256] MODSIGN: Couldn't get UEFI MokListRT
[ 1.620315] Couldn't get size: 0x800000000000000e
[ 1.622317] MODSIGN: Couldn't get UEFI dbx list
[ 1.624446] zswap: loaded using pool lzo/zbud
[ 1.628185] Key type big_key registered
[ 1.629937] Key type trusted registered
[ 1.632012] Key type encrypted registered
[ 1.633668] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[ 1.638009] ima: Allocated hash algorithm: sha1
[ 1.639272] evm: HMAC attrs: 0x1
[ 1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323)
[ 1.646247] Freeing unused kernel memory: 5824K
[ 1.657870] Checked W+X mappings: passed, no W+X pages found
[ 1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.660250]
[ 1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu
[ 1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 1.668204] Call trace:
[ 1.668981] dump_backtrace+0x0/0x198
[ 1.670212] show_stack+0x24/0x30
[ 1.671282] dump_stack+0x98/0xc8
[ 1.672433] panic+0x128/0x2b0
[ 1.673502] do_exit+0x75c/0xa80
[ 1.674749] do_group_exit+0x40/0xb0
[ 1.676191] get_signal+0x114/0x6e8
[ 1.677867] do_signal+0x18c/0x240
[ 1.679498] do_notify_resume+0xd0/0x328
[ 1.681302] work_pending+0x8/0x10
[ 1.682771] SMP: stopping secondary CPUs
[ 1.684624] Kernel Offset: disabled
[ 1.686119] CPU features: 0x04802008
[ 1.687353] Memory Limit: none
[ 1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.688453]
Full example log: https://autopkgtest.ubuntu.com/results/autopkgtest-bionic/bionic/arm64/c/chromium-browser/20210511_093739_ea3f2@/log.gz
We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible.
The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5)
The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11)
The other thing someone suggested we try is that MDS mitigation is enabled in the broken cloud, so we could disable it. No idea if that makes sense tbh.
That's about all we have right now. |
|
| 2021-06-11 16:18:27 |
Iain Lane |
summary |
[scalingstack bos01] bionic instances always fail to boot on eMAGs in this cloud |
[scalingstack bos01] bionic (arm64) instances always fail to boot on eMAGs in this cloud |
|
| 2021-06-14 12:24:10 |
Lucas Kanashiro |
bug |
|
|
added subscriber Christian Ehrhardt |
| 2021-06-23 01:33:16 |
Po-Hsu Lin |
bug |
|
|
added subscriber Po-Hsu Lin |
| 2021-06-23 21:11:38 |
Paul Collins |
bug |
|
|
added subscriber The Canonical Sysadmins |
| 2021-06-24 16:42:24 |
dann frazier |
bug task added |
|
linux (Ubuntu) |
|
| 2021-06-24 16:42:33 |
dann frazier |
qemu (Ubuntu): status |
New |
Invalid |
|
| 2021-06-25 18:46:15 |
dann frazier |
linux (Ubuntu): status |
New |
Triaged |
|
| 2021-08-17 16:48:00 |
Taihsiang Ho |
bug |
|
|
added subscriber Taihsiang Ho |
