This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack.
When we boot bionic Ubuntu cloud images in "bos01", they always fail like this:
...
[ 1.585611] Key type dns_resolver registered
[ 1.587408] registered taskstats version 1
[ 1.588913] Loading compiled-in X.509 certificates
[ 1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd0178c9ab722f3ae7b392f7714ac4'
[ 1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37625abec039ef2bf521249b969'
[ 1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e0737e31163a466ad7b70a850c19'
[ 1.610861] Couldn't get size: 0x800000000000000e
[ 1.613413] MODSIGN: Couldn't get UEFI db list
[ 1.615920] Couldn't get size: 0x800000000000000e
[ 1.618256] MODSIGN: Couldn't get UEFI MokListRT
[ 1.620315] Couldn't get size: 0x800000000000000e
[ 1.622317] MODSIGN: Couldn't get UEFI dbx list
[ 1.624446] zswap: loaded using pool lzo/zbud
[ 1.628185] Key type big_key registered
[ 1.629937] Key type trusted registered
[ 1.632012] Key type encrypted registered
[ 1.633668] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[ 1.638009] ima: Allocated hash algorithm: sha1
[ 1.639272] evm: HMAC attrs: 0x1
[ 1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323)
[ 1.646247] Freeing unused kernel memory: 5824K
[ 1.657870] Checked W+X mappings: passed, no W+X pages found
[ 1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.660250]
[ 1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu
[ 1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 1.668204] Call trace:
[ 1.668981] dump_backtrace+0x0/0x198
[ 1.670212] show_stack+0x24/0x30
[ 1.671282] dump_stack+0x98/0xc8
[ 1.672433] panic+0x128/0x2b0
[ 1.673502] do_exit+0x75c/0xa80
[ 1.674749] do_group_exit+0x40/0xb0
[ 1.676191] get_signal+0x114/0x6e8
[ 1.677867] do_signal+0x18c/0x240
[ 1.679498] do_notify_resume+0xd0/0x328
[ 1.681302] work_pending+0x8/0x10
[ 1.682771] SMP: stopping secondary CPUs
[ 1.684624] Kernel Offset: disabled
[ 1.686119] CPU features: 0x04802008
[ 1.687353] Memory Limit: none
[ 1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.688453]
We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible.
The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5)
The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11)
This is something to do with the configuration or software versions running here, since we have identical hardware running in an adjacent cloud region but with different versions of the cloud/virt stack.
When we boot bionic Ubuntu cloud images in "bos01", they always fail like this:
... 8c9ab722f3ae7b3 92f7714ac4' 625abec039ef2bf 521249b969' 37e31163a466ad7 b70a850c19' 0x0/0x198 0x24/0x30 0x98/0xc8 exit+0x40/ 0xb0 0x114/0x6e8 0x18c/0x240 resume+ 0xd0/0x328 0x8/0x10
[ 1.585611] Key type dns_resolver registered
[ 1.587408] registered taskstats version 1
[ 1.588913] Loading compiled-in X.509 certificates
[ 1.592668] Loaded X.509 cert 'Build time autogenerated kernel key: 4a4a555bc5fd017
[ 1.598866] Loaded X.509 cert 'Canonical Ltd. Live Patch Signing: 14df34d1a87cf37
[ 1.605389] Loaded X.509 cert 'Canonical Ltd. Kernel Module Signing: 88f752e560a1e07
[ 1.610861] Couldn't get size: 0x800000000000000e
[ 1.613413] MODSIGN: Couldn't get UEFI db list
[ 1.615920] Couldn't get size: 0x800000000000000e
[ 1.618256] MODSIGN: Couldn't get UEFI MokListRT
[ 1.620315] Couldn't get size: 0x800000000000000e
[ 1.622317] MODSIGN: Couldn't get UEFI dbx list
[ 1.624446] zswap: loaded using pool lzo/zbud
[ 1.628185] Key type big_key registered
[ 1.629937] Key type trusted registered
[ 1.632012] Key type encrypted registered
[ 1.633668] AppArmor: AppArmor sha1 policy hashing enabled
[ 1.635625] ima: No TPM chip found, activating TPM-bypass! (rc=-19)
[ 1.638009] ima: Allocated hash algorithm: sha1
[ 1.639272] evm: HMAC attrs: 0x1
[ 1.640875] rtc-efi rtc-efi: setting system clock to 2021-05-11 09:28:43 UTC (1620725323)
[ 1.646247] Freeing unused kernel memory: 5824K
[ 1.657870] Checked W+X mappings: passed, no W+X pages found
[ 1.660250] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.660250]
[ 1.663294] CPU: 1 PID: 1 Comm: init Not tainted 4.15.0-142-generic #146-Ubuntu
[ 1.665939] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[ 1.668204] Call trace:
[ 1.668981] dump_backtrace+
[ 1.670212] show_stack+
[ 1.671282] dump_stack+
[ 1.672433] panic+0x128/0x2b0
[ 1.673502] do_exit+0x75c/0xa80
[ 1.674749] do_group_
[ 1.676191] get_signal+
[ 1.677867] do_signal+
[ 1.679498] do_notify_
[ 1.681302] work_pending+
[ 1.682771] SMP: stopping secondary CPUs
[ 1.684624] Kernel Offset: disabled
[ 1.686119] CPU features: 0x04802008
[ 1.687353] Memory Limit: none
[ 1.688453] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000005
[ 1.688453]
Full example log: https:/ /autopkgtest. ubuntu. com/results/ autopkgtest- bionic/ bionic/ arm64/c/ chromium- browser/ 20210511_ 093739_ ea3f2@/ log.gz
We tried to get IS to roll things back to match the working bos02, but they said it's too different and not possible.
The working cloud is running Mitaka from the cloud archive on Xenial (qemu 2.5)
The broken cloud is running Queens from the cloud archive on Xenial (qemu 2.11)
That's about all we have right now.