guest kernel panic in live migration test
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Confirmed
|
High
|
Unassigned | ||
Victoria |
New
|
Undecided
|
Unassigned |
Bug Description
There are various kernel panics visible in the guest in the nova-live-migration job. It is so far mostly visible on stable/victoria .
Example run: https:/
Nova stack trace:
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Nov 08 16:43:33.983139 ubuntu-
Guest kernel panic:
2021-11-08 16:41:35,132 99547 DEBUG [tempest.
body=
[ 15.293919] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 15.298512] BUG: unable to handle page fault for address: ffff91bdc256c400
[ 15.299353] #PF: supervisor instruction fetch in kernel mode
[ 15.299943] #PF: error_code(0x0011) - permissions violation
[ 15.300902] PGD 5e01067 P4D 5e01067 PUD 5e02067 PMD 80000000024001e3
[ 15.302056] Oops: 0011 [#1] SMP NOPTI
[ 15.302770] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 5.3.0-26-generic #28~18.04.1-Ubuntu
[ 15.303549] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 15.305100] RIP: 0010:0xffff91bd
[ 15.305758] Code: 6f 72 69 74 79 20 66 6f 72 20 25 64 20 28 25 73 29 00 25 64 20 28 25 73 29 20 6f 6c 64 20 70 72 69 6f 72 69 74 79 20 25 64 2c <20> 6e 65 77 20 70 72 69 6f 72 69 74 79 20 25 64 0a 00 70 72 6f 63
[ 15.307322] RSP: 0018:ffffa59240
[ 15.307835] RAX: ffff91bdc256c400 RBX: ffff91bdc762b4c0 RCX: ffff91bdc3239900
[ 15.308457] RDX: 0000000000000400 RSI: ffffa59240053df8 RDI: ffff91bdc67e6d88
[ 15.309109] RBP: ffffa59240053e48 R08: 0000000000000000 R09: 0000000000000001
[ 15.309725] R10: ffffa59240053bc0 R11: 000000000430d5a0 R12: ffff91bdc762b510
[ 15.310344] R13: ffffa59240053df8 R14: 000000000000000a R15: 0000000000000202
[ 15.311176] FS: 000000000000000
[ 15.311892] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 15.312403] CR2: ffff91bdc256c400 CR3: 0000000002f6c000 CR4: 00000000000006f0
[ 15.313336] Call Trace:
[ 15.314947] ? rcu_core+
[ 15.315635] rcu_core_
[ 15.316138] __do_softirq+
[ 15.316573] run_ksoftirqd+
[ 15.316952] smpboot_
[ 15.317326] kthread+0x121/0x140
[ 15.317648] ? sort_range+
[ 15.317980] ? kthread_
[ 15.318337] ret_from_
[ 15.318819] Modules linked in: ip_tables x_tables nls_utf8 nls_iso8859_1 nls_ascii isofs hid_generic usbhid hid virtio_rng virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm virtio_scsi virtio_net net_failover failover virtio_input virtio_blk qemu_fw_cfg 9pnet_virtio 9pnet pcnet32 8139cp mii ne2k_pci 8390 e1000
[ 15.322811] CR2: ffff91bdc256c400
[ 15.324234] ---[ end trace 73d738baa971ca73 ]---
[ 15.324797] RIP: 0010:0xffff91bd
[ 15.325176] Code: 6f 72 69 74 79 20 66 6f 72 20 25 64 20 28 25 73 29 00 25 64 20 28 25 73 29 20 6f 6c 64 20 70 72 69 6f 72 69 74 79 20 25 64 2c <20> 6e 65 77 20 70 72 69 6f 72 69 74 79 20 25 64 0a 00 70 72 6f 63
[ 15.326679] RSP: 0018:ffffa59240
[ 15.327142] RAX: ffff91bdc256c400 RBX: ffff91bdc762b4c0 RCX: ffff91bdc3239900
[ 15.327742] RDX: 0000000000000400 RSI: ffffa59240053df8 RDI: ffff91bdc67e6d88
[ 15.328342] RBP: ffffa59240053e48 R08: 0000000000000000 R09: 0000000000000001
[ 15.328964] R10: ffffa59240053bc0 R11: 000000000430d5a0 R12: ffff91bdc762b510
[ 15.329563] R13: ffffa59240053df8 R14: 000000000000000a R15: 0000000000000202
[ 15.330167] FS: 000000000000000
[ 15.330854] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 15.331348] CR2: ffff91bdc256c400 CR3: 0000000002f6c000 CR4: 00000000000006f0
[ 15.332107] Kernel panic - not syncing: Fatal exception in interrupt
[ 15.333470] Kernel Offset: 0x34800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000
[ 15.334628] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
Additional hits: https:/
description: | updated |
description: | updated |
tags: | added: gate-failure |
Changed in nova: | |
status: | New → Invalid |
We noticed that on wallaby,xena and master cirros 0.5.2 is used while on victoria cirros 0.5.1. But we ruled out this as the possible source of the problem by trying 0.5.2 on victoria and seeing the same panics https:/ /review. opendev. org/c/openstack /nova/+ /817173