[Ubuntu 1810] Migrating Power9 guest back to source causes kernel Oops inside guest

Bug #1810392 reported by bugproxy on 2019-01-03
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
High
Canonical Kernel Team
linux (Ubuntu)
Undecided
Ubuntu on IBM Power Systems Bug Triage

Bug Description

---Problem Description---
Migrating back Power9 to source cause Kernel Oops inside guest

Contact Information = Balamuruhan S / <email address hidden>

---uname output---
source and target host Kernel: 4.18.0-8-generic
Guest Kernel: 4.18.0-8-generic

Machine Type = Boston

---Debugger---
A debugger is not configured

---Steps to Reproduce---

Due to Bug 172310, Turn off the firewalld in source and target host machines.

1. Define a healthy guest and placed image in NFS share.

2. Mounted the image location in destination and start the guest to perform migration.

3. Perform migration from source to target,
08:07:54 INFO | Migrating VM avocado-vt-vm1-migration from qemu:///system to qemu+ssh://9.40.194.13/system
08:07:54 DEBUG| Running virsh command: migrate --live --domain avocado-vt-vm1-migration --desturi qemu+ssh://9.40.194.13/system --timeout 60
08:07:54 INFO | Running '/usr/bin/virsh -c 'qemu:///system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu+ssh://9.40.194.13/system --timeout 60'
08:08:21 DEBUG| [stdout]
08:08:21 INFO | Command '/usr/bin/virsh -c 'qemu:///system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu+ssh://9.40.194.13/system --timeout 60' finished with 0 after 26.9465551376s
08:08:21 DEBUG| status: 0
08:08:21 DEBUG| stdout:
08:08:21 DEBUG| stderr:
08:08:21 INFO | Migration exit status: 0

4. check VM uptime after successful migration,
08:07:44 INFO | Check VM uptime before migration: 57.66

5. Migration is successful from source to destination, now perform migration back to source

08:08:49 INFO | Migrating VM avocado-vt-vm1-migration from qemu+ssh://9.40.194.13/system to qemu:///system
08:08:49 DEBUG| Running virsh command: migrate --live --domain avocado-vt-vm1-migration --desturi qemu:///system --timeout 60
08:08:49 INFO | Running '/usr/bin/virsh -c 'qemu+ssh://9.40.194.13/system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu:///system --timeout 60'
08:09:22 DEBUG| [stdout]
08:09:22 INFO | Command '/usr/bin/virsh -c 'qemu+ssh://9.40.194.13/system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu:///system --timeout 60' finished with 0 after 32.8640539646s
08:09:22 DEBUG| status: 0
08:09:22 DEBUG| stdout:
08:09:22 DEBUG| stderr:
08:09:22 INFO | Migration exit status: 0

6. check uptime of VM after migration and it is observed that VM have gone for reboot

08:10:22 INFO | Check VM uptime after migrating back to source: 31.09

5. By checking the VM console Call Trace is observed as mentioned below

The time stamp during the call trace observed inside guest matches with the timestamp during which
migration is triggered.

Stack trace output:
 no

Attachment:
1. VM sosreport
2. source and target sosreport
3. VM console log with call trace
4. Guest xml
5. Test job log

Oops output:
 2018-10-26 08:08:26: root@ubuntu1810:~#
2018-10-26 08:09:11: [ 144.512549] Unable to handle kernel paging request for data at address 0x71ff9672fa0e0989
2018-10-26 08:09:11: [ 144.512655] Faulting instruction address: 0xc0000000003af7cc
2018-10-26 08:09:11: [ 144.512741] Oops: Kernel access of bad area, sig: 11 [#1]
2018-10-26 08:09:11: [ 144.512789] LE SMP NR_CPUS=2048 NUMA pSeries
2018-10-26 08:09:11: [ 144.512859] Modules linked in: iscsi_target_mod target_core_mod xt_CHECKSUM iptable_mangle ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bpfilter kvm dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto crct10dif_vpmsum sch_fq_codel nfsd ib_iser auth_rpcgss rdma_cm nfs_acl iw_cm lockd ib_cm grace ib_core sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear virtio_net net_failover crc32c_vpmsum virtio_blk virtio_scsi failover
2018-10-26 08:09:11: [ 144.513801] CPU: 5 PID: 1072 Comm: systemd-network Not tainted 4.18.0-8-generic #9-Ubuntu
2018-10-26 08:09:11: [ 144.513872] NIP: c0000000003af7cc LR: c0000000003af7b8 CTR: c000000000c9ea80
2018-10-26 08:09:11: [ 144.513943] REGS: c0000000f19ab910 TRAP: 0380 Not tainted (4.18.0-8-generic)
2018-10-26 08:09:11: [ 144.514025] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 24002824 XER: 00000000
2018-10-26 08:09:11: [ 144.514102] CFAR: c0000000000174a0 IRQMASK: 0
2018-10-26 08:09:11: [ 144.514102] GPR00: c0000000003af670 c0000000f19abb90 c00000000178c800 0000000000000000
2018-10-26 08:09:11: [ 144.514102] GPR04: 0000000000000000 0000000000000007 0000000000000000 0000000000000006
2018-10-26 08:09:11: [ 144.514102] GPR08: b1ff967250115c09 0000000000000000 0000000000000000 c000000000d71000
2018-10-26 08:09:11: [ 144.514102] GPR12: c000000000c9ea80 c00000000fff8a00 0000000000000000 0000000000000000
2018-10-26 08:09:11: [ 144.514102] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
2018-10-26 08:09:11: [ 144.514102] GPR20: 0000000000000000 0000000000000000 0000000000000000 c0000000f19abde0
2018-10-26 08:09:11: [ 144.514102] GPR24: b1ff967250115c09 c0000000aa1f5580 71ff9672fa0e0989 c0000000fd017380
2018-10-26 08:09:11: [ 144.514102] GPR28: c000000000b5d188 00000000006000c0 c0000000aa1f5580 c0000000058a7c80
2018-10-26 08:09:11: [ 144.515013] NIP [c0000000003af7cc] kmem_cache_alloc+0x30c/0x360
2018-10-26 08:09:11: [ 144.515102] LR [c0000000003af7b8] kmem_cache_alloc+0x2f8/0x360
2018-10-26 08:09:11: [ 144.515186] Call Trace:
2018-10-26 08:09:11: [ 144.515224] [c0000000f19abb90] [c0000000003af670] kmem_cache_alloc+0x1b0/0x360 (unreliable)
2018-10-26 08:09:11: [ 144.515337] [c0000000f19abc00] [c000000000b5d188] sk_prot_alloc+0x58/0x200
2018-10-26 08:09:11: [ 144.515426] [c0000000f19abc50] [c000000000b5d3e4] sk_alloc+0x54/0x220
2018-10-26 08:09:11: [ 144.515514] [c0000000f19abca0] [c000000000c9e8f0] unix_create1+0xe0/0x270
2018-10-26 08:09:11: [ 144.515601] [c0000000f19abcf0] [c000000000c9eb98] unix_create+0x118/0x150
2018-10-26 08:09:11: [ 144.515689] [c0000000f19abd10] [c000000000b58234] __sock_create+0x134/0x2c0
2018-10-26 08:09:11: [ 144.515775] [c0000000f19abdc0] [c000000000b59f90] __sys_socket+0x60/0x160
2018-10-26 08:09:11: [ 144.515860] [c0000000f19abe10] [c000000000b5a0bc] sys_socket+0x2c/0x40
2018-10-26 08:09:11: [ 144.515948] [c0000000f19abe30] [c00000000000b288] system_call+0x5c/0x70
2018-10-26 08:09:11: [ 144.516032] Instruction dump:
2018-10-26 08:09:11: [ 144.516088] 409efe1c 7c98c278 38a50001 7f3ac278 fb490000 f8aa0000 4bc67cc5 60000000
2018-10-26 08:09:11: [ 144.516195] 7fb9c040 419e0028 813f0020 e91f0138 <7d5a482a> 7d3a4a14 7d4a4278 7fa95040
2018-10-26 08:09:11: [ 144.516314] ---[ end trace ea4650f5afbe0331 ]---

Userspace tool common name: Qemu, Libvirt

Userspace rpm:

Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu4 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu3 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.12+dfsg-3ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.12+dfsg-3ubuntu7 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.12+dfsg-3ubuntu7 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-data 1:2.12+dfsg-3ubuntu7 all QEMU full system emulation (data files)
ii qemu-system-gui 1:2.12+dfsg-3ubuntu7 ppc64el QEMU full system emulation binaries (user interface and audio support)
ii qemu-system-ppc 1:2.12+dfsg-3ubuntu7 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.12+dfsg-3ubuntu7 ppc64el QEMU utilities

Libvirt:
# dpkg -l | grep libvirt
ii gir1.2-libvirt-glib-1.0:ppc64el 1.0.0-1 ppc64el GObject introspection files for the libvirt-glib library
ii gir1.2-libvirt-sandbox-1.0 0.5.1+git20160404-1 ppc64el GObject introspection files for the libvirt-sandbox library
ii libvirt-clients 4.6.0-2ubuntu3 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.6.0-2ubuntu3 ppc64el Virtualization daemon
ii libvirt-daemon-driver-storage-gluster 4.6.0-2ubuntu3 ppc64el Virtualization daemon glusterfs storage driver
ii libvirt-daemon-driver-storage-rbd 4.6.0-2ubuntu3 ppc64el Virtualization daemon RBD storage driver
ii libvirt-daemon-driver-storage-sheepdog 4.6.0-2ubuntu3 ppc64el Virtualization daemon Sheedog storage driver
ii libvirt-daemon-driver-storage-zfs 4.6.0-2ubuntu3 ppc64el Virtualization daemon ZFS storage driver
ii libvirt-daemon-system 4.6.0-2ubuntu3 ppc64el Libvirt daemon configuration files
ii libvirt-dbus 1.2.0-1 ppc64el libvirt D-Bus API bindings
ii libvirt-dev:ppc64el 4.6.0-2ubuntu3 ppc64el development files for the libvirt library
ii libvirt-doc 4.6.0-2ubuntu3 all documentation for the libvirt library
ii libvirt-glib-1.0-0:ppc64el 1.0.0-1 ppc64el libvirt GLib and GObject mapping library
ii libvirt-glib-1.0-dev:ppc64el 1.0.0-1 ppc64el Development files for the libvirt-glib library
ii libvirt-ocaml 0.6.1.4-2build1 ppc64el OCaml bindings for libvirt (runtime)
ii libvirt-ocaml-dev 0.6.1.4-2build1 ppc64el OCaml bindings for libvirt (development files)
ii libvirt-sandbox-1.0-5 0.5.1+git20160404-1 ppc64el Application sandbox toolkit shared library
ii libvirt-sandbox-1.0-dev 0.5.1+git20160404-1 ppc64el Development files for libvirt-sandbox library
ii libvirt-sanlock 4.6.0-2ubuntu3 ppc64el Sanlock plugin for virtlockd
ii libvirt-wireshark 4.6.0-2ubuntu3 ppc64el Wireshark dissector for the libvirt protocol
ii libvirt0:ppc64el 4.6.0-2ubuntu3 ppc64el library for interfacing with different virtualization systems
ii libvirtodbc0 6.1.6+repack-0ubuntu9 ppc64el high-performance database - ODBC libraries
ii libvirtualpg-dev:ppc64el 2.0.0~rc0-1 ppc64el VirtualPG development files
ii libvirtualpg0:ppc64el 2.0.0~rc0-1 ppc64el VirtualPG shared library
ii python-libvirt 4.6.0-1 ppc64el libvirt Python bindings

bugproxy (bugproxy) on 2019-01-03
tags: added: architecture-ppc64le bugnameltc-172756 severity-high targetmilestone-inin---
Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → kernel-package (Ubuntu)
Changed in ubuntu-power-systems:
importance: Undecided → High
affects: kernel-package (Ubuntu) → linux (Ubuntu)
Changed in ubuntu-power-systems:
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
Andrew Cloke (andrew-cloke) wrote :

Assigning to kernel team initially, although this may be a qemu/libvirt issue.

Manoj Iyer (manjo) wrote :

IBM, is this reproducible with the latest Bionic 4.18 kernel as well ?

Changed in ubuntu-power-systems:
status: New → Triaged
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers