Test 069 from ubuntu_zfs_xfs_generic causes kernel panic on Artful zKVM

[Po-Hsu Lin]

This issue can be easily reproduced on our zKVM node (s2lp6g004) by running the ubuntu_zfs_xfs_generic test suite.

The "virsh list" command will show that it's "crashed" after this happens.

09:09:49 INFO | Running: /home/ubuntu/autotest/client/tests/ubuntu_zfs_xfs_generic/ubuntu_zfs_xfs_generic.sh 069 /home/ubuntu/autotest/client/tmp/ubuntu_zfs_xfs_generic/src
09:09:49 DEBUG| Running '/home/ubuntu/autotest/client/tests/ubuntu_zfs_xfs_generic/ubuntu_zfs_xfs_generic.sh 069 /home/ubuntu/autotest/client/tmp/ubuntu_zfs_xfs_generic/src'
09:09:49 DEBUG| [stdout] Creating zfs pool testpool..
09:09:49 DEBUG| [stdout] Creating zfs file systems test and scratch in testpool..
09:09:49 DEBUG| [stdout] VDEVs in /home/ubuntu/autotest/client/tmp/ubuntu_zfs_xfs_generic/src
09:09:49 DEBUG| [stdout] FSTYP -- zfs
09:09:49 DEBUG| [stdout] PLATFORM -- Linux/s390x s2lp6g004 4.12.0-12-generic
09:09:49 DEBUG| [stdout] MKFS_OPTIONS -- testpool/scratch
09:09:49 DEBUG| [stdout] MOUNT_OPTIONS -- testpool/scratch /mnt/scratch
09:09:49 DEBUG| [stdout]
Connection to s2lp6g004 closed by remote host.

ubuntu@s2lp6g004:~$ [ 1471.034292] Out of memory: Kill process 4199 (python) score 24 or sacrifice child
[ 1471.034298] Killed process 4493 (sh) total-vm:2268kB, anon-rss:0kB, file-rss:504kB, shmem-rss:0kB
[ 1471.067172] Out of memory: Kill process 4199 (python) score 24 or sacrifice child
[ 1471.067176] Killed process 4199 (python) total-vm:122940kB, anon-rss:0kB, file-rss:1420kB, shmem-rss:0kB
[ 1471.069927] Out of memory: Kill process 4496 (autotest-local) score 8 or sacrifice child
[ 1471.069931] Killed process 14411 (autotest-local) total-vm:21872kB, anon-rss:192kB, file-rss:1008kB, shmem-rss:0kB
[ 1471.082995] Out of memory: Kill process 4496 (autotest-local) score 8 or sacrifice child
[ 1471.082997] Killed process 4497 (autotest-local) total-vm:21872kB, anon-rss:0kB, file-rss:360kB, shmem-rss:0kB
[ 1471.104838] Out of memory: Kill process 4496 (autotest-local) score 8 or sacrifice child
[ 1471.104842] Killed process 4498 (autotest-local) total-vm:21872kB, anon-rss:0kB, file-rss:360kB, shmem-rss:0kB
[ 1471.132957] Out of memory: Kill process 4496 (autotest-local) score 8 or sacrifice child
[ 1471.132960] Killed process 4496 (autotest-local) total-vm:21872kB, anon-rss:0kB, file-rss:3144kB, shmem-rss:0kB
[ 1471.154586] Out of memory: Kill process 14413 (autotest-local) score 7 or sacrifice child
[ 1471.154588] Killed process 14413 (autotest-local) total-vm:21872kB, anon-rss:0kB, file-rss:360kB, shmem-rss:0kB
[ 1471.174634] Out of memory: Kill process 14414 (autotest-local) score 7 or sacrifice child
[ 1471.174637] Killed process 14414 (autotest-local) total-vm:21872kB, anon-rss:0kB, file-rss:360kB, shmem-rss:0kB
[ 1471.177964] Out of memory: Kill process 2482 (ovs-vswitchd) score 3 or sacrifice child
[ 1471.177966] Killed process 2482 (ovs-vswitchd) total-vm:7448kB, anon-rss:856kB, file-rss:6412kB, shmem-rss:0kB
[ 1471.208758] Out of memory: Kill process 4148 (sshd) score 2 or sacrifice child
[ 1471.208760] Killed process 4198 (sshd) total-vm:10648kB, anon-rss:0kB, file-rss:2032kB, shmem-rss:0kB
[ 1471.303038] Out of memory: Kill process 14807 (check) score 2 or sacrifice child
[ 1471.303041] Killed process 15112 (069) total-vm:5832kB, anon-rss:0kB, file-rss:1316kB, shmem-rss:0kB
[ 1471.332361] Out of memory: Kill process 14807 (check) score 2 or sacrifice child
[ 1471.332364] Killed process 14807 (check) total-vm:5820kB, anon-rss:0kB, file-rss:1192kB, shmem-rss:0kB
[ 1471.352704] Out of memory: Kill process 3407 (bash) score 1 or sacrifice child
[ 1471.352706] Killed process 3407 (bash) total-vm:5404kB, anon-rss:0kB, file-rss:1444kB, shmem-rss:0kB
[ 1471.407768] Out of memory: Kill process 3386 (systemd) score 1 or sacrifice child
[ 1471.407772] Killed process 3387 ((sd-pam)) total-vm:13084kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 1471.427874] Out of memory: Kill process 3386 (systemd) score 1 or sacrifice child
[ 1471.427877] Killed process 3386 (systemd) total-vm:11252kB, anon-rss:0kB, file-rss:2204kB, shmem-rss:0kB
[ 1471.448535] Out of memory: Kill process 2390 (polkitd) score 1 or sacrifice child
[ 1471.448541] Killed process 2390 (polkitd) total-vm:231004kB, anon-rss:0kB, file-rss:1856kB, shmem-rss:0kB
[ 1471.464663] Out of memory: Kill process 2343 (accounts-daemon) score 1 or sacrifice child
[ 1471.464667] Killed process 2343 (accounts-daemon) total-vm:230840kB, anon-rss:0kB, file-rss:1796kB, shmem-rss:0kB
[ 1471.493818] Out of memory: Kill process 4148 (sshd) score 1 or sacrifice child
[ 1471.493820] Killed process 4148 (sshd) total-vm:10648kB, anon-rss:0kB, file-rss:1572kB, shmem-rss:0kB
[ 1471.512756] Out of memory: Kill process 2345 (rsyslogd) score 1 or sacrifice child
[ 1471.512763] Killed process 2345 (rsyslogd) total-vm:225636kB, anon-rss:0kB, file-rss:1632kB, shmem-rss:0kB
[ 1471.641242] Out of memory: Kill process 2646 (systemd-resolve) score 1 or sacrifice child
[ 1471.641249] Killed process 2646 (systemd-resolve) total-vm:9072kB, anon-rss:0kB, file-rss:1820kB, shmem-rss:0kB
[ 1471.641423] Out of memory: Kill process 2335 (systemd-logind) score 1 or sacrifice child
[ 1471.641426] Killed process 2335 (systemd-logind) total-vm:8948kB, anon-rss:0kB, file-rss:1860kB, shmem-rss:0kB
[ 1471.671121] Out of memory: Kill process 2826 (login) score 1 or sacrifice child
[ 1471.671123] Killed process 2826 (login) total-vm:4308kB, anon-rss:0kB, file-rss:1836kB, shmem-rss:0kB
[ 1471.751204] Out of memory: Kill process 704 (systemd-journal) score 1 or sacrifice child
[ 1471.751208] Killed process 704 (systemd-journal) total-vm:27008kB, anon-rss:0kB, file-rss:1720kB, shmem-rss:0kB
[ 1471.751308] Out of memory: Kill process 4495 (sudo) score 1 or sacrifice child
[ 1471.751311] Killed process 4495 (sudo) total-vm:4668kB, anon-rss:0kB, file-rss:1644kB, shmem-rss:0kB
[ 1471.771307] Out of memory: Kill process 2169 (rpcbind) score 1 or sacrifice child
[ 1471.771310] Killed process 2169 (rpcbind) total-vm:6528kB, anon-rss:0kB, file-rss:1596kB, shmem-rss:0kB
[ 1471.791131] Out of memory: Kill process 2380 (irqbalance) score 1 or sacrifice child
[ 1471.791136] Killed process 2380 (irqbalance) total-vm:6452kB, anon-rss:0kB, file-rss:1716kB, shmem-rss:0kB
[ 1471.861210] Out of memory: Kill process 2408 (ovsdb-server) score 1 or sacrifice child
[ 1471.861212] Killed process 2408 (ovsdb-server) total-vm:5484kB, anon-rss:0kB, file-rss:1276kB, shmem-rss:0kB
[ 1471.861288] Out of memory: Kill process 3199 (exim4) score 1 or sacrifice child
[ 1471.861291] Killed process 3199 (exim4) total-vm:12224kB, anon-rss:0kB, file-rss:1072kB, shmem-rss:0kB
[ 1471.863078] Out of memory: Kill process 2339 (cron) score 1 or sacrifice child
[ 1471.863081] Killed process 2339 (cron) total-vm:3192kB, anon-rss:0kB, file-rss:1644kB, shmem-rss:0kB
[ 1471.881184] Out of memory: Kill process 2168 (systemd-timesyn) score 1 or sacrifice child
[ 1471.881189] Killed process 2168 (systemd-timesyn) total-vm:82908kB, anon-rss:0kB, file-rss:1144kB, shmem-rss:0kB
[ 1471.885203] Out of memory: Kill process 2491 (systemd-network) score 1 or sacrifice child
[ 1471.885207] Killed process 2491 (systemd-network) total-vm:12316kB, anon-rss:0kB, file-rss:1476kB, shmem-rss:0kB
[ 1471.911136] Out of memory: Kill process 2596 (rpc.mountd) score 1 or sacrifice child
[ 1471.911139] Killed process 2596 (rpc.mountd) total-vm:5060kB, anon-rss:0kB, file-rss:1096kB, shmem-rss:0kB
[ 1471.941127] Out of memory: Kill process 2772 (iscsid) score 0 or sacrifice child
[ 1471.941133] Killed process 2772 (iscsid) total-vm:4444kB, anon-rss:40kB, file-rss:1488kB, shmem-rss:12kB
[ 1471.971190] Out of memory: Kill process 2344 (atd) score 0 or sacrifice child
[ 1471.971194] Killed process 2344 (atd) total-vm:3168kB, anon-rss:0kB, file-rss:1452kB, shmem-rss:0kB
[ 1471.991217] Out of memory: Kill process 2341 (zed) score 0 or sacrifice child
[ 1471.991220] Killed process 2341 (zed) total-vm:6652kB, anon-rss:0kB, file-rss:1436kB, shmem-rss:0kB
[ 1471.994305] Out of memory: Kill process 14419 (ubuntu_zfs_xfs_) score 0 or sacrifice child
[ 1471.994307] Killed process 14419 (ubuntu_zfs_xfs_) total-vm:3480kB, anon-rss:0kB, file-rss:1316kB, shmem-rss:0kB
[ 1472.031186] Out of memory: Kill process 3155 (dnsmasq) score 0 or sacrifice child
[ 1472.031190] Killed process 3155 (dnsmasq) total-vm:6932kB, anon-rss:0kB, file-rss:1044kB, shmem-rss:0kB
[ 1472.061128] Out of memory: Kill process 722 (blkmapd) score 0 or sacrifice child
[ 1472.061131] Killed process 722 (blkmapd) total-vm:3948kB, anon-rss:0kB, file-rss:1304kB, shmem-rss:0kB
[ 1472.081206] Out of memory: Kill process 2160 (rpc.idmapd) score 0 or sacrifice child
[ 1472.081211] Killed process 2160 (rpc.idmapd) total-vm:2992kB, anon-rss:0kB, file-rss:1116kB, shmem-rss:0kB
[ 1472.081428] Out of memory: Kill process 2346 (lxcfs) score 0 or sacrifice child
[ 1472.081433] Killed process 2346 (lxcfs) total-vm:150496kB, anon-rss:0kB, file-rss:840kB, shmem-rss:0kB
[ 1472.086279] Out of memory: Kill process 2407 (monitor) score 0 or sacrifice child
[ 1472.086281] Killed process 2407 (monitor) total-vm:5220kB, anon-rss:0kB, file-rss:708kB, shmem-rss:0kB
[ 1472.101168] Out of memory: Kill process 730 (lvmetad) score 0 or sacrifice child
[ 1472.101172] Killed process 730 (lvmetad) total-vm:77712kB, anon-rss:0kB, file-rss:636kB, shmem-rss:0kB
[ 1472.104798] Out of memory: Kill process 2481 (monitor) score 0 or sacrifice child
[ 1472.104800] Killed process 2481 (monitor) total-vm:7284kB, anon-rss:0kB, file-rss:116kB, shmem-rss:0kB
[ 1472.151128] Out of memory: Kill process 15366 (append_writer) score 0 or sacrifice child
[ 1472.151130] Killed process 15366 (append_writer) total-vm:936kB, anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[ 1472.181124] Out of memory: Kill process 2348 (dbus-daemon) score 0 or sacrifice child
[ 1472.181126] Killed process 2348 (dbus-daemon) total-vm:5936kB, anon-rss:0kB, file-rss:1780kB, shmem-rss:0kB
[ 1472.191147] Out of memory: Kill process 2379 (snapd) score 0 or sacrifice child
[ 1472.191155] Killed process 2379 (snapd) total-vm:135952kB, anon-rss:0kB, file-rss:472kB, shmem-rss:0kB
[ 1472.212067] Out of memory: Kill process 2773 (iscsid) score 0 or sacrifice child
[ 1472.212070] Killed process 2773 (iscsid) total-vm:4948kB, anon-rss:1244kB, file-rss:3540kB, shmem-rss:24kB
[ 1472.223452] Kernel panic - not syncing: Out of memory and no killable processes...
[ 1472.223452]
[ 1472.223454] CPU: 0 PID: 2428 Comm: snapd Tainted: P O 4.12.0-12-generic #13-Ubuntu
[ 1472.223455] Hardware name: IBM 2964 N63 400 (KVM/Linux)
[ 1472.223456] Stack:
[ 1472.223456] 0000000029e47420 0000000029e474b0 0000000000000003 0000000000000000
[ 1472.223458] 0000000029e47550 0000000029e474c8 0000000029e474c8 0000000000000020
[ 1472.223459] 0000000000dd7dc0 0000000000000040 0000000033bf3710 000000000000000a
[ 1472.223460] 000000000000000c 0000000029e47518 0000000000000000 0000000000000000
[ 1472.223462] 0000000000875e38 000000000011319e 0000000029e474b0 0000000029e474f8
[ 1472.223463] Call Trace:
[ 1472.223464] ([<000000000011305a>] show_trace+0x62/0x78)
[ 1472.223465] [<0000000000113140>] show_stack+0x60/0xd8
[ 1472.223467] [<00000000005a0502>] dump_stack+0x82/0xb0
[ 1472.223468] [<00000000002a7efc>] panic+0x104/0x240
[ 1472.223469] [<00000000002b11f2>] out_of_memory+0x3aa/0x578
[ 1472.223470] [<00000000002b794e>] __alloc_pages_nodemask+0xefe/0xf90
[ 1472.223472] [<000000000031d0cc>] alloc_pages_vma+0xd4/0x2f8
[ 1472.223473] [<00000000002ebbe2>] wp_page_copy+0x47a/0x6f8
[ 1472.223474] [<00000000002edcae>] do_wp_page+0xa6/0x690
[ 1472.223475] [<00000000002f18a8>] __handle_mm_fault+0xbf0/0x11f0
[ 1472.223476] [<00000000002f1f7a>] handle_mm_fault+0xd2/0x1a0
[ 1472.223477] [<0000000000121d32>] do_protection_exception+0x212/0x488
[ 1472.223479] [<0000000000864f54>] pgm_check_handler+0x150/0x1a4
[ 1472.223480] [<000000000016a078>] mm_release+0xe0/0x160
[ 1472.223481] ([<0000000000169ff2>] mm_release+0x5a/0x160)
[ 1472.223482] [<000000000017037e>] do_exit+0x16e/0xbd0
[ 1472.223483] [<0000000000170ea0>] do_group_exit+0x50/0xe8
[ 1472.223484] [<000000000017e6e8>] get_signal+0x358/0x638
[ 1472.223485] [<00000000001079d8>] do_signal+0x60/0x678
[ 1472.223486] [<0000000000864d00>] system_call+0x1a8/0x27c

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: linux-image-4.12.0-12-generic 4.12.0-12.13
ProcVersionSignature: Ubuntu 4.12.0-12.13-generic 4.12.8
Uname: Linux 4.12.0-12-generic s390x
NonfreeKernelModules: zfs zunicode zavl zcommon znvpair
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.6-0ubuntu7
Architecture: s390x
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Tue Aug 29 09:00:53 2017
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1:
PciMultimedia:

ProcEnviron:
 TERM=vt220
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C
 SHELL=/bin/bash
ProcFB: Error: [Errno 2] No such file or directory: '/proc/fb'
ProcKernelCmdLine: root=UUID=d86dc8de-6ac8-4e4b-aba8-92872a209a8c crashkernel=196M
RelatedPackageVersions:
 linux-restricted-modules-4.12.0-12-generic N/A
 linux-backports-modules-4.12.0-12-generic N/A
 linux-firmware 1.167
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1713653

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Po-Hsu Lin (cypressyew)]

This issue does not exist after expanding the memory from 1G to 2G.
Thus I'm closing this.