| 2022-04-22 04:44:04 |
Po-Hsu Lin |
description |
Issue found on kernel04 with kernel 5.4.0-110-generic
Test passed, but the test will gets killed by the autotest timeout setting (2100 seconds) in the end.
Summary:
Stressors run: 229
Skipped: 3, binderfs pci smi
Failed: 0,
Oopsed: 0,
Oomed: 0,
Passed: 226, access af-alg affinity aio aiol alarm bad-altstack bad-ioctl bigheap branch brk cache cap chattr chdir chmod chown chroot clock close context cpu crypt cyclic daemon dccp dentry dev dev-shm dir dirdeep dirmany dnotify dup dynlib enosys env epoll eventfd exit-group fallocate fanotify fault fcntl fiemap fifo file-ioctl filename flock fork fp-error fpunch fstat full funcret futex get getdent getrandom goto handle hash hdd hrtimers icache icmp-flood inode-flags inotify io iomix ioprio io-uring ipsec-mb itimer judy key kill klog kvm landlock lease link list loadavg locka lockbus lockf lockofd loop madvise malloc mcontend membarrier memfd memhotplug memrate memthrash mergesort mincore misaligned mknod mlock mmap mmapaddr mmapfixed mmapfork mmaphuge mmapmany mq mremap msg msync munmap nanosleep netdev netlink-proc netlink-task nice null open pageswap personality physpage pidfd ping-sock pipe pipeherd pkey poll prctl prefetch procfs pthread ptrace pty radixsort randlist ramfs rawdev rawpkt rawsock rawudp readahead reboot rename resched revio rlimit rmap rseq rtc schedpolicy sctp seal seccomp secretmem seek sem sem-sysv sendfile session set shellsort shm shm-sysv sigabrt sigchld sigfd sigfpe sigio signal signest sigpending sigpipe sigq sigrt sigsegv sigsuspend sigtrap skiplist sleep sock sockabuse sockdiag sockmany softlockup sparsematrix splice stackmmap stream swap switch symlink sync-file syncload sysbadaddr sysfs tee timer timerfd tlb-shootdown tmpfs tree tsearch tun udp udp-flood unshare urandom userfaultfd usersyscall utime vdso vecwide verity vfork vm vm-addr vm-rw vm-segv vm-splice wait x86syscall yield zero zombie
Badret: 0,
Tests took 471 seconds to run
There are some stress-ng related process in D state:
$ ps aux | grep stress
root 2247 0.0 0.0 27280 984 pts/1 S 02:28 0:00 /usr/bin/python2 -u autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_stress_smoke_test/control
root 2248 0.0 0.0 27280 984 pts/1 S 02:28 0:00 /usr/bin/python2 -u autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_stress_smoke_test/control
root 152197 0.0 0.5 46080 21204 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152196 6 0 1 152196 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152201 0.0 0.7 46080 27428 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152200 6 0 1 152200 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152377 0.0 0.7 46080 27384 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152376 6 0 1 152376 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152407 0.0 0.6 46080 27020 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152406 6 0 1 152406 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152431 0.0 0.7 46080 28092 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152430 6 0 1 152430 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152505 0.0 0.5 36364 21268 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152504 6 0 1 152504 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 191513 0.0 0.0 9480 964 pts/1 D 02:37 0:00 swapoff -a /home/ubuntu/autotest/client/tmp/ubuntu_stress_smoke_test/src/stress-ng/swap.img
Looks like it's because of the swapoff command went wrong, I can see the following trace in dmesg:
[ 664.537025] ------------[ cut here ]------------
[ 664.537029] kernel BUG at mm/zswap.c:896!
[ 664.537079] illegal operation: 0001 ilc:1 [#1] SMP
[ 664.537083] Modules linked in: sctp vhost_net tap vhost_vsock vmw_vsock_virtio_transport_common vhost vsock zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) dccp_ipv4 dccp tgr192 streebog_generic sm3_generic sha3_generic rmd320 rmd256 rmd160 rmd128 nhpoly1305 poly1305_generic michael_mic md4 cmac ccm algif_rng twofish_generic twofish_common tea sm4_generic serpent_generic seed khazad fcrypt des_generic cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis algif_skcipher algif_hash aegis128 algif_aead af_alg nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat nf_tables nfnetlink ip6table_filter ip6_tables aufs iptable_mangle xt_CHECKSUM iptable_nat xt_MASQUERADE xt_tcpudp bridge stp llc iptable_filter bpfilter openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 binfmt_misc dm_multipath scsi_dh_rdac
[ 664.537139] scsi_dh_emc scsi_dh_alua vmur vfio_ccw vfio_mdev mdev vfio_iommu_type1 vfio sch_fq_codel nfsd drm drm_panel_orientation_quirks auth_rpcgss nfs_acl lockd i2c_core grace sunrpc ip_tables x_tables btrfs zstd_compress zlib_deflate raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear pkey zcrypt crc32_vx_s390 ghash_s390 prng aes_s390 des_s390 libdes qeth_l2 sha512_s390 sha256_s390 sha1_s390 qeth sha_common dasd_eckd_mod qdio dasd_mod ccwgroup
[ 664.537198] CPU: 2 PID: 152087 Comm: apport Tainted: P O 5.4.0-110-generic #124-Ubuntu
[ 664.537200] Hardware name: IBM 2964 N63 400 (z/VM 6.4.0)
[ 664.537202] Krnl PSW : 0704c00180000000 0000000034e6beba (zswap_writeback_entry+0x32a/0x340)
[ 664.537209] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[ 664.537212] Krnl GPRS: 0000000000000003 00000000e597a200 ffffffffffffffea 0000000000000316
[ 664.537213] 0000000000000cea 0000000000000000 000003e0ffffffea 00000000f1a60328
[ 664.537215] 000003d080a46940 00000000f1a60320 000000000000c410 000000008d7c8b98
[ 664.537216] 00000000e597a200 000000009a353b00 0000000034e6bd78 000003e00e4630a8
[ 664.537225] Krnl Code: 0000000034e6beac: c0e50032c4aa brasl %r14,00000000354c4800
0000000034e6beb2: a7f4ffaa brc 15,0000000034e6be06
#0000000034e6beb6: a7f40001 brc 15,0000000034e6beb8
>0000000034e6beba: ec81ffff00d9 aghik %r8,%r1,-1
0000000034e6bec0: a7f4fefe brc 15,0000000034e6bcbc
0000000034e6bec4: 0707 bcr 0,%r7
0000000034e6bec6: 0707 bcr 0,%r7
0000000034e6bec8: 0707 bcr 0,%r7
[ 664.537240] Call Trace:
[ 664.537242] ([<0000000034e6bd78>] zswap_writeback_entry+0x1e8/0x340)
[ 664.537246] [<0000000034eab55c>] zbud_reclaim_page+0x14c/0x2b0
[ 664.537248] [<0000000034eab700>] zbud_zpool_shrink+0x40/0x80
[ 664.537249] [<0000000034e6c162>] zswap_frontswap_store+0x292/0x5c0
[ 664.537252] [<0000000034e6a494>] __frontswap_store+0xb4/0x1a0
[ 664.537255] [<0000000034e618aa>] swap_writepage+0x5a/0xc0
[ 664.537259] [<0000000034e07bf8>] pageout.isra.0+0x118/0x3e0
[ 664.537261] [<0000000034e0bcec>] shrink_page_list+0xa3c/0xe90
[ 664.537264] [<0000000034e0cd40>] shrink_inactive_list+0x1f0/0x4e0
[ 664.537266] [<0000000034e0d934>] shrink_node_memcg+0x2f4/0x3f0
[ 664.537268] [<0000000034e0daf4>] shrink_node+0xc4/0x4d0
[ 664.537270] [<0000000034e0dfde>] do_try_to_free_pages+0xde/0x400
[ 664.537272] [<0000000034e0e3ee>] try_to_free_pages+0xee/0x230
[ 664.537274] [<0000000034e5af7c>] __alloc_pages_slowpath+0x31c/0xe90
[ 664.537276] [<0000000034e5bda4>] __alloc_pages_nodemask+0x2b4/0x330
[ 664.537279] [<0000000034e77e2a>] alloc_pages_vma+0x9a/0x280
[ 664.537281] [<0000000034e62ad4>] __read_swap_cache_async+0x194/0x270
[ 664.537283] [<0000000034e62bda>] read_swap_cache_async+0x2a/0x60
[ 664.537285] [<0000000034e62e3a>] swap_cluster_readahead+0x22a/0x2e0
[ 664.537287] [<0000000034e6331e>] swapin_readahead+0x2ce/0x410
[ 664.537292] [<0000000034e37594>] do_swap_page+0x1f4/0x880
[ 664.537294] [<0000000034e3938e>] __handle_mm_fault+0x7ee/0x910
[ 664.537296] [<0000000034e39576>] handle_mm_fault+0xc6/0x180
[ 664.537299] [<0000000034c11910>] do_dat_exception+0x120/0x3d0
[ 664.537305] [<00000000354d75de>] pgm_check_handler+0x1da/0x230
[ 664.537306] Last Breaking-Event-Address:
[ 664.537307] [<0000000034e6beb6>] zswap_writeback_entry+0x326/0x340
[ 664.537310] ---[ end trace 2f637439fb06e842 ]--- |
Issue found on kernel04 with kernel 5.4.0-110-generic
Test passed, but the test will gets killed by the autotest timeout setting (2100 seconds) in the end.
Summary:
Stressors run: 229
Skipped: 3, binderfs pci smi
Failed: 0,
Oopsed: 0,
Oomed: 0,
Passed: 226, access af-alg affinity aio aiol alarm bad-altstack bad-ioctl bigheap branch brk cache cap chattr chdir chmod chown chroot clock close context cpu crypt cyclic daemon dccp dentry dev dev-shm dir dirdeep dirmany dnotify dup dynlib enosys env epoll eventfd exit-group fallocate fanotify fault fcntl fiemap fifo file-ioctl filename flock fork fp-error fpunch fstat full funcret futex get getdent getrandom goto handle hash hdd hrtimers icache icmp-flood inode-flags inotify io iomix ioprio io-uring ipsec-mb itimer judy key kill klog kvm landlock lease link list loadavg locka lockbus lockf lockofd loop madvise malloc mcontend membarrier memfd memhotplug memrate memthrash mergesort mincore misaligned mknod mlock mmap mmapaddr mmapfixed mmapfork mmaphuge mmapmany mq mremap msg msync munmap nanosleep netdev netlink-proc netlink-task nice null open pageswap personality physpage pidfd ping-sock pipe pipeherd pkey poll prctl prefetch procfs pthread ptrace pty radixsort randlist ramfs rawdev rawpkt rawsock rawudp readahead reboot rename resched revio rlimit rmap rseq rtc schedpolicy sctp seal seccomp secretmem seek sem sem-sysv sendfile session set shellsort shm shm-sysv sigabrt sigchld sigfd sigfpe sigio signal signest sigpending sigpipe sigq sigrt sigsegv sigsuspend sigtrap skiplist sleep sock sockabuse sockdiag sockmany softlockup sparsematrix splice stackmmap stream swap switch symlink sync-file syncload sysbadaddr sysfs tee timer timerfd tlb-shootdown tmpfs tree tsearch tun udp udp-flood unshare urandom userfaultfd usersyscall utime vdso vecwide verity vfork vm vm-addr vm-rw vm-segv vm-splice wait x86syscall yield zero zombie
Badret: 0,
Tests took 471 seconds to run
There are some stress-ng related process in D state:
$ ps aux | grep stress
root 2247 0.0 0.0 27280 984 pts/1 S 02:28 0:00 /usr/bin/python2 -u autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_stress_smoke_test/control
root 2248 0.0 0.0 27280 984 pts/1 S 02:28 0:00 /usr/bin/python2 -u autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_stress_smoke_test/control
root 152197 0.0 0.5 46080 21204 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152196 6 0 1 152196 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152201 0.0 0.7 46080 27428 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152200 6 0 1 152200 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152377 0.0 0.7 46080 27384 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152376 6 0 1 152376 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152407 0.0 0.6 46080 27020 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152406 6 0 1 152406 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152431 0.0 0.7 46080 28092 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152430 6 0 1 152430 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 152505 0.0 0.5 36364 21268 ? D 02:34 0:00 /usr/bin/python3 /usr/share/apport/apport 152504 6 0 1 152504 !home!ubuntu!autotest!client!tmp!ubuntu_stress_smoke_test!src!stress-ng!stress-ng
root 191513 0.0 0.0 9480 964 pts/1 D 02:37 0:00 swapoff -a /home/ubuntu/autotest/client/tmp/ubuntu_stress_smoke_test/src/stress-ng/swap.img
Looks like it's because of the swapoff command went wrong, I can see the following trace in dmesg:
[ 664.537025] ------------[ cut here ]------------
[ 664.537029] kernel BUG at mm/zswap.c:896!
[ 664.537079] illegal operation: 0001 ilc:1 [#1] SMP
[ 664.537083] Modules linked in: sctp vhost_net tap vhost_vsock vmw_vsock_virtio_transport_common vhost vsock zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) dccp_ipv4 dccp tgr192 streebog_generic sm3_generic sha3_generic rmd320 rmd256 rmd160 rmd128 nhpoly1305 poly1305_generic michael_mic md4 cmac ccm algif_rng twofish_generic twofish_common tea sm4_generic serpent_generic seed khazad fcrypt des_generic cast6_generic cast5_generic cast_common camellia_generic blowfish_generic blowfish_common anubis algif_skcipher algif_hash aegis128 algif_aead af_alg nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter overlay xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat nf_tables nfnetlink ip6table_filter ip6_tables aufs iptable_mangle xt_CHECKSUM iptable_nat xt_MASQUERADE xt_tcpudp bridge stp llc iptable_filter bpfilter openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 binfmt_misc dm_multipath scsi_dh_rdac
[ 664.537139] scsi_dh_emc scsi_dh_alua vmur vfio_ccw vfio_mdev mdev vfio_iommu_type1 vfio sch_fq_codel nfsd drm drm_panel_orientation_quirks auth_rpcgss nfs_acl lockd i2c_core grace sunrpc ip_tables x_tables btrfs zstd_compress zlib_deflate raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear pkey zcrypt crc32_vx_s390 ghash_s390 prng aes_s390 des_s390 libdes qeth_l2 sha512_s390 sha256_s390 sha1_s390 qeth sha_common dasd_eckd_mod qdio dasd_mod ccwgroup
[ 664.537198] CPU: 2 PID: 152087 Comm: apport Tainted: P O 5.4.0-110-generic #124-Ubuntu
[ 664.537200] Hardware name: IBM 2964 N63 400 (z/VM 6.4.0)
[ 664.537202] Krnl PSW : 0704c00180000000 0000000034e6beba (zswap_writeback_entry+0x32a/0x340)
[ 664.537209] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[ 664.537212] Krnl GPRS: 0000000000000003 00000000e597a200 ffffffffffffffea 0000000000000316
[ 664.537213] 0000000000000cea 0000000000000000 000003e0ffffffea 00000000f1a60328
[ 664.537215] 000003d080a46940 00000000f1a60320 000000000000c410 000000008d7c8b98
[ 664.537216] 00000000e597a200 000000009a353b00 0000000034e6bd78 000003e00e4630a8
[ 664.537225] Krnl Code: 0000000034e6beac: c0e50032c4aa brasl %r14,00000000354c4800
0000000034e6beb2: a7f4ffaa brc 15,0000000034e6be06
#0000000034e6beb6: a7f40001 brc 15,0000000034e6beb8
>0000000034e6beba: ec81ffff00d9 aghik %r8,%r1,-1
0000000034e6bec0: a7f4fefe brc 15,0000000034e6bcbc
0000000034e6bec4: 0707 bcr 0,%r7
0000000034e6bec6: 0707 bcr 0,%r7
0000000034e6bec8: 0707 bcr 0,%r7
[ 664.537240] Call Trace:
[ 664.537242] ([<0000000034e6bd78>] zswap_writeback_entry+0x1e8/0x340)
[ 664.537246] [<0000000034eab55c>] zbud_reclaim_page+0x14c/0x2b0
[ 664.537248] [<0000000034eab700>] zbud_zpool_shrink+0x40/0x80
[ 664.537249] [<0000000034e6c162>] zswap_frontswap_store+0x292/0x5c0
[ 664.537252] [<0000000034e6a494>] __frontswap_store+0xb4/0x1a0
[ 664.537255] [<0000000034e618aa>] swap_writepage+0x5a/0xc0
[ 664.537259] [<0000000034e07bf8>] pageout.isra.0+0x118/0x3e0
[ 664.537261] [<0000000034e0bcec>] shrink_page_list+0xa3c/0xe90
[ 664.537264] [<0000000034e0cd40>] shrink_inactive_list+0x1f0/0x4e0
[ 664.537266] [<0000000034e0d934>] shrink_node_memcg+0x2f4/0x3f0
[ 664.537268] [<0000000034e0daf4>] shrink_node+0xc4/0x4d0
[ 664.537270] [<0000000034e0dfde>] do_try_to_free_pages+0xde/0x400
[ 664.537272] [<0000000034e0e3ee>] try_to_free_pages+0xee/0x230
[ 664.537274] [<0000000034e5af7c>] __alloc_pages_slowpath+0x31c/0xe90
[ 664.537276] [<0000000034e5bda4>] __alloc_pages_nodemask+0x2b4/0x330
[ 664.537279] [<0000000034e77e2a>] alloc_pages_vma+0x9a/0x280
[ 664.537281] [<0000000034e62ad4>] __read_swap_cache_async+0x194/0x270
[ 664.537283] [<0000000034e62bda>] read_swap_cache_async+0x2a/0x60
[ 664.537285] [<0000000034e62e3a>] swap_cluster_readahead+0x22a/0x2e0
[ 664.537287] [<0000000034e6331e>] swapin_readahead+0x2ce/0x410
[ 664.537292] [<0000000034e37594>] do_swap_page+0x1f4/0x880
[ 664.537294] [<0000000034e3938e>] __handle_mm_fault+0x7ee/0x910
[ 664.537296] [<0000000034e39576>] handle_mm_fault+0xc6/0x180
[ 664.537299] [<0000000034c11910>] do_dat_exception+0x120/0x3d0
[ 664.537305] [<00000000354d75de>] pgm_check_handler+0x1da/0x230
[ 664.537306] Last Breaking-Event-Address:
[ 664.537307] [<0000000034e6beb6>] zswap_writeback_entry+0x326/0x340
[ 664.537310] ---[ end trace 2f637439fb06e842 ]---
When this happens, the reboot process will be blocking by the following message for a while:
A stop job is running for /swapfile |
|
