kernel trace found with ubuntu_bpf test on Eoan s390x LPAR

[Po-Hsu Lin]

This trace can be seen right after you started the test on s390x LPAR.

 **********************************************************
 ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
 ** **
 ** trace_printk() being used. Allocating extra memory. **
 ** **
 ** This means that this is a DEBUG kernel and it is **
 ** unsafe for production use. **
 ** **
 ** If you see this message and you are not debugging **
 ** the kernel, report this immediately to your vendor! **
 ** **
 ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
 **********************************************************
 WARNING: CPU: 0 PID: 3159 at mm/page_alloc.c:4702 __alloc_pages_nodemask+0x25e/0x318
 Modules linked in: xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat nf_tables nfnetlink ip6table_filter ip6_tables iptable_mangle xt_CHECKSUM iptable_nat xt_MASQUERADE xt_tcpudp bridge iptable_filter bpfilter aufs overlay 8021q garp stp mrp llc openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 binfmt_misc zfs(PO) zunicode(PO) zavl(PO) icp(PO) zlua(PO) zcommon(PO) znvpair(PO) spl(O) genwqe_card crc_itu_t chsc_sch eadm_sch ctcm fsm vfio_ccw vfio_mdev mdev vfio_iommu_type1 vfio sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables btrfs zstd_compress zlib_deflate raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear dm_service_time pkey zcrypt crc32_vx_s390 ghash_s390 prng mlx4_en mlx4_ib ib_uverbs aes_s390 ib_core ptp pps_core qeth_l2 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common mlx4_core zfcp scsi_transport_fc qeth qdio ccwgroup
  dasd_eckd_mod dasd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath
 CPU: 0 PID: 3159 Comm: test_verifier Tainted: P O 5.3.0-21-generic #22-Ubuntu
 Hardware name: IBM 2964 N63 400 (LPAR)
 Krnl PSW : 0704c00180000000 00000002fb832c9e (__alloc_pages_nodemask+0x25e/0x318)
            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
 Krnl GPRS: 00000003b7efcac0 0000000000040dc0 0000000000040dc0 000000000000000a
            0000000000000000 0000000000000000 000000000000000a 00000002fb6386b8
            00000000003d0900 0000000000000dc0 000003e00b067000 000003e00b067000
            00000003a57a8000 000002aa0d1b28e0 000003e002b6bb28 000003e002b6baa8
 Krnl Code: 00000002fb832c92: a7212000 tmll %r2,8192
            00000002fb832c96: a774ff87 brc 7,2fb832ba4
           #00000002fb832c9a: a7f40001 brc 15,2fb832c9c
           >00000002fb832c9e: a7890000 lghi %r8,0
            00000002fb832ca2: a7f4ff83 brc 15,2fb832ba8
            00000002fb832ca6: a7180000 lhi %r1,0
            00000002fb832caa: a7f4ff1b brc 15,2fb832ae0
            00000002fb832cae: e31003400004 lg %r1,832
 Call Trace:
 ([<000003e002b6bb88>] 0x3e002b6bb88)
  [<00000002fb7ff3a0>] kmalloc_order_trace+0x40/0x100
  [<00000002fb6386b8>] bpf_int_jit_compile+0x70/0x240
  [<00000002fb7828ce>] bpf_prog_select_runtime+0x13e/0x190
  [<00000002fb78614c>] bpf_prog_load+0x2dc/0x6a8
  [<00000002fb786ca6>] __do_sys_bpf+0x78e/0x10c0
  [<00000002fbe82228>] system_call+0xdc/0x2c8
 Last Breaking-Event-Address:
  [<00000002fb832c9a>] __alloc_pages_nodemask+0x25a/0x318
 ---[ end trace 1d8b7179967b64ff ]---

The test itself will get killed with oom-killer later, thus generating a return code 137 for the test.

All the following tests in this test suite will be skipped.

[Po-Hsu Lin (cypressyew)]

This can be reproduced with 5.3.0-20 on the same node as well:

 WARNING: CPU: 1 PID: 2799 at mm/page_alloc.c:4702 __alloc_pages_nodemask+0x25e/0x318
 Modules linked in: xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_mangle ip6table_nat nf_tables nfnetlink ip6table_filter ip6_tables iptable_mangle xt_CHECKSUM iptable_nat xt_MASQUERADE xt_tcpudp bridge iptable_filter bpfilter aufs overlay 8021q garp stp mrp llc openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 binfmt_misc zfs(PO) zunicode(PO) zavl(PO) icp(PO) zlua(PO) zcommon(PO) znvpair(PO) spl(O) genwqe_card crc_itu_t chsc_sch eadm_sch ctcm fsm vfio_ccw vfio_mdev mdev vfio_iommu_type1 vfio sch_fq_codel nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables btrfs zstd_compress zlib_deflate raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 linear dm_service_time pkey zcrypt crc32_vx_s390 ghash_s390 prng aes_s390 mlx4_ib ib_uverbs mlx4_en qeth_l2 des_s390 des_generic ptp pps_core ib_core sha512_s390 sha256_s390 sha1_s390 sha_common mlx4_core zfcp scsi_transport_fc qeth qdio ccwgroup
  dasd_eckd_mod dasd_mod scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath
 CPU: 1 PID: 2799 Comm: test_verifier Tainted: P O 5.3.0-20-generic #21-Ubuntu
 Hardware name: IBM 2964 N63 400 (LPAR)
 Krnl PSW : 0704c00180000000 00000002ec206c9e (__alloc_pages_nodemask+0x25e/0x318)
            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
 Krnl GPRS: da15f4fd64a5a47d 0000000000040dc0 0000000000040dc0 000000000000000a
            0000000000000000 0000000000000000 000000000000000a 00000002ec00c6b8
            00000000003d0900 0000000000000dc0 000003e00a16b000 000003e00a16b000
            00000003a6c20000 000002aa09e798e0 000003e00279bb28 000003e00279baa8
 Krnl Code: 00000002ec206c92: a7212000 tmll %r2,8192
            00000002ec206c96: a774ff87 brc 7,2ec206ba4
           #00000002ec206c9a: a7f40001 brc 15,2ec206c9c
           >00000002ec206c9e: a7890000 lghi %r8,0
            00000002ec206ca2: a7f4ff83 brc 15,2ec206ba8
            00000002ec206ca6: a7180000 lhi %r1,0
            00000002ec206caa: a7f4ff1b brc 15,2ec206ae0
            00000002ec206cae: e31003400004 lg %r1,832
 Call Trace:
 ([<000003e00279bb88>] 0x3e00279bb88)
  [<00000002ec1d33a0>] kmalloc_order_trace+0x40/0x100
  [<00000002ec00c6b8>] bpf_int_jit_compile+0x70/0x240
  [<00000002ec1568ce>] bpf_prog_select_runtime+0x13e/0x190
  [<00000002ec15a14c>] bpf_prog_load+0x2dc/0x6a8
  [<00000002ec15aca6>] __do_sys_bpf+0x78e/0x10c0
  [<00000002ec856228>] system_call+0xdc/0x2c8
 Last Breaking-Event-Address:
  [<00000002ec206c9a>] __alloc_pages_nodemask+0x25a/0x318
 ---[ end trace d7b63e2466552ed4 ]---

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1851829

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Sean Feole (sfeole)]

Sam, can you point me to the last failure we have recorded? I could not find it in the logs.

[Po-Hsu Lin (cypressyew)]

Hi Sean,
there is no report available on the result page, as the job was terminated abnormally.

For the log on jenkins, you can see that it suddenly stopped:
  04:43:31 DEBUG| [stdout] Fork 100 tasks to 'test_hashmap'
  04:43:31 DEBUG| [stdout] Fork 100 tasks to 'test_hashmap_percpu'
  + ARCHIVE=/var/lib/jenkins/jobs/sru-1__E_s390x.LPAR
  generic__using_s2lp4__for_kernel/builds/3/archive
  + scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -r
  ubuntu@s2lp4:kernel-test-results /var/lib/jenkins/jobs/sru-1__E_s390x.LPAR-
  generic__using_s2lp4__for_kernel/builds/3/archive

http://10.246.72.4:8080/view/sut-s2lp4/job/sru-1__E_s390x.LPAR-generic__using_s2lp4__for_kernel/3/consoleText

Or here if the link above is not working: https://pastebin.ubuntu.com/p/jCDvWBXSKw/

This issue can be reproduced if you run this manually.

[Po-Hsu Lin (cypressyew)]

It's the test_map test, see bug 1856163