StarlingX

kernel Oops (memory corruption) in low latency kernel

Bug #1810973 reported by Wendy Mitchell
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Lin Shuicheng

Bug Description

Brief Description
----------------
kern Tracebacks (errors and alerts) on compute host (compute-3) requiring reboot
Subsequent instantiation schedules but unsuccessful)

Linux version 3.10.0-862.11.6.rt56.819.el7.tis.43.x86_64 (<email address hidden>) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP PREEMPT RT Sat Dec 29 22:01:14 EST 2018

Severity
--------
Normal

Operations ~this time
------------------
Step 1
[2019-01-05 20:35:44,943] Create a flavor with 2 vcpus, 0G ephemera disk, 512M swap disk
 Test Step 2:
[2019-01-05 20:35:49,803]Add following extra specs: {'aggregate_instance_extra_specs:storage': 'remote', 'hw:cpu_policy': 'dedicated'}
Test Step 3:
[2019-01-05 20:35:55,379] Boot a vm (from tis-centos-image feac8876-d3a9-47b0-a4e0-921144cb2fa1)
eg.
[2019-01-05 20:36:04,902] 703 INFO MainThread vm_helper.boot_vm :: nova boot --flavor 5dd0ff1c-f584-42c7-b067-19ce45c756a1 --key-name keypair-tenant2 --image feac8876-d3a9-47b0-a4e0-921144cb2fa1 --nic net-id=ed805453-a34f-4de5-adc1-dbc35a8c8ed5,vif-model=virtio --nic net-id=2802c966-88dc-4c9b-a0b5-5b8893823a6c,vif-model=virtio tenant2-migration_test-8 --poll

Test Step 4:
[2019-01-05 20:39:16,850]Attach volume to vm

Test Step 5:
[2019-01-05 20:39:55,808] Auto mount ephemeral, swap, and attached volume

Test Step 6:
[2019-01-05 20:40:08,120] Create files under vm disks: ['/', 'none', '/mnt/vdc']

Test Step 7:
2019-01-05 20:40:28,357] Live migrate VM (in this example the originating ost is compute-3 and target host is compute-1)

The instance become active on compute-1 and the instance is deleted.

Then, attempted to launch subsequent instance at [2019-01-05 20:42:23,617]
'nova ... boot --flavor 2fa0d844-6721-406e-8cfa-7e289c2ff05f --boot-volume cf49c4aa-eed3-4e60-817d-a6426cab6268 --key-name keypair-tenant2 --nic net-id=ed805453-a34f-4de5-adc1-dbc35a8c8ed5,vif-model=virtio --nic net-id=2802c966-88dc-4c9b-a0b5-5b8893823a6c,vif-model=virtio tenant2-migration_test-10 --poll'

Actual Behavior
----------------
kernel Oops issue starting here (recursive fault)
+ subsequent instance launch attempt passed the scheduler filter and host (compute-3) was selected then rejected here:
2019-01-05 21:13:14.319 76060 INFO nova.utils [req-fa608d01-f058-4c50-87e7-308b2eb34961 237f3c3a0ef94371b17e4ad749ffcccf f044282b0bf9493f8e7d912371e5babd - default default] ComputeFilter: (compute-3) REJECT: host has not been heard from in a while

2019-01-05T20:41:18.322 compute-3 kernel: alert [16424.208325] BUG: unable to handle kernel paging request at ffffffff84331490
2019-01-05T20:41:18.322 compute-3 kernel: alert [16424.208330] IP: [<ffffffff843315cc>] radix_tree_node_alloc+0xac/0xd0
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208331] PGD 53e14067 PUD 53e15063 PMD 532000e1
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208333] Oops: 0003 [#1] PREEMPT SMP
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208353] Modules linked in: cuse fuse xt_REDIRECT nf_nat_redirect ip6table_raw ip6table_mangle xt_nat xt_conntrack xt_mark xt_connmark iptable_raw xt_comment iptable_nat xt_CHECKSUM iptable_mangle nbd dm_mod ebtable_filter ebtables tun openvswitch nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c binfmt_misc vfio_pci virtio_net nfsv3 nfs fscache nfsd auth_rpcgss nfs_acl lockd grace cls_u32 sch_sfq sch_htb ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support sunrpc intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper cryptd joydev lpc_ich mei_me i2c_i801 mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler vfio_iommu_type1 vfio ip_tables ext4 mbcache jbd2 xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) mlx4_en(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci igb i2c_algo_bit i2c_core isci ixgbe(O) dca tpm_tis(O) tpm_tis_core(O) tpm(O) i40e(O) e1000e(O)
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208367] CPU: 0 PID: 214665 Comm: parted Kdump: loaded Tainted: G O ------------ 3.10.0-862.11.6.rt56.819.el7.tis.43.x86_64 #1
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208368] Hardware name: Intel Corporation W2600CR/W2600CR, BIOS SE5C600.86B.02.04.0003.102320141138 10/23/2014
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208369] task: ffff9b18b15c5d00 ti: ffff9b18b3634000 task.ti: ffff9b18b3634000
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208370] RIP: 0010:[<ffffffff843315cc>] [<ffffffff843315cc>] radix_tree_node_alloc+0xac/0xd0
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208371] RSP: 0018:ffff9b18b3637908 EFLAGS: 00010206
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208372] RAX: ffffffff84331480 RBX: 0000000000000020 RCX: 0000000228b7ba0f
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208372] RDX: ffff9b1d9e016800 RSI: 000000000001f5ff RDI: ffff9b1d9d818db0
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208372] RBP: ffff9b18b3637918 R08: 000000000000001b R09: 000000003fffffff
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208373] R10: fffff54f9e225680 R11: 0000000000000000 R12: ffff9b18a5ec2fe8
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208373] R13: 000000000001f5ff R14: ffff9b18a5ec30c8 R15: ffff9b1d9d818db0
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208374] FS: 00007fd442bf8880(0000) GS:ffff9b1d9e000000(0000) knlGS:0000000000000000
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208375] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208375] CR2: ffffffff84331490 CR3: 0000000328d2a000 CR4: 00000000001607f0
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208376] Call Trace:
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208380] [<ffffffff841e6461>] ? __mem_cgroup_commit_charge.constprop.55+0xa1/0x2f0
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208382] [<ffffffff8433229d>] __radix_tree_create+0xcd/0x270
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208385] [<ffffffff841777e3>] page_cache_tree_insert+0x43/0x170
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208386] [<ffffffff841781fe>] __add_to_page_cache_locked+0xae/0x1d0
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208387] [<ffffffff84178377>] add_to_page_cache_lru+0x37/0xb0
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208389] [<ffffffff8423ab77>] mpage_readpages+0xd7/0x170
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208391] [<ffffffff842343c0>] ? I_BDEV+0x10/0x10
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208393] [<ffffffff841cc8a8>] ? alloc_pages_current+0x98/0x110
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208395] [<ffffffff84234ccd>] blkdev_readpages+0x1d/0x20
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208397] [<ffffffff841853f2>] __do_page_cache_readahead+0x1f2/0x280
2019-01-05T20:41:18.322 compute-3 kernel: warning [16424.208398] [<ffffffff84185734>] ondemand_readahead+0x254/0x260
2019-01-05T20:41:24.064 compute-3 kernel: warning [16424.208399] [<ffffffff84185a44>] page_cache_sync_readahead+0x44/0xb0
2019-01-05T20:41:24.064 compute-3 kernel: warning [16424.208400] [<ffffffff84179312>] generic_file_aio_read+0x2c2/0x7b0
2019-01-05T20:41:24.064 compute-3 kernel: warning [16424.208402] [<ffffffff8423511c>] blkdev_aio_read+0x4c/0x70
2019-01-05T20:41:24.064 compute-3 kernel: warning [16424.208404] [<ffffffff841f36c3>] do_sync_read+0x93/0xe0
2019-01-05T20:41:24.064 compute-3 kernel: warning [16424.208405] [<ffffffff841f40ff>] vfs_read+0x9f/0x170
2019-01-05T20:41:24.064 compute-3 kernel: warning [16424.208406] [<ffffffff841f4fcf>] SyS_read+0x7f/0xf0
2019-01-05T20:41:24.064 compute-3 kernel: warning [16424.208407] [<ffffffff841760c3>] ? context_tracking_user_exit+0x13/0x20
2019-01-05T20:41:24.064 compute-3 kernel: warning [16424.208410] [<ffffffff8480129d>] tracesys+0xa3/0xc9
2019-01-05T20:41:24.064 compute-3 kernel: warning [16424.208419] Code: 38 c0 ff ff 83 e1 08 75 31 48 8b 92 38 c0 ff ff 80 e6 02 75 25 48 85 c0 75 9f eb 89 0f 1f 00 48 8b 42 08 48 8b 48 10 48 89 4a 08 <48> c7 40 10 00 00 00 00 83 2a 01 eb b5 0f 0b 48 89 45 f0 e8 3c
2019-01-05T20:41:24.064 compute-3 kernel: alert [16424.208420] RIP [<ffffffff843315cc>] radix_tree_node_alloc+0xac/0xd0
2019-01-05T20:41:24.065 compute-3 kernel: warning [16424.208420] RSP <ffff9b18b3637908>
2019-01-05T20:41:24.065 compute-3 kernel: warning [16424.208421] CR2: ffffffff84331490
2019-01-05T20:41:24.065 compute-3 kernel: warning [16424.637273] ---[ end trace 0000000000000002 ]---
2019-01-05T20:41:24.065 compute-3 kernel: info [16424.708746] note: parted[214665] exited with preempt_count 1
2019-01-05T20:41:24.065 compute-3 kernel: err [16424.708935] BUG: scheduling while atomic: parted/214665/0x10000002
2019-01-05T20:41:24.065 compute-3 kernel: warning [16424.708960] Modules linked in: cuse fuse xt_REDIRECT nf_nat_redirect ip6table_raw ip6table_mangle xt_nat xt_conntrack xt_mark xt_connmark iptable_raw xt_comment iptable_nat xt_CHECKSUM iptable_mangle nbd dm_mod ebtable_filter ebtables tun openvswitch nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c binfmt_misc vfio_pci virtio_net nfsv3 nfs fscache nfsd auth_rpcgss nfs_acl lockd grace cls_u32 sch_sfq sch_htb ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support sunrpc intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper cryptd joydev lpc_ich mei_me i2c_i801 mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler vfio_iommu_type1 vfio ip_tables ext4 mbcache jbd2 xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) mlx4_en(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci igb i2c_algo_bit i2c_core isci ixgbe(O) dca tpm_tis(O) tpm_tis_core(O) tpm(O) i40e(O) e1000e(O)
2019-01-05T20:41:24.065 compute-3 kernel: warning [16424.708978] CPU: 0 PID: 214665 Comm: parted Kdump: loaded Tainted: G D O ------------ 3.10.0-862.11.6.rt56.819.el7.tis.43.x86_64 #1
2019-01-05T20:41:24.065 compute-3 kernel: warning [16424.708978] Hardware name: Intel Corporation W2600CR/W2600CR, BIOS SE5C600.86B.02.04.0003.102320141138 10/23/2014
2019-01-05T20:41:24.065 compute-3 kernel: warning [16424.708979] Call Trace:
2019-01-05T20:41:24.065 compute-3 kernel: warning [16424.708986] [<ffffffff847f4f46>] dump_stack+0x19/0x1b
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.708989] [<ffffffff847ef7da>] __schedule_bug+0x62/0x70
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.708991] [<ffffffff847fccea>] __schedule+0x7ca/0x960
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.708993] [<ffffffff84218184>] ? mntput+0x24/0x40
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.708996] [<ffffffff840bd8c9>] __cond_resched+0x29/0x50
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.708997] [<ffffffff847fd398>] _cond_resched+0x48/0x50
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.708999] [<ffffffff840a8280>] task_work_run+0xc0/0xe0
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709001] [<ffffffff84086184>] do_exit+0x2d4/0xaa0
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709003] [<ffffffff84083047>] ? kmsg_dump+0xd7/0x100
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709005] [<ffffffff84331490>] ? radix_tree_node_rcu_free+0x10/0x40
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709007] [<ffffffff84020b8a>] oops_end+0xaa/0x110
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709009] [<ffffffff847ee7c2>] no_context+0x285/0x2a8
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709030] [<ffffffff84331490>] ? radix_tree_node_rcu_free+0x10/0x40
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709031] [<ffffffff847ee85d>] __bad_area_nosemaphore+0x78/0x1d5
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709032] [<ffffffff847ee090>] ? pud_page_vaddr+0xd/0x45
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709033] [<ffffffff84331490>] ? radix_tree_node_rcu_free+0x10/0x40
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709034] [<ffffffff847ee9ce>] bad_area_nosemaphore+0x14/0x16
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709035] [<ffffffff840621ee>] __do_page_fault+0xbe/0x4b0
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709037] [<ffffffff8417a8a3>] ? mempool_alloc+0x63/0x150
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709038] [<ffffffff8406265f>] do_page_fault+0x3f/0x90
2019-01-05T20:41:24.066 compute-3 kernel: warning [16424.709039] [<ffffffff84800648>] page_fault+0x28/0x30
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709041] [<ffffffff84331480>] ? radix_tree_preload+0x40/0x40
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709041] [<ffffffff843315cc>] ? radix_tree_node_alloc+0xac/0xd0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709044] [<ffffffff841e6461>] ? __mem_cgroup_commit_charge.constprop.55+0xa1/0x2f0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709045] [<ffffffff8433229d>] __radix_tree_create+0xcd/0x270
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709046] [<ffffffff841777e3>] page_cache_tree_insert+0x43/0x170
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709047] [<ffffffff841781fe>] __add_to_page_cache_locked+0xae/0x1d0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709048] [<ffffffff84178377>] add_to_page_cache_lru+0x37/0xb0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709051] [<ffffffff8423ab77>] mpage_readpages+0xd7/0x170
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709052] [<ffffffff842343c0>] ? I_BDEV+0x10/0x10
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709054] [<ffffffff841cc8a8>] ? alloc_pages_current+0x98/0x110
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709056] [<ffffffff84234ccd>] blkdev_readpages+0x1d/0x20
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709057] [<ffffffff841853f2>] __do_page_cache_readahead+0x1f2/0x280
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709058] [<ffffffff84185734>] ondemand_readahead+0x254/0x260
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709059] [<ffffffff84185a44>] page_cache_sync_readahead+0x44/0xb0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709060] [<ffffffff84179312>] generic_file_aio_read+0x2c2/0x7b0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709061] [<ffffffff8423511c>] blkdev_aio_read+0x4c/0x70
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709063] [<ffffffff841f36c3>] do_sync_read+0x93/0xe0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709064] [<ffffffff841f40ff>] vfs_read+0x9f/0x170
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709065] [<ffffffff841f4fcf>] SyS_read+0x7f/0xf0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709066] [<ffffffff841760c3>] ? context_tracking_user_exit+0x13/0x20
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.709067] [<ffffffff8480129d>] tracesys+0xa3/0xc9
2019-01-05T20:41:24.067 compute-3 kernel: alert [16424.746410] BUG: unable to handle kernel paging request at 0000000228b7ba1f
2019-01-05T20:41:24.067 compute-3 kernel: alert [16424.746412] IP: [<ffffffff843315c4>] radix_tree_node_alloc+0xa4/0xd0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.746413] PGD 32b71d067 PUD 0
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.746414] Oops: 0000 [#2] PREEMPT SMP
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.746426] Modules linked in: cuse fuse xt_REDIRECT nf_nat_redirect ip6table_raw ip6table_mangle xt_nat xt_conntrack xt_mark xt_connmark iptable_raw xt_comment iptable_nat xt_CHECKSUM iptable_mangle nbd dm_mod ebtable_filter ebtables tun openvswitch nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c binfmt_misc vfio_pci virtio_net nfsv3 nfs fscache nfsd auth_rpcgss nfs_acl lockd grace cls_u32 sch_sfq sch_htb ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support sunrpc intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper cryptd joydev lpc_ich mei_me i2c_i801 mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler vfio_iommu_type1 vfio ip_tables ext4 mbcache jbd2 xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) mlx4_en(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci igb i2c_algo_bit i2c_core isci ixgbe(O) dca tpm_tis(O) tpm_tis_core(O) tpm(O) i40e(O) e1000e(O)
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.746436] CPU: 0 PID: 214672 Comm: blkid Kdump: loaded Tainted: G D W O ------------ 3.10.0-862.11.6.rt56.819.el7.tis.43.x86_64 #1
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.746436] Hardware name: Intel Corporation W2600CR/W2600CR, BIOS SE5C600.86B.02.04.0003.102320141138 10/23/2014
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.746437] task: ffff9b18a4a6dd00 ti: ffff9b18b0d14000 task.ti: ffff9b18b0d14000
2019-01-05T20:41:24.067 compute-3 kernel: warning [16424.746438] RIP: 0010:[<ffffffff843315c4>] [<ffffffff843315c4>] radix_tree_node_alloc+0xa4/0xd0
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746439] RSP: 0018:ffff9b18b0d17918 EFLAGS: 00010206
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746439] RAX: 0000000228b7ba0f RBX: 0000000000000020 RCX: 0000000000000014
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746440] RDX: ffff9b1d9e016800 RSI: 0000000000000040 RDI: ffff9b1d9d819530
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746440] RBP: ffff9b18b0d17928 R08: 0000000000000005 R09: 000000000000003f
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746440] R10: fffff54f8cc46ac0 R11: 0000000000000000 R12: ffff9b18a5ec1da8
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746441] R13: 0000000000000040 R14: ffff9b18a5ec1dd8 R15: ffff9b1d9d819530
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746442] FS: 00007f8fef6d4780(0000) GS:ffff9b1d9e000000(0000) knlGS:0000000000000000
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746442] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746443] CR2: 0000000228b7ba1f CR3: 0000000328ca8000 CR4: 00000000001607f0
2019-01-05T20:41:24.068 compute-3 kernel: warning [16424.746443] Call Trace:
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746445] [<ffffffff841e6461>] ? __mem_cgroup_commit_charge.constprop.55+0xa1/0x2f0
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746446] [<ffffffff8433229d>] __radix_tree_create+0xcd/0x270
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746448] [<ffffffff841777e3>] page_cache_tree_insert+0x43/0x170
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746449] [<ffffffff841781fe>] __add_to_page_cache_locked+0xae/0x1d0
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746450] [<ffffffff84178377>] add_to_page_cache_lru+0x37/0xb0
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746451] [<ffffffff8423ab77>] mpage_readpages+0xd7/0x170
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746452] [<ffffffff842343c0>] ? I_BDEV+0x10/0x10
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746454] [<ffffffff841cc8a8>] ? alloc_pages_current+0x98/0x110
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746455] [<ffffffff84234ccd>] blkdev_readpages+0x1d/0x20
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746456] [<ffffffff841853f2>] __do_page_cache_readahead+0x1f2/0x280
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746457] [<ffffffff841859c1>] force_page_cache_readahead+0xa1/0xe0
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746458] [<ffffffff84185a97>] page_cache_sync_readahead+0x97/0xb0
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746459] [<ffffffff84179312>] generic_file_aio_read+0x2c2/0x7b0
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746460] [<ffffffff8423511c>] blkdev_aio_read+0x4c/0x70
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746462] [<ffffffff841f36c3>] do_sync_read+0x93/0xe0
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746463] [<ffffffff841f40ff>] vfs_read+0x9f/0x170
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746464] [<ffffffff841f4fcf>] SyS_read+0x7f/0xf0
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746465] [<ffffffff841760c3>] ? context_tracking_user_exit+0x13/0x20
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746466] [<ffffffff8480129d>] tracesys+0xa3/0xc9
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746475] Code: 44 c0 ff ff 01 48 8b 8a 38 c0 ff ff 83 e1 08 75 31 48 8b 92 38 c0 ff ff 80 e6 02 75 25 48 85 c0 75 9f eb 89 0f 1f 00 48 8b 42 08 <48> 8b 48 10 48 89 4a 08 48 c7 40 10 00 00 00 00 83 2a 01 eb b5
2019-01-05T20:41:24.069 compute-3 kernel: alert [16424.746476] RIP [<ffffffff843315c4>] radix_tree_node_alloc+0xa4/0xd0
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746476] RSP <ffff9b18b0d17918>
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746476] CR2: 0000000228b7ba1f
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.746478] ---[ end trace 0000000000000003 ]---
2019-01-05T20:41:24.069 compute-3 kernel: info [16424.815657] note: blkid[214672] exited with preempt_count 1
2019-01-05T20:41:24.069 compute-3 kernel: err [16424.815747] BUG: scheduling while atomic: blkid/214672/0x10000002
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.815771] Modules linked in: cuse fuse xt_REDIRECT nf_nat_redirect ip6table_raw ip6table_mangle xt_nat xt_conntrack xt_mark xt_connmark iptable_raw xt_comment iptable_nat xt_CHECKSUM iptable_mangle nbd dm_mod ebtable_filter ebtables tun openvswitch nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c binfmt_misc vfio_pci virtio_net nfsv3 nfs fscache nfsd auth_rpcgss nfs_acl lockd grace cls_u32 sch_sfq sch_htb ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support sunrpc intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper cryptd joydev lpc_ich mei_me i2c_i801 mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler vfio_iommu_type1 vfio ip_tables ext4 mbcache jbd2 xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) mlx4_en(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci igb i2c_algo_bit i2c_core isci ixgbe(O) dca tpm_tis(O) tpm_tis_core(O) tpm(O) i40e(O) e1000e(O)
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.815789] CPU: 0 PID: 214672 Comm: blkid Kdump: loaded Tainted: G D W O ------------ 3.10.0-862.11.6.rt56.819.el7.tis.43.x86_64 #1
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.815790] Hardware name: Intel Corporation W2600CR/W2600CR, BIOS SE5C600.86B.02.04.0003.102320141138 10/23/2014
2019-01-05T20:41:24.069 compute-3 kernel: warning [16424.815791] Call Trace:
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815797] [<ffffffff847f4f46>] dump_stack+0x19/0x1b
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815799] [<ffffffff847ef7da>] __schedule_bug+0x62/0x70
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815801] [<ffffffff847fccea>] __schedule+0x7ca/0x960
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815804] [<ffffffff84218184>] ? mntput+0x24/0x40
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815806] [<ffffffff840bd8c9>] __cond_resched+0x29/0x50
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815808] [<ffffffff847fd398>] _cond_resched+0x48/0x50
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815810] [<ffffffff840a8280>] task_work_run+0xc0/0xe0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815812] [<ffffffff84086184>] do_exit+0x2d4/0xaa0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815814] [<ffffffff84083047>] ? kmsg_dump+0xd7/0x100
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815816] [<ffffffff84020b8a>] oops_end+0xaa/0x110
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815817] [<ffffffff847ee7c2>] no_context+0x285/0x2a8
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815819] [<ffffffff847ee85d>] __bad_area_nosemaphore+0x78/0x1d5
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815820] [<ffffffff840bc73b>] ? migrate_enable+0xdb/0x210
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815821] [<ffffffff847ee9ce>] bad_area_nosemaphore+0x14/0x16
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815823] [<ffffffff840621ee>] __do_page_fault+0xbe/0x4b0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815826] [<ffffffff8417a8a3>] ? mempool_alloc+0x63/0x150
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815827] [<ffffffff8406265f>] do_page_fault+0x3f/0x90
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815828] [<ffffffff84800648>] page_fault+0x28/0x30
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815831] [<ffffffff843315c4>] ? radix_tree_node_alloc+0xa4/0xd0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815833] [<ffffffff841e6461>] ? __mem_cgroup_commit_charge.constprop.55+0xa1/0x2f0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815834] [<ffffffff8433229d>] __radix_tree_create+0xcd/0x270
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815835] [<ffffffff841777e3>] page_cache_tree_insert+0x43/0x170
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815836] [<ffffffff841781fe>] __add_to_page_cache_locked+0xae/0x1d0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815837] [<ffffffff84178377>] add_to_page_cache_lru+0x37/0xb0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815839] [<ffffffff8423ab77>] mpage_readpages+0xd7/0x170
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815840] [<ffffffff842343c0>] ? I_BDEV+0x10/0x10
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815843] [<ffffffff841cc8a8>] ? alloc_pages_current+0x98/0x110
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815844] [<ffffffff84234ccd>] blkdev_readpages+0x1d/0x20
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815845] [<ffffffff841853f2>] __do_page_cache_readahead+0x1f2/0x280
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815846] [<ffffffff841859c1>] force_page_cache_readahead+0xa1/0xe0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815848] [<ffffffff84185a97>] page_cache_sync_readahead+0x97/0xb0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815849] [<ffffffff84179312>] generic_file_aio_read+0x2c2/0x7b0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815850] [<ffffffff8423511c>] blkdev_aio_read+0x4c/0x70
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815852] [<ffffffff841f36c3>] do_sync_read+0x93/0xe0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815854] [<ffffffff841f40ff>] vfs_read+0x9f/0x170
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815855] [<ffffffff841f4fcf>] SyS_read+0x7f/0xf0
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815856] [<ffffffff841760c3>] ? context_tracking_user_exit+0x13/0x20
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.815857] [<ffffffff8480129d>] tracesys+0xa3/0xc9
2019-01-05T20:41:24.070 compute-3 kernel: err [16424.899166] BUG: scheduling while atomic: blkid/214672/0x10000002
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899178] Modules linked in: cuse fuse xt_REDIRECT nf_nat_redirect ip6table_raw ip6table_mangle xt_nat xt_conntrack xt_mark xt_connmark iptable_raw xt_comment iptable_nat xt_CHECKSUM iptable_mangle nbd dm_mod ebtable_filter ebtables tun openvswitch nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c binfmt_misc vfio_pci virtio_net nfsv3 nfs fscache nfsd auth_rpcgss nfs_acl lockd grace cls_u32 sch_sfq sch_htb ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support sunrpc intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper cryptd joydev lpc_ich mei_me i2c_i801 mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler vfio_iommu_type1 vfio ip_tables ext4 mbcache jbd2 xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) mlx4_en(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci igb i2c_algo_bit i2c_core isci ixgbe(O) dca tpm_tis(O) tpm_tis_core(O) tpm(O) i40e(O) e1000e(O)
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899188] CPU: 0 PID: 214672 Comm: blkid Kdump: loaded Tainted: G D W O ------------ 3.10.0-862.11.6.rt56.819.el7.tis.43.x86_64 #1
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899189] Hardware name: Intel Corporation W2600CR/W2600CR, BIOS SE5C600.86B.02.04.0003.102320141138 10/23/2014
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899189] Call Trace:
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899192] [<ffffffff847f4f46>] dump_stack+0x19/0x1b
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899193] [<ffffffff847ef7da>] __schedule_bug+0x62/0x70
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899195] [<ffffffff847fccea>] __schedule+0x7ca/0x960
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899196] [<ffffffff84218184>] ? mntput+0x24/0x40
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899197] [<ffffffff840bd8c9>] __cond_resched+0x29/0x50
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899198] [<ffffffff847fd398>] _cond_resched+0x48/0x50
2019-01-05T20:41:24.070 compute-3 kernel: warning [16424.899199] [<ffffffff840a8280>] task_work_run+0xc0/0xe0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899201] [<ffffffff84086184>] do_exit+0x2d4/0xaa0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899202] [<ffffffff84083047>] ? kmsg_dump+0xd7/0x100
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899203] [<ffffffff84020b8a>] oops_end+0xaa/0x110
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899204] [<ffffffff847ee7c2>] no_context+0x285/0x2a8
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899205] [<ffffffff847ee85d>] __bad_area_nosemaphore+0x78/0x1d5
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899206] [<ffffffff840bc73b>] ? migrate_enable+0xdb/0x210
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899207] [<ffffffff847ee9ce>] bad_area_nosemaphore+0x14/0x16
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899208] [<ffffffff840621ee>] __do_page_fault+0xbe/0x4b0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899210] [<ffffffff8417a8a3>] ? mempool_alloc+0x63/0x150
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899211] [<ffffffff8406265f>] do_page_fault+0x3f/0x90
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899212] [<ffffffff84800648>] page_fault+0x28/0x30
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899213] [<ffffffff843315c4>] ? radix_tree_node_alloc+0xa4/0xd0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899214] [<ffffffff841e6461>] ? __mem_cgroup_commit_charge.constprop.55+0xa1/0x2f0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899215] [<ffffffff8433229d>] __radix_tree_create+0xcd/0x270
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899216] [<ffffffff841777e3>] page_cache_tree_insert+0x43/0x170
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899217] [<ffffffff841781fe>] __add_to_page_cache_locked+0xae/0x1d0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899218] [<ffffffff84178377>] add_to_page_cache_lru+0x37/0xb0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899219] [<ffffffff8423ab77>] mpage_readpages+0xd7/0x170
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899220] [<ffffffff842343c0>] ? I_BDEV+0x10/0x10
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899222] [<ffffffff841cc8a8>] ? alloc_pages_current+0x98/0x110
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899223] [<ffffffff84234ccd>] blkdev_readpages+0x1d/0x20
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899224] [<ffffffff841853f2>] __do_page_cache_readahead+0x1f2/0x280
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899225] [<ffffffff841859c1>] force_page_cache_readahead+0xa1/0xe0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899226] [<ffffffff84185a97>] page_cache_sync_readahead+0x97/0xb0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899227] [<ffffffff84179312>] generic_file_aio_read+0x2c2/0x7b0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899229] [<ffffffff8423511c>] blkdev_aio_read+0x4c/0x70
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899230] [<ffffffff841f36c3>] do_sync_read+0x93/0xe0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899231] [<ffffffff841f40ff>] vfs_read+0x9f/0x170
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899232] [<ffffffff841f4fcf>] SyS_read+0x7f/0xf0
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899233] [<ffffffff841760c3>] ? context_tracking_user_exit+0x13/0x20
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899234] [<ffffffff8480129d>] tracesys+0xa3/0xc9
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899261] ------------[ cut here ]------------
2019-01-05T20:41:24.071 compute-3 kernel: crit [16424.899261] kernel BUG at kernel/rtmutex.c:966!
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899262] invalid opcode: 0000 [#3] PREEMPT SMP
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899273] Modules linked in: cuse fuse xt_REDIRECT nf_nat_redirect ip6table_raw ip6table_mangle xt_nat xt_conntrack xt_mark xt_connmark iptable_raw xt_comment iptable_nat xt_CHECKSUM iptable_mangle nbd dm_mod ebtable_filter ebtables tun openvswitch nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c binfmt_misc vfio_pci virtio_net nfsv3 nfs fscache nfsd auth_rpcgss nfs_acl lockd grace cls_u32 sch_sfq sch_htb ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support sunrpc intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper cryptd joydev lpc_ich mei_me i2c_i801 mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler vfio_iommu_type1 vfio ip_tables ext4 mbcache jbd2 xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) mlx4_en(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci igb i2c_algo_bit i2c_core isci ixgbe(O) dca tpm_tis(O) tpm_tis_core(O) tpm(O) i40e(O) e1000e(O)
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899282] CPU: 0 PID: 214672 Comm: blkid Kdump: loaded Tainted: G D W O ------------ 3.10.0-862.11.6.rt56.819.el7.tis.43.x86_64 #1
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899283] Hardware name: Intel Corporation W2600CR/W2600CR, BIOS SE5C600.86B.02.04.0003.102320141138 10/23/2014
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899283] task: ffff9b18a4a6dd00 ti: ffff9b18b0d14000 task.ti: ffff9b18b0d14000
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899286] RIP: 0010:[<ffffffff847fde67>] [<ffffffff847fde67>] rt_spin_lock_slowlock+0x357/0x360
2019-01-05T20:41:24.071 compute-3 kernel: warning [16424.899286] RSP: 0018:ffff9b18b0d17230 EFLAGS: 00010046
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899287] RAX: ffff9b18a4a6dd00 RBX: ffff9b1d9d819540 RCX: ffff9b18a4a6dd00
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899287] RDX: 0000000000000000 RSI: ffff9b18a4a6dd00 RDI: ffff9b1d9d819540
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899288] RBP: ffff9b18b0d172c8 R08: ffff9b1d9d819558 R09: ffff9b18a4a6dd01
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899288] R10: ffff9b1d9d819528 R11: 0000000000000001 R12: ffff9b18a4a6dd00
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899289] R13: ffff9b1d9d819530 R14: ffff9b18b0d17250 R15: 0000000000000286
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899289] FS: 00007f8fef6d4780(0000) GS:ffff9b1d9e000000(0000) knlGS:0000000000000000
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899290] CR2: 00007ffea0a1fe90 CR3: 00000003316cc000 CR4: 00000000001607f0
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899291] Call Trace:
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899292] [<ffffffff8480129d>] ? tracesys+0xa3/0xc9
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899294] [<ffffffff847fef65>] rt_spin_lock+0x25/0x30
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899296] [<ffffffff841830cc>] tag_pages_for_writeback+0x3c/0xc0
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899296] [<ffffffff84183c22>] write_cache_pages+0x112/0x530
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899298] [<ffffffff84182eb0>] ? global_dirtyable_memory+0x70/0x70
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899299] [<ffffffff840bdb09>] ? ttwu_do_wakeup+0x19/0x120
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899300] [<ffffffff847ff4ad>] ? _raw_spin_unlock_irqrestore+0x5d/0x70
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899302] [<ffffffff840c141c>] ? try_to_wake_up+0x6c/0x5e0
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899303] [<ffffffff84184090>] generic_writepages+0x50/0x80
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899304] [<ffffffff84234d05>] blkdev_writepages+0x35/0x40
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899304] [<ffffffff84184fc4>] do_writepages+0x24/0x50
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899306] [<ffffffff84178ea5>] __filemap_fdatawrite_range+0x65/0x80
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899307] [<ffffffff84178f30>] filemap_write_and_wait+0x40/0x90
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899308] [<ffffffff842359ff>] __sync_blockdev+0x1f/0x40
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899309] [<ffffffff84235d4c>] __blkdev_put+0x5c/0x1a0
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899310] [<ffffffff842367fe>] blkdev_put+0x4e/0x140
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899311] [<ffffffff842369a5>] blkdev_close+0x25/0x30
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899312] [<ffffffff841f654d>] __fput+0xed/0x270
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899313] [<ffffffff841f67be>] ____fput+0xe/0x10
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899314] [<ffffffff840a827b>] task_work_run+0xbb/0xe0
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899315] [<ffffffff84086184>] do_exit+0x2d4/0xaa0
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899316] [<ffffffff84083047>] ? kmsg_dump+0xd7/0x100
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899318] [<ffffffff84020b8a>] oops_end+0xaa/0x110
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899319] [<ffffffff847ee7c2>] no_context+0x285/0x2a8
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899320] [<ffffffff847ee85d>] __bad_area_nosemaphore+0x78/0x1d5
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899321] [<ffffffff840bc73b>] ? migrate_enable+0xdb/0x210
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899322] [<ffffffff847ee9ce>] bad_area_nosemaphore+0x14/0x16
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899323] [<ffffffff840621ee>] __do_page_fault+0xbe/0x4b0
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899324] [<ffffffff8417a8a3>] ? mempool_alloc+0x63/0x150
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899325] [<ffffffff8406265f>] do_page_fault+0x3f/0x90
2019-01-05T20:41:24.072 compute-3 kernel: warning [16424.899326] [<ffffffff84800648>] page_fault+0x28/0x30
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899327] [<ffffffff843315c4>] ? radix_tree_node_alloc+0xa4/0xd0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899328] [<ffffffff841e6461>] ? __mem_cgroup_commit_charge.constprop.55+0xa1/0x2f0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899329] [<ffffffff8433229d>] __radix_tree_create+0xcd/0x270
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899330] [<ffffffff841777e3>] page_cache_tree_insert+0x43/0x170
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899331] [<ffffffff841781fe>] __add_to_page_cache_locked+0xae/0x1d0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899332] [<ffffffff84178377>] add_to_page_cache_lru+0x37/0xb0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899333] [<ffffffff8423ab77>] mpage_readpages+0xd7/0x170
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899334] [<ffffffff842343c0>] ? I_BDEV+0x10/0x10
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899336] [<ffffffff841cc8a8>] ? alloc_pages_current+0x98/0x110
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899337] [<ffffffff84234ccd>] blkdev_readpages+0x1d/0x20
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899338] [<ffffffff841853f2>] __do_page_cache_readahead+0x1f2/0x280
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899339] [<ffffffff841859c1>] force_page_cache_readahead+0xa1/0xe0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899340] [<ffffffff84185a97>] page_cache_sync_readahead+0x97/0xb0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899341] [<ffffffff84179312>] generic_file_aio_read+0x2c2/0x7b0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899342] [<ffffffff8423511c>] blkdev_aio_read+0x4c/0x70
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899343] [<ffffffff841f36c3>] do_sync_read+0x93/0xe0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899344] [<ffffffff841f40ff>] vfs_read+0x9f/0x170
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899345] [<ffffffff841f4fcf>] SyS_read+0x7f/0xf0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899346] [<ffffffff841760c3>] ? context_tracking_user_exit+0x13/0x20
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899347] [<ffffffff8480129d>] tracesys+0xa3/0xc9
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899356] Code: 1f 44 00 00 e8 db f0 ff ff e9 c4 fd ff ff 66 0f 1f 44 00 00 e8 0b ca 8f ff e8 46 13 88 ff 0f 0b 0f 1f 40 00 e8 bb f0 ff ff eb 9e <0f> 0b 0f 1f 80 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 57 65
2019-01-05T20:41:24.073 compute-3 kernel: alert [16424.899357] RIP [<ffffffff847fde67>] rt_spin_lock_slowlock+0x357/0x360
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899357] RSP <ffff9b18b0d17230>
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.899359] ---[ end trace 0000000000000004 ]---

Reproducibility
---------------
TBD

System Configuration
--------------------
Remote storage

Branch/Pull Time/Commit
-----------------------
2019-01-04_20-18-00

Timestamp/Logs
--------------
see inline

Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :
Download full text (60.5 KiB)

kernel trace con't (and kern.log being attached
2019-01-05T20:41:24.073 compute-3 kernel: alert [16424.970653] Fixing recursive fault but reboot is needed!
2019-01-05T20:41:24.073 compute-3 kernel: err [16424.970656] BUG: scheduling while atomic: blkid/214672/0x00000003
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970682] Modules linked in: cuse fuse xt_REDIRECT nf_nat_redirect ip6table_raw ip6table_mangle xt_nat xt_conntrack xt_mark xt_connmark iptable_raw xt_comment iptable_nat xt_CHECKSUM iptable_mangle nbd dm_mod ebtable_filter ebtables tun openvswitch nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c binfmt_misc vfio_pci virtio_net nfsv3 nfs fscache nfsd auth_rpcgss nfs_acl lockd grace cls_u32 sch_sfq sch_htb ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support sunrpc intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel glue_helper lrw gf128mul ablk_helper cryptd joydev lpc_ich mei_me i2c_i801 mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler vfio_iommu_type1 vfio ip_tables ext4 mbcache jbd2 xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) mlx4_en(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlx4_ib(O) mlx4_core(O) devlink rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_ucm(O) ib_uverbs(O) ib_cm(O) ib_core(O) mlx_compat(O) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci igb i2c_algo_bit i2c_core isci ixgbe(O) dca tpm_tis(O) tpm_tis_core(O) tpm(O) i40e(O) e1000e(O)
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970700] CPU: 0 PID: 214672 Comm: blkid Kdump: loaded Tainted: G D W O ------------ 3.10.0-862.11.6.rt56.819.el7.tis.43.x86_64 #1
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970700] Hardware name: Intel Corporation W2600CR/W2600CR, BIOS SE5C600.86B.02.04.0003.102320141138 10/23/2014
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970701] Call Trace:
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970707] [<ffffffff847f4f46>] dump_stack+0x19/0x1b
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970710] [<ffffffff847ef7da>] __schedule_bug+0x62/0x70
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970712] [<ffffffff847fccea>] __schedule+0x7ca/0x960
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970714] [<ffffffff840828bf>] ? vprintk_default+0x1f/0x30
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970716] [<ffffffff847ef2d1>] ? printk+0x60/0x77
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970717] [<ffffffff847fceb0>] schedule+0x30/0xa0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970718] [<ffffffff840868e6>] do_exit+0xa36/0xaa0
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970719] [<ffffffff84083047>] ? kmsg_dump+0xd7/0x100
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970722] [<ffffffff84020b8a>] oops_end+0xaa/0x110
2019-01-05T20:41:24.073 compute-3 kernel: warning [16424.970723] [<ffffffff84020d2b>] die+0x4b/0x70
2019-01-05T20:41:24.074 compute-3 kernel: warning [16424.97072...

Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

Observation by Jim Somerville

Somerville, Jim added a comment
If you scroll down in the logs to just before the reboot, you'll see jbd2 journalling stuff in the traceback.

This could be:

 [fs] jbd2: fix use after free in jbd2_journal_start_reserved() (Lukas Czerner) [1442044]

Fixed in the 957 kernel.

Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking as release gating; needs investigation by Cindy's kernel/OS team.

Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
assignee: nobody → Bruce Jones (brucej)
tags: added: stx.2019.03 stx.distro.other
Bruce Jones (brucej)
Changed in starlingx:
assignee: Bruce Jones (brucej) → Cindy Xie (xxie1)
Lin Shuicheng (shuicheng)
Changed in starlingx:
assignee: Cindy Xie (xxie1) → Lin Shuicheng (shuicheng)
Ken Young (kenyis)
tags: added: stx.2019.05
removed: stx.2019.03
Revision history for this message
Lin Shuicheng (shuicheng) wrote :

As Jim's comment, the issue should have been fixed with 957 kernel, which is included in CentOS 7.6 upgrade. Let's check the issue again after CentOS 7.6 code merged back to master.

Revision history for this message
Erich Cordoba (ericho) wrote :

Can https://bugs.launchpad.net/starlingx/+bug/1815541 be considered a duplicated of this one?

Revision history for this message
Lin Shuicheng (shuicheng) wrote :

Hi Erich, per Jim's comment, 1815541 is duplicated of 1814595. And 1814595 is different as this one (1810973).
Both issue are jbd2 related in stack, but there are different cause.

Revision history for this message
Lin Shuicheng (shuicheng) wrote :

Mark as fixed since centos76 is merged back to master.

Changed in starlingx:
status: Triaged → Fix Committed
Ghada Khalil (gkhalil)
Changed in starlingx:
status: Fix Committed → Fix Released
Ken Young (kenyis)
tags: added: stx.2.0
removed: stx.2019.05
Ghada Khalil (gkhalil)
tags: added: stx.retestneeded
Revision history for this message
Wendy Mitchell (wmitchellwr) wrote :

retested in BUILD_ID="20190410T013000Z"

tags: removed: stx.retestneeded
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.