virtual machine locks up when accessing mounted share

[Jason Grammenos]

I have a virtual machine that locks up every time any file access is done to the mounted ocsfs2 shared volume. the volume is backed by vmware multiwriter virtual disk.

I do not have a lot of diagnostics other than the various dmesg logs

it seems to be an ocfs2 issue or at least ocfs2 is "crashing"
when this happens the ssh session locks up and is un usable.
and as a result of monitoring services that try to read disk status the cpu load goes high
attempts to reboot the vm during this state sometimes result in

user@hostname:~$ sudo shutdown -r now
Failed to start reboot.target: Connection timed out
See system logs and 'systemctl status reboot.target' for details.

lsb_release -rd
Description: Ubuntu 16.04.7 LTS
Release: 16.04

apt-cache policy ocfs2-tools
ocfs2-tools:
  Installed: 1.6.4-3.1

number #1

[Sep 4 14:33] BUG: unable to handle kernel paging request at ffffffffc0504088
[ +0.000101] IP: [<ffffffff8186955e>] _raw_spin_lock+0xe/0x70
[ +0.000100] PGD 1e0f067 PUD 1e11067 PMD 33756067 PTE 778f3061
[ +0.000048] Oops: 0003 [#1] SMP
[ +0.000027] Modules linked in: ocfs2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue mptctl configfs vmw_balloon joydev input_leds serio_raw i2c_piix4 shpchp mac_hid vmw_vsock_vmci_transport vsock vmw_vmci ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse mptspi mptscsih mptbase scsi_transport_spi vmwgfx ahci ttm drm_kms_helper syscopyarea vmxnet3 sysfillrect sysimgblt fb_sys_fops libahci drm pata_acpi fjes
[ +0.000591] CPU: 0 PID: 5568 Comm: touch Not tainted 4.4.0-189-generic #219-Ubuntu
[ +0.000049] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ +0.000067] task: ffff880034f25580 ti: ffff8800767c4000 task.ti: ffff8800767c4000
[ +0.000047] RIP: 0010:[<ffffffff8186955e>] [<ffffffff8186955e>] _raw_spin_lock+0xe/0x70
[ +0.000057] RSP: 0018:ffff8800767c7998 EFLAGS: 00010286
[ +0.000035] RAX: 0000000000020000 RBX: ffffffffc0504000 RCX: 00000000ffffffff
[ +0.000045] RDX: 0000000080000000 RSI: 0000000000000009 RDI: ffffffffc0504088
[ +0.000044] RBP: ffff8800767c7998 R08: 0000000000000000 R09: 0000000000000000
[ +0.000045] R10: ffff880078ea5f00 R11: 0000000000008000 R12: ffffffffc0504088
[ +0.000044] R13: ffff880077a63aa8 R14: ffff880077a510d0 R15: 00000000ffffffff
[ +0.000047] FS: 00007fe66bc9e700(0000) GS:ffff88007b600000(0000) knlGS:0000000000000000
[ +0.000050] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ +0.000117] CR2: ffffffffc0504088 CR3: 00000000763c0000 CR4: 0000000000160670
[ +0.000203] Stack:
[ +0.000018] ffff8800767c79b8 ffffffff8123953e 0000000000000009 ffff880077a63000
[ +0.000055] ffff8800767c7a38 ffffffffc0582b77 ffffffffc053a09a 0000000000000000
[ +0.000054] ffff8800784b7c98 ffff8800784b7ce0 0000000100000000 ffff8800784b7bc8
[ +0.000054] Call Trace:
[ +0.000084] [<ffffffff8123953e>] igrab+0x1e/0x50
[ +0.000393] [<ffffffffc0582b77>] ocfs2_get_system_file_inode+0x67/0x300 [ocfs2]
[ +0.001444] [<ffffffffc053a09a>] ? ocfs2_read_virt_blocks+0x1ca/0x310 [ocfs2]
[ +0.001374] [<ffffffffc0527830>] ? dx_leaf_sort_swap+0x40/0x40 [ocfs2]
[ +0.001347] [<ffffffffc056b43e>] ocfs2_reserve_suballoc_bits+0x4e/0x430 [ocfs2]
[ +0.001329] [<ffffffffc056b8bb>] ocfs2_steal_resource+0x9b/0x130 [ocfs2]
[ +0.001313] [<ffffffffc056bd15>] ocfs2_reserve_new_inode+0xb5/0x370 [ocfs2]
[ +0.001294] [<ffffffffc054ec5e>] ocfs2_mknod+0x4be/0x10e0 [ocfs2]
[ +0.001296] [<ffffffffc05313a3>] ? ocfs2_should_refresh_lock_res+0x113/0x160 [ocfs2]
[ +0.001300] [<ffffffffc054fa1c>] ocfs2_create+0x4c/0x160 [ocfs2]
[ +0.001279] [<ffffffff8122b43d>] vfs_create+0x12d/0x190
[ +0.001790] [<ffffffff8122dba2>] path_openat+0x1232/0x1360
[ +0.001198] [<ffffffff8122f8a1>] do_filp_open+0x91/0x100
[ +0.001217] [<ffffffff811d6d19>] ? do_mmap+0x309/0x480
[ +0.001132] [<ffffffff8123d657>] ? __alloc_fd+0xc7/0x190
[ +0.001110] [<ffffffff8121da28>] do_sys_open+0x138/0x2a0
[ +0.001099] [<ffffffff8106eeb1>] ? __do_page_fault+0x1c1/0x410
[ +0.001043] [<ffffffff8121dbae>] SyS_open+0x1e/0x20
[ +0.001029] [<ffffffff8186989b>] entry_SYSCALL_64_fastpath+0x22/0xd0
[ +0.001036] Code: e2 fe 8d 72 02 0f b7 f6 0f 1f 80 00 00 00 00 eb d2 66 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 b8 00 00 02 00 <f0> 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 41 89 d0 0f b7
[ +0.003175] RIP [<ffffffff8186955e>] _raw_spin_lock+0xe/0x70
[ +0.001009] RSP <ffff8800767c7998>
[ +0.000996] CR2: ffffffffc0504088
[ +0.002732] ---[ end trace c3584abd4a8a5250 ]---

#2

[Sep 3 04:09] general protection fault: 0000 [#1] SMP
[ +0.000346] Modules linked in: mptctl ocfs2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs vmw_balloon joydev input_leds serio_raw shpchp i2c_piix4 vmw_vsock_vmci_transport vsock vmw_vmci mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse vmwgfx ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci vmxnet3 libahci mptspi mptscsih drm pata_acpi mptbase scsi_transport_spi fjes
[ +0.001534] CPU: 0 PID: 1553 Comm: java Not tainted 4.4.0-189-generic #219-Ubuntu
[ +0.000049] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ +0.000067] task: ffff880073863900 ti: ffff8800798e0000 task.ti: ffff8800798e0000
[ +0.000047] RIP: 0010:[<ffffffff8186955e>] [<ffffffff8186955e>] _raw_spin_lock+0xe/0x70
[ +0.000086] RSP: 0018:ffff8800798e3998 EFLAGS: 00010206
[ +0.000035] RAX: 0000000000020000 RBX: 6d612d7972616e69 RCX: 00000000ffffffff
[ +0.000046] RDX: 0000000080000000 RSI: 0000000000000009 RDI: 6d612d7972616ef1
[ +0.000045] RBP: ffff8800798e3998 R08: 0000000000000000 R09: 0000000000000000
[ +0.000045] R10: ffff880075729060 R11: 0000000000008000 R12: 6d612d7972616ef1
[ +0.000044] R13: ffff8800748aeaa8 R14: ffff880076399650 R15: 00000000ffffffff
[ +0.000046] FS: 00007f9a50ad9700(0000) GS:ffff88007b600000(0000) knlGS:0000000000000000
[ +0.000051] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ +0.000037] CR2: 00007f9a1d4f2000 CR3: 0000000079936000 CR4: 0000000000160670
[ +0.000160] Stack:
[ +0.000019] ffff8800798e39b8 ffffffff8123953e 0000000000000009 ffff8800748ae000
[ +0.000055] ffff8800798e3a38 ffffffffc057bb77 ffffffffc053309a 0000000000000000
[ +0.000054] ffff88007958e498 ffff88007958e4e0 0000000100000000 ffff88007958e3c8
[ +0.000055] Call Trace:
[ +0.000025] [<ffffffff8123953e>] igrab+0x1e/0x50
[ +0.000084] [<ffffffffc057bb77>] ocfs2_get_system_file_inode+0x67/0x300 [ocfs2]
[ +0.000088] [<ffffffffc053309a>] ? ocfs2_read_virt_blocks+0x1ca/0x310 [ocfs2]
[ +0.000072] [<ffffffffc0520830>] ? dx_leaf_sort_swap+0x40/0x40 [ocfs2]
[ +0.000075] [<ffffffffc056443e>] ocfs2_reserve_suballoc_bits+0x4e/0x430 [ocfs2]
[ +0.000074] [<ffffffffc05648bb>] ocfs2_steal_resource+0x9b/0x130 [ocfs2]
[ +0.001659] [<ffffffffc0564d15>] ocfs2_reserve_new_inode+0xb5/0x370 [ocfs2]
[ +0.001326] [<ffffffffc0547c5e>] ocfs2_mknod+0x4be/0x10e0 [ocfs2]
[ +0.001622] [<ffffffffc052a3a3>] ? ocfs2_should_refresh_lock_res+0x113/0x160 [ocfs2]
[ +0.001571] [<ffffffffc0548a1c>] ocfs2_create+0x4c/0x160 [ocfs2]
[ +0.001318] [<ffffffff8122b43d>] vfs_create+0x12d/0x190
[ +0.001218] [<ffffffff8122dba2>] path_openat+0x1232/0x1360
[ +0.001255] [<ffffffff8123a44c>] ? file_update_time+0xcc/0x110
[ +0.001167] [<ffffffff8122f8a1>] do_filp_open+0x91/0x100
[ +0.001132] [<ffffffff8121e3d5>] ? new_sync_write+0xa5/0xf0
[ +0.001131] [<ffffffff8123d657>] ? __alloc_fd+0xc7/0x190
[ +0.001072] [<ffffffff8121da28>] do_sys_open+0x138/0x2a0
[ +0.001040] [<ffffffff8121dbae>] SyS_open+0x1e/0x20
[ +0.001028] [<ffffffff8186989b>] entry_SYSCALL_64_fastpath+0x22/0xd0
[ +0.001041] Code: e2 fe 8d 72 02 0f b7 f6 0f 1f 80 00 00 00 00 eb d2 66 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 b8 00 00 02 00 <f0> 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 41 89 d0 0f b7
[ +0.003174] RIP [<ffffffff8186955e>] _raw_spin_lock+0xe/0x70
[ +0.001007] RSP <ffff8800798e3998>
[ +0.002867] ---[ end trace 6fd21d2a8e763939 ]---

#3

[Sep 4 03:56] (java,5847,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000098] (java,5847,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000047] (java,5847,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000046] (java,5847,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000049] (java,5847,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000040] (java,5847,1):ocfs2_create:679 ERROR: status = -22
[ +8.987127] (java,2088,0):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000093] (java,2088,0):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000146] (java,2088,0):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000047] (java,2088,0):ocfs2_mknod:314 ERROR: status = -22
[ +0.000040] (java,2088,0):ocfs2_mknod:499 ERROR: status = -22
[ +0.000039] (java,2088,0):ocfs2_create:679 ERROR: status = -22
[ +9.060788] (java,2090,0):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000065] (java,2090,0):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000062] (java,2090,0):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000047] (java,2090,0):ocfs2_mknod:314 ERROR: status = -22
[ +0.000039] (java,2090,0):ocfs2_mknod:499 ERROR: status = -22
[ +0.000039] (java,2090,0):ocfs2_create:679 ERROR: status = -22
[Sep 4 03:57] (java,2088,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000060] (java,2088,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000045] (java,2088,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000044] (java,2088,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000039] (java,2088,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000037] (java,2088,1):ocfs2_create:679 ERROR: status = -22
[ +15.482181] (java,2088,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000095] (java,2088,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000157] (java,2088,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000162] (java,2088,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000148] (java,2088,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000148] (java,2088,1):ocfs2_create:679 ERROR: status = -22
[Sep 4 04:13] (java,5848,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000097] (java,5848,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000191] (java,5848,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000902] (java,5848,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000771] (java,5848,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000726] (java,5848,1):ocfs2_create:679 ERROR: status = -22
[ +18.618350] (java,5846,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000767] (java,5846,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000645] (java,5846,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000639] (java,5846,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000650] (java,5846,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000631] (java,5846,1):ocfs2_create:679 ERROR: status = -22
[Sep 4 04:14] (java,5848,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000717] (java,5848,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000671] (java,5848,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000677] (java,5848,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000677] (java,5848,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000692] (java,5848,1):ocfs2_create:679 ERROR: status = -22
[ +6.859084] (java,5849,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.001405] (java,5849,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.001280] (java,5849,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.001289] (java,5849,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.001327] (java,5849,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.001296] (java,5849,1):ocfs2_create:679 ERROR: status = -22
[ +15.674889] (java,5848,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000767] (java,5848,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000693] (java,5848,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000651] (java,5848,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000641] (java,5848,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000680] (java,5848,1):ocfs2_create:679 ERROR: status = -22
[ +8.095135] (java,5849,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000872] (java,5849,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000666] (java,5849,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000655] (java,5849,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000648] (java,5849,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000645] (java,5849,1):ocfs2_create:679 ERROR: status = -22
[Sep 4 04:17] (java,2071,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.001019] (java,2071,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000387] (java,2071,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000353] (java,2071,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000345] (java,2071,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000343] (java,2071,1):ocfs2_create:679 ERROR: status = -22
[ +36.327665] (java,2076,0):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000794] (java,2076,0):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000658] (java,2076,0):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000653] (java,2076,0):ocfs2_mknod:314 ERROR: status = -22
[ +0.000716] (java,2076,0):ocfs2_mknod:499 ERROR: status = -22
[ +0.000652] (java,2076,0):ocfs2_create:679 ERROR: status = -22
[Sep 4 04:18] (java,2077,0):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000434] (java,2077,0):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000346] (java,2077,0):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000337] (java,2077,0):ocfs2_mknod:314 ERROR: status = -22
[ +0.000349] (java,2077,0):ocfs2_mknod:499 ERROR: status = -22
[ +0.000333] (java,2077,0):ocfs2_create:679 ERROR: status = -22
[ +18.813455] (java,2069,1):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000750] (java,2069,1):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000706] (java,2069,1):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000705] (java,2069,1):ocfs2_mknod:314 ERROR: status = -22
[ +0.000642] (java,2069,1):ocfs2_mknod:499 ERROR: status = -22
[ +0.000674] (java,2069,1):ocfs2_create:679 ERROR: status = -22
[Sep 4 04:19] (java,2071,0):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000777] (java,2071,0):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000658] (java,2071,0):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000650] (java,2071,0):ocfs2_mknod:314 ERROR: status = -22
[ +0.000657] (java,2071,0):ocfs2_mknod:499 ERROR: status = -22
[ +0.000644] (java,2071,0):ocfs2_create:679 ERROR: status = -22
[ +34.510080] (java,2072,0):ocfs2_reserve_suballoc_bits:796 ERROR: status = -22
[ +0.000811] (java,2072,0):ocfs2_reserve_new_inode:1120 ERROR: status = -22
[ +0.000676] (java,2072,0):ocfs2_reserve_new_inode:1132 ERROR: status = -22
[ +0.000663] (java,2072,0):ocfs2_mknod:314 ERROR: status = -22
[ +0.000654] (java,2072,0):ocfs2_mknod:499 ERROR: status = -22
[ +0.000686] (java,2072,0):ocfs2_create:679 ERROR: status = -22
[Sep 4 07:12] BUG: unable to handle kernel paging request at ffffffffc046a088
[ +0.000540] IP: [<ffffffff8186955e>] _raw_spin_lock+0xe/0x70
[ +0.000472] PGD 1e0f067 PUD 1e11067 PMD 79abe067 PTE 77bc9061
[ +0.000475] Oops: 0003 [#1] SMP
[ +0.000455] Modules linked in: mptctl ocfs2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue configfs vmw_balloon joydev input_leds serio_raw shpchp i2c_piix4 vmw_vsock_vmci_transport vsock vmw_vmci mac_hid ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse vmwgfx ttm drm_kms_helper syscopyarea sysfillrect sysimgblt ahci fb_sys_fops drm mptspi mptscsih vmxnet3 mptbase libahci scsi_transport_spi pata_acpi fjes
[ +0.004701] CPU: 0 PID: 2090 Comm: java Not tainted 4.4.0-189-generic #219-Ubuntu
[ +0.000653] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
[ +0.001325] task: ffff880078e0d580 ti: ffff88007792c000 task.ti: ffff88007792c000
[ +0.000695] RIP: 0010:[<ffffffff8186955e>] [<ffffffff8186955e>] _raw_spin_lock+0xe/0x70
[ +0.000730] RSP: 0018:ffff88007792f998 EFLAGS: 00010286
[ +0.000738] RAX: 0000000000020000 RBX: ffffffffc046a000 RCX: 00000000ffffffff
[ +0.000731] RDX: 0000000080000000 RSI: 0000000000000009 RDI: ffffffffc046a088
[ +0.000741] RBP: ffff88007792f998 R08: 0000000000000000 R09: 0000000000000000
[ +0.000747] R10: ffff880078f15c00 R11: 0000000000008000 R12: ffffffffc046a088
[ +0.000745] R13: ffff8800768e8aa8 R14: ffff8800748bcc50 R15: 00000000ffffffff
[ +0.000763] FS: 00007fd5da2ae700(0000) GS:ffff88007b600000(0000) knlGS:0000000000000000
[ +0.000804] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ +0.000779] CR2: ffffffffc046a088 CR3: 0000000078e2e000 CR4: 0000000000160670
[ +0.000868] Stack:
[ +0.000799] ffff88007792f9b8 ffffffff8123953e 0000000000000009 ffff8800768e8000
[ +0.000811] ffff88007792fa38 ffffffffc0594b77 ffffffffc054c09a 0000000000000000
[ +0.000850] ffff880079193c98 ffff880079193ce0 0000000100000000 ffff880079193bc8
[ +0.000851] Call Trace:
[ +0.000832] [<ffffffff8123953e>] igrab+0x1e/0x50
[ +0.001077] [<ffffffffc0594b77>] ocfs2_get_system_file_inode+0x67/0x300 [ocfs2]
[ +0.000884] [<ffffffffc054c09a>] ? ocfs2_read_virt_blocks+0x1ca/0x310 [ocfs2]
[ +0.000866] [<ffffffffc0539830>] ? dx_leaf_sort_swap+0x40/0x40 [ocfs2]
[ +0.000928] [<ffffffffc057d43e>] ocfs2_reserve_suballoc_bits+0x4e/0x430 [ocfs2]
[ +0.000934] [<ffffffffc057d8bb>] ocfs2_steal_resource+0x9b/0x130 [ocfs2]
[ +0.000896] [<ffffffffc057dd15>] ocfs2_reserve_new_inode+0xb5/0x370 [ocfs2]
[ +0.000886] [<ffffffffc0560c5e>] ocfs2_mknod+0x4be/0x10e0 [ocfs2]
[ +0.000891] [<ffffffffc05433a3>] ? ocfs2_should_refresh_lock_res+0x113/0x160 [ocfs2]
[ +0.000924] [<ffffffffc0561a1c>] ocfs2_create+0x4c/0x160 [ocfs2]
[ +0.000860] [<ffffffff8122b43d>] vfs_create+0x12d/0x190
[ +0.000834] [<ffffffff8122dba2>] path_openat+0x1232/0x1360
[ +0.000812] [<ffffffff810bcb53>] ? update_curr+0xe3/0x170
[ +0.000787] [<ffffffff8122f8a1>] do_filp_open+0x91/0x100
[ +0.000785] [<ffffffff818649dd>] ? __schedule+0x30d/0x810
[ +0.000753] [<ffffffff818649d1>] ? __schedule+0x301/0x810
[ +0.000723] [<ffffffff8123d657>] ? __alloc_fd+0xc7/0x190
[ +0.000732] [<ffffffff8121da28>] do_sys_open+0x138/0x2a0
[ +0.000692] [<ffffffff81864f15>] ? schedule+0x35/0x80
[ +0.000672] [<ffffffff8121dbae>] SyS_open+0x1e/0x20
[ +0.000636] [<ffffffff8186989b>] entry_SYSCALL_64_fastpath+0x22/0xd0
[ +0.000640] Code: e2 fe 8d 72 02 0f b7 f6 0f 1f 80 00 00 00 00 eb d2 66 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 b8 00 00 02 00 <f0> 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 41 89 d0 0f b7
[ +0.002030] RIP [<ffffffff8186955e>] _raw_spin_lock+0xe/0x70
[ +0.000658] RSP <ffff88007792f998>
[ +0.000639] CR2: ffffffffc046a088
[ +0.001869] ---[ end trace a0b6053f39d0bb33 ]---

[Paride Legovini (paride)]

Hello Jason and thanks for this bug report. This actually looks like a kernel issue rather than a bug in the userspace ocfs2-tools. I added a "linux" (kernel) task to reflect this, and set the ocfs2-tools to Incomplete for the moment.

[Jason Grammenos (jason.grammenos.agility)]

at the time when i filed the bug I had not yet determined that it was a kernel issue. I am now sure, and there is a proper kernel bug filed for the issue:

https://bugs.launchpad.net/ubuntu/+source/linux-signed/+bug/1895010,

close, marked as duplicate, etc. Do as you see fit.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1894247

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Jason Grammenos (jason.grammenos.agility)]

I cannot run then command as the vm where the issue occurred is no longer running the problematic kernel. And yes this appears by all intents and purpose to be a kernel level regression