| 2019-06-24 10:28:28 |
TJ |
description |
18.04 amd64, 4.15.0-50-lowlatency.
The system has a number of local autofs mounts created and managed by systemd through .automount units.
Earlier today I had finished working on one such ("SourceCode") and it had auto-unmounted after the idle time. A short time after that I tried to use that auto-mount directory again ("/home/tj/SourceCode") and the shell hung. I tried from several more shells with different commands with the same result.
Initially couldn't see any clues in the logs but after a while the kernel dumped a couple of stack traces of which I include the first here.
After the stack-traces appeared the shell was able to report an error for other auto-mounts that haven't hung but trying to access the problematic "/home/tj/SourceCode" still hangs the shell, and trying to manually mount the affected file-system to the same location also hangs but works for another mount-point:
ls: cannot access '/home/tj/Projects/': Host is down
Jun 24 11:01:02 T300CHI kernel: INFO: task ls:29630 blocked for more than 120 seconds.
Jun 24 11:01:02 T300CHI kernel: Tainted: G OE 4.15.0-50-lowlatency #54-Ubuntu
Jun 24 11:01:02 T300CHI kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 24 11:01:02 T300CHI kernel: Call Trace:
Jun 24 11:01:02 T300CHI kernel: __schedule+0x293/0x8a0
Jun 24 11:01:02 T300CHI kernel: schedule+0x33/0x90
Jun 24 11:01:02 T300CHI kernel: schedule_timeout+0x1e6/0x380
Jun 24 11:01:02 T300CHI kernel: ? autofs4_wait+0x451/0x870 [autofs4]
Jun 24 11:01:02 T300CHI kernel: wait_for_completion+0xac/0x130
Jun 24 11:01:02 T300CHI kernel: ? wake_up_q+0x80/0x80
Jun 24 11:01:02 T300CHI kernel: autofs4_expire_wait+0xa0/0x160 [autofs4]
Jun 24 11:01:02 T300CHI kernel: do_expire_wait+0x155/0x210 [autofs4]
Jun 24 11:01:02 T300CHI kernel: ? __d_alloc+0x122/0x1d0
Jun 24 11:01:02 T300CHI kernel: autofs4_d_manage+0x76/0x1a0 [autofs4]
Jun 24 11:01:02 T300CHI kernel: follow_managed+0xce/0x310
Jun 24 11:01:02 T300CHI kernel: lookup_fast+0x115/0x340
Jun 24 11:01:02 T300CHI kernel: walk_component+0x49/0x470
Jun 24 11:01:02 T300CHI kernel: path_lookupat+0x84/0x1f0
Jun 24 11:01:02 T300CHI kernel: filename_lookup+0xb6/0x190
Jun 24 11:01:02 T300CHI kernel: ? mem_cgroup_commit_charge+0x82/0x520
Jun 24 11:01:02 T300CHI kernel: ? __check_object_size+0xaf/0x1b0
Jun 24 11:01:02 T300CHI kernel: ? strncpy_from_user+0x4d/0x170
Jun 24 11:01:02 T300CHI kernel: user_path_at_empty+0x36/0x40
Jun 24 11:01:02 T300CHI kernel: ? user_path_at_empty+0x36/0x40
Jun 24 11:01:02 T300CHI kernel: vfs_statx+0x76/0xe0
Jun 24 11:01:02 T300CHI kernel: SYSC_newstat+0x3d/0x70
Jun 24 11:01:02 T300CHI kernel: ? __do_page_fault+0x292/0x520
Jun 24 11:01:02 T300CHI kernel: SyS_newstat+0xe/0x10
Jun 24 11:01:02 T300CHI kernel: do_syscall_64+0x73/0x130
Jun 24 11:01:02 T300CHI kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Jun 24 11:01:02 T300CHI kernel: RIP: 0033:0x7f4a02814775
Jun 24 11:01:02 T300CHI kernel: RSP: 002b:00007fff7c274b98 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
Jun 24 11:01:02 T300CHI kernel: RAX: ffffffffffffffda RBX: 000055fb8dd08640 RCX: 00007f4a02814775
Jun 24 11:01:02 T300CHI kernel: RDX: 000055fb8dd08658 RSI: 000055fb8dd08658 RDI: 00007fff7c276d68
Jun 24 11:01:02 T300CHI kernel: RBP: 00007fff7c274f40 R08: 0000000000000000 R09: 00007fff7c276d72
Jun 24 11:01:02 T300CHI kernel: R10: 000000007c276d00 R11: 0000000000000246 R12: 00007fff7c276d68
Jun 24 11:01:02 T300CHI kernel: R13: 0000000000000001 R14: 0000000000000000 R15: 000055fb8dd08658 |
18.04 amd64, 4.15.0-50-lowlatency.
The system has a number of local autofs mounts created and managed by systemd through .automount units.
Earlier today I had finished working on one such ("SourceCode") and it had auto-unmounted after the idle time. A short time after that I tried to use that auto-mount directory again ("/home/tj/SourceCode") and the shell hung. I tried from several more shells with different commands with the same result.
Initially couldn't see any clues in the logs but after a while the kernel dumped a couple of stack traces of which I include the first here.
After the stack-traces appeared the shell was able to report an error for other auto-mounts that haven't hung but trying to access the problematic "/home/tj/SourceCode" still hangs the shell, and trying to manually mount the affected file-system to the same location also hangs but works for another mount-point:
ls: cannot access '/home/tj/Projects/': Host is down
In trying to clean up I manually stopped all the systemd .mount/.automount units. As soon as I stopped the home-tj-SourceCode.automount unit the mount succeeded and the shells un-hung.
Not sure if this is caused by the systemd autofs handling or there was some race condition behind it, but it seems to have recovered.
Jun 24 11:01:02 T300CHI kernel: INFO: task ls:29630 blocked for more than 120 seconds.
Jun 24 11:01:02 T300CHI kernel: Tainted: G OE 4.15.0-50-lowlatency #54-Ubuntu
Jun 24 11:01:02 T300CHI kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jun 24 11:01:02 T300CHI kernel: Call Trace:
Jun 24 11:01:02 T300CHI kernel: __schedule+0x293/0x8a0
Jun 24 11:01:02 T300CHI kernel: schedule+0x33/0x90
Jun 24 11:01:02 T300CHI kernel: schedule_timeout+0x1e6/0x380
Jun 24 11:01:02 T300CHI kernel: ? autofs4_wait+0x451/0x870 [autofs4]
Jun 24 11:01:02 T300CHI kernel: wait_for_completion+0xac/0x130
Jun 24 11:01:02 T300CHI kernel: ? wake_up_q+0x80/0x80
Jun 24 11:01:02 T300CHI kernel: autofs4_expire_wait+0xa0/0x160 [autofs4]
Jun 24 11:01:02 T300CHI kernel: do_expire_wait+0x155/0x210 [autofs4]
Jun 24 11:01:02 T300CHI kernel: ? __d_alloc+0x122/0x1d0
Jun 24 11:01:02 T300CHI kernel: autofs4_d_manage+0x76/0x1a0 [autofs4]
Jun 24 11:01:02 T300CHI kernel: follow_managed+0xce/0x310
Jun 24 11:01:02 T300CHI kernel: lookup_fast+0x115/0x340
Jun 24 11:01:02 T300CHI kernel: walk_component+0x49/0x470
Jun 24 11:01:02 T300CHI kernel: path_lookupat+0x84/0x1f0
Jun 24 11:01:02 T300CHI kernel: filename_lookup+0xb6/0x190
Jun 24 11:01:02 T300CHI kernel: ? mem_cgroup_commit_charge+0x82/0x520
Jun 24 11:01:02 T300CHI kernel: ? __check_object_size+0xaf/0x1b0
Jun 24 11:01:02 T300CHI kernel: ? strncpy_from_user+0x4d/0x170
Jun 24 11:01:02 T300CHI kernel: user_path_at_empty+0x36/0x40
Jun 24 11:01:02 T300CHI kernel: ? user_path_at_empty+0x36/0x40
Jun 24 11:01:02 T300CHI kernel: vfs_statx+0x76/0xe0
Jun 24 11:01:02 T300CHI kernel: SYSC_newstat+0x3d/0x70
Jun 24 11:01:02 T300CHI kernel: ? __do_page_fault+0x292/0x520
Jun 24 11:01:02 T300CHI kernel: SyS_newstat+0xe/0x10
Jun 24 11:01:02 T300CHI kernel: do_syscall_64+0x73/0x130
Jun 24 11:01:02 T300CHI kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Jun 24 11:01:02 T300CHI kernel: RIP: 0033:0x7f4a02814775
Jun 24 11:01:02 T300CHI kernel: RSP: 002b:00007fff7c274b98 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
Jun 24 11:01:02 T300CHI kernel: RAX: ffffffffffffffda RBX: 000055fb8dd08640 RCX: 00007f4a02814775
Jun 24 11:01:02 T300CHI kernel: RDX: 000055fb8dd08658 RSI: 000055fb8dd08658 RDI: 00007fff7c276d68
Jun 24 11:01:02 T300CHI kernel: RBP: 00007fff7c274f40 R08: 0000000000000000 R09: 00007fff7c276d72
Jun 24 11:01:02 T300CHI kernel: R10: 000000007c276d00 R11: 0000000000000246 R12: 00007fff7c276d68
Jun 24 11:01:02 T300CHI kernel: R13: 0000000000000001 R14: 0000000000000000 R15: 000055fb8dd08658 |
|
