Comment 7 for bug 1872021

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote : Re: commissioning fails due to hung tasks setting up ipmitool

Summary...

For Ubuntu Bionic, dpkg triggers for systemd (237-3ubuntu10.39) might have caused systemd to hang:

[ 363.776878] wait_for_completion+0xba/0x140
[ 363.776890] __flush_work+0x15b/0x210
[ 363.776901] flush_delayed_work+0x41/0x50
[ 363.776908] fsnotify_wait_marks_destroyed+0x15/0x20
[ 363.776912] fsnotify_destroy_group+0x48/0xd0
[ 363.776917] inotify_release+0x1e/0x50
[ 363.776923] __fput+0xea/0x220
[ 363.776929] ____fput+0xe/0x10
[ 363.776935] task_work_run+0x9d/0xc0
[ 363.776942] exit_to_usermode_loop+0xc0/0xd0
[ 363.776947] do_syscall_64+0x121/0x130
[ 363.776954] entry_SYSCALL_64_after_hwframe+0x3d/0xa2

and

[ 364.050206] wait_for_completion+0xba/0x140
[ 364.050238] __synchronize_srcu.part.13+0x85/0xb0
[ 364.050248] synchronize_srcu+0x66/0xe0
[ 364.050256] fsnotify_mark_destroy_workfn+0x7b/0xe0
[ 364.050262] process_one_work+0x1de/0x420
[ 364.050267] worker_thread+0x228/0x410
[ 364.050272] kthread+0x121/0x140

and

[ 364.326985] wait_for_completion+0xba/0x140
[ 364.326988] __synchronize_srcu.part.13+0x85/0xb0
[ 364.326993] synchronize_srcu+0x66/0xe0
[ 364.326995] ? synchronize_srcu+0x66/0xe0
[ 364.326996] fsnotify_connector_destroy_workfn+0x4a/0x80
[ 364.326998] process_one_work+0x1de/0x420
[ 364.326999] worker_thread+0x253/0x410
[ 364.327001] kthread+0x121/0x140

All stack traces seem to come from "fsnotify" subsystem and waiting on delayed work (completion) for fsnotify marks destruction after a inotify_release() was called. Completion did not happen for the past 2 minutes. Without a kernel dump it is hard to tell if completion was still ok - due to kthread being overloaded doing scheduled work and/or the marks group destruction - or there was a dead lock for the completion due to a kernel bug.

If this is reproducible, I think that having a kernel dump would help identifying the issue. I'm letting the kernel team to handle this and marking all other issues as dealt per previous comments.