All stack traces seem to come from "fsnotify" subsystem and waiting on delayed work (completion) for fsnotify marks destruction after a inotify_release() was called. Completion did not happen for the past 2 minutes. Without a kernel dump it is hard to tell if completion was still ok - due to kthread being overloaded doing scheduled work and/or the marks group destruction - or there was a dead lock for the completion due to a kernel bug.
If this is reproducible, I think that having a kernel dump would help identifying the issue. I'm letting the kernel team to handle this and marking all other issues as dealt per previous comments.
Summary...
For Ubuntu Bionic, dpkg triggers for systemd (237-3ubuntu10.39) might have caused systemd to hang:
[ 363.776878] wait_for_ completion+ 0xba/0x140 work+0x15b/ 0x210 work+0x41/ 0x50 wait_marks_ destroyed+ 0x15/0x20 destroy_ group+0x48/ 0xd0 release+ 0x1e/0x50 run+0x9d/ 0xc0 usermode_ loop+0xc0/ 0xd0 64+0x121/ 0x130 64_after_ hwframe+ 0x3d/0xa2
[ 363.776890] __flush_
[ 363.776901] flush_delayed_
[ 363.776908] fsnotify_
[ 363.776912] fsnotify_
[ 363.776917] inotify_
[ 363.776923] __fput+0xea/0x220
[ 363.776929] ____fput+0xe/0x10
[ 363.776935] task_work_
[ 363.776942] exit_to_
[ 363.776947] do_syscall_
[ 363.776954] entry_SYSCALL_
and
[ 364.050206] wait_for_ completion+ 0xba/0x140 srcu.part. 13+0x85/ 0xb0 srcu+0x66/ 0xe0 mark_destroy_ workfn+ 0x7b/0xe0 one_work+ 0x1de/0x420 thread+ 0x228/0x410
[ 364.050238] __synchronize_
[ 364.050248] synchronize_
[ 364.050256] fsnotify_
[ 364.050262] process_
[ 364.050267] worker_
[ 364.050272] kthread+0x121/0x140
and
[ 364.326985] wait_for_ completion+ 0xba/0x140 srcu.part. 13+0x85/ 0xb0 srcu+0x66/ 0xe0 srcu+0x66/ 0xe0 connector_ destroy_ workfn+ 0x4a/0x80 one_work+ 0x1de/0x420 thread+ 0x253/0x410
[ 364.326988] __synchronize_
[ 364.326993] synchronize_
[ 364.326995] ? synchronize_
[ 364.326996] fsnotify_
[ 364.326998] process_
[ 364.326999] worker_
[ 364.327001] kthread+0x121/0x140
All stack traces seem to come from "fsnotify" subsystem and waiting on delayed work (completion) for fsnotify marks destruction after a inotify_release() was called. Completion did not happen for the past 2 minutes. Without a kernel dump it is hard to tell if completion was still ok - due to kthread being overloaded doing scheduled work and/or the marks group destruction - or there was a dead lock for the completion due to a kernel bug.
If this is reproducible, I think that having a kernel dump would help identifying the issue. I'm letting the kernel team to handle this and marking all other issues as dealt per previous comments.