containerd-shim deadlocks, then crashes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
containerd (Ubuntu) |
Expired
|
Undecided
|
Unassigned |
Bug Description
I'm using docker-compose to wrangle a bunch of containers, and quite often one of my containers hangs.
When I inspect the process with strace, I see it's blocked on write(2, "...").
/proc/$
strace -p $containerd_
After enough time passes (around 5 or 10 minutes?) I see that containerd-shim crash with a SIGABRT. This time I had strace still attached:
strace: Process 861273 attached
futex(0xad3848, FUTEX_WAIT_PRIVATE, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGABRT {si_signo=SIGABRT, si_code=SI_USER, si_pid=867057, si_uid=0} ---
nanosleep(
nanosleep(
write(2, "SIGABRT: abort", 14) = 14
write(2, "\n", 1) = 1
write(2, "PC=", 3) = 3
write(2, "0x45c791", 8) = 8
write(2, " m=", 3) = 3
write(2, "0", 1) = 1
write(2, " sigcode=", 9) = 9
write(2, "0", 1) = 1
write(2, "\n", 1) = 1
write(2, "\n", 1) = 1
write(2, "goroutine ", 10) = 10
write(2, "0", 1) = 1
write(2, " [", 2) = 2
write(2, "idle", 4) = 4
write(2, "]:\n", 3) = 3
write(2, "runtime.futex", 13) = 13
...
write(2, "rflags ", 7) = 7
write(2, "0x286", 5) = 5
write(2, "\n", 1) = 1
write(2, "cs ", 7) = 7
write(2, "0x33", 4) = 4
write(2, "\n", 1) = 1
write(2, "fs ", 7) = 7
write(2, "0x0", 3) = 3
write(2, "\n", 1) = 1
write(2, "gs ", 7) = 7
write(2, "0x0", 3) = 3
write(2, "\n", 1) = 1
exit_group(2) = ?
+++ exited with 2 +++
(Full strace log attached, unless I forget)
It would be nice if I could read that Go traceback somewhere instead of looking at truncated strace writes, but I don't know where.
journalctl -u containerd shows only this:
rugs. 15 12:10:32 blynas containerd[1133]: time="2020-
rugs. 15 12:10:33 blynas containerd[1133]: time="2020-
rugs. 15 12:10:34 blynas containerd[1133]: time="2020-
rugs. 15 12:11:03 blynas containerd[1133]: time="2020-
rugs. 15 12:20:09 blynas containerd[1133]: time="2020-
rugs. 15 12:20:09 blynas containerd[1133]: time="2020-
rugs. 15 12:20:12 blynas containerd[1133]: time="2020-
ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: containerd 1.3.3-0ubuntu2
ProcVersionSign
Uname: Linux 5.4.0-47-generic x86_64
NonfreeKernelMo
ApportVersion: 2.20.11-0ubuntu27.8
Architecture: amd64
CasperMD5CheckR
CurrentDesktop: ubuntu:GNOME
Date: Tue Sep 15 12:20:56 2020
EcryptfsInUse: Yes
InstallationDate: Installed on 2019-06-12 (460 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
SourcePackage: containerd
UpgradeStatus: Upgraded to focal on 2020-04-24 (143 days ago)
Hello Marius,
Thank you for taking the time to file a bug report.
Since there is not enough information in your report to begin triage or to
differentiate between a local configuration problem and a bug in Ubuntu, I
am marking this bug as "Incomplete".
I know you had trouble in getting the strace output already but, if you don't mind, could you please provide us a short reproducer ? This way me or some other person is able to reproduce it locally and chose preferred tools to debug the issue.
Please change status back to New once you're done so this can be triaged again by someone from the Ubuntu Server team.
Thanks a lot.
-rafaeldtinoco