4.4.0-83-generic + Docker + EC2 frequently crashes at cgroup_rmdir GPF
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
docker (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned | ||
linux (Ubuntu) |
Confirmed
|
High
|
Unassigned |
Bug Description
We run xenial-based Docker container hosts on EC2 with Amazon ECS. Recently we refreshed our base image, we started to see frequent panic.
Hosts run Amazon ECS Agent, and the agent automatically creates or destroys Docker container based on requests onto ECS cluster.
I think this crash is caused by Docker-related operations, because crashing at cgroups.
Also, we're running several different cluster with another EC2 instance types, using same image. This problem is only reproducing at t2.small instances. (We also run c4.large and m4.* clusters)
Our previous image ran 4.4.0-79-generic, and we see no problem with 79.
[30558.783899] general protection fault: 0000 [#1] SMP
[30558.784056] Modules linked in: veth binfmt_misc xt_nat xt_comment xt_tcpudp ipt_MASQUERADE nf_nat_
[30558.784056] CPU: 0 PID: 1 Comm: systemd Not tainted 4.4.0-83-generic #106-Ubuntu
[30558.784056] Hardware name: Xen HVM domU, BIOS 4.2.amazon 02/16/2017
[30558.784056] task: ffff88007c4e8000 ti: ffff88007c4e4000 task.ti: ffff88007c4e4000
[30558.784056] RIP: 0010:[<
[30558.784056] RSP: 0018:ffff88007c
[30558.784056] RAX: ffff8800114481bd RBX: ffff88002827ba50 RCX: ffff88007ab8d150
[30558.784056] RDX: 00111e7e00ffff88 RSI: ffff88002827ba54 RDI: ffffffff8217745c
[30558.784056] RBP: ffff88007c4e7e60 R08: 0000000000000020 R09: ffff88007c4e7e70
[30558.784056] R10: 000000000637760b R11: ffff880011829a80 R12: ffff88007ab8d000
[30558.784056] R13: 0000000000000000 R14: 0000559b48b2dcc0 R15: 00000000ffffff9c
[30558.784056] FS: 00007f29cf0db8c
[30558.784056] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[30558.784056] CR2: 00007fa466d58180 CR3: 000000007c190000 CR4: 00000000001406f0
[30558.784056] Stack:
[30558.784056] ffff88002827ba50 ffff88002827ba50 ffff8800373e70d0 0000559b48b2dcc0
[30558.784056] ffff88007c4e7e80 ffffffff811194b3 ffff88002827ba50 0000000000000000
[30558.784056] ffff88007c4e7ea0 ffffffff8128ddcd ffff880011829a80 0000000000000000
[30558.784056] Call Trace:
[30558.784056] [<ffffffff81119
[30558.784056] [<ffffffff8128d
[30558.784056] [<ffffffff8121b
[30558.784056] [<ffffffff8121f
[30558.784056] [<ffffffff81220
[30558.784056] [<ffffffff81840
[30558.784056] Code: 74 fd 48 c7 c7 5c 74 17 82 e8 8e 72 72 00 49 8b 94 24 50 01 00 00 49 8d 8c 24 50 01 00 00 48 39 d1 48 8d 42 f0 74 18 48 8b 50 08 <c6> 82 b0 01 00 00 01 48 8b 50 10 48 39 d1 48 8d 42 f0 75 e8 49
[30558.784056] RIP [<ffffffff81119
[30558.784056] RSP <ffff88007c4e7e40>
[30558.960828] ---[ end trace 7634e03ff94e8934 ]---
[30558.964811] Kernel panic - not syncing: Fatal exception in interrupt
[30558.968805] Kernel Offset: disabled
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-
ProcVersionSign
Uname: Linux 4.4.0-83-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jul 6 10:22 seq
crw-rw---- 1 root audio 116, 33 Jul 6 10:22 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
Date: Thu Jul 6 10:27:37 2017
Ec2AMI: ami-34100353
Ec2AMIManifest: (unknown)
Ec2Availability
Ec2InstanceType: t2.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Xen HVM domU
PciMultimedia:
ProcEnviron:
SHELL=/bin/bash
TERM=screen-
PATH=(custom, no user)
LANG=en_US.UTF-8
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=
RelatedPackageV
linux-
linux-
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog:
dmi.bios.date: 02/16/2017
dmi.bios.vendor: Xen
dmi.bios.version: 4.2.amazon
dmi.chassis.type: 1
dmi.chassis.vendor: Xen
dmi.modalias: dmi:bvnXen:
dmi.product.name: HVM domU
dmi.product.
dmi.sys.vendor: Xen
This change was made by a bot.