Ubuntu 15.10 Crashing Frequently on EC2 Instances w/ Enhanced Networking
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Won't Fix
|
Critical
|
Naeil Ezzoueidi |
Bug Description
Lots of details and history of the problem here: https:/
10 of my webservers have started crashing immediately following the 15.10 upgrade. As far as what exactly defines a "crash", Instance Status Checks fail, and I can no longer SSH to the machine. Background daemons running on the system stop responding, and nothing is written to the logs.
After weeks of working with the AWS team, I finally fixed a netconsole issue via "echo 7 > /proc/sys/
[21410.260077] general protection fault: 0000 [#1] SMP
[21410.261976] Modules linked in: isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_
[21410.264054] CPU: 0 PID: 26957 Comm: apache2 Not tainted 4.2.0-23-generic #28-Ubuntu
[21410.264054] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/07/2015
[21410.264054] task: ffff8803f9809b80 ti: ffff8803f999c000 task.ti: ffff8803f999c000
[21410.264054] RIP: 0010:[<
[21410.264054] RSP: 0000:ffff8803ff
[21410.264054] RAX: dead000000200200 RBX: ffff8803ff20e9c0 RCX: ffff8803ff203ec8
[21410.264054] RDX: ffff8803ff203ec8 RSI: 0000000000011fc0 RDI: ffff8803ff20e9c0
[21410.264054] RBP: ffff8803ff203f08 R08: 000000000000a77a R09: 0000000000000000
[21410.264054] R10: 0000000000000020 R11: 0000000000000004 R12: 000000000000007c
[21410.264054] R13: ffffffff8172aaf0 R14: 0000000000000000 R15: ffff8803af955be0
[21410.264054] FS: 00007fb0ce6e878
[21410.264054] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[21410.264054] CR2: 00007fb0ce51e130 CR3: 00000003fb233000 CR4: 00000000001406f0
[21410.264054] Stack:
[21410.264054] ffff8803ff203eb8 ffff8803ff20f5f8 ffff8803ff20f3f8 ffff8803ff20f1f8
[21410.264054] ffff8803ff20e9f8 ffff8803af955b58 dead000000200200 00000000f60fabc0
[21410.264054] 0000000000011fc0 0000000000000001 ffffffff81c0b0c8 0000000000000001
[21410.264054] Call Trace:
[21410.264054] <IRQ>
[21410.264054] [<ffffffff8107f
[21410.264054] [<ffffffff8107f
[21410.264054] [<ffffffff814a4
[21410.264054] [<ffffffff817f1
[21410.264054] <EOI>
[21410.264054] Code: 81 e6 00 00 20 00 48 85 d2 48 89 45 b8 0f 85 30 01 00 00 4c 89 7b 08 0f 1f 44 00 00 49 8b 07 49 8b 57 08 48 85 c0 48 89 02 74 04 <48> 89 50 08 41 f6 47 2a 10 48 b8 00 02 20 00 00 00 ad de 49 c7
[21410.264054] RIP [<ffffffff810e5
[21410.264054] RSP <ffff8803ff203e98>
I don't have a vmcore at the moment, but I'm trying to get one from AWS and should have one in the next couple of days. This is happening frequently and repeatedly since I first upgraded to 15.10 on early December.
ubuntu@
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.10
Release: 15.10
Codename: wily
ubuntu@
Linux xxx-web-xx 4.2.0-23-generic #28-Ubuntu SMP Sun Dec 27 17:47:31 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@
ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: linux-image-
ProcVersionSign
Uname: Linux 4.2.0-23-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jan 14 15:42 seq
crw-rw---- 1 root audio 116, 33 Jan 14 15:42 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.19.1-0ubuntu5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Thu Jan 14 21:31:14 2016
Ec2AMI: ami-d5e7adbf
Ec2AMIManifest: (unknown)
Ec2Availability
Ec2InstanceType: m4.xlarge
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Xen HVM domU
PciMultimedia:
ProcEnviron:
TERM=xterm-
PATH=(custom, no user)
XDG_RUNTIME_
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
0 cirrusdrmfb
1 xen
ProcKernelCmdLine: BOOT_IMAGE=
RelatedPackageV
linux-
linux-
linux-firmware 1.149.3
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
UpgradeStatus: Upgraded to wily on 2015-12-15 (29 days ago)
dmi.bios.date: 12/07/2015
dmi.bios.vendor: Xen
dmi.bios.version: 4.2.amazon
dmi.chassis.type: 1
dmi.chassis.vendor: Xen
dmi.modalias: dmi:bvnXen:
dmi.product.name: HVM domU
dmi.product.
dmi.sys.vendor: Xen
Changed in linux (Ubuntu): | |
importance: | Undecided → Critical |
Changed in linux (Ubuntu): | |
assignee: | nobody → Alberto Salvia Novella (es20490446e) |
status: | Confirmed → In Progress |
Changed in linux (Ubuntu): | |
status: | In Progress → Triaged |
assignee: | Alberto Salvia Novella (es20490446e) → nobody |
Changed in linux (Ubuntu): | |
assignee: | nobody → Na3iL (naeilzoueidi) |
This change was made by a bot.