Dropped packets on EC2, "xen_netfront: xennet: skb rides the rocket: x slots"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Fix Released
|
Medium
|
Unassigned | ||
Trusty |
Fix Released
|
Medium
|
Unassigned | ||
Utopic |
Fix Released
|
Medium
|
Unassigned |
Bug Description
Running Ubuntu 14.04 LTS on EC2, we see a lot of the following in the kernel log:
xen_netfront: xennet: skb rides the rocket: 19 slots
Each of these messages corresponds to a dropped TX packet, and eventually causes our application's connections to break and timeout.
The problem appears when network load increases. We have Node.js processes doing pubsub with a Redis server, and these are most visibly affected, showing frequent connection loss. The processes talk to each other using the private addresses EC2 allocates to the machines.
Notably, the default MTU on the network interface seems to have gone up from 1500 on 13.10, to 9000 in 14.04 LTS. Reducing the MTU back to 1500 seems to drastically reduce dropped packets. (Can't say for certain if it completely eliminates the problem.)
The machines we run are started from ami-896c96fe.
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-
ProcVersionSign
Uname: Linux 3.13.0-24-generic x86_64
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 May 9 09:01 seq
crw-rw---- 1 root audio 116, 33 May 9 09:01 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory: 'iw'
Date: Fri May 9 09:11:18 2014
Ec2AMI: ami-896c96fe
Ec2AMIManifest: (unknown)
Ec2Availability
Ec2InstanceType: c3.large
Ec2Kernel: aki-52a34525
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
PciMultimedia:
ProcFB:
ProcKernelCmdLine: root=LABEL=
RelatedPackageV
linux-
linux-
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 May 9 09:54 seq
crw-rw---- 1 root audio 116, 33 May 9 09:54 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg: [ 24.724129] init: plymouth-
DistroRelease: Ubuntu 14.04
Ec2AMI: ami-896c96fe
Ec2AMIManifest: (unknown)
Ec2Availability
Ec2InstanceType: c3.large
Ec2Kernel: aki-52a34525
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
Package: linux (not installed)
PciMultimedia:
ProcFB:
ProcKernelCmdLine: root=LABEL=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty ec2-images
Uname: Linux 3.13.0-24-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
break-fix: - 97a6d1bb2b658ac
break-fix: - 11d3d2a16cc1f05
CVE References
Changed in linux (Ubuntu): | |
importance: | Undecided → Medium |
Changed in linux (Ubuntu): | |
assignee: | nobody → Stefan Bader (smb) |
status: | Confirmed → In Progress |
description: | updated |
tags: | added: kernel-bug-break-fix |
Changed in linux (Ubuntu Trusty): | |
status: | Triaged → Confirmed |
Changed in linux (Ubuntu Utopic): | |
status: | Triaged → Confirmed |
Changed in linux (Ubuntu): | |
status: | In Progress → Fix Committed |
Changed in linux (Ubuntu): | |
status: | Fix Committed → Fix Released |
description: | updated |
Changed in linux (Ubuntu Utopic): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu Trusty): | |
status: | Confirmed → Fix Committed |
Changed in linux (Ubuntu Utopic): | |
status: | Fix Committed → Fix Released |
tags: | removed: kernel-bug-break-fix |
This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:
apport-collect 1317811
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.