kswapd0 consumes 100% CPU

[Alex]

I'm running AWS EC2 t2.micro instance with below ubuntu version.
ubuntu-wily-15.10-amd64-server-20151203 (ami-7d948911)
4.2.0-23-generic

Periodically around once per day my instance consumes 100% CPU by kswapd0 kernell process.
Instance restart helps, otherwise this continues.

I have no swap configured.

Could be related to #721896

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: linux-image-4.2.0-23-generic 4.2.0-23.28
ProcVersionSignature: User Name 4.2.0-23.28-generic 4.2.6
Uname: Linux 4.2.0-23-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jan 9 07:44 seq
 crw-rw---- 1 root audio 116, 33 Jan 9 07:44 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.19.1-0ubuntu5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Sat Jan 9 20:32:49 2016
Ec2AMI: ami-7d948911
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: eu-central-1a
Ec2InstanceType: t2.micro
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Xen HVM domU
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:
 0 cirrusdrmfb
 1 xen
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-23-generic root=UUID=73008cf2-451d-4df0-a2e6-16ffb91670c7 ro console=tty1 console=ttyS0
RelatedPackageVersions:
 linux-restricted-modules-4.2.0-23-generic N/A
 linux-backports-modules-4.2.0-23-generic N/A
 linux-firmware 1.149.3
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UdevLog: Error: [Errno 2] No such file or directory: '/var/log/udev'
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/07/2015
dmi.bios.vendor: Xen
dmi.bios.version: 4.2.amazon
dmi.chassis.type: 1
dmi.chassis.vendor: Xen
dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/07/2015:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr:
dmi.product.name: HVM domU
dmi.product.version: 4.2.amazon
dmi.sys.vendor: Xen

[Brad Figg (brad-figg)]

This change was made by a bot.

[penalvch (penalvch)]

Alex, thank you for reporting this and helping make Ubuntu better.

At your earliest convenience, could you please test the latest upstream kernel available from the very top line at the top of the page from http://kernel.ubuntu.com/~kernel-ppa/mainline/?C=N;O=D (the release names are irrelevant for testing, and please do not test the daily folder)? Install instructions are available at https://wiki.ubuntu.com/Kernel/MainlineBuilds . This will allow additional upstream developers to examine the issue.

If testing on your main install would be inconvenient, one may:
1) Install Ubuntu to a different partition and then test this there.
2) Backup, or clone the primary install.

If the latest kernel did not allow you to test to the issue (ex. you couldn't boot into the OS) please make a comment in your report about this, and continue to test the next most recent kernel version until you can test to the issue. Once you've tested the upstream kernel, please comment on which kernel version specifically you tested. If this issue is fixed in the mainline kernel, please add the following tags by clicking on the yellow circle with a black pencil icon, next to the word Tags, located at the bottom of the report description:
kernel-fixed-upstream
kernel-fixed-upstream-X.Y-rcZ

Where X, and Y are the first two numbers of the kernel version, and Z is the release candidate number if it exists.

If the mainline kernel does not fix the issue, please add the following tags:
kernel-bug-exists-upstream
kernel-bug-exists-upstream-X.Y-rcZ

Please note, an error to install the kernel does not fit the criteria of kernel-bug-exists-upstream.

Once testing of the latest upstream kernel is complete, please mark this report's Status as Confirmed. Please let us know your results.

Thank you for your understanding.

[Alex (alex-golonzovsky)]

I've recreated an instance from scratch with ami-c4140aa8.
Issue still happens, however less frequently - it happened 3 times during last week.

I've proceeded with latest kernell update.
Installed 4.4.0-040400.

Will follow on results below.

[Alex (alex-golonzovsky)]

Unfortunately cannot proceed without linux-image-extra package. Rolling back to previous version.

Will try to setup new instance with latest 4.4.

[Alex (alex-golonzovsky)]

Issue still persists, see utilization graph on mostly idle instance. All peaks are caused by mentioned issue. I have restart configured as a workaround, when cpu hits 100 for 30 min.

[penalvch (penalvch)]

Alex, to clarify, if you were to add swap back in, would kswapd0 still consume 100% CPU when at the same load levels across resources?

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]