OOM with lots of free swap

[Nate Eldredge]

On a vps with 512MB RAM and a 2 GB swapfile, I am experiencing 100% reproducible OOM when running duplicity. duplicity uses a fair amount of memory, about 300 MB at the time of OOM, but there should be plenty of swap to accommodate it.

I have attached a syslog excerpt. Note "Free swap = 2004684kB".

If I am reading the log message correctly, "order=2" at the top means this was an attempt to allocate 4 pages, i.e. 16KB. It certainly seems that it should be possible to satisfy this by swapping. I know fragmentation is a possible issue but it seems implausible that we couldn't find 4 pages together.

This doesn't seem to happen when running other memory-hogging processes. I wrote a small program that malloc's 1 GB and writes to all of it. It runs fine, even as root. It is slow because of swapping but the oom-killer does not get invoked. So I am not sure what is special about duplicity. It's an ordinary user-space backup utility written in Python; it reads files, encrypts data, and sends it out over ssh. The only thing I can think of is that it runs gpg to do the encryption, and I believe gpg will mlock() some memory to keep keys secure, but that should be a really small amount.

Happy to supply more information or test things.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-59-generic 4.4.0-59.80
ProcVersionSignature: Ubuntu 4.4.0-59.80-generic 4.4.35
Uname: Linux 4.4.0-59-generic x86_64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jan 21 19:31 seq
 crw-rw---- 1 root audio 116, 33 Jan 21 19:31 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Sun Jan 29 03:37:31 2017
HibernationDevice: RESUME=UUID=7c1226a1-9254-49c2-b4e6-64d204affd15
IwConfig:
 lo no wireless extensions.

 eth1 no wireless extensions.

 eth0 no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1:
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: root=/dev/xvda1 ro quiet splash console=hvc0
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-59-generic N/A
 linux-backports-modules-4.4.0-59-generic N/A
 linux-firmware 1.157.6
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to xenial on 2016-09-19 (132 days ago)
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jan 21 19:31 seq
 crw-rw---- 1 root audio 116, 33 Jan 21 19:31 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
HibernationDevice: RESUME=UUID=7c1226a1-9254-49c2-b4e6-64d204affd15
IwConfig:
 lo no wireless extensions.

 eth1 no wireless extensions.

 eth0 no wireless extensions.
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1:
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: root=/dev/xvda1 ro quiet splash console=hvc0
ProcVersionSignature: Ubuntu 4.4.0-59.80-generic 4.4.35
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-59-generic N/A
 linux-backports-modules-4.4.0-59-generic N/A
 linux-firmware 1.157.6
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
Uname: Linux 4.4.0-59-generic x86_64
UpgradeStatus: Upgraded to xenial on 2016-09-19 (132 days ago)
UserGroups:

_MarkForUpload: True

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1660098

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Nate Eldredge (nate-thatsmathematics)]

apport information

[Nate Eldredge (nate-thatsmathematics)]

apport information

[Nate Eldredge (nate-thatsmathematics)]

apport information

[Nate Eldredge (nate-thatsmathematics)]

apport information

[Nate Eldredge (nate-thatsmathematics)]

apport information

[Nate Eldredge (nate-thatsmathematics)]

apport information

[Nate Eldredge (nate-thatsmathematics)]

apport information

[Nate Eldredge (nate-thatsmathematics)]

apport information

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.10 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.10-rc6

[Nate Eldredge (nate-thatsmathematics)]

On Mon, 30 Jan 2017, Joseph Salisbury wrote:

> Did this issue start happening after an update/upgrade? Was there a
> prior kernel version where you were not having this particular problem?

Good thought. After checking my logs, it seems to have started right
after upgrading the kernel from 4.4.0-57-generic to -59. And now I have
found https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842 so I
think this is a duplicate of that.

Thanks for the help!

--
Nate Eldredge
<email address hidden>