| 2017-02-20 16:19:52 |
Iain Buclaw |
bug |
|
|
added bug |
| 2017-02-20 16:30:14 |
Brad Figg |
linux (Ubuntu): status |
New |
Confirmed |
|
| 2017-02-20 16:32:02 |
Iain Buclaw |
attachment added |
|
linux-44059.oom.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4822833/+files/linux-44059.oom.txt |
|
| 2017-02-20 16:33:14 |
Iain Buclaw |
attachment added |
|
linux-48034.oom.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4822834/+files/linux-48034.oom.txt |
|
| 2017-02-20 18:34:06 |
Iain Buclaw |
attachment added |
|
slabinfo.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4822861/+files/slabinfo.txt |
|
| 2017-02-20 18:36:05 |
Iain Buclaw |
attachment removed |
slabinfo.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4822861/+files/slabinfo.txt |
|
|
| 2017-02-20 18:41:22 |
Iain Buclaw |
attachment added |
|
slabtop.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4822862/+files/slabtop.txt |
|
| 2017-02-21 16:16:54 |
Iain Buclaw |
attachment added |
|
linux-48036.oom.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4823499/+files/linux-48036.oom.txt |
|
| 2017-02-21 17:52:59 |
Joseph Salisbury |
linux (Ubuntu): importance |
Undecided |
High |
|
| 2017-02-21 17:55:14 |
Joseph Salisbury |
tags |
amd64 apport-bug xenial |
amd64 apport-bug kernel-da-key xenial |
|
| 2017-02-22 08:12:30 |
Iain Buclaw |
description |
Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59.
Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34.
First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads.
However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34.
The symptoms I'm seeing are:
Upgrading 4.4.0-57 -> 4.4.0-59:
- /proc/meminfo:Buffers: Up from 9GB to 15GB
- /proc/meminfo:Cached: Up from 5GB to 10GB
- /proc/meminfo:SReclaimable: Down from 15GB to 5GB.
- /proc/meminfo:SUnreclaim: No notable change
Upgrading 4.4.0-57 -> 4.8.0-34:
- /proc/meminfo:Buffers: Up from 9GB to 14GB
- /proc/meminfo:Cached: Down from 5GB to 2GB
- /proc/meminfo:SReclaimable: Down from 15GB to 14GB.
- /proc/meminfo:SUnreclaim: No notable change
Downgrading 4.8.0-34 -> 4.4.0-63: (however uptime is only 1 hour)
- /proc/meminfo:Buffers: No notable change
- /proc/meminfo:Cached: No notable change
- /proc/meminfo:SReclaimable: No notable change
- /proc/meminfo:SUnreclaim: No notable change
Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs.
Downgrading to 4.4.0-57 also works.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-63-generic 4.4.0-63.84
ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44
Uname: Linux 4.4.0-63-generic x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
Date: Mon Feb 20 16:15:56 2017
InstallationDate: Installed on 2012-06-04 (1721 days ago)
InstallationMedia:
IwConfig:
lo no wireless extensions.
eth0 no wireless extensions.
Lsusb: Error: [Errno 2] No such file or directory: 'lsusb'
MachineType: System manufacturer System Product Name
PciMultimedia:
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw
RelatedPackageVersions:
linux-restricted-modules-4.4.0-63-generic N/A
linux-backports-modules-4.4.0-63-generic N/A
linux-firmware 1.157.8
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago)
dmi.bios.date: 10/17/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1106
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: P8H67-M PRO
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer |
Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59.
Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34.
First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads.
However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34.
The symptoms I'm seeing are:
Upgrading 4.4.0-57 -> 4.4.0-59:
- /proc/meminfo:Buffers: Up from 9GB to 15GB
- /proc/meminfo:Cached: Up from 5GB to 10GB
- /proc/meminfo:SReclaimable: Down from 15GB to 5GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Upgrading 4.4.0-57 -> 4.4.0-63:
- /proc/meminfo:Buffers: Up from 9GB to 26GB
- /proc/meminfo:Cached: Down from 5GB to 300MB
- /proc/meminfo:SReclaimable: Down from 15GB to 2GB
- /proc/meminfo:SUnreclaim: Down from 50MB to 30MB
Upgrading 4.4.0-57 -> 4.8.0-34:
- /proc/meminfo:Buffers: Up from 9GB to 14GB
- /proc/meminfo:Cached: Down from 5GB to 2GB
- /proc/meminfo:SReclaimable: Down from 15GB to 14GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs.
Downgrading to 4.4.0-57 also works.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-63-generic 4.4.0-63.84
ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44
Uname: Linux 4.4.0-63-generic x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
Date: Mon Feb 20 16:15:56 2017
InstallationDate: Installed on 2012-06-04 (1721 days ago)
InstallationMedia:
IwConfig:
lo no wireless extensions.
eth0 no wireless extensions.
Lsusb: Error: [Errno 2] No such file or directory: 'lsusb'
MachineType: System manufacturer System Product Name
PciMultimedia:
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw
RelatedPackageVersions:
linux-restricted-modules-4.4.0-63-generic N/A
linux-backports-modules-4.4.0-63-generic N/A
linux-firmware 1.157.8
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago)
dmi.bios.date: 10/17/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1106
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: P8H67-M PRO
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer |
|
| 2017-02-22 08:15:46 |
Iain Buclaw |
description |
Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59.
Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34.
First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing no change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads.
However, on closer inspection, there is definitely a notable difference between 4.4.0-59 and 4.4.0-63/4.8.0-34.
The symptoms I'm seeing are:
Upgrading 4.4.0-57 -> 4.4.0-59:
- /proc/meminfo:Buffers: Up from 9GB to 15GB
- /proc/meminfo:Cached: Up from 5GB to 10GB
- /proc/meminfo:SReclaimable: Down from 15GB to 5GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Upgrading 4.4.0-57 -> 4.4.0-63:
- /proc/meminfo:Buffers: Up from 9GB to 26GB
- /proc/meminfo:Cached: Down from 5GB to 300MB
- /proc/meminfo:SReclaimable: Down from 15GB to 2GB
- /proc/meminfo:SUnreclaim: Down from 50MB to 30MB
Upgrading 4.4.0-57 -> 4.8.0-34:
- /proc/meminfo:Buffers: Up from 9GB to 14GB
- /proc/meminfo:Cached: Down from 5GB to 2GB
- /proc/meminfo:SReclaimable: Down from 15GB to 14GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs.
Downgrading to 4.4.0-57 also works.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-63-generic 4.4.0-63.84
ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44
Uname: Linux 4.4.0-63-generic x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
Date: Mon Feb 20 16:15:56 2017
InstallationDate: Installed on 2012-06-04 (1721 days ago)
InstallationMedia:
IwConfig:
lo no wireless extensions.
eth0 no wireless extensions.
Lsusb: Error: [Errno 2] No such file or directory: 'lsusb'
MachineType: System manufacturer System Product Name
PciMultimedia:
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw
RelatedPackageVersions:
linux-restricted-modules-4.4.0-63-generic N/A
linux-backports-modules-4.4.0-63-generic N/A
linux-firmware 1.157.8
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago)
dmi.bios.date: 10/17/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1106
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: P8H67-M PRO
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer |
Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59.
Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34.
First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads.
These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34.
The symptoms I'm seeing are:
Upgrading 4.4.0-57 -> 4.4.0-59:
- /proc/meminfo:Buffers: Up from 9GB to 15GB
- /proc/meminfo:Cached: Up from 5GB to 10GB
- /proc/meminfo:SReclaimable: Down from 15GB to 5GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Upgrading 4.4.0-57 -> 4.4.0-63:
- /proc/meminfo:Buffers: Up from 9GB to 26GB
- /proc/meminfo:Cached: Down from 5GB to 300MB
- /proc/meminfo:SReclaimable: Down from 15GB to 2GB
- /proc/meminfo:SUnreclaim: Down from 50MB to 30MB
Upgrading 4.4.0-57 -> 4.8.0-34:
- /proc/meminfo:Buffers: Up from 9GB to 14GB
- /proc/meminfo:Cached: Down from 5GB to 2GB
- /proc/meminfo:SReclaimable: Down from 15GB to 14GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs.
Downgrading to 4.4.0-57 also works.
Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far to different from what I expect to be normal on these particular servers where I'm experiencing crashes.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-63-generic 4.4.0-63.84
ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44
Uname: Linux 4.4.0-63-generic x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
Date: Mon Feb 20 16:15:56 2017
InstallationDate: Installed on 2012-06-04 (1721 days ago)
InstallationMedia:
IwConfig:
lo no wireless extensions.
eth0 no wireless extensions.
Lsusb: Error: [Errno 2] No such file or directory: 'lsusb'
MachineType: System manufacturer System Product Name
PciMultimedia:
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw
RelatedPackageVersions:
linux-restricted-modules-4.4.0-63-generic N/A
linux-backports-modules-4.4.0-63-generic N/A
linux-firmware 1.157.8
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago)
dmi.bios.date: 10/17/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1106
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: P8H67-M PRO
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer |
|
| 2017-02-22 08:16:42 |
Iain Buclaw |
description |
Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59.
Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34.
First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads.
These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34.
The symptoms I'm seeing are:
Upgrading 4.4.0-57 -> 4.4.0-59:
- /proc/meminfo:Buffers: Up from 9GB to 15GB
- /proc/meminfo:Cached: Up from 5GB to 10GB
- /proc/meminfo:SReclaimable: Down from 15GB to 5GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Upgrading 4.4.0-57 -> 4.4.0-63:
- /proc/meminfo:Buffers: Up from 9GB to 26GB
- /proc/meminfo:Cached: Down from 5GB to 300MB
- /proc/meminfo:SReclaimable: Down from 15GB to 2GB
- /proc/meminfo:SUnreclaim: Down from 50MB to 30MB
Upgrading 4.4.0-57 -> 4.8.0-34:
- /proc/meminfo:Buffers: Up from 9GB to 14GB
- /proc/meminfo:Cached: Down from 5GB to 2GB
- /proc/meminfo:SReclaimable: Down from 15GB to 14GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs.
Downgrading to 4.4.0-57 also works.
Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far to different from what I expect to be normal on these particular servers where I'm experiencing crashes.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-63-generic 4.4.0-63.84
ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44
Uname: Linux 4.4.0-63-generic x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
Date: Mon Feb 20 16:15:56 2017
InstallationDate: Installed on 2012-06-04 (1721 days ago)
InstallationMedia:
IwConfig:
lo no wireless extensions.
eth0 no wireless extensions.
Lsusb: Error: [Errno 2] No such file or directory: 'lsusb'
MachineType: System manufacturer System Product Name
PciMultimedia:
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw
RelatedPackageVersions:
linux-restricted-modules-4.4.0-63-generic N/A
linux-backports-modules-4.4.0-63-generic N/A
linux-firmware 1.157.8
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago)
dmi.bios.date: 10/17/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1106
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: P8H67-M PRO
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer |
Same as #1655842 - Started seeing oom-killer on multiple servers upgraded to 4.4.0-59.
Unlike #1655842, also seeing the same oom-killer on multiple servers updated to 4.8.0-34.
First upgraded them all 4.8 servers 4.8.0-36, then downgraded a few to 4.4.0-63. I am seeing an even more pronounced change in the memory usage, so I can only assume that 4.4.0-63 is also bugged with the same problem as 4.4.0-59 and 4.8.0-34. Either because #1655842 is not fixed, or it is only fixed for certain kinds of workloads.
These are the changes I'm seeing in our memory graphs between 4.4.0-59 and 4.4.0-63/4.8.0-34.
The symptoms I'm seeing are:
Upgrading 4.4.0-57 -> 4.4.0-59:
- /proc/meminfo:Buffers: Up from 9GB to 15GB
- /proc/meminfo:Cached: Up from 5GB to 10GB
- /proc/meminfo:SReclaimable: Down from 15GB to 5GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Upgrading 4.4.0-57 -> 4.4.0-63:
- /proc/meminfo:Buffers: Up from 9GB to 26GB
- /proc/meminfo:Cached: Down from 5GB to 300MB
- /proc/meminfo:SReclaimable: Down from 15GB to 2GB
- /proc/meminfo:SUnreclaim: Down from 50MB to 30MB
Upgrading 4.4.0-57 -> 4.8.0-34:
- /proc/meminfo:Buffers: Up from 9GB to 14GB
- /proc/meminfo:Cached: Down from 5GB to 2GB
- /proc/meminfo:SReclaimable: Down from 15GB to 14GB
- /proc/meminfo:SUnreclaim: Staying at 50MB
Setting vm.vfs_cache_pressure = 300 seems to have a positive effect of not causing OOMs.
Downgrading to 4.4.0-57 also works.
Will also note that I haven't had a definitive OOM in 4.4.0-63. But the shift in memory usage is far too much from what I expect to be normal on these particular servers where I'm experiencing crashes.
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-63-generic 4.4.0-63.84
ProcVersionSignature: Ubuntu 4.4.0-63.84-generic 4.4.44
Uname: Linux 4.4.0-63-generic x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
Date: Mon Feb 20 16:15:56 2017
InstallationDate: Installed on 2012-06-04 (1721 days ago)
InstallationMedia:
IwConfig:
lo no wireless extensions.
eth0 no wireless extensions.
Lsusb: Error: [Errno 2] No such file or directory: 'lsusb'
MachineType: System manufacturer System Product Name
PciMultimedia:
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-63-generic root=UUID=b790930f-ad81-4b27-a353-a4b3d6a29007 ro nomodeset nomdmonddf nomdmonisw
RelatedPackageVersions:
linux-restricted-modules-4.4.0-63-generic N/A
linux-backports-modules-4.4.0-63-generic N/A
linux-firmware 1.157.8
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to xenial on 2017-02-16 (4 days ago)
dmi.bios.date: 10/17/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1106
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: P8H67-M PRO
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1106:bd10/17/2011:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP8H67-MPRO:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer |
|
| 2017-02-23 10:29:36 |
Iain Buclaw |
attachment added |
|
linux-48036.oom2.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4825131/+files/linux-48036.oom2.txt |
|
| 2017-02-23 21:07:30 |
Oliver O. |
bug |
|
|
added subscriber Oliver O. |
| 2017-02-24 13:39:24 |
Iain Buclaw |
attachment added |
|
linux-48036.oom3.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4825946/+files/linux-48036.oom3.txt |
|
| 2017-02-27 11:16:07 |
Iain Buclaw |
attachment added |
|
linux-48036.oom4.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4827637/+files/linux-48036.oom4.txt |
|
| 2017-02-27 11:16:22 |
Iain Buclaw |
attachment added |
|
linux-48036.oom5.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4827639/+files/linux-48036.oom5.txt |
|
| 2017-02-27 11:16:38 |
Iain Buclaw |
attachment added |
|
linux-48036.oom6.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4827640/+files/linux-48036.oom6.txt |
|
| 2017-02-27 11:16:53 |
Iain Buclaw |
attachment added |
|
linux-48036.oom7.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1666260/+attachment/4827641/+files/linux-48036.oom7.txt |
|
| 2017-03-21 08:12:35 |
Rasmus Larsen |
bug |
|
|
added subscriber Rasmus Larsen |
| 2017-04-04 19:26:26 |
Oliver O. |
removed subscriber Oliver O. |
|
|
|
| 2017-12-16 11:45:31 |
Iain Buclaw |
tags |
amd64 apport-bug kernel-da-key xenial |
amd64 apport-bug kernel-da-key kernel-fixed-upstream xenial |
|
