| 2015-09-24 07:31:39 |
Oskar Liljeblad |
bug |
|
|
added bug |
| 2015-09-24 08:00:06 |
Brad Figg |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2015-09-24 08:00:07 |
Brad Figg |
tags |
|
trusty |
|
| 2015-09-24 12:59:10 |
Oskar Liljeblad |
tags |
trusty |
apport-collected trusty |
|
| 2015-09-24 12:59:11 |
Oskar Liljeblad |
description |
Slab and SUnreclaim values in /proc/meminfo keep increasing. On one servers it reached 85% of physical memory after 14 days - but on most other servers it increases more slowly. I checked /proc/slabinfo and almost all allocations were in kmalloc-512. So I enabled "slub_debug=U,kmalloc-512" on one server, and after only 24h of uptime 11% of the memory was used by kmalloc-512 and unreclaimable. With debugging enabled I could see the following in /sys/kernel/slab/kmalloc-512/alloc_calls:
521294 storvsc_queuecommand+0x359/0x790 [hv_storvsc] age=161922/955116/20882927 pid=1-41545
All other counters were below 2000. In /sys/kernel/slab/kmalloc-512/free_calls I see the following:
516823 <not-available> age=4315783846 pid=0
The hv_storvsc module is for Hyper-V. We are (unfortunately) running Hyper-V 6.3.9600.16384 with Microsoft System Center 2012 R2 Update rollup 3 for all the servers with this issue.
Kernels are stock linux-image-3.13.0-63-generic, 3.13.0-63.103, x86_64, from Ubuntu 14.04 LTS . /proc/version_signature contains:
Ubuntu 3.13.0-63.103-generic 3.13.11-ckt25
No output from lspci -vnvn. The problem described above happens on both single and multicore virtual machines. CPU in hypervisors are E5-2630 v2 @ 2.60GHz. Let me know if you need more info or if I can do more debugging.
Regards,
Oskar Liljeblad |
Slab and SUnreclaim values in /proc/meminfo keep increasing. On one servers it reached 85% of physical memory after 14 days - but on most other servers it increases more slowly. I checked /proc/slabinfo and almost all allocations were in kmalloc-512. So I enabled "slub_debug=U,kmalloc-512" on one server, and after only 24h of uptime 11% of the memory was used by kmalloc-512 and unreclaimable. With debugging enabled I could see the following in /sys/kernel/slab/kmalloc-512/alloc_calls:
521294 storvsc_queuecommand+0x359/0x790 [hv_storvsc] age=161922/955116/20882927 pid=1-41545
All other counters were below 2000. In /sys/kernel/slab/kmalloc-512/free_calls I see the following:
516823 <not-available> age=4315783846 pid=0
The hv_storvsc module is for Hyper-V. We are (unfortunately) running Hyper-V 6.3.9600.16384 with Microsoft System Center 2012 R2 Update rollup 3 for all the servers with this issue.
Kernels are stock linux-image-3.13.0-63-generic, 3.13.0-63.103, x86_64, from Ubuntu 14.04 LTS . /proc/version_signature contains:
Ubuntu 3.13.0-63.103-generic 3.13.11-ckt25
No output from lspci -vnvn. The problem described above happens on both single and multicore virtual machines. CPU in hypervisors are E5-2630 v2 @ 2.60GHz. Let me know if you need more info or if I can do more debugging.
Regards,
Oskar Liljeblad
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 24 00:31 seq
crw-rw---- 1 root audio 116, 33 Sep 24 00:31 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.13
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg:
[59081.977909] systemd-udevd[26480]: starting version 204
[59124.051974] init: systemd-logind main process (756) killed by TERM signal
DistroRelease: Ubuntu 14.04
InstallationDate: Installed on 2014-09-09 (380 days ago)
InstallationMedia: Ubuntu-Server 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.3)
IwConfig:
eth0 no wireless extensions.
eth1 no wireless extensions.
lo no wireless extensions.
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Microsoft Corporation Virtual Machine
Package: linux (not installed)
PciMultimedia:
ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.13.0-63-generic.efi.signed root=UUID=f4d228d6-2eee-40fc-bf3f-633e46fa8301 ro slub_debug=U,kmalloc-512
ProcVersionSignature: Ubuntu 3.13.0-63.103-generic 3.13.11-ckt25
RelatedPackageVersions:
linux-restricted-modules-3.13.0-63-generic N/A
linux-backports-modules-3.13.0-63-generic N/A
linux-firmware 1.127.15
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.13.0-63-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
WifiSyslog:
Sep 24 02:06:19 adm-backup1 dhclient: message repeated 1447 times: [ DHCPREQUEST of 10.40.128.9 on eth0 to 192.0.2.253 port 67 (xid=0x429dad4)]
Sep 24 02:06:37 adm-backup1 dhclient: DHCPREQUEST of 10.40.128.9 on eth0 to 255.255.255.255 port 67 (xid=0x429dad4)
Sep 24 02:06:37 adm-backup1 dhclient: DHCPACK of 10.40.128.9 from 192.0.2.253
Sep 24 02:06:37 adm-backup1 dhclient: bound to 10.40.128.9 -- renewal in 44877 seconds.
_MarkForUpload: True
dmi.bios.date: 11/26/2012
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v1.0
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v1.0
dmi.chassis.asset.tag: 6126-4244-1659-0314-3158-3955-44
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v1.0
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev1.0:bd11/26/2012:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev1.0:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev1.0:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev1.0:
dmi.product.name: Virtual Machine
dmi.product.version: Hyper-V UEFI Release v1.0
dmi.sys.vendor: Microsoft Corporation |
|
| 2015-09-24 12:59:12 |
Oskar Liljeblad |
attachment added |
|
BootDmesg.txt https://bugs.launchpad.net/bugs/1499203/+attachment/4473285/+files/BootDmesg.txt |
|
| 2015-09-24 12:59:14 |
Oskar Liljeblad |
attachment added |
|
ProcCpuinfo.txt https://bugs.launchpad.net/bugs/1499203/+attachment/4473286/+files/ProcCpuinfo.txt |
|
| 2015-09-24 12:59:15 |
Oskar Liljeblad |
attachment added |
|
ProcEnviron.txt https://bugs.launchpad.net/bugs/1499203/+attachment/4473287/+files/ProcEnviron.txt |
|
| 2015-09-24 12:59:17 |
Oskar Liljeblad |
attachment added |
|
ProcInterrupts.txt https://bugs.launchpad.net/bugs/1499203/+attachment/4473288/+files/ProcInterrupts.txt |
|
| 2015-09-24 12:59:19 |
Oskar Liljeblad |
attachment added |
|
ProcModules.txt https://bugs.launchpad.net/bugs/1499203/+attachment/4473289/+files/ProcModules.txt |
|
| 2015-09-24 12:59:20 |
Oskar Liljeblad |
attachment added |
|
UdevDb.txt https://bugs.launchpad.net/bugs/1499203/+attachment/4473290/+files/UdevDb.txt |
|
| 2015-09-24 12:59:21 |
Oskar Liljeblad |
attachment added |
|
UdevLog.txt https://bugs.launchpad.net/bugs/1499203/+attachment/4473291/+files/UdevLog.txt |
|
| 2015-09-25 09:33:56 |
Oskar Liljeblad |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2015-09-30 17:05:16 |
Joseph Salisbury |
linux (Ubuntu): importance |
Undecided |
Medium |
|
| 2015-09-30 17:05:36 |
Joseph Salisbury |
tags |
apport-collected trusty |
apport-collected kernel-da-key kernel-hyper-v trusty |
|
| 2015-09-30 17:12:54 |
Joseph Salisbury |
nominated for series |
|
Ubuntu Trusty |
|
| 2015-09-30 17:12:54 |
Joseph Salisbury |
bug task added |
|
linux (Ubuntu Trusty) |
|
| 2015-09-30 17:13:02 |
Joseph Salisbury |
linux (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2015-09-30 17:13:04 |
Joseph Salisbury |
linux (Ubuntu): importance |
Medium |
High |
|
| 2015-09-30 17:13:06 |
Joseph Salisbury |
linux (Ubuntu Trusty): status |
New |
Confirmed |
|
| 2015-10-15 13:08:02 |
Svein.Erik Lund |
bug |
|
|
added subscriber Svein.Erik Lund |
| 2015-12-11 19:47:22 |
Joseph Salisbury |
linux (Ubuntu): assignee |
|
Joseph Salisbury (jsalisbury) |
|
| 2015-12-11 19:47:24 |
Joseph Salisbury |
linux (Ubuntu Trusty): assignee |
|
Joseph Salisbury (jsalisbury) |
|
| 2015-12-11 19:47:29 |
Joseph Salisbury |
linux (Ubuntu Trusty): status |
Confirmed |
In Progress |
|
| 2015-12-11 19:47:32 |
Joseph Salisbury |
linux (Ubuntu): status |
Confirmed |
In Progress |
|
| 2016-03-01 18:29:45 |
Joseph Salisbury |
linux (Ubuntu): status |
In Progress |
Fix Released |
|
| 2016-03-01 18:29:48 |
Joseph Salisbury |
linux (Ubuntu Trusty): status |
In Progress |
Fix Released |
|
