10-15% iowait on kvm instances on Ubuntu 14.04

[William Van Hevelingen]

After upgrading from Ubuntu 12.04 to Ubuntu 14.04 we began seeing 10-15% iowait on kvm guests. The guests had little to no load and yet the shell was laggy and commands took forever to respond (assuming you could even type them)

# uname -a
Linux kvm-andromeda 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC
2014 x86_64 x86_64 x86_64 GNU/Linux

# kvm --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.1), Copyright (c)
2003-2008 Fabrice Bellard

I posted to Ubuntu Server and they suggested that I upgrade my kernel version.

Relevant Mailing List discussion:

https://lists.ubuntu.com/archives/ubuntu-server/2014-July/006925.html

After installing a 3.14 kernel the performance returned to normal.

kvm-andromeda:~# uname -a
Linux kvm-andromeda 3.14.1-031401-generic #201404141220 SMP Mon Apr 14 16:21:48 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
---
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
DistroRelease: Ubuntu 14.04
Package: linux (not installed)
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
Tags: trusty
Uname: Linux 3.14.1-031401-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: root
_MarkForUpload: True
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jul 21 13:53 seq
 crw-rw---- 1 root audio 116, 33 Jul 21 13:53 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 14.04
HibernationDevice: RESUME=/dev/mapper/bootdisk-swap_1
IwConfig: Error: [Errno 2] No such file or directory
MachineType: Supermicro H8DG6/H8DGi
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.13.0-29-generic root=/dev/mapper/bootdisk-root ro quiet splash rootdelay=120 nomdmonddf nomdmonisw vt.handoff=7
ProcVersionSignature: Ubuntu 3.13.0-29.53-generic 3.13.11.2
RelatedPackageVersions:
 linux-restricted-modules-3.13.0-29-generic N/A
 linux-backports-modules-3.13.0-29-generic N/A
 linux-firmware 1.127.2
RfKill: Error: [Errno 2] No such file or directory
Tags: trusty
Uname: Linux 3.13.0-29-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: root
_MarkForUpload: True
dmi.bios.date: 11/25/2013
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 3.5
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: H8DG6/H8DGi
dmi.board.vendor: Supermicro
dmi.board.version: 1.0
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 17
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 1234567890
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3.5:bd11/25/2013:svnSupermicro:pnH8DG6/H8DGi:pvr1234567890:rvnSupermicro:rnH8DG6/H8DGi:rvr1.0:cvnSupermicro:ct17:cvr1234567890:
dmi.product.name: H8DG6/H8DGi
dmi.product.version: 1234567890
dmi.sys.vendor: Supermicro

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1344430

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[William Van Hevelingen (blkperl)]

I can boot into the old kernel and run apport-collect if you need that as well.

[Brad Figg (brad-figg)]

@william,

Yes, please boot into the broken kernel and run apport-collect 1344430.

[Stefan Bader (smb)]

Some things (like the hardware of the host) we get through apport-collect. It would be good to get a better understanding of the guest usage. Like how many guests, how much memory they get, how many vcpus. virtio disks and network and how are the guests connected to the network (NAT bridge, transparent bridge, openvswitch?). Are vdisk using raw images or qcov2, image files or partitions or logical volumes.
Maybe then we can also replicate the issue in a test environment. Up to now I am not sure I ever was able to see the same issue or not. Oh and was there a specific stat or combination of stats that hints this is happening. Laggy terminal is hard to test for in some scripted way.

[Chris J Arges (arges)]

One thing we are looking at is potential issues with KSM and KVM. Can you try disabling KSM and see if the results remain the same:
- edit /etc/default/qemu:
- change KSM_ENABLED to 0
- sudo service qemu restart
I would recommend doing this when your VMs are shutdown; and ensuring nothing else in your system is using KSM.

Thanks,

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

apport information

[William Van Hevelingen (blkperl)]

Rebooted into the old kernel and disabled KSM. Then ran apport-collect.

[Chris J Arges (arges)]

I believe I've found the fix for this issue for 3.13.
If you can, please test the kernel posted on comment #1 on this bug:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1346917
Make sure KSM is enabled; and any workarounds for this bug are disabled.

If this fixes the issue for you, you are welcome to mark this bug as a duplicate of 1346917.

Thanks!

[Joseph Salisbury (jsalisbury)]

Marking incomplete until requested testing is complete.