Hi Ubuntu,
Just found out today that when using Ubuntu Xenial HWE kernel 4.13 in a VM on top of Proxmox VE 5.1.x, the direct synchronous write performance is 93% slower than on kernel 4.4. Kernel 4.13 direct sync 4KB write could only achieve 303KB/s while kernel 4.4 could achieve 4.5MB/s.
The VM is using ext4 filesystem and running on top of ZFS ZVOL on ZFS RAID1 with virtio-scsi, qemu disk cache none, and qemu disk discard enabled. The underlying physical disks are Samsung Enterprise SSD SM863a 480GB series.
The reason I need to run kernel 4.13 to mount CephFS Luminous with kernel client. Hopefully CephFS Luminous kernel client will be back ported to kernel 4.4 in the near future.
Is this from Ubuntu Xenial kernel 4.13 performance regression or from Proxmox VE 5.1 kernel 4.13 virtio-scsi performance regression?
The test results:
##
# ubuntu xenial kernel 4.13. the direct synchronous write performance is 93% slower than kernel 4.4
##
root@localhost:~# uname -a
Linux localhost 4.13.0-37-generic #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@localhost:~# dd if=/dev/zero of=/mnt/test/test bs=4K count=256 oflag=direct,sync status=progress; rm /mnt/test/test
1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.46369 s, 303 kB/s
##
# ubuntu xenial kernel 4.4. the direct synchronous write performance is 14x faster than kernel 4.13
##
root@localhost:~# uname -a
Linux localhost 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@localhost:~# dd if=/dev/zero of=/mnt/test/test bs=4K count=256 oflag=direct,sync status=progress; rm /mnt/test/test
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231176 s, 4.5 MB/s
ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.13.0-37-generic 4.13.0-37.42~16.04.1
ProcVersionSignature: Ubuntu 4.13.0-37.42~16.04.1-generic 4.13.13
Uname: Linux 4.13.0-37-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.15
Architecture: amd64
Date: Sun Mar 25 13:43:34 2018
InstallationDate: Installed on 2018-03-22 (2 days ago)
InstallationMedia: Ubuntu-Server 16.04.4 LTS "Xenial Xerus" - Release amd64 (20180228)
ProcEnviron:
TERM=screen-bce
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: linux-hwe
UpgradeStatus: No upgrade log present (probably fresh install)
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 May 7 22:23 seq
crw-rw---- 1 root audio 116, 33 May 7 22:23 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.16
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
HibernationDevice: RESUME=/dev/mapper/localhost-swap
InstallationDate: Installed on 2018-03-22 (47 days ago)
InstallationMedia: Ubuntu-Server 16.04.4 LTS "Xenial Xerus" - Release amd64 (20180228)
IwConfig: Error: [Errno 2] No such file or directory
Lsusb:
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.13.0-39-generic root=/dev/mapper/localhost-root ro
ProcVersionSignature: Ubuntu 4.13.0-39.44~16.04.1-generic 4.13.16
RelatedPackageVersions:
linux-restricted-modules-4.13.0-39-generic N/A
linux-backports-modules-4.13.0-39-generic N/A
linux-firmware 1.157.17
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
Uname: Linux 4.13.0-39-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-2.11
dmi.modalias: dmi:bvnSeaBIOS:bvrrel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-2.11:cvnQEMU:ct1:cvrpc-i440fx-2.11:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-2.11
dmi.sys.vendor: QEMU
---
AlsaDevices:
total 0
crw-rw----+ 1 root audio 116, 1 May 9 01:40 seq
crw-rw----+ 1 root audio 116, 33 May 9 01:40 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 18.04
HibernationDevice: RESUME=/dev/mapper/localhost--vg-swap_1
InstallationDate: Installed on 2018-04-27 (11 days ago)
InstallationMedia: Ubuntu-Server 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-20-generic root=/dev/mapper/hostname--vg-root ro
ProcVersionSignature: Ubuntu 4.15.0-20.21-generic 4.15.17
RelatedPackageVersions:
linux-restricted-modules-4.15.0-20-generic N/A
linux-backports-modules-4.15.0-20-generic N/A
linux-firmware 1.173
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic
Uname: Linux 4.15.0-20-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-2.11
dmi.modalias: dmi:bvnSeaBIOS:bvrrel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-2.11:cvnQEMU:ct1:cvrpc-i440fx-2.11:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-2.11
dmi.sys.vendor: QEMU
Just tested again on new Ubuntu 18.04 with kernel 4.15. The direct synchronous small 4K write performance is still slower than kernel 4.4 on Ubuntu 16.04.
Are kernel 4.13 and 4.15 having write performance issue on qemu 2.11.x?
##
# ubuntu bionic kernel 4.15. the direct synchronous write performance is still 93% slower than kernel 4.4 on ubuntu xenial.
##
root@localhost:~# dd if=/dev/zero of=/tmp/test bs=4K count=256 oflag=direct,sync status=progress; rm /tmp/test
1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.504 s, 299 kB/s
root@localhost:~# dd if=/dev/zero of=/tmp/test bs=4M count=256 oflag=direct,sync status=progress; rm /tmp/test
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.00597 s, 357 MB/s
root@localhost:~# uname -a
Linux localhost 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux