Optane PM access times are much longer than they should be

[steve heller]

I have 4 128GB Optane DC Persistent Memory devices in my server, assigned to AppDirect access (i.e., accessed via memory-mapping at the cache line level), and have run into a severe performance issue with the latest kernel 5.0.0-31.

I realize that not many people have this storage type as yet, but it is becoming increasingly important in the server world, so I think this bug is pretty important.

Bug details:

When running with kernel 5.0.0-29, write access times to the Optane PM devices are in the sub-microsecond range, as they should be.

When running with kernel 5.0.0-31, write access times are much slower, averaging in the 30 microsecond range, until a large number of writes (see below) have been executed. Then they return to normal speed.

My application is a hash table implementation optimized for the Optane PM devices (see webpage at www.threemisses.com). Since it is a hash table, it has very poor locality of reference, and is filled "randomly", so I don't know exactly what the sequence of writes is.

However, I can say that the speed is very slow until the fill ratio of the hash table is about 50%, then it returns to the normal sub-microsecond range.

ProblemType: Bug
DistroRelease: Ubuntu 19.04
Package: linux-image-5.0.0-31-generic 5.0.0-31.33
ProcVersionSignature: Ubuntu 5.0.0-31.33-generic 5.0.21
Uname: Linux 5.0.0-31-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.10-0ubuntu27.1
Architecture: amd64
Date: Tue Oct 15 09:01:06 2019
InstallationDate: Installed on 2019-09-02 (42 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
IwConfig:
 lo no wireless extensions.

 eno0 no wireless extensions.

 enp181s0f1 no wireless extensions.
MachineType: Supermicro SYS-7039A-I
ProcEnviron:
 LD_LIBRARY_PATH=<set>
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 EFI VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.0.0-31-generic root=UUID=3239f211-86a3-44bb-8251-62037ebea848 ro transparent_hugepage=always quiet splash nomodeset vt.handoff=1
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-5.0.0-31-generic N/A
 linux-backports-modules-5.0.0-31-generic N/A
 linux-firmware 1.178.3
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 03/06/2019
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 3.0a
dmi.board.asset.tag: ZM193S000653
dmi.board.name: X11DAi-N
dmi.board.vendor: Supermicro
dmi.board.version: 1.02
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 1
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 123456789
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr3.0a:bd03/06/2019:svnSupermicro:pnSYS-7039A-I:pvr123456789:rvnSupermicro:rnX11DAi-N:rvr1.02:cvnSupermicro:ct1:cvr123456789:
dmi.product.family: SMC X11
dmi.product.name: SYS-7039A-I
dmi.product.sku: 097A15D9
dmi.product.version: 123456789
dmi.sys.vendor: Supermicro

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[steve heller (technovelist)]

I have confirmed that the bug still exists in 5.6.3.

[steve heller (technovelist)]

I believe this bug is due to a change that causes files on a dax volume to be allocated as sparse files, so that every write to a new allocation unit causes another physical allocation. The previous behavior was to allocate the whole file up to the allocated size as one operation.

This is just by observing the behavior and running experiments to try to figure it out. I haven't looked at the code.

[Kai-Heng Feng (kaihengfeng)]

Please test latest mainline kernel:
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4-rc4/

And raise the issue to mailing list if the issue persists.

[steve heller (technovelist)]

On Thu, 24 Oct 2019 15:21:15 -0000, Kai-Heng Feng <email address hidden> wrote:

>Please test latest mainline kernel:
>https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4-rc4/
>
>And raise the issue to mailing list if the issue persists.

I'll do that as soon as possible, maybe tonight or tomorrow.
Thanks!
------------
Steve Heller

[steve heller (technovelist)]

https://lore.kernel.org/<email address hidden>

[steve heller (technovelist)]

Is there an estimate as to when you are likely to pick up this fix?

[steve heller (technovelist)]

This seems to be working correctly now here:

https://lore.kernel.org/<email address hidden>/