Slow du performance on Lustre for large directories

[Timothy Day]

Quoting the upstream bug report:

"The gnulib function dirent_inode_sort_may_be_useful() should return
false for Lustre (i.e. #define S_MAGIC_LUSTRE 0x0BD00BD0 as seen
in lustre/include/uapi/linux/lustre/lustre_user.h in the Lustre source
tree as LL_SUPER_MAGIC [1]). Sorting dirents negatively impacts du
performance on Lustre because it interferes with Lustre's ability to
prefetch file metadata (via statahead).

For context, Lustre is an open-source (GPLv2) out-of-tree Linux filesystem
commonly used for HPC applications."

This patch was merged upstream as:

https://github.com/coreutils/gnulib/commit/578b8d7dc5e3fc00d308660fa60bb529a2e42bb3

Could we cherry-pick this to Ubuntu gnulib?

[Ojas (ojasgadre)]

patch returns false for lustre

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lustre-fix.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Nick Rosbrook (enr0n)]

Hi Timothy,

Can you provide some more context on (a) what problem you are facing that this patch solves, and (b) which release(s) of Ubuntu you want fixed? In principal, the patch seems fine, but if you are requesting a backport to a stable release, we need to follow the SRU process[1], which among other things, requires a test plan that demonstrates the bug before the fix, and demonstrates the problem is solved after the fix[2].

I am unsubscribing Ubuntu sponsors for now, because this is not yet actionable on that front.

[1] https://documentation.ubuntu.com/project/SRU/stable-release-updates/index.html
[2] https://documentation.ubuntu.com/project/SRU/reference/bug-template/#reference-sru-bug-template

[Timothy Day (tim-day-387)]

(a) When using the Lustre filesystem on Ubuntu, du performance on large directories (i.e. anything larger than 10k files) is very slow. This is because gnulib will sort dirents in large directories. This harms the Lustre client's attempt to prefetch metadata from the Lustre server. The bug fix above disables this sorting in gnulib (similar to NFS or tmpfs). This improves the performance around 9x.

(b) U22 and U24, ideally.

Regarding a test plan - Testing the fix requires setting up a Lustre client and Lustre server. Lustre is not packaged with Ubuntu by default. So you would need to build Lustre from source, if you want to test it yourself. However, I've personally validated the patch. How should we approach a test plan in this scenario?

[Patrick Robert Iacob (iacobp)]

Hi Timothy. I've been working on getting this patched in. The patches seem to build alright. Currently working on confirming the performance improvements and quantifying them before writing an sru. I think in this case a sufficient test plan could be simply building Luster, setting up a new volume, creating a few thousand files, and measuring du performance between patched and unpatched gnulib.
Do you know if this is an issue that affects luster when installed as a client or server on ubuntu? Just the client would be alot easier to test as it seems lustre does not currently support any version of ubuntu as a server and has some trouble building the latest lts.