fs_fill in fs from ubuntu_ltp failed with ENOSPC when testing btrfs on B-5.3 ARM64 / P9, B-5.4, H-5.11 aws

[Po-Hsu Lin]

Issue found on B-5.3 with arm64 node appleton-kernel (5.3.0-53.47~18.04.1-generic)
This issue cannot be found on amd64 / i386 / P8 node (on P8 it's failing with TIMEOUT, bug 1878763), but can be found on another P9 node (baltar)

 tst_test.c:1316: TINFO: Testing on btrfs
 tst_mkfs.c:90: TINFO: Formatting /dev/loop2 with btrfs opts='' extra opts=''
 tst_test.c:1250: TINFO: Timeout per run is 0h 05m 00s
 fs_fill.c:103: TINFO: Running 10 writer threads
 tst_fill_fs.c:39: TINFO: open(): ENOSPC (28)
 tst_fill_fs.c:39: TINFO: open(): ENOSPC (28)
 tst_fill_fs.c:39: TINFO: open(): ENOSPC (28)
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 tst_fill_fs.c:39: TINFO: open(): ENOSPC (28)
 tst_fill_fs.c:39: TINFO: open(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread9/file0
 tst_fill_fs.c:39: TINFO: open(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread4/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread7/file0
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread2/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread6/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread10/file0
 safe_macros.c:366: TBROK: fs_fill.c:55: unlink(mntpoint/thread4/file0) failed: ENOSPC (28)

Please find attachment for the test log.

[Po-Hsu Lin (cypressyew)]

This can be found on 5.6.0-1023.23-oem on node amaura as well (passed on lagalla)

[Kelsey Steele (kelsey-steele)]

Spotted on Focal aws : 5.4.0-1026.26 : amd64

[Krzysztof Kozlowski (krzk)]

Found on groovy/aws 5.8.0-1036.38 and 5.8.0-1035.37 (AWS: t3a.2xlarge)

[Kleber Sacilotto de Souza (kleber-souza)]

Found with focal/linux 5.4.0-85.95 on arm64 node dazzle.

[Po-Hsu Lin (cypressyew)]

Found on hirsute/aws 5.11.0-1022.23 (AWS: t3a.2xlarge)

[Po-Hsu Lin (cypressyew)]

With Impish 5.13.0-23.23 node entei, it's failing with timeout on btrfs:

 tst_test.c:988: TINFO: Formatting /dev/loop3 with btrfs opts='' extra opts=''
 tst_test.c:1423: TINFO: Timeout per run is 0h 15m 00s
 fs_fill.c:103: TINFO: Running 130 writer threads
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread99/file0
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread3/file0
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread6/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread97/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread90/file0
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread44/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread9/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread40/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread87/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread8/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread43/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread10/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread85/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread24/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread78/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread35/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread80/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread126/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread119/file0
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread84/file0
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread93/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread30/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread55/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread64/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread5/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread65/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread19/file0
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 tst_fill_fs.c:59: TINFO: write(): ENOSPC (28)
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread105/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread95/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread113/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread66/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread61/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread92/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread125/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread60/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread26/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread121/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread68/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread56/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/thread73/file0
 fs_fill.c:53: TINFO: Unlinking mntpoint/th...

[Luke Nowakowski-Krijger (lukenow)]

Found on 2022.01.31/bionic/linux/4.15.0-169.177 on dryden.ppc64el

[Portia Stephens (portias)]

Found on linux-xilinx-zynqmp-5.15.0-1020.24/arm64 on zcu102 and zcu104

[Po-Hsu Lin (cypressyew)]

Despite we're having this fs_fill failure on XFS (bug 2024161) after LTP test suite update, this issue with btrfs is still visible on some instances / kernels

e.g. n2d-standard-2 with b-gcp-5.4.0-1111.120~18.04.1