ubuntu_ltp:fs:gf15 growfiles.c:134: Test failed (Hit max errors value of 1)

[Tim Gardner]

growfiles(gf15): 68703 growfiles.c/2262: 77415 tlibio.c/969 writev(8, iov, 1) nbyte:24346 returned=16219
growfiles(gf15): 68703 growfiles.c/1765: 77415 Hit max errors value of 1
gf15 1 TFAIL : growfiles.c:134: Test failed

[Po-Hsu Lin (cypressyew)]

Comment moved from bug 1936305:

Found on F-AWS-5.13 5.13.0-1008.9~20.04.2

Failed on certain instances:
  * c3.xlarge
  * c4.large
  * c5n.large
  * c6g.8xlarge
  * i3.metal
  * i3en.24xlarge
  * r5.large
  * r5.metal
  * t2.small
  * t3.medium

Test log:
Checking for required user/group ids

'nobody' user id and group found.
'bin' user id and group found.
'daemon' user id and group found.
Users group found.
Sys group found.
Required users/groups exist.
no big block device was specified on commandline.
Tests which require a big block device are disabled.
You can specify it with option -z
INFO: Test start time: Mon Dec 20 22:04:11 UTC 2021
COMMAND: /opt/ltp/bin/ltp-pan -q -e -S -a 63192 -n 63192 -f /tmp/ltp-LCg0vIt0Lj/alltests -l /dev/null -C /dev/null -T /dev/null
LOG File: /dev/null
FAILED COMMAND File: /dev/null
TCONF COMMAND File: /dev/null
Running tests.......
growfiles(gf15): 63234 growfiles.c/2262: 85703 tlibio.c/1002 pwrite(8, buf, 35574, 2122966446) returned=35410
growfiles(gf15): 63234 growfiles.c/1765: 85703 Hit max errors value of 1
gf15 1 TFAIL : growfiles.c:134: Test failed
INFO: ltp-pan reported some tests FAIL
LTP Version: 20210927
INFO: Test end time: Mon Dec 20 22:04:30 UTC 2021

[Po-Hsu Lin (cypressyew)]

Found on F-gkeop-5.15.0-1017.22~20.04.1 with n1-standard-64 and t2d-standard-4

[Thibf (thibf)]

Found on j-gkeop-5.15.0-1036.42 on multiple instances, restarting tests to see if it's flaky.

[Hui Wang (hui.wang)]

Probably need to set CONFIG_SCSI_VIRTIO to Y?

[Hui Wang (hui.wang)]

ran the testcase multiple times, still failed.

@Po-Hsu,

Do you know how to create vm and run the test manually?

[Po-Hsu Lin (cypressyew)]

Worked on this with Hui today, this issue seems to be caused by the insufficient disk space.

I have done some investigation on e2-highcpu-8 which has 10G disk. Here is the disk free space on this instance with different kernels:
* 1.9G free on / with J-gkeop
* 3.2G free on / with J-gke

A manual test for gf15 on e2-highcpu-8 with J-gkeop-5.15.0-1036.42 shows it will eat up all the free space very quickly and fail with the following error message in the end:
  growfiles(gf15): 62846 growfiles.c/2262: 78769 tlibio.c/739 write(3, buf, 4336) ret:-1, errno=28 No space left on device
  gf15 1 TFAIL : growfiles.c:134: Test failed

Although the error message is a bit different from what we see on the J-gkeop-5.15.0-1036.42 test report:
  growfiles(gf15): 58147 growfiles.c/2262: 81180 tlibio.c/1007 pwrite(8, buf, 46928, 2011908881) returned=13551
  growfiles(gf15): 58147 growfiles.c/1765: 81180 Hit max errors value of 1

But I think the disk space is the key. From this commit in LTP https://github.com/linux-test-project/ltp/commit/71791643b48de70cbff77ee4f9281fc6486b1814 it mentioned that:
> If -i option is set to zero, some growfiles tests (e.g. gf01,
> gf[10-11], gf[14-15], gf18) will grow every file size to 2G
> before stopping test.

And this is the command we use for gf15 test:
  gf15 growfiles -W gf15 -b -e 1 -u -r 1-49600 -I r -u -i 0 -L 120 -f Lgfile1 -d $TMPDIR

After free up some space, make 2.2G free on / with J-gkeop this test will pass.

The reason why J-gkeop used up more space then J-gke is because linux-firmware depends on linux-image-gkeop. I am not sure why we are installing linux-frmware package (which took 1G space) on this cloud kernel. If we can get rid of it this test should be able to pass.

[Hui Wang (hui.wang)]

The linux-meta package of j-gkeop has that dependency:

hwang4@hwang4-ThinkPad-T14s-Gen-2a:~/test/gkeop/new/linux-meta$ git diff
diff --git a/debian/control.d/gkeop b/debian/control.d/gkeop
index e27ebd3..0e78c21 100644
--- a/debian/control.d/gkeop
+++ b/debian/control.d/gkeop
@@ -12,7 +12,6 @@ Section: kernel
 Provides: ${dkms:zfs-modules}
 Depends: ${misc:Depends},
  linux-image-${kernel-abi-version}-gkeop,
- linux-firmware,
  intel-microcode,
  amd64-microcode
 Recommends:

[Po-Hsu Lin (cypressyew)]

@hui.wang
I have added df -h command to the test and restarted those failing instance, the free disk space are just 1.9G, I will add a hint with bug 1936305 (No space left on device) for them this time.
Meanwhile I will discuss with other TNT folks to see if we can bump the disk size.