ubuntu_vfat_stress constantly failing on Azure Standard_D11_v2, Standard_D2_v3

[Po-Hsu Lin]

With Bionic 4.15.0-1035.36 Azure kernel
The ubunut_vfat_stress test keep failing on these two instances (fail rate 3/3 on each instances)

For Standard_D11_v2, the failure could be found on Azure 4.15.0-1035.36~16.04.1 as well (Standard_D2_v3 passed with it)

Log:
timestamp=1543832442 localtime=Dec 03 10:20:42 Command <LOG=/tmp/vfat-falure.log STRESS_NG=/home/azure/autotest/client/tmp/ubuntu_vfat_stress/src/stress-ng/stress-ng DURATION=10s bash -c /home/azure/autotest/client/tests/ubuntu_vfat_stress/ubuntu_vfat_stress.sh /home/azure/autotest/client/tmp/ubuntu_vfat_stress/src 2>&1> failed, rc=1, Command returned non-zero exit status
  * Command:
      LOG=/tmp/vfat-falure.log
      STRESS_NG=/home/azure/autotest/client/tmp/ubuntu_vfat_stress/src/stress-ng
      /stress-ng DURATION=10s bash -c
      /home/azure/autotest/client/tests/ubuntu_vfat_stress/ubuntu_vfat_stress.sh
      /home/azure/autotest/client/tmp/ubuntu_vfat_stress/src 2>&1
  Exit status: 1
  Duration: 5.34092283249

  stdout:
  Testing: --verify --times --metrics-brief --syslog --keep-name -t 10s --hdd 2 --hdd-opts sync,wr-rnd,rd-rnd,fadv-willneed,fadv-rnd --lockf 2 --seek 2 --aio 2 --aio-requests 32 --dentry 2 --dir 2 --dentry-order stride --fallocate 2 --fstat 2 --dentries 100 --lease 2 --open 2 --rename 2 --hdd-bytes 4M --fallocate-bytes 2M --chdir 2 --rename 2 --hdd-write-size 512
  Created loop image vfat-test-13123/vfat-image-13123.img on /dev/loop0
  mkfs.fat 4.1 (2017-01-24)

  --------------------------------------------------------------------------------
  VFAT options: allow_utime=20
  Stress test: /home/azure/autotest/client/tmp/ubuntu_vfat_stress/src/stress-ng/stress-ng --verify --times --metrics-brief --syslog --keep-name -t 10s --hdd 2 --hdd-opts sync,wr-rnd,rd-rnd,fadv-willneed,fadv-rnd --lockf 2 --seek 2 --aio 2 --aio-requests 32 --dentry 2 --dir 2 --dentry-order stride --fallocate 2 --fstat 2 --dentries 100 --lease 2 --open 2 --rename 2 --hdd-bytes 4M --fallocate-bytes 2M --chdir 2 --rename 2 --hdd-write-size 512
  VFAT_IMAGE: vfat-test-13123
  Image Size: 256M
  Loop device: /dev/loop0
  Mount point: /mnt/vfat-test-13123
  Date: Mon Dec 3 10:20:37 UTC 2018
  Host: b-la-azure-4-15-0-Standard-D11-v2-vfat-stress
  Kernel: 4.15.0-1035-azure #36-Ubuntu SMP Fri Nov 30 12:47:16 UTC 2018
  Machine: b-la-azure-4-15-0-Standard-D11-v2-vfat-stress x86_64 x86_64
  CPUs online: 2
  CPUs total: 2
  Page size: 4096
  Pages avail: 3493871
  Pages total: 3590614
  --------------------------------------------------------------------------------

  Started, PID 13175
  stress-ng now terminated
  Stopped, PID 13175 (after 0 seconds)
  umounting vfat /dev/loop0 /mnt/vfat-test-13123
  umount: /mnt/vfat-test-13123: target is busy.
  umount vfat /mnt/vfat-test-13123 failed
  destroying loop /dev/loop0
  rmdir: failed to remove '/mnt/vfat-test-13123': Device or resource busy
  ================================================================================
  /mnt/vfat-test-13123 is already mounted!

Syslog could be found here: https://pastebin.ubuntu.com/p/C7RC6fXNnQ/

[Colin Ian King (colin-king)]

Seems like stress-ng is in the process of spawning off child stressors after it has checked that all the stressors have been terminated and we are left with some residual processes still accessing the mounted vfat filesystem. Use lsof to check for these and retry the kill until all stress-ng process are gone.

Fix commtted: http://kernel.ubuntu.com/git/ubuntu/autotest-client-tests.git/commit/?id=a699cab26ef3bba531faed774a0b880470552fca

[Po-Hsu Lin (cypressyew)]

Re-testing on Standard_D11_v2, Standard_D2_v3

[Po-Hsu Lin (cypressyew)]

Passed as expected!
Thanks.