directory-concurrent.sh.ext4 in ubuntu_ecryptfs failed on Atrful and Bionic ThunderX ARM64

[Po-Hsu Lin]

This directory-concurrent.sh.ext4 test will fail with 4.13 and 4.15 kernel on a ThunderX ARM64 node (passed on the other Moonshot node):

Steps:
  1. git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest-client-tests -b master-next
  2. git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest
  3. rm -fr autotest/client/tests
  4. ln -sf ~/autotest-client-tests autotest/client/tests
  5. AUTOTEST_PATH=/home/ubuntu/autotest sudo -E autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_ecryptfs/control

Output:
 Running 'tests/run_one.sh -K -t directory-concurrent.sh -b 1000000 -D /mnt/image -l /mnt/lower -u /mnt/upper -f ext4'
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 directory-concurrent FAIL

 Test Summary:
 0 passed
 1 failed

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-image-4.15.0-17-generic 4.15.0-17.18
ProcVersionSignature: User Name 4.15.0-17.18-generic 4.15.17
Uname: Linux 4.15.0-17-generic aarch64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 20 09:29 seq
 crw-rw---- 1 root audio 116, 33 Apr 20 09:29 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu5
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDmesg:

Date: Fri Apr 20 09:40:05 2018
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Cavium ThunderX CRB
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:
 0 EFI VGA
 1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-17-generic root=UUID=ddcb8b36-6267-4392-a1f9-74d9203b742e ro acpi=force iommu.passthrough=1
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-17-generic N/A
 linux-backports-modules-4.15.0-17-generic N/A
 linux-firmware 1.173
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/12/2012
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 5.11
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: To be filled by O.E.M.
dmi.board.vendor: To be filled by O.E.M.
dmi.board.version: To be filled by O.E.M.
dmi.chassis.asset.tag: To be filled by O.E.M.
dmi.chassis.type: 0
dmi.chassis.vendor: Cavium
dmi.chassis.version: To be filled by O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr5.11:bd12/12/2012:svnCavium:pnThunderXCRB:pvrTobefilledbyO.E.M.:rvnTobefilledbyO.E.M.:rnTobefilledbyO.E.M.:rvrTobefilledbyO.E.M.:cvnCavium:ct0:cvrTobefilledbyO.E.M.:
dmi.product.family: Default string
dmi.product.name: ThunderX CRB
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Cavium

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1765653

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Andy Whitcroft (apw)]

This bug was nominated against a series that is no longer supported, ie artful. The bug task representing the artful nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Po-Hsu Lin (cypressyew)]

If you run the directory-concurrent.sh for ext4 at the very beginning, it will pass.

But if you run the trunc-file test against ext4 first and then this one, it will fail. This can be easily reproduced.

[Manoj Iyer (manjo)]

This is a test case issue, and the failure is a false positive. ecryptfs/tests/kernel/directory-concurrent/test.c hang_check() the parent uses select() to check to see if the child process has completed. The child process lets the parent know that its done by writing "EXIT" to a pipe. select() is non blocking while read() is blocking, so the select() would timeout after 30s if nothing was read from the pipe, this timeout state is used to detect a hang in rmdir/mkdir. But the test does not take into account of the fact that there is no guarantee that the the order of execution after a fork() will be child first and then parent. If the parent executes before the child, we hit these false positives.

The sync between parent and child can be achieved by calling wait(NULL) before select(), ie let the parent wait for the child state to change, but if there really is a hang in rmdir/mkdir the parent would hang indefinitely, and that would defeat the tests purpose of detecting hangs.

I found that as a workaround, adding a sleep(1) after write() in the child helps with resolving this issue. I tested this multiple times successfully.

[Manoj Iyer (manjo)]

With this patch applied, the test case directory-current does not report these false positives.

ubuntu@recht:~/ecryptfs/tests$ sudo ./run_tests.sh -K -t trunc-file.sh -b 1000000 -D /mnt/image -l /mnt/lower -u /mnt/upper -f ext4 ; sleep 3s; sudo ./run_tests.sh -K -t directory-concurrent.sh -b 1000000 -D /mnt/image -l /mnt/lower -u /mnt/upper -f ext4
Running eCryptfs filesystem tests on ext4
trunc-file pass

Test Summary:
1 passed
0 failed
Running eCryptfs filesystem tests on ext4
directory-concurrent pass

Test Summary:
1 passed
0 failed

[Sean Feole (sfeole)]

As the ecryptfs support has been dropped in Bionic+(https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1798359) marking bug invalid