directory-concurrent.sh.ext4 in ubuntu_ecryptfs failed on Atrful and Bionic ThunderX ARM64

Bug #1765653 reported by Po-Hsu Lin on 2018-04-20
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
eCryptfs
Undecided
Tyler Hicks
ubuntu-kernel-tests
Undecided
Unassigned
linux (Ubuntu)
Medium
Unassigned
Artful
Medium
Unassigned
Bionic
Medium
Unassigned

Bug Description

This directory-concurrent.sh.ext4 test will fail with 4.13 and 4.15 kernel on a ThunderX ARM64 node (passed on the other Moonshot node):

Steps:
  1. git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest-client-tests -b master-next
  2. git clone --depth=1 git://kernel.ubuntu.com/ubuntu/autotest
  3. rm -fr autotest/client/tests
  4. ln -sf ~/autotest-client-tests autotest/client/tests
  5. AUTOTEST_PATH=/home/ubuntu/autotest sudo -E autotest/client/autotest-local --verbose autotest/client/tests/ubuntu_ecryptfs/control

Output:
 Running 'tests/run_one.sh -K -t directory-concurrent.sh -b 1000000 -D /mnt/image -l /mnt/lower -u /mnt/upper -f ext4'
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing mkdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 Timed out after 30 seconds doing rmdir() - possible eCryptfs hang
 directory-concurrent FAIL

 Test Summary:
 0 passed
 1 failed

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-image-4.15.0-17-generic 4.15.0-17.18
ProcVersionSignature: User Name 4.15.0-17.18-generic 4.15.17
Uname: Linux 4.15.0-17-generic aarch64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Apr 20 09:29 seq
 crw-rw---- 1 root audio 116, 33 Apr 20 09:29 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu5
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDmesg:

Date: Fri Apr 20 09:40:05 2018
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Cavium ThunderX CRB
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcFB:
 0 EFI VGA
 1 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-17-generic root=UUID=ddcb8b36-6267-4392-a1f9-74d9203b742e ro acpi=force iommu.passthrough=1
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-17-generic N/A
 linux-backports-modules-4.15.0-17-generic N/A
 linux-firmware 1.173
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/12/2012
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 5.11
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: To be filled by O.E.M.
dmi.board.vendor: To be filled by O.E.M.
dmi.board.version: To be filled by O.E.M.
dmi.chassis.asset.tag: To be filled by O.E.M.
dmi.chassis.type: 0
dmi.chassis.vendor: Cavium
dmi.chassis.version: To be filled by O.E.M.
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr5.11:bd12/12/2012:svnCavium:pnThunderXCRB:pvrTobefilledbyO.E.M.:rvnTobefilledbyO.E.M.:rnTobefilledbyO.E.M.:rvrTobefilledbyO.E.M.:cvnCavium:ct0:cvrTobefilledbyO.E.M.:
dmi.product.family: Default string
dmi.product.name: ThunderX CRB
dmi.product.version: To be filled by O.E.M.
dmi.sys.vendor: Cavium

Po-Hsu Lin (cypressyew) wrote :
Po-Hsu Lin (cypressyew) on 2018-04-20
summary: directory-concurrent.sh.ext4 in ubuntu_ecryptfs failed on Atrful and
- Bionic
+ Bionic ARM64
description: updated
summary: directory-concurrent.sh.ext4 in ubuntu_ecryptfs failed on Atrful and
- Bionic ARM64
+ Bionic ThunderX ARM64
description: updated

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1765653

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Changed in linux (Ubuntu Artful):
status: New → Triaged
Changed in linux (Ubuntu Bionic):
status: Incomplete → Triaged
importance: Undecided → Medium
Changed in linux (Ubuntu Artful):
importance: Undecided → Medium
tags: added: kernel-da-key

This bug was nominated against a series that is no longer supported, ie artful. The bug task representing the artful nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu Artful):
status: Triaged → Won't Fix
Po-Hsu Lin (cypressyew) wrote :

If you run the directory-concurrent.sh for ext4 at the very beginning, it will pass.

But if you run the trunc-file test against ext4 first and then this one, it will fail. This can be easily reproduced.

Manoj Iyer (manjo) wrote :

This is a test case issue, and the failure is a false positive. ecryptfs/tests/kernel/directory-concurrent/test.c hang_check() the parent uses select() to check to see if the child process has completed. The child process lets the parent know that its done by writing "EXIT" to a pipe. select() is non blocking while read() is blocking, so the select() would timeout after 30s if nothing was read from the pipe, this timeout state is used to detect a hang in rmdir/mkdir. But the test does not take into account of the fact that there is no guarantee that the the order of execution after a fork() will be child first and then parent. If the parent executes before the child, we hit these false positives.

The sync between parent and child can be achieved by calling wait(NULL) before select(), ie let the parent wait for the child state to change, but if there really is a hang in rmdir/mkdir the parent would hang indefinitely, and that would defeat the tests purpose of detecting hangs.

I found that as a workaround, adding a sleep(1) after write() in the child helps with resolving this issue. I tested this multiple times successfully.

Manoj Iyer (manjo) wrote :

With this patch applied, the test case directory-current does not report these false positives.

ubuntu@recht:~/ecryptfs/tests$ sudo ./run_tests.sh -K -t trunc-file.sh -b 1000000 -D /mnt/image -l /mnt/lower -u /mnt/upper -f ext4 ; sleep 3s; sudo ./run_tests.sh -K -t directory-concurrent.sh -b 1000000 -D /mnt/image -l /mnt/lower -u /mnt/upper -f ext4
Running eCryptfs filesystem tests on ext4
trunc-file pass

Test Summary:
1 passed
0 failed
Running eCryptfs filesystem tests on ext4
directory-concurrent pass

Test Summary:
1 passed
0 failed

Manoj Iyer (manjo) on 2018-09-14
Changed in linux (Ubuntu Bionic):
status: Triaged → Invalid
Changed in linux (Ubuntu):
status: Triaged → Invalid
Tyler Hicks (tyhicks) on 2018-09-14
Changed in ecryptfs:
assignee: nobody → Tyler Hicks (tyhicks)
status: New → Triaged
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers