Server locks up listing files on mounted volume

[Pieter Coetzee]

## Using latest Centos Plus 7.3 kernel:
kernel-plus-3.10.0-514.21.1.el7.centos.plus.x86_64

## sestatus selinux disabled
SELinux status: disabled

#> modprobe ecryptfs

#> modinfo ecryptfs
filename: /lib/modules/3.10.0-514.21.1.el7.centos.plus.x86_64/kernel/fs/ecryptfs/ecryptfs.ko
license: GPL
rhelversion: 7.3
srcversion: D8F287B49D8815C77E03B2A
vermagic: 3.10.0-514.21.1.el7.centos.plus.x86_64 SMP mod_unload modversions
signer: CentOS Linux kernel signing key
sig_key: 63:61:15:6D:94:4C:B1:0B:FE:49:86:1C:47:08:33:42:5D:C6:4E:74

## Using utils from epel-testing repo:
ecryptfs-utils-111-5.el7.x86_64

#> ecryptfs-setup-private
#> ecryptfs-add-passphrase --fnek
#> chmod +s /sbin/mount.ecryptfs*

#> ecryptfs-mount-private
Enter your login passphrase:
Inserted auth tok with sig [4bf790158acc1a6d] into the user session keyring

#> mount
/var/lib/mysql/.Private on /var/lib/mysql/data type ecryptfs (rw,nosuid,nodev,relatime,ecryptfs_fnek_sig=f97cd4d9e90f8d23,ecryptfs_sig=4bf790158acc1a6d,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_unlink_sigs)

#> ls -latr /var/lib/mysql
drwx------ 2 mysql mysql 102 Jun 20 16:27 .ecryptfs
drwx------ 16 mysql mysql 4096 Jun 21 12:03 .Private
drwx------ 16 mysql mysql 4096 Jun 21 12:03 data
-rw-r----- 1 mysql mysql 86 Jun 21 12:05 relay-log.info
drwxr-xr-x. 46 root root 4096 Jun 21 15:14 ..
srwxrwxrwx 1 mysql mysql 0 Jun 21 15:15 mysql.sock
drwxr-xr-x 5 mysql mysql 106 Jun 21 15:15 .
-rw------- 1 mysql mysql 4152 Jun 21 15:44 .bash_history

#> getfacl /var/lib/mysql/
getfacl: Removing leading '/' from absolute path names
# file: var/lib/mysql/
# owner: mysql
# group: mysql
user::rwx
group::r-x
other::r-x

## Can copy files in and out from /var/lib/mysql/data
For eaxample:
cp -avf /var/mysql/* /var/lib/mysql/data/

## Can start service mysql
#> systemctl start mysql

## Streaming syslog to external server
#> tail -f /var/log/rsyslog.log
2017-06-21T16:20:53+10:00 systemd: Starting MySQL Community Server...
2017-06-21T16:20:57+10:00 systemd: Started MySQL Community Server.

*** Please help: Unable to resolve the following
Server freezes and hangs after any of the following:
$>ls -la /var/lib/mysql/data/*
$>getfacl /var/lib/mysql/data
#> mysql
mysql> show databases;
mysql> use mysql;

*** No information after server lockup ****
No further information recorded by remote syslog.
No response to magic SysRq keys
SSH session dropped with broken pipe

Requires hard power down restart. No debugging information of any kind found.
Tested on AWS with transparent encrypted volumes as well as kvm with luks encrypted drives.

*** Any thoughts, fixes, recommendations??

Thanks heaps

Server hangs and drops ssh

[Jason Xing (wlxing)]

Does it work well when you create/remove some files in /var/lib/mysql? Can you see the contents of files in that directory when mounting/unmounting?

I don't know how you can mount from <.Private> to <mysql>? I'm trying to reproduce...

[Pieter Coetzee (pcoetzee)]

Hi Jason,

I followed the instructions from the wiki in order to create the mount:

https://wiki.archlinux.org/index.php/ECryptfs

"The mount point ("upper directory") for the encrypted folder will be at ~/Private by default, however you can manually change this right after the setup command has finished running, by doing:
$ mv ~/Private /path/to/new/folder
$ echo /path/to/new/folder > ~/.ecryptfs/Private.mnt"

i.e.
#> su - mysql
$> mv ~/Private /var/lib/mysql/data
$> echo /var/lib/mysql/data > ~/.ecryptfs/Private.mnt

* I can not enter the mounted dir /var/lib/mysql/data in order to create or remove files.
* I can copy files into the mounted drive /var/lib/mysql/data directory and also out from the mounted drive
* I can not remove/delete files from the mounted directory
* The .Private dir appears to get populated with encrypted data

For example:
$> ls -latr /var/lib/mysql/.Private/
total 1126988
-rw-rw---- 1 mysql mysql 12288 Sep 21 2016 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUYS3s-lAKefWI9kFl2EtfrU--
drwx------ 2 mysql mysql 8192 Jun 6 13:07 ECRYPTFS_FNEK_ENCRYPTED.FXbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUwuVE.Gar3U0VrJGI1iCWBcYyIgYDxJi33xH82.4OBx--
drwx------ 2 mysql mysql 12288 Jun 6 15:49 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUik5mmKXwWLAUK4w1Jo5V5U--
drwx------ 2 mysql mysql 97 Jun 6 15:49 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUTI6BCTA5AT3yxhvXnNDc1---
drwx------ 2 mysql mysql 97 Jun 6 15:49 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUtENkTvY9AyHa7nFMbB-ExU--
drwx------ 2 mysql mysql 97 Jun 6 15:49 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUI4mX-6UV57AQs-e75Nm7zU--
drwx------ 2 mysql mysql 97 Jun 6 15:49 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUjdx38rncIgb4Bhstqzrf6E--
drwx------ 2 mysql mysql 97 Jun 6 15:49 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUjm03VHpaDHsx9f74mWx4nU--
drwx------ 2 mysql mysql 12288 Jun 6 15:49 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUtxQR4hGMFujLmhZh62NGJ---
drwx------ 2 mysql mysql 8192 Jun 6 15:49 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUmr.SoEKt-8K7QVEwSsV0lk--
drwx------ 2 mysql mysql 97 Jun 6 15:54 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUpRWLpx0YOiDyghjh3uEPyk--
drwx------ 2 mysql mysql 97 Jun 6 15:54 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUghwO6ut2Cmy2Qxu.6f0TD---
drwx------ 2 mysql mysql 188416 Jun 6 15:54 ECRYPTFS_FNEK_ENCRYPTED.FXbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUUVDmLTIVZRCkzC4HYmLPQ8a4SPvDEfP6dZoQk8yuQgc-
drwx------ 2 mysql mysql 233472 Jun 6 15:54 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUDrHV7wtWDlUTcij1yjEhBU--
-rw-rw---- 1 mysql mysql 12288 Jun 20 12:28 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUT1zhFGypSUVVuxZ60Rov-U--
-rw-rw---- 1 mysql mysql 12288 Jun 20 12:28 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0yq6bUTnpJebJNSFj0twnV1w3acU--
-rw-rw---- 1 mysql mysql 536879104 Jun 20 12:28 ECRYPTFS_FNEK_ENCRYPTED.FWbtTBHNuEyB6kTg8F.wxHMvzsx5XP0...

[Pieter Coetzee (pcoetzee)]

If tried to set reboot on oops and kernel panic of three seconds but it does not appear to be hitting a kernel panic.

It appears to me more like all mounts (hard drive devices??) are instantly dropped the moment you try do anything inside the ecryptfs mounted volume:

On the virtual machine screen console I can only see the following after the lockup:

No irq handler for vector (irq -1)
[ 1.916935] mce: Unable to init device /dev/mcelog (rc: -5)
Please enter your passphrase for disk QEMU_HARDDISK (luks-3c587311-1fdf-4c41-a108-f8aa303d488a)!

## I have tried two different AWS servers and two different KVM machines with the same results. I have also source compiled the last 5 ecryptfs-utils release from the archives to no avail.

Thank you very much for you time.

Kind regards,
Pieter

Only way out of this situation is to power off and power back on.

[Jason Xing (wlxing)]

Hello,

I strongly recommend using "sudo mount -t <encrypted> <decrypted>" instead of using "ecryptfs-mount-private", then you could create or remove files. You could try that command and then see what will happen.

Regards,
Jason

[Pieter Coetzee (pcoetzee)]

Tried various mounting options as well as various mount points. I followed the steps for manual mounting but none of these actions appears to have any affect and the end result of a system crash without any error or warning is always and consistently the same.

-bash-4.2$ sudo mount -t ecryptfs /var/lib/mysql/.Private /var/lib/mysql/data

Select key type to use for newly created files:
 1) passphrase
 2) openssl
Selection: 1
Passphrase:
Select cipher:
 1) aes: blocksize = 16; min keysize = 16; max keysize = 32
 2) blowfish: blocksize = 8; min keysize = 16; max keysize = 56
 3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24
 4) twofish: blocksize = 16; min keysize = 16; max keysize = 32
 5) cast6: blocksize = 16; min keysize = 16; max keysize = 32
 6) cast5: blocksize = 8; min keysize = 5; max keysize = 16
Selection [aes]: 1
Select key bytes:
 1) 16
 2) 32
 3) 24
Selection [16]: 1
Enable plaintext passthrough (y/n) [n]: y
Enable filename encryption (y/n) [n]: n
Attempting to mount with the following options:
  ecryptfs_unlink_sigs
  ecryptfs_passthrough
  ecryptfs_key_bytes=16
  ecryptfs_cipher=aes
  ecryptfs_sig=4bf790158acc1a6d
Mounted eCryptfs

-bash-4.2$ mount
...
tmpfs on /run/user/1004 type tmpfs (rw,nosuid,nodev,relatime,size=368528k,mode=700,uid=1004,gid=1004)
/var/lib/mysql/.Private on /var/lib/mysql/data type ecryptfs (rw,relatime,ecryptfs_sig=4bf790158acc1a6d,ecryptfs_cipher=aes,ecryptfs_key_bytes=16,ecryptfs_passthrough,ecryptfs_unlink_sigs)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=368528k,mode=700,uid=1001,gid=1001)

-bash-4.2$ pwd

/var/lib/mysql

-bash-4.2$ ls -latr
total 16
drwx------ 2 mysql mysql 102 Jun 20 16:27 .ecryptfs
-rw-r----- 1 mysql mysql 86 Jun 21 12:05 relay-log.info
-rw------- 1 mysql mysql 4671 Jun 21 17:13 .bash_history
drwxr-xr-x. 46 root root 4096 Jun 22 10:46 ..
drwx------ 2 mysql mysql 6 Jun 22 11:08 .Private
drwx------ 2 mysql mysql 4096 Jun 22 11:08 data
drwxr-xr-x 5 mysql mysql 89 Jun 23 10:10 .

-bash-4.2$ cd data

-bash-4.2$ ls

### System Crash####

Thank you very much for your time.

Kind regards,
Pieter

[Jason Xing (wlxing)]

I mean create two directories and do manual mount again.It seems that you've tried this method. Okay, that's really weird.Do you have some messages in the log (checking by "dmesg")?

@Tyler Can you take a look at this. Thanks.

[Pieter Coetzee (pcoetzee)]

No, unfortunately as stated before - no errors, no logs and no information whatsoever.

I've enabled remote syslog and I have also tried to capture something with $> dmesg -wH but alas to no avail.

There are simply no error or debugging information reported on anything as far as I can see.

Kind regards,
Pieter

[Pieter Coetzee (pcoetzee)]

Managed to get the following from our journal logs recorded just prior to the system crash.

Jun 23 14:13:53 oca-test-stable2-luks sshd[3996]: Passphrase file wrapped
Jun 23 14:13:53 oca-test-stable2-luks sshd[3996]: pam_ecryptfs: Unable to rewrap passphrase file
Jun 23 14:13:53 oca-test-stable2-luks sshd[3996]: Failed to detect wrapped passphrase version: Permission denied
Jun 23 14:13:53 oca-test-stable2-luks sshd[3996]: Error attempting to unwrap passphrase from file [/var/lib/mysql/.ecryptfs/wrapped-passphrase]; rc = [-13]
Jun 23 14:13:53 oca-test-stable2-luks sshd[3996]: Error adding passphrase key token to user session keyring; rc = [-5]
Jun 23 14:13:59 oca-test-stable2-luks sudo[4026]: mysql : TTY=pts/1 ; PWD=/var/lib/mysql ; USER=root ; COMMAND=/bin/su
Jun 23 14:15:34 oca-test-stable2-luks sudo[4160]: mysql : TTY=pts/1 ; PWD=/var/lib/mysql ; USER=root ; COMMAND=/usr/bin/ng-journalctl --sudo -f
Jun 23 14:16:22 oca-test-stable2-luks ecryptfs-insert-wrapped-passphrase-into-keyring[4183]: Incorrect wrapping key for file [/var/lib/mysql/.ecryptfs/wrapped-passphrase]
Jun 23 14:16:22 oca-test-stable2-luks ecryptfs-insert-wrapped-passphrase-into-keyring[4183]: Error attempting to unwrap passphrase from file [/var/lib/mysql/.ecryptfs/wrapped-passphrase]; rc = [-5]
Jun 23 14:17:02 oca-test-stable2-luks sudo[4204]: mysql : TTY=pts/1 ; PWD=/var/lib/mysql ; USER=root ; COMMAND=/usr/bin/ng-journalctl --sudo -f
Timeout, server oca-test-stable2-luks not responding.

[Pieter Coetzee (pcoetzee)]

I think the Pam issues relating to automount may be a red herring.

I tried creating the crypted drive with the following:

#> ecryptfs-setup-private --nopwcheck --noautomount

Which does not generate the following pam messages:
: pam_ecryptfs: Unable to rewrap passphrase file
: Failed to detect wrapped passphrase version: Permission denied
: Error attempting to unwrap passphrase from file [/var/lib/mysql/.ecryptfs/wrapped-passphrase]; rc = [-13]
: Error adding passphrase key token to user session keyring; rc = [-5]

But alas the server crashes in exactly the same way.

I've tried setting permissions. Recreating the procedure. Modifying the pam files etc as discussed by other technologists on forums but unfortunately are yet to stumble across a workaround that works.

Kind regards,
Pieter

[Jason Xing (wlxing)]

Let me organize my thoughts. The purpose is that you want to use mysql in the mounted directory, right?
If so, please do this step by step.
1. Create two directories, say, <encrypted> and <decrypted>.
2. Execute "sudo mount -t ecryptfs <encrypted> <decrypted>"
3. Then use basic file operations command and manipulate mysql in <decrypted> to see what will happen.