mysqld is terminating if we give different service name in PAM plugin group mapping

[Ramesh Sivaraman]

mysqld is terminating if we give different service name in PAM plugin group mapping. In this testcase replaced 'mysqld' service name with 'xyz'

Testcase

INSTALL PLUGIN auth_pam SONAME 'auth_pam.so';
CREATE USER ''@'' IDENTIFIED WITH auth_pam AS 'xyz, pam_db1=db1_dev';
flush privileges;

When we try to login using any OS user mysqld will terminate without writing anything in error log.

$ ./bin/mysql -umytest -p --socket=/home/ramesh/Percona-Server-5.6.25-rel73.1-d0661a9.Linux.x86_64-debug/socket.sock -e" SELECT USER(), CURRENT_USER()"
Enter password:
ERROR 2013 (HY000): Lost connection to MySQL server during query
$

[Sergei Glushchenko (sergei.glushchenko)]

I am not able to reproduce the crash. I am getting 'Access denied' error. Can you please share full config (including /etc/pam.d/* files) and the stack trace if possible.

[Ramesh Sivaraman (rameshvs02)]

Using binary tar ball testing

1) startup command

/home/ramesh/Percona-Server-5.6.25-rel73.1-d0661a9.Linux.x86_64-debug/bin/mysqld --no-defaults --core-file --innodb_buffer_pool_size=2147483648 --basedir=/home/ramesh/Percona-Server-5.6.25-rel73.1-d0661a9.Linux.x86_64-debug --tmpdir=/home/ramesh/Percona-Server-5.6.25-rel73.1-d0661a9.Linux.x86_64-debug/data --datadir=/home/ramesh/Percona-Server-5.6.25-rel73.1-d0661a9.Linux.x86_64-debug/data --socket=/home/ramesh/Percona-Server-5.6.25-rel73.1-d0661a9.Linux.x86_64-debug/socket.sock --port=12894 --log-error=/home/ramesh/Percona-Server-5.6.25-rel73.1-d0661a9.Linux.x86_64-debug/log/master.err 2>&1 &

2) remove anonymous users

delete from mysql.user where user='';

3) Install PAM plugin and create plam plugin user.

INSTALL PLUGIN auth_pam SONAME 'auth_pam.so';
CREATE USER ''@'' IDENTIFIED WITH auth_pam AS 'xyz, pam_db1=db1_dev';
flush privileges;

4) Run mysql -A -umytest -S/home/ramesh/Percona-Server-5.6.25-rel73.1-d0661a9.Linux.x86_64-debug/socket.sock test -p

PAM configuration

$ cat /etc/pam.d/mysqld
auth required pam_warn.so
auth required pam_unix.so audit
account required pam_unix.so audit
$

Attached strace info

[Ramesh Sivaraman (rameshvs02)]

GDB info. Also attached full bt info

(gdb) bt
#0 0x00007f7113bfac41 in pam_sm_authenticate () from /lib/security/pam_ecryptfs.so
#1 0x00007f71d5db2dcf in ?? () from /lib/x86_64-linux-gnu/libpam.so.0
#2 0x00007f71d5db265d in pam_authenticate () from /lib/x86_64-linux-gnu/libpam.so.0
#3 0x00007f71d5fea685 in authenticate_user_with_pam_server (vio=0x7f71d46a4450, info=0x7f71d46a4468)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/plugin/percona-pam-for-mysql/src/auth_pam_common.c:145
#4 0x00000000007689f7 in do_auth_once (thd=0x231ebd0, auth_plugin_name=0x7f71d46a47a0, mpvio=0x7f71d46a4450)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_acl.cc:11254
#5 0x0000000000769091 in acl_authenticate (thd=0x231ebd0, com_change_user_pkt_len=0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_acl.cc:11412
#6 0x00000000007a9138 in check_connection (thd=0x231ebd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1213
#7 0x00000000007a92e5 in login_connection (thd=0x231ebd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1283
#8 0x00000000007a9a2f in thd_prepare_connection (thd=0x231ebd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1452
#9 0x00000000007a9f4f in do_handle_one_connection (thd_arg=0x231ebd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1531
#10 0x00000000007a99f8 in handle_one_connection (arg=0x231ebd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1444
#11 0x0000000000dd113d in pfs_spawn_thread (arg=0x7f9cd30)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/storage/perfschema/pfs.cc:1860
#12 0x00007f71f1abe6aa in start_thread (arg=0x7f71d46a5700) at pthread_create.c:333
#13 0x00007f71f070ceed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)

[Sergei Glushchenko (sergei.glushchenko)]

Looks similar to upstream https://bugs.mysql.com/bug.php?id=74310

[Ramesh Sivaraman (rameshvs02)]

GDB info after installing debug symbols

(gdb) bt
#0 pam_sm_authenticate (pamh=0x7f5be4017cd0, flags=<optimized out>, argc=1, argv=0x7f5be4022e30) at pam_ecryptfs.c:142
#1 0x00007f5c0f1eddcf in _pam_dispatch_aux (use_cached_chain=<optimized out>, resumed=<optimized out>, h=<optimized out>, flags=0, pamh=0x7f5be4017cd0)
    at pam_dispatch.c:110
#2 _pam_dispatch (pamh=pamh@entry=0x7f5be4017cd0, flags=flags@entry=0, choice=choice@entry=1) at pam_dispatch.c:395
#3 0x00007f5c0f1ed65d in pam_authenticate (pamh=0x7f5be4017cd0, flags=0) at pam_auth.c:34
#4 0x00007f5c0f425685 in authenticate_user_with_pam_server (vio=0x7f5c0f666450, info=0x7f5c0f666468)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/plugin/percona-pam-for-mysql/src/auth_pam_common.c:145
#5 0x00000000007689f7 in do_auth_once (thd=0x3137bd0, auth_plugin_name=0x7f5c0f6667a0, mpvio=0x7f5c0f666450)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_acl.cc:11254
#6 0x0000000000769091 in acl_authenticate (thd=0x3137bd0, com_change_user_pkt_len=0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_acl.cc:11412
#7 0x00000000007a9138 in check_connection (thd=0x3137bd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1213
#8 0x00000000007a92e5 in login_connection (thd=0x3137bd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1283
#9 0x00000000007a9a2f in thd_prepare_connection (thd=0x3137bd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1452
#10 0x00000000007a9f4f in do_handle_one_connection (thd_arg=0x3137bd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1531
#11 0x00000000007a99f8 in handle_one_connection (arg=0x3137bd0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/sql/sql_connect.cc:1444
#12 0x0000000000dd113d in pfs_spawn_thread (arg=0x8de67d0)
    at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.25-73.1/storage/perfschema/pfs.cc:1860
#13 0x00007f5cbf15a6aa in start_thread (arg=0x7f5c0f667700) at pthread_create.c:333
#14 0x00007f5cbdda8eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)

[Sergei Glushchenko (sergei.glushchenko)]

It turned out to be stack overflow.

When ecryptfs_utils installed, pam_ecryptfs.so got itself written to several pam.d config files:

# grep pam_ecrypt -r /etc/pam.d/
/etc/pam.d/common-auth:auth optional pam_ecryptfs.so unwrap
/etc/pam.d/common-session:session optional pam_ecryptfs.so unwrap
/etc/pam.d/common-session-noninteractive:session optional pam_ecryptfs.so unwrap
/etc/pam.d/common-password:password optional pam_ecryptfs.so

It is not used when we specify mysqld as service name (because there is /etc/pam.d/mysqld), but it is used when we specify 'xyz', because there is no /etc/pam.d/xyz.

As we can see from the stack trace mysqld crashes in pam_ecryptfs.so:

#0 pam_sm_authenticate (pamh=0x7f5be4017cd0, flags=<optimized out>, argc=1, argv=0x7f5be4022e30) at pam_ecryptfs.c:142

Lets take a look at pam_sm_authenticate:

PAM_EXTERN int pam_sm_authenticate(pam_handle_t *pamh, int flags, int argc,
       const char **argv)
{
 uid_t uid = 0, oeuid = 0;
 long ngroups_max = sysconf(_SC_NGROUPS_MAX);
 gid_t gid = 0, oegid = 0, groups[ngroups_max+1];
 int ngids = 0;
 char *homedir = NULL;
 const char *username;
 char *passphrase = NULL;
 char salt[ECRYPTFS_SALT_SIZE];
 char salt_hex[ECRYPTFS_SALT_SIZE_HEX];
 char *auth_tok_sig = NULL;
 char *private_mnt = NULL;
 pid_t child_pid, tmp_pid;
 long rc;

 rc = pam_get_user(pamh, &username, NULL);

 ...

especially at lines

 long ngroups_max = sysconf(_SC_NGROUPS_MAX);
 gid_t gid = 0, oegid = 0, groups[ngroups_max+1];

sysconf(_SC_NGROUPS_MAX) is 65536, sizeof(gid_t) is 4.
So, 262144 bytes is allocated on stack.

Now

mysql> select @@thread_stack;

+----------------+
| @@thread_stack |
+----------------+
| 262144 |
+----------------+
1 row in set (0.01 sec)

Default mysql stack size in not enough to handle pam_ecryptfs. Workaround is to increase MySQL stack size by setting --thread-stack variable.

[Laurynas Biveinis (laurynas-biveinis)]

Sergei, should we convert this to a doc bug?

Also, might be worth posting your analysis to https://bugs.mysql.com/bug.php?id=74310

[Roel Van de Paar (roel11)]

Marked as doc bug

[Hrvoje Matijakovic (hrvojem)]

Fixed here: https://github.com/percona/percona-server/pull/352

[Hrvoje Matijakovic (hrvojem)]

Fixed here: https://github.com/percona/percona-server/pull/358
https://github.com/percona/percona-server/pull/359

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-949