PAM Plugin Fails on CentOS 5

[Jervin R]

As documentation stands, the PAM configuration /etc/pam.d/mysqld is not sufficient to authenticate properly. An internal failure with pam_unix.so will prevent a successful login even when authentication is a success.

http://www.percona.com/doc/percona-pam-for-mysql/manual.html#configuring-pam-for-mysql

[root@sbxc05 ~]# man pam_succeed
No manual entry for pam_succeed
[root@sbxc05 ~]# cat /etc/redhat-release
CentOS release 5.10 (Final)
[root@sbxc05 ~]# mysql -e 'select @@version, @@version_comment'
+-------------+-----------------------------------------------------+
| @@version | @@version_comment |
+-------------+-----------------------------------------------------+
| 5.5.35-33.0 | Percona Server (GPL), Release rel33.0, Revision 611 |
+-------------+-----------------------------------------------------+

How to repeat: simply follow http://www.mysqlperformanceblog.com/2013/08/14/getting-percona-pam-to-work-with-percona-server-its-client-apps/ using the versions above.

Workaround:

Add pam_permit.so in the PAM configuration:

auth required pam_warn.so
auth required pam_unix.so
account required pam_unix.so
account required pam_permit.so

pam_permit.so seems to simply ignore the internal failure, password verification still works as expected.

[root@sbxc05 ~]# mysql -uap_user -pwrongpass
ERROR 1045 (28000): Access denied for user 'ap_user'@'localhost' (using password: YES)
[root@sbxc05 ~]# mysql -uap_user -pp3rc0na
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 31
Server version: 5.5.35-33.0 Percona Server (GPL), Release rel33.0, Revision 611

Copyright (c) 2009-2013 Percona LLC and/or its affiliates
Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

On the other hand, as the man page of pam_permit.so using this module is dangerous, hence I am filing this bug to potentially document the workaround and fix the internal failure.

[Jervin R (revin)]

An strace output of bad PAM authentication in CentOS 5, more specifically it fails on:

readlink("/proc/self/exe", 0x2af628896e70, 4096) = -1 EACCES (Permission denied)

[Jervin R (revin)]

Good PAM authentication strace on CentOS 6

[Laurynas Biveinis (laurynas-biveinis)]

I'd look first at why reading /proc/self/exe failed. Patching the PAM config to hide this failure does not feel right.

[Raghavendra D Prabhu (raghavendra-prabhu)]

>> readlink("/proc/self/exe", 0x2af628896e70, 4096) = -1 EACCES (Permission denied)

Selinux or setuid/sgid? Either of those are usual culprits here.

[Jervin R (revin)]

Raghu,

SELINUX is disabled. If you look at the logs I attached, a good authentication procedure was able to do seteuid/setuid before the PAM checks - the bad one does not.

[Nilnandan Joshi (nilnandan-joshi)]

Confirmed with CentOS 5.9.

If we can add "account required pam_permit.so" in /etc/pam.d/mysqld then it will work otherwise it will not allow to login with linux password.

Without pam_permit.so,

[root@localhost ~]# mysql -u ap_user -p
Enter password:
ERROR 1045 (28000): Access denied for user 'ap_user'@'localhost' (using password: YES)

With pam_permit.so in /etc/pam.d/mysqld

[root@localhost ~]# mysql -u ap_user -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 5.5.36-34.2 Percona Server (GPL), Release rel34.2, Revision 648

Copyright (c) 2009-2014 Percona LLC and/or its affiliates
Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

[Laurynas Biveinis (laurynas-biveinis)]

I don't see why this bug has been set to "Confirmed." It still looks like a system configuration issue, and a Wishlist documentation bug at best.

Nil, can you post the complete /etc/pam.d/mysqld files you used?

Some superficial googling turned up some advice that, if I'm not misunderstanding it, all auth/account stacks must be finished by pam_permit/pam_deny in the configs.

[Nilnandan Joshi (nilnandan-joshi)]

Hi Laurynas,

On CentOS 6/Ubuntu 13.04, it is working fine with below settings in /etc/pam.d/mysqld
as per http://www.mysqlperformanceblog.com/2013/08/14/getting-percona-pam-to-work-with-percona-server-its-client-apps/

auth required pam_warn.so
auth required pam_unix.so audit
account required pam_unix.so audit

But on CentOS 5.9, its not working with same settings. To make it work we have to add, pam_permit.so

auth required pam_warn.so
auth required pam_unix.so audit
account required pam_unix.so audit
account required pam_permit.so

So I'm not sure if it's problem with Docs (that we need add pam_permit.so for CentOS 5.X) or it's bug.

[Laurynas Biveinis (laurynas-biveinis)]

Thus it seems that further work is required to verify this bug.

[Valerii Kravchuk (valerii-kravchuk)]

What kind of work? There is a clear repeatable test case on CentOS 5 that shows that software does not work as PAM manual described, while it works this way on CentOS 6. So, one only has to decide what to fix - PAM manual, to take CentOS 5 case into account, or Percona Server, to work on CentOS 5 the way PAM manual describes. What exactly non-developer can further verify here?

[Laurynas Biveinis (laurynas-biveinis)]

Valerii -

I had in mind that it is not clear what kind of bug is this: whether it is a system misconfiguration issue on CentOS 5, in which case it is a doc bug, or a PAM plugin code bug that manifests on CentOS 5 only. Does answering this question - by verifying whether the PAM config used is really correct, which can be performed by a non-developer - fall outside the scope of bug verification?

[Valerii Kravchuk (valerii-kravchuk)]

Well, it's stock CentOS 5 vs stock CentOS 6, they are both the same, and the save version of our software works on 6 but not on 5, with the same configuration. Both bug reported and Nil just did it all (PAM configuration steps) by the manual.

If any special configuration is needed for CentOS 5, our manual for PAM should say this explicitly. It was already proved that one has to add:

account required pam_permit.so

on CentOS 5 to make PAM work. Proved and double checked that this is not needed on CentOS 6 or other Linux. So, we clearly have a problem to fix in http://www.percona.com/doc/percona-pam-for-mysql/manual.html#configuring-pam-for-mysql that says nothing about pam_permit.so or CentOS 5 specifically. Current state must be properly documented, even if eventually PAM is going to be fixed to make this extra module not needed.

I am not sure how to prove there is internal failure to fix (strace, something else). Specific instructions would be really helpful.

[Sergei Glushchenko (sergei.glushchenko)]

This is definitely not a PAM plugin bug. It is likely not a documentation issue.

The key is EACCESS reading /proc/self/exe. Failed setuid/setgid can be a reason.

1. Please attach full strace of mysqld. Comment #5 states that there are setuid failures however they don't present in attached logs.
2. Make sure that mysqld operates with user/group which have access to mysqld binary

[Launchpad Janitor (janitor)]

[Expired for Percona Server 5.5 because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for Percona Server 5.6 because there has been no activity for 60 days.]

[Sergei Glushchenko (sergei.glushchenko)]

From my experience PAM plugin works great on Cent OS 5 when Percona Server started with root privileges.

[Jervin R (revin)]

Sergei,

I think that is the problem, PAM plugin should work without the need to start MYSQL as root :-). At least this should be doc bug based on comment #8, if it is *expected* to work without pam_permit.so, that takes as back to why the plugin does not work with CentOS 5 and we already have a repeatable test case right?

[Nilnandan Joshi (nilnandan-joshi)]

Please find attached strace output while testing with and without pam_permit.so on CentOS 5
You can see

readlink("/proc/self/exe", 0x2b3915612cc0, 4096) = -1 EACCES (Permission denied)

and

sendto(29, "K\0\0\4\377\25\4#28000Access denied for u"..., 79, MSG_DONTWAIT, NULL, 0) = 79
shutdown(29, 2 /* send and receive */) = 0
close(29) = 0
futex(0x12f6d64, FUTEX_WAIT_PRIVATE, 29, NULL

[Laurynas Biveinis (laurynas-biveinis)]

Sergei, can you look at the strace log?

[Sergei Glushchenko (sergei.glushchenko)]

I've done little debugging and found that bug doesn't related to
failed readlink. It turned out to be just pam_unix.so is broken on
Centos 5 for non-root caller. If one is using the trick with adding
mysql user to shadow group or changing /etc/shadow's owner group to
mysql, then mysqld can access /etc/shadow and authenticate user
against it. That is how 'auth' module of pam_unix.so does. It gets
password hash from /etc/shadow and compares it with password user
trying to login with. Then 'account' module of auth_pam.so should
check whether the user is valid. Validation includes checks that
user is existent and it's password is not blank, and it's account
isn't expired and so on. It should be done in our case, again,
against /etc/shadow.

The trouble here is that pam_unix from CentOS 5 is patched by RedHat
such a way that 'account' module first verifies caller's (mysqld)
process UID (mysql in our case) and if it is not 0 (not root) fires
PAM_INGNORE.

"Ignore underlying account module regardless of whether the control
flag is required, optional, or sufficient" -- that is how PAM_IGNORE
described.

So, PAM falls back to pam_deny.so for 'account'. Now adding
'pam_permit.so' makes user to be able to bypass 'account' without
any checks.

To summarize.

1. We have nothing to fix in Percona PAM plugin.
2. pam_unix.so in CentOS 5 is broken.
3. Adding pam_permit.so makes user to bypass account checks, which
   is security concern.

[Laurynas Biveinis (laurynas-biveinis)]

Thanks Sergei.

Assigning to Hrvoje to document this.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3089