Comment 27 for bug 1574900

TL;DR
- pam_mysql.c buf in pam_mysql_check_passwd() is overflowing
- my_make_scrambled_password() is NOT returning content that can be compared to what is stored in the mysql DB when using PASSWORD().
- my_make_scrambled_password_sha1() seems to be the right one to use, as it returns a string of hex values, but it's not exported

Details:

It's buf that is overflowing in pam_mysql.c:
/* PASSWORD */
case 2: {
        char buf[42];
...
        my_make_scrambled_password(buf, passwd, strlen(passwd));
        syslog(LOG_AUTHPRIV | LOG_ERR, PAM_MYSQL_LOG_PREFIX "andreas: row0=%s buf=%s passwd=%s", row[0], buf, passwd);

I added some simple debugging above, and got;
May 11 22:08:42 yakkety-pam-mysql vsftpd: pam_mysql - andreas: row0=*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 buf=$5$EI#015W7T^j'x#00173VPb#016g#025J$8cyy7LNV0Uhg0RCY1OV0OcrbqCB7eaYBsRmWnBJmzT2 passwd=password

row0 matches what's in the DB:
mysql> SELECT pass FROM accounts WHERE username = 'user';
+-------------------------------------------+
| pass |
+-------------------------------------------+
| *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 |

You can see that buf contains something in a very different format than what is stored in mysql. It's definitely not in hex format, it's much larger than 42 and I'm not even sure if it's supposed to be \0 terminated. I haven't found docs for my_make_scrambled_password() yet.

That magic number 42 comes from mysql's sql/auth/password.c for a *different* function:
/*
    MySQL 4.1.1 password hashing: SHA conversion (see RFC 2289, 3174) twice
    applied to the password string, and then produced octet sequence is
    converted to hex string.
    The result of this function is used as return value from PASSWORD() and
    is stored in the database.
  SYNOPSIS
    my_make_scrambled_password_sha1()
    buf OUT buffer of size 2*SHA1_HASH_SIZE + 2 to store hex string
    password IN password string
    pass_len IN length of password string
*/

Where:
./include/sha1.h:#define SHA1_HASH_SIZE 20 /* Hash size in bytes */

so buf has size 2*20+2 = 42. (what about null termination?)

BUT that Synopsis is for my_make_scrambled_password_sha1(), *NOT* my_make_scrambled_password().

my_make_scrambled_password() seems to be something very different:
void my_make_scrambled_password(char *to, const char *password,
                                size_t pass_len)
{

  char salt[CRYPT_SALT_LENGTH + 1];

  generate_user_salt(salt, CRYPT_SALT_LENGTH + 1);
  my_crypt_genhash(to,
                     CRYPT_MAX_PASSWORD_SIZE,
                     password,
                     pass_len,
                     salt,
                     0);

}

CRYPT_MAX_PASSWORD_SIZE is much bigger than 42, and in any case the result is not a string of hex values.

Further digging needed, but I'm EOD now :)