I installed all as outlined in the bug and first was able to verify that the fix in proposed fixes the make_scrambled_password issue.

Lacking an ehcp setup I created a trvial DB for auth like this:

/etc/pam.d/vsftpd (please don't mind the nons ecure setup on my test env):
auth required pam_mysql.so user=root passwd="" host=localhost db=test table=ftpaccounts usercolumn=ftpusername passwdcolumn=password crypt=2
 account required pam_mysql.so user=root passwd="" host=localhost db=test table=ftpaccounts usercolumn=ftpusername passwdcolumn=password crypt=2

SQL via mysql:
create database test;
use test;
create table ftpaccounts (
    -> user_id int(6) not null auto_increment,
    -> password varchar(16) not null,
    -> primary key (user_id),
    -> key user_id (user_id) );
insert into ftpaccounts(ftpusername, password) values ('ubuntu', encrypt('ubuntu'));

First I was trying a wrong PW, to see what happens then:
Mar 20 14:08:59 xenial-test vsftpd[6447]: pam_mysql - MySQL error (Access denied for user 'ubuntu'@'localhost' (using password: YES))

But then with that in place use ftp to connect.
$ ftp 10.0.4.174
Connected to 10.0.4.174.
220 Welcome to vsFTPd Server
Name (10.0.4.174:paelzer): ubuntu
331 Please specify the password.
Password:
*** stack smashing detected ***: /usr/sbin/vsftpd terminated
Login failed.

Now this is reported on the client, but the main vsftp process is just fine.
Maybe it spawns a process on each login.

No report on the vsftpd host in journal at all.
I "normalized" myself step by step.

First I replaced the custom vsftpd.conf in the description with the default one from the package.
=> Same issue.

Resetting the /etc/pam.d/vsftpd made it working - at least that.

I also made a try keeping the content of the default pam, but then added the pam_mysql lines at the end.

Eventually I found that of the two lines the "auth" one is the killing one.
Is working ok:
account required pam_mysql.so user=root passwd="" host=localhost db=test table=ftpaccounts usercolumn=ftpusername passwdcolumn=password crypt=2
Triggers the bug:
auth required pam_mysql.so user=root passwd="" host=localhost db=test table=ftpaccounts usercolumn=ftpusername passwdcolumn=password crypt=2

Then I was installing vsftpd-dbg and attached gdb to it.
As I expected it to fork its childs per connection I set
(gdb) set follow-fork-mode child

It seems it spawns two childs, and the second one is the one dying badly (not the first).
Ok, then we try:
(gdb) set detach-on-fork off
With that on it is a bit of juggling processes but I finally reached this:

That still looks like a breakage in /lib/security/pam_mysql.so more than anything else.
Add "libpam-mysql-dbgsym" on top and look again:

#0  0x00007f31168d5428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f31168d702a in __GI_abort () at abort.c:89
#2  0x00007f31169177ea in __libc_message (do_abort=do_abort@entry=1, Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/libcrypt-2.23.so...done.
Reading symbols from /usr/lib/debug/.build-id/4d/7f52f335dc9665c2dcf308ce6514a6ae86dede.debug...done.
Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/librt-2.23.so...done.
Reading symbols from /usr/lib/debug/lib/security/pam_mysql.so...done.
Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/libm-2.23.so...done.
Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/libnss_compat-2.23.so...done.
Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/libnss_nis-2.23.so...done.
Reading symbols from /usr/lib/debug/lib/x86_64-linux-gnu/libnss_files-2.23.so...done.

    fmt=fmt@entry=0x7f3116a2e8a2 "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007f31169b856c in __GI___fortify_fail (msg=<optimized out>, msg@entry=0x7f3116a2e884 "stack smashing detected")
    at fortify_fail.c:37
#4  0x00007f31169b8510 in __stack_chk_fail () at stack_chk_fail.c:28
#5  0x00007f311473cea1 in pam_mysql_check_passwd (ctx=0x55cbdb28a600, user=<optimized out>, passwd=<optimized out>, 
    null_inhibited=null_inhibited@entry=1) at pam_mysql.c:2729
#6  0x00007f311473d22b in pam_sm_authenticate (pamh=0x55cbdb2685d0, flags=0, argc=<optimized out>, argv=0x55cbdb271260)
    at pam_mysql.c:3381
#7  0x00007f311751eea6 in ?? () from /lib/x86_64-linux-gnu/libpam.so.0
#8  0x00007f311751e61d in pam_authenticate () from /lib/x86_64-linux-gnu/libpam.so.0
#9  0x000055cbda07558c in vsf_sysdep_check_auth (p_user_str=p_user_str@entry=0x7ffe7a4f86c8, 
    p_pass_str=p_pass_str@entry=0x7ffe7a4f8610, p_remote_host=p_remote_host@entry=0x7ffe7a4f8790) at sysdeputil.c:387
#10 0x000055cbda06d366 in handle_local_login (p_pass_str=0x7ffe7a4f8610, p_user_str=0x7ffe7a4f86c8, p_sess=0x7ffe7a4f8670)
    at privops.c:387
#11 handle_login (p_pass_str=0x7ffe7a4f8610, p_user_str=0x7ffe7a4f86c8, p_sess=0x7ffe7a4f8670) at privops.c:338
#12 vsf_privop_do_login (p_sess=p_sess@entry=0x7ffe7a4f8670, p_pass_str=p_pass_str@entry=0x7ffe7a4f8610) at privops.c:257
#13 0x000055cbda06c733 in process_login_req (p_sess=0x7ffe7a4f8670) at twoprocess.c:318
#14 vsf_two_process_start (p_sess=0x7ffe7a4f8670) at twoprocess.c:112
#15 0x000055cbda061678 in main (argc=<optimized out>, argv=<optimized out>) at main.c:252

So it seems the function pam_mysql_check_passwd is the stack smashing one.
vsftp in that case is only a trigger.

I wanted to compare that to the newer 0.8 version, but the changes are so vast that one can hardly see individual changes.


With a file like that and sull sources for zesty I could selectively install the 0.8 version
cat /etc/apt/preferences
Package: libpam-mysql
Pin: release n=zesty
Pin-Priority: 990

Package: *
Pin: release n=xenial
Pin-Priority: 900

Package: *
Pin: release o=Ubuntu
Pin-Priority: -10


With the 0.8 in place it does no more break with stack smashing (but instead tells me my login is incorrect - which I think is wrong, but I already have a severely modified setup, so might be unimportant).


TL;DR:
- repro as above
- bug in libpam_mysql
- stack smashing function pam_mysql_check_passwd