vsftpd hangs with SIGCHLD when pam_exec.so is used

[Markus Wigge]

[ Impact ]

 * User impact: under certain conditions using pam_exec the vsftp server
   just hangs

 * Reason: when running sub-processes on login through pam_exec a process
   is spawned. That can confuse vsftp if that child ends triggering SIGCHLD but
   already been picked up by e.g. pam_exec.so itself.

 * Fix: The fix uses waitpid over wait to be able to pass options. With that
   it sets WNOHANG in vsf_sysutil_wait except if it is explicitly called
   to wait as done in common_do_login for the pre-login child.
   Therefore these other calls now allow it to "return immediately if
   no child has exited" as defined for WNOHANG in [1]

[1]: https://manpages.ubuntu.com/manpages/noble/en/man2/wait.2.html

[ Test Plan ]

# install
$ apt install lftp vsftpd
# change config
$ sed -i.old '1 i\account optional pam_exec.so debug quiet /root/foo.sh\' /etc/pam.d/vsftpd
# script to run
$ cat > /root/foo.sh << EOF
#!/bin/bash
/bin/true
touch /tmp/brooks-was-here
/bin/true
EOF
$ chmod +x /root/foo.sh
# enable ssl
$ sed -i -s -e 's/ssl_enable=NO/ssl_enable=YES/' /etc/vsftpd.conf
$ systemctl restart vsftpd.service
# Place a file there
$ echo foobar > /home/ubuntu/egal
# set test PW to ubuntu user
echo 'ubuntu:ubuntu' | chpasswd

# Using it with ftps (and ignore cert verification as it is the snakeoil cert)

To verify the test config, if you run this in a second console you should see it calling the script as yo uact on the server.
$ tail -f /var/log/auth.log
...
2024-07-16T07:30:37.966553+00:00 o vsftpd: pam_exec(vsftpd:account): Calling /root/foo.sh ...

Good case (Noble / Oracular):

root@n:~# lftp 127.0.0.1
lftp 127.0.0.1:~> set ftp:ssl-force true
lftp 127.0.0.1:~> set ssl:verify-certificate false
lftp 127.0.0.1:~> login ubuntu ubuntu
lftp ubuntu@127.0.0.1:~> dir
-rw-r--r-- 1 0 0 7 Jul 16 07:30 egal
lftp ubuntu@127.0.0.1:~> get egal
7 bytes transferred
lftp ubuntu@127.0.0.1:~>
exit
root@n:~# cat egal
foobar

Bad case (Focal and Jammy)
root@j:~# lftp 127.0.0.1
lftp 127.0.0.1:~> set ftp:ssl-force true
lftp 127.0.0.1:~> set ssl:verify-certificate false
lftp 127.0.0.1:~> login ubuntu ubuntu
lftp ubuntu@127.0.0.1:~> dir
`ls' at 0 [Sending commands...]

[ Where problems could occur ]

* This changes signal handling for SIGCHLD.
  The code now returns cleanly if there was nobody to wait for, which formerly
  would have caused a the main process to die "Child died, so we'll do the same"
  That is intentionally changed for the condition of the child already being
  consumed.
  If there is a use case of the child leaving which was meant to terminate
  (unlikely, this is an unclean die call) it might no more happen now.

[ Other Info ]

* The code is the same (only no change rebuilds) still, this does not occur in
  Noble and Oracular. At least not with the current test setup. That is slightly
  disturbing.
  Also in the reproduction we've seen that it only occurred with FTPS, but that
  is not conceptually tied to the problem, it might only be yet another detail
  that changes the timing and size of the signal race window.
  Of course we can assume that it is just a race and the window is
  different there, but then should we not fix it? Or we can assume something
  else e.g. pam_exec has changed behavior to mask the issue and hence no vsftpd
  change is needed there. I think it is wasted to research this for ages, but
  it leaves some uncertainty.

---

When you try to run a script with pam_exec.so on login vsftpd freezes with SIGCHLD.

This was fixed in 2015 by redhat and never adopted to Debian/Ubunutu.

See also:
- https://bugzilla.redhat.com/show_bug.cgi?id=1198259
- https://git.centos.org/rpms/vsftpd/blob/54ac5fac29fcc1bb68f2e96e63ecfda655286ff8/f/SOURCES/0026-Prevent-hanging-in-SIGCHLD-handler.patch

[dapospis (dapospis-redhat-bugs)]

The issue seems to appear also on RHEL7.
+++ This bug was initially created as a clone of Bug #1092877 +++

Description of problem:
The vsftpd hangs when the pam_exec.so is added to the /etc/pam.d/vsftpd.

---------------------------------------------
...
...
session include password-auth
session optional pam_exec.so /bin/echo
---------------------------------------------

Version-Release number of selected component (if applicable):
vsftpd-3.0.2-9.el7

How reproducible:
always

Steps to Reproduce:
1. add pam_exec.so calling echo to /etc/pam.d/vsftpd
2. add session_support=YES to /etc/vsftpd/vsftpd.conf
3. restart vsftpd
4. connect with a client to the vsftpd
5. login and send e.g. ls command

Actual results:
The vsftpd hangs in SIGCHLD handler

Expected results:
The vsftpd will work as intended.

[errata-xmlrpc (errata-xmlrpc-redhat-bugs)]

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2015-2446.html

[Mitchell Dzurick (mitchdz)]

Thank you for taking the time to make this bug report Markus!

[Mitchell Dzurick (mitchdz)]

What version of Ubuntu are you seeing this bug on? Unfortunately upstream/debian has been pretty dormant for a while, in-fact we are ahead of debian right now, so it may make sense to apply this patch.

Unfortunately it looks like even upstream does not have this patch applied.

[Mitchell Dzurick (mitchdz)]

Markus, if I were to create a PPA with this patch applied, would you be willing to help test that out?

[Markus Wigge (markus-cultcom)]

Hello,

actually I am running "bookworm" and noticed that the packet is quite
abandoned so I looked at the Ubuntu repository which had a newer release
of vsftpd but also quite old.

I build the ubuntu package on debian bookworm an noticed the same error.
Then I rebuilt the same package with the attached patch applied and now
it is working as expected. My custom script is run by pam_exec.so on
login as it was intended.

It would be great to see it applied upstream in Ubunutu and Debian as well.

Just add a line to /etc/pam.d/vsftpd like this:
account optional pam_exec.so debug quiet /root/my_script.sh

And simply run "logger test" inside.

Without the patch the vsftpd hangs as soon as someone logs in.

Kind Regards,
Markus

Am 15.06.24 um 02:50 schrieb Mitchell Dzurick:
> Markus, if I were to create a PPA with this patch applied, would you be
> willing to help test that out?
>
> ** Changed in: vsftpd (Ubuntu)
> Status: New => Triaged
>

[Mitchell Dzurick (mitchdz)]

Markus, would you mind making a bug report upstream to see if this could be included?

[Markus Wigge (markus-cultcom)]

Well, there is a bug report from 2020 about the problem:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=952421

Am 18.06.24 um 21:01 schrieb Mitchell Dzurick:
> Markus, would you mind making a bug report upstream to see if this could
> be included?
>

[Paride Legovini (paride)]

Looks like upstream development is *very* slow, and I could not find an issue tracker or a public vcs repo. Given that we have Ubuntu delta already (we're already ahead of Debian, which is still packaging 3.0.3), I believe it makes sense to add the patch to the Ubuntu package.

[Mitchell Dzurick (mitchdz)]

Markus, if we prepare a package with the patch, would you be willing to help out testing the patch?

[Markus Wigge (markus-cultcom)]

When it is needed I can try to install it on my server.

Am 21.06.24 um 07:33 schrieb Mitchell Dzurick:
> Markus, if we prepare a package with the patch, would you be willing to
> help out testing the patch?
>

[Mitchell Dzurick (mitchdz)]

Thanks Markus, I'll make a PPA for that right now.

[Mitchell Dzurick (mitchdz)]

Just kicked off a build at [0].

[0] - https://launchpad.net/~mitchdz/+archive/ubuntu/lp2069324-vsftpd-sigchld

[Bryce Harrington (bryce)]

Hi Markus - looks like Mitchell's PPA build succeeded. Have you had a chance to install it on your server and test that it isn't showing the crash?

[Markus Wigge (markus-cultcom)]

Sorry, I have no oracular to install the package. Only debian bookworm
where it is not installable due to missing dependencies.
I only uses the ubuntu sources to build my own patched version for debian.

Am 03.07.24 um 06:14 schrieb Bryce Harrington:
> Hi Markus - looks like Mitchell's PPA build succeeded. Have you had a
> chance to install it on your server and test that it isn't showing the
> crash?
>

[Mitchell Dzurick (mitchdz)]

Thank you for your time in attempting to install the package Markus.

I agree with Paride in https://bugs.launchpad.net/ubuntu/+source/vsftpd/+bug/2069324/comments/8, we should go ahead and implement this fix on our side.

I'll attempt a quick repro tomorrow, but I'm pretty booked so if it takes substantial time then I'll defer the reproduction and validation for someone else. If repro is easy, I'll go ahead and stage the change back to Focal.

[Markus Wigge (markus-cultcom)]

Hello,

reproduction is quite simple. Just add a pam_exec.so to PAM-config to
run a small script.
"exit 0" might do it.

Or even simpler:
account optional pam_exec.so debug quiet /bin/true

Without the Patch vsftp hangs after a successful login. With the patch
it continues running.

Regards,
Markus

Am 10.07.24 um 06:10 schrieb Mitchell Dzurick:
> Thank you for your time in attempting to install the package Markus.
>
> I agree with Paride in
> https://bugs.launchpad.net/ubuntu/+source/vsftpd/+bug/2069324/comments/8,
> we should go ahead and implement this fix on our side.
>
> I'll attempt a quick repro tomorrow, but I'm pretty booked so if it
> takes substantial time then I'll defer the reproduction and validation
> for someone else. If repro is easy, I'll go ahead and stage the change
> back to Focal.
>

[Christian Ehrhardt  (paelzer)]

Hi,
first of all thanks for all the prep work.
The patch LGTM and I agree we should add it.
I was happy to see the suggestions on how to reproduce, but while writing it out in more detail could not reproduce the hang. The following are the steps I drafted to an SRU templates "how to test and verify" steps. But the hang does not occur. Could you have a look where my steps diverge from what you had in mind?

# Install ftp server
$ sudo apt install vsftpd

# set a user with password, I use ubuntu:ubuntu in the examples here
$ passwd ubuntu

# add testfile to the users home dir
$ echo foobar > /home/ubuntu/egal

# Check with FTP if the normal setup works
root@o:~# ftp 127.0.0.1
Connected to 127.0.0.1.
220 (vsFTPd 3.0.5)
Name (127.0.0.1:root): ubuntu
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> dir
229 Entering Extended Passive Mode (|||47570|)
150 Here comes the directory listing.
-rw-r--r-- 1 0 0 7 Jul 12 06:59 egal
226 Directory send OK.
ftp> get egal
local: egal remote: egal
229 Entering Extended Passive Mode (|||21047|)
150 Opening BINARY mode data connection for egal (7 bytes).
100% |**************************************************************************************************************************************************| 7 45.87 KiB/s 00:00 ETA
226 Transfer complete.
7 bytes received in 00:00 (8.49 KiB/s)
ftp> ^D
221 Goodbye.
root@o:~# cat egal
foobar
root@o:~# rm egal

# Now set up the failure condition
#1 add a few exec commands on pam
$ echo "account optional pam_exec.so debug quiet /bin/true" | sudo tee -a /etc/pam.d/vsftpd

# Restart server to be sure to pick up everything
$ sudo systemctl restart vsftpd.service

# Log in again to see the hang
$ ftp 127.0.0.1

^^ but this does not fall into the hang, I can get the file and quit normally.

It is not that the setup is totally useless, in /var/log/auth.log I see on the login
2024-07-12T07:27:01.030634+00:00 o vsftpd: pam_exec(vsftpd:account): Calling /bin/true ...

Do you have an advice what you do different to reproduce the issue?

P.S. along evaluating this I found many more changes we should apply to our ftp servers. Not all suitable for SRUs, but going forward I want to still improve them. To find the required time for that effort I internally filed SD-1755 to not fall through the cracks forever.

[Markus Wigge (markus-cultcom)]

Hello,

I acutally run a shell script /root/foo.sh which runs several commands.
Maybe that is a difference.
And I only use FTPS to connect.

Regards,
Markus

Am 12.07.24 um 09:33 schrieb Christian Ehrhardt :
> Hi,
> first of all thanks for all the prep work.
> The patch LGTM and I agree we should add it.
> I was happy to see the suggestions on how to reproduce, but while writing it out in more detail could not reproduce the hang. The following are the steps I drafted to an SRU templates "how to test and verify" steps. But the hang does not occur. Could you have a look where my steps diverge from what you had in mind?
>
>
> # Install ftp server
> $ sudo apt install vsftpd
>
> # set a user with password, I use ubuntu:ubuntu in the examples here
> $ passwd ubuntu
>
> # add testfile to the users home dir
> $ echo foobar > /home/ubuntu/egal
>
> # Check with FTP if the normal setup works
> root@o:~# ftp 127.0.0.1
> Connected to 127.0.0.1.
> 220 (vsFTPd 3.0.5)
> Name (127.0.0.1:root): ubuntu
> 331 Please specify the password.
> Password:
> 230 Login successful.
> Remote system type is UNIX.
> Using binary mode to transfer files.
> ftp> dir
> 229 Entering Extended Passive Mode (|||47570|)
> 150 Here comes the directory listing.
> -rw-r--r-- 1 0 0 7 Jul 12 06:59 egal
> 226 Directory send OK.
> ftp> get egal
> local: egal remote: egal
> 229 Entering Extended Passive Mode (|||21047|)
> 150 Opening BINARY mode data connection for egal (7 bytes).
> 100% |**************************************************************************************************************************************************| 7 45.87 KiB/s 00:00 ETA
> 226 Transfer complete.
> 7 bytes received in 00:00 (8.49 KiB/s)
> ftp> ^D
> 221 Goodbye.
> root@o:~# cat egal
> foobar
> root@o:~# rm egal
>
>
> # Now set up the failure condition
> #1 add a few exec commands on pam
> $ echo "account optional pam_exec.so debug quiet /bin/true" | sudo tee -a /etc/pam.d/vsftpd
>
> # Restart server to be sure to pick up everything
> $ sudo systemctl restart vsftpd.service
>
> # Log in again to see the hang
> $ ftp 127.0.0.1
>
> ^^ but this does not fall into the hang, I can get the file and quit
> normally.
>
> It is not that the setup is totally useless, in /var/log/auth.log I see on the login
> 2024-07-12T07:27:01.030634+00:00 o vsftpd: pam_exec(vsftpd:account): Calling /bin/true ...
>
>
> Do you have an advice what you do different to reproduce the issue?
>
>
> P.S. along evaluating this I found many more changes we should apply to our ftp servers. Not all suitable for SRUs, but going forward I want to still improve them. To find the required time for that effort I internally filed SD-1755 to not fall through the cracks forever.
>
> ** Changed in: vsftpd (Ubuntu)
> Status: Triaged => Incomplete
>

[Christian Ehrhardt  (paelzer)]

Thanks Markus

#1 trying with this now:
in /etc/pam.d/vsftpd:
account optional pam_exec.so debug quiet /root/foo.sh

And
$ chmod +x /root/foo.sh
$ cat /root/foo.sh
#!/bin/bash
/bin/true
touch /tmp/brooks-was-here
/bin/true

By default vsftp is configured for the snakeoil certificate to get started easily, but with ssl disabled. So enable it and restart.

$ sed -i -s -e 's/ssl_enable=NO/ssl_enable=YES/' /etc/vsftpd.conf
$ sudo systemctl restart vsftpd.service

# Using it with ftps (and ignore cert verification as it is the snakeoil cert)

root@o:~# lftp 127.0.0.1
lftp 127.0.0.1:~> set ftp:ssl-force true
lftp 127.0.0.1:~> set ssl:verify-certificate false
lftp 127.0.0.1:~> login ubuntu ubuntu
lftp ubuntu@127.0.0.1:~> dir
-rw-r--r-- 1 0 0 7 Jul 12 06:59 egal
lftp ubuntu@127.0.0.1:~> get egal
7 bytes transferred
lftp ubuntu@127.0.0.1:~>
exit
root@o:~# cat egal
foobar
root@o:~# ll /tmp/brooks-was-here
-rw------- 1 root root 0 Jul 15 12:04 /tmp/brooks-was-here
root@o:~# date
Mon Jul 15 12:04:32 UTC 2024

I checked ftps was enabled e.g. a dir without it was not working:
  530 Non-anonymous sessions must use encryption.

Sadly that means it still works fine.
We still need more from you about what is needed to trigger this hang :-/

[Christian Ehrhardt  (paelzer)]

I was double checking this on other releases and got the hang on focal.
I could indeed only trigger this with FTPS on focal, but AFAICS the same config as on Oracular.
That motivated me, so I tried fresh F,J,N,O systems which in theory should all be the same but obviously behave different.

And I found it is not affecting Noble and higher.
That is weird in itself, as there the only difference is that they got two no change rebuilds (for a CVE and for libssl3t64)

I was updating the steps to recreate as I used them for this.
For >=Noble this either really is no issue due to yet unknown reasons, or on >Noble we need adapted test steps that would trigger it there.

# install
$ apt install lftp vsftpd
# change config
$ sed -i.old '1 i\account optional pam_exec.so debug quiet /root/foo.sh\' /etc/pam.d/vsftpd
# script to run
$ cat > /root/foo.sh << EOF
#!/bin/bash
/bin/true
touch /tmp/brooks-was-here
/bin/true
EOF
$ chmod +x /root/foo.sh
# enable ssl
$ sed -i -s -e 's/ssl_enable=NO/ssl_enable=YES/' /etc/vsftpd.conf
$ systemctl restart vsftpd.service
# Place a file there
$ echo foobar > /home/ubuntu/egal
# set test PW to ubuntu user
echo 'ubuntu:ubuntu' | chpasswd

# Using it with ftps (and ignore cert verification as it is the snakeoil cert)

To verify the test config, if you run this in a second console you should see it calling the script as yo uact on the server.
$ tail -f /var/log/auth.log
...
2024-07-16T07:30:37.966553+00:00 o vsftpd: pam_exec(vsftpd:account): Calling /root/foo.sh ...

Good case (Noble / Oracular):

root@n:~# lftp 127.0.0.1
lftp 127.0.0.1:~> set ftp:ssl-force true
lftp 127.0.0.1:~> set ssl:verify-certificate false
lftp 127.0.0.1:~> login ubuntu ubuntu
lftp ubuntu@127.0.0.1:~> dir
-rw-r--r-- 1 0 0 7 Jul 16 07:30 egal
lftp ubuntu@127.0.0.1:~> get egal
7 bytes transferred
lftp ubuntu@127.0.0.1:~>
exit
root@n:~# cat egal
foobar

Bad case (Focal and Jammy)
root@j:~# lftp 127.0.0.1
lftp 127.0.0.1:~> set ftp:ssl-force true
lftp 127.0.0.1:~> set ssl:verify-certificate false
lftp 127.0.0.1:~> login ubuntu ubuntu
lftp ubuntu@127.0.0.1:~> dir
`ls' at 0 [Sending commands...]

^^ here it hangs

[Christian Ehrhardt  (paelzer)]

MRs approved, uploaded for Focal and Jammy.

Now waiting for the SRU team to find time for reviewing it in the -unapproved queue.