RLimitCPU has no effect in Apache

[Neil Van Dyke]

Binary package hint: apache2

The Apache "RLimitCPU" directive has no effect on in the Ubuntu packaging of Apache 2.2.8. We have reproduced this problem on multiple Ubuntu 8.04 systems, including a freshly-installed one.

We have verified that it *does* work on the same machine when using an unmodified upstream source build of 2.2.8. We have also verified that it works on Debian "stable" (using Debian packaging of Apache 2.2.9).

This arguably constitutes a DoS security vulnerabilitys, since the Ubuntu packaging of Apache is not preventing a runaway process from taking down the server as a correctly operating Apache (including upstream) does.

The cause appears to be in either Ubuntu-specific (or Debian-specific) patches to 2.2.8 in the Ubuntu/Debian-specific configuration setup in Ubuntu packaging of 2.2.8.

If the problem can be fixed in the Ubuntu packaging of Apache as an update to 8.04, so that we could use it on our server, that would be great. Otherwise, we will have to move to a build of upstream Apache or move away from Ubuntu.

Thank you.

Description: Ubuntu 8.04.2
Release: 8.04

ii apache2 2.2.8-1ubuntu0.9 Next generation, scalable, extendable web server
ii apache2-mpm-worker 2.2.8-1ubuntu0.9 High speed threaded model for Apache HTTPD
ii apache2-utils 2.2.8-1ubuntu0.9 utility programs for webservers
ii apache2.2-common 2.2.8-1ubuntu0.9 Next generation, scalable, extendable web server

[Chuck Short (zulcss)]

Do you have a script or a cgi that tests this bug?

Thanks
chuck

[Neil Van Dyke (neil-neilvandyke)]

Chuck Short wrote at 07/02/2009 09:51 AM:
> Do you have a script or a cgi that tests this bug?
>

The following "cgi-bin" script can be used to trigger "RLimitCPU" in a
correctly functioning Apache. Setting the limits to 2 seconds of CPU
time typically permits around 10 seconds of real time to watch the time
grow in "top". On a correctly functioning Apache, the process
terminates shortly after 2 seconds of CPU time are incurred. Thanks.

#!/bin/sh
echo "Content-Type: text/html"
echo ""
echo "BEFORE"
while true ; do
    expr 1 + 1 > /dev/null
done
echo "AFTER"

[Neil Van Dyke (neil-neilvandyke)]

I just verified that Ubuntu's 9.04's packaging of Apache 2.2.11 also exhibits this problem.

Looking through the Ubuntu patches to upstream Apache 2.2.8 (where we initially noticed the problem), I haven't yet found an obvious cause.

The people who did the packaging or patches on this would be able to debug this faster than me. Otherwise, I'll have to start tracing through unfamiliar Apache source to debug it that way.

[Neil Van Dyke (neil-neilvandyke)]

Why is the status of this still Incomplete?

I realize that the holiday weekend here in the US probably interrupted work, but it's now almost a week idling on what appears to be an Ubuntu-specific security/stability problem for Apache servers.

I really need to know if Ubuntu has an imminent solution.

I can't tell whether anyone of the dozens of people copied on this bug has tried to reproduce the problem yet.

I'm afraid this problem and then lack of response is seriously damaging my users' confidence in Ubuntu Server, after they recently switched to it.

Thank you.

[Kees Cook (kees)]

this is Ubuntu-specific?

[Kees Cook (kees)]

What are the specific configurations you're using so that developers can set up a test to reproduce what you're seeing?

[Kees Cook (kees)]

Based on http://httpd.apache.org/docs/2.0/mod/core.html#rlimitcpu I set a 2 second soft and hard limit on the default website /etc/apache2/sites-enabled/000-default:
<VirtualHost *:80>
 ServerAdmin webmaster@localhost
 RLimitCPU 2 2
...

I put the example script above into /usr/lib/cgi-bin/test.cgi, and ran it. It was correctly killed, observed via strace:
...
[pid 10848] write(1, "2\n", 2) = 2
[pid 10848] close(1) = 0
[pid 10848] munmap(0x7f4f160ed000, 4096) = 0
[pid 10848] close(2) = 0
[pid 10848] exit_group(0) = ?
Process 10848 detached
<... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 10848
--- SIGCHLD (Child exited) @ 0 (0) ---
+++ killed by SIGKILL +++
Process 5234 detached

[Kees Cook (kees)]

As an added note, you can examine a process's rlimits via /proc/$pid/limits

[Neil Van Dyke (neil-neilvandyke)]

* Simply take a fresh Ubuntu 8.04 install (which gets Apache 2.2.8), add "RLimitCPU 2 2" to the "default" Apache site file, drop the script above into the "cgi-bin" dir, and run the CGI. You'll see that the CGI process is *not* killed.

* Do the same thing on a Debian "stable" system (which gets Apache 2.2.9), and the CGI process *is* killed.

* Rig up a build of upstream Apache 2.2.8 (no Ubuntu patches) to use esssentially the same config file tree as a fresh Ubuntu 8.04 install (and running on the same Ubuntu 8.04 installed system), and the CGI process *is* killed.

At this point, unless I made a mistake while troubleshooting, the problem appears to be in Ubuntu-specific patches to Apache.

[Neil Van Dyke (neil-neilvandyke)]

(I had not seen Kees Cook's failure to reproduce before I posted my last message.)

I have just reproduced the problem with a fresh install of 9.04 on an X86 box. (I do not have a fresh 8.04 install at the moment.)

I am at a loss to explain why Kees Cook could not reproduce the problem. I wonder whether he was using a fresh install of an LTS version, or some other configuration.

I request that this bug be reopened.

[Andrew Mitchell (ajmitch)]

I've tested this on hardy (i386) with all updates installed. The only difference I can see at a glance is i386 vs amd64 with regards to testing this.
I've tested with the same method as Kees, in that /etc/apache2/sites-enabled/000-default has:
<VirtualHost *>
        ServerAdmin webmaster@localhost
        RLimitCPU 2 2
...

The apache2 process was restarted with /etc/init.d/apache2 restart, with the same CGI script installed.
After hitting the URL, the process is definitely running without being killed, and the limit has been set on that child process:

:0:> sudo cat /proc/2089/limits
Limit Soft Limit Hard Limit Units
Max cpu time 2 2 ms

With this, the kernel doesn't appear to be killing the process quickly, however I suspect that this is due to system time vs CPU time - the example given spends a fair bit of time forking processes, but once it reaches 0:02.00 in top, the kernel kills the process, even though about a minute of wall clock time has elapsed.

[Kees Cook (kees)]

Can you show that your CPU time (ps auwwx | grep test.cgi) is exceeding
the limits set (or lack of set limit) for the process that Apache spawns
(cat /proc/$(pidof test.cgi)/limits)?

[Neil Van Dyke (neil-neilvandyke)]

I appreciate the attempts Ubuntu people have made to reproduce the problem, and I'm baffled that myself and my users are still easily reproducing the problem.

I once again reproduced the problem on one of my Ubuntu configurations, and observed through "/proc/<pid>/limits" that the limits are "unlimited" on the process.

We have had multiple people reproduce this problem here, on multiple systems, at multiple sites, with fresh installs, on both i386 and amd64, and everyone saw the same erroneous behavior with Ubuntu packagings but not with upstream. Nor did Debian exhibit this problem. We have also been aware that the limit was in CPU seconds, not wall clock time. (I was very skeptical myself, when they first called me in as a fresh pair of eyes, after very experienced people there were stumped.)

I suppose the only thing to do at this point, as far as this bug report is concerned,are: (1) for me to start once again with a fresh Ubuntu install, and this time to carefully log each step in a form appropriate for this bug report (perhaps video it, too! :); and (2) for me to get the problematic Apache process debuggable and locate the erroneous behavior that way.

Realistically, this problem is so bizarre and has been so time-consuming that (though I wouldn't attribute "fault" til we know the cause), my users will also have to consider known-good options, such as running upstream Apache or switching the distro to Debian "stable" or RHEL/CentOS.

[Kees Cook (kees)]

I have re-tested on i386 hardy and got the same results as ajmitch: the test program was correctly killed. Neil, if you can post steps to reproduce, I would be very thankful. Sorry it's not any easy problem to uncover. :(

[Neil Van Dyke (neil-neilvandyke)]

I have just reproduced the problem with a fresh install with latest updates, and kept a detailed log as I did so. Perhaps someone can spot something I am doing wrong? I would assume that I'm doing something wrong, except comparable things work on Debian and with upstream, and my users also experienced the problem independently before I did. A fresh pair of eyes on this would be appreciated!

* Download Ubuntu Server 8.04.2 for i386, and burn it to a CD-R. I used
  the following, as fetched on 2009-07-01:
  http://ftp-mirror.internap.com/pub/ubuntu-releases/hardy/ubuntu-8.04.2-server-i386.iso

* Use machine IBM/Lenovo ThinkPad T60 with Intel Core Duo (32-bit X86).
  (Note that we have also observed the same behavior on 64-bit server
  hardware.)

* Plug machine into Ethernet.

* Boot the CD-R and do the following:

* Language: English

* From CD boot menu, select: Check CD for defects. Result: "The CD-ROM
  integrity test was successful. The CD-ROM is valid." Reboot system.

* Language: English

* From CD boot menu, select: Test memory. Reboot after a lot of
  successful testing passing.

* Language: English

* From CD boot menu, select: Install Ubuntu Server.

* Choose language: English

* Country: United States

* Detect keyboard layout: No.

* Keyboard origin: USA

* Keyboard layout: USA

* Wait for some device scanning and installing and DHCP.

* Hostname: myserver

* Time zone: Eastern

* Partition disks: Guided - use entire disk. Write changes to disk.

* Wait for partitioning and fs creation.

* Full name of new user: John Smith

* Username: john

* Supply password.

* No HTTP proxy.

* Additional software selection: OpenSSH server

* Wait, then remove CD when ejected, then let reboot.

* SSH into "myserver" as "john" from another machine. Subsequent
  commands are in this shell unless otherwise specified.

* sudo su -

* apt-get update

* apt-get upgrade

* Note that the following packages are upgraded (at approx. 10pm EDT
  2009-07-10):

  apparmor apparmor-utils apt apt-utils base-files cpp-4.2 cron dash
  file gcc-4.2-base initscripts installation-report libcurl3-gnutls
  libgcc1 libgnutls13 libkrb53 libldap-2.4-2 libmagic1 libsasl2-2
  libsasl2-modules libssl0.9.8 libstdc++6 libvolume-id0
  linux-image-2.6.24-23-server linux-ubuntu-modules-2.6.24-23-server
  lsb-base lsb-release ntpdate python-apt sudo sysv-rc sysvutils tasksel
  tasksel-data tzdata udev update-manager-core

* shutdown -r now

* Wait for reboot.

* SSH into "myserver" as "john" from another machine. Subsequent
  commands are in this shell unless otherwise specified.

* sudo su -

* apt-get install apache2

* Create file "/usr/lib/cgi-bin/forever" with the contents:

---- CUT HERE ----
#!/bin/sh
echo "Content-Type: text/html"
echo ""
echo "BEFORE"
while true ; do
    expr 1 + 1 > /dev/null
done
echo "AFTER"
---- CUT HERE ----

* chmod 0755 /usr/lib/cgi-bin/forever

* /etc/init.d/apache2 stop

* Edit "/etc/apache2/sites-available/default" to add an erroneous
  directive to the top of the file: "XXX"

* /etc/init.d/apache2 start

* Verify that the start failed because of the erroneous directive, which
  confirms this config file really is being used.

* Ed...

[Kees Cook (kees)]

Can you attach the virtual host config that you are seeing the issue with?

[Neil Van Dyke (neil-neilvandyke)]

Here is the virtual host config from the procedure documented in:
https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/394350/comments/15

It is file:
/etc/apache2/sites-available/default

[Jeroen Ooms (jeroenooms)]

I am experiencing exactly the same problem as reported by Neal in 2009. I am running the latest Apache2 from Ubuntu 10.10 repository. RLimitCPU does not kill any of my processes.

[Neil Van Dyke (neil-neilvandyke)]

I was asked privately whether I found a solution. We wasted too much time trying to convince people that we had a credible problem report, and then it seemed that no one was prepared to actually do anything with the information. After that time waste, we decided to give up on RLimitCPU on Ubuntu, and focused our energies elsewhere.

[Serge Hallyn (serge-hallyn)]

Odd that others are having trouble reproducing. It reproduces for me on a hardy uec image. /proc/$$/limits for the test.cgi process shows unlimited for cpu, and the script is never killed.

[Serge Hallyn (serge-hallyn)]

strace -f -o/tmp/output /etc/init.d/apache2 start

followed by triggering the script - nowhere in /tmp/output does RLIMIT_CPU get set.

[Serge Hallyn (serge-hallyn)]

verified under natty as well

[Serge Hallyn (serge-hallyn)]

The actual setrlimit is done by apr. Apache source ships with its own coyp of the apr source under srclib, but that does not get compiled (or used).

Apache definately sees the RLimit_CPU configuration and sets internal variables accordingly. I've yet to instrument the libapr1-dev code itself to see why it is not calling setrlimit.

[Serge Hallyn (serge-hallyn)]

At last, here we go.

The file /root/apache2-2.2.17/modules/generators/mod_cgi.c has hooks to set rlimits on spawned off instances. However, test.cgi is being executed by /root/apache2-2.2.17/modules/generators/mod_cgid.c. It has those hooks, but they are #ifdef'd out.

[Serge Hallyn (serge-hallyn)]

The debian sid package also ifdefs them out, so if limits are working on debian it must be because they are configured to use mod_cgi and not mod_cgid?