mysql configuration should be adjusted to help prevent against chained attacks against LAMP stack

[Todd Smith]

SRU Justification (apparmor)

1. impact of the bug is medium for stable releases. While no applications shipped in Ubuntu are directly affected by this, it would be good if our LTS release provided a more secure user-tmp abstraction for people deploying new profiles on Ubuntu 10.04 LTS.

2. This has been addressed during the maverick development cycle.

3. Patch is small. It places 'owner' in front of /tmp/** and /var/tmp/** as well as requiring 'owner' for @{HOME}/tmp/ and its files and subdirectories.

4. TEST CASE:
$ cp /usr/share/example-content/Kubuntu_leaflet.jpg /tmp
$ sudo chown root:root /tmp/Kubuntu_leaflet.jpg
$ sudo aa-enforce /etc/apparmor.d/usr.bin.firefox
$ firefox /tmp/Kubuntu_leaflet.jpg

At this point, firefox will not display the image and something like the following should be in dmesg:
[ 1298.220693] type=1503 audit(1288797298.697:138): operation="open" pid=2948 parent=2944 profile="/usr/lib/firefox-3.6.12/firefox-*bin" requested_mask="::r" denied_mask="::r" fsuid=1000 ouid=0 name="/tmp/Kubuntu_leaflet.jpg"

5. This will regress if a confined application tries to access files owned by another user in /tmp (indeed, that is the protection we want ;) and when someone confines two different applications that a) run under differing user ids and b) interact with each other by one writing to /tmp and the other reading that file from /tmp. I imagine that there are very few users who would be affected by this. On the desktop, the evince profile is affected at all by this change because it explicitly allows read access to any files with an extension that it has support for. Firefox's profile is disabled by default.

This is a change requiring the most testing and thought. I maintain it is an important proactive fix for Lucid. It has been in maverick for several months with no reported regressions once we decided on the right approach. Once in -proposed, I plan to run the QRT tests on all AppArmor confined applications in Lucid to verify no regressions.

Binary package hint: apparmor

I have reported this to the CERT/Bugtraq system so you may have been contacted by them. It was a large bug report so something may have fallen though the cracks.

The problem is AppArmor rule sets do not adequately protect a LAMP environment from attacks. Exploit code has been written which bypasses AppAmoror rule sets to obtain remote code execution. The exploit can be obtained here (https://sitewat.ch/Exploits/nuke_exploit.txt).

The attack scenario:
Back before AppArmor it was common to see sql injection attacks against PHP/MySQL like this:
Vulnerable code:
<?php
mysql_query("select name from user where id=".$_GET[id]);
?>
Exploit:
http://localhost/sql_inj.php?id=0 union select "<?php eval($_GET[e]);?>" into outfile "/var/www/backdoor.php"

AppArmor stops this attack, which is impressive. However, there is a flaw in this security system. In my exploit i am dropping the file in "/tmp/theme.php" then i use a Local File Include vulnerability (LFI) to execute this php file. The problem is that BOTH MySQL and Apache have access to /tmp/. The line "#include <abstractions/user-tmp>" in the usr.sbin.mysqld is the source of the vulnerability. The patch is very simple, mysql should have its own tmp folder that only the mysqld process has access to.

This whole concept of process separation to prevent attacks is completely undermined by creating "unions" between processes in the form of these header files. In fact every time you see an #include in an app armor rule set, its a point of weakness. I hope to give a another killer blackhat/defcon talk, this time i am talking about about my exploit and these abuses against apparmor.

Thanks,
Michael Brooks.

[Jamie Strandboge (jdstrand)]

Thank you for using Ubuntu and reporting a bug.

The initial title for this bug is misleading: "Bypass AppArmor ruleset of MySQL allows for remote code execution". First, the MySQL profile does not allow MySQL to execute code in /tmp or other places and none of the profile's ruleset was bypassed. As you said, it does allow writing files to /tmp (see below for possible improvements). Second, Ubuntu does not ship a profile for apache, php or phpnuke. While there is an example profile for phpsysinfo (that is in complain mode), it does not use the user-tmp abstraction. Third, the attack relies upon php being able to include a file from /tmp. This should be considered a broken php configuration.

That said, this bug does make a good point: administrators creating AppArmor profiles and configuring their systems need to be mindful of interactions between applications. As demonstrated, AppArmor can both be configured to prevent attacks in vulnerable applications as well as misconfigured to allow said attacks.

As described, there are several things which could help prevent this attack from succeeding, remembering that mysql runs as the 'mysql' user and apache runs as the 'www-data' user in Ubuntu:
1. include_path could be used to disallow including files from /tmp. Because of the wide range of applications that user's can install, Ubuntu cannot ship with a hardened include_path, but it should be considered standard/best practice to adjust this for precisely the bug you mentioned. Also, DAC should have prevented writing to /var/www in the first place (though by your description, AppArmor caught it first).
2. as mentioned, MySQL could be configured to use a different tmp directory with mode 700, then normal DAC would have prevented this attack. This can be done by adjusting the 'tmpdir' variable in mysql.cnf. This should be considered for inclusion in Ubuntu.
3. a umask of 077 could be used for MySQL in which case normal DAC would have prevented this. This should be considered for inclusion in Ubuntu
4. the php application could have been protected with AppArmor, disallowing reads from /tmp

I do consider it a bug that the LAMP stack is allowed to interact in this manner. MySQL does need a temporary directory to do its work, and the problem is not in AppArmor (we can't deny access to MySQL's scratch area without breaking it) but in the default configuration for MySQL in Ubuntu. Adjusting MySQL's umask to 077 is probably the simplest and safest change that could be made, and would have prevented this attack.

I am going to open a task against mysql for 2 and 3 above. I do not believe this is a bug in AppArmor because, as mentioned, it is simply allowing the necessary access to MySQL's scratch area and Ubuntu does not ship a profile for this vulnerable php application, apache or php (if an administrator writes one, then it is the administrator's responsibility to understand the interactions between the software in use on his/her system).

That said, I am going to leave the apparmor task open for now, because one improvement could be considered in the user-tmp abstraction: we could use an owner match. If both MySQL and the php application were confined by...

[Todd Smith (td-smith)]

Jamie Strandboge,

Your response is in depth and I agree that your patches would break my exploit in its current form. I highly disagree with the following statement:
"I do not believe this is a bug in AppArmor because, as mentioned, it is simply allowing the necessary access to MySQL's scratch area and Ubuntu does not ship a profile for this vulnerable php application, apache or php (if an administrator writes one, then it is the administrator's responsibility to understand the interactions between the software in use on his/her system)."

The vast majority of administrators will use the default AppArmor rule sets provided to them and will probably be unaware of its existence. Although Ubuntu only ships with MySQL rules the principal of exploit chaining is to take advantage of weak points in the system as a whole. In a sense exploit chaining is used to exploit the weakest links in a chain of security systems. The lack of rule sets is every much of a vulnerability as an ineffective one. The acknowledgment of this bypass of an AppArmor rule set would in turn be an acknowledgment of a fundamental design flaw in AppArmor and this is a difficult statement to make in public. Although my exploit and future exploits will make this statement clear enough. I have no doubt AppArmor dev team will react to this.

It should also be noted that more strict forms of processes separation such as SELinux, Virtual Machines and Chroots do not suffer from this same attack. The exploit that I posted, doesn't work under Fedora. This is because SELinux will not allow PHP to include owned by another user such as mysqld or any other process that that is exploited. I think this is the simplest and most widely effective patch against my attack pattern. Perhaps AppArmor can inherit this elegance.

Thanks,
Michael Brooks

[Jamie Strandboge (jdstrand)]

Thanks for your response. My point 4 attempted to address your point:
4. the php application could have been protected with AppArmor, disallowing reads from /tmp

You cannot attribute a bug to AppArmor for a profile that does not exist. The AppArmor model is to confine processes, not data. Because apache does not have a profile in the default installation, AppArmor cannot confine it. Therefore it is not a bug in AppArmor but rather a missing security feature in distributions like Ubuntu that have AppArmor available but do not have a profile for a particular application, in this case apache. However, this is true of any system that doesn't have full labeling and confinement. Fedora happens to have policy for apache, but this attack would presumably work on Fedora if it was against a different application for which policy was not written (such as lighttpd). Simply put, one cannot bypass a rule that does not exist. The MySQL ruleset was not bypassed -- MySQL needs an area to write files and the ruleset allows that. Admittedly, it is a bug in the MySQL Debian/Ubuntu packaging that allows this attack to succeed and this will be addressed. If Apache shipped an enforcing profile which used the user-tmp abstraction as is, only then could you attribute that the intended AppArmor protections were bypassed. This bug highlights that the user-tmp abstraction is too loose for when people have profiled the applications involved and are depending on it to work to thwart chained attacks, and I committed a fix for that to trunk which requires 'owner' matching which should largely address this.

Fedora happens to have a MAC feature available in its distribution (see below for why Ubuntu made a different choice) for apache. Unfortunately, due to packaging practices in Ubuntu and Debian, it is very difficult to confine apache in a general enough fashion for users to install and use web applications that provides any significant security benefit. This is a known issue and something that is being worked on (it is on the Ubuntu security team's roadmap) in various ways. I won't go into those details further in this bug, but if you are interested, I encourage you to discuss this in the ubuntu-hardened mailing list and/or IRC channel.

AIUI, this vulnerability does not work on Fedora because apache is confined with SELinux. Apache in Ubuntu is not confined by AppArmor. In Fedora, users are often required to adjust SELinux policy to get their applications to work. In Ubuntu, we choose to not break those applications, but instead provide an example of how to confine a web application that administrators can use to opt into the AppArmor security feature. Our stance is that if a security feature breaks default and common configurations, users will turn off the feature. While it is possible to only disable the Apache profile, if people are frustrated enough by a too strict AppArmor profile, they will disable all of AppArmor on the system, which will weaken the system overall since there are several profiles enforced on a typical Ubuntu system (such as the aforementioned MySQL profile). This is a distribution choice.

[Todd Smith (td-smith)]

Hey Jamie,

For the most part I agree with your stance and I am happy to see the summary update. I also totally agree with this statement:
"Our stance is that if a security feature[SELinux] breaks default and common configurations, users will turn off the feature."

PHP-Nuke will not run on a default Fedora system because of SELinux and I think that the most common response is for people to disable it all together. I agree that a security measure like this should be avoided at all costs in Ubuntu. I think that we can both agree that there is a common ground in terms of security and usability. I will keep an eye on this problem and see that it matures properly.

You are correct AppArmor doesn't have a feature to protect the context in which data is accessed like SELinux, and it would be nice if it did. My argument is that AppArmor with its current feature set can be configured to break my exploit, but other proven security measures can also be used to address this issue. I would like to be involved with Hardened Ubuntu to help find a good solution to these problems.

Thanks Again,
Michael

[Todd Smith (td-smith)]

Hey Mr Cook,

I am very interested in how this patch. Before your roll it out i want to
make sure there isn't a trivial bypass. Please keep me informed, I want
Ubuntu be rock solid.

Thanks,
Michael Brooks

On Fri, Jun 4, 2010 at 5:12 PM, Kees Cook <email address hidden> wrote:

> ** Changed in: apparmor (Ubuntu)
> Importance: Undecided => Medium
>
> ** Changed in: mysql-dfsg-5.1 (Ubuntu)
> Importance: Undecided => Medium
>
> ** Changed in: apparmor (Ubuntu)
> Assignee: (unassigned) => Jamie Strandboge (jdstrand)
>
> --
> mysql configuration should be adjusted to help prevent against chained
> attacks against LAMP stack
> https://bugs.launchpad.net/bugs/578922
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in “apparmor” package in Ubuntu: Fix Committed
> Status in “mysql-dfsg-5.1” package in Ubuntu: Triaged
>
> Bug description:
> Binary package hint: apparmor
>
> I have reported this to the CERT/Bugtraq system so you may have been
> contacted by them. It was a large bug report so something may have fallen
> though the cracks.
>
> The problem is AppArmor rule sets do not adequately protect a LAMP
> environment from attacks. Exploit code has been written which bypasses
> AppAmoror rule sets to obtain remote code execution. The exploit can be
> obtained here (https://sitewat.ch/Exploits/nuke_exploit.txt).
>
> The attack scenario:
> Back before AppArmor it was common to see sql injection attacks against
> PHP/MySQL like this:
> Vulnerable code:
> <?php
> mysql_query("select name from user where id=".$_GET[id]);
> ?>
> Exploit:
> http://localhost/sql_inj.php?id=0 union select "<?php eval($_GET[e]);?>"
> into outfile "/var/www/backdoor.php"
>
> AppArmor stops this attack, which is impressive. However, there is a
> flaw in this security system. In my exploit i am dropping the file in
> "/tmp/theme.php" then i use a Local File Include vulnerability (LFI) to
> execute this php file. The problem is that BOTH MySQL and Apache have
> access to /tmp/. The line "#include <abstractions/user-tmp>" in the
> usr.sbin.mysqld is the source of the vulnerability. The patch is very
> simple, mysql should have its own tmp folder that only the mysqld process
> has access to.
>
> This whole concept of process separation to prevent attacks is completely
> undermined by creating "unions" between processes in the form of these
> header files. In fact every time you see an #include in an app armor rule
> set, its a point of weakness. I hope to give a another killer
> blackhat/defcon talk, this time i am talking about about my exploit and
> these abuses against apparmor.
>
> Thanks,
> Michael Brooks.
>
>
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/578922/+subscribe
>

[Launchpad Janitor (janitor)]

This bug was fixed in the package apparmor - 2.5.1~pre1393-0ubuntu1

---------------
apparmor (2.5.1~pre1393-0ubuntu1) maverick; urgency=low

  * Update to upstream bzr revision 1393 from lp:apparmor/2.5.
    * add dbus-session abstraction (LP: #566207)
    * require owner in user-tmp abstraction (LP: #578922)
    * don't use uninitialized $opt_s (LP: #582075)
    * allow thunderbird 3 in abstractions/ubuntu-email (LP: #590462)
    * allow gmplayer in abstractions/ubuntu-media-players (LP: #591421)
  * debian/control: updated branches.
  * debian/patches/0001-local-includes.patch: backported patch from trunk to
    allow local administrators to customize their profiles without modifying
    a shipped profile
  * debian/rules:
    - don't pass RELEASE to libapparmor's 'make install' as it breaks the
      build and isn't used by the Makfile anyway
    - install apparmor.d/local/README in apparmor, not apparmor-profiles
    - don't install apparmor.d/local/usr.sbin.ntpd
  * Drop the following patches already included upstream:
    - 0001-lp538561.patch
    - 0002-aalogprof-warnings.patch
    - 0003-fix-memleaks.patch
    - 0004-lp549557.patch
    - 0005-lp538661.patch
    - 0006-lp611248.patch
 -- Jamie Strandboge <email address hidden> Thu, 05 Aug 2010 16:10:46 -0500

[Jamie Strandboge (jdstrand)]

I talked to our server team about this, and they said that changing the temp directory for MySQL is actually bug #375371. I am going to mark the MySQL task as "Invalid" here (for lack of a better category) and encourage discussion of moving the temporary directory be moved to bug #375371. If that bug becomes "Won't Fix" we should reopen the MySQL task in this one for setting the MySQL umask.

[Todd Smith (td-smith)]

I talked to 2 developers that are on the AppArmor team after my Defcon talk
and they have a fix in the Linux Mainline. Channing the MySQL's temp
directory is probably unnecessary due to AppArmor improvements, although I
haven't gotten around to testing it. I plan on doing an extensive test very
soon. You should expect additions to this security related bug report.

On Mon, Aug 16, 2010 at 5:33 AM, Jamie Strandboge <email address hidden> wrote:

> I talked to our server team about this, and they said that changing the
> temp directory for MySQL is actually bug #375371. I am going to mark the
> MySQL task as "Invalid" here (for lack of a better category) and
> encourage discussion of moving the temporary directory be moved to bug
> #375371. If that bug becomes "Won't Fix" we should reopen the MySQL task
> in this one for setting the MySQL umask.
>
> ** Changed in: mysql-dfsg-5.1 (Ubuntu)
> Status: Triaged => Won't Fix
>
> --
> mysql configuration should be adjusted to help prevent against chained
> attacks against LAMP stack
> https://bugs.launchpad.net/bugs/578922
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in “apparmor” package in Ubuntu: Fix Released
> Status in “mysql-dfsg-5.1” package in Ubuntu: Won't Fix
>
> Bug description:
> Binary package hint: apparmor
>
> I have reported this to the CERT/Bugtraq system so you may have been
> contacted by them. It was a large bug report so something may have fallen
> though the cracks.
>
> The problem is AppArmor rule sets do not adequately protect a LAMP
> environment from attacks. Exploit code has been written which bypasses
> AppAmoror rule sets to obtain remote code execution. The exploit can be
> obtained here (https://sitewat.ch/Exploits/nuke_exploit.txt).
>
> The attack scenario:
> Back before AppArmor it was common to see sql injection attacks against
> PHP/MySQL like this:
> Vulnerable code:
> <?php
> mysql_query("select name from user where id=".$_GET[id]);
> ?>
> Exploit:
> http://localhost/sql_inj.php?id=0 union select "<?php eval($_GET[e]);?>"
> into outfile "/var/www/backdoor.php"
>
> AppArmor stops this attack, which is impressive. However, there is a
> flaw in this security system. In my exploit i am dropping the file in
> "/tmp/theme.php" then i use a Local File Include vulnerability (LFI) to
> execute this php file. The problem is that BOTH MySQL and Apache have
> access to /tmp/. The line "#include <abstractions/user-tmp>" in the
> usr.sbin.mysqld is the source of the vulnerability. The patch is very
> simple, mysql should have its own tmp folder that only the mysqld process
> has access to.
>
> This whole concept of process separation to prevent attacks is completely
> undermined by creating "unions" between processes in the form of these
> header files. In fact every time you see an #include in an app armor rule
> set, its a point of weakness. I hope to give a another killer
> blackhat/defcon talk, this time i am talking about about my exploit and
> these abuses against apparmor.
>
> Thanks,
> Michael Brooks.
>
>
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/ap...

[Jamie Strandboge (jdstrand)]

SRU Justification (apparmor)

1. impact of the bug is medium for stable releases. While no applications shipped in Ubuntu are directly affected by this, it would be good if our LTS release provided a more secure user-tmp abstraction for people deploying new profiles on Ubuntu 10.04 LTS.

2. This has been addressed during the maverick development cycle.

3. Patch is small. It places 'owner' in front of /tmp/** and /var/tmp/** as well as requiring 'owner' for @{HOME}/tmp/ and its files and subdirectories.

4. TEST CASE:
$ cp /usr/share/example-content/Kubuntu_leaflet.jpg /tmp
$ sudo chown root:root /tmp/Kubuntu_leaflet.jpg
$ sudo aa-enforce /etc/apparmor.d/usr.bin.firefox
$ firefox /tmp/Kubuntu_leaflet.jpg

At this point, firefox will not display the image and something like the following should be in dmesg:
[ 1298.220693] type=1503 audit(1288797298.697:138): operation="open" pid=2948 parent=2944 profile="/usr/lib/firefox-3.6.12/firefox-*bin" requested_mask="::r" denied_mask="::r" fsuid=1000 ouid=0 name="/tmp/Kubuntu_leaflet.jpg"

5. This will regress if a confined application tries to access files owned by another user in /tmp (indeed, that is the protection we want ;) and when someone confines two different applications that a) run under differing user ids and b) interact with each other by one writing to /tmp and the other reading that file from /tmp. I imagine that there are very few users who would be affected by this. On the desktop, the evince profile is affected at all by this change because it explicitly allows read access to any files with an extension that it has support for. Firefox's profile is disabled by default.

This is a change requiring the most testing and thought. I maintain it is an important proactive fix for Lucid. It has been in maverick for several months with no reported regressions once we decided on the right approach. Once in -proposed, I plan to run the QRT tests on all AppArmor confined applications in Lucid to verify no regressions.

[Martin Pitt (pitti)]

Accepted apparmor into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Jamie Strandboge (jdstrand)]

Upgraded to 2.5.1-0ubuntu0.10.04.1 in lucid-proposed and this issue is resolved.

[Launchpad Janitor (janitor)]

This bug was fixed in the package apparmor - 2.5.1-0ubuntu0.10.04.1

---------------
apparmor (2.5.1-0ubuntu0.10.04.1) lucid-proposed; urgency=low

  * Backport 2.5.1-0ubuntu0.10.10.1 from maverick for userspace tools to work
    with newer kernels (LP: #660077)
    NOTE: user-tmp now uses 'owner' match, so non-default profiles will have
    to be adjusted when 2 separately confined applications that both use the
    user-tmp abstraction depend on being able to cooperatively share files
    with each other in /tmp or /var/tmp.
  * remove the following patches (features not appropriate for SRU):
    - 0002-add-chromium-browser.patch
    - 0003-local-includes.patch
    - 0004-ubuntu-abstractions-updates.patch
  * debian/rules (this makes it the same as what was shipped in 10.04 LTS
    release):
    - don't ship aa-update-browser and its man page (requires
      0004-ubuntu-abstractions-updates.patch)
    - don't ship apparmor.d/local/ (requires 0003-local-includes.patch)
    - don't use dh_apparmor (not in Ubuntu 10.04 LTS)
    - don't ship chromium profile
  * remove debian/profiles/chromium-browser
  * remove debian/aa-update-browser*
  * debian/apparmor-profiles.postinst: revert to that in lucid release
    (requires dh_apparmor and 0002-add-chromium-browser.patch)
  * remove debian/apparmor-profiles.postrm: doesn't make sense without
    0002-add-chromium-browser.patch
  * debian/control:
    - revert Build-Depends on debhelper (>= 5)
    - revert Standards-Version to 3.8.4
    - revert Vcs-Bzr
    - use Conflicts/Replaces version that was in Ubuntu 10.04 LTS
  * debian/patches/0011-lucid-compat-dbus.patch: move /var/lib/dbus/machine-id
    back into dbus, since profiles on 10.04 LTS expect it there
  * debian/patches/0012-lucid-compat-kde.patch: add kde4-config to kde
    abstraction, since the firefox profile on Ubuntu 10.04 LTS expects it to
    be there

apparmor (2.5.1-0ubuntu0.10.10.2) maverick-proposed; urgency=low

  * New upstream release (LP: #660077)
    - The following patches were refreshed:
      + 0001-fix-release.patch
      + 0003-local-includes.patch
      + 0004-ubuntu-abstractions-updates.patch
      + 0008-lp648900.patch: renamed as 0005-lp648900.patch
    - The following patches were dropped (included upstream):
      + 0005-lp601583.patch
      + 0006-network-interface-enumeration.patch
      + 0007-gnome-updates.patch
  * debian/patches/0006-testsuite-fixes.patch: testsuite fixes from head
    of 2.5 branch. These are needed for QRT and SRU testing (LP: #652211)
  * debian/patches/0007-honor-cflags.patch: have the parser makefile honor
    CFLAGS environment variable. Brings back missing symbols for the retracer
  * debian/patches/0008-lp652674.patch: fix warnings for messages without
    denied or requested masks (LP: #652674)
  * debian/apparmor.init: fix path to aa-status (LP: #654841)
  * debian/apport/source_apparmor.py: apport hook should use
    root_command_hook() for running apparmor_status (LP: #655529)
  * debian/apport/source_apparmor.py: use ProcKernelCmdline and don't clobber
    cmdline details (LP: #657091)
  * debian/{rules,control}: move apache2 abstractions into the base package
    so we can put ...