Double free in libapache2-mod-auth-pgsql causes Apache to crash

Bug #1272857 reported by mubm
48
This bug affects 6 people
Affects Status Importance Assigned to Milestone
libapache2-mod-auth-pgsql (Debian)
Fix Released
Unknown
libapache2-mod-auth-pgsql (Ubuntu)
Fix Released
Medium
mubm
Trusty
Fix Released
Medium
Andreas Hasenack

Bug Description

[Impact]
The libapache2-mod-auth-pgsql module will trigger frequent segfaults in apache if used in conjunction with a CGI script.

[Test Case]

* install the packages on the Ubuntu release you are testing:
$ sudo apt install apache2 libapache2-mod-auth-pgsql postgresql

* create the database and populate it with the test user:
$ sudo -u postgres -H createdb userdb
$ sudo -u postgres -H psql userdb -c "CREATE TABLE UserLogin (Username text, ApachePassword text);"
$ sudo -u postgres -H psql userdb -c "INSERT INTO UserLogin VALUES ('ubuntu', 'secret');"

* Create the DB user the module will use and grant access to the user table:
$ sudo -u postgres -H psql postgres -c "CREATE ROLE www UNENCRYPTED PASSWORD 'password' NOSUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN;"
$ sudo -u postgres -H psql userdb -c "GRANT SELECT ON TABLE userlogin TO www;"

* Create /etc/apache2/conf-available/authpgtest.conf with the following content:
Alias /authpgtest /export/scratch/authpgtest
<Directory /export/scratch/authpgtest/>
  Options +ExecCGI +FollowSymLinks
  AddHandler cgi-script .pl
  AuthType basic
  AuthName "My Auth"
  Require valid-user
  AuthBasicProvider pgsql
  Auth_PG_authoritative On
  Auth_PG_host 127.0.0.1
  Auth_PG_port 5432
  Auth_PG_user www
  Auth_PG_pwd password
  Auth_PG_database userdb
  Auth_PG_encrypted off
  Auth_PG_pwd_table UserLogin
  Auth_PG_uid_field Username
  Auth_PG_pwd_field ApachePassword
</Directory>

* Enable this new configuration:
$ sudo a2enconf authpgtest.conf

* Enable the auth-pgsql and cgi modules and then restart apache:
$ for n in 000_auth_pgsql cgi; do sudo a2enmod $n; done
$ sudo service apache2 restart

* Create the CGI directory for our script:
$ sudo mkdir -p /export/scratch/authpgtest

* Create the CGI script /export/scratch/authpgtest/hw.pl with the following contents:
#!/usr/bin/perl
print "Content-type: text/html\n\n";
print "Hello, World!\n";

* Make it executable:
$ sudo chmod 0755 /export/scratch/authpgtest/hw.pl

* Access the http://ubuntu:secret@localhost/authpgtest/hw.pl URL a few times while tailing /var/log/apache/error.log. After a few tries it will fail, and apache will log a segfault:
$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
Hello, World!
$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
Hello, World!
$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
curl: (52) Empty reply from server

In /var/log/apache2/error.log:
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007fa9340007c8 ***
[Wed Jul 19 20:43:57.077960 2017] [core:notice] [pid 10926:tid 140365262006144] AH00051: child pid 10930 exit signal Aborted (6), possible coredump in /etc/apache2

After installing the fixed libapache2-mod-auth-pgsql package, all attempts will work.

[Regression Potential]
This patch is already being used in Ubuntu releases higher than trusty, all the way to artful, and also in Debian.

This is a very old module that hasn't been built in a while (see [other info] below. It's possible that just by rebuilding it with the new environment available in Trusty could introduce unknowns. Hopefully, if that happens, it will be immediately noticed by the people who use it and will test this SRU.

[Other Info]
This module hasn't been rebuilt since vivid and seems unmaintained, being at version 2.0.3 since the precise days:
 libapache2-mod-auth-pgsql | 2.0.3-5build2 | precise
 libapache2-mod-auth-pgsql | 2.0.3-6 | trusty
 libapache2-mod-auth-pgsql | 2.0.3-6.1 | vivid
 libapache2-mod-auth-pgsql | 2.0.3-6.1 | xenial
 libapache2-mod-auth-pgsql | 2.0.3-6.1 | yakkety
 libapache2-mod-auth-pgsql | 2.0.3-6.1 | zesty
 libapache2-mod-auth-pgsql | 2.0.3-6.1ubuntu1 | artful

- Debian's last changelog entry is from August 2013
- Fedora killed it in July 2011
- I couldn't find it in SuSE

Revision history for this message
mubm (mubm) wrote :
Changed in apache2 (Ubuntu):
assignee: nobody → mubm (mubm)
information type: Private → Public
Revision history for this message
Apport retracing service (apport) wrote :

StacktraceTop:
 daemon_signal_handler (sig=1) at mod_cgid.c:573
 <signal handler called>
 __accept_nocancel () at ../sysdeps/unix/syscall-template.S:81
 cgid_server (data=data@entry=0x7f19808a4de0) at mod_cgid.c:686
 cgid_start (p=0x7f19809dd028, main_server=0x7f19808a4de0, procnew=0x7f19809ba0e8) at mod_cgid.c:876

Revision history for this message
Apport retracing service (apport) wrote : Stacktrace.txt
Revision history for this message
Apport retracing service (apport) wrote : StacktraceSource.txt
Revision history for this message
Apport retracing service (apport) wrote : ThreadStacktrace.txt
Changed in apache2 (Ubuntu):
importance: Undecided → Medium
tags: removed: need-amd64-retrace
Revision history for this message
Launchpad Janitor (janitor) wrote : Re: apache2 crashed with SIGSEGV in <signal handler called>()

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in apache2 (Ubuntu):
status: New → Confirmed
Revision history for this message
Christian Bachmaier (chkis) wrote :

Calling a cgi-sript using mod_cgid unter Trusty Thar delivers segementation faults nearly on every single request in error.log:

[Tue Mar 25 10:30:07.598453 2014] [core:notice] [pid 1178:tid 140037490894720] AH00051: child pid 7555 exit signal Segmentation fault (11), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': double free or corruption (!prev): 0x00007f5cdc0019a0 ***
[Tue Mar 25 10:30:09.601954 2014] [core:notice] [pid 1178:tid 140037490894720] AH00051: child pid 7593 exit signal Segmentation fault (11), possible coredump in /etc/apache2

However, this is not always and mostly the content is correctly displayed in the client browser.

Revision history for this message
Christian Bachmaier (chkis) wrote :

Calling a python cgi script delivers in error.log (with all regular ubuntu updates/patches installed):

*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007ff1a80004b8 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007ff1a80004b8 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007ff1a8017840 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007ff1a8017840 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007ff1a8015a20 ***
*** Error in `/usr/sbin/apache2': double free or corruption (!prev): 0x00007ff1a80019a0 ***
[Fri Apr 04 08:43:44.969263 2014] [core:notice] [pid 1764:tid 140676438046592] AH00051: child pid 1769 exit signal Segmentation fault (11), possible coredump in /etc/apache2

Revision history for this message
Christian Bachmaier (chkis) wrote :

Now I could sharpen the problem. It only shows up if using AuthPG libapache2-mod-auth-pgsql in connection with a cgi script. To reproduce put the following in /etc/apache2/conf.d/authpgtest.conf and then execute a2enconf authpgtest.conf and service apache2 reload .

Alias /authpgtest /export/scratch/authpgtest
<Directory /export/scratch/authpgtest/>
  Options +ExecCGI +FollowSymLinks
  AddHandler cgi-script .pl

  AuthName "TestRealm"
  AuthType Basic
  AuthBasicProvider pgsql
  Auth_PG_host /var/run/postgresql
  Auth_PG_port 5432
  Auth_PG_database authpgtestdb
  Auth_PG_pwd_table ident_users
  Auth_PG_uid_field uname
  Auth_PG_pwd_field password
  Auth_PG_encrypted off
  Require valid-user
</Directory>

Of course the postgres db authpgtestdb with table ident_users with columns uname and password have to exist, or the params adapted.
Under /export/scratch/authpgtest/hw.pl put the follwing.

#!/usr/bin/perl

print "Content-type: text/html\n\n";
print "Hello, World!\n";

Then call http://<host>/authpgtest/hw.pl in your browser while tail -f tail -f /var/log/apache2/error.log is running. Press the reload button of the browser a few times.

Without using AuthPG-Realm I think the errors do not show up. Unfortunately, I need this for production and a workaround seems not to doable.

Chris

Revision history for this message
Tim Dawborn (tim-dawborn) wrote :

I'm seeing exactly the same thing as #9.

<LocationMatch "/git/setup(\.git)?">
  AuthName "Git repository: setup"
  AuthBasicAuthoritative Off
  AuthBasicProvider pgsql
  AuthType Basic
  AuthUserFile /dev/null
  require valid-user

  Auth_PG_host XXX
  Auth_PG_port XXX
  Auth_PG_user XXX
  Auth_PG_pwd XXX
  Auth_PG_database XXX
  Auth_PG_encrypted on
  Auth_PG_hash_type MD5

  Auth_PG_pwd_table "view_users, view_groups"
  Auth_PG_uid_field view_users.login
  Auth_PG_pwd_field view_users.hashed_password
  Auth_PG_pwd_whereclause "AND view_users.login = view_groups.login AND view_groups.group_name = 'XXX'"
</LocationMatch>

[Wed Jun 04 15:23:35.226264 2014] [core:notice] [pid 22392:tid 139964758747008] AH00051: child pid 27013 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f4bf4038c50 ***
[Wed Jun 04 15:23:36.228084 2014] [core:notice] [pid 22392:tid 139964758747008] AH00051: child pid 27012 exit signal Aborted (6), possible coredump in /etc/apache2
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f4bf4038c50 ***
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007f4bf4038c50 ***

Revision history for this message
Tim Dawborn (tim-dawborn) wrote :

Whoops, forgot the CGI script component of the Apache config:

# Git HTTP Backend
SetEnv GIT_HTTP_EXPORT_ALL 1
SetEnv GIT_PROJECT_ROOT XXX
ScriptAlias /git /usr/lib/git-core/git-http-backend

Revision history for this message
Andreas Gleißner (gleissner) wrote :

Hello,

I believe to have identified the problem and
propose a fix (see attachment).

The module has a global variable PGconn *pg_conn, which is a pointer to
a PostgreSQL connection.
The code determines if there is an active connection by checking if
pg_conn is non-NULL.
However, the connection is closed by calling PQfinish(pg_conn) without
resetting pg_conn to NULL.
The documentation of libpq says that PQfinish frees the memory used by
the PGConn object.
Hence, when Apache calls check_password the second time, the code
falsely assumes (as pg_conn != NULL) that there is an active connection
and tries to access the previously freed PGconn object, which results in
a segmentation fault.
The same problem applies to the pointer PGresult *pg_result, for which
PQclear(pg_result) is called.

My proposed solution consists in simply resetting pg_conn = NULL after
each call of PQfinish(pg_conn) and resetting pg_result = NULL after each
call of PQclear(pg_result).

Andreas

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "doublefree.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Revision history for this message
Christian Bachmaier (chkis) wrote :

I am using the code of Andreas in #12 now in a productive environment. So far without any problems. Could someone assign the bug to the package maintainer? It seems like I have not the option to do that here. Thank you, Chris.

Revision history for this message
Marco Nenciarini (mnencia) wrote :

I confirm the bug. I'll add the patch to the Debian package soon.

Revision history for this message
Robie Basak (racb) wrote :

Looks like this was fixed in 2.0.3-6.1 which was synced from Debian in Vivid, and so is also fixed in Wily.

If someone can prepare a backport, please follow the steps at https://wiki.ubuntu.com/StableReleaseUpdates#Procedure to have 14.04 updated.

affects: apache2 (Ubuntu) → libapache2-mod-auth-pgsql (Ubuntu)
Robie Basak (racb)
summary: - apache2 crashed with SIGSEGV in <signal handler called>()
+ Double free in libapache2-mod-auth-pgsql causes Apache to crash
Changed in libapache2-mod-auth-pgsql (Ubuntu):
status: Confirmed → Fix Released
Changed in libapache2-mod-auth-pgsql (Ubuntu Trusty):
status: New → Triaged
importance: Undecided → Medium
Changed in libapache2-mod-auth-pgsql (Debian):
status: Unknown → Fix Released
Revision history for this message
Peder Stray (pstray) wrote :

Still no build for Trusty. I would really appreciate if someone could fix that.

tags: added: server-next
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I have a branch (https://code.launchpad.net/~ahasenack/ubuntu/+source/libapache2-mod-auth-pgsql/+git/libapache2-mod-auth-pgsql/+ref/trusty-check-null-double-free-1698758-1272857) to SRU this fix and bug #1698758 at the same time. Just waiting on the latter to land in artful, then the SRU floodgates can open.

Changed in libapache2-mod-auth-pgsql (Ubuntu Trusty):
assignee: nobody → Andreas Hasenack (ahasenack)
status: Triaged → In Progress
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

debdiff for trusty

description: updated
description: updated
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello mubm, or anyone else affected,

Accepted libapache2-mod-auth-pgsql into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libapache2-mod-auth-pgsql/2.0.3-6ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libapache2-mod-auth-pgsql (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Trusty verification

Confirmed the segfault with libapache2-mod-auth-pgsql 2.0.3-6:
ubuntu@trusty-mod-auth-pgsql-double-free-1272857:~$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
Hello, World!
ubuntu@trusty-mod-auth-pgsql-double-free-1272857:~$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
Hello, World!
ubuntu@trusty-mod-auth-pgsql-double-free-1272857:~$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
curl: (52) Empty reply from server

logs:
*** Error in `/usr/sbin/apache2': free(): invalid pointer: 0x00007effd80007c8 ***
[Thu Aug 03 14:03:55.357288 2017] [core:notice] [pid 6943:tid 139637886596992] AH00051: child pid 6947 exit signal Aborted (6), possible coredump in /etc/apache2

Installing the version from proposed:
(...)
Get:1 http://br.archive.ubuntu.com/ubuntu/ trusty-proposed/main libapache2-mod-auth-pgsql amd64 2.0.3-6ubuntu0.1 [18.6 kB]
Fetched 18.6 kB in 0s (1,000 kB/s)
(Reading database ... 26196 files and directories currently installed.)
Preparing to unpack .../libapache2-mod-auth-pgsql_2.0.3-6ubuntu0.1_amd64.deb ...
Unpacking libapache2-mod-auth-pgsql (2.0.3-6ubuntu0.1) over (2.0.3-6) ...
Setting up libapache2-mod-auth-pgsql (2.0.3-6ubuntu0.1) ...
apache2_invoke 000_auth_pgsql: already enabled
 * Restarting web server apache2

Repeating the test several times, no crash:
ubuntu@trusty-mod-auth-pgsql-double-free-1272857:~$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
Hello, World!
ubuntu@trusty-mod-auth-pgsql-double-free-1272857:~$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
Hello, World!
ubuntu@trusty-mod-auth-pgsql-double-free-1272857:~$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
Hello, World!
ubuntu@trusty-mod-auth-pgsql-double-free-1272857:~$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
Hello, World!
ubuntu@trusty-mod-auth-pgsql-double-free-1272857:~$ curl -f http://ubuntu:secret@localhost/authpgtest/hw.pl
Hello, World!

tags: added: verification-done-xenial
removed: verification-needed
tags: added: verification-done-trusty
removed: verification-done-xenial
tags: added: verification-done
Revision history for this message
Sebastien Bacher (seb128) wrote :

(it doesn't look like there is anything left to review/upload so unsubscribing sponsors)

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libapache2-mod-auth-pgsql - 2.0.3-6ubuntu0.1

---------------
libapache2-mod-auth-pgsql (2.0.3-6ubuntu0.1) trusty; urgency=medium

  * d/p/fixdoublefree.patch: set freed pointers to NULL before subsequent
    checks against NULL. (LP: #1272857)
  * d/p/crypt-check-null-1698758.patch: check for a NULL return from crypt(3)
    (LP: #1698758)

 -- Andreas Hasenack <email address hidden> Thu, 22 Jun 2017 16:54:09 -0300

Changed in libapache2-mod-auth-pgsql (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for libapache2-mod-auth-pgsql has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.