symlink attack vulnerability in init/helper scripts

[Christian Ehrhardt ]

Please keep information from this thread confidential until 2017-11-09, and
please reply with the information requested below. As a special exception, I
encourage you to share it with your distribution's own security team.

The PostgreSQL security team has learned of a class of vulnerabilities
affecting major PostgreSQL distributions represented here. The security team
is disclosing this to packagers in advance of the public so each of you can
find and fix vulnerable code in your distributions.

Scripts that the root user calls to start PostgreSQL are often vulnerable to
symlink attack through their log file handling. If the postgres user can
replace the log file with a symbolic link, vulnerable scripts will append to
the link target, as root, at the next postmaster start. (The link can point
anywhere in the filesystem, and the target need not yet exist.) Some scripts
also chown() or chmod() the log file, and one can attack them to redirect
those calls to arbitrary targets. By writing a file like /etc/ld.so.preload,
one can exploit this to run arbitrary code as root. Some distribution scripts
make this simple to exploit, while others leave only a narrow race window.
Scripts PostgreSQL distributes in contrib/start-scripts are vulnerable.
PostgreSQL will use CVE-2017-12172 for the contrib/start-scripts
vulnerability; if you maintain your own vulnerable scripts, I recommend
acquiring your own CVE.

The problem arises when root alters files located in a directory writable to a
non-root user. (If any parent directory is writable to a non-root user, that
triggers the same problem.) I know of three ways to close the vulnerability:

1. Perform the filesystem operations as the directory owner UID, not as root.
   Prefer this method. The contrib/start-scripts fix uses it; the fix simply
   moves log creation inside the "su postgres" command.

2. Create the file with open(filename, O_EXCL | O_CREAT, mode) only. Do not
   pass the filename to any other mutating system call, particularly not
   chown() or chmod(). (It's fine to issue fchown() or fchmod().) This has
   no known advantage over method (1) and is inconvenient in shell scripts.
   Under these conditions, there is no safe, portable way to open for writing
   an already-existing file. This method does not suffice if any higher
   parent directory is writable to a non-root user.

3. Move the file to a root-owned directory hierarchy. This is good for new
   designs, but it may be too disruptive in a security patch.

Note that testing for the presence of symlinks is insufficient, because it
invariably suffers from a TOCTOU race condition.

This vulnerability already saw limited public recognition in
https://security-tracker.debian.org/tracker/CVE-2016-1255. The fix linked
from that page thwarted the most potent attacks, but I think it was
incomplete. After the sysopen() and before the chown(), an attacker can
replace the chown() target with a symlink to an important root-owned file.

I have attached the contrib/start-scripts fix for your reference. This fix is
bolder than I would use in a distribution. It creates postgres-owned log
files instead of root-owned files, and users must manually chown or chmod
their old log files once when installing the fix. That is okay since
installing new contrib/start-scripts is manual. An automated distribution
update would likely choose to handle the ownership change transparently.

Disclosure Timeline:
- 2017-11-06: Starting on this date, I welcome pushing fixes to public source
  repositories. The contrib/start-scripts fix will appear in the PostgreSQL
  git repository. Please continue to refrain from making announcements or
  otherwise discussing the fixes in public.
- 2017-11-09: I welcome free discussion of any aspect of this vulnerability,
  in any forum. I encourage you to make any announcements on this date.

Please reply in the next week or two with answers to the following:
1. Which PostgreSQL distribution(s) do you handle?
2. Link to the source repository containing your start scripts, if public.
3. Are your scripts vulnerable?
4. If yes, do you plan to fix your scripts the week of 2017-11-09?

I encourage you to discuss implementation of your fix on this thread. When
you have a fix ready, I recommend posting the patch to this thread. This
allows us all to review and look for holes.

If you have questions or information that you prefer not to share with the
entire -packagers community, feel free to start a thread on
<email address hidden> to discuss the matter alone with the security team.

If you credit issue reporters, please credit Antoine Scemama of Brainloop. (I
did that when requesting the contrib/start-scripts CVE.)

Thanks,
nm

---

Christoph replied for Debian:
Re: Noah Misch 2017-10-10 <email address hidden>
> The problem arises when root alters files located in a directory writable to a
> non-root user. (If any parent directory is writable to a non-root user, that
> triggers the same problem.) I know of three ways to close the vulnerability:
>
> 1. Perform the filesystem operations as the directory owner UID, not as root.
> 2. Create the file with open(filename, O_EXCL | O_CREAT, mode) only.
> 3. Move the file to a root-owned directory hierarchy.

4. Use lchown()

> This vulnerability already saw limited public recognition in
> https://security-tracker.debian.org/tracker/CVE-2016-1255. The fix linked
> from that page thwarted the most potent attacks, but I think it was
> incomplete. After the sysopen() and before the chown(), an attacker can
> replace the chown() target with a symlink to an important root-owned file.

Direct link to patch in question:
https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/commit/?id=c8989206ec360f199400c74f129f7b4cb878c1ee

Thanks for the analysis there. When patching it for CVE-2016-1255, I
was mostly thinking of clusters owned by random non-postgres users
(postgresql-common can handle clusters owned by arbitrary users) which
would likely have uid >= 500, so chown() wouldn't be invoked, but of
course the core problem is that it presents a postgres->root privilege
escalation.

I think the simple fix there is

use POSIX qw(lchown);
...
lchown $info{'owneruid'}, $g, $info{'logfile'} if (defined $g);

pg_ctlcluster has unfortunately grown over the years to be horrible
spaghetti code that changes between invoking user (likely root) and
the cluster owner back and forth a few times. I've always meant to
refactor it (but there's only 24h a day). Hopefully there's not more
problems of that sort in there.

> Please reply in the next week or two with answers to the following:
> 1. Which PostgreSQL distribution(s) do you handle?

Debian, apt.postgresql.org, indirectly also Ubuntu.

> 2. Link to the source repository containing your start scripts, if public.

The full pg_ctlcluster file is there, if anyone wants to have a look:
https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/tree/pg_ctlcluster

Another source of problems might be pg_createcluster.

> 3. Are your scripts vulnerable?

Yes.

> 4. If yes, do you plan to fix your scripts the week of 2017-11-09?

Yes.

---

Related and on top there is [1], fixed by [2] which we would be open in Trusty + Xenial (but fixed in zesty).

[1]: https://security-tracker.debian.org/tracker/CVE-2016-1255.
[2]: https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/commit/?id=c8989206ec360f199400c74f129f7b4cb878c1ee

[Christoph Berg (myon)]

Planned Debian fix: https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/commit/?id=8b4d0a889a8287181c4bdf46462db9b737a6e25d

Christoph

[Christian Ehrhardt  (paelzer)]

Thanks Christoph!
I already reviewed it from your mail to pgsql-packagers a few minutes ago and it LGTM.
I also already pinged our security folks to check if that is enough or if they'd need more to go for it now.

[Marc Deslauriers (mdeslaur)]

Did this get its own CVE number yet?

[Christian Ehrhardt  (paelzer)]

No expert in CVEs, but isn't it:
CVE-2016-1255 => https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/commit/?id=c8989206ec360f19
CVE-2017-12172 => https://anonscm.debian.org/cgit/pkg-postgresql/postgresql-common.git/commit/?id=8b4d0a889a82871

[Marc Deslauriers (mdeslaur)]

CVE-2017-12172 is for PostgreSQL's init scripts.
The Debian script is a different script containing a similar issue, it should get its own CVE number.

[Christoph Berg (myon)]

Re: Marc Deslauriers 2017-11-08 <email address hidden>
> CVE-2017-12172 is for PostgreSQL's init scripts.
> The Debian script is a different script containing a similar issue, it should get its own CVE number.

Just got one from Debian: CVE-2017-8806

Will update Git in a few minutes.

Christoph

[Launchpad Janitor (janitor)]

This bug was fixed in the package postgresql-common - 173ubuntu0.1

---------------
postgresql-common (173ubuntu0.1) xenial-security; urgency=medium

  * SECURITY UPDATE: symlink attack vulnerability
    - drop privileges when creating log file in pg_ctlcluster.
    - c8989206ec360f199400c74f129f7b4cb878c1ee
    - CVE-2016-1255
  * SECURITY UPDATE: symlink attack vulnerability in init/helper scripts
    (LP: #1727209)
    - use lchown instead of chown in pg_createcluster, pg_ctlcluster,
      pg_upgradecluster.
    - 8b4d0a889a8287181c4bdf46462db9b737a6e25d
    - No CVE number

 -- Marc Deslauriers <email address hidden> Wed, 08 Nov 2017 08:17:29 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package postgresql-common - 154ubuntu1.1

---------------
postgresql-common (154ubuntu1.1) trusty-security; urgency=medium

  * SECURITY UPDATE: symlink attack vulnerability
    - drop privileges when creating log file in pg_ctlcluster.
    - c8989206ec360f199400c74f129f7b4cb878c1ee
    - CVE-2016-1255
  * SECURITY UPDATE: symlink attack vulnerability in init/helper scripts
    (LP: #1727209)
    - use lchown instead of chown in pg_createcluster, pg_ctlcluster,
      pg_upgradecluster.
    - 8b4d0a889a8287181c4bdf46462db9b737a6e25d
    - No CVE number

 -- Marc Deslauriers <email address hidden> Wed, 08 Nov 2017 08:22:48 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package postgresql-common - 184ubuntu1.1

---------------
postgresql-common (184ubuntu1.1) artful-security; urgency=medium

  * SECURITY UPDATE: symlink attack vulnerability in init/helper scripts
    (LP: #1727209)
    - use lchown instead of chown in pg_createcluster, pg_ctlcluster,
      pg_upgradecluster.
    - 8b4d0a889a8287181c4bdf46462db9b737a6e25d
    - No CVE number

 -- Marc Deslauriers <email address hidden> Wed, 08 Nov 2017 08:06:51 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package postgresql-common - 179ubuntu0.1

---------------
postgresql-common (179ubuntu0.1) zesty-security; urgency=medium

  * SECURITY UPDATE: symlink attack vulnerability in init/helper scripts
    (LP: #1727209)
    - use lchown instead of chown in pg_createcluster, pg_ctlcluster,
      pg_upgradecluster.
    - 8b4d0a889a8287181c4bdf46462db9b737a6e25d
    - No CVE number

 -- Marc Deslauriers <email address hidden> Wed, 08 Nov 2017 08:12:07 -0500