Logs.var.log.mysql.error.log.txt contains usernames and passwords

[BCB]

MySQL has some logic for ensuring passwords aren't written to the logs, detailed at https://dev.mysql.com/doc/refman/5.7/en/password-logging.html (passwords are rewritten before they are logged). However, a failed grant statement is written unaltered to the error log, bypassing the password rewriting logic.

[Impact]
Ubuntu's bug reporting system will suggest uploading the error log to a bug report. This can lead to user credentials written in plain text in public bug reports.

[Test case]
(note/todo: I had a simpler test for this, but can't find the exact syntax for it)
* Add the following to the server config:
plugin-load=validate_password.so
validate-password=FORCE_PLUS_PERMANENT
and restart the server
* Log in and run GRANT ALL ON *.* TO 'user'@'localhost' IDENTIFIED BY '123';
* Observe statement failing because it doesn't follow password validation rules
* Run "ubuntu-bug mysql-server"
* Choose "View Report"
* Search for "123"

Expected behavior:
Password is scrambled or otherwise not written to the apport report

Actual behavior:
The entire failed grant statement is written to the apport report

[Regression Potential]
The fix replaces all lines in the log that contain any of the terms mentioned on the password-logging site, so it will rewrite more lines than strictly necessary, potentially making debugging harder.

[Original description]
Your automated bug reports are posting Logs.var.log.mysql.error.log.txt in clear text. These logs may contain PII as well as user credentials.

[Seth Arnold (seth-arnold)]

Thanks for the report; I found two instances in our bugs with the following messages:

[Warning] Did not write failed 'GRANT ALL PRIVILEGES ON `phpmyadmin`.* TO `phpmyadmin`@'localhost' IDENTIFIED BY 'password'' into binary log while granting/revoking privileges in databases.
[Warning] Did not write failed 'grant all privileges on wordpress.* to wordpressuser@localhost identified by "password"' into binary log while granting/revoking privileges in databases.

(I've replaced the passwords with "password".)

Are there other instances of passwords or usernames that go into this log?

Thanks

[Lars Tangvald (lars-tangvald)]

With MySQL 5.7, if you run mysqld --initialize yourself, for a new database, it will generate a random root password and put it in the log (users will be required to change this password on first login), but the normal package installation will either set unix socket authentication or a password chosen during configuration.

[Lars Tangvald (lars-tangvald)]

Verified on MySQL 5.7; Password logging should follow the rules specified on https://dev.mysql.com/doc/refman/5.7/en/password-logging.html, but it seems it's not caught correctly when the grant statement fails. I'll send this upstream.

The error log will contain usernames for failed logins, but I can't think of much else in the way of PII it would contain.

[BCB (fdajkddcek)]

I just checked the aprox 39 mysql-5.7 bug reports with xxx.error.log.txt
attached and did not see anymore.

I locked my report as my user name and a few random inserts were appeared
in the log.
"ssunderlin"
"Stephen Sunderlin"

If you can remove/replace/ or just delete that error log you can make my
report public again.

https://i255940206.restricted.launchpadlibrarian.net/255940206/Logs.var.log.mysql.error.log.txt?token=vjZn2FHWMBn7TDH8QfsM7fBf9Xccw9sv

Thank you.

On Tue, Apr 26, 2016 at 4:28 AM, Lars Tangvald <email address hidden>
wrote:

> Verified on MySQL 5.7; Password logging should follow the rules
> specified on https://dev.mysql.com/doc/refman/5.7/en/password-
> logging.html, but it seems it's not caught correctly when the grant
> statement fails. I'll send this upstream.
>
> The error log will contain usernames for failed logins, but I can't
> think of much else in the way of PII it would contain.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1574458
>
> Title:
> Logs.var.log.mysql.error.log.txt contains usernames and passwords
>
> Status in mariadb-10.0 package in Ubuntu:
> New
> Status in mariadb-5.5 package in Ubuntu:
> New
> Status in mysql-5.5 package in Ubuntu:
> New
> Status in mysql-5.6 package in Ubuntu:
> New
> Status in mysql-5.7 package in Ubuntu:
> New
>
> Bug description:
> Your automated bug reports are posting
> Logs.var.log.mysql.error.log.txt in clear text. These logs may
> contain PII as well as user credentials.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/mariadb-10.0/+bug/1574458/+subscriptions
>

[BCB (fdajkddcek)]

I just checked the aprox 39 mysql-5.7 bug reports with xxx.error.log.txt
attached and did not see anymore.

I locked my report as my user name and a few random inserts were appeared
in the log.
"ssunderlin"
"Stephen Sunderlin"

If you can remove/replace/ or just delete that error log you can make my
report public again.

https://i255940206.restricted.launchpadlibrarian.net/255940206/Logs.var.log.mysql.error.log.txt?token=vjZn2FHWMBn7TDH8QfsM7fBf9Xccw9sv

Thank you.

On Mon, Apr 25, 2016 at 11:08 PM, Seth Arnold <email address hidden>
wrote:

> Thanks for the report; I found two instances in our bugs with the
> following messages:
>
> [Warning] Did not write failed 'GRANT ALL PRIVILEGES ON `phpmyadmin`.* TO
> `phpmyadmin`@'localhost' IDENTIFIED BY 'password'' into binary log while
> granting/revoking privileges in databases.
> [Warning] Did not write failed 'grant all privileges on wordpress.* to
> wordpressuser@localhost identified by "password"' into binary log while
> granting/revoking privileges in databases.
>
> (I've replaced the passwords with "password".)
>
> Are there other instances of passwords or usernames that go into this
> log?
>
> Thanks
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1574458
>
> Title:
> Logs.var.log.mysql.error.log.txt contains usernames and passwords
>
> Status in mariadb-10.0 package in Ubuntu:
> New
> Status in mariadb-5.5 package in Ubuntu:
> New
> Status in mysql-5.5 package in Ubuntu:
> New
> Status in mysql-5.6 package in Ubuntu:
> New
> Status in mysql-5.7 package in Ubuntu:
> New
>
> Bug description:
> Your automated bug reports are posting
> Logs.var.log.mysql.error.log.txt in clear text. These logs may
> contain PII as well as user credentials.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/mariadb-10.0/+bug/1574458/+subscriptions
>

[Seth Arnold (seth-arnold)]

BCB, it's a bit ironic; because the bug is private, I can't get to it to delete the attachment; and because the url is now visible, it's easily downloaded even without privileges.

The right column of the launchpad interface, near the bottom of the column, is a box with all the attachments; hit 'edit' next to Logs.var... and the page that shows up will have a 'delete' option of some sort.

Thanks

[BCB (fdajkddcek)]

well that was easy. Thanks.

On Wed, Apr 27, 2016 at 6:00 PM, Seth Arnold <email address hidden>
wrote:

> BCB, it's a bit ironic; because the bug is private, I can't get to it to
> delete the attachment; and because the url is now visible, it's easily
> downloaded even without privileges.
>
> The right column of the launchpad interface, near the bottom of the
> column, is a box with all the attachments; hit 'edit' next to
> Logs.var... and the page that shows up will have a 'delete' option of
> some sort.
>
> Thanks
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1574458
>
> Title:
> Logs.var.log.mysql.error.log.txt contains usernames and passwords
>
> Status in mariadb-10.0 package in Ubuntu:
> New
> Status in mariadb-5.5 package in Ubuntu:
> New
> Status in mysql-5.5 package in Ubuntu:
> New
> Status in mysql-5.6 package in Ubuntu:
> New
> Status in mysql-5.7 package in Ubuntu:
> New
>
> Bug description:
> Your automated bug reports are posting
> Logs.var.log.mysql.error.log.txt in clear text. These logs may
> contain PII as well as user credentials.
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/mariadb-10.0/+bug/1574458/+subscriptions
>

[Lars Tangvald (lars-tangvald)]

Until this is fixed upstream, as a mitigation effort we'll change the apport hook script to filter out (with a notice that it's been done) any lines in the error log that might contain private info.

[Robie Basak (racb)]

Password logging aside, there's an open question about whether logging usernames is acceptable. I'd say it is, since usernames are generally actually webapps and thus are useful debugging information. Users are already prompted to see the report before sending it, and actively choose to send it. Dropping usernames would go in the direction of crippling useful reporting. Since users already choose to send reports, those bothered by this could just not send them.

Comments appreciated, but I'd look to the Ubuntu security team to make a final decision.

[Seth Arnold (seth-arnold)]

Since the usernames I've seen so far were all from applications I never really thought about it being a breach of privacy to include those in the logs. Filtering usernames seems like a good idea to me but I'm bothered way less about it than e.g. passwords.

Thanks

[Robie Basak (racb)]

The apport hook is no longer sending passwords as of 5.7.13-0ubuntu1 in Yakkety. Adding a task for Xenial.

I'm preparing an SRU for Xenial that will include this fix.

Seth, alternatively do you want it in the security pocket? Though that would mess with my planned SRU a little. We can arrange something if needed.

[Robie Basak (racb)]

Uploaded mysql-5.7 to Xenial. SRU team: you may find it easier to review if you verify my uploaded delta is the same as https://git.launchpad.net/~racb/ubuntu/+source/mysql-5.7/log/?h=mysql-5.7/ubuntu/xenial and then use that.

[Adam Conrad (adconrad)]

Hello BCB, or anyone else affected,

Accepted mysql-5.7 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.7/5.7.12-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Christian Ehrhardt  (paelzer)]

Verified that my testcase works on Xenial container, upgraded to proposed and checked again.
Log now contains:
--- Line containing protected term grant stripped from log by apport hook. Ref. Launchpad bug #1574458

So, verified

[Robie Basak (racb)]

For mysql-5.7, I verified actual behaviour in xenial, and then upgraded to xenial-proposed. Now the apport report instead says "--- Line containing protected term grant stripped from log by apport hook. Ref. Launchpad bug #1574458" instead of revealing the password as expected.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.7 - 5.7.13-0ubuntu0.16.04.2

---------------
mysql-5.7 (5.7.13-0ubuntu0.16.04.2) xenial-security; urgency=medium

  * SECURITY UPDATE: Update to 5.7.13 to fix security issues (LP: #1604796)
    - http://www.oracle.com/technetwork/security-advisory/cpujul2016-2881720.html
    - CVE-2016-3424
    - CVE-2016-3459
    - CVE-2016-3477
    - CVE-2016-3486
    - CVE-2016-3501
    - CVE-2016-3518
    - CVE-2016-3521
    - CVE-2016-3588
    - CVE-2016-3614
    - CVE-2016-3615
    - CVE-2016-5436
    - CVE-2016-5437
    - CVE-2016-5439
    - CVE-2016-5440
    - CVE-2016-5441
    - CVE-2016-5442
    - CVE-2016-5443
  * debian/patches/mysql-export-scramble.patch: removed, upstream.

 -- Marc Deslauriers <email address hidden> Wed, 20 Jul 2016 08:44:25 -0400

[Nish Aravamudan (nacc)]

mysql-5.5, mysql-5.6, mariadb-5.5 are all not in 16.04.

[Nish Aravamudan (nacc)]

@racb, @lars-tangvald:

I think I have set the various tasks correctly now. But I'm not sure if anyone has verified if the issue is present in Trusty (older versions of the MySQL packages) or in MariaDB's packaging.

[Christian Ehrhardt  (paelzer)]

Checked 5.6 in Trusty it does not have the related apport hook that adds error.log
5.5 in Trusty has that hook so it generally applies there.

[Nish Aravamudan (nacc)]

I believe the 4 remaining tasks are now confirmed (my check was looking at the source package to see if the apport hook existed, and if it does whether it had the same fix as src:mysql-5.7 to filter the logs (they all filter the conf files already)).

[Andreas Hasenack (ahasenack)]

The mysql-5.5 trusty task is still open. Lars, can you pick that one up perpahs?

Mariadb is in universe and probably won't get an update.

[Otto Kekäläinen (otto)]

MariaDB hasn't been using passwords by default for a very long time, and most likely this bug never affected the MariaDB versions in Debian nor Ubuntu.