data corruption on arm64 and ppc64el

Bug #1427406 reported by dann frazier on 2015-03-02
20
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL Server
Unknown
Unknown
mariadb-10.0 (Debian)
Fix Released
Unknown
mariadb-10.0 (Ubuntu)
Undecided
Unassigned
Trusty
Undecided
Unassigned
Utopic
Undecided
Unassigned
Vivid
Undecided
Unassigned
Wily
Undecided
Unassigned
mariadb-5.5 (Ubuntu)
Undecided
Unassigned
Trusty
Critical
Unassigned
Utopic
Undecided
Unassigned
mysql-5.5 (Ubuntu)
Undecided
Unassigned
Trusty
Critical
Unassigned
Utopic
Critical
Unassigned
mysql-5.6 (Ubuntu)
Critical
Unassigned
Trusty
Critical
Unassigned
Utopic
Critical
Unassigned
Vivid
Critical
Unassigned
Wily
Critical
Unassigned

Bug Description

I've verified that the following MariaDB issue also impacts MySQL:
  https://mariadb.atlassian.net/browse/MDEV-6615

[Impact]
Data corruption on arm64 and ppc64el

[Test Case]
Reproducibility varies across SoCs but, at least on one SoC, it reliably fails within seconds with these sysbench commands:

sysbench --num-threads=128 --max-requests=0 --max-time=1000 --test=oltp prepare
sysbench --num-threads=128 --max-requests=0 --max-time=1000 --test=oltp run

On failure, mysql will crash and sysbench will report a connection loss to the database.

[Regression Risk]
The fix touch some low level locking code, so there is always the risk that it introduces a problem on other platforms. However, this fix has been included in MariaDB since the 10.0.13 release in Aug, 2014 so it has had some real world testing.

dann frazier (dannf) wrote :
tags: added: patch
dann frazier (dannf) wrote :
Changed in mysql-5.6 (Ubuntu Trusty):
status: New → Confirmed
importance: Undecided → Critical
dann frazier (dannf) wrote :

Apparently the fix I've proposed here caused regressions in MariaDB and was reworked:
  https://mariadb.atlassian.net/browse/MDEV-7026
Unfortunately, the reworking reintroduces the problem on arm64:
  https://mariadb.atlassian.net/browse/MDEV-7658

dann frazier (dannf) on 2015-03-04
Changed in mariadb-5.5 (Ubuntu Trusty):
importance: Undecided → Critical
status: New → Confirmed
Changed in mysql-5.6 (Ubuntu Vivid):
assignee: nobody → recep (rbasak)
Changed in mysql-5.6 (Ubuntu Vivid):
assignee: recep (rbasak) → Robie Basak (racb)
Robie Basak (racb) wrote :

This one has been assigned to me, but I don't really have any action to take unless someone can present a patch. If it's MySQL upstream related, then trying ryeng on #debian-mysql on OFTC may be helpful.

Changed in mysql-5.6 (Ubuntu Vivid):
assignee: Robie Basak (racb) → nobody
dann frazier (dannf) on 2015-03-27
no longer affects: mariadb-5.5 (Ubuntu Vivid)
dann frazier (dannf) wrote :

I've prepared a git branch against Debian's mysql-5.6 tree w/ fixes that pass my test on ARM64:
  https://github.com/dannf/mysql-5.6.git

I should note that this was really a mechanical port w/ blackbox testing. I'm not an expert in the semantics at this level - and the code here had diverged quite a bit between MariaDB and MySQL.

I've verified this builds on armhf, i386 and amd64 (including whatever build-time test suites run), but I haven't done any further testing on those architectures.

Robie Basak (racb) wrote :

Oleg put my concerns very well in the IRC meeting just now:

17:12 <strikov> gaughen: patch is related to reworking locking primitives which means that the fact that it fixes exact issue doesn't mean that it won't break anything else (unexpectedly).

17:12 <strikov> gaughen: it means that we either need to have good enough test suite or (better) wait for upstreaming the change

dann frazier (dannf) on 2015-05-29
no longer affects: mariadb-5.5 (Ubuntu Wily)
dann frazier (dannf) wrote :

Fixes for mariadb-5.5 and mariadb-10.0 are now available upstream:
  https://mariadb.atlassian.net/browse/MDEV-7658

Changed in mariadb-10.0 (Debian):
status: Unknown → New
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.6 - 5.6.24-0ubuntu3

---------------
mysql-5.6 (5.6.24-0ubuntu3) wily; urgency=medium

  * d/p/MDEV-{6450,6483,7026,7658}-*: Introduce full memory barrier support
    for arm64 and ppc64el to fix corruption/hang issues (LP: #1427406).

 -- dann frazier <email address hidden> Tue, 09 Jun 2015 11:47:35 -0600

Changed in mysql-5.6 (Ubuntu Wily):
status: Confirmed → Fix Released
dann frazier (dannf) on 2015-06-11
Changed in mysql-5.6 (Ubuntu Vivid):
status: Confirmed → In Progress
dann frazier (dannf) on 2015-06-12
Changed in mysql-5.6 (Ubuntu Utopic):
status: New → In Progress
dann frazier (dannf) on 2015-06-12
Changed in mysql-5.6 (Ubuntu Trusty):
status: Confirmed → In Progress
dann frazier (dannf) on 2015-06-15
Changed in mariadb-5.5 (Ubuntu Utopic):
status: New → In Progress
status: In Progress → New
Changed in mysql-5.5 (Ubuntu Utopic):
status: New → In Progress
description: updated
summary: - data corruption on arm64
+ data corruption on arm64 and ppc64el
dann frazier (dannf) on 2015-06-15
Changed in mysql-5.5 (Ubuntu Trusty):
status: New → In Progress
Changed in mysql-5.6 (Ubuntu Utopic):
importance: Undecided → Critical
Changed in mysql-5.5 (Ubuntu Trusty):
importance: Undecided → Critical
Changed in mysql-5.5 (Ubuntu Utopic):
importance: Undecided → Critical
Changed in mariadb-10.0 (Debian):
status: New → Fix Released
Chris J Arges (arges) wrote :

Please re-upload without also including the orig tarball (since this doesn't change). You can verify this in your .changes file before uploading.

Chris J Arges (arges) wrote :

Actually these looked ok, go ahead and re-upload. Sorry about the noise.

On Wed, Jul 1, 2015 at 9:33 AM, Chris J Arges
<email address hidden> wrote:
> Actually these looked ok, go ahead and re-upload. Sorry about the noise.

 Done.

Hello dann, or anyone else affected,

Accepted mysql-5.6 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.6/5.6.19-0ubuntu0.14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.6 (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Changed in mysql-5.6 (Ubuntu Utopic):
status: In Progress → Fix Committed
Chris J Arges (arges) wrote :

Hello dann, or anyone else affected,

Accepted mysql-5.6 into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.6/5.6.19-1~exp1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.6 (Ubuntu Vivid):
status: In Progress → Fix Committed
Chris J Arges (arges) wrote :

Hello dann, or anyone else affected,

Accepted mysql-5.6 into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.6/5.6.24-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.5 (Ubuntu Trusty):
status: In Progress → Fix Committed
Chris J Arges (arges) wrote :

Hello dann, or anyone else affected,

Accepted mysql-5.5 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.5/5.5.43-0ubuntu0.14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of mysql-5.6 from vivid-proposed was performed and bug 1471486 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and tag 1471486 "bot-stop-nagging". Thanks!

tags: added: verification-failed
Robie Basak (racb) wrote :

I don't think that bug is sufficient to block SRU verification without further reports or confirmation, since nothing I know of deletes /etc/mysql/my.cnf.fallback so I think it has likely been deleted manually on the reporter's system. Removing verification-failed since I don't want this SRU cancelled, but perhaps we should wait a little longer before we confirm verification-done here.

tags: removed: verification-failed
dann frazier (dannf) wrote :

I've verified mysql-5.6/vivid.

dann frazier (dannf) wrote :

Also verified mysql-5.6/utopic.

Hello dann, or anyone else affected,

Accepted mysql-5.5 into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.5/5.5.43-0ubuntu0.14.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.5 (Ubuntu Utopic):
status: In Progress → Fix Committed
dann frazier (dannf) wrote :

mysql-5.5/trusty verified.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mariadb-10.0 - 10.0.20-0ubuntu0.15.04.1

---------------
mariadb-10.0 (10.0.20-0ubuntu0.15.04.1) vivid-security; urgency=low

  * SECURITY UPDATE: Update to 10.0.20 (via .18 and .19) fixes security issues:
    - CVE-2015-3152: Client command line option --ssl-verify-server-cert (and
      MYSQL_OPT_SSL_VERIFY_SERVER_CERT option of the client API) when used
      together with --ssl will ensure that the established connection is
      SSL-encrypted and the MariaDB server has a valid certificate.
      (LP: #1464895)
    - CVE-2014-8964: bundled PCRE contained heap-based buffer overflow
      vulnerability that allowed the server to crash or have other unspecified
      impact via a crafted regular expression made possible with the
      REGEXP_SUBSTR function (MDEV-8006).
    - CVE-2015-0501
    - CVE-2015-2571
    - CVE-2015-0505
    - CVE-2015-0499
    (LP: #1451677)
  * New release includes fix for memory corruption on arm64 (LP: #1427406)
  * Upstream also includes lots of line ending changes (from CRLF -> LF)

 -- Otto Kekäläinen <email address hidden> Fri, 03 Jul 2015 17:39:42 +0300

Changed in mariadb-10.0 (Ubuntu Vivid):
status: New → Fix Released
dann frazier (dannf) wrote :

mysql-5.5/utopic verified.

The last remaining mysql update is mysql-5.6/trusty which I had to reupload to fix a FTBFS issue unrelated to this update (LP: #1419262).

Changed in mariadb-10.0 (Ubuntu Utopic):
status: New → Invalid
Changed in mariadb-10.0 (Ubuntu Trusty):
status: New → Invalid
Changed in mariadb-10.0 (Ubuntu Wily):
status: New → Triaged
dann frazier (dannf) wrote :

Fixed by inclusion of micro release 5.5.44.

Changed in mariadb-5.5 (Ubuntu Trusty):
status: Confirmed → Fix Released
Changed in mariadb-5.5 (Ubuntu Utopic):
status: New → Fix Released
dann frazier (dannf) on 2015-07-15
Changed in mariadb-5.5 (Ubuntu):
status: New → Fix Released
Changed in mysql-5.5 (Ubuntu):
status: New → Invalid
Changed in mariadb-5.5 (Ubuntu):
status: Fix Released → Invalid
Marc Deslauriers (mdeslaur) wrote :

I'm sorry, but I don't see how we can carry these patches in the Ubuntu MySQL packages.

Since Oracle no longer discloses details of their MySQL security vulnerabilities, we have no other choice than to upgrade to their latest upstream version when they publish vulnerability details. This implies that we are relying on their internal testing for each release.

Adding these patches will prevent us from being able to update to a new MySQL version as soon as it is available as the patches will require porting and testing. Diverging from upstream also means we aren't actually running the code that has passed their testing.

On top of that, there is no clear indication these patches will actually end up in the next MySQL version, as there is a contributor agreement issue.

Due to these reasons, I have to object to carrying these patches in Ubuntu.

Marc Deslauriers (mdeslaur) wrote :

After discussing my concerns with dannf and rbasak on irc, we have come to the following:

- We are aware of this issue, and are making reasonable efforts to ensure that users are not impacted. However, as highlighted by point (5) below, there may be circumstances in which we have to regress this fix in order to ensure a security patch will be applied.

- dannf's team has commitment from Oracle to fix broken archs in some future version
- patches are only applied to affected archs
- MySQL updates are usually available before Oracle's quarterly security notice is published
(Quarterly security notice can be viewed here: http://www.oracle.com/technetwork/topics/security/alerts-086861.html )

1) dannf's team will figure out how to be notified of a new micro release
2) dannf's team will update a PPA w/ the new micro release before quarterly security notice (~6 weeks generally, but can be immediate)
* In the event where a new MySQL version is published at the same time as the quarterly security notice, dannf's team will update the PPA with updated patches no more than 2 working days after publication
3) dannf's team will test the updated PPA on arm64 (ppc64el will not be explicitly tested)

4) security team will pull updated patches from ppa when preparing security updates. Security updates will not be tested on affected architectures

5) Security updates will not be held back if there is a problem with "the patch". If necessary, arm64/ppc64el users will be regressed by the security team issuing an update with the patch dropped. If the patch is dropped, a notice will be added to the Ubuntu Security Notice.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.5 - 5.5.44-0ubuntu0.14.10.1

---------------
mysql-5.5 (5.5.44-0ubuntu0.14.10.1) utopic-security; urgency=medium

  * SECURITY UPDATE: Update to 5.5.44 to fix security issues (LP: #1475294)
    - http://www.oracle.com/technetwork/topics/security/cpujul2015-2367936.html
    - CVE-2015-2582
    - CVE-2015-2620
    - CVE-2015-2643
    - CVE-2015-2648
    - CVE-2015-4737
    - CVE-2015-4752
    - CVE-2015-4757

 -- Marc Deslauriers <email address hidden> Thu, 16 Jul 2015 11:52:48 -0400

Changed in mysql-5.5 (Ubuntu Utopic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.5 - 5.5.44-0ubuntu0.14.04.1

---------------
mysql-5.5 (5.5.44-0ubuntu0.14.04.1) trusty-security; urgency=medium

  * SECURITY UPDATE: Update to 5.5.44 to fix security issues (LP: #1475294)
    - http://www.oracle.com/technetwork/topics/security/cpujul2015-2367936.html
    - CVE-2015-2582
    - CVE-2015-2620
    - CVE-2015-2643
    - CVE-2015-2648
    - CVE-2015-4737
    - CVE-2015-4752
    - CVE-2015-4757

 -- Marc Deslauriers <email address hidden> Thu, 16 Jul 2015 13:36:50 -0400

Changed in mysql-5.5 (Ubuntu Trusty):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mariadb-10.0 - 10.0.20-0ubuntu0.15.04.1

---------------
mariadb-10.0 (10.0.20-0ubuntu0.15.04.1) vivid-security; urgency=low

  * SECURITY UPDATE: Update to 10.0.20 (via .18 and .19) fixes security issues:
    - CVE-2015-3152: Client command line option --ssl-verify-server-cert (and
      MYSQL_OPT_SSL_VERIFY_SERVER_CERT option of the client API) when used
      together with --ssl will ensure that the established connection is
      SSL-encrypted and the MariaDB server has a valid certificate.
      (LP: #1464895)
    - CVE-2014-8964: bundled PCRE contained heap-based buffer overflow
      vulnerability that allowed the server to crash or have other unspecified
      impact via a crafted regular expression made possible with the
      REGEXP_SUBSTR function (MDEV-8006).
    - CVE-2015-0501
    - CVE-2015-2571
    - CVE-2015-0505
    - CVE-2015-0499
    (LP: #1451677)
  * New release includes fix for memory corruption on arm64 (LP: #1427406)
  * Upstream also includes lots of line ending changes (from CRLF -> LF)

 -- Otto Kekäläinen <email address hidden> Fri, 03 Jul 2015 17:39:42 +0300

Changed in mariadb-10.0 (Ubuntu Wily):
status: Triaged → Fix Released
Rolf Leggewie (r0lf) wrote :

utopic has seen the end of its life and is no longer receiving any updates. Marking the utopic task for this ticket as "Won't Fix".

Changed in mysql-5.6 (Ubuntu Utopic):
status: Fix Committed → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.