data corruption on arm64 and ppc64el

[dann frazier]

I've verified that the following MariaDB issue also impacts MySQL:
  https://mariadb.atlassian.net/browse/MDEV-6615

[Impact]
Data corruption on arm64 and ppc64el

[Test Case]
Reproducibility varies across SoCs but, at least on one SoC, it reliably fails within seconds with these sysbench commands:

sysbench --num-threads=128 --max-requests=0 --max-time=1000 --test=oltp prepare
sysbench --num-threads=128 --max-requests=0 --max-time=1000 --test=oltp run

On failure, mysql will crash and sysbench will report a connection loss to the database.

[Regression Risk]
The fix touch some low level locking code, so there is always the risk that it introduces a problem on other platforms. However, this fix has been included in MariaDB since the 10.0.13 release in Aug, 2014 so it has had some real world testing.

[dann frazier (dannf)]

Apparently the fix I've proposed here caused regressions in MariaDB and was reworked:
  https://mariadb.atlassian.net/browse/MDEV-7026
Unfortunately, the reworking reintroduces the problem on arm64:
  https://mariadb.atlassian.net/browse/MDEV-7658

[Robie Basak (racb)]

This one has been assigned to me, but I don't really have any action to take unless someone can present a patch. If it's MySQL upstream related, then trying ryeng on #debian-mysql on OFTC may be helpful.

[dann frazier (dannf)]

I've prepared a git branch against Debian's mysql-5.6 tree w/ fixes that pass my test on ARM64:
  https://github.com/dannf/mysql-5.6.git

I should note that this was really a mechanical port w/ blackbox testing. I'm not an expert in the semantics at this level - and the code here had diverged quite a bit between MariaDB and MySQL.

I've verified this builds on armhf, i386 and amd64 (including whatever build-time test suites run), but I haven't done any further testing on those architectures.

[Robie Basak (racb)]

Oleg put my concerns very well in the IRC meeting just now:

17:12 <strikov> gaughen: patch is related to reworking locking primitives which means that the fact that it fixes exact issue doesn't mean that it won't break anything else (unexpectedly).

17:12 <strikov> gaughen: it means that we either need to have good enough test suite or (better) wait for upstreaming the change

[dann frazier (dannf)]

Fixes for mariadb-5.5 and mariadb-10.0 are now available upstream:
  https://mariadb.atlassian.net/browse/MDEV-7658

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.6 - 5.6.24-0ubuntu3

---------------
mysql-5.6 (5.6.24-0ubuntu3) wily; urgency=medium

  * d/p/MDEV-{6450,6483,7026,7658}-*: Introduce full memory barrier support
    for arm64 and ppc64el to fix corruption/hang issues (LP: #1427406).

 -- dann frazier <email address hidden> Tue, 09 Jun 2015 11:47:35 -0600

[Chris J Arges (arges)]

Please re-upload without also including the orig tarball (since this doesn't change). You can verify this in your .changes file before uploading.

[Chris J Arges (arges)]

Actually these looked ok, go ahead and re-upload. Sorry about the noise.

[dann frazier (dannf)]

On Wed, Jul 1, 2015 at 9:33 AM, Chris J Arges
<email address hidden> wrote:
> Actually these looked ok, go ahead and re-upload. Sorry about the noise.

 Done.

[Chris J Arges (arges)]

Hello dann, or anyone else affected,

Accepted mysql-5.6 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.6/5.6.19-0ubuntu0.14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello dann, or anyone else affected,

Accepted mysql-5.6 into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.6/5.6.19-1~exp1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello dann, or anyone else affected,

Accepted mysql-5.6 into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.6/5.6.24-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello dann, or anyone else affected,

Accepted mysql-5.5 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.5/5.5.43-0ubuntu0.14.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ubuntu Foundations Team Bug Bot (crichton)]

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of mysql-5.6 from vivid-proposed was performed and bug 1471486 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and tag 1471486 "bot-stop-nagging". Thanks!

[Robie Basak (racb)]

I don't think that bug is sufficient to block SRU verification without further reports or confirmation, since nothing I know of deletes /etc/mysql/my.cnf.fallback so I think it has likely been deleted manually on the reporter's system. Removing verification-failed since I don't want this SRU cancelled, but perhaps we should wait a little longer before we confirm verification-done here.

[dann frazier (dannf)]

I've verified mysql-5.6/vivid.

[dann frazier (dannf)]

Also verified mysql-5.6/utopic.

[Chris J Arges (arges)]

Hello dann, or anyone else affected,

Accepted mysql-5.5 into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.5/5.5.43-0ubuntu0.14.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[dann frazier (dannf)]

mysql-5.5/trusty verified.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mariadb-10.0 - 10.0.20-0ubuntu0.15.04.1

---------------
mariadb-10.0 (10.0.20-0ubuntu0.15.04.1) vivid-security; urgency=low

  * SECURITY UPDATE: Update to 10.0.20 (via .18 and .19) fixes security issues:
    - CVE-2015-3152: Client command line option --ssl-verify-server-cert (and
      MYSQL_OPT_SSL_VERIFY_SERVER_CERT option of the client API) when used
      together with --ssl will ensure that the established connection is
      SSL-encrypted and the MariaDB server has a valid certificate.
      (LP: #1464895)
    - CVE-2014-8964: bundled PCRE contained heap-based buffer overflow
      vulnerability that allowed the server to crash or have other unspecified
      impact via a crafted regular expression made possible with the
      REGEXP_SUBSTR function (MDEV-8006).
    - CVE-2015-0501
    - CVE-2015-2571
    - CVE-2015-0505
    - CVE-2015-0499
    (LP: #1451677)
  * New release includes fix for memory corruption on arm64 (LP: #1427406)
  * Upstream also includes lots of line ending changes (from CRLF -> LF)

 -- Otto Kekäläinen <email address hidden> Fri, 03 Jul 2015 17:39:42 +0300

[dann frazier (dannf)]

mysql-5.5/utopic verified.

The last remaining mysql update is mysql-5.6/trusty which I had to reupload to fix a FTBFS issue unrelated to this update (LP: #1419262).

[dann frazier (dannf)]

Fixed by inclusion of micro release 5.5.44.

[Marc Deslauriers (mdeslaur)]

I'm sorry, but I don't see how we can carry these patches in the Ubuntu MySQL packages.

Since Oracle no longer discloses details of their MySQL security vulnerabilities, we have no other choice than to upgrade to their latest upstream version when they publish vulnerability details. This implies that we are relying on their internal testing for each release.

Adding these patches will prevent us from being able to update to a new MySQL version as soon as it is available as the patches will require porting and testing. Diverging from upstream also means we aren't actually running the code that has passed their testing.

On top of that, there is no clear indication these patches will actually end up in the next MySQL version, as there is a contributor agreement issue.

Due to these reasons, I have to object to carrying these patches in Ubuntu.

[Marc Deslauriers (mdeslaur)]

After discussing my concerns with dannf and rbasak on irc, we have come to the following:

- We are aware of this issue, and are making reasonable efforts to ensure that users are not impacted. However, as highlighted by point (5) below, there may be circumstances in which we have to regress this fix in order to ensure a security patch will be applied.

- dannf's team has commitment from Oracle to fix broken archs in some future version
- patches are only applied to affected archs
- MySQL updates are usually available before Oracle's quarterly security notice is published
(Quarterly security notice can be viewed here: http://www.oracle.com/technetwork/topics/security/alerts-086861.html )

1) dannf's team will figure out how to be notified of a new micro release
2) dannf's team will update a PPA w/ the new micro release before quarterly security notice (~6 weeks generally, but can be immediate)
* In the event where a new MySQL version is published at the same time as the quarterly security notice, dannf's team will update the PPA with updated patches no more than 2 working days after publication
3) dannf's team will test the updated PPA on arm64 (ppc64el will not be explicitly tested)

4) security team will pull updated patches from ppa when preparing security updates. Security updates will not be tested on affected architectures

5) Security updates will not be held back if there is a problem with "the patch". If necessary, arm64/ppc64el users will be regressed by the security team issuing an update with the patch dropped. If the patch is dropped, a notice will be added to the Ubuntu Security Notice.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.5 - 5.5.44-0ubuntu0.14.10.1

---------------
mysql-5.5 (5.5.44-0ubuntu0.14.10.1) utopic-security; urgency=medium

  * SECURITY UPDATE: Update to 5.5.44 to fix security issues (LP: #1475294)
    - http://www.oracle.com/technetwork/topics/security/cpujul2015-2367936.html
    - CVE-2015-2582
    - CVE-2015-2620
    - CVE-2015-2643
    - CVE-2015-2648
    - CVE-2015-4737
    - CVE-2015-4752
    - CVE-2015-4757

 -- Marc Deslauriers <email address hidden> Thu, 16 Jul 2015 11:52:48 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.5 - 5.5.44-0ubuntu0.14.04.1

---------------
mysql-5.5 (5.5.44-0ubuntu0.14.04.1) trusty-security; urgency=medium

  * SECURITY UPDATE: Update to 5.5.44 to fix security issues (LP: #1475294)
    - http://www.oracle.com/technetwork/topics/security/cpujul2015-2367936.html
    - CVE-2015-2582
    - CVE-2015-2620
    - CVE-2015-2643
    - CVE-2015-2648
    - CVE-2015-4737
    - CVE-2015-4752
    - CVE-2015-4757

 -- Marc Deslauriers <email address hidden> Thu, 16 Jul 2015 13:36:50 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package mariadb-10.0 - 10.0.20-0ubuntu0.15.04.1

---------------
mariadb-10.0 (10.0.20-0ubuntu0.15.04.1) vivid-security; urgency=low

  * SECURITY UPDATE: Update to 10.0.20 (via .18 and .19) fixes security issues:
    - CVE-2015-3152: Client command line option --ssl-verify-server-cert (and
      MYSQL_OPT_SSL_VERIFY_SERVER_CERT option of the client API) when used
      together with --ssl will ensure that the established connection is
      SSL-encrypted and the MariaDB server has a valid certificate.
      (LP: #1464895)
    - CVE-2014-8964: bundled PCRE contained heap-based buffer overflow
      vulnerability that allowed the server to crash or have other unspecified
      impact via a crafted regular expression made possible with the
      REGEXP_SUBSTR function (MDEV-8006).
    - CVE-2015-0501
    - CVE-2015-2571
    - CVE-2015-0505
    - CVE-2015-0499
    (LP: #1451677)
  * New release includes fix for memory corruption on arm64 (LP: #1427406)
  * Upstream also includes lots of line ending changes (from CRLF -> LF)

 -- Otto Kekäläinen <email address hidden> Fri, 03 Jul 2015 17:39:42 +0300

[Rolf Leggewie (r0lf)]

utopic has seen the end of its life and is no longer receiving any updates. Marking the utopic task for this ticket as "Won't Fix".

[Christian Ehrhardt  (paelzer)]

Vivid is end of life, marking as won't fix.
There is one "lost" update - which was [1].
But that was lost in a follow on security update - see c#26 + c#27 for details.

That makes mysql-5.6 in Trusty "triaged" instead of "Fix Committed"
It does not seem a later stable release has intregated that particular fix you had - maybe an alternative.
@Dannf - any plan to reroll these - or will this go to Won't Fix?
After all that time I'll mark won't fix now, please update if this is still work in progress (or a zombie that should be WIP).

[1]: https://launchpad.net/ubuntu/+source/mysql-5.6/5.6.19-0ubuntu0.14.04.2

[dann frazier (dannf)]

On Wed, Oct 25, 2017 at 7:56 AM, ChristianEhrhardt
<email address hidden> wrote:
> Vivid is end of life, marking as won't fix.
> There is one "lost" update - which was [1].
> But that was lost in a follow on security update - see c#26 + c#27 for details.
>
> That makes mysql-5.6 in Trusty "triaged" instead of "Fix Committed"
> It does not seem a later stable release has intregated that particular fix you had - maybe an alternative.
> @Dannf - any plan to reroll these - or will this go to Won't Fix?

I believe this was later fixed, in a different way, by an upstream
point release.

From https://dev.mysql.com/doc/relnotes/mysql/5.6/en/news-5-6-27.html :
 "InnoDB: A data corruption occurred on ARM64. GCC builtins did not
issue the correct fences when setting or unsetting the lock word. (Bug
#21102971, Bug #76135)"

I'll therefore go ahead and mark trusty as Fix Released.