mysql_upgrade is called twice concurrently on upgrade from 14.04

[Robie Basak]

Due to the old upstart script still being active, mysql_upgrade is called both via upstart on daemon start and by mysql-server-5.7.postinst when upgrading on a system that is running upstart. This seems to cause various mysql_upgrade failures. mysql_upgrade will alter the structure of the system tables, particularly performance_schema, which it will delete and recreate. If one process deletes it, then the other creates it, the first will also try to create it, throwing an error.

[Impact]

Error causes mysql_upgrade to fail, which in turn makes the package installation fail.

[Test case]

Difficult to test fully since it's a race condition bug with two upgrade processes running together.
* Install mysql-server in Ubuntu Trusty
* Upgrade system to Xenial

Expected behavior:
The server installs, and the database is upgraded to support 5.7

Actual behavior:
Occasionally fails with errors like 'performance_schema does not exist' or 'performance_schema already exists'.

[Regression potential]

Debian-start is made a no-op script, so if it did something useful we haven't noticed, that's lost.

---
Let's use this bug to track this particular root cause, since it isn't clear that all the other classes of report will be cleared up by this fix although we think that they may be. We can dupe them in as needed as we find out.

[Lars Tangvald (lars-tangvald)]

The old debian-start script used by upstart runs mysql_upgrade, checks for crashed tables and checks for passwordless root accounts (printing a warning if one is found).

In 5.7, d/postinst will secure passwordless root accounts with unix socket auth and run mysql_upgrade, and mysql_upgrade itself will check for and fix any crashed tables, so this script is no longer needed.

Additionally this could cause mysql_upgrade to be executed by both upstart and postinst on upgrades, causing various errors.

The solution is to change the script in 5.7 to a no-op

[Lars Tangvald (lars-tangvald)]

Change d/additions/debian-start to no-op, since it's no longer needed and was running mysql_upgrade, possibly concurrently with d/postinst

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "0001-Change-d-additions-debian-start-to-no-op.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.7 - 5.7.13-0ubuntu1

---------------
mysql-5.7 (5.7.13-0ubuntu1) yakkety; urgency=medium

  * Ubuntu merge from Debian VCS 1cdae6c. Remaining changes:
    - Revert "Build with libmecab-dev" since libmecab-dev is in
      universe.
  * Drop changes:
    - mysql-server-5.7: depend on apparmor (>= 2.10) for Trusty upgrade
      path (all upgrade paths go through Xenial; in hindsight this
      should have been a Breaks anyway).
    - Upstreamed to Debian:
      + d/p/allowemptyport.patch: update corresponding test
      + postinst: ignore mysql_upgrade's exit status 2
      + Export my_make_scrambled_password
      + Client commands in postinst should ignore custom config
      + Enable setting password for existing database if it was empty
      + Quote $rootpw correctly
      + Add libnuma-dev, libaio-dev to libmysqld-dev deps
      + d/p/fix-man-page-links.patch: drop (fixed upstream)
      + Add d/NEWS

mysql-5.7 (5.7.13-1~exp1) UNRELEASED; urgency=medium

  [ Robie Basak ]
  * Import upstream version 5.7.13.
  * Drop d/libmysqlclient18.lintian-overrides as the problem is genuine
    and should not be hidden. This addresses #812812 in part, but does
    not close it.
  * AppArmor profile refactoring and updates for 5.7.
  * Rename apport hook from 5.6 to 5.7.
  * debian/watch file update to 5.7.
  * d/rules: replace 5.6 with 5.7.
  * Drop libmysqld-pic.
  * d/rules: use dh_install --list-missing.
  * d/rules: some dh_install excludes.
  * d/rules: drop large set of comments.
  * d/rules configure: switch to INSTALL_LAYOUT=DEB.
  * d/rules configure: build WITH_INNODB_MEMCACHED.
  * d/rules configure: build with supplied boost.
  * d/rules configure: build with BUILD_CONFIG=mysql_release.
  * d/rules configure: drop SYSTEM_TYPE=.
  * d/rules: chmod changes.
  * mysqld.conf: move to non-deprecated name.
  * Rename debian/*5.6* to 5.7.
  * Adjust installation debian/additions/mysql-systemd-start.
  * Drop d/mysql-client-5.7.examples.
  * Drop some lintian overrides.
  * d/control: switch to 5.7.
  * libmysqlclient soname bump.
  * Update manpage formatting lintian overrides.
  * Drop d/p/fix-mysqlhotcopy-test-failure.patch.
  * Drop d/p/spelling.patch.
  * Drop d/p/fix_standalone_tests.patch.
  * Drop d/p/disable_tests.patch.
  * Drop d/p/scripts__mysqld_safe.sh__signals.patch.
  * Update Hurd and kFreeBSD patches.
  * Add d/p/fix-mtr-search-paths.patch.
  * Non-functional (documentation) 5.6->5.7 substitutions.
  * Drop comment from .install files.
  * Apport hook: install under new name.
  * mysql-testsuite-5.7 installation adjustments for 5.7.
  * Binary installation tweaks.
  * d/p/mysql-test-run-paths: adjust test suite for generic binary
    paths.
  * Move testsuite binaries to testsuite package.
  * Build with libmecab-dev.
  * Manpage changes for 5.7.
  * dep8 tests: s/5.6/5.7/.
  * dep8 test: adjust exclude list.
  * dep8 test: adjust test runner path.
  * dep8 test: fix typo (non-functional).
  * d/mysql-server-5.7.dirs: clean syntax.
  * TODO: d/copyright: misc updates for 5.7.
  * d/copyright: boost.
  * Mark scripts executable.
  * Drop gawk build dependency.
  * Add new MySQL 5.7 build de...

[Robie Basak (racb)]

Uploaded to Xenial. SRU team: you may find it easier to review if you verify my uploaded delta is the same as https://git.launchpad.net/~racb/ubuntu/+source/mysql-5.7/log/?h=mysql-5.7/ubuntu/xenial and then use that.

[Adam Conrad (adconrad)]

Hello Robie, or anyone else affected,

Accepted mysql-5.7 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.7/5.7.12-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Verified. This is tricky to test because it is a race condition, but I have verified that it appears that mysql_upgrade isn't being called twice, and I have also ran a test upgrade five times with no issues.

I examined /var/log/upstart/mysql.log before and after a test upgrade. Previously, we were printing "Checking for tables which need an upgrade, are corrupt or were not closed cleanly." there when running mysql_upgrade from the upstart job. After a test upgrade to xenial-proposed, we longer see this string, so it seems likely that mysql-start was updated at the correct time. I also manually checked /var/log/apt/term.log after a release upgrade to xenial-proposed and this appears to be consistent with this. mysqld is running (still on upstart) after an upgrade. After a reboot, systemd has started it as expected.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.7 - 5.7.13-0ubuntu0.16.04.2

---------------
mysql-5.7 (5.7.13-0ubuntu0.16.04.2) xenial-security; urgency=medium

  * SECURITY UPDATE: Update to 5.7.13 to fix security issues (LP: #1604796)
    - http://www.oracle.com/technetwork/security-advisory/cpujul2016-2881720.html
    - CVE-2016-3424
    - CVE-2016-3459
    - CVE-2016-3477
    - CVE-2016-3486
    - CVE-2016-3501
    - CVE-2016-3518
    - CVE-2016-3521
    - CVE-2016-3588
    - CVE-2016-3614
    - CVE-2016-3615
    - CVE-2016-5436
    - CVE-2016-5437
    - CVE-2016-5439
    - CVE-2016-5440
    - CVE-2016-5441
    - CVE-2016-5442
    - CVE-2016-5443
  * debian/patches/mysql-export-scramble.patch: removed, upstream.

 -- Marc Deslauriers <email address hidden> Wed, 20 Jul 2016 08:44:25 -0400

[WhyteHorse (whytehorse)]

This bug is back. I'm on 16.04 and got a failed upgrade of mysql.