Bug #1571865 “mysql fails to start after upgrade if previous def...” : Bugs : mysql-5.7 package : Ubuntu

Revision history for this message

Joel Parke (joelparke) wrote on 2016-04-18:

#1

.etc.apparmor.d.usr.sbin.mysqld.txt Edit (1.5 KiB, text/plain; charset="utf-8")
AptOrdering.txt Edit (37.8 KiB, text/plain; charset="utf-8")
Dependencies.txt Edit (3.3 KiB, text/plain; charset="utf-8")
Df.txt Edit (551 bytes, text/plain; charset="utf-8")
Dmesg.txt Edit (34.8 KiB, text/plain; charset="utf-8")
DpkgHistoryLog.txt Edit (40.9 KiB, text/plain; charset="utf-8")
DpkgTerminalLog.gz Edit (36.8 KiB, application/x-gzip)
HookError_ubuntu.txt Edit (783 bytes, text/plain; charset="utf-8")
JournalErrors.txt Edit (73.8 KiB, text/plain; charset="utf-8")
KernLog.txt Edit (4.2 KiB, text/plain; charset="utf-8")
Logs.var.log.mysql.error.log.txt Edit (25.8 KiB, text/plain; charset="utf-8")
MySQLConf.etc.mysql.my.cnf.txt Edit (4.8 KiB, text/plain; charset="utf-8")
MySQLConf.etc.mysql.mysql.cnf.txt Edit (682 bytes, text/plain; charset="utf-8")
MySQLConf.etc.mysql.mysql.conf.d.mysqld.cnf.txt Edit (3.0 KiB, text/plain; charset="utf-8")

Revision history for this message

Robie Basak (racb) wrote on 2016-04-19:

#2

From log:

2016-04-18T22:13:30.875218Z 0 [Warning] System table 'plugin' is expected to be transactional.
2016-04-18T22:13:30.875411Z 0 [ERROR] Function 'innodb' already exists
2016-04-18T22:13:30.875459Z 0 [Warning] Couldn't load plugin named 'innodb' with soname 'ha_innodb.so'.
2016-04-18T22:13:30.875477Z 0 [ERROR] Function 'federated' already exists
2016-04-18T22:13:30.875487Z 0 [Warning] Couldn't load plugin named 'federated' with soname 'ha_federated.so'.
2016-04-18T22:13:30.875498Z 0 [ERROR] Function 'blackhole' already exists
2016-04-18T22:13:30.875506Z 0 [Warning] Couldn't load plugin named 'blackhole' with soname 'ha_blackhole.so'.
2016-04-18T22:13:30.875517Z 0 [ERROR] Function 'archive' already exists
2016-04-18T22:13:30.875525Z 0 [Warning] Couldn't load plugin named 'archive' with soname 'ha_archive.so'.
2016-04-18T22:13:30.876258Z 0 [ERROR] unknown variable 'key_buffer=16K'
2016-04-18T22:13:30.876347Z 0 [ERROR] Aborting

Robie Basak (racb) on 2016-04-21

description:	updated
summary:	- package mysql-server-5.7 5.7.11-0ubuntu6 failed to install/upgrade: - subprocess installed post-installation script returned error exit status - 1 + mysql fails to start after upgrade if previous defaults were customised
Changed in mysql-5.7 (Ubuntu):
status:	New → Triaged
importance:	Undecided → High

Robie Basak (racb) on 2016-04-21

description:

updated

Mathew Hodson (mhodson) on 2016-04-22

tags:	added: dist-upgrade
Changed in ubuntu-release-notes:
status:	New → Fix Released

Stein Rune Risa (steinrr-deactivatedaccount) on 2016-04-22

Changed in mysql-5.7 (Ubuntu):
status:	Triaged → Fix Released

Revision history for this message

Stein Rune Risa (steinrr-deactivatedaccount) wrote on 2016-04-22:

#3

This was wrongly marked as fixed by me. Please rollback that change.

Mathew Hodson (mhodson) on 2016-04-23

Changed in mysql-5.7 (Ubuntu):
status:	Fix Released → Triaged

Robie Basak (racb) on 2016-04-25

Changed in mysql-5.7 (Ubuntu Xenial):
importance:	Undecided → High
status:	New → Triaged

Revision history for this message

Doug Smythies (dsmythies) wrote on 2016-04-26:

#4

In my case there was no customization. All 3 of my effected 16.04 servers were fresh installs of 16.04 during the development phase of 16.04. This was all two or three weeks ago. I finally sorted out the mess, by re-installing mysql.

Robie Basak (racb) on 2016-04-29

Changed in mysql-5.7 (Ubuntu):
assignee:	nobody → Lars Tangvald (lars-tangvald)
Changed in mysql-5.7 (Ubuntu Xenial):
assignee:	nobody → Lars Tangvald (lars-tangvald)
Changed in mysql-5.7 (Ubuntu):
status:	Triaged → In Progress

Robie Basak (racb) on 2016-04-29

Changed in mysql-5.7 (Ubuntu):
milestone:	none → ubuntu-16.05

Revision history for this message

Mitch Claborn (mitch-news) wrote on 2016-06-21:

#5

Re: [Workaround Option 3/3]

I tried this method, placing my local customizations in /etc/mysql/mysql.conf.d/local.cnf but mysqld did not pick those up. I had to name my file zzlocal.cnf. I'm guessing it reads those file in alpha order, with the later files overriding the earlier.

Revision history for this message

Stephen Baker (stephenbaker75) wrote on 2016-06-22: Re: [Bug 1571865] Re: mysql fails to start after upgrade if previous defaults were customised

#6

Download full text (6.6 KiB)

I had spent a few hours on this with no resolution. Even with the mysql
service running as root, it failed to get permissions to write to a data
directory other than /var/lib/mysql (or whatever the default is if that's
wrong). Even with apparmor configured to allow the new data directory, and
even in complain mode, it still failed to read/write files in the new
datadir or couldn't open a socket.

My fix was moving my data to the location; mounting a device to
/var/lib/mysql.

On Tue, Jun 21, 2016 at 11:43 AM, Mitch Claborn <email address hidden>
wrote:

> Re: [Workaround Option 3/3]
>
> I tried this method, placing my local customizations in
> /etc/mysql/mysql.conf.d/local.cnf but mysqld did not pick those up. I
> had to name my file zzlocal.cnf. I'm guessing it reads those file in
> alpha order, with the later files overriding the earlier.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1573878).
> https://bugs.launchpad.net/bugs/1571865
>
> Title:
> mysql fails to start after upgrade if previous defaults were
> customised
>
> Status in Release Notes for Ubuntu:
> Fix Released
> Status in mysql-5.7 package in Ubuntu:
> In Progress
> Status in mysql-5.7 source package in Xenial:
> Triaged
>
> Bug description:
> In 14.04 (both in 5.5 and 5.6), the default /etc/mysql/my.cnf shipped
> with options "key-buffer" and "myisam-recover". In 5.7, these option
> names have been removed and replaced with "key-buffer-size" and
> "myisam-recover-options" instead. If a user customised
> /etc/mysql/my.cnf before, then the entire file is preserved, including
> the removed options, causing mysqld to fail to start after upgrade to
> 5.7 (eg. when upgrading to 16.04).
>
> If your customisations were made in 15.04 or 15.10 and
> /etc/mysql/my.cnf.migrated does not exist, then the workarounds below
> are still essentially the same but with a couple of exceptions:
>
> 1. Instead of editing /etc/mysql/my.cnf.migrated, edit the file you
> originally changed directly. This may be /etc/mysql/my.cnf (through
> the symlink), or a file you changed or added in either
> /etc/mysql/conf.d/ or /etc/mysql/mysql.conf.d/. The command "grep -Er
> 'key.buffer|myisam.recover' /etc/mysql" may help you in locating this.
>
> 2. No need to run update-alternatives to remove use of
> /etc/mysql/my.cnf.migrated.
>
> [Workaround Option 1/3]
>
> To reset your MySQL configuration back to defaults, type "sudo update-
> alternatives --remove my.cnf /etc/mysql/my.cnf.migrated" after the
> upgrade. Then use "sudo service mysql start" to start the MySQL daemon
> and "sudo apt-get -f install" to recover your system packaging state.
>
> This option is not available if /etc/mysql/my.cnf.migrated doesn't
> exist on your system, for example if your customisations were made on
> 15.04 or 15.10.
>
> [Workaround Option 2/3]
>
> For a quick fix while retaining your existing customised
> configuration, edit the [mysqld] section /etc/mysql/my.cnf.migrated as
> follows. But see the caveats detailed below and consider Workaround
> Option 3/3 instead first.
>
> 1. Replace "key_buf...

I had spent a few hours on this with no resolution.  Even with the mysql
service running as root, it failed to get permissions to write to a data
directory other than /var/lib/mysql (or whatever the default is if that's
wrong).  Even with apparmor configured to allow the new data directory, and
even in complain mode, it still failed to read/write files in the new
datadir or couldn't open a socket.

My fix was moving my data to the location; mounting a device to
/var/lib/mysql.

On Tue, Jun 21, 2016 at 11:43 AM, Mitch Claborn <1571865@bugs.launchpad.net>
wrote:

> Re: [Workaround Option 3/3]
>
> I tried this method, placing my local customizations in
> /etc/mysql/mysql.conf.d/local.cnf but mysqld did not pick those up. I
> had to name my file zzlocal.cnf. I'm guessing it reads those file in
> alpha order, with the later files overriding the earlier.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1573878).
> https://bugs.launchpad.net/bugs/1571865
>
> Title:
>   mysql fails to start after upgrade if previous defaults were
>   customised
>
> Status in Release Notes for Ubuntu:
>   Fix Released
> Status in mysql-5.7 package in Ubuntu:
>   In Progress
> Status in mysql-5.7 source package in Xenial:
>   Triaged
>
> Bug description:
>   In 14.04 (both in 5.5 and 5.6), the default /etc/mysql/my.cnf shipped
>   with options "key-buffer" and "myisam-recover". In 5.7, these option
>   names have been removed and replaced with "key-buffer-size" and
>   "myisam-recover-options" instead. If a user customised
>   /etc/mysql/my.cnf before, then the entire file is preserved, including
>   the removed options, causing mysqld to fail to start after upgrade to
>   5.7 (eg. when upgrading to 16.04).
>
>   If your customisations were made in 15.04 or 15.10 and
>   /etc/mysql/my.cnf.migrated does not exist, then the workarounds below
>   are still essentially the same but with a couple of exceptions:
>
>   1. Instead of editing /etc/mysql/my.cnf.migrated, edit the file you
>   originally changed directly. This may be /etc/mysql/my.cnf (through
>   the symlink), or a file you changed or added in either
>   /etc/mysql/conf.d/ or /etc/mysql/mysql.conf.d/. The command "grep -Er
>   'key.buffer|myisam.recover' /etc/mysql" may help you in locating this.
>
>   2. No need to run update-alternatives to remove use of
>   /etc/mysql/my.cnf.migrated.
>
>   [Workaround Option 1/3]
>
>   To reset your MySQL configuration back to defaults, type "sudo update-
>   alternatives --remove my.cnf /etc/mysql/my.cnf.migrated" after the
>   upgrade. Then use "sudo service mysql start" to start the MySQL daemon
>   and "sudo apt-get -f install" to recover your system packaging state.
>
>   This option is not available if /etc/mysql/my.cnf.migrated doesn't
>   exist on your system, for example if your customisations were made on
>   15.04 or 15.10.
>
>   [Workaround Option 2/3]
>
>   For a quick fix while retaining your existing customised
>   configuration, edit the [mysqld] section /etc/mysql/my.cnf.migrated as
>   follows. But see the caveats detailed below and consider Workaround
>   Option 3/3 instead first.
>
>   1. Replace "key_buffer" with "key_buffer_size". Note that there is a
>   second occurrance of "key_buffer" under the [isamchk] section at the
>   end of the file; changing this second occurrance is not necessary.
>
>   2. Replace "myisam-recover" with "myisam-recover-options".
>
>   Then use "sudo service mysql start" to start the MySQL daemon again
>   and "sudo apt-get -f install" to recover your system packaging state.
>
>   However, this workaround does not put you in the best place for future
>   upgrades, since packaging will continue to not be able to perfectly
>   update this file while preserving your modifications. Additionally
>   there may be parts of your previously customised configuration that
>   still will not work with MySQL 5.7.
>
>   To make future upgrades smoother in the future, consider following the
>   next workaround option instead.
>
>   [Workaround Option 3/3]
>
>   Examine /etc/mysql/my.cnf.migrated for the customisations you made
>   previously. You can find an original version of /etc/mysql/my.cnf as
>   shipped with 14.04 at: http://bazaar.launchpad.net/~ubuntu-
>
> branches/ubuntu/trusty/mysql-5.5/trusty/view/head:/debian/additions/my.cnf
>
>   Determine the changes you made to /etc/mysql/my.cnf. Taking only these
>   changes and not the default contents of this file, add just your
>   customisations into a new file at /etc/mysql/mysql.conf.d/local.cnf
>   (preferred) and/or by editing /etc/mysql/mysql.conf.d/mysqld.cnf (to
>   be avoided if possible) if necessary.
>
>   Run: "sudo update-alternatives --remove my.cnf
>   /etc/mysql/my.cnf.migrated" to switch to the new configuration scheme.
>
>   Run: "sudo service mysql start" to start the MySQL daemon and "sudo
>   apt-get -f install" to recover your system packaging state.
>
>   [Original Description]
>
>   Upgrading from 15.10 to 16.04 fails here
>   Not sure if this is related to a bug report already reported.
>
>   ProblemType: Package
>   DistroRelease: Ubuntu 16.04
>   Package: mysql-server-5.7 5.7.11-0ubuntu6
>   ProcVersionSignature: Ubuntu 3.19.0-30.34-generic 3.19.8-ckt6
>   Uname: Linux 3.19.0-30-generic x86_64
>   ApportVersion: 2.20.1-0ubuntu2
>   Architecture: amd64
>   Date: Mon Apr 18 18:13:33 2016
>   ErrorMessage: subprocess installed post-installation script returned
> error exit status 1
>   InstallationDate: Installed on 2014-04-18 (731 days ago)
>   InstallationMedia:
>
>   Logs.var.log.daemon.log:
>
>   MySQLConf.etc.mysql.conf.d.mysql.cnf: [mysql]
>   MySQLConf.etc.mysql.conf.d.mysqld_safe_syslog.cnf:
>    [mysqld_safe]
>    syslog
>   MySQLConf.etc.mysql.conf.d.mysqldump.cnf:
>    [mysqldump]
>    quick
>    quote-names
>    max_allowed_packet   = 16M
>   MySQLConf.etc.mysql.mysql.conf.d.mysqld_safe_syslog.cnf:
>    [mysqld_safe]
>    syslog
>   MySQLVarLibDirListing: ['debian-5.7.flag', 'debian-5.5.flag',
> 'debian-5.6.flag', 'ib_logfile1', 'drupal8', 'servermail', 'ib_logfile0',
> 'auto.cnf', 'risenlif_risenlife2', 'dynazu_wiki', 'performance_schema',
> 'ibdata1', 'phpmyadmin', 'ib_buffer_pool', 'mysql_upgrade_info',
> 'parke_wiki', 'tracker', 'mysql']
>   ProcCmdline: root=LABEL=DOROOT ro
>   RelatedPackageVersions:
>    dpkg 1.18.4ubuntu1
>    apt  1.2.10ubuntu1
>   SourcePackage: mysql-5.7
>   Title: package mysql-server-5.7 5.7.11-0ubuntu6 failed to
> install/upgrade: subprocess installed post-installation script returned
> error exit status 1
>   UpgradeStatus: Upgraded to xenial on 2016-04-18 (0 days ago)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-release-notes/+bug/1571865/+subscriptions
>

Robie Basak (racb) on 2016-07-12

Changed in mysql-5.7 (Ubuntu):
status:	In Progress → Fix Committed

Robie Basak (racb) on 2016-07-13

Changed in mysql-5.7 (Ubuntu Xenial):
status:	Triaged → In Progress
assignee:	Lars Tangvald (lars-tangvald) → Robie Basak (racb)

Revision history for this message

Launchpad Janitor (janitor) wrote on 2016-07-13:

#7

This bug was fixed in the package mysql-5.7 - 5.7.13-0ubuntu3

---------------
mysql-5.7 (5.7.13-0ubuntu3) yakkety; urgency=medium

  * Automatically replace removed config options from 5.5 and old 5.6
    installations (LP: #1571865). Cherry-picked from Debian VCS ef3f92b.
    Thanks to Lars Tangvald.

-- Robie Basak <email address hidden> Tue, 12 Jul 2016 17:45:00 +0100

Changed in mysql-5.7 (Ubuntu):
status:	Fix Committed → Fix Released

Robie Basak (racb) on 2016-07-14

Changed in mysql-5.7 (Ubuntu Xenial):
milestone:	none → ubuntu-16.04.1

Lars Tangvald (lars-tangvald) on 2016-07-14

description:

updated

Lars Tangvald (lars-tangvald) on 2016-07-14

description:

updated

Revision history for this message

Robie Basak (racb) wrote on 2016-07-14:

#8

Thanks Mitch. I've updated the workaround accordingly.

description:	updated
description:	updated

Revision history for this message

Robie Basak (racb) wrote on 2016-07-14:

#9

Uploaded to Xenial. SRU team: you may find it easier to review if you verify my uploaded delta is the same as https://git.launchpad.net/~racb/ubuntu/+source/mysql-5.7/log/?h=mysql-5.7/ubuntu/xenial and then use that.

Revision history for this message

Adam Conrad (adconrad) wrote on 2016-07-15: Please test proposed package

#10

Hello Joel, or anyone else affected,

Accepted mysql-5.7 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-5.7/5.7.12-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.7 (Ubuntu Xenial):
status:	In Progress → Fix Committed
tags:	added: verification-needed

Revision history for this message

Francisco José Cañizares Santofimia (telefrancisco) wrote on 2016-07-17:

#11

Latest package in SUR doesn't solves the problem for me. Moreover, log in /var/log/mysql.log is empty. Can you tell me how can I provide debug info?

Revision history for this message

maghfurin (furin-xiitkj) wrote on 2016-07-18: Re: [Bug 1571865] Re: mysql fails to start after upgrade if previous defaults were customised

#12

Download full text (7.9 KiB)

furin@furin-K43SD:~$ sudo apt-get install -f
[sudo] password for furin:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer
required:
  linux-headers-4.4.0-22 linux-headers-4.4.0-22-generic
  linux-headers-4.4.0-24 linux-headers-4.4.0-24-generic
  linux-image-4.4.0-22-generic linux-image-4.4.0-24-generic
  linux-image-extra-4.4.0-22-generic linux-image-extra-4.4.0-24-generic
  mysql-server-5.7 mysql-server-core-5.7
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up mysql-server-5.7 (5.7.12-0ubuntu1.1) ...
mysql_upgrade: Got error: 1045: Access denied for user 'root'@'localhost'
(using password: NO) while connecting to the MySQL server
Upgrade process encountered error and will not continue.
mysql_upgrade failed with exit status 11
dpkg: error processing package mysql-server-5.7 (--configure):
subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
mysql-server-5.7
E: Sub-process /usr/bin/dpkg returned an error code (1)

On 18 July 2016 at 05:50, Francisco José Cañizares Santofimia <
<email address hidden>> wrote:

> Latest package in SUR doesn't solves the problem for me. Moreover, log
> in /var/log/mysql.log is empty. Can you tell me how can I provide debug
> info?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1571865
>
> Title:
> mysql fails to start after upgrade if previous defaults were
> customised
>
> Status in Release Notes for Ubuntu:
> Fix Released
> Status in mysql-5.7 package in Ubuntu:
> Fix Released
> Status in mysql-5.7 source package in Xenial:
> Fix Committed
>
> Bug description:
> In 14.04 (both in 5.5 and 5.6), the default /etc/mysql/my.cnf shipped
> with options "key-buffer" and "myisam-recover". In 5.7, these option
> names have been removed and replaced with "key-buffer-size" and
> "myisam-recover-options" instead. If a user customised
> /etc/mysql/my.cnf before, then the entire file is preserved, including
> the removed options, causing mysqld to fail to start after upgrade to
> 5.7 (eg. when upgrading to 16.04).
>
> [Impact]
> Server will fail to start, causing upgrade/installation of MySQL to fail.
>
> [Test case]
> 1. Install mysql-server in Ubuntu Trusty
> 2. Edit /etc/mysql/my.cnf and save it (can just add a comment)
> 3. Upgrade distro to Xenial
>
> Expected behavior:
> Server upgrades and starts normally
>
> Actual behavior:
> Server fails to upgrade, because it can't start, throwing an error about
> 'unknown option key_buffer'
>
> [Regression Potential]
> * If the sed command is faulty in some way it could mangle the options,
> leading to the server not starting and installation failing
>
> [Workarounds]
>
> If your customisations were made in 15.04 or 15.10 and
> /etc/mysql/my.cnf.migrated does not exist, then the workarounds below
> are still es...

furin@furin-K43SD:~$ sudo apt-get install -f
[sudo] password for furin:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer
required:
  linux-headers-4.4.0-22 linux-headers-4.4.0-22-generic
  linux-headers-4.4.0-24 linux-headers-4.4.0-24-generic
  linux-image-4.4.0-22-generic linux-image-4.4.0-24-generic
  linux-image-extra-4.4.0-22-generic linux-image-extra-4.4.0-24-generic
  mysql-server-5.7 mysql-server-core-5.7
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up mysql-server-5.7 (5.7.12-0ubuntu1.1) ...
mysql_upgrade: Got error: 1045: Access denied for user 'root'@'localhost'
(using password: NO) while connecting to the MySQL server
Upgrade process encountered error and will not continue.
mysql_upgrade failed with exit status 11
dpkg: error processing package mysql-server-5.7 (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 mysql-server-5.7
E: Sub-process /usr/bin/dpkg returned an error code (1)

On 18 July 2016 at 05:50, Francisco José Cañizares Santofimia <
telefrancisco@gmail.com> wrote:

> Latest package in SUR doesn't solves the problem for me. Moreover, log
> in /var/log/mysql.log is empty. Can you tell me how can I provide debug
> info?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1571865
>
> Title:
>   mysql fails to start after upgrade if previous defaults were
>   customised
>
> Status in Release Notes for Ubuntu:
>   Fix Released
> Status in mysql-5.7 package in Ubuntu:
>   Fix Released
> Status in mysql-5.7 source package in Xenial:
>   Fix Committed
>
> Bug description:
>   In 14.04 (both in 5.5 and 5.6), the default /etc/mysql/my.cnf shipped
>   with options "key-buffer" and "myisam-recover". In 5.7, these option
>   names have been removed and replaced with "key-buffer-size" and
>   "myisam-recover-options" instead. If a user customised
>   /etc/mysql/my.cnf before, then the entire file is preserved, including
>   the removed options, causing mysqld to fail to start after upgrade to
>   5.7 (eg. when upgrading to 16.04).
>
>   [Impact]
>   Server will fail to start, causing upgrade/installation of MySQL to fail.
>
>   [Test case]
>   1. Install mysql-server in Ubuntu Trusty
>   2. Edit /etc/mysql/my.cnf and save it (can just add a comment)
>   3. Upgrade distro to Xenial
>
>   Expected behavior:
>   Server upgrades and starts normally
>
>   Actual behavior:
>   Server fails to upgrade, because it can't start, throwing an error about
> 'unknown option key_buffer'
>
>   [Regression Potential]
>   * If the sed command is faulty in some way it could mangle the options,
>   leading to the server not starting and installation failing
>
>   [Workarounds]
>
>   If your customisations were made in 15.04 or 15.10 and
>   /etc/mysql/my.cnf.migrated does not exist, then the workarounds below
>   are still essentially the same but with a couple of exceptions:
>
>   1. Instead of editing /etc/mysql/my.cnf.migrated, edit the file you
>   originally changed directly. This may be /etc/mysql/my.cnf (through
>   the symlink), or a file you changed or added in either
>   /etc/mysql/conf.d/ or /etc/mysql/mysql.conf.d/. The command "grep -Er
>   'key.buffer|myisam.recover' /etc/mysql" may help you in locating this.
>
>   2. No need to run update-alternatives to remove use of
>   /etc/mysql/my.cnf.migrated.
>
>   [Workaround Option 1/3]
>
>   To reset your MySQL configuration back to defaults, type "sudo update-
>   alternatives --remove my.cnf /etc/mysql/my.cnf.migrated" after the
>   upgrade. Then use "sudo service mysql start" to start the MySQL daemon
>   and "sudo apt-get -f install" to recover your system packaging state.
>
>   This option is not available if /etc/mysql/my.cnf.migrated doesn't
>   exist on your system, for example if your customisations were made on
>   15.04 or 15.10.
>
>   [Workaround Option 2/3]
>
>   For a quick fix while retaining your existing customised
>   configuration, edit the [mysqld] section /etc/mysql/my.cnf.migrated as
>   follows. But see the caveats detailed below and consider Workaround
>   Option 3/3 instead first.
>
>   1. Replace "key_buffer" with "key_buffer_size". Note that there is a
>   second occurrance of "key_buffer" under the [isamchk] section at the
>   end of the file; changing this second occurrance is not necessary.
>
>   2. Replace "myisam-recover" with "myisam-recover-options".
>
>   Then use "sudo service mysql start" to start the MySQL daemon again
>   and "sudo apt-get -f install" to recover your system packaging state.
>
>   However, this workaround does not put you in the best place for future
>   upgrades, since packaging will continue to not be able to perfectly
>   update this file while preserving your modifications. Additionally
>   there may be parts of your previously customised configuration that
>   still will not work with MySQL 5.7.
>
>   To make future upgrades smoother in the future, consider following the
>   next workaround option instead.
>
>   [Workaround Option 3/3]
>
>   Examine /etc/mysql/my.cnf.migrated for the customisations you made
>   previously. You can find an original version of /etc/mysql/my.cnf as
>   shipped with 14.04 at: http://bazaar.launchpad.net/~ubuntu-
>
> branches/ubuntu/trusty/mysql-5.5/trusty/view/head:/debian/additions/my.cnf
>
>   Determine the changes you made to /etc/mysql/my.cnf. Taking only these
>   changes and not the default contents of this file, add just your
>   customisations into a new file at /etc/mysql/mysql.conf.d/zzlocal.cnf
>   (preferred) and/or by editing /etc/mysql/mysql.conf.d/mysqld.cnf (to
>   be avoided if possible) if necessary.
>
>   Run: "sudo update-alternatives --remove my.cnf
>   /etc/mysql/my.cnf.migrated" to switch to the new configuration scheme.
>
>   Run: "sudo service mysql start" to start the MySQL daemon and "sudo
>   apt-get -f install" to recover your system packaging state.
>
>   [Original Description]
>
>   Upgrading from 15.10 to 16.04 fails here
>   Not sure if this is related to a bug report already reported.
>
>   ProblemType: Package
>   DistroRelease: Ubuntu 16.04
>   Package: mysql-server-5.7 5.7.11-0ubuntu6
>   ProcVersionSignature: Ubuntu 3.19.0-30.34-generic 3.19.8-ckt6
>   Uname: Linux 3.19.0-30-generic x86_64
>   ApportVersion: 2.20.1-0ubuntu2
>   Architecture: amd64
>   Date: Mon Apr 18 18:13:33 2016
>   ErrorMessage: subprocess installed post-installation script returned
> error exit status 1
>   InstallationDate: Installed on 2014-04-18 (731 days ago)
>   InstallationMedia:
>
>   Logs.var.log.daemon.log:
>
>   MySQLConf.etc.mysql.conf.d.mysql.cnf: [mysql]
>   MySQLConf.etc.mysql.conf.d.mysqld_safe_syslog.cnf:
>    [mysqld_safe]
>    syslog
>   MySQLConf.etc.mysql.conf.d.mysqldump.cnf:
>    [mysqldump]
>    quick
>    quote-names
>    max_allowed_packet   = 16M
>   MySQLConf.etc.mysql.mysql.conf.d.mysqld_safe_syslog.cnf:
>    [mysqld_safe]
>    syslog
>   MySQLVarLibDirListing: ['debian-5.7.flag', 'debian-5.5.flag',
> 'debian-5.6.flag', 'ib_logfile1', 'drupal8', 'servermail', 'ib_logfile0',
> 'auto.cnf', 'risenlif_risenlife2', 'dynazu_wiki', 'performance_schema',
> 'ibdata1', 'phpmyadmin', 'ib_buffer_pool', 'mysql_upgrade_info',
> 'parke_wiki', 'tracker', 'mysql']
>   ProcCmdline: root=LABEL=DOROOT ro
>   RelatedPackageVersions:
>    dpkg 1.18.4ubuntu1
>    apt  1.2.10ubuntu1
>   SourcePackage: mysql-5.7
>   Title: package mysql-server-5.7 5.7.11-0ubuntu6 failed to
> install/upgrade: subprocess installed post-installation script returned
> error exit status 1
>   UpgradeStatus: Upgraded to xenial on 2016-04-18 (0 days ago)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-release-notes/+bug/1571865/+subscriptions
>

Revision history for this message

Francisco José Cañizares Santofimia (telefrancisco) wrote on 2016-07-18:

#13

#12 You're still using the "old" package (5.7.12-0ubuntu1.1) that fails but I'm talking about the one in proposed (see #10) (5.7.12-0ubuntu1.2).

Revision history for this message

Wellington Torrejais da Silva (wellington1993) wrote on 2016-07-19:

#14

This:
insserv: warning: current start runlevel(s) (empty) of script `mysql' overrides LSB defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `mysql' overrides LSB defaults (0 1 6).

Solved:
sudo update-rc.d -f mysql remove && sudo update-rc.d mysql defaults

But this still remains:
""
Configurando mysql-server-5.7 (5.7.12-0ubuntu1.2) ...
Renaming removed key_buffer and myisam-recover options (if present)
mysql_upgrade: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) while connecting to the MySQL server
Upgrade process encountered error and will not continue.
mysql_upgrade failed with exit status 11
dpkg: erro ao processar o pacote mysql-server-5.7 (--configure):
sub-processo script post-installation instalado retornou estado de saída de erro 1
dpkg: problemas com dependências impedem a configuração de mysql-server:
mysql-server depende de mysql-server-5.7; porém:
Pacote mysql-server-5.7 não está configurado ainda.

dpkg: erro ao processar o pacote mysql-server (--configure):
problemas de dependência - deixando desconfigurado
Nenhum relatório apport escrito pois a mensagem de erro indica que é um erro de seguimento de um erro anterior.
Erros foram encontrados durante o processamento de:
mysql-server-5.7
mysql-server
E: Sub-process /usr/bin/dpkg returned an error code (1)
""

Any ideas?

Revision history for this message

Wellington Torrejais da Silva (wellington1993) wrote on 2016-07-19:

#15

Comment #14 Solved after remove and reinstall (Update not work).
Sorry #14 unnecessary comment.

Thanks!

Revision history for this message

Robie Basak (racb) wrote on 2016-07-20:

#16

I've spent some time testing this fix in Xenial. It was broken for two reasons:

1) I failed to update the version string check in mysql-server-5.7.postinst in preparing the SRU, so the migration code never ran. This isn't a regression, just a failure to fix the problem as intended.

2) On an upgrade from Trusty to Xenial, mysql-server-5.7 itself isn't being upgrade, so again the migration code never ran.

The fix is to change the match to:
if dpkg --compare-versions "$2" le "5.7.13-0ubuntu0~"; then

I tested 5.7.13-0ubuntu0.16.04.1 from ppa:ubuntu-security-proposed as follows:

I hacked the postinst in the binary package directly as above, to save time. I used "apt-get --download-only" to pull all the binaries generated from src:mysql:5.7, and placed the modified set (just mysql-server-5.7, the others were just straight copies) into a local repository, created Packages and Release with apt-ftparchive, and then pointed apt to using "deb [trusted=yes] file:///root/repo /". I then used this in place of the PPA for testing.

The cases I tested were:

Trusty to Xenial upgrade without local conffile modification: PASS
Trusty to Xenial upgrade with local conffile modification: PASS
Trusty to old Xenial package with local conffile modification (expected fail), then upgrade to new Xenial package: PASS
Xenial to new package upgrade without local conffile modification: PASS
Xenial to new package upgrade with local conffile modification: PASS
Fresh Xenial new package install: PASS

The fix is now working as expected.

Francisco, thank you for testing the proposed update. Please note that there are a variety of reasons that mysqld could fail to start. What I am tracking in this bug is the specific case of an unrelated customisation in 14.04 causing mysqld to fail to start after upgrade to 16.04 because previously valid default options will be preserved in the upgrade and are now deprecated.

This is what this proposed fix fixes: it detects those specific configuration directives and renames them, which should improve the situation for most users in this fairly common case.

If there are other root causes that impact users, then I'd like to fix them too, but we need to identify what those root causes are. To save confusion, we should track them sepearately in their own bugs.

Logs should be in /var/log/mysql/error.log. If this is not the case, you could be upgrading from a very old customised configuration where the default configuration did not provide this. You need "log_error = /var/log/mysql/error.log" directive. If your system is this old, it'll probably be in /etc/mysql/my.cnf.migrated unless you have updated your configuration to use the new scheme since. I hope this helps, but if you have a reproducible unrelated issue, please continue in a fresh bug report.

I've spent some time testing this fix in Xenial. It was broken for two reasons:

1) I failed to update the version string check in mysql-server-5.7.postinst in preparing the SRU, so the migration code never ran. This isn't a regression, just a failure to fix the problem as intended.

2) On an upgrade from Trusty to Xenial, mysql-server-5.7 itself isn't being upgrade, so again the migration code never ran.

The fix is to change the match to:
    if dpkg --compare-versions "$2" le "5.7.13-0ubuntu0~"; then

I tested 5.7.13-0ubuntu0.16.04.1 from ppa:ubuntu-security-proposed as follows:

I hacked the postinst in the binary package directly as above, to save time. I used "apt-get --download-only" to pull all the binaries generated from src:mysql:5.7, and placed the modified set (just mysql-server-5.7, the others were just straight copies) into a local repository, created Packages and Release with apt-ftparchive, and then pointed apt to using "deb [trusted=yes] file:///root/repo /". I then used this in place of the PPA for testing.

The cases I tested were:

Trusty to Xenial upgrade without local conffile modification: PASS
Trusty to Xenial upgrade with local conffile modification: PASS
Trusty to old Xenial package with local conffile modification (expected fail), then upgrade to new Xenial package: PASS
Xenial to new package upgrade without local conffile modification: PASS
Xenial to new package upgrade with local conffile modification: PASS
Fresh Xenial new package install: PASS

The fix is now working as expected.

Francisco, thank you for testing the proposed update. Please note that there are a variety of reasons that mysqld could fail to start. What I am tracking in this bug is the specific case of an unrelated customisation in 14.04 causing mysqld to fail to start after upgrade to 16.04 because previously valid default options will be preserved in the upgrade and are now deprecated.

This is what this proposed fix fixes: it detects those specific configuration directives and renames them, which should improve the situation for most users in this fairly common case.

If there are other root causes that impact users, then I'd like to fix them too, but we need to identify what those root causes are. To save confusion, we should track them sepearately in their own bugs.

Logs should be in /var/log/mysql/error.log. If this is not the case, you could be upgrading from a very old customised configuration where the default configuration did not provide this. You need "log_error = /var/log/mysql/error.log" directive. If your system is this old, it'll probably be in /etc/mysql/my.cnf.migrated unless you have updated your configuration to use the new scheme since. I hope this helps, but if you have a reproducible unrelated issue, please continue in a fresh bug report.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2016-07-21:

#17

This bug was fixed in the package mysql-5.7 - 5.7.13-0ubuntu0.16.04.2

---------------
mysql-5.7 (5.7.13-0ubuntu0.16.04.2) xenial-security; urgency=medium

  * SECURITY UPDATE: Update to 5.7.13 to fix security issues (LP: #1604796)
    - http://www.oracle.com/technetwork/security-advisory/cpujul2016-2881720.html
    - CVE-2016-3424
    - CVE-2016-3459
    - CVE-2016-3477
    - CVE-2016-3486
    - CVE-2016-3501
    - CVE-2016-3518
    - CVE-2016-3521
    - CVE-2016-3588
    - CVE-2016-3614
    - CVE-2016-3615
    - CVE-2016-5436
    - CVE-2016-5437
    - CVE-2016-5439
    - CVE-2016-5440
    - CVE-2016-5441
    - CVE-2016-5442
    - CVE-2016-5443
  * debian/patches/mysql-export-scramble.patch: removed, upstream.

-- Marc Deslauriers <email address hidden> Wed, 20 Jul 2016 08:44:25 -0400

Changed in mysql-5.7 (Ubuntu Xenial):
status:	Fix Committed → Fix Released

Revision history for this message

Rocko (rockorequin) wrote on 2016-07-25:

#18

I found that when it tried to install mysql-5.7 - 5.7.13-0ubuntu0.16.04.2, dpkg just hung forever at postinst (in this case, forever means overnight). The only way around this was to kill -9 the two mysqld processes that ran (I think one was already running, and then another ran when I killed the first).

Then I upgraded to yakkety and it installed mysql-server-5.7_5.7.13-0ubuntu4_amd64.deb and I had exactly the same problem. Killing the mysqld processes as before allowed dpkg to continue.

I found some other issues too when I tried reinstalling from scratch - /etc/mysql/my.cnf has the line "!includedir /etc/mysql/conf.d/", but /etc/mysql/conf.d no longer exists if you have purged mysql before reinstalling, so the server won't start up and configuration therefore fails. I had to comment the line out manually to get it to work.

Revision history for this message

Mohamed Hafez (mohamed-m-m-hafez) wrote on 2016-08-01:

#19

Hi, so just to clarify, for those of us using do-release-upgrade from 14.04 to 16.04.1, we don't need to worry about manually renaming "key-buffer-size" and "myisam-recover-options", and all that will automatically be taken care of for us?

Revision history for this message

Robie Basak (racb) wrote on 2016-08-01:

#20

@Rocko

Sounds like you have an issue unrelated to this bug. See http://www.ubuntu.com/support/community for community support options, or if you think there is a different bug in Ubuntu, then please file that with full details.

@Mohamed

In short, yes. If you don't have a customised configuration, then the upgrade should switch to the latest configuration automatically. If you have a locally customised configuration, then you will end up with a /etc/mysql/my.cnf.migrated file active containing your previous configuration with those two options renamed. It is not necessary to make any further changes, but I advise following "Workaround Option 3/3" anyway to put you in the best place for future upgrades.

Revision history for this message

Mohamed Hafez (mohamed-m-m-hafez) wrote on 2016-09-08:

#21

So, if I understand correctly, after the upgrade, I'll still have a my.cnf file *exactly* as I left it, but the config file mysql will be using will be a my.cnf.migrated file, which will be a best-guess at auto-migrating my custom config. Is this correct?

After double-checking my.cnf.migrated and doing Workaround 3/3, can I just rename my.cnf.migrated to my.cnf and discard the old version, and be good to go?

Revision history for this message

Christian Deligant (christian-deligant) wrote on 2016-10-10:

#22

Why couldn't mysql-server just throw a warning on an unknown variable and start with a default?
Especially during a release update, mysql is critical when many packages rely on a DB-based content to upgrade themselves (Mediawiki, otrs, etc)

Revision history for this message

Lars Tangvald (lars-tangvald) wrote on 2016-10-11:

#23

Christian:
There are too many things that could go wrong for users if the server itself ignores something in the config file and uses what it considers a safe default.

For the packaging we are working on working around the issue of the server not starting in cases where users have disabled the automatic starting/stopping of services, and this could potentially be extended to include cases where the config contains invalid options (i.e. print a warning and don't try to run mysql_upgrade instead of throwing an error).

But this might not cover all the use cases you mention, since some of those packages might be expecting the server to be up and running.

Revision history for this message

Christian Deligant (christian-deligant) wrote on 2016-10-11:

#24

I understand your concern.

My installation had a separate cnf file in the conf.d directory with a couple of personal settings affecting the mysqld, among them the old slow-query variables.

This was done in order to have the my.cnf upgrade smoothly among time.

What happened is that mysql-server did not restart, probably throwing a complain but without stopping (and upgrading from 14.04 to 16.04 means 2900 MB of stuff to download, running on an SSD you really don't even notice what is passing on the screen!), the other packages complaining about not being able to connect to the socket... not exactly a good hint about the server being off. Only at the very end installation ended up with 2 packages misconfigured...

I suggest instead of that behaviour to hang the upgrade in case mysql-server is not restarting with a notification showing the offending settings in order for the sysadmin to understand what is going wrong and act before going on.

Hope this suggestion will be accepted

Revision history for this message

Robie Basak (racb) wrote on 2016-10-11:

#25

On Tue, Oct 11, 2016 at 09:37:56AM -0000, Christian Deligant wrote:
> What happened is that mysql-server did not restart, probably throwing a
> complain but without stopping (and upgrading from 14.04 to 16.04 means
> 2900 MB of stuff to download, running on an SSD you really don't even
> notice what is passing on the screen!), the other packages complaining
> about not being able to connect to the socket... not exactly a good hint
> about the server being off. Only at the very end installation ended up
> with 2 packages misconfigured...

The upgrade does its best to upgrade everything that is unaffected.
Other package unable to connect to the socket should be correctable with
the same "sudo apt-get -f install" after you have fixed the problem
manually after your upgrade. If there is some reason that didn't work
for you, then that's a separate bug that can be fixed (whether in mysql
or the dependent packages).

> I suggest instead of that behaviour to hang the upgrade in case mysql-
> server is not restarting with a notification showing the offending
> settings in order for the sysadmin to understand what is going wrong and
> act before going on.

I don't think this would be acceptable. Say ten packages with similar
challenges follow your solution. Then it would become impossible for you
to complete your upgrade unattended. You'd need to babysit the whole
process. This would be incredibly tedious. Instead, why not fix all the
manual-intervention-required issues at once at the end?

Is there some reason why the solution I described above does not work
for you?

Revision history for this message

Christian Deligant (christian-deligant) wrote on 2016-10-11:

#26

@Robie Basak
The main problem is that all the packages that rely on an active mysql-server to upgrade themselves will stop and ask what to do, thus you anyway need to babysit them :)

At this point, having the upgrade of Mysql-Server stopping if it was on and could not restart will only stop the upgrade once.

I don't think a release upgrade has to be done unattended...

Revision history for this message

oscar (esstampida) wrote on 2017-02-10:

#27

ñ

Revision history for this message

Robie Basak (racb) wrote on 2017-03-28:

#28

Bug 1676605 referred to this from comment 18:

> I found some other issues too when I tried reinstalling from scratch - /etc/mysql/my.cnf has the line "!includedir /etc/mysql/conf.d/", but /etc/mysql/conf.d no longer exists if you have purged mysql before reinstalling, so the server won't start up and configuration therefore fails. I had to comment the line out manually to get it to work.

I wondered if there was a bug here, but I couldn't figure out how to reproduce this. Purging mysql-common does drop /etc/mysql/conf.d/, but reinstalling it brings it back. If you can figure out how to reproduce it, please file a separate bug for that.

Ubuntu
mysql-5.7 package

mysql fails to start after upgrade if previous defaults were customised

Bug Description

CVE References

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

	Status	Importance	Assigned to	Milestone
Release Notes for Ubuntu	Fix Released	Undecided	Unassigned
mysql-5.7 (Ubuntu)	Fix Released	High	Lars Tangvald	Ubuntu ubuntu-16.05
Xenial	Fix Released	High	Robie Basak	Ubuntu ubuntu-16.04.1

Ubuntumysql-5.7 package

mysql fails to start after upgrade if previous defaults were customised

Bug Description

CVE References

Duplicates of this bug

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
mysql-5.7 package