mysql-server-5.7.postinst races daemon start with mysql_upgrade

[Maris Nikos]

Same error message as in older 5.7.x. This is on 5.7.29

ProblemType: Package
DistroRelease: Ubuntu 16.04
Package: mysql-server-5.7 5.7.29-0ubuntu0.16.04.1
ProcVersionSignature: Ubuntu 4.15.0-76.86~16.04.1-generic 4.15.18
Uname: Linux 4.15.0-76-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.21
Architecture: amd64
Date: Fri Feb 14 18:10:29 2020
ErrorMessage: subprocess installed post-installation script returned error exit status 1
InstallationDate: Installed on 2017-11-01 (835 days ago)
InstallationMedia: Ubuntu 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801)
Logs.var.log.daemon.log:

MySQLConf.etc.mysql.conf.d.mysql.cnf:
MySQLConf.etc.mysql.conf.d.mysqldump.cnf:
 [mysqldump]
 quick
 quote-names
 max_allowed_packet = 16M
MySQLConf.etc.mysql.mysql.conf.d.mysqld_safe_syslog.cnf:
 [mysqld_safe]
 syslog
MySQLVarLibDirListing: ['ntwine', 'private_key.pem', 'client-cert.pem', 'test12062', 'sys', 'debian-5.7.flag', 'ib_buffer_pool', 'ca.pem', 'ib_logfile0', 'ibdata1', 'mysql_upgrade_info', 'server-cert.pem', 'test7728', 'ib_logfile1', 'auto.cnf', 'performance_schema', 'public_key.pem', 'session', 'mysql', 'ca-key.pem', 'client-key.pem', 'server-key.pem']
ProcCmdline: BOOT_IMAGE=/boot/vmlinuz-4.15.0-76-generic root=UUID=326a67ad-0b90-46ba-9f18-3f1e7f52ca4a ro quiet splash vt.handoff=7
RelatedPackageVersions:
 dpkg 1.18.4ubuntu1.4
 apt 1.2.29ubuntu0.1
SourcePackage: mysql-5.7
Title: package mysql-server-5.7 5.7.29-0ubuntu0.16.04.1 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1
UpgradeStatus: No upgrade log present (probably fresh install)
upstart.mysql.override: manual

[Christian Ehrhardt  (paelzer)]

From your log:
mysql_upgrade: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) while connecting to the MySQL server

I'd call it a dup of bug 1592669, but that is meant to be fixed.
Why does it show up "again".

Subscribing Lars and Robie to take a mysql-look

[Hans Rakers (hrak)]

I just ran into this same issue with 5.7.29-0ubuntu0.16.04.1. It is caused by a race condition.

It is a rather large MySQL database, so stopping & starting up the daemon takes some time. Tried 'apt-get -f install' multiple times to no avail.

After going over the code in /var/lib/dpkg/info/mysql-server-5.7.postinst, i managed to "fix" it by adding a 'sleep 5' at line 317. Sleep 3 didn't cut it, it had to be sleep 5 :D

Like so:

    if ! test_sysmaint_access; then
      invoke stop
      set_mysql_pw "debian-sys-maint" "$pass"
      invoke start
      echo "sleeping 5 secs"
      sleep 5
    fi
    # mysql_upgrade returns exit status 2 if the database is already upgraded
    # (LP: #1566406) so ignore its exit status if it is 2.
    result=0
    mysql_upgrade --defaults-file=/etc/mysql/debian.cnf || result=$?

The way to finally get rid of these issues is to add something in this script to check if the instance is ready for connections before starting mysql_upgrade.

[Lars Tangvald (lars-tangvald)]

Hi,

Thanks for the tip!

This logic has changed significantly in Ubuntu 18+ to use new MySQL
functionality to make the upgrade process more robust (using a method of
starting the server that doesn't continue until it's ready for
connections). 20.04 has MySQL 8.0, which has deprecated mysql_upgrade
(upgrades are handled automatically by the server).

Any waiting period would be an unreliable fix (some people would need
longer wait still, and for everyone else it becomes an annoying delay),
and I'm not sure it would be doable to backport major changes in the
maintainer scripts to Ubuntu 16.

--

Lars

On 20/02/2020 14:49, Hans Rakers wrote:
> I just ran into this same issue with 5.7.29-0ubuntu0.16.04.1. It is
> caused by a race condition.
>
> It is a rather large MySQL database, so stopping & starting up the
> daemon takes some time. Tried 'apt-get -f install' multiple times to no
> avail.
>
> After going over the code in /var/lib/dpkg/info/mysql-
> server-5.7.postinst, i managed to "fix" it by adding a 'sleep 5' at line
> 317. Sleep 3 didn't cut it, it had to be sleep 5 :D
>
> Like so:
>
> if ! test_sysmaint_access; then
> invoke stop
> set_mysql_pw "debian-sys-maint" "$pass"
> invoke start
> echo "sleeping 5 secs"
> sleep 5
> fi
> # mysql_upgrade returns exit status 2 if the database is already upgraded
> # (LP: #1566406) so ignore its exit status if it is 2.
> result=0
> mysql_upgrade --defaults-file=/etc/mysql/debian.cnf || result=$?
>
> The way to finally get rid of these issues is to add something in this
> script to check if the instance is ready for connections before starting
> mysql_upgrade.
>

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Hans' race seems valid. I don't think it's necessarily the case that Maris had exactly the same problem. There's not enough detail in the logs to determine that for sure. But to make progress, I think it's OK to assume this is the case.

As Lars said, this is fixed in 18.04 onwards. I don't think a fix to 16.04 is warranted due to the regression risk. The workaround for anyone affected was described by Hans in comment 3.

If anyone disagrees, please start by addressing Lars' concern in comment 4 with a specific patch.

[Maris Nikos (nickmaris)]

To clarify I am not technical and I just know it showed an error :)

On Mon, Feb 24, 2020, 19:35 Robie Basak <email address hidden> wrote:

> Thank you for taking the time to report this bug and helping to make
> Ubuntu better.
>
> Hans' race seems valid. I don't think it's necessarily the case that
> Maris had exactly the same problem. There's not enough detail in the
> logs to determine that for sure. But to make progress, I think it's OK
> to assume this is the case.
>
> As Lars said, this is fixed in 18.04 onwards. I don't think a fix to
> 16.04 is warranted due to the regression risk. The workaround for anyone
> affected was described by Hans in comment 3.
>
> If anyone disagrees, please start by addressing Lars' concern in comment
> 4 with a specific patch.
>
> ** Changed in: mysql-5.7 (Ubuntu)
> Status: New => Fix Released
>
> ** Also affects: mysql-5.7 (Ubuntu Xenial)
> Importance: Undecided
> Status: New
>
> ** Changed in: mysql-5.7 (Ubuntu Xenial)
> Status: New => Won't Fix
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1863318
>
> Title:
> mysql-server-5.7.postinst races daemon start with mysql_upgrade
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/mysql-5.7/+bug/1863318/+subscriptions
>