postinst runs mysql_upgrade before server is properly started ( Can't connect [...] '/var/run/mysqld/mysqld.sock)

[Pas]

Hello,

The corresponding apt error message looks like this:

...
Do you want to continue? [Y/n]
Setting up mysql-server-5.7 (5.7.17-0ubuntu0.16.04.1) ...
Renaming removed key_buffer and myisam-recover options (if present)
mysql_upgrade: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) while connecting to the MySQL server
Upgrade process encountered error and will not continue.
mysql_upgrade failed with exit status 11
dpkg: error processing package mysql-server-5.7 (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of mysql-server:
 mysql-server depends on mysql-server-5.7; however:
  Package mysql-server-5.7 is not configured yet.

dpkg: error processing package mysql-server (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                                                                                                          Errors were encountered while processing:
 mysql-server-5.7
 mysql-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

The workaround is to put a `sleep 3` on the 320. line of mysql-server-5.7.postinst script (right before `mysql_upgrade --defaults-file=/etc/mysql/debian.cnf || result=$?` )

The root cause seems to be that invoke-rc.d does not execute systemctl --wait, and systemctl without wait doesn't wait until the database server daemon is up and listening.

And of course it could be mitigated by using a systemd socket for mysql.

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

I think we have seen some other reports of mysql_upgrade failures like this but so far had been unable to reproduce and did not have a technical explanation. Thank you very much for providing this. We can probably duplicate other reports to this bug.

Separately, we're overhauling how mysql_upgrade is called due to bug 1592669. Our planned solution for that bug is to no longer use invoke-rc.d to start mysqld before running mysql_upgrade at all. Instead we'll run a "private" daemon that runs with networking and ACLs disabled in a root-only private socket so that mysql_upgrade can then run unhindered, and then shut it down again. So when that bug is resolved, I think this (looks valid) bug will become moot.

I subscribed Lars for his attention.

Thank you again for the report.

[Pas (pasthelod)]

Thanks for the quick reply and attention.

Yes, I saw a few similar reports, but I felt opening a separate and terser bug might be better than just commenting on one of them. And while looking at them I saw the plan regarding the upgrade process. Yes, a private instance will be much better.

Feel free to close this, I mainly reported this so maybe others will find the workaround useful.

[Robie Basak (racb)]

On Wed, Mar 08, 2017 at 06:41:58PM -0000, Pas wrote:
> Feel free to close this, I mainly reported this so maybe others will
> find the workaround useful.

I think this bug should remain open until we either fix the race or we
replace the daemon start with the private one as planned. Since the bug
is currently still present and valid, and is fundamentally a different
underlying issue from the other one. For example, if we struggled to fix
the policy-rc.d issue for whatever reason, we might still be introduce a
"wait for mysqld to be ready or time out" call before running
mysql_upgrade, fixing this bug but not the other.

[Lars Tangvald (lars-tangvald)]

Yeah, the fix for the policy-rc.d issue should fix this as well.

Are you able to reproduce this reliably, or is it just occasional?

[Pas (pasthelod)]

I tried it about 2-3 times, before confirming that it won't go away by itself.

Basically the sequence of events was: apt full-upgrade, it failed, then dpkg --configure -a which of course failed too, apt full-upgrade again, it failed again, then I scrolled up to read the error, I ran mysql_upgrade manually, it ran without error, so I tried the configure again, but it failed. Then I confirmed that I have 5.7 up and running, but the postinst is just not having it. So then I just did the usual strace -f -e execve -s 300 dpkg --configure -a, and then I edited the postinst file, and filed the bug after it worked.

[Christian Ehrhardt  (paelzer)]

I've seen some of these kind of issues as well.

The reports I've seen so far often end with retrying until it works, yet I like the temporary workaround you outlined with the sleep in the postinst for those where restarting won't help.

I'll slightly modify the subject to make this more "findable" and tagged it "triage" to make this the spot to dup similar bugs to.

I look forward to the reworked upgrade handling.

[OMAR MAGALHAES (omarmagalhaes)]

At first a was trying update my password becouse I have problem in authenticate phpmyadmin, and after with mysql workbench.
as I am learnig use linux ubuntu, maybe this related problem was my fault.

Thank you