Upstart does not successfully start mysql on boot

[Will Brooks]

After upgrading from 9.10 to 10.4 mysql no longer starts on boot. Manually running start mysql starts the server just fine. However, this is not an option because this is a production server. As a workaround I disabled the upstartlink in /etc/init.d for mysql and copied the old 9.10 mysql init script. Problem temporarily solved.

[Will Brooks (willbrooks)]

I believe this thread is relevant:

http://ubuntuforums.org/showthread.php?t=1470738

Their solution is not adequate for me because I do not wish to bind mysql to eth0.

[GerryG (kanab2004)]

As shown in
http://ubuntuforums.org/showthread.php?t=1470738
many people are effected by this, if mysqld must be bound to an interface other than "lo". Most of the problems appear to results from a timing issue, where upstart attempts to start the mysqld daemon before the specific network interface is available. Towards this, another posted workaround changes the condition in /etc/init/mysql from:
  start on (net-device-up and local-filesystems)
to
  start on (net-device-up IFACE=eth0 and local-filesystems)

Obviously, this is not good because it conditions starting mysqld on a specific interface, and any future changes there would break unless you remembered to change this script. A better solution would be a conditional that required ALL network interfaces to be up before starting mysqld. While this might impose a slight additional delay in some systems, it would be robust. I was unable to find such a conditional.

[Will Brooks (willbrooks)]

GerryG, you mention this but I wanted to confirm that it worked for me.

I changed /etc/init/mysql.conf

start on (net-device-up
          and local-filesystems
         and runlevel[2345])
stop on runlevel [016]

to

start on (net-device-up IFACE=lo and local-filesystems)
stop on runlevel [016]

This worked for me and there are probably caveats I haven't had problems with yet. Also of note I removed "and runlevel[2345]" because on my system (a production server that only boots to runlevel 3) mysql will not start with this line present. Not sure if this is an issue for systems that boot to runlevel 5.

[GerryG (kanab2004)]

Will Brooks,
I had assumed that "net-device-up" conditioned on only the first, which would likely be "lo". But, perhaps not. I'll try that IFACE=lo. The following explains my reasoning.

Could you tell me your bind for mysqld in /etc/mysql/my.cnf, from the four options below? There are some slight differences between them, depending on your config elsewhere and the resulting timing in your specific system, together with the use of TCP versus a Unix socket. Certain combinations can produce a somewhat similar failure to start mysql on boot, only slightly related to the script changes you made. In one server here, mysql is only used internally, and I bind to localhost. That one does not require any changes to /etc/init/mysql.conf. The other requires remote mysql access, binds to the machine name, and needs the mysql.conf changes I mentioned. The interaction between different binds and the upstart changes has produced some confusion in what is needed to fix this problem in any specific system, as most people do not supply enough information and the diagnostics are poor.

localhost
127.0.0.1
{specific IP}
{name that is host or DNS resolved to IP}

[Will Brooks (willbrooks)]

GerryG

In my.cnf I bind to 127.0.0.1 as I do not need external connections to mysql for this particular server. Thanks for the reasoning and posts.

Will

[GerryG (kanab2004)]

Will, that probably explains why your fix works. As I don't know the logic in starting the interfaces, it may be possible that lo is sometimes not the first to start and that caused your issue. Hopefully, my description may allow others to determine where their problem is at.

Also, I do have a note that localhost is a magic word, slightly different from 127.0.0.1 in this context, and that there may be a performance difference that you may want to check into.

[Max Barry (max-maxbarry)]

Experienced this problem following a simple upgrade of a few packages including mysql (see below). Afterward mysql did not start on boot.

I tried appending "IFACE=lo" to "net-device-up" as recommended above, but this made no difference. To fix I had to remove the line "and runlevel[2345]".

Not sure why this makes any difference as "runlevel" returns "N 2". But now mysql starts on boot.

My fix:

--- /etc/init/mysql.conf.bak 2010-07-28 10:29:51.733012958 +1000
+++ /etc/init/mysql.conf 2010-07-28 10:45:11.099834819 +1000
@@ -4,8 +4,7 @@
 author "Mario Limonciello <email address hidden>"

 start on (net-device-up
- and local-filesystems
- and runlevel[2345])
+ and local-filesystems)
 stop on runlevel [016]

From /var/log/apt/history.log:

Start-Date: 2010-07-27 15:32:25
Install: samba-common-bin (3.4.7~dfsg-1ubuntu3.1)
Upgrade: libkrb5-3 (1.8.1+dfsg-2ubuntu0.1, 1.8.1+dfsg-2ubuntu0.2), libkrb5support0 (1.8.1+dfsg-2ubuntu0.1, 1.8.1+dfsg-2ubuntu0.2), gparted (0.5.1-1ubuntu2, 0.5.1-1ubuntu3), mysql-client-core-5.1 (5.1.41-3ubuntu12.3, 5.1.41-3ubuntu12.4), apt-transport-https (0.7.25.3ubuntu9, 0.7.25.3ubuntu9.1), xserver-common (1.7.6-2ubuntu7.2, 1.7.6-2ubuntu7.3), mysql-server-5.1 (5.1.41-3ubuntu12.3, 5.1.41-3ubuntu12.4), python-apt (0.7.94.2ubuntu6.1, 0.7.94.2ubuntu6.2), mysql-client-5.1 (5.1.41-3ubuntu12.3, 5.1.41-3ubuntu12.4), mysql-common (5.1.41-3ubuntu12.3, 5.1.41-3ubuntu12.4), xserver-xorg-core (1.7.6-2ubuntu7.2, 1.7.6-2ubuntu7.3), libsmbclient (3.4.7~dfsg-1ubuntu3, 3.4.7~dfsg-1ubuntu3.1), libk5crypto3 (1.8.1+dfsg-2ubuntu0.1, 1.8.1+dfsg-2ubuntu0.2), apt-utils (0.7.25.3ubuntu9, 0.7.25.3ubuntu9.1), mysql-server-core-5.1 (5.1.41-3ubuntu12.3, 5.1.41-3ubuntu12.4), apt (0.7.25.3ubuntu9, 0.7.25.3ubuntu9.1), firefox (3.6.6+nobinonly-0ubuntu0.10.04.1, 3.6.8+build1+nobinonly-0ubuntu0.10.04.1), gdm (2.30.2.is.2.30.0-0ubuntu2, 2.30.2.is.2.30.0-0ubuntu3), mysql-server (5.1.41-3ubuntu12.3, 5.1.41-3ubuntu12.4), firefox-gnome-support (3.6.6+nobinonly-0ubuntu0.10.04.1, 3.6.8+build1+nobinonly-0ubuntu0.10.04.1), samba (3.4.7~dfsg-1ubuntu3, 3.4.7~dfsg-1ubuntu3.1), libwbclient0 (3.4.7~dfsg-1ubuntu3, 3.4.7~dfsg-1ubuntu3.1), firefox-3.5 (3.6.6+nobinonly-0ubuntu0.10.04.1, 3.6.8+build1+nobinonly-0ubuntu0.10.04.1), libmysqlclient16 (5.1.41-3ubuntu12.3, 5.1.41-3ubuntu12.4), libfreetype6 (2.3.11-1ubuntu2, 2.3.11-1ubuntu2.1), samba-common (3.4.7~dfsg-1ubuntu3, 3.4.7~dfsg-1ubuntu3.1), firefox-branding (3.6.6+nobinonly-0ubuntu0.10.04.1, 3.6.8+build1+nobinonly-0ubuntu0.10.04.1), libgssapi-krb5-2 (1.8.1+dfsg-2ubuntu0.1, 1.8.1+dfsg-2ubuntu0.2)
End-Date: 2010-07-27 15:35:41

[GerryG (kanab2004)]

Max,
curious...I should have mentioned this earlier.

"and runlevel[2345]"

I checked the original installed scripts in /etc/init/mysql.conf for three machines and none of them have that conditional. However, some other scripts do use it, and it apparently works there. Perhaps this is another issue to look out for, though as you noted, it looks like it shouldn't effect it. Looking further, it appears that processes managed by the new upstart no longer reflect their runlevel in sysv-rc-conf, but apparently (maybe) use that conditional in the /etc/init scripts.

I was previously unable to find much documentation on the upstart changes. Looks like I'll have to revisit that. At this point this bug thread is getting muddled.

Ubuntu 10.04.1 LTS
Linux 2.6.32-24-generic #38-Ubuntu SMP Mon Jul 5 09:20:59 UTC 2010 x86_64 GNU/Linux
Linux 2.6.32-21-generic-pae #32-Ubuntu SMP Fri Apr 16 09:39:35 UTC 2010 i686 GNU/Linux

[cdenley (c-denley)]

"and runlevel[2345]"

should be

"and runlevel [2345]"

You need the space.

[Will Brooks (willbrooks)]

Ok, upgraded my server today and seems like the problem is fixed.

Start-Date: 2010-08-02 18:16:54
Upgrade: mysql-client (5.1.41-3ubuntu12.5, 5.1.41-3ubuntu12.6), mysql-client-core-5.1 (5.1.41-3ubuntu12.5, 5.1.41-3ubuntu12.6), mysql-server-5.1 (5.1.41-3ubuntu12.5, 5.1.41-3ubuntu12.6), mysql-client-5.1 (5.1.41-3ubuntu12.5, 5.1.41-3ubuntu12.6), mysql-common (5.1.41-3ubuntu12.5, 5.1.41-3ubuntu12.6), libmysqlclient-dev (5.1.41-3ubuntu12.5, 5.1.41-3ubuntu12.6), mysql-server-core-5.1 (5.1.41-3ubuntu12.5, 5.1.41-3ubuntu12.6), mysql-server (5.1.41-3ubuntu12.5, 5.1.41-3ubuntu12.6), libmysqlclient16 (5.1.41-3ubuntu12.5, 5.1.41-3ubuntu12.6)
End-Date: 2010-08-02 18:17:52

[Monkey (monkey-libre)]

I´ve closed this bug seeing the last comment.
Thank You for making Ubuntu better.

[GerryG (kanab2004)]

To Monkey:
I agree on closing. Mine also seems to be working as expected now.