mysql does not always create a Unix socket on startup

[Jason Grammenos]

ubuntu 14.04 lts

after rebooting the system, mysql fails to start correctly and does no create the appropriate .sock file.
the mysqld process is running.

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

mysql should restart correctly after a system reboot

this issue does not occur everytime, and seems like a potential race condition issue.
restarting the mysql service after boot fixes the issue (service mysql restart)

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: mysql-server 5.5.41-0ubuntu0.14.04.1
ProcVersionSignature: Ubuntu 3.13.0-46.75-generic 3.13.11-ckt15
Uname: Linux 3.13.0-46-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.7
Architecture: amd64
Date: Thu Feb 26 10:39:25 2015
Logs.var.log.daemon.log:

MySQLConf.etc.mysql.conf.d..keepme:
MySQLConf.etc.mysql.conf.d.mysqld.safe.syslog.cnf:
 [mysqld_safe]
 syslog
MySQLVarLibDirListing: False
PackageArchitecture: all
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
SourcePackage: mysql-5.5
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Simon Déziel (sdeziel)]

This problem is seriously harming our production where we never know if MySQL will be functional after a reboot. The problem seems to happen 50% of the time.

Here are some observations when the problem occurs:

# status mysql
mysql start/running, process 825

# ll /var/run/mysqld/
total 0
drwxr-xr-x 2 mysql root 40 Feb 27 09:28 ./
drwxr-xr-x 18 root root 600 Feb 27 09:35 ../

# ps aux | grep mysql
mysql 825 0.0 2.5 624108 52628 ? Ssl 09:27 0:00 /usr/sbin/mysqld
root 2040 0.0 0.0 11740 920 pts/0 S+ 09:36 0:00 grep --color=auto mysql

# mysqladmin ping
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!

[Simon Déziel (sdeziel)]

Further testing showed this issue only manifests during a reboot.

Doing a restart loop after bootup does *not* trigger the issue.

[Simon Déziel (sdeziel)]

# System reboot signal

150303 10:15:04 [Note] /usr/sbin/mysqld: Normal shutdown
150303 10:15:04 [Note] Event Scheduler: Purging the queue. 0 events
150303 10:15:04 InnoDB: Starting shutdown...
150303 10:15:05 InnoDB: Shutdown completed; log sequence number 1595745
150303 10:15:05 [Note] /usr/sbin/mysqld: Shutdown complete

# Machine is back

150303 10:15:26 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150303 10:15:26 [Note] Plugin 'FEDERATED' is disabled.
150303 10:15:26 InnoDB: The InnoDB memory heap is disabled
150303 10:15:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150303 10:15:26 InnoDB: Compressed tables use zlib 1.2.8
150303 10:15:26 InnoDB: Using Linux native AIO
150303 10:15:26 InnoDB: Initializing buffer pool, size = 128.0M
150303 10:15:26 InnoDB: Completed initialization of buffer pool
150303 10:15:26 InnoDB: highest supported file format is Barracuda.
150303 10:15:26 InnoDB: Waiting for the background threads to start
150303 10:15:26 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150303 10:15:26 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead.
150303 10:15:26 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
150303 10:15:26 [Note] Plugin 'FEDERATED' is disabled.
150303 10:15:26 InnoDB: The InnoDB memory heap is disabled
150303 10:15:26 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150303 10:15:26 InnoDB: Compressed tables use zlib 1.2.8
150303 10:15:26 InnoDB: Using Linux native AIO
150303 10:15:26 InnoDB: Initializing buffer pool, size = 128.0M
150303 10:15:26 InnoDB: Completed initialization of buffer pool
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
150303 10:15:26 InnoDB: Retrying to lock the first data file
150303 10:15:27 InnoDB: 5.5.41 started; log sequence number 1595745
150303 10:15:27 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
150303 10:15:27 [Note] - '127.0.0.1' resolves to '127.0.0.1';
150303 10:15:27 [Note] Server socket created on IP: '127.0.0.1'.
150303 10:15:27 [Note] Event Scheduler: Loaded 0 events
150303 10:15:27 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.41-0ubuntu0.14.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
InnoDB: Unable to lock ./ibdata1, error: 11

# "error: 11" goes on and on

150303 10:17:08 InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
150303 10:17:08 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
150303 10:17:08 InnoDB: Could not open or create data files.
15030...

[Simon Déziel (sdeziel)]

Comment #5 contains the steps needed to reproduce the issue. I'm now attaching the error.log of MySQL.

[Marek Vybíral (cccp1989)]

This is only workaround and probably not the most elegant solution, but it's working. I simply added 'service mysql restart' at the end of mysql startup script in /etc/init/mysql.conf.