mysql will not start after 5.5 to 5.6 upgrade

[km]

I just did a double Ubuntu upgraded 14.04->14.10->15.04, which sent me from mysql server 5.5.43 to 5.6.24.

Now if I run "service mysql start" it hangs and the mysql log shows

150516 09:47:00 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

/usr/sbin/mysqld: Can't read dir of '/etc/mysql/mysql.conf.d/' (Errcode: 13 - Permission denied)
Fatal error in defaults handling. Program aborted
150516 09:47:00 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Yet as far as permission

# ls -ld /etc/mysql/mysql.conf.d
drwxr-xr-x 2 root root 4096 May 15 03:56 /etc/mysql/mysql.conf.d
# ls -l /etc/mysql/mysql.conf.d
total 8
-rw-r--r-- 1 root root 3027 Feb 17 14:51 mysqld.cnf
-rw-r--r-- 1 root root 21 Feb 11 10:10 mysqld_safe_syslog.cnf
---
ApportVersion: 2.17.2-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
DistroRelease: Ubuntu 15.04
InstallationDate: Installed on 2012-12-14 (884 days ago)
InstallationMedia: Ubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.5)
Logs.var.log.daemon.log:

MySQLConf.etc.mysql.conf.d.mysql.cnf: [mysql]
MySQLConf.etc.mysql.conf.d.mysqld.safe.syslog.cnf:
 [mysqld_safe]
 syslog
MySQLConf.etc.mysql.conf.d.mysqldump.cnf:
 [mysqldump]
 quick
 quote-names
 max_allowed_packet = 16M
MySQLConf.etc.mysql.mysql.conf.d.mysqld.safe.syslog.cnf:
 [mysqld_safe]
 syslog
MySQLVarLibDirListing: False
NonfreeKernelModules: nvidia
Package: mysql-5.6 (not installed)
ProcCmdline: BOOT_IMAGE=/boot/vmlinuz-3.19.0-17-generic root=UUID=0659f1b3-4361-442f-bf62-b9b3393a3b95 ro quiet splash crashkernel=384M-:128M crashkernel=384M-:128M crashkernel=384M-:128M crashkernel=384M-:128M crashkernel=384M-:128M vt.handoff=7
ProcVersionSignature: Ubuntu 3.19.0-17.17-generic 3.19.6
Tags: vivid
Uname: Linux 3.19.0-17-generic x86_64
UpgradeStatus: Upgraded to vivid on 2015-05-15 (3 days ago)
UserGroups: adm bumblebee cdrom debian-tor dip disk libvirtd lpadmin plugdev sambashare sudo tape vboxusers
_MarkForUpload: True
modified.conffile..etc.apparmor.d.usr.sbin.mysqld: [modified]
mtime.conffile..etc.apparmor.d.usr.sbin.mysqld: 2013-08-18T11:48:56.903852

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command, as it will automatically gather debugging information, in a terminal:

apport-collect 1455773

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

Please also attach /etc/apparmor.d/usr.sbin.mysqld (if the apport-collect command above does not already attach it) and check /var/log/kern.log for any apparmor denials, and attach those here if there are any.

[km (km-mathcs)]

apport information

[km (km-mathcs)]

apport information

[km (km-mathcs)]

apport information

[km (km-mathcs)]

apport information

[km (km-mathcs)]

apport information

[km (km-mathcs)]

apport information

[km (km-mathcs)]

apport information

[km (km-mathcs)]

apport information

[Robie Basak (racb)]

Thanks. You're getting AppArmor denials there for /etc/mysql/mysql.conf.d/ even though the AppArmor profile in /etc/apparmor.d/usr.sbin.mysqld looks correct. Is there some reason this hasn't been reloaded, or do you have any local overrides present? Have you rebooted since you upgraded?

[km (km-mathcs)]

It does in fact appear to be an apparmor problem, the kern.log has multiple

May 18 08:51:39 orac kernel: [79075.568386] audit: type=1400 audit(1431953499.358:167): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/etc/mysql/mysql.conf.d/" pid=7960 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=122 ouid=0

[km (km-mathcs)]

i have rebooted several times. I'm not aware of local overrides, certainly not done manually.

I do have one concern. During the upgrade it asked me several times about a mysql password which I declined to enter.

[km (km-mathcs)]

I found the problem. Apparmor still had the 5.5 profile which survived the upgrade. However, the mysql config directory has moved and needed the 5.6 profile for Apparmor.

[Robie Basak (racb)]

@km

Thank you for reporting back. Does this mean that it was a problem specific to customisations you had made to your system, or is this a problem that would apply to all users? If the latter, do you now understand how I might be able to reproduce the issue?

[km (km-mathcs)]

Can't be sure, but I think its a general problem. If you want to try and reproduce it try:

* Start with 14.04 with mysql server installed, configured, and running.
* /etc/apparmor.d should contain a profile file usr.sbin.mysqld
* Do the upgrade first to 14.10 and then to 15.04
* When doing the upgrades you will be asked if you want to supply a password for mysql server. Just say continue
(motivation, you are not trying to change mysqld password)
* check /etc/apparmor.d. If it has the old usr.sbin.mysqld (and an uninstalled dist file) you have hit the "bug"

The old apparmor (5.5) profile cannot work with the updated (5.6) mysqld.

[Robie Basak (racb)]

> If it has the old usr.sbin.mysqld (and an uninstalled dist file) you have hit the "bug"

By "uninstalled dist file", do you mean usr.sbin.mysqld.dpkg-dist? I expect your scenario to occur if the user previously modifies the apparmor profile. In this case, on upgrade, the user should be prompted to merge customisations manually and asked whether to use the old or new file. In the case of a non-interactive upgrade, the old file will be used, and the new one left renamed so that the user can fix it later manually.

If the old file is used, then you will hit the issue you've described, but this isn't a bug because Debian policy says that old customised configurations must not be blindly clobbered and we have no finer-grained way to bring forward those customisations.

Your report said (automatically generated by apport):

modified.conffile..etc.apparmor.d.usr.sbin.mysqld: [modified]
mtime.conffile..etc.apparmor.d.usr.sbin.mysqld: 2013-08-18T11:48:56.903852

So I think this is exactly what happened to you when you or some script changed the old apparmor profile back on 18 August 2013. So I'll mark this bug Invalid on that basis.

[km (km-mathcs)]

The old one was not modified to the best of my knowledge. I didn't even know there was one or that apparmor was involved in mysql.