mysqld fails to start on upgrade 7.10->8.04

[Robin Joinson]

The problem seems to be caused by Apparmor. For example in /var/log/syslog:

Mar 19 20:04:39 lyra kernel: [ 6979.352113] audit(1205957079.490:28): operation=
"inode_permission" request_mask="r::" denied_mask="r::" name="/opt/share/mysql/english/errmsg.sys" pid=3861 profile="/usr/sbin/mysqld" namespace="default"
Mar 19 20:04:39 lyra mysqld[3863]: 080319 20:04:39 [ERROR] Can't find messagefile '/usr/share/mysql/english/errmsg.sys'
Mar 19 20:04:39 lyra mysqld[3863]: 080319 20:04:39 [ERROR] AbortingMar 19 20:04:39 lyra mysqld[3863]: Mar 19 20:04:39 lyra mysqld_safe[3865]: ended

[deleted AppArmor removal example]
please instead of removing apparmor, just disable a profile:
  sudo touch /etc/apparmod.d/disable/usr.sbin.mysqld
  sudo /etc/init.d/apparmor reload

the mysql server components then upgraded OK and the server started.

I don't know about Apparmor, but from the looks of syslog, it looks as though there was a problem with the apparmor acls; I noticed that during the mysql server install, a script appears to load up the apparmor rules and these rules appear to be preventing a clean mysqld upgrade.

Previous upgrades of mysqld under Gutsy on this machine have been problem free.

[Mathias Gug (mathiaz)]

On Wed, Jun 04, 2008 at 07:40:37PM -0000, Launchpad Bug Tracker wrote:
> You have been subscribed to a public bug:
>
> The problem seems to be caused by Apparmor. For example in
> /var/log/syslog:
>
> Mar 19 20:04:39 lyra kernel: [ 6979.352113] audit(1205957079.490:28): operation=
> "inode_permission" request_mask="r::" denied_mask="r::" name="/opt/share/mysql/english/errmsg.sys" pid=3861 profile="/usr/sbin/mysqld" namespace="default"
> Mar 19 20:04:39 lyra mysqld[3863]: 080319 20:04:39 [ERROR] Can't find messagefile '/usr/share/mysql/english/errmsg.sys'
> Mar 19 20:04:39 lyra mysqld[3863]: 080319 20:04:39 [ERROR] AbortingMar 19 20:04:39 lyra mysqld[3863]: Mar 19 20:04:39 lyra mysqld_safe[3865]: ended
>

Which version of mysql were you upgrading from ? Did you install
apparmor-profiles ?

> Did
>
> apt-get remove --purge apparmor
> apt-get -f install
>
> the mysql server components then upgraded OK and the server started.
>

You don't have to purge apparmor - you can put mysqld profile in
complain mode. See https://wiki.ubuntu.com/DebuggingApparmor for more
information.

  status incomplete

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

[Robin Joinson (robin-joinson)]

I didn't really want apparmor to be installed, but thanks for the tip. Nope, I didn't install apparmor-profiles; I don't believe apparmor is necessary for running mysql on Ubuntu.

I was upgrading the whole system, to the best of my recollection, the version numbers were from 5.0.45 under Gutsy to 5.0.51a under Hardy.

The error arose after I clicked on the 'upgrade system' icon on the desktop. It seemed at the time that either the apparmor or the mysql package (or both) did not have the correct dependency settings and/or upgrade checks.

If you are making a mysql server upgrade (or install) depend in any way on apparmor (I think that is a bad idea) you need to have a mysql-server package dependency set, to ensure that the correct apparmor components are present on the system for mysql. The mysql-server package's pre-install script should also check that the correct apparmour configuration settings are made to allow the mysql server to start. A more informative post-install script message wouldn't do any harm either! Likewise, the apparmor package needs to check if there are any services for which it needs to install special configuration files, so it doesn't prevent them from starting.

IMHO, the package management system ought to make it unnecessary for a sysadmin to have to figure out: (1) That there is something new on the system, namely apparmor, preventing a mysqld startup. (2) That an apparmor 'mysql profile' must be installed, sought out and put into 'complain mode' before the mysql server will start.

This bug frightened me. For a minute, I thought I had been magically spirited away to my Solaris box!

Many thanks for looking into this, by the way. Your efforts are very much appreciated!

--Robin

[Kees Cook (kees)]

If your mysql is installed out of /opt, it is not an Ubuntu version. :)

[Robin Joinson (robin-joinson)]

MySQL isn't installed in /opt. The myslqd is the Ubuntu one in /usr/sbin/mysqld. On this system, /usr/share is symlinked to /opt/share and has been since the system was installed (and prior to the mysql package being installed). The mysql package was upgraded a number of times with apt-get dist-upgrade and there was no problem until app armour came along. Since I purged apparmor, there hasn't been another problem. I think you have a problem with package dependencies. I don't think that the symlinked /usr/share is the cause of the problem.

[Mathias Gug (mathiaz)]

On Fri, Jun 06, 2008 at 10:59:13PM -0000, Robin Joinson wrote:
> MySQL isn't installed in /opt. The myslqd is the Ubuntu one in
> /usr/sbin/mysqld. On this system, /usr/share is symlinked to /opt/share
> and has been since the system was installed (and prior to the mysql
> package being installed). The mysql package was upgraded a number of
> times with apt-get dist-upgrade and there was no problem until app
> armour came along. Since I purged apparmor, there hasn't been another
> problem. I think you have a problem with package dependencies. I don't
> think that the symlinked /usr/share is the cause of the problem.

That's exactly the problem. AppArmor follows symlinks when applying
the rules in a profile. You'd either have to update the mysqld profile
to allow access to /opt/share or remove the symlink.

--
Mathias Gug
Ubuntu Developer http://www.ubuntu.com

[Robin Joinson (robin-joinson)]

Ok, so apparmor doesn't like some local condition on my machine. The substance of my bug report is still the same.

The package maintainer should configure the apparmor package so that it does these things:

- at the pre-install phase, check for affected services and warn the admin when a service doesn't meet the conditions required by the package. Consider bailing out of the install at this stage, if possible/appropriate.
- attempt to ensure, at the install phase, that all conditions are met, or if this is not possible, at least ensure that a running service isn't crippled, for example by automating the step to put apparmor into 'complain mode' for that service.
- check at the post-install phase that all conditions were, in fact, met and if not, warn the admin.

If the package maintainer doesn't do this, the admin has to spend time figuring out that:

- a new kind of kernel message in the syslog is caused by a new package (apparmor).
- apparmor is writing these messages because some of its defaults for a service profile do not match what is on the system.
- apparmor is denying access to file/s essential for a service to start.
- a default 'apparmor profile' must be updated somehow, to accommodate the state of the system (I didn't get this far before I purged apparmor).

Since the preconditions for file access are known by the package maintainer, why can't the package check these things?

Anyhow, thanks for the information about apparmor. It looks as though it might be a very useful security tool, but I would want to read about it in depth and consider the impact very carefully before running it on a production box.

All the best,

--Robin

[Jamie Strandboge (jdstrand)]

The package does quite a bit to make sure things don't break. Please see:
https://wiki.ubuntu.com/ApparmorProfileMigration

Unfortunately, there is no way to reasonably detect every situation a
user might have changed his or her system, however, the above migration
strategies catch most cases (by far). As such, the decision was made to
protect users where possible by default. Documentation is found in the
package, wiki, and apparmor protections are mentioned in the release
notes.

[Robin Joinson (robin-joinson)]

Ok Jamie,

From my point of view, I've had the opportunity to raise and discuss all my concerns and I believe the position of the Ubunu team is that things will stay the same in the Apparmor package.

I guess my 'bug' is closed.

Nevertheless, many thanks to the team for taking the time and effort to correspond with me about this.

Best Regards,

--Robin