Apparmor refuses mysqld open on /etc/mysql/my.cnf.migrated causing it to fail to start

[Richard]

[Triage Notes]

This bug tracks reports of the following AppArmor denial:

Aug 7 18:22:17 c505 kernel: [ 1485.490459] audit: type=1400 audit(1470619337.926:98): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/etc/mysql/my.cnf.migrated" pid=11219 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=109 ouid=0

If you are seeing a denial of /etc/mysql/my.cnf.migrated in this manner, then the underlying issue may affect you too.

If you are seeing a denial of some other file, then you have some other AppArmor-related problem that is *not* the issue tracked in this bug. If you believe that your failure is a bug in Ubuntu, please look for other bugs and file a new bug if there isn't a matching one that tracks your different issue.

[Original Desription]

SYMPTOM
mysql fails to start

IMPACT
no sql for you

DIAGNOSE:
service mysql start
Job for mysql.service failed because the control process exited with error code. See "systemctl status mysql.service" and "journalctl -xe" for details.

- systemd has all sort of craziness going on so stop that
service mysql stop

- look at what apparmor is up to
tail -F /var/log/syslog | grep -i apparmor

- then start again
service mysql start

output is:
- mysql cannot read /etc/mysql
Aug 7 18:22:17 c505 kernel: [ 1485.490459] audit: type=1400 audit(1470619337.926:98): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/etc/mysql/my.cnf.migrated" pid=11219 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=109 ouid=0

GET_AROUND:
sudo nano /etc/apparmor.d/usr.sbin.mysqld
line 25 after:
# Allow config access
  /etc/mysql/** r,
-add line:
  /etc/mysql/* r,

systemctl restart apparmor.service

service mysql stop
service mysql start

- problem is gone

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: mysql-server-5.7 5.7.13-0ubuntu0.16.04.2
ProcVersionSignature: Ubuntu 4.4.0-31.50-generic 4.4.13
Uname: Linux 4.4.0-31-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
Date: Sun Aug 7 18:31:32 2016
Logs.var.log.daemon.log:

MySQLConf.etc.mysql.conf.d.mysql.cnf: [mysql]
MySQLConf.etc.mysql.conf.d.mysqld_safe_syslog.cnf:
 [mysqld_safe]
 syslog
MySQLConf.etc.mysql.conf.d.mysqldump.cnf:
 [mysqldump]
 quick
 quote-names
 max_allowed_packet = 16M
MySQLConf.etc.mysql.mysql.conf.d.mysqld_safe_syslog.cnf:
 [mysqld_safe]
 syslog
MySQLVarLibDirListing: ['debian-5.0.flag', 'ib_buffer_pool', 'test', 'ib_logfile1', 'xpc21.err', 'auto.cnf', 'mysql_upgrade_info', 'bugzilla', 'zabbix', 'debian-5.5.flag', 'debian-5.1.flag', 'redmine_default', 'roundcube', 'ib_logfile0', 'ibtmp1', 'sys', 'mysql', 'ibdata1', 'bugs', 'debian-5.7.flag', 'owncloud', 'performance_schema']
ProcEnviron:
 SHELL=/bin/bash
 TERM=xterm
 PATH=(custom, user)
 LANG=en_US.UTF-8
 LANGUAGE=en_US.UTF-8
SourcePackage: mysql-5.7
UpgradeStatus: Upgraded to xenial on 2016-06-13 (56 days ago)
modified.conffile..etc.apparmor.d.usr.sbin.mysqld: [modified]
mtime.conffile..etc.apparmor.d.usr.sbin.mysqld: 2016-08-07T18:24:24.264357

[Robie Basak (racb)]

Thank you for your report.

I believe this has happened because you have a locally modified /etc/apparmor.d/usr.sbin.mysqld. The one we ship has "/etc/mysql/** r" which includes /etc/my.cnf.migrated.

If you still think this is a bug in Ubuntu, please explain, provide exact steps to reproduce and then change the bug status back to New. Thanks!

[Richard (ismail-a)]

I did change it when there were other apparmor bugs.

The problem is that this line, that I have, does not work:
/etc/mysql/** r

Adding this line fixes the problem:
  /etc/mysql/* r,

I don't know apparmor well, but it appears two stars do not work here.

[Robie Basak (racb)]

According to http://wiki.apparmor.net/index.php/QuickProfileLanguage#File_Globbing, /etc/mysql/** will match any file or directory in or below /etc/mysql/. So this should work. If it isn't working for you, I suspect that you have an unrelated issue that inadvertently got fixed when you edited the profile.

We haven't had any other reports claiming this. The lack of other reports suggests to me your suggestion cannot be the root cause - otherwise every user upgrading from a customised my.cnf in Trusty would be affected and this is clearly not the case.

If you disagree, then please provide exact steps to reproduce so that this can be independently verified, and then change the bug status back to New.

[Richard (ismail-a)]

That is true, if I remove my line again, it still works.

mysql-server-5.7:amd64 was updated on 160724

When my server was rebooted on 2016-08-07, mysql would not start with the following syslog output:
c505 kernel: [ 1485.490459] audit: type=1400 audit(1470619337.926:98): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/etc/mysql/my.cnf.migrated" pid=11219 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=109 ouid=0

The following commands were required to successfully start:
service mysql stop
systemctl restart apparmor.service
service mysql start

mysql was running, but after reboot (as in abrupt power failure) it would not start.

[Seth Arnold (seth-arnold)]

On Mon, Aug 08, 2016 at 09:05:14PM -0000, Richard wrote:
> The following commands were required to successfully start:
> service mysql stop
> systemctl restart apparmor.service
> service mysql start

Please note that "systemctl restart apparmor.service" is dangerous;
systemd implements "restart" as "stop ; start", which unloads apparmor
profiles, then reloads them. Whatever was running confined is no longer
confined afterwards. "reload" is safe.

Thanks

[Richard (ismail-a)]

Aha, so it will likely fail on next reboot again.

Another mysql-server troubleshooting session in my near future :)

[Michelle Liu (anonmily)]

I have the same issue, but this time, with Ubuntu Xenial 16.04 running on Vagrant, for both the ubuntu/xenial64 and the bento/ubuntu-16.04 images. Unfortunately, Richard's fix didn't work for me either, and I had no choice but to revert back to 14.04.

[Gary Mercer (gmercer2012)]

Ok Robie Basak, You can now consider the original post an official bug report. I am getting the same log entries as Richard except the fsuid for my server log is 107. This is the second time on version 16.04.1 and first time on 16.04.2 that this has happened. The only permanent fix I have found os to do a clean install of the server software and hope it goes longer than last install before it blows up.

I believe the conflict is the way apparmor deals with mysql in the start up phase at boot. One curious behavior that I have found is that there seems to be a connection with the introduction of webmin server to the queue at startup.

In the future, please do not discount a bug report simply because only one user has reported the issue. If one user has experienced the issue, then it is a near certainty that at least 10% of the user base has run into the same exact issue. They just don't have the time or experience to detail the circumstances that brought up the issue in the first place. Not all users are programmers.

[Paul Sauve (paulbgd)]

This happens to me as well, 16.04 with the following log:

Mar 2 16:39:06 paulpc kernel: [ 1622.002136] audit: type=1400 audit(1488494346.712:224): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/19772/status" pid=19772 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=121 ouid=121
Mar 2 16:39:06 paulpc kernel: [ 1622.002172] audit: type=1400 audit(1488494346.712:225): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=19772 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=121 ouid=0
Mar 2 16:39:06 paulpc kernel: [ 1622.002228] audit: type=1400 audit(1488494346.712:226): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/19772/status" pid=19772 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=121 ouid=121

[haliphax (haliphax)]

Same thing is happening to me. Just started after some package upgrades. I have never touched the AppArmor config nor the MySQL config. I wound up disabling AppArmor in the meantime just so that I can get my database server up again.

Mar 03 20:09:28 oddnetwork.org audit[4215]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/4215/status" pid=4215 comm="mysqld" requested_mask
Mar 03 20:09:28 oddnetwork.org kernel: audit: type=1400 audit(1488593368.094:695): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/4215/status" pi
Mar 03 20:09:28 oddnetwork.org audit[4215]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=4215 comm="mysqld" reques
Mar 03 20:09:28 oddnetwork.org kernel: audit: type=1400 audit(1488593368.098:696): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/n
Mar 03 20:09:28 oddnetwork.org audit[4215]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/4215/status" pid=4215 comm="mysqld" requested_mask
Mar 03 20:09:28 oddnetwork.org kernel: audit: type=1400 audit(1488593368.102:697): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/4215/status" pi
Mar 03 20:09:28 oddnetwork.org systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE
r

[Gary Mercer (gmercer2012)]

Ok. You guys have had way too long to at least post something besides "this isn't a bug" retort to the original post. BTW, that was back in August of last year. Most of us who have taken the time to look into solutions to the issue would like to know that at least some programmer somewhere in the Ubuntu universe acknowledge that there might be a problem and that it is being looked into. The downside to this is that this is what turns people off from what really is a good operating system.

[Robie Basak (racb)]

@Gary

Look, I want to help. Please see "How to Report Bugs Effectively" at http://www.chiark.greenend.org.uk/~sgtatham/bugs.html for an excellent explanation as to why I cannot help right now.

> You can now consider the original post an official bug report.

Unfortunately the bug status will have to be marked Incomplete until we have steps to reproduce the problem. I want it to be clear that developers do not have enough information to help with this right now, as they don't know how to reproduce the problem and nobody has any hunch as to what is really happening. Thus we also don't know if it is a bug in Ubuntu's mysql-5.7 packaging (or in Ubuntu at all) or not, since we don't know if it is something about how you set up the system which results in this problem.

You mentioned webmin, which is not part of Ubuntu. Is it possible that you and Richard both have webmin in common, and that it is the use of webmin that results in the problem (perhaps a race?) rather than a bug in Ubuntu?

@Paul
@haliphax

It seems that you both have a different error message. Though it is related to AppArmor, the root cause for your problem is likely to be different. It looks more likely to me that you are affected by bug 1658239 rather than this one.

[Robie Basak (racb)]

Importance -> Low since we appear to have had only three reports in six months, and it's coming up to a year since 16.04 was released. Therefore I think it's reasonable to assume that this only affects unusual end-user configurations, since typically when a bug affecting more MySQL users appears we receive orders of magnitude more reports.

Richard, Michelle or Gary: if any you can work out steps to reproduce the problem (perhaps tweaking timing as needed if it's a race), then I'd be happy to look into it further. Once you have commented with further information about how to reproduce the bug, please change the bug status to Confirmed. Otherwise, feel free to use continue using this bug as a communication point for others affected but please leave it marked Incomplete.

[Gary Mercer (gmercer2012)]

ok. The "bug" is reproducible. It has happened on every time i have installed ubuntu 16.04 (all iterations). It is quirky because if you do a reboot three times in a row, the error message goes away and apparmor allows mysql to start without any errors. on the same machine with exactly the same hardware and configurations, 16 fails, 15 fails, 14 fails half the time, and 12 runs. Every failure is a combination of apparmor and mysql. My test system is to do 4 fresh installs and configurations on each version of ubuntu.

If this is not a bug in the operating system, then I have been using the wrong definition of what a bug in software is for the last 42 years. It is a bug if it happens at all whether you can repeat the case or not. Unless I send you my machine, you cant reproduce it or we wouldnt be exchanging messages.

Bottom line, I am having a problem with Ubuntu and the way apparmor interacts with other parts of the operating system. Since Im not a linux programmer, nor do i have the time to chase this rabbit down the hole, I have two choices. One, get windows server, or two, get a new machine.

Thank you for your time you have graciously given me. I do appreciate it. Im frustrated at the issue and not you. please forgive me for being rude or offending you. It was not my intention.

Gary

Sent from my iPad

> On Mar 6, 2017, at 5:07 AM, Robie Basak <email address hidden> wrote:
>
> @Gary
>
> Look, I want to help. Please see "How to Report Bugs Effectively" at
> http://www.chiark.greenend.org.uk/~sgtatham/bugs.html for an excellent
> explanation as to why I cannot help right now.
>
>> You can now consider the original post an official bug report.
>
> Unfortunately the bug status will have to be marked Incomplete until we
> have steps to reproduce the problem. I want it to be clear that
> developers do not have enough information to help with this right now,
> as they don't know how to reproduce the problem and nobody has any hunch
> as to what is really happening. Thus we also don't know if it is a bug
> in Ubuntu's mysql-5.7 packaging (or in Ubuntu at all) or not, since we
> don't know if it is something about how you set up the system which
> results in this problem.
>
> You mentioned webmin, which is not part of Ubuntu. Is it possible that
> you and Richard both have webmin in common, and that it is the use of
> webmin that results in the problem (perhaps a race?) rather than a bug
> in Ubuntu?
>
> @Paul
> @haliphax
>
> It seems that you both have a different error message. Though it is
> related to AppArmor, the root cause for your problem is likely to be
> different. It looks more likely to me that you are affected by bug
> 1658239 rather than this one.
>
> ** Summary changed:
>
> - mysql server fails to start due to faulty apparmor profile
> + Apparmor refuses mysqld open on /etc/mysql/my.cnf.migrated causing it to fail to start
>
> ** Description changed:
>
> + [Triage Notes]
> +
> + This bug tracks reports of the following AppArmor denial:
> +
> + Aug 7 18:22:17 c505 kernel: [ 1485.490459] audit: type=1400
> + audit(1470619337.926:98): apparmor="DENIED" operation="open"
> + profile="/usr/sbin/mysqld" name="/etc/...

[Robie Basak (racb)]

On Mon, Mar 06, 2017 at 09:11:06PM -0000, Gary Mercer wrote:
> ok. The "bug" is reproducible. It has happened on every time i have
> installed ubuntu 16.04 (all iterations). It is quirky because if you do
> a reboot three times in a row, the error message goes away and apparmor
> allows mysql to start without any errors. on the same machine with
> exactly the same hardware and configurations, 16 fails, 15 fails, 14
> fails half the time, and 12 runs. Every failure is a combination of
> apparmor and mysql. My test system is to do 4 fresh installs and
> configurations on each version of ubuntu.

By "reproducible" we usually mean that someone else can reproduce the
issue given the same starting position (usually a fresh install or
pristine cloud image) and following exactly the same instructions.

It sounds like you're able to reproduce it from a fresh installation. Is
this the case? If so, it would help if you could write down the exact
steps (with no ambiguity) of exactly what you're doing from a fresh
installation to lead to the problem, and exactly what you're doing to
confirm that the problem exists. For example, don't say "install mysql",
say "type: apt-get install mysql-server-5.7". I do know how to install
mysql, but there are many ways of doing things, and I need to know
exactly which of those ways you are using for every single step.

> If this is not a bug in the operating system, then I have been using the
> wrong definition of what a bug in software is for the last 42 years. It
> is a bug if it happens at all whether you can repeat the case or not.
> Unless I send you my machine, you cant reproduce it or we wouldnt be
> exchanging messages.

Sure - it does sound like a race condition at the moment. But I still
need instructions on how to reproduce the problem even if it doesn't
occur every time.

> [...]

> Thank you for your time you have graciously given me. I do appreciate
> it. Im frustrated at the issue and not you. please forgive me for being
> rude or offending you. It was not my intention.

You're welcome. I understand it's frustrating.

[Mark Thompson (alajjana)]

I am not sure this will help anyone but i got this problem to go away via: (I cannot cut and paste my logs so you may have to fill in the middle sections)

I got the apparmor utilities and put mysql into complain only mode.
::aa-complain mysql
While this did help mysql run it didnt get much farther so i tried
::mysql_safe

which told me that /var/run/mysql directory did not exist.
::mkdir /var/run/mysql
::systemctl stop mysql (as it said it was running ... *shrugs* ... moving on)
::systemctl start mysql

it started with out any problems

[RN (rnickle-hccoll)]

I just ran into Robie's problem at my site. I have a 16.04 server setup in a VM running mysql 5.7.17 Community Edition.

About two weeks ago, we started having the mysqld throw apparmor errors and die every day at a random time.

What appears to be happening is the daily apt update is trying to update mysql.

The random time is a consequence of it being run by the systemd timer for cron.daily.

For some reason, part of that update involves a configuration change to mysqld.

/etc$ sudo apt-get upgrade --simulate | grep -i sql
Conf mysql-server-5.7 (5.7.17-0ubuntu0.16.04.1 Ubuntu:16.04/xenial-updates, Ubuntu:16.04/xenial-security [amd64])

At that time, either during shutdown or reload, mysqld is trying to check the status of itself (or the previous instance?) and is being rejected by apparmor:

Mar 22 11:57:45 SERVERNAME kernel: [16429106.693912] audit: type=1400 audit(1490198265.911:133): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/7135/status" pid=7135 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111

[Mike van der Sluis (mike-something)]

I ran into this problem when I upgraded my VM from 14.04 to 16.04.

mysql server no longer started, and after a re-install with

sudo apt purge mysql-server mysql-server-5.7 mysql-server-core-5.7
sudo apt install mysql-server

showed the errors stated

I shut them up with:
sudo nano /etc/apparmor.d/usr.sbin.mysqld

editing the system resource access section to:

# Allow system resource access
  /sys/devices/system/cpu/ r,
  /sys/devices/system/node/ r,
  /sys/devices/system/node/** r,
  /proc/** r,

Now it starts with no more app-armor blocks

[Paul Payne (payneio)]

Same problem here. mike-something's solution fixed the issue for me. I believe rnickle-hccoll's description is correct.

[Paul Payne (payneio)]

Actually, I'm getting a further apparmor error:
AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld"

[Paul Payne (payneio)]

I don't know what the deal is. I can now start the mysqld with `sudo systemctl start mysql`. However, when I interact with apt, e.g. `sudo apt-get autoremove`, it attempts to complete the mysql configuration step:

Setting up mysql-server-5.7 (5.7.17-0ubuntu0.16.04.2) ...
mysql_upgrade: Got error: 2002: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) while connecting to the MySQL server
Upgrade process encountered error and will not continue.
mysql_upgrade failed with exit status 11

When looking at syslog, it appears mysql is being shut down before this part of the upgrade.

[Gary Mercer (gmercer2012)]

Hello all. I have been trying to figure out just what is going on like all of you. The problem is that Oracle has messed with mysql to the point that it no longer works like it should. The fix is Mariadb, or any other database program. I'm not into conspiracies but Oracle has a vested interest in killing mySql. Reminds me of the Digital Research and Microsoft wars from long ago. Either way, when i installed mariadb i got no hits from apparmor. Two different machines with exactly the same results before and after, one a server and one a desktop.

You can try to figure out what piece of code in mySql needs to be fixed or just get mariadb and get back to other more important things. Good Luck everyone.

[Seth Arnold (seth-arnold)]

On Tue, Mar 28, 2017 at 04:26:52PM -0000, Paul Payne wrote:
> Actually, I'm getting a further apparmor error:
> AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld"

This is not an error message. This is a status message that reports a new
AppArmor policy has been loaded.

Thanks

[Robie Basak (racb)]

@RN

> I just ran into Robie's problem at my site.

No, you didn't. You didn't get the message with name="/etc/mysql/my.cnf.migrated" so you are not affected by this bug. You are hitting a different bug - perhaps bug 1658239.

[Lars Tangvald (lars-tangvald)]

I see this now on a plain, fresh install of MySQL 5.7:
[Fri Mar 31 12:50:59 2017] audit: type=1400 audit(1490957460.075:78): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/4988/status" pid=4988 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=112 ouid=112

As Robie notes, it's from bug 1658239.
However, this error has no noticeable effect on the running of the server. It comes up as it should, including after a service restart or a system reboot, so I don't think it's really related to this bug report.

It might be causing problems that aren't trivial to reproduce, but if so it should be fixed with the fix to apparmor upstream.

For those of you seeing issues that don't include errors when reading files in /etc/mysql, do you see anything in the journal log or MySQL error log?

[Lars Tangvald (lars-tangvald)]

(This might be best put in a separate report, but adding it here for now)
The behavior was added for 5.7 with this change
https://github.com/mysql/mysql-server/commit/1bba7015030a1cdd9eba21d48e7fdd1bad16ebef

It is there to prevent one server process from setting the same socket file as another, so if you have two server processes running and they are set to use the same socket file, shutting down one can remove the socket file for the other (making it appear to have died).

[Zach Appelbaum (kritias)]

Happened to me on a fresh DO droplet.

Ubuntu 16.04.2 LTS
mysql Ver 14.14 Distrib 5.7.17 (x86_64)

After two or three days of normal functioning. We had to start optimizing MyISAM. So the first thing we changed

log_slow_queries = /var/log/mysql/mysql-slow.log
long_query_time = 3
log-queries-not-using-indexes

service mysql restart started failing from there.

Side note. We changed key_buffer_size to
key_buffer_size = 768M

And restarted without problems the first day.

[Seth Arnold (seth-arnold)]

Zach, this bug appears to be growing to encompass too many things. I suggest filing a new bug report if you think you've got an issue, and if you think it's AppArmor related, please be sure to include any DENIED lines from dmesg or audit logs.

Thanks

[Robert Kofler (robert-kofler)]

I just realized some strange behaviour with my
mysqld Ver 5.7.17-0ubuntu0.16.04.1 for Linux on x86_64 ((Ubuntu))

this my.cmf parameter "default-character-set=utf8" in /etc/mysql/mysql.conf.d/mysqld.cnf
caused this dmesg output:

 audit: type=1400 audit(1491639131.263:200): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/8039/status" pid=8039 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=107 ouid=107

btw:
/etc/mysql contains a link, I'm not whether this is allowed by apparmor setting:

drwxr-xr-x 2 root root 4096 Apr 7 20:07 conf.d/
-rw------- 1 root root 317 Feb 21 20:47 debian.cnf
-rwxr-xr-x 1 root root 120 Jan 18 22:46 debian-start*
lrwxrwxrwx 1 root root 24 Feb 21 20:43 my.cnf -> /etc/alternatives/my.cnf
-rw-r--r-- 1 root root 839 Apr 20 2016 my.cnf.fallback
-rw-r--r-- 1 root root 682 Apr 20 2016 mysql.cnf
drwxr-xr-x 2 root root 4096 Apr 8 10:14 mysql.conf.d/

but /etc/apparmor.d/usr.sbin.mysqld only specifies an "r" NOT "lr" for its config-dir.

# Allow config access
  /etc/mysql/** r,

(but I'm no apparmor expert)

[Valdisvi (valdis-vitolins)]

Actually I got similar problem, when tuning setting I added
table_cache = 256
into /etc/mysql/mysql.conf.d/mysqld.cnf file

[Robie Basak (racb)]

I think it's time to mark this bug Invalid. We have had a bunch of "me too" comments that are clearly and incorrectly describing bugs other than the one described here.

Conversely the only credible explanation of the problem reported (failure to start due to AppArmor denial of /etc/mysql/my.cnf.migrated) is reported to have gone away by itself and we haven't had any reports of anyone having independently reproduced or even just experienced the same problem.

Unfortunately the misguided comments are making it impossible to follow this bug. If it turns out that the bug is real, then I suggest filing a separate and full report (referring to this bug) to get a developer's attention, since I'll be unsubscribing from this bug as I've seen no signal amongst the noise except from the original reporter.

[Zach Appelbaum (kritias)]

How are they incorrect? It's pretty simple. Change a my.cnf setting once or twice and restart. Then AppArmor comes in and messes everything up. If you said that this should be reported to AppArmor then say that instead.

[Gary Mercer (gmercer2012)]

Sorry Robie but you are still wrong. It is a bug but mostly because mysql
is more the problem and not Ubuntu. Changing databases does fix the bug.
Changing ubuntu doesn't make the bug go away.

Gary

On Apr 21, 2017 6:39 AM, "Robie Basak" <email address hidden> wrote:

> I think it's time to mark this bug Invalid. We have had a bunch of "me
> too" comments that are clearly and incorrectly describing bugs other
> than the one described here.
>
> Conversely the only credible explanation of the problem reported
> (failure to start due to AppArmor denial of /etc/mysql/my.cnf.migrated)
> is reported to have gone away by itself and we haven't had any reports
> of anyone having independently reproduced or even just experienced the
> same problem.
>
> Unfortunately the misguided comments are making it impossible to follow
> this bug. If it turns out that the bug is real, then I suggest filing a
> separate and full report (referring to this bug) to get a developer's
> attention, since I'll be unsubscribing from this bug as I've seen no
> signal amongst the noise except from the original reporter.
>
> ** Changed in: mysql-5.7 (Ubuntu)
> Status: Incomplete => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1610765
>
> Title:
> Apparmor refuses mysqld open on /etc/mysql/my.cnf.migrated causing it
> to fail to start
>
> Status in mysql-5.7 package in Ubuntu:
> Invalid
>
> Bug description:
> [Triage Notes]
>
> This bug tracks reports of the following AppArmor denial:
>
> Aug 7 18:22:17 c505 kernel: [ 1485.490459] audit: type=1400
> audit(1470619337.926:98): apparmor="DENIED" operation="open"
> profile="/usr/sbin/mysqld" name="/etc/mysql/my.cnf.migrated" pid=11219
> comm="mysqld" requested_mask="r" denied_mask="r" fsuid=109 ouid=0
>
> If you are seeing a denial of /etc/mysql/my.cnf.migrated in this
> manner, then the underlying issue may affect you too.
>
> If you are seeing a denial of some other file, then you have some
> other AppArmor-related problem that is *not* the issue tracked in this
> bug. If you believe that your failure is a bug in Ubuntu, please look
> for other bugs and file a new bug if there isn't a matching one that
> tracks your different issue.
>
> [Original Desription]
>
> SYMPTOM
> mysql fails to start
>
> IMPACT
> no sql for you
>
> DIAGNOSE:
> service mysql start
> Job for mysql.service failed because the control process exited with
> error code. See "systemctl status mysql.service" and "journalctl -xe" for
> details.
>
> - systemd has all sort of craziness going on so stop that
> service mysql stop
>
> - look at what apparmor is up to
> tail -F /var/log/syslog | grep -i apparmor
>
> - then start again
> service mysql start
>
> output is:
> - mysql cannot read /etc/mysql
> Aug 7 18:22:17 c505 kernel: [ 1485.490459] audit: type=1400
> audit(1470619337.926:98): apparmor="DENIED" operation="open"
> profile="/usr/sbin/mysqld" name="/etc/mysql/my.cnf.migrated" pid=11219
> comm="mysqld" requested_mask="r" denied_mask="r" fsuid=109 ouid=0
>
> GET_AROUND:
> sudo nano /etc/a...

[Jarod (jarod42)]

MySQL started to throw apparmor errors after enabling slow-query-logs.
I haven't used this log option for quite some time. So it probably sat for a few years (and several migrations) uncommented in my my.cnf.

It turned out that the variable name had changes. The entry used to be "log_slow_queries = /var/..." now it needs to be "slow-query-log-file = /var/..."

[cripperz (cripperz)]

Just bumped into this error with my Ubuntu 16.04 LTS Server with mysql-5.7

-- Unit mysql.service has begun starting up.
Aug 09 19:49:46 ns3 audit[3405]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3405/status" pid=340
Aug 09 19:49:46 ns3 audit[3405]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/"
Aug 09 19:49:46 ns3 audit[3405]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/3405/status" pid=340
Aug 09 19:49:46 ns3 kernel: audit: type=1400 audit(1502279386.154:69): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld
Aug 09 19:49:46 ns3 kernel: audit: type=1400 audit(1502279386.154:70): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld
Aug 09 19:49:46 ns3 kernel: audit: type=1400 audit(1502279386.154:71): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld
Aug 09 19:49:47 ns3 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE

[Andreas Hasenack (ahasenack)]

@cripperz, can you please try regenerating the apparmor cache and reloading it, via this command:

sudo apparmor_parser -r -T -W -v /etc/apparmor.d/usr.sbin.mysqld

Then restart mysql:

sudo service mysql restart

[cripperz (cripperz)]

Aug 18 20:07:53 ns3 audit[7045]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/7045/status" pid=7045 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
I just stumbled again into the problem. took the chance to try your suggest @andreas

below error

Aug 18 20:07:53 ns3 kernel: audit: type=1400 audit(1503058073.118:64): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/7045/status" pid=7045 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 18 20:07:53 ns3 audit[7045]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=7045 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=0
Aug 18 20:07:53 ns3 audit[7045]: AVC apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/7045/status" pid=7045 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 18 20:07:53 ns3 kernel: audit: type=1400 audit(1503058073.122:65): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=7045 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=0
Aug 18 20:07:53 ns3 kernel: audit: type=1400 audit(1503058073.122:66): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/proc/7045/status" pid=7045 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=111 ouid=111
Aug 18 20:07:54 ns3 systemd[1]: mysql.service: Main process exited, code=exited, status=1/FAILURE

[Andreas Hasenack (ahasenack)]

Where is the output of "sudo apparmor_parser -r -T -W -v /etc/apparmor.d/usr.sbin.mysqld"?

[cripperz (cripperz)]

@andreas
Just a description how this happened again. Usually it would happen after the server lost power and reboot this error would come back. Or the VM abruptly off without a proper shutdown then i will hit into this issue.

root@ns3:/etc/mysql#apparmor_parser -r -T -W -v /etc/apparmor.d/usr.sbin.mysqld
Replacement succeeded for "/usr/sbin/mysqld".
root@ns3:/etc/mysql#

For testing sake, i added below in the /etc/apparmor.d/usr.sbin.mysqld

  /sys/devices/system/node/* r,
  /sys/devices/system/node/ r,
  /sys/devices/system/node/node0/meminfo r,

  /proc/** rk,
  /proc/* rk,

then i get

Aug 19 09:32:34 ns3 kernel: [ 9028.392694] audit: type=1400 audit(1503106354.767:603): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/mysqld" pid=6139 comm="apparmor_parser"

[Andreas Hasenack (ahasenack)]

The DENIED of /proc/<pid>/status is tracked in bug #1658239
The DENIED of /sys/devices/system/node/ is tracked in bug bug #1658233

@cripperz I think the above cover the changes you had to made to your profile. Let's track these issues over there and keep this bug for apparmor errors regarding the /etc/mysql/my.cnf.migrated file specifically.

[Richard (ismail-a)]

So, I have become very good at checking the MySQL setup, and in my case (I recently restarted my server) I had AppArmor properly setup but it still would not work

My previous actions, as pointed out, disabled AppArmor why it after that started working
@seth-arnold Seth Arnold pointed out the correct solution in #6

The proper solution is:

       systemctl reload apparmor.service

- which applies the current and extensively checked settings, this apply which apparently had not been done, and this keeps AppArmor active and MySQL working

The question might be why are the settings not applied even on a server reboot?

[Andreas Hasenack (ahasenack)]

I don't know if it's the case here, but I've seen *new* apparmor profiles not being applied because the apparmor cache file had, for some reason, a more recent timestamp. That made the system use the cache file instead of the correct changed config file.

[Robie Basak (racb)]

My position is still the same. It seems that everyone piling on to this bug report with a "me too" is not affected by the same root cause as the original reporter, who had a problem with a locally customised AppArmor profile that ended up incorrect and which we identified was not a bug.

The status set against this bug is only one status and cannot track the status of each of your separate issues.

If you have a different AppArmor MySQL related problem, this is not the bug for you. Please read my triage notes at the top of the bug description.

It does not help to add comments for your unrelated issues. Few developers will read about them here. Please file separate bug reports as instructed.

[Zach Appelbaum (kritias)]

Where should we post it then? AppArmor bug tracker? It's pretty annoying to change 1 setting on mysql.cnf and then be denied restarting the MySQL server because of that. Are my distro settings too tight? is it apparmor? is it mysql?

[Gary Mercer (gmercer2012)]

Basak. You were wrong at the beginning and your still wrong. The bug is
mysql and apparmor dont work together. Period. I havent had any problems at
all no matter what configuration i had as soon as I uninstalled mysql and
loaded another database. Please quit telling people there is nothing wrong
with mysql.

On Feb 9, 2018 9:05 AM, "Zach Appelbaum" <email address hidden> wrote:

> Where should we post it then? AppArmor bug tracker? It's pretty annoying
> to change 1 setting on mysql.cnf and then be denied restarting the MySQL
> server because of that. Are my distro settings too tight? is it
> apparmor? is it mysql?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1610765
>
> Title:
> Apparmor refuses mysqld open on /etc/mysql/my.cnf.migrated causing it
> to fail to start
>
> Status in mysql-5.7 package in Ubuntu:
> Invalid
>
> Bug description:
> [Triage Notes]
>
> This bug tracks reports of the following AppArmor denial:
>
> Aug 7 18:22:17 c505 kernel: [ 1485.490459] audit: type=1400
> audit(1470619337.926:98): apparmor="DENIED" operation="open"
> profile="/usr/sbin/mysqld" name="/etc/mysql/my.cnf.migrated" pid=11219
> comm="mysqld" requested_mask="r" denied_mask="r" fsuid=109 ouid=0
>
> If you are seeing a denial of /etc/mysql/my.cnf.migrated in this
> manner, then the underlying issue may affect you too.
>
> If you are seeing a denial of some other file, then you have some
> other AppArmor-related problem that is *not* the issue tracked in this
> bug. If you believe that your failure is a bug in Ubuntu, please look
> for other bugs and file a new bug if there isn't a matching one that
> tracks your different issue.
>
> [Original Desription]
>
> SYMPTOM
> mysql fails to start
>
> IMPACT
> no sql for you
>
> DIAGNOSE:
> service mysql start
> Job for mysql.service failed because the control process exited with
> error code. See "systemctl status mysql.service" and "journalctl -xe" for
> details.
>
> - systemd has all sort of craziness going on so stop that
> service mysql stop
>
> - look at what apparmor is up to
> tail -F /var/log/syslog | grep -i apparmor
>
> - then start again
> service mysql start
>
> output is:
> - mysql cannot read /etc/mysql
> Aug 7 18:22:17 c505 kernel: [ 1485.490459] audit: type=1400
> audit(1470619337.926:98): apparmor="DENIED" operation="open"
> profile="/usr/sbin/mysqld" name="/etc/mysql/my.cnf.migrated" pid=11219
> comm="mysqld" requested_mask="r" denied_mask="r" fsuid=109 ouid=0
>
> GET_AROUND:
> sudo nano /etc/apparmor.d/usr.sbin.mysqld
> line 25 after:
> # Allow config access
> /etc/mysql/** r,
> -add line:
> /etc/mysql/* r,
>
> systemctl restart apparmor.service
>
> service mysql stop
> service mysql start
>
> - problem is gone
>
> ProblemType: Bug
> DistroRelease: Ubuntu 16.04
> Package: mysql-server-5.7 5.7.13-0ubuntu0.16.04.2
> ProcVersionSignature: Ubuntu 4.4.0-31.50-generic 4.4.13
> Uname: Linux 4.4.0-31-generic x86_64
> ApportVersion: 2.20.1-0ubuntu2.1
> Architecture: amd64
> Date: Sun Aug 7 18:31:32 2016
> Logs...

[Zach Appelbaum (kritias)]

Let's tone it down a bit Gary. He's both right and wrong. He wants people to discuss the same root issue and this thread is mixed up with several "me too" comments that may or may not be the same issue.

In my case it was simple as editing mysql.cnf and start getting AppArmor DENIED errors.

[David Dombrowsky (davek)]

After all that, my mysqld is still broken. I see no "valid" ticket for this bug linked here, if this one is invalid. Where is the bug report for the root cause?

[Seth Arnold (seth-arnold)]

David, the trouble is most of the comments here are for various different local configuration changes. You'd be best served by filing a new bug. If you have DENIED lines from AppArmor, make sure your logs include them.

Thanks