Comment 2 for bug 1735424

Revision history for this message
Christian Ehrhardt  (paelzer) wrote : Re: package mysql-server is compiled without cmake flag -DWITH_NUMA=ON

Hi Mathias,
thank you for your report - it is an interesting one.

I think (but I subscribe Lars and Robie for more expertise) that you are right that it is not working but wrong on the current assumption why. Let me outline why I think so.

[1]: defines it needs WITH_NUMA, but since MySQL 5.7.17 it is set by CMAKE "based on whether the current platform has NUMA support."
Now with that in mind what it states: " For the innodb_numa_interleave option to be available".
So what happens in your case is NOT "innodb_numa_interleave - unknown option", instead it understands and try to follow and only later then fails to actually execute it:
"Failed to set NUMA memory policy of buffer pool page frames to MPOL_INTERLEAVE (error: Invalid argument)."
I think the availability of libnuma-dev in [2] is enough to make CMake enable WITH_NUMA.
And the issue you face is that the system you are on is not numa aware/capable and/or it might be a permission issue.

So far that is only a theory - so please if you know better correct me :-)

Next I thought it might be a platform thing, so I tried with numactl and set up a KVM guest with fake numa to work for sure.

Container (no numa)
# sudo numactl --interleave 0-1 echo foo
libnuma: Warning: node argument 1 out of range

Guest (fake numa)
$ sudo numactl --interleave 0-1 echo foo
foo

So on the latter I re-ran the test with mysql that you reported (thanks for the steps to reproduce). And it still triggers.
So far +1 on confirmed for the actual issue.

I went into root as I thought it is a permission issue, but it isn't (usually user mysql).

But then I saw there is a set of apparmor denials, none
[ 792.298983] audit: type=1400 audit(1512141533.009:33): apparmor="DENIED" operation="open" profile="/usr/sbin/mysqld" name="/sys/devices/system/node/" pid=3699 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=113 ouid=0

That is numa related, maybe it does some checks up front.
So I disabled apparmor for mysqld
$ sudo aa-disable /usr/sbin/mysqld
And e voila:

2017-12-01T15:16:13.126125Z 0 [Note] InnoDB: Setting NUMA memory policy to MPOL_INTERLEAVE
2017-12-01T15:16:13.132921Z 0 [Note] InnoDB: Setting NUMA memory policy to MPOL_DEFAULT
2017-12-01T15:16:13.132949Z 0 [Note] InnoDB: Completed initialization of buffer pool

I checked the Denies and wrote rules for them.
If you add the following to the "/etc/apparmor.d/usr.sbin.mysqld" profile you will be good.

# numa enabling
  /sys/devices/system/node/ r,
  /sys/devices/system/node/** r,
  @{PROC}/@{pid}/status r,

Then reload with:
$ sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.mysqld

So it is not a build issue, but instead an imperfect apparmor profile.
I leave the inclusion of the fix to our mysql packagers (subscribing them).

[1]: https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_numa_interleave
[2]: https://launchpadlibrarian.net/341325097/buildlog_ubuntu-artful-amd64.mysql-5.7_5.7.20-0ubuntu0.17.10.1_BUILDING.txt.gz