When run from systemd-nspawn, installation of mysql-server fails because postinst fails to shut down server

[Matt Coleman]

Creating a fresh Bionic directory with `debootstrap` and then attempting to install `mysql-server` inside a `systemd-nspawn` container fails with the following message:

Setting up mysql-server-5.7 (5.7.21-1ubuntu1) ...
invoke-rc.d: could not determine current runlevel
 * Stopping MySQL database server mysqld [ OK ]
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Renaming removed key_buffer and myisam-recover options (if present)
Error: Unable to shut down server with process id 532
dpkg: error processing package mysql-server-5.7 (--configure):
 installed mysql-server-5.7 package post-installation script subprocess returned error exit status 1

Steps to reproduce:
1. debootstrap bionic testmysql
2. rm testmysql/etc/resolv.conf
2. systemd-nspawn -D testmyql --bind /etc/resolv.conf /bin/bash -c 'apt update && apt install mysql-server'

[Christian Ehrhardt  (paelzer)]

my nspawn failed to reproduce this (probably too old on xenial), but doing the same in a chroot instead of systemd-nspawn works fine.

Setting up mysql-server-5.7 (5.7.21-1ubuntu1) ...
invoke-rc.d: could not determine current runlevel
 * Stopping MySQL database server mysqld [ OK ]
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Renaming removed key_buffer and myisam-recover options (if present)
Cannot open /proc/net/unix: No such file or directory
Cannot open /proc/mounts
Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service.

So it seems to be something special to nspawn.

The code in the postinst that does this looks like that.

stop_server(){
  local tmpdir=$1
  # Send kill signal
  server_pid=$(cat "$tmpdir/mysqld.pid")
  kill "$server_pid"

  for i in $(seq 1 60); do
    sleep 0.1 # A full second is too long, but we need to give the server _some_ time.
    if ! $(ps $server_pid >/dev/null 2>&1); then
      return 0
    fi
    sleep 1
  done
  # The server hasn't shut down in a timely manner
  echo "Error: Unable to shut down server with process id $server_pid" >&2
  return 1
}

So it essentially is "just" a kill and waiting until the pid is gone

Knowing that a few questions:
1. if you run an interactive bash in nspawn and do the same install would that work
2. if (1) is the same, then after the fail what is going on with the pid it failed on
   2.1 any dmesg/syslog entries about it?
   2.2 if you send a kill manually what happens?
   2.3 is the process in any odd state (a full ps axlf from inside the nspawn might be useful)

Setting up a bionic to check the nspawn there ...

[Christian Ehrhardt  (paelzer)]

ubuntu@b-test:~$ sudo systemd-nspawn -D testmysql --bind /etc/resolv.conf /bin/bash
Spawning container testmysql on /home/ubuntu/testmysql.
Press ^] three times within 1s to kill container.
Host and machine ids are equal (92544cb0ba5946158c7c4f9b57691fe3): refusing to link journals
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

Ok, in there with a shell

Ok, it takes a long time but works:
Setting up mysql-server-5.7 (5.7.21-1ubuntu1) ...
invoke-rc.d: could not determine current runlevel
 * Stopping MySQL database server mysqld [ OK ]
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Renaming removed key_buffer and myisam-recover options (if present)
Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service.
invoke-rc.d: could not determine current runlevel

So this likely needs debug on your system why it fails for you.
Adding systemd task for being nspawn related.

[Robie Basak (racb)]

We did change how mysql-server-5.7.postinst handles internal server start and stop in Bionic in order to fix bug 1592669.

[Matt Coleman (mcoleman)]

It behaves differently (the installation succeeds) if you install mysql-server from a bash prompt inside the container. Running the installation as a parameter to `systemd-nspawn` will fail. I have tested this on a 16.04.3 system and an 18.04 system.

I just noticed something especially weird: if I add `dpkg --configure -a` to the end of the `systemd-nspawn` command, then the `apt install` command will reliably finish.

Prepare a base container to be used for the tests:
1. debootstrap bionic bionic
2. rm bionic/etc/resolv.conf

This will always fail:
sudo rm -rf testmysql && sudo cp -a bionic testmysql && sudo systemd-nspawn -D testmysql bash -c 'apt update && apt install mysql-server'

This will always succeed:
sudo rm -rf testmysql && sudo cp -a bionic testmysql && sudo systemd-nspawn -D testmysql bash -c 'apt update && apt install mysql-server; dpkg --configure -a'

[Matt Coleman (mcoleman)]

It also succeeds if I replace `dpkg --configure -a` with `ls`:
rm -rf testmysql && cp -a bionic testmysql && systemd-nspawn -D testmysql bash -c 'apt update && apt install mysql-server; ls'

[Matt Coleman (mcoleman)]

Thanks for pointing out the changes to server start/stop, Robie. Adding the `--as-pid2` parameter to `systemd-nspawn` allows it to work.