systemd abort

[Daniel Black]

############################

sudo systemctl stop mariadb

# needs to be fast, do not sleep more than 1 sec

sudo systemctl start mariadb

# delay does not matter

sudo systemctl disable mariadb

# delay does not matter

sudo systemctl enable mariadb

# delay does not matter

sudo systemctl status mariadb

############################

Here the service starts showing the infamous

############################

● mariadb.service - MariaDB database server
    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor
preset: enabled)
   Drop-In: /etc/systemd/system/mariadb.service.d
            └─migrated-from-my.cnf-settings.conf
    Active: inactive (dead) since Wed 2015-10-14 15:04:58 UTC; 58s ago
  Main PID: 880 (mysqld)
    Status: "Taking your SQL requests now..."
    CGroup: /system.slice/mariadb.service
            └─880 /usr/sbin/mysqld

############################

If I then shut down the MariaDB server, e.g. via the shutdown command
(or just SIGTERM the process), I get

############################

Broadcast message from systemd-journald@ubuntu-vivid-amd64 (Wed
2015-10-14 15:14:18 UTC):

systemd[1]: Caught <ABRT>, dumped core as pid 960.

Broadcast message from systemd-journald@ubuntu-vivid-amd64 (Wed
2015-10-14 15:14:18 UTC):

systemd[1]: Freezing execution.

############################

systemd:

ii libpam-systemd:amd64 219-7ubuntu3
amd64 system and service manager - PAM module
ii libsystemd0:amd64 219-7ubuntu3
amd64 systemd utility library
ii systemd 219-7ubuntu3
amd64 system and service manager
ii systemd-sysv 219-7ubuntu3
amd64 system and service manager - SysV links

with also occurs on Redhat - backtrace: https://bugzilla.redhat.com/show_bug.cgi?id=1271832

[Daniel (daniel-redhat-bugs)]

Description of problem:

rapidly stop / starting a service segfaults systemd. Result was rather limited system functionality

$ sudo reboot
Failed to start reboot.target: Activation of org.freedesktop.systemd1 timed out
Failed to open /dev/initctl: No such device or address
Failed to talk to init daemon.

Version-Release number of selected component (if applicable):

Name : systemd
Arch : x86_64
Epoch : 0
Version : 219
Release : 24.fc22

mariadb-server rpm (built from 10.1 branch - https://github.com/MariaDB/server/tree/10.1) - cmake . -DBUILD_CONFIG=mysql_release -DRPM=fedora22 && make -j 8 package

Steps to Reproduce:

############################

sudo systemctl stop mariadb;sudo systemctl start mariadb

# needs to be fast, do not sleep more than 1 sec

# delay does not matter

sudo systemctl disable mariadb

# delay does not matter

sudo systemctl enable mariadb

# delay does not matter

sudo systemctl status mariadb

############################

Here the service starts showing the infamous

############################

● mariadb.service - MariaDB database server
    Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor
preset: enabled)
   Drop-In: /etc/systemd/system/mariadb.service.d
            └─migrated-from-my.cnf-settings.conf
    Active: inactive (dead) since Wed 2015-10-14 15:04:58 UTC; 58s ago
  Main PID: 880 (mysqld)
    Status: "Taking your SQL requests now..."
    CGroup: /system.slice/mariadb.service
            └─880 /usr/sbin/mysqld

############################

If I then shut down the MariaDB server, e.g. via the shutdown command
(or just SIGTERM the process),

Actual results:

17:40 root@spaceman ~ # systemctl stop mariadb.service ; systemctl start mariadb.service

17:40 root@spaceman ~ # systemctl status -l mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─migrated-from-my.cnf-settings.conf
   Active: active (running) since Wed 2015-10-14 17:40:30 CEST; 3s ago
  Process: 8715 ExecStartPre=/usr/bin/sync (code=exited, status=0/SUCCESS)
 Main PID: 8748 (mysqld)
   Status: "Taking your SQL requests now..."
   CGroup: /system.slice/mariadb.service
           └─8748 /usr/sbin/mysqld

Oct 14 17:40:28 spaceman systemd[1]: Starting MariaDB database server...
Oct 14 17:40:29 spaceman mysqld[8748]: 2015-10-14 17:40:29 139704082127040 [Note] /usr/sbin/mysqld (mysqld 10.1.8-MariaDB-log) starting as process 8748 ...
Oct 14 17:40:30 spaceman systemd[1]: Started MariaDB database server.

17:40 root@spaceman ~ # systemctl stop mariadb.service ; systemctl start mariadb.service

17:41 root@spaceman ~ # systemctl disable mariadb.service ; systemctl enable mariadb.service
Removed symlink /etc/systemd/system/mysqld.service.
Removed symlink /etc/systemd/system/multi-user.target.wants/mariadb.service.
Removed symlink /etc/systemd/system/mysql.service.
Created symlink from /etc/systemd/system/mysql.service to /usr/lib/systemd/system/mariadb.service.
Created symlink from /etc/systemd/sys...

[Daniel Black (daniel-black)]

note mariadb-10.1 is not an ubuntu package. Its a mariadb.com produced package produced from the trunk of https://github.com/MariaDB/server/tree/10.1 - build output http://buildbot.askmonty.org/buildbot/builders/kvm-deb-vivid-amd64/builds/356/steps/test/logs/stdio is a build.

[Martin Pitt (pitti)]

Is there an apt repository for these mariadb packages somewhere, or at least some debs?

[Daniel Black (daniel-black)]

http://hasky.askmonty.org/archive/pack/10.1/build-8889/kvm-deb-vivid-amd64/debs/binary/

[Daniel (daniel-redhat-bugs)]

rpms: http://hasky.askmonty.org/archive/pack/10.1/build-8889/kvm-rpm-fedora22-amd64/rpms/ - mariadb-server

[Daniel (daniel-redhat-bugs)]

Created attachment 1083254
core.9204.gz

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Elena Stepanova (elenst)]

Correction to the initial scenario: it turns out that the service restart (quick or not) is irrelevant. It was not obvious because the problem happens sporadically, so after adding a delay the test was passing by pure chance.

Further experiments have shown that to get the pseudo-dead service, it is enough to do

# After reboot, mariadb service is already running
sudo systemctl disable mariadb
# optional delay
sudo systemctl enable mariadb
# optional delay
sudo systemctl status mariadb

# At this point we have the "dead" service, further shutdown causes systemd abort.

On my machine it failed in ~30-40% of runs, but it is highly volatile.

[Daniel (daniel-redhat-bugs)]

mariadb reference: https://mariadb.atlassian.net/browse/MDEV-8956

from Elena:

Correction to the initial scenario: it turns out that the service restart (quick or not) is irrelevant. It was not obvious because the problem happens sporadically, so after adding a delay the test was passing by pure chance.

Further experiments have shown that to get the pseudo-dead service, it is enough to do

# After reboot, mariadb service is already running
sudo systemctl disable mariadb
# optional delay
sudo systemctl enable mariadb
# optional delay
sudo systemctl status mariadb

# At this point we have the "dead" service, further shutdown causes systemd abort.

On my machine it failed in ~30-40% of runs, but it is highly volatile.

[Fedora (fedora-redhat-bugs)]

Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.