Cannot configure krb5-kdc on Ubuntu Jammy 22.04.01, "Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142."

[Surfrock66]

I have a fresh install of Ubuntu Server 22.04.01 LTS. After installing the server and running all updates, I run the following command:

apt -y install slapd ldap-utils schema2ldif sasl2-bin libsasl2-modules-gssapi-mit krb5-kdc-ldap krb5-admin-server krb5-kdc

This will be installing krb5-kdc 1.19.2-2.

This is in preparation for setting up an OpenLDAP server, a Kerberos server with an LDAP backend, and saslauthd for pass-through authentication. krb5-kdc was auto-selected when running the steps in the guide here in my development environment: https://ubuntu.com/server/docs/service-kerberos-with-openldap-backend When installing that, I get the following in the output:

Setting up krb5-kdc (1.19.2-2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/krb5-kdc.service → /lib/systemd/system/krb5-kdc.service.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.

I do get the prompts for the realm, kdc, and admin server hostnames, and they are reflected in /etc/krb5.conf. If I then run the following:

dpkg-reconfigure krb5-kdc

I am prompted for whether I want the package to create the Kerberos KDC configuration automatically, and when I say yes, it then repeats the following error:

Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.

I cannot find any further debug in the syslog or anything to indicate what the root cause is; the list of packages here are all installed together on a separate development server where I experimented with the configuration I will be deploying here in production so I don't think it's incompatible packages in the install list, but I am open to feedback on that.

[Steve Langasek (vorlon)]

systemctl is a non-optional component of an Ubuntu 22.04 system. What does `which systemctl` return for you? What does `systemctl` return when you run it?

[Surfrock66 (surfrock66)]

/usr/bin/systemctl

It's definitely installed; I've just stopped and started slapd with it.

[Surfrock66 (surfrock66)]

Running it with no arguments gives me a giant list of services, targets, etc. All the normal systemd stuff you would expect. I can even see krb5-kdc red and failed in the list.

[Steve Langasek (vorlon)]

then this must be a bug in the init-system-helpers package providing the deb-systemd-invoke command; reassigning.

[Surfrock66 (surfrock66)]

Just a follow up, I'm getting the same issue on Ubuntu 22.10 as well. I reimaged the server from scratch and tried to install the following, which had the same issue:

apt -y install db-util db5.3-util krb5-admin-server krb5-config krb5-kdc krb5-kdc-ldap krb5-user ldap-utils libgssrpc4 libkadm5clnt-mit12 libkadm5srv-mit12 libkdb5-10 libltdl7 libodbc2 libsasl2-modules-gssapi-mit libverto-libevent1 libverto1 sasl2-bin schema2ldif slapd

None of those should cause a conflict. I did a "dpkg-reconfigure" on all of them, and the only one with an issue is krb5-kdc, and on 22.10 the error is a slightly different line, "Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145." That's line 145 instead of 142, seems like it's that systemd-invoke line, is there a way for me to get additional debug from that command?

[Sergio Durigan Junior (sergiodj)]

I did a little debugging, and the problem happens because krb5-kdc.service fails to start with:

Feb 02 15:22:34 krb5-test systemd[1]: Starting Kerberos 5 Key Distribution Center...
Feb 02 15:22:34 krb5-test krb5kdc[3957]: Cannot open DB2 database '/var/lib/krb5kdc/principal': No such file or directory - while initializing database for realm LXD
Feb 02 15:22:34 krb5-test krb5kdc[3957]: krb5kdc: cannot initialize realm LXD - see log file for details
Feb 02 15:22:34 krb5-test systemd[1]: krb5-kdc.service: Control process exited, code=exited, status=1/FAILURE
Feb 02 15:22:34 krb5-test systemd[1]: krb5-kdc.service: Failed with result 'exit-code'.
Feb 02 15:22:34 krb5-test systemd[1]: Failed to start Kerberos 5 Key Distribution Center.

systemd-invoke will try to run systemctl like this:

systemctl --quiet --system restart krb5-kdc.service

which fails because of the problem mentioned above.

I don't think this is a problem with init-system-helpers, but rather an issue with krb5-kdc indeed. Also, I believe it's worth reporting this bug to Debian, since they suffer from it too.

[Sergio Durigan Junior (sergiodj)]

FWIW, because Focal's deb-systemd-invoke doesn't use --quiet we end up getting a clearer error there:

# dpkg-reconfigure krb5-kdc
Job for krb5-kdc.service failed because the control process exited with error code.
See "systemctl status krb5-kdc.service" and "journalctl -xe" for details.
invoke-rc.d: initscript krb5-kdc, action "start" failed.
● krb5-kdc.service - Kerberos 5 Key Distribution Center
     Loaded: loaded (/lib/systemd/system/krb5-kdc.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-02-02 18:32:34 UTC; 7ms ago
    Process: 2000 ExecStart=/usr/sbin/krb5kdc -P /var/run/krb5-kdc.pid $DAEMON_ARGS (code=exited, status=1/FAILURE)
        CPU: 10ms

Feb 02 18:32:34 bla systemd[1]: Starting Kerberos 5 Key Distribution Center...
Feb 02 18:32:34 bla krb5kdc[2000]: Cannot open DB2 database '/var/lib/krb5kdc/principal': No such file or directory - while initializing database for realm LXD
Feb 02 18:32:34 bla krb5kdc[2000]: krb5kdc: cannot initialize realm LXD - see log file for details
Feb 02 18:32:34 bla systemd[1]: krb5-kdc.service: Control process exited, code=exited, status=1/FAILURE
Feb 02 18:32:34 bla systemd[1]: krb5-kdc.service: Failed with result 'exit-code'.
Feb 02 18:32:34 bla systemd[1]: Failed to start Kerberos 5 Key Distribution Center.

[Sam Hartman (hartmans)]

>>>>> "Sergio" == Sergio Durigan Junior <email address hidden> writes:
    Sergio> systemd-invoke will try to run systemctl like this:

    Sergio> systemctl --quiet --system restart krb5-kdc.service

    Sergio> which fails because of the problem mentioned above.

So, there was a bug that made its way to the Debian TC asking what the
behavior ought to be when a maintainer script tried to restart a unit
and the unit failed.
The conclusion of that bug was that there is no general
policy--sometimes you want the maintainer script to fail, sometimes you
do not.
At least that was my recollection.

krb5-kdc is a service where you probably don't want the unit
restarting/starting to be a failure to cause the maintainer script to
fail.
How to I explain that to debhelper?

--Sam

[Surfrock66 (surfrock66)]

There is a bunch of interesting order-of-events issues I'm discovering with what I'm doing, and because of that it is creating errors that are obscured in the packaging process. I don't know if there's a fix, or just some alerts, etc. The package failure appears to be because I did NOT set up a realm; intending to use ldap as the backend, I figured I would NOT have krb5-kdc config create an initial realm. This means when it tries to start the service, I get this in the logs:

Cannot open DB2 database '/var/lib/krb5kdc/principal': No such file or directory - while initializing database for realm SUBDOMAIN.DOMAIN.COM

The realm is defined by the install of krb5-config, so it knows the realm it wants to use. So, fine, maybe that's expected; then I go in and run krb5_ldap_util to create the realm, and THAT led to another error...the tool doesn't support TLS. I get "Confidentiality required while initializing database" which indicates a TLS error. Disabled forcing of tls on the ldap server and I could initialize the realm, stash everything needed in keyfiles, and I was off to the races.

I don't know if there is a packaging fix (other than the advice from the maintainers above about handling the systemd calls knowing they will fail) but it's been interesting to troubleshoot.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[mai ling (ml35)]

same error when installing `zram-tools` under jammy 22.04.3 LTS

$ sudo eatmydata apt install zram-tools
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  bc
The following NEW packages will be installed:
  bc zram-tools
0 upgraded, 2 newly installed, 0 to remove and 33 not upgraded.
Need to get 93.3 kB of archives.
After this operation, 249 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 bc amd64 1.07.1-3build1 [87.6 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu jammy/universe amd64 zram-tools all 0.3.3.1-1 [5,676 B]
Fetched 93.3 kB in 0s (522 kB/s)
Selecting previously unselected package bc.
(Reading database ... 48059 files and directories currently installed.)
Preparing to unpack .../bc_1.07.1-3build1_amd64.deb ...
Unpacking bc (1.07.1-3build1) ...
Selecting previously unselected package zram-tools.
Preparing to unpack .../zram-tools_0.3.3.1-1_all.deb ...
Unpacking zram-tools (0.3.3.1-1) ...
Setting up bc (1.07.1-3build1) ...
Setting up zram-tools (0.3.3.1-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/zramswap.service → /lib/systemd/system/zramswap.service.
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for doc-base (0.11.1) ...
Processing 1 added doc-base file...

this is line 142 in /usr/bin/deb-systemd-invoke

            system('systemctl', '--quiet', @instance_args, $action, @start_units) == 0 or die("Could not execute systemctl: $!");

$ dpkg -S /usr/bin/deb-systemd-invoke
init-system-helpers: /usr/bin/deb-systemd-invoke

$ apt policy $(dpkg -S /usr/bin/deb-systemd-invoke|cut -d: -f1)
init-system-helpers:
  Installed: 1.62
  Candidate: 1.62
  Version table:
 *** 1.62 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu jammy/main i386 Packages
        100 /var/lib/dpkg/status

[mai ling (ml35)]

probably because the service failed to start?

$ journalctl --no-hostname -u zramswap
Dec 07 20:55:42 systemd[1]: Starting Linux zramswap setup...
Dec 07 20:55:42 root[452655]: Starting Zram
Dec 07 20:55:42 zramswap[452655]: <13>Dec 7 20:55:42 root: Starting Zram
Dec 07 20:55:42 zramswap[452656]: modprobe: FATAL: Module zram not found in directory /lib/modules/5.15.0-89-generic
Dec 07 20:55:42 root[452657]: Error: inserting the zram kernel module
Dec 07 20:55:42 zramswap[452657]: <13>Dec 7 20:55:42 root: Error: inserting the zram kernel module
Dec 07 20:55:42 systemd[1]: zramswap.service: Main process exited, code=exited, status=1/FAILURE
Dec 07 20:55:42 systemd[1]: zramswap.service: Failed with result 'exit-code'.
Dec 07 20:55:42 systemd[1]: Failed to start Linux zramswap setup.

[mai ling (ml35)]

this in turn fails when linux modules extra is not installed (it's the one that ships the zram module on which the package installed relies on)

[mai ling (ml35)]

so perhaps the message should be more descriptive such as "we tried to start the $some.service, but it failed", instead of misleading "Could not execute systemctl".

and also should print, "oh, by the way, this is the complete output and exit error code of invoking systemctl <list of all arguments we have passed to systemctl>"

[Dan MacDonald (allcoms)]

This bug also affects installing proftpd under 22.04.

apt install proftpd
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'proftpd-core' instead of 'proftpd'
proftpd-core is already the newest version (1.3.7c+dfsg-1build1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n]
[master d41b86764d] saving uncommitted changes in /etc prior to apt run
 Author: cseadmin <email address hidden>
 1 file changed, 1 insertion(+), 1 deletion(-)
Setting up proftpd-core (1.3.7c+dfsg-1build1) ...
usermod: no changes
Synchronizing state of proftpd.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable proftpd
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.
dpkg: error processing package proftpd-core (--configure):
 installed proftpd-core package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 proftpd-core
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Sergio Durigan Junior (sergiodj)]

I'm changing my opinion here.

I feel like this is indeed a problem with how init-system-helpers (more specifically, deb-systemd-invoke) warns users about errors. Since it uses "--quiet" when invoking systemctl, I believe it needs to be a bit more verbose to explain what happened.

What's interesting that I can't reproduce the apt failure. For example, "apt install proftpd" will warn me about deb-systemd-invoke, but the command will finish successfully. ISTR having seen this behaviour before, but I don't remember my conclusion at the time.

Anyway, this needs to be forwarded to Debian. I don't believe Ubuntu should diverge from Debian in this case.

[Alexander Stohr (alexander-stohr)]

hey, i removed the "quiet" and did a bit of alternate outputs. the action is 'start' for my failing 'xrdp'.

Job for xrdp.service failed because the control process exited with error code.
See "systemctl status xrdp.service" and "journalctl -xeu xrdp.service" for details.
Could not execute systemctl: - start at /usr/bin/deb-systemd-invoke line 143.

[NetVicious (netvicious)]

Same problem here with Ubuntu 22.04.4 LTS in a brand new AWS EC2 directly born as Ubuntu 22.

Setting up proftpd-core (1.3.7c+dfsg-1build1) ...
usermod: no changes
Synchronizing state of proftpd.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable proftpd
Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.
dpkg: error processing package proftpd-core (--configure):
 installed proftpd-core package post-installation script subprocess returned error exit status 1

I fixed it editing /var/lib/dpkg/info/proftpd-core.postinst and commenting the deb-systemd-invoke commands, running the apt install another time and letting the postint script as it was after the package was installed correctly.

[Mitchell Dzurick (mitchdz)]

netvicious or Dan, do you have the AMI that you used to produce that error with proftpd-core?

I also tried reproduction on ami-0dffe9017aa8424a2 in eu-north-1 and was not able to reproduce that failure.

I just booted a fresh t3.micro instance with the above AMI, then proceeded to run `apt update -y && apt install -y proftpd`.

[Mitchell Dzurick (mitchdz)]

Also, the various packages here have a similar error message but the error is probably caused by different things. This is a generic error message from init-system-helpers.

Now, with that said, I agree init-system-helpers should be more verbose when this error happens.