snapd fails to reload LXD snap

[Simon Fels]

The update of snapd to rev 22991 seems to break a

$ snap restart --reload lxd

The commands fails with:

error: cannot perform the following tasks:
- Run service command "reload-or-restart" for running services of snap "lxd" (systemctl command [reload-or-restart snap.lxd.daemon.unix.socket snap.lxd.daemon.service] failed with exit status 1: stderr:
Job failed. See "journalctl -xe" for details.)

Looking at the system logs shows

The job identifier is 922.
Oct 08 17:36:57 j0 systemd[1]: snap.lxd.daemon.unix.socket: Socket service snap.lxd.daemon.service already active, refusing.
Oct 08 17:36:57 j0 systemd[1]: Failed to listen on Socket unix for snap application lxd.daemon.

This was working with previous snapd versions and causes our test suites to fail starting today on Oct 8

Trying to reload only the lxd.daemon command fails in the same way

snap restart --reload lxd.daemon
error: cannot perform the following tasks:
- Run service command "reload-or-restart" for services ["daemon"] of snap "lxd" (systemctl command [reload-or-restart snap.lxd.daemon.unix.socket snap.lxd.daemon.service] failed with exit status 1: stderr:
Job failed. See "journalctl -xe" for details.)

Steps to reproduce:

1. Create a new LXD instance $ lxc launch ubuntu:j j0
2. Refresh snaps within the instance an ensure rev 22991 of snapd is installed
3. Initialize lxd: $ lxd init --auto
4. Try to reload LXD: $ snap restart --reload lxd

[Thomas Parrott (tomparrott)]

Could this be due to an attempted fix for https://bugs.launchpad.net/snapd/+bug/2028141 ?

[Chuan Li (lccn)]

A customer using Anbox reproduced the same issue.
When the customer tried to add a new LXD node, the logs from the new node are as follows.

Oct 21 09:14:23 ip-172 systemd[1]: Reloading Service for snap application lxd.daemon...
Oct 21 09:14:23 ip-172 systemd[1]: Reloaded Service for snap application lxd.daemon.
Oct 21 09:14:23 ip-172 snapd[61076]: taskrunner.go:299: Change 14 task (Run service command "reload-or-restart" for services ["daemon"] of snap "lxd") failed: systemctl command [reload-or-restart snap.lxd.daemon.unix.socket snap.lxd.daemon.service] failed with exit status 1: stderr:
Oct 21 09:14:23 ip-172 snapd[61076]: Job failed. See "journalctl -xe" for details.

Ubuntu 22.04.5 LTS

snap list
lxd 5.0.2-838e1b2 24326 5.0/stable canonical** disabled,in-cohort,held
lxd 5.0.3-80aeff7 29353 5.0/stable canonical** in-cohort,held
snapd 2.59.5 19459 latest/stable canonical** snapd,disabled
snapd 2.63 21761 latest/stable canonical** snapd

[Simon Déziel (sdeziel)]

It seems that during a `snap restart --reload`, snap is trying to `reload-or-restart` both systemd units at once but the `.socket` one doesn't support reloading as we can see:

```
# systemctl reload-or-restart snap.lxd.daemon.unix.socket; echo $?
Job failed. See "journalctl -xe" for details.
1
```

Similar issue for a simple `reload` of a `.socket` unit:

```
# systemctl reload snap.lxd.daemon.unix.socket; echo $?
Failed to reload snap.lxd.daemon.unix.socket: Job type reload is not applicable for unit snap.lxd.daemon.unix.socket.
3
```

As such, I guess the current workaround is to replace `snap restart --reload lxd` by the more low-level `systemctl reload snap.lxd.daemon.service` which accomplishes just what's needed.

[Abdullah Ayman Elsharnouby (sharnouby)]

A customer using Anbox reproduced the same issue also:
2024-11-12 16:59:38 Tuning LXD ZFS based storage pool
Storage volume backups created
Storage volume images created
error: cannot perform the following tasks:
- Run service command "reload-or-restart" for services ["daemon"] of snap "lxd" (systemctl command [reload-or-restart snap.lxd.daemon.unix.socket snap.lxd.daemon.service] failed with exit status 1: stderr:
Job failed. See "journalctl -xe" for details.)

[Philip Meulengracht (the-meulengracht)]

Hey!

We've reproduced this and opened a PR to deal with this bug

https://github.com/canonical/snapd/pull/14724

[Philip Meulengracht (the-meulengracht)]

The PR has landed in master and is scheduled for snapd 2.67 release