lxd upgrade hung during upgrade

[Pavel Francírek]

Upgrade (apt upgrade) hung on Ubuntu 18.04.1 for lxd=3.0.2-0ubuntu1~18.04.1.
No zfs system.

I had to downgrade (to 3.0.0-0ubuntu4), rebuild config (profile and network bridge) and import containers.

Don't know what log might interested you. For example, there was lots of "lxd.service: Found left-over process 24952 (lxd) in control group while starting unit. Ignoring." in syslog.

No problem with previous version (3.0.1-0ubuntu1~18.04.1).

[Stéphane Graber (stgraber)]

The entire output of "journalctl -u lxd" would be useful as would /var/log/lxd/lxd.log (and any older version of that file available in the directory).

[Pavel Francírek (pavel-toplist)]

Here they are. As you can see upgrade started at Nov 06 15:02 and at about 15:50 I downngraded and started older version.

[Stéphane Graber (stgraber)]

It's kinda hard to tell what's going on because of all that systemd mess.
It looks to me like there was somehow a leftover process from LXD 3.0.1 that'd keep the database open and prevent the 3.0.2 version from properly starting up and applying the update but without a system to poke at, it's hard to tell.

I also see a whole lot of reports of "kill -9" (SIGKILL) happening to the LXD process, it's not something that LXD itself or its packaging would do, so I'm wondering where that was coming from.

Would you consider attempting the upgrade to 3.0.2 again?

If it works, then I'm afraid there's not going to be enough information in here for us to track it down and fix it, so we'd likely close this issue and see if anyone else hits this on a system where we can grab some more information as it's happening.

If it doesn't work and LXD gets stuck again, can you try:
 - sudo systemctl stop lxd.socket lxd.service
 - sudo pkill -9 lxd
 - sudo lxd --debug --group lxd

That'd make sure there's no leftover LXD processes on the system then start LXD outside of the systemd unit and in debug mode. If something bad happens during the upgrade, that'd then make it pretty clear what's going on.

[Pavel Francírek (pavel-toplist)]

Yes, SIGKILLs were sent by hand in attempt to stop lxd after it became unresponsive (both "lxc list" and "systemctl stop .." commands hung too).

I understand the point "If it doesn't work" but problem was that I was not even able to upgrade to 3.0.2 version (apt upgrade hung during install) and don't know in was state the installation was.

Maybe I try to stop lxd before upgrade but we need some time to prepare for bad scenario.

[Launchpad Janitor (janitor)]

[Expired for lxd (Ubuntu) because there has been no activity for 60 days.]