apt dist-upgrade does not restart package qemu-guest-agent

[Jan Kratochvíl]

[Impact]
When upgrading qemu-guest-agent, the service is not restarted. This can cause an issue in certain scenarios such as using Terraform where the servers get automatically updated, and then get lost due to qemu-guest-agent not restarting.

[Fix]
The fix is to remove --no-start from dh_installsystemd

This is a backport from the following debian commit - https://salsa.debian.org/qemu-team/qemu/-/commit/eb0eba431ad4524a768a4e5cef4d4f6f1ad8a649

[Test Case]
set of steps to reproduce the issue. With the fix, qemu-guest-agent should be active after reinstalling.

$ lxc launch ubuntu:jammy j --vm -c limits.memory=8GB
$ lxc shell j
# apt update && apt install -y uvtool uvtool-libvirt
# su - ubuntu
$ ssh-keygen
$ uvt-simplestreams-libvirt sync release=jammy arch=amd64
$ uvt-kvm create test release=jammy --memory 2024
$ uvt-kvm wait test
$ uvt-kvm ssh test
# apt install qemu-guest-agent
# systemctl start qemu-guest-agent
# apt install --reinstall qemu-guest-agent
# systemctl status qemu-guest-agent

[Things that can go wrong]
- Security considerations of automatically starting qemu-guest-agent. It should be presumed that users understand the package they are installing.
- If the hypervisor can change settings to no longer support qemu-guest-agent service, then attempting to start the service will fail. I'm not sure if this scenario is possible.

[boe (fritzenboe)]

Hi,

I can confirm the issue. We have several virtual machines with 22.04 (upgraded from 18.04) with the same issue on each qemu-guest-agent upgrade. In the follow example the service is stopped with he upgrade of the package at '2023-07-17 13:21:02'

root@vm1:~# grep -i qemu /var/log/dpkg.log
2023-07-03 20:58:18 upgrade qemu-guest-agent:amd64 1:4.2-3ubuntu6.27 1:6.2+dfsg-2ubuntu6.11
2023-07-03 20:58:18 status half-configured qemu-guest-agent:amd64 1:4.2-3ubuntu6.27
2023-07-03 20:58:18 status unpacked qemu-guest-agent:amd64 1:4.2-3ubuntu6.27
2023-07-03 20:58:18 status half-installed qemu-guest-agent:amd64 1:4.2-3ubuntu6.27
2023-07-03 20:59:18 status unpacked qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.11
2023-07-03 21:00:06 configure qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.11 <none>
2023-07-03 21:00:06 status unpacked qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.11
2023-07-03 21:00:06 status half-configured qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.11
2023-07-03 21:00:06 status installed qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.11
2023-07-17 13:21:02 upgrade qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.11 1:6.2+dfsg-2ubuntu6.12
2023-07-17 13:21:02 status half-configured qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.11
2023-07-17 13:21:02 status unpacked qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.11
2023-07-17 13:21:02 status half-installed qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.11
2023-07-17 13:21:02 status unpacked qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.12
2023-07-17 13:21:05 configure qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.12 <none>
2023-07-17 13:21:05 status unpacked qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.12
2023-07-17 13:21:05 status half-configured qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.12
2023-07-17 13:21:05 status installed qemu-guest-agent:amd64 1:6.2+dfsg-2ubuntu6.12

root@vm1:~# systemctl status qemu-guest-agent.service
○ qemu-guest-agent.service - QEMU Guest Agent
Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static)
Active: inactive (dead) since Mon 2023-07-17 13:21:02 CEST; 2 days ago
Main PID: 763 (code=exited, status=0/SUCCESS)
CPU: 58ms

Jul 11 05:42:42 vm1 qemu-ga[763]: info: guest-fsfreeze called
Jul 12 05:44:36 vm1 qemu-ga[763]: info: guest-fsfreeze called
Jul 13 04:43:06 vm1 qemu-ga[763]: info: guest-fsfreeze called
Jul 14 05:57:13 vm1 qemu-ga[763]: info: guest-fsfreeze called
Jul 15 05:46:10 vm1 qemu-ga[763]: info: guest-fsfreeze called
Jul 16 06:22:59 vm1 qemu-ga[763]: info: guest-fsfreeze called
Jul 17 05:40:00 vm1 qemu-ga[763]: info: guest-fsfreeze called
Jul 17 13:21:02 vm1 systemd[1]: Stopping QEMU Guest Agent...
Jul 17 13:21:02 vm1 systemd[1]: qemu-guest-agent.service: Deactivated successfully.
Jul 17 13:21:02 vm1 systemd[1]: Stopped QEMU Guest Agent.

Kind regards,
Achim

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[David Kalnischkies (donkult)]

APT itself is not involved with (re)starting services. The individual packages do this in their maintainer scripts, so reassigning to qemu (which contains qemu-guest-agent).

[Lena Voytek (lvoytek)]

Thanks for the bug report. I can confirm this happens after a qemu upgrade and after installing qemu-guest-agent in mantic, lunar, and jammy. Marking as such

[Sergio Durigan Junior (sergiodj)]

Thank you for taking the time to report the bug.

Indeed, I can confirm that qemu-guest-agent.service won't restart after upgrading. For the sake of completeness, here are the steps to verify:

$ lxc launch ubuntu:jammy j --vm -c limits.memory=8GB

Make sure to extend the rootfs size of the VM if needed.

$ lxc shell j
# apt update && apt install -y uvtool uvtool-libvirt
# ...
# su - ubuntu
$ ssh-keygen
$ uvt-simplestreams-libvirt sync release=jammy arch=amd64
$ uvt-kvm create test release=jammy --memory 2024
$ uvt-kvm wait test
$ uvt-kvm ssh test

You can also spawn other releases like Lunar or Mantic. They are all affected by the issue.

Basically, a good way to reproduce the bug is to start the agent, and then simply reinstall the package:

# apt install qemu-guest-agent
# systemctl start qemu-guest-agent
# apt install --reinstall qemu-guest-agent

You will notice that the service doesn't come back up.

Now, the root cause of this problem is the use of "--no-start" in the following d/rules line:

dh_installsystemd -a -pqemu-guest-agent --no-start --no-enable

According to dh_installsystemd:

--no-start
    Do not start the unit file after upgrades and after initial installation (the latter is only relevant for services without a corresponding init script).

Moreover, since qemu is using debhelper-compat 12 or greater, the default option for dh_installsystemd is:

--restart-after-upgrade
    Do not stop the unit file until after the package upgrade has been completed. This is the default behaviour in compat 10.

So, the way I see it, there are two ways to possibly fix this problem:

1) Use --no-restart-after-upgrade, which, contrary to what its name says, will stop the service in the prerm script (which is already being done by default), and start the service again after upgrade.

2) Remove the --no-start option from the d/rules line. I tested it here and it doesn't seem to have any negative side-effects.

I'm still running a few tests and will report back later. This is something that I intend to bring to the Debian qemu maintainer's attention as well.

[Sergio Durigan Junior (sergiodj)]

After some tests, option (1) is out of the table: even using --no-restart-after-upgrade won't work if we still pass --no-start to dh_installsystemd. The way forward here is to remove --no-start from dh_installsystem.

[Sergio Durigan Junior (sergiodj)]

Submitted an MR to Debian's qemu:

https://salsa.debian.org/qemu-team/qemu/-/merge_requests/37

I'd like to get the maintainer's opinion before moving forward with this.

I'm not convinced that we'll be able to SRU this into Jammy, but at least on Mantic it should be fixed.

[Sergio Durigan Junior (sergiodj)]

The MR was accepted and uploaded to Debian unstable. I intend to merge the package soon, which should fix the issue on Mantic.

I still don't know if this is SRU material, though. Have to think about it...

[Patrick Kaiser (argonius)]

Just wanted to mention, that this is a not so trivial issue. When using terraform to deploy an ubuntu22.04 cloud-init image you will end up that:
1. vm is created
2. it automatically upgrades qemu-guest-agent
3. and does not start it
4. terraform lost the connection to the vm as the guest agent is not running anymore

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:8.0.4+dfsg-1ubuntu1

---------------
qemu (1:8.0.4+dfsg-1ubuntu1) mantic; urgency=medium

  * Merge with Debian unstable. Remaining changes:
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-system-common.qemu-kvm.service: systemd unit to call
        qemu-kvm-init
      - d/qemu-system-common.install: install helper script
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: call dh_installinit and dh_installsystemd for qemu-kvm
    - Distribution specific machine type
      (LP 1304107 1621042 1776189 1761372 1761372 1776189)
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types containing release versioned machine attributes
      - Add an info about -hpb machine type in debian/qemu-system-x86.NEWS
      - ubuntu-q35 alias added to auto-select the most recent q35 ubuntu type
    - Enable nesting by default
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
        [ No more strictly needed, but required for backward compatibility ]
    - tolerate ipxe size change on migrations to >=18.04 (LP 1713490)
      - d/p/ubuntu/pre-bionic-256k-ipxe-efi-roms.patch: old machine types
        reference 256k path
      - d/control-in: depend on ipxe-qemu-256k-compat-efi-roms to be able to
        handle incoming migrations from former releases.
    - Ease the use of module retention on upgrades (LP 1913421)
      - debian/qemu-block-extra.postinst: enable mount unit on install/upgrade
    - Remaining GCC-12 FTBFS (LP 1988710 + LP 1921664)
      + d/p/u/qboot-Disable-LTO-for-ELF-binary-build-step.patch:
        fix qboot FTBFS with LTO

 -- Sergio Durigan Junior <email address hidden> Mon, 14 Aug 2023 16:28:34 -0400

[Timo Aaltonen (tjaalton)]

Hello Jan, or anyone else affected,

Accepted qemu into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:7.2+dfsg-5ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

note that the jammy upload was rejected, because there was an older upload already on the queue, and I figured that was more important to get in first (a regression in the previous update)

[Mitchell Dzurick (mitchdz)]

Thanks Timo! I just made another MP for Jammy, so will wait for that one.

In the meantime I tested the new lunar proposed package.

$ lxc launch ubuntu:lunar l --vm -c limits.memory=8GB
$ lxc shell l
# apt update && apt install -y uvtool uvtool-libvirt
# su - ubuntu
$ ssh-keygen
$ uvt-simplestreams-libvirt sync release=lunar arch=amd64
$ uvt-kvm create test release=lunar --memory 2024
$ uvt-kvm wait test
$ uvt-kvm ssh test
$ sudo su
# apt update && apt install -y qemu-guest-agent
# dpkg -l qemu-guest-agent | grep qemu-guest-agent | awk '{print $3}'
1:7.2+dfsg-5ubuntu2.2
# systemctl start qemu-guest-agent
# apt install --reinstall qemu-guest-agent
# systemctl status qemu-guest-agent | grep Active
     Active: inactive (dead)
# systemctl start qemu-guest-agent
# systemctl status qemu-guest-agent | grep Active -A1
     Active: active (running) since Fri 2023-09-15 19:13:49 UTC; 2s ago
   Main PID: 3638 (qemu-ga)
# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF
# apt install -y qemu-guest-agent -t lunar-proposed
# dpkg -l qemu-guest-agent | grep qemu-guest-agent | awk '{print $3}'
1:7.2+dfsg-5ubuntu2.3
# systemctl status qemu-guest-agent | grep Active -A1
     Active: active (running) since Fri 2023-09-15 19:14:05 UTC; 11s ago
   Main PID: 3787 (qemu-ga)

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:7.2+dfsg-5ubuntu2.3

---------------
qemu (1:7.2+dfsg-5ubuntu2.3) lunar; urgency=medium

  * d/rules: remove --no-start for qemu-guest-agent (LP: #2028124)

 -- Mitchell Dzurick <email address hidden> Thu, 31 Aug 2023 05:38:41 -0700

[Mitchell Dzurick (mitchdz)]

Thanks Brian! This bug is fixed in Lunar now. I'm waiting on Jammy as there's another package in -proposed. I'll keep an eye on it and upload the Jammy package once the Jammy queue is open.

[Andreas Hasenack (ahasenack)]

> - Security considerations of automatically starting qemu-guest-agent. It should be presumed that users
> understand the package they are installing.

What's interesting with this change:

- dh_installsystemd -pqemu-guest-agent --no-start --no-enable
+ dh_installsystemd -pqemu-guest-agent --no-enable

Is that qemu-guest-agent is still not started upon fresh install.

Checking what is called during first install:

17:22:46 3.244 deb-systemd-inv 7897 7833 0 /usr/bin/deb-systemd-invoke start qemu-guest-agent.service
17:22:46 3.252 sh 7898 7897 0 /bin/sh -c systemctl --system is-enabled -- 'qemu-guest-agent.service'
17:22:46 3.252 systemctl 7899 7898 0 /usr/bin/systemctl --system is-enabled -- qemu-guest-agent.service
17:22:46 3.256 systemctl 7900 7897 0 /usr/bin/systemctl --system --quiet is-active -- qemu-guest-agent.service

Interesting, there is a "systemctl is-enabled" call.

And indeed, deb-systemd-invoke will not start the service if it's not enabled:

deb-systemd-invoke:105 (jammy)
# If the job is disabled and is not currently running, the job is not started or restarted.

And of course, closer inspection of the output when installing the package would have told me that already:
Selecting previously unselected package qemu-guest-agent.
(Reading database ... 69980 files and directories currently installed.)
Preparing to unpack .../qemu-guest-agent_1%3a6.2+dfsg-2ubuntu6.15~jammy1_amd64.deb ...
Unpacking qemu-guest-agent (1:6.2+dfsg-2ubuntu6.15~jammy1) ...
Setting up qemu-guest-agent (1:6.2+dfsg-2ubuntu6.15~jammy1) ...
qemu-guest-agent.service is a disabled or a static unit, not starting it. <===============
Processing triggers for man-db (2.10.2-1) ...

And that's why you had to include a specific start of the service in your test case right after installing the package.

[Andreas Hasenack (ahasenack)]

Hello Jan, or anyone else affected,

Accepted qemu into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:6.2+dfsg-2ubuntu6.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:6.2+dfsg-2ubuntu6.15) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

casper/unknown (s390x)
freedom-maker/unknown (s390x)
ganeti/3.0.2-1ubuntu1 (armhf)
nova/unknown (arm64)
sbuild/unknown (arm64)
systemd/unknown (s390x)
vmdb2/unknown (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Mitchell Dzurick (mitchdz)]

Thanks Andreas! I think this behavior is fine. It isn't a regression in 1:6.2+dfsg-2ubuntu6.15, and this bug is targeting upgrading. If anything, it's a little bit more verbose so users know that they should start the service manually.

Guides I see online[0] tell you to start the service after install already, and I can't imagine a scenario that you have the capability to install this package and not be able to run systemctl start after it.

It is a good point though, I'd like to have the behavior of the service started on install in future releases so I'll take a look and see if there's a more elegant solution here that we can propose to Debian. My immediate thought is to add back in --no-start, and then manually run `systemctl start qemu-guest-agent` in .postinst so that way we can skip using deb-systemd-invoke which cares about the service state. That solution doesn't feel very elegant so I'd like to do a little more research.

[0] - https://pve.proxmox.com/wiki/Qemu-guest-agent#Linux

[Mitchell Dzurick (mitchdz)]

Jammy Tested

$ lxc launch ubuntu:jammy j --vm -c limits.memory=8GB
$ lxc shell j
# apt update && apt install -y uvtool uvtool-libvirt
# su - ubuntu
$ ssh-keygen
$ uvt-simplestreams-libvirt sync release=jammy arch=amd64
$ uvt-kvm create test release=jammy --memory 2024
$ uvt-kvm wait test
$ uvt-kvm ssh test
$ sudo su
# apt update && apt install -y qemu-guest-agent
# dpkg -l qemu-guest-agent | grep qemu-guest-agent | awk '{print $3}'
1:6.2+dfsg-2ubuntu6.14
# systemctl start qemu-guest-agent
# apt install --reinstall qemu-guest-agent
# systemctl status qemu-guest-agent | grep Active
     Active: inactive (dead)
# systemctl start qemu-guest-agent
# systemctl status qemu-guest-agent | grep Active -A1
     Active: active (running) since Thu 2023-10-05 00:32:54 UTC; 3s ago
   Main PID: 1730 (qemu-ga)
# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF
# apt update -y && apt install -y qemu-guest-agent -t jammy-proposed
# dpkg -l qemu-guest-agent | grep qemu-guest-agent | awk '{print $3}'
1:6.2+dfsg-2ubuntu6.15
# systemctl status qemu-guest-agent | grep Active -A1
     Active: active (running) since Thu 2023-10-05 00:34:41 UTC; 17s ago
   Main PID: 2319 (qemu-ga)

Also tested upgrading to future packages works for restarting the service

# wget https://launchpad.net/ubuntu/+archive/primary/+files/qemu-guest-agent_6.2+dfsg-2ubuntu6.15_amd64.deb
# mkdir tmp
# dpkg-deb -R qemu-guest-agent_6.2+dfsg-2ubuntu6.15_amd64.deb tmp/
# sed -i s/6.15/6.16/g tmp/DEBIAN/control
# dpkg-deb -b tmp/ 6.16.deb
# dpkg -i 6.16.deb
# systemctl status qemu-guest-agent | grep Active -A1
     Active: active (running) since Thu 2023-10-05 00:42:32 UTC; 12s ago
   Main PID: 2571 (qemu-ga)

As mentioned before, a fresh install creates a new warning message

# apt purge qemu-guest-agent
# apt install -y qemu-guest-agent -t jammy-proposed
...
qemu-guest-agent.service is a disabled or a static unit, not starting it.
...

This is not a regression as it has been present and this is how the package used to work. I think this is ultimately a benefit as it makes the user more aware that the service will not be started.

[Andreas Hasenack (ahasenack)]

I retriggered the failed qemu tests, and I think they look good now, but I'll let the SRU report[1] go green before acting on it.

1. https://ubuntu-archive-team.ubuntu.com/pending-sru.html

[Mitchell Dzurick (mitchdz)]

Andreas, the tests are green, can we go ahead with phasing or is there more testing that should be done?

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:6.2+dfsg-2ubuntu6.15

---------------
qemu (1:6.2+dfsg-2ubuntu6.15) jammy; urgency=medium

  * d/rules: remove --no-start for qemu-guest-agent (LP: #2028124)

 -- Mitchell Dzurick <email address hidden> Fri, 15 Sep 2023 14:39:05 -0400