qemu-guest-agent / systemctl enable failure

[Vincent JARDIN]

qemu-guest-agent service cannot be enabled. It fails with:

$ sudo systemctl enable qemu-guest-agent
Synchronizing state of qemu-guest-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable qemu-guest-agent
The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
Alias= settings in the [Install] section, and DefaultInstance= for template
units). This means they are not meant to be enabled using systemctl.

version
=======
$ apt info qemu-guest-agent
Package: qemu-guest-agent
Version: 1:4.2-3ubuntu6.1
Priority: optional
Section: universe/misc
Source: qemu
Origin: Ubuntu

solution
========
$ cat /usr/lib/systemd/system/qemu-guest-agent.service
[Unit]
Description=QEMU Guest Agent
BindsTo=dev-virtio\x2dports-org.qemu.guest_agent.0.device
After=dev-virtio\x2dports-org.qemu.guest_agent.0.device

[Service]
ExecStart=-/usr/sbin/qemu-ga
Restart=always
RestartSec=0

[Install]
XXX add here the proper required target.

[Dan Streetman (ddstreet)]

> sudo systemctl enable qemu-guest-agent

that's not how you use it; it's a static service. You don't manually enable/disable it, nor do you start/stop it.

the package provides a udev rule to setup SYSTEMD_WANTS for the agent for any 'org.qemu.guest_agent.0' device, in /lib/udev/rules.d/60-qemu-guest-agent.rules.

See docs here: https://wiki.libvirt.org/page/Qemu_guest_agent

[Paride Legovini (paride)]

Hi Vincent,

I think the systemd service is working as intended. Other than the output you pasted, running `systemctl enable qemu-guest-agent` prints the following:

[...]
Possible reasons for having this kind of units are:
• A unit may be statically enabled by being symlinked from another unit's
  .wants/ or .requires/ directory.
• A unit's purpose may be to act as a helper for some other unit which has
  a requirement dependency on it.
• A unit may be started when needed via activation (socket, path, timer,
  D-Bus, udev, scripted systemctl call, ...).
• In case of template units, the unit is meant to be enabled with some

qemu-guest-agent falls in the third case, and it's activate by the udev rule defined in:

  /lib/udev/rules.d/60-qemu-guest-agent.rules

which is installed by the same package.

I'm setting the status to Incomplete for the moment. If my reply is not enough to solve your issue and you believe you found a bug in Ubuntu please explain the problem you are facing more in detail, possibly with steps to reproduce it, then change the report status back to New and we'll look at it again. Thanks!

[Vincent JARDIN (vjardin)]

Thanks Dan and Paride for your followup. I'd like this issue to be invalid, but it seems qemu agent is not run by udev:

qm agent 100 ping && echo ok
QEMU guest agent is not running

vm$ sudo systemctl start qemu-guest-agent

qm agent 100 ping && echo ok
ok

it seems that:
===
cat /lib/udev/rules.d/60-qemu-guest-agent.rules 1 ↵
SUBSYSTEM=="virtio-ports", ATTR{name}=="org.qemu.guest_agent.0", \
  TAG+="systemd" ENV{SYSTEMD_WANTS}="qemu-guest-agent.service"
===

is not enough to kick qemu-ga:
vm$ ps ax | grep qemu-ga 373283 ? Ss 0:00 /usr/sbin/qemu-ga

So, if it is not a systemctl / qemu packaging issue, what am I missing ?

thank you,

[Paride Legovini (paride)]

Hi,

I set it to Invalid then. I didn't see Dan had already done so when I replied to the bug and changed its status.

The problem you describe may depend on how the guest is started, as the udev rule has a trigger on virtio-ports. If you believe this is a bug and not a configuration problem please file a new bug report describing how to create the VM and possibly attaching the libvirt XML domain definition. Thanks!

[Christian Ehrhardt  (paelzer)]

The trigger it needs is the virtio port being set up.
Do you have:
  /dev/virtio-ports/org.qemu.guest_agent.0
in your guest?

If not your guest definition will need it, for config examples see
  https://wiki.qemu.org/Features/GuestAgent
  https://wiki.libvirt.org/page/Qemu_guest_agent

If you have that /dev file then the service should have been started by the udev rule.

[Vincent JARDIN (vjardin)]

Hi Christian,

the guest is properly set:
# ls -la /dev/virtio-ports/org.qemu.guest_agent.0
lrwxrwxrwx 1 root root 11 May 25 19:07 /dev/virtio-ports/org.qemu.guest_agent.0 -> ../vport2p1
# lsof /dev/virtio-ports/org.qemu.guest_agent.0

In order to get the "qm agent 100 ping && echo ok" working, the following is needed:
# systemctl start qemu-guest-agent
# lsof /dev/virtio-ports/org.qemu.guest_agent.0
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
qemu-ga 444278 root 4u CHR 243,1 0t0 123 /dev/virtio-ports/../vport2p1

So, it is more likely a udev issue. Something is not properly set with the default settings.

[Christian Ehrhardt  (paelzer)]

Odd, it worked on all systems I've seen so far - it is not generally failing.

Can you check your `systemd-analyze dump` output if it contains something like the following:

-> Unit qemu-guest-agent.service:
        Description: QEMU Guest Agent
        Instance: n/a
        Unit Load State: loaded
        Unit Active State: active
        State Change Timestamp: Mon 2020-06-15 06:42:38 UTC
        Inactive Exit Timestamp: Mon 2020-06-15 06:42:38 UTC
        Active Enter Timestamp: Mon 2020-06-15 06:42:38 UTC
        Active Exit Timestamp: Mon 2020-06-15 06:42:38 UTC
        Inactive Enter Timestamp: Mon 2020-06-15 06:42:38 UTC
        May GC: no
        Need Daemon Reload: no
        Transient: no
        Perpetual: no
        Garbage Collection Mode: inactive
        Slice: system.slice
        CGroup: /system.slice/qemu-guest-agent.service
        CGroup realized: yes
        CGroup realized mask: memory devices pids bpf-firewall bpf-devices
        CGroup own mask: memory pids
        Invocation ID: 240ad62495ab42118bd00254da1d7f12
        Fragment Path: /lib/systemd/system/qemu-guest-agent.service
        Condition Timestamp: Mon 2020-06-15 06:42:38 UTC
        Condition Result: yes
        Assert Timestamp: Mon 2020-06-15 06:42:38 UTC
        Assert Result: yes
        Requires: sysinit.target (origin-default)
        Requires: system.slice (origin-file)
        BindsTo: dev-virtio\x2dports-org.qemu.guest_agent.0.device (origin-file)
        WantedBy: sys-devices-pci0000:00-0000:00:02.2-0000:03:00.0-virtio1-virtio\x2dports-vport1p1.device (destination-udev)
....
-> Unit sys-devices-pci0000:00-0000:00:02.2-0000:03:00.0-virtio1-virtio\x2dports-vport1p1.device:
        Description: /sys/devices/pci0000:00/0000:00:02.2/0000:03:00.0/virtio1/virtio-ports/vport1p1
        Instance: n/a
        Unit Load State: loaded
        Unit Active State: active
        State Change Timestamp: Tue 2020-06-09 11:11:56 UTC
        Inactive Exit Timestamp: Tue 2020-06-09 11:11:56 UTC
        Active Enter Timestamp: Tue 2020-06-09 11:11:56 UTC
        Active Exit Timestamp: n/a
        Inactive Enter Timestamp: n/a
        May GC: no
        Need Daemon Reload: no
        Transient: no
        Perpetual: no
        Garbage Collection Mode: inactive
        Slice: n/a
        CGroup: n/a
        CGroup realized: no
        Invocation ID: cb54e03654164f7e8b5d0019576e7767
        Following Set Member: dev-vport1p1.device
        Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
        Wants: qemu-guest-agent.service (origin-udev)
        References: qemu-guest-agent.service (origin-udev)
        StopWhenUnneeded: no
        RefuseManualStart: no
        RefuseManualStop: no
        DefaultDependencies: yes
        OnFailureJobMode: replace
        IgnoreOnIsolate: yes
        Device State: plugged
        Sysfs Path: /sys/devices/pci0000:00/0000:00:02.2/0000:03:00.0/virtio1/virtio-ports/vport1p1
        Found: found-udev
        udev SYSTEMD_WANTS: qemu-guest-agent.service

[Vincent JARDIN (vjardin)]

I installed twice a Ubuntu 20.04 / Server on two different proxmox systems and for both I get this same issue. So, on my side, it seems to happen for 100% of the cases.

Regarding:
  # sudo systemd-analyze dump

I do not have any "Unit qemu-guest-agent.service":
  # sudo systemd-analyze dump | grep qemu
 Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
 Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
-> Unit dev-virtio\x2dports-org.qemu.guest_agent.0.device:
 Description: /dev/virtio-ports/org.qemu.guest_agent.0

but the following:
-> Unit dev-vport2p1.device:
        Description: /dev/vport2p1
        Instance: n/a
        Unit Load State: loaded
        Unit Active State: active
        State Change Timestamp: Mon 2020-05-25 19:07:45 UTC
        Inactive Exit Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Enter Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Exit Timestamp: n/a
        Inactive Enter Timestamp: n/a
        May GC: no
        Need Daemon Reload: no
        Transient: no
        Perpetual: no
        Garbage Collection Mode: inactive
        Slice: n/a
        CGroup: n/a
        CGroup realized: no
        Invocation ID: 843668b952cb48bc95327db579fc3393
        Following: sys-devices-pci0000:00-0000:00:08.0-virtio2-virtio\x2dports-vport2p1.device
        Following Set Member: sys-devices-pci0000:00-0000:00:08.0-virtio2-virtio\x2dports-vport2p1.device
        Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
        StopWhenUnneeded: no
        RefuseManualStart: no
        RefuseManualStop: no
        DefaultDependencies: yes
        OnFailureJobMode: replace
        IgnoreOnIsolate: yes
        Device State: plugged
        Sysfs Path: /sys/devices/pci0000:00/0000:00:08.0/virtio2/virtio-ports/vport2p1
        Found: found-udev
[...]
-> Unit sys-devices-pci0000:00-0000:00:08.0-virtio2-virtio\x2dports-vport2p1.device:
        Description: /sys/devices/pci0000:00/0000:00:08.0/virtio2/virtio-ports/vport2p1
        Instance: n/a
        Unit Load State: loaded
        Unit Active State: active
        State Change Timestamp: Mon 2020-05-25 19:07:45 UTC
        Inactive Exit Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Enter Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Exit Timestamp: n/a
        Inactive Enter Timestamp: n/a
        May GC: no
        Need Daemon Reload: no
        Transient: no
        Perpetual: no
        Garbage Collection Mode: inactive
        Slice: n/a
        CGroup: n/a
        CGroup realized: no
        Invocation ID: 286cd2fc5950453c841840794c43c957
        Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
        Following Set Member: dev-vport2p1.device
        StopWhenUnneeded: no
        RefuseManualStart: no
        RefuseManualStop: no
        DefaultDependencies: yes
        OnFailureJobMode: replace
        IgnoreOnIsolate: yes
        Device State: plugged
        Sysfs Path: /sys/devices/pci0000:00/0000:00:08.0/virtio2/virtio-ports/vport2p1
        Found: found-udev
[...]
-> Unit dev-virtio\x2dports-org.qemu.guest_agent.0.device:
        Description: /dev/virtio-ports/o...

[Christian Ehrhardt  (paelzer)]

For reference, the same grep that you did:

ubuntu@focal:~$ sudo systemd-analyze dump | grep qemu
        RequiredBy: qemu-guest-agent.service (destination-default)
        Before: qemu-guest-agent.service (destination-default)
        ReferencedBy: qemu-guest-agent.service (destination-default)
-> Unit qemu-guest-agent.service:
        CGroup: /system.slice/qemu-guest-agent.service
        Fragment Path: /lib/systemd/system/qemu-guest-agent.service
        BindsTo: dev-virtio\x2dports-org.qemu.guest_agent.0.device (origin-file)
        After: dev-virtio\x2dports-org.qemu.guest_agent.0.device (origin-file)
        References: dev-virtio\x2dports-org.qemu.guest_agent.0.device (origin-file)
                Command Line: /usr/sbin/qemu-ga
        Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
        Before: qemu-guest-agent.service (destination-default)
        ReferencedBy: qemu-guest-agent.service (destination-default)
        Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
        Wants: qemu-guest-agent.service (origin-udev)
        References: qemu-guest-agent.service (origin-udev)
        udev SYSTEMD_WANTS: qemu-guest-agent.service
        ConflictedBy: qemu-guest-agent.service (destination-default)
        After: qemu-guest-agent.service (destination-default)
        ReferencedBy: qemu-guest-agent.service (destination-default)
-> Unit dev-virtio\x2dports-org.qemu.guest_agent.0.device:
        Description: /dev/virtio-ports/org.qemu.guest_agent.0
        BoundBy: qemu-guest-agent.service (destination-file)
        Before: qemu-guest-agent.service (destination-file)
        ReferencedBy: qemu-guest-agent.service (destination-file)
        Before: qemu-guest-agent.service (destination-file)
        ReferencedBy: qemu-guest-agent.service (destination-file)
        RequiredBy: qemu-guest-agent.service (destination-file)
        Before: qemu-guest-agent.service (destination-file)
        ReferencedBy: qemu-guest-agent.service (destination-file)

[Christian Ehrhardt  (paelzer)]

In your case it finds the device and tries to follow:
        Following Set Member: sys-devices-pci0000:00-0000:00:08.0-virtio2-virtio\x2dports-vport2p1.device
        Following Set Member: dev-vport2p1.device
It knows it is udev related
        Found: found-udev
But your path is different:
        Sysfs Path: /sys/devices/pci0000:00/0000:00:08.0/virtio2/virtio-ports/vport2p1

You miss (the following is only in my log) to reach:
        Wants: qemu-guest-agent.service (origin-udev)
        References: qemu-guest-agent.service (origin-udev)
        ...
        udev SYSTEMD_WANTS: qemu-guest-agent.service

It seems you have multiple virtio-serial ports.
Yours is "vport2p1"?
What is the first one?
Maybe an extra console?
Mine looks like:
$ ll /dev/virtio-ports/
total 0
drwxr-xr-x 2 root root 60 Jun 16 05:19 ./
drwxr-xr-x 18 root root 3920 Jun 16 05:19 ../
lrwxrwxrwx 1 root root 11 Jun 16 05:19 org.qemu.guest_agent.0 -> ../vport1p1

Hmm ... proxmox ... maybe that is something special in the setup and that is why you see it but no one else so far?

Do you have a guest XML of libvirt in the host that you can share so one can try to set up the same over here?

[Vincent JARDIN (vjardin)]

There is a single virtio port:

# ls -la /dev/virtio-ports/
lrwxrwxrwx 1 root root 11 May 25 19:07 org.qemu.guest_agent.0 -> ../vport2p1
# ls -la /dev/vport*
crw------- 1 root root 243, 1 May 25 19:07 /dev/vport2p1

As you can notice, there is 1 single deivce, but it starts from vport2p1.

It can be confirmed with the the proxmox's qemu command line (there is no virsh/libvirt):
/usr/bin/kvm -id 100 -name XYZ
  -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait
  -mon chardev=qmp,mode=control
  -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5
  -mon chardev=qmp-event,mode=control
  -pidfile /var/run/qemu-server/100.pid
  -daemonize
  -smbios type=1,uuid=xyz-2b19-4f25-xyz-xyz
  -smp 4,sockets=1,cores=4,maxcpus=4
  -nodefaults
  -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg
  -vnc unix:/var/run/qemu-server/100.vnc,password
  -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep
  -m 2048
  -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e
  -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f
  -device vmgenid,guid=6a11a495-xyz-4773-xyz-xyz
  -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2
  -device usb-tablet,id=tablet,bus=uhci.0,port=1
  -device VGA,id=vga,bus=pci.0,addr=0x2
  -chardev socket,path=/var/run/qemu-server/100.qga,server,nowait,id=qga0
  -device virtio-serial,id=qga0,bus=pci.0,addr=0x8
  -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3
  -iscsi initiator-name=iqn.1993-08.org.debian:01:xyz
  -drive file=/var/lib/vz/template/iso/ubuntu-20.04-live-server-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=threads
  -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200
  -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5
  -drive file=/var/lib/vz/images/100/vm-100-disk-0.qcow2,if=none,id=drive-scsi0,format=qcow2,cache=none,aio=native,detect-zeroes=on
  -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100
  -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on
  -device virtio-net-pci,mac=00:ca:fe:00:20:13,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
  -netdev type=tap,id=net1,ifname=tap100i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on
  -device virtio-net-pci,mac=00:ca:fe:00:20:12,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301
  -machine type=pc+pve0

[Christian Ehrhardt  (paelzer)]

Hmm, not too many differences to mine:

-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \

The -pci being a difference is an alias to change per underlying architecture - it still would be the same thing eventually.

Is the following much different in your case?
$ sudo lshw -C input
  *-pnp00:01
       product: PnP device PNP0303
       physical id: 2
       capabilities: pnp
       configuration: driver=i8042 kbd
  *-pnp00:02
       product: PnP device PNP0f13
       physical id: 3
       capabilities: pnp
       configuration: driver=i8042 aux

[Vincent JARDIN (vjardin)]

You welcome, here it is:

$ sudo lshw -C input
  *-usb
       description: Human interface device
       product: QEMU USB Tablet
       vendor: QEMU
       physical id: 1
       bus info: usb@1:1
       version: 0.00
       serial: 28754-0000:00:01.2-1
       capabilities: usb-2.00
       configuration: driver=usbhid maxpower=100mA speed=12Mbit/s
  *-pnp00:01
       product: PnP device PNP0303
       physical id: 2
       capabilities: pnp
       configuration: driver=i8042 kbd
  *-pnp00:02
       product: PnP device PNP0f13
       physical id: 3
       capabilities: pnp
       configuration: driver=i8042 aux

[Christian Ehrhardt  (paelzer)]

hmm thanks vincent, nothing that would obviously be too suspicious.
I wonder if I need to setup a proxmox VM somewhere ...

[Vincent JARDIN (vjardin)]

The setup is quite straightforward, but it requires a standalone server:
  https://www.proxmox.com/en/downloads/item/proxmox-ve-6-2-iso-installer

If you can't find any environment for such setup, ping me thru a private email (vjardin AT freeDOTfr), I'll look for a mean we could share.

Since udev has to be used, I'll try some more logs/debugging too...

[Christian Ehrhardt  (paelzer)]

I installed a proxmox 6.2 for testing purposes.
On that I installed a Ubuntu 20.04 Guest.
The setup has a checkbox for "qemu-agent" which I ticked.
(also worth as a test of the new installer in that env)

Installed qmeu-guest-agent in there and rebooted.
E voila - up and running.

It does not run right after install, probably since the udev event was issued before the rule existed. But I'm not sure if that is a big issue ...

@Vincent - as I understood you in your case the service comes never up, not even on a reboot- right?

[Vincent JARDIN (vjardin)]

right.

I'll download the "current" proxmox iso 6.2 and re-do a fresh installation too... just in case. I did twice this installation ~1 month a go.

I'll capture and put more inside for debugging. I'll keep you posted. I hope I am not hunting a ghost...

[Vincent JARDIN (vjardin)]

Hi Christian,

sorry for the delay, I had a HW issue before getting the new server up with proxmox. Starting from a fresh setup, you are right, it works without any issues.

So let's ignore this request, there is no need of any systemd modification.

thanks again,

[Christian Ehrhardt  (paelzer)]

Thanks for coming back Vincent, I'll mark as invalid then.

[Paride Legovini (paride)]

Thanks Vincent for following up and glad to hear the problem is gone. I'm setting this report to Invalid then. Should you face new issues do not hesitate to open a new one.