I installed twice a Ubuntu 20.04 / Server on two different proxmox systems and for both I get this same issue. So, on my side, it seems to happen for 100% of the cases.

Regarding:
  # sudo systemd-analyze dump

I do not have any "Unit qemu-guest-agent.service":
  # sudo systemd-analyze dump | grep qemu
	Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
	Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
-> Unit dev-virtio\x2dports-org.qemu.guest_agent.0.device:
	Description: /dev/virtio-ports/org.qemu.guest_agent.0

but the following:
-> Unit dev-vport2p1.device:
        Description: /dev/vport2p1
        Instance: n/a
        Unit Load State: loaded
        Unit Active State: active
        State Change Timestamp: Mon 2020-05-25 19:07:45 UTC
        Inactive Exit Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Enter Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Exit Timestamp: n/a
        Inactive Enter Timestamp: n/a
        May GC: no
        Need Daemon Reload: no
        Transient: no
        Perpetual: no
        Garbage Collection Mode: inactive
        Slice: n/a
        CGroup: n/a
        CGroup realized: no
        Invocation ID: 843668b952cb48bc95327db579fc3393
        Following: sys-devices-pci0000:00-0000:00:08.0-virtio2-virtio\x2dports-vport2p1.device
        Following Set Member: sys-devices-pci0000:00-0000:00:08.0-virtio2-virtio\x2dports-vport2p1.device
        Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
        StopWhenUnneeded: no
        RefuseManualStart: no
        RefuseManualStop: no
        DefaultDependencies: yes
        OnFailureJobMode: replace
        IgnoreOnIsolate: yes
        Device State: plugged
        Sysfs Path: /sys/devices/pci0000:00/0000:00:08.0/virtio2/virtio-ports/vport2p1
        Found: found-udev
[...]
-> Unit sys-devices-pci0000:00-0000:00:08.0-virtio2-virtio\x2dports-vport2p1.device:
        Description: /sys/devices/pci0000:00/0000:00:08.0/virtio2/virtio-ports/vport2p1
        Instance: n/a
        Unit Load State: loaded
        Unit Active State: active
        State Change Timestamp: Mon 2020-05-25 19:07:45 UTC
        Inactive Exit Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Enter Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Exit Timestamp: n/a
        Inactive Enter Timestamp: n/a
        May GC: no
        Need Daemon Reload: no
        Transient: no
        Perpetual: no
        Garbage Collection Mode: inactive
        Slice: n/a
        CGroup: n/a
        CGroup realized: no
        Invocation ID: 286cd2fc5950453c841840794c43c957
        Following Set Member: dev-virtio\x2dports-org.qemu.guest_agent.0.device
        Following Set Member: dev-vport2p1.device
        StopWhenUnneeded: no
        RefuseManualStart: no
        RefuseManualStop: no
        DefaultDependencies: yes
        OnFailureJobMode: replace
        IgnoreOnIsolate: yes
        Device State: plugged
        Sysfs Path: /sys/devices/pci0000:00/0000:00:08.0/virtio2/virtio-ports/vport2p1
        Found: found-udev
[...]
-> Unit dev-virtio\x2dports-org.qemu.guest_agent.0.device:
        Description: /dev/virtio-ports/org.qemu.guest_agent.0
        Instance: n/a
        Unit Load State: loaded
        Unit Active State: active
        State Change Timestamp: Mon 2020-05-25 19:07:45 UTC
        Inactive Exit Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Enter Timestamp: Mon 2020-05-25 19:07:45 UTC
        Active Exit Timestamp: n/a
        Inactive Enter Timestamp: n/a
        May GC: no
        Need Daemon Reload: no
        Transient: no
        Perpetual: no
        Garbage Collection Mode: inactive
        Slice: n/a
        CGroup: n/a
        CGroup realized: no
        Invocation ID: f97e7149150d4db585951d7c843685ca
        Following: sys-devices-pci0000:00-0000:00:08.0-virtio2-virtio\x2dports-vport2p1.device
        Following Set Member: sys-devices-pci0000:00-0000:00:08.0-virtio2-virtio\x2dports-vport2p1.device
        Following Set Member: dev-vport2p1.device
        StopWhenUnneeded: no
        RefuseManualStart: no
        RefuseManualStop: no
        DefaultDependencies: yes
        OnFailureJobMode: replace
        IgnoreOnIsolate: yes
        Device State: plugged
        Sysfs Path: /sys/devices/pci0000:00/0000:00:08.0/virtio2/virtio-ports/vport2p1
        Found: found-udev

best regards,