hv-fcopy-daemon service fails to start if /dev/vmbus/hv_fcopy doesn't exist

[Dan Watkins]

On VMs launched in Azure, /dev/vmbus/hv_fcopy doesn't exist, so hv-fcopy-daemon fails to start. This means that boot is degraded by default:

$ systemctl list-units --failed --no-legend
hv-fcopy-daemon.service loaded failed failed Hyper-V File Copy Protocol Daemon

$ systemctl is-system-running
degraded

which makes detecting other failures much harder to automate.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-cloud-tools-common 4.15.0-15.16
ProcVersionSignature: User Name 4.15.0-1004.4-azure 4.15.15
Uname: Linux 4.15.0-1004-azure x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access '/dev/snd/': No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
CRDA: N/A
Date: Mon Apr 23 16:27:47 2018
Dependencies:

IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Microsoft Corporation Virtual Machine
PackageArchitecture: all
PciMultimedia:

ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-1004-azure root=UUID=fda9466a-9068-447f-8572-6d8d310eabd3 ro console=tty1 console=ttyS0 earlyprintk=ttyS0 rootdelay=300
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-1004-azure N/A
 linux-backports-modules-4.15.0-1004-azure N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/02/2017
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 090007
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: 7.0
dmi.chassis.asset.tag: 7783-7084-3265-9085-8269-3286-77
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: 7.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr090007:bd06/02/2017:svnMicrosoftCorporation:pnVirtualMachine:pvr7.0:rvnMicrosoftCorporation:rnVirtualMachine:rvr7.0:cvnMicrosoftCorporation:ct3:cvr7.0:
dmi.product.name: Virtual Machine
dmi.product.uuid: 44FD5ABF-DE16-AD4A-B2E0-6E70167950AF
dmi.product.version: 7.0
dmi.sys.vendor: Microsoft Corporation

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.16 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.17-rc2

[Eduardo Bonato (ebonato)]

Hi Folks,
I can confirm that bug still on Latest kernel 4.18.

~$ sudo dpkg -l | grep azure
ii linux-azure 4.18.0.1013.12 amd64 Complete Linux kernel for Azure systems.
ii linux-azure-cloud-tools-4.15.0-1036 4.15.0-1036.38 amd64 Linux kernel version specific cloud tools for version 4.15.0-1036
ii linux-azure-cloud-tools-4.18.0-1011 4.18.0-1011.11~18.04.1 amd64 Linux kernel version specific cloud tools for version 4.18.0-1011
ii linux-azure-cloud-tools-4.18.0-1013 4.18.0-1013.13~18.04.1 amd64 Linux kernel version specific cloud tools for version 4.18.0-1013
ii linux-azure-headers-4.15.0-1036 4.15.0-1036.38 all Header files related to Linux kernel version 4.15.0
ii linux-azure-headers-4.18.0-1011 4.18.0-1011.11~18.04.1 all Header files related to Linux kernel version 4.18.0
ii linux-azure-headers-4.18.0-1013 4.18.0-1013.13~18.04.1 all Header files related to Linux kernel version 4.18.0
ii linux-azure-tools-4.15.0-1036 4.15.0-1036.38 amd64 Linux kernel version specific tools for version 4.15.0-1036
ii linux-azure-tools-4.18.0-1011 4.18.0-1011.11~18.04.1 amd64 Linux kernel version specific tools for version 4.18.0-1011
ii linux-azure-tools-4.18.0-1013 4.18.0-1013.13~18.04.1 amd64 Linux kernel version specific tools for version 4.18.0-1013
ii linux-cloud-tools-4.15.0-1036-azure 4.15.0-1036.38 amd64 Linux kernel version specific cloud tools for version 4.15.0-1036
ii linux-cloud-tools-4.18.0-1011-azure 4.18.0-1011.11~18.04.1 amd64 Linux kernel version specific cloud tools for version 4.18.0-1011
ii linux-cloud-tools-4.18.0-1013-azure 4.18.0-1013.13~18.04.1 amd64 Linux kernel version specific cloud tools for version 4.18.0-1013
ii linux-cloud-tools-azure 4.18.0.1013.12 amd64 Linux kernel versioned cloud tools for Azure systems.
ii linux-headers-4.15.0-1036-azure 4.15.0-1036.38 amd64 Linux kernel headers for version 4.15.0 on 64 bit x86 SMP
ii linux-headers-4.18.0-1011-azure 4.18.0-1011.11~18.04.1 amd64 Linux kernel headers for version 4.18.0 on 64 bit x86 SMP
ii linux-headers-4.18.0-1013-azure 4.18.0-1013.13~18.04.1 amd64 Linux kernel headers for version 4.18.0 on 64 bit x86 SMP
ii linux-headers-azure 4.18.0.1013.12 amd64 Linux kernel headers for Azure systems.
rc linux-image-4.15.0-1030-azure 4.15.0-1030.31 amd64 Signed kernel image azure
rc linux-image-4.15.0-1031-azure 4.15.0-1031.32 amd64 Signed kernel image azure
rc linux-image-4.15.0-1032-azure 4.15.0-1032.33 ...

[Eduardo Bonato (ebonato)]

According to this MS Documentation:
https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/reference/integration-services

The hv_fcopy_daemon have a low impact on VM.
The role of this daemon isn't necessary to a Azure VM, as you will not use it to copy files from/to VM (This could be necessary if you are directly on Hyper-V Console )

Looking into debian sources:
https://sources.debian.org/src/linux/5.0.2-1~exp1/debian/hyperv-daemons.hv-fcopy-daemon.service/

You will see that a different condition exist to run this service daemon:
ConditionPathExists=/dev/vmbus/hv_fcopy

So just add this extra condition to ubuntu service description on
/lib/systemd/system/hv-fcopy-daemon.service

Restart your VM and you will see it is no more degraded.

[Bearsh (bearsh)]

@ebonato
in focal, the service fails because of that condition:
$ systemctl status hv-fcopy-daemon.service
● hv-fcopy-daemon.service - Hyper-V File Copy Protocol Daemon
     Loaded: loaded (/lib/systemd/system/hv-fcopy-daemon.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Mon 2020-11-09 10:03:59 UTC; 58s ago
             └─ ConditionPathExists=/dev/vmbus/hv_fcopy was not met

any advise?

[Brian Murray (brian-murray)]

Ubuntu 17.10 (Artful Aardvark) has reached end of life, so this bug will not be fixed for that specific release.

[Yuri Sevatz (yuri-sevatz)]

I was just looking into the hv_fcopy_daemon behavior for another reason (trying to support this on Gentoo), and I stumbled on this bug. Everything here mentioned above seems to be absolutely correct:

- Make sure the corresponding "Integration Services" setting is enabled in Hyper-V Manager, in this case:
  Launch "Hyper-V Manager" -> Right-Click Virtual Machine -> "Settings..." -> "Integration Services" -> Make sure "Guest Services" checkbox is Enabled)
- If the VM is already running, restart the VM, or you can run `systemctl start hv_fcopy_daemon`.

But I noticed when I installed my own Ubuntu 20.04, 22.04. 24.04 test machines on Windows 11, it appears that the prior mentioned "Guest services" checkbox is *disabled* by default after using either "New..." or "Quick Create..." when creating any virtual machine in latest Hyper-V Manager.

This appears to be both the source of confusion for users and the intended design according to microsoft's documentation, therefore according to Microsoft, the ConditionPathExists is definitely needed: Because those paths are dynamically craeted and destroyed whenever the corresponding kernel module is inserted/removed ( `/dev/vmbus/{hv_kvp,hv_vss,hv_fcopy}` )

[Yuri Sevatz (yuri-sevatz)]

There's one extra thought I had that could perhaps make this experience a lot better for users:

- If distros wanted to make units have WantedBy= on the same device they are writing BindsTo= on, then systemd might be able to dynamically start/stop the service automatically at runtime, whenever users enable/disable the individual "Integration Services" settings checkboxes in Hyper-V Manager on the host.

However, while this trick probably would work great for hv_kvp_daemon, hv_vss_daemon, and hv_fcopy_daemon on kernels LESS THAN 6.11; I discovered that on kernels GREATER OR EQUAL to 6.11, it appears that writing `ConditionPathExists=/dev/vmbus/hv_fcopy` is probably wrong now. This is due to the fact the driver for hv_fcopy has since been removed from Kernel 6.11 and has since been replaced with a uio driver located at `/sys/bus/vmbus/devices/eb765408-105f-49b6-b4aa-c123b64d17d4/uio` :

https://github.com/torvalds/linux/commit/ec314f61e4fc2d3dd6ea78aa18a5ac276eb1a8e3

https://github.com/torvalds/linux/commit/82b0945ce2c2d636d5e893ad50210875c929f257

So it appears that in Ubuntu releases that have Kernel 6.11, users may still be stumbling on this bugreport, precisely because ubuntu is incorrectly using `ConditionPathExists=/dev/vmbus/hv_fcopy` with the new `hv_fcopy_uio_daemon`, but accidentally kept the old path from `hv_fcopy_daemon`

IMO, for now it might make sense to just set ConditionPathExists= on that new path, specifically on these newer kernels.

[Yuri Sevatz (yuri-sevatz)]

Ah -- Please disregard what i said in my last comment about suggesting any changes to how the hv_*.service units should starts dynamically, because it actually already does this on `WantedBy=` when the corresponding device for the kernel module is created; because a udev rule already does this.

However, I think this too is an area where the hv_fcopy kernel module no longer exists either. So that specific udev rule for `vmbus/hv_fcopy` too likely needs a new strategy.

[Yuri Sevatz (yuri-sevatz)]

I just verified the behavior on all Ubuntu Pro versions for 16.04, 18.04, 20.04, 22.04:

- The behavior works great on all of these provided that the user remembers to check off "Guest services" in the Hyper-V Manager VM's Settings. Additionally, hv-fcopy-daemon.service, hv-kvp-daemon.service, and hv-vss-daemon.service were all smart enough to start/stop dynamically themselves at runtime whenever the respective integration services checkbox in the Hyper-V Manager are enabled/disabled.

^ So the bug described appears fixed in those releases.

I also tested Ubuntu Pro 24.04, and there's definitely an issue still that I had touched on in my last comment. In more detail:

1) `/lib/udev/rules.d/60-fcopy-daemon.rules` attempts to tag a virtual device in systemd aligning with the `vmbus/hv_fcopy` driver, and was previously how this systemd virtual device setup a dynamic "Wants" on hv-fcopy-daemon.service whenever the "Guest services" checkbox is enabled/disabled in Hyer-V Manager.

2) Because the "vmbus/hv_fcopy" driver is removed in kernel >= 6.11, the kernel never creates `/dev/vmbus/hv_fcopy` anymore, then it is impossible to satisfy the condition which hv-fcopy-daemon.service depends upon during startup, so it never willingly starts itself when booting.

3) Even when I get everything in my hyper-v VM setup correctly with "Guest Services" enabled, it appears that `/sys/bus/vmbus/devices/eb765408-105f-49b6-b4aa-c123b64d17d4/uio` path is never created; although the parent folder is dynamically created/destroyed anytime "Guest Services" is enabled/disabled (just no "uio" file inside it).

4) Manually launching `hv_fcopy_uio_daemon -n` as root when "Guest Services" is enabled in Hyper-V Settings causes it to exit with an error stating that it expects the above path to exist.

@ebonato This might require a separate patch for 24.04 and above, due to all those releases running kernels >= 6.11. hv_fcopy_uio_daemon just doesn't work at all on those kernels. It's probably related to this ticket in the overall symptom, although the issue now appears to be more caused by Microsoft's recent "uio" refactoring in kernel 6.11 and above.