[Hyper-V] KVP daemon fails to run scripts

Bug #1766857 reported by Ionut Lenghel
46
This bug affects 8 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
High
Unassigned
Bionic
Confirmed
High
Unassigned
Cosmic
Confirmed
High
Unassigned
linux-azure (Ubuntu)
Confirmed
High
Unassigned
Bionic
Confirmed
High
Unassigned
Cosmic
Confirmed
High
Unassigned

Bug Description

While testing Bionic daily build with kernel 4.15.0-20-generic we saw that there are 2 issues with the KVP daemon:

1. KVP daemon crash after boot. Opened a different bug for this: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1779640

2. After the KVP daemon is being started the following messages appear:

Apr 25 04:45:25 bionicDaily hv_kvp_daemon[1895]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 25 04:45:25 bionicDaily hv_kvp_daemon[1895]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found

The above binaries are present on the system, but the their actual path is /usr/sbin/hv_get_dns_info and /usr/sbin/hv_get_dhcp_info. Either the hv_get_dhcp_info and hv_get_dns_info binaries should be placed in the location where the daemon is looking for (/usr/libexec/hypervkvpd/), or the daemon should be set to search for the binaries in the /usr/sbin directory.

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1766857

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: bionic
Chris Valean (cvalean)
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Changed in linux (Ubuntu):
importance: Undecided → High
Changed in linux-azure (Ubuntu Bionic):
importance: Undecided → High
tags: added: kernel-da-key kernel-hyper-v
Revision history for this message
Joseph Salisbury (jsalisbury) wrote : Re: [Hyper-V] KVP daemon fails to start

Did this bug just start happening with the 4.15.0-20 kernel? Were there prior 18.04 kernels that did not exhibit this bug?

Revision history for this message
Ionut Lenghel (ilenghel) wrote :

I tested back to the daily build from 2nd of March and I can confirm the second issue (regarding the binaries path), but not the first one (regarding the daemon crash). On the build from the 2nd of March the linux-tools and linux-cloud-tools version is 4.15.0-10.11

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-azure (Ubuntu Bionic):
status: New → Confirmed
Changed in linux-azure (Ubuntu):
status: New → Confirmed
Revision history for this message
giacomo (chiaravalli) wrote :

I confirm the second, the daemon hv_kvp_daemon not find the files:
/usr/libexec/hypervkvpd/hv_get_dns_info
/usr/libexec/hypervkvpd/hv_get_dhcp_info
With linux-cloud-tools version 4.15.0-22.24

Revision history for this message
Joshua R. Poulson (jrp) wrote :

In my own VMs, this does not affect linux-azure, it only affects the generic 4.15 kernel.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Commit 7fa32e5ec28b was mentioned in the descriptions. However, that commit has been in mainlien since 4.15-rc3, so it's always been in 18.04.

We may want to split the two issues in to two separate bugs, so they each get addressed.

Do both issues still happen with the latest Bionic kernel? Per comment #6, it sounds liek the second issues still happens. How about the first issue, where the KVP daemon crashes after approximatively 2 minutes of uptime?

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Commit 7fa32e5ec28b was actually mentioned in bug 1664663, an I don't think it factors into this bug.

Changed in linux (Ubuntu):
status: Confirmed → In Progress
Changed in linux (Ubuntu Bionic):
status: Confirmed → In Progress
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux-azure (Ubuntu):
status: Confirmed → Invalid
Changed in linux-azure (Ubuntu Bionic):
status: Confirmed → Invalid
Guido (mcnomad81)
information type: Public → Public Security
information type: Public Security → Public
Revision history for this message
Ionut Lenghel (ilenghel) wrote :

I have tested Bionic with kernel 4.15.0-22-generic, linux-cloud-tools-4.15.0-22.24 and linux-tools-4.15.0-22.24 and both issues still occur.

While testing the above mentioned kernel, I saw the following behavior:

1. Right after booting the system the KVP daemon reports it is active (running). The command "systemctl status hv-kvp-daemon" returns the following output:

hv-kvp-daemon.service - Hyper-V KVP Protocol Daemon
   Loaded: loaded (/lib/systemd/system/hv-kvp-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-06-07 11:43:22 UTC; 55s ago
 Main PID: 1363 (hv_kvp_daemon)
    Tasks: 1 (limit: 4496)
   CGroup: /system.slice/hv-kvp-daemon.service
           └─1363 /usr/lib/linux-tools/4.15.0-22-generic/hv_kvp_daemon -n

Jun 07 11:43:22 bionic systemd[1]: Started Hyper-V KVP Protocol Daemon.
Jun 07 11:43:22 bionic KVP[1363]: KVP starting; pid is:1363
Jun 07 11:43:22 bionic KVP[1363]: KVP LIC Version: 3.1

2. After approximately 2 minutes the KVP daemon enters in the failed state (issue 1).
3. Manually starting the daemon with the command "systemctl start hv-kvp-daemon" works perfectly fine.
4. "systemctl status hv-kvp-daemon" will now show the second issue:

hv-kvp-daemon.service - Hyper-V KVP Protocol Daemon
   Loaded: loaded (/lib/systemd/system/hv-kvp-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-06-07 11:47:29 UTC; 8s ago
 Main PID: 1995 (hv_kvp_daemon)
    Tasks: 1 (limit: 4496)
   CGroup: /system.slice/hv-kvp-daemon.service
           └─1995 /usr/lib/linux-tools/4.15.0-22-generic/hv_kvp_daemon -n

Jun 07 11:47:29 bionic systemd[1]: Started Hyper-V KVP Protocol Daemon.
Jun 07 11:47:29 bionic KVP[1995]: KVP starting; pid is:1995
Jun 07 11:47:29 bionic KVP[1995]: KVP LIC Version: 3.1
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found

5. Rebooting the system after this point will no longer trigger the first issue, only the second one. Even stopping the VM and turning it back ON does NOT trigger the first issue again.

Could it be that while the hv-kvp-daemon service shows the two files (hv_get_dns_info and hv_get_dhcp_info) are not found, it will not enter in the failed state? Meaning that the two apparently separate issues, are actually related. I am not 100% sure about this.

In order to trigger the first issue again, the Integration Service corresponding to the KVP daemon has to be disabled and re-enabled, and the VM has to be rebooted. This is just the way I managed to reproduce the first issue again, I am not sure if there are other ways to trigger it again.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Since they might be related, lets focus on the second issue first, which is cause by the missing files.

I'll set up a test system to try and reproduce.

It seems like this is a regression, but it would be good to confirm. Can you test these two kernels?

Last 4.14 Bionic kernel:
https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/ppa2/+build/14223014

First 4.15 Bionic kernel:
https://launchpad.net/ubuntu/+source/linux/4.15.0-9.10/+build/14326410

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

I built a test kernel which can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1766857/

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

Revision history for this message
Ionut Lenghel (ilenghel) wrote :

Regarding comment #11:
I have tested both kernels provided with the following results:

- 4.14 Bionic kernel (4.14.0-16-generic) does NOT exhibit issue #2 (missing files), but it does exhibit issue #1 (KVP daemon failure after boot). Linux-cloud-tools version installed is 4.14.0-16.19.

- 4.15 Bionic kernel (4.15.0-9-generic) behaves the same as 4.14, meaning that issue #2 (missing files) is NOT present, but issue #1 (KVP daemon failure after boot) shows up. Linux-cloud-tools version is 4.15.0-9.10.

Regarding comment #12:
I have tested the kernel from the link and it seems that this kernel does NOT exhibit issue #2 (but it does exhibit issue #1).

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

At this point, we should probably have two separate bugs to address each problem. Can we keep this bug for the second issue and create a new bug for the first issue?

Revision history for this message
Ionut Lenghel (ilenghel) wrote :

I have opened a different bug for the first issue: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1779640

Regarding the second issue, as mentioned in comment #13, the tested kernels did not exhibit issue #2.

description: updated
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Did you notice any other issues that occur when testing the test kernel posted in comment #12. The second issue posted in the bug is resolved by changing the definition of KVP_CONFIG_LOC in hv_kvp_daemon.c:

---
 tools/hv/hv_kvp_daemon.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
index 4c99c57..e969057 100644
--- a/tools/hv/hv_kvp_daemon.c
+++ b/tools/hv/hv_kvp_daemon.c
@@ -95,7 +95,7 @@ static struct utsname uts_buf;
 #define KVP_CONFIG_LOC "/var/lib/hyperv"

 #ifndef KVP_SCRIPTS_PATH
-#define KVP_SCRIPTS_PATH "/usr/libexec/hypervkvpd/"
+#define KVP_SCRIPTS_PATH "/usr/sbin/"
 #endif

 #define KVP_NET_DIR "/sys/class/net/"
--
2.7.4

Revision history for this message
Chris Valean (cvalean) wrote :

Joe, for the 2nd issue initially reported here, about "After the KVP daemon is being started the following messages appear".
We've noticed a regression for this in 18.10 - daily's and even GA.

Can you please add Cosmic 18.10 to the bug here and re-open or should I send a new bug report for it?
Thank you!

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Hi Chris, I added a Cosmic 18.10 bug task. Does this bug only happen in Cosmic, or is it still happening in Bionic?

Revision history for this message
Chris Valean (cvalean) wrote :

Issue with 'Invalid kvp dependency scripts path /usr/libexec/hypervkvpd' remains in latest kernel-image bionic 4.15.0-38.41.

Verified azure-kernel 4.15.0-1030.31 for bionic, same VM and environment, and the issue does *not* appear.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Hmm, I wonder if the linux-azure kernel has and extra patch(s) that the generic kernel does not. I'll do some comparisons.

Changed in linux (Ubuntu Cosmic):
status: In Progress → Confirmed
Changed in linux (Ubuntu Bionic):
status: In Progress → Confirmed
Changed in linux (Ubuntu):
status: In Progress → Confirmed
assignee: Joseph Salisbury (jsalisbury) → nobody
Changed in linux (Ubuntu Bionic):
assignee: Joseph Salisbury (jsalisbury) → nobody
Changed in linux (Ubuntu Cosmic):
assignee: Joseph Salisbury (jsalisbury) → nobody
Brad Figg (brad-figg)
tags: added: cscc
Revision history for this message
Robert Collins (lifeless) wrote :

Just created a new VM and wondered why enhanced mode logins in hyper-V weren't working and encountered this.

$ uname -a
Linux cognite-lifeless 5.3.0-45-generic #37-Ubuntu SMP Thu Mar 26 20:41:27 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 19.10
Release: 19.10
Codename: eoan

Revision history for this message
Robert Collins (lifeless) wrote :

Can confirm the Azure kernel also fixes enhanced sessions when combined with appropriate windows side metadata shenanigans.

Florian Wolff (flocom)
no longer affects: focal (Ubuntu)
no longer affects: focal (Ubuntu Bionic)
no longer affects: focal (Ubuntu Cosmic)
Revision history for this message
Florian Wolff (flocom) wrote :

Uups, wrong tagging. This bug does still affect linux (Ubuntu Focal)

Revision history for this message
Murph (murph9000) wrote :

I'm seeing the problem with /usr/libexec/hypervkvpd being used by hv_kvp_daemon, when the scripts are in /usr/sbin on a newly created Ubuntu 22.04 VM (on Windows 11 Pro 22H2 Hyper-V), on the linux-azure kernel.

murph@Ubuntu-Virtual-Machine:~$ uname -a
Linux Ubuntu-Virtual-Machine 5.15.0-1035-azure #42-Ubuntu SMP Tue Feb 28 19:41:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
murph@Ubuntu-Virtual-Machine:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
murph@Ubuntu-Virtual-Machine:~$

There's an endless stream of log noise in syslog, at the fairly high rate of 8 logs per 10 seconds, with the same thing repeated each time:

Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35236]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35238]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35246]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35248]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35256]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35258]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35266]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 6 06:56:31 Ubuntu-Virtual-Machine hv_kvp_daemon[35268]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found

Murph (murph9000)
tags: added: jammy
Murph (murph9000)
Changed in linux-azure (Ubuntu):
status: Invalid → Confirmed
Revision history for this message
Murph (murph9000) wrote :

Same problem with a newly created Ubuntu 20.04 VM (Windows 11 Pro 22H2 Hyper-V), on the linux-generic kernel. The same 8 logs per 10 seconds as seen on jammy, for the scripts not found in /usr/libexec/hyperkvpd.

murph@Ubuntu-Virtual-Machine:~$ uname -a
Linux Ubuntu-Virtual-Machine 5.4.0-146-generic #163-Ubuntu SMP Fri Mar 17 18:26:02 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
murph@Ubuntu-Virtual-Machine:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal
murph@Ubuntu-Virtual-Machine:~$

tags: added: focal
Revision history for this message
Murph (murph9000) wrote :

So, I've tried to dig into this a little to figure out why there's a mismatch between hv_kvp_daemon and the installed location of the hv_* scripts, and it looks like it must be somewhere on the packaging side of things rather than the build side. In both of the following Makefiles, the scripts which were throwing errors (hv_get_dhcp_info and hv_get_dns_info) look like they should be correctly installed in /usr/libexec/hypervkvpd. The daemons are still getting put into /usr/sbin by the Makefile, and they might be better living under libexec as well, since they don't really belong in PATH; but it's the location of the scripts which is a noisy bug that breaks some Hyper-V functionality.

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/tree/tools/hv/Makefile
https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-azure/+git/jammy/tree/tools/hv/Makefile

Something in the packaging phase is erroneously moving those scripts from /usr/libexec/hyperkvpd to /usr/sbin, but I'm not entirely sure just what is doing that, only that it doesn't seem to be on the build side.

Revision history for this message
Murph (murph9000) wrote (last edit ):

With some more digging, I believe I've found the root of the bug.

https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/tree/debian/rules.d/3-binary-indep.mk

(see also 2-binary-arch.mk for the hv_* daemon binaries, and how they get into their installed location)

That makefile is installing the scripts into $(cloudsbin), which looks like it defaults to /usr/sbin, contrary to the path embedded in the hv_kvp_daemon binary and the corresponding Makefile over in jammy/tools/hv.

I think hv_* should live in /usr/libexec/hypervkvpd, and only lsvmbus should live in /usr/sbin (as it has some diagnostic utility). I think rules.d/* needs to changed to eliminate the path mismatch involved in this bug. If consensus says these things should live in /usr/sbin, then I'm ok with that too (and the path embedded in the hv_kvp_daemon binary needs to be changed).

Murph (murph9000)
Changed in linux-azure (Ubuntu Bionic):
status: Invalid → Confirmed
Changed in linux-azure (Ubuntu Cosmic):
status: Invalid → Confirmed
Murph (murph9000)
summary: - [Hyper-V] KVP daemon fails to start
+ [Hyper-V] KVP daemon fails to run scripts
Revision history for this message
Murph (murph9000) wrote :

I've just checked the kinetic source, and the same path mismatch seems to exist between debian/rules.d/3-binary-indep.mk and tools/hv/*

tags: added: kinetic
Revision history for this message
Murph (murph9000) wrote :

Checked lunar source, looks like it has the same problem.

tags: added: lunar
Revision history for this message
Murph (murph9000) wrote :

I've done a little more digging. The use of /usr/libexec/hypervkvpd for the child scripts comes from upstream.

https://github.com/torvalds/linux/blob/master/tools/hv/Makefile
https://github.com/torvalds/linux/blob/master/tools/hv/hv_kvp_daemon.c

If the governing principle is to adhere to the default Linux & Debian paths (where reasonable), then the child scripts (hv_*.sh) should live there in Ubuntu. I.e. this is a packaging error that should be fixed in Ubuntu's debian/rules.d/3-binary-indep.mk.

The default install location (in torvalds/linux) for the 3 hv_*_daemon binaries is /usr/sbin, but the hv_*.sh scripts install into /usr/libexec/hyperkvpd. For Debian, the hv_*_daemon binaries are in their hyperv-daemons package and install in /usr/sbin, but it seems that they have a long standing bug (#919350) where they don't provide any of the /usr/libexec/hyperkvpd scripts.

tags: added: packaging
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.