[Hyper-V] KVP daemon fails to start

Bug #1766857 reported by Ionut Lenghel on 2018-04-25
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
High
Joseph Salisbury
Bionic
High
Joseph Salisbury
Cosmic
High
Joseph Salisbury
linux-azure (Ubuntu)
High
Unassigned
Bionic
High
Unassigned
Cosmic
High
Unassigned

Bug Description

While testing Bionic daily build with kernel 4.15.0-20-generic we saw that there are 2 issues with the KVP daemon:

1. KVP daemon crash after boot. Opened a different bug for this: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1779640

2. After the KVP daemon is being started the following messages appear:

Apr 25 04:45:25 bionicDaily hv_kvp_daemon[1895]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Apr 25 04:45:25 bionicDaily hv_kvp_daemon[1895]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found

The above binaries are present on the system, but the their actual path is /usr/sbin/hv_get_dns_info and /usr/sbin/hv_get_dhcp_info. Either the hv_get_dhcp_info and hv_get_dns_info binaries should be placed in the location where the daemon is looking for (/usr/libexec/hypervkvpd/), or the daemon should be set to search for the binaries in the /usr/sbin directory.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1766857

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: bionic
Chris Valean (cvalean) on 2018-04-25
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Changed in linux (Ubuntu):
importance: Undecided → High
Changed in linux-azure (Ubuntu Bionic):
importance: Undecided → High
tags: added: kernel-da-key kernel-hyper-v
Joseph Salisbury (jsalisbury) wrote :

Did this bug just start happening with the 4.15.0-20 kernel? Were there prior 18.04 kernels that did not exhibit this bug?

Ionut Lenghel (ilenghel) wrote :

I tested back to the daily build from 2nd of March and I can confirm the second issue (regarding the binaries path), but not the first one (regarding the daemon crash). On the build from the 2nd of March the linux-tools and linux-cloud-tools version is 4.15.0-10.11

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-azure (Ubuntu Bionic):
status: New → Confirmed
Changed in linux-azure (Ubuntu):
status: New → Confirmed
giacomo (chiaravalli) wrote :

I confirm the second, the daemon hv_kvp_daemon not find the files:
/usr/libexec/hypervkvpd/hv_get_dns_info
/usr/libexec/hypervkvpd/hv_get_dhcp_info
With linux-cloud-tools version 4.15.0-22.24

Joshua R. Poulson (jrp) wrote :

In my own VMs, this does not affect linux-azure, it only affects the generic 4.15 kernel.

Joseph Salisbury (jsalisbury) wrote :

Commit 7fa32e5ec28b was mentioned in the descriptions. However, that commit has been in mainlien since 4.15-rc3, so it's always been in 18.04.

We may want to split the two issues in to two separate bugs, so they each get addressed.

Do both issues still happen with the latest Bionic kernel? Per comment #6, it sounds liek the second issues still happens. How about the first issue, where the KVP daemon crashes after approximatively 2 minutes of uptime?

Joseph Salisbury (jsalisbury) wrote :

Commit 7fa32e5ec28b was actually mentioned in bug 1664663, an I don't think it factors into this bug.

Changed in linux (Ubuntu):
status: Confirmed → In Progress
Changed in linux (Ubuntu Bionic):
status: Confirmed → In Progress
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux-azure (Ubuntu):
status: Confirmed → Invalid
Changed in linux-azure (Ubuntu Bionic):
status: Confirmed → Invalid
Guido (mcnomad81) on 2018-06-07
information type: Public → Public Security
information type: Public Security → Public
Ionut Lenghel (ilenghel) wrote :

I have tested Bionic with kernel 4.15.0-22-generic, linux-cloud-tools-4.15.0-22.24 and linux-tools-4.15.0-22.24 and both issues still occur.

While testing the above mentioned kernel, I saw the following behavior:

1. Right after booting the system the KVP daemon reports it is active (running). The command "systemctl status hv-kvp-daemon" returns the following output:

hv-kvp-daemon.service - Hyper-V KVP Protocol Daemon
   Loaded: loaded (/lib/systemd/system/hv-kvp-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-06-07 11:43:22 UTC; 55s ago
 Main PID: 1363 (hv_kvp_daemon)
    Tasks: 1 (limit: 4496)
   CGroup: /system.slice/hv-kvp-daemon.service
           └─1363 /usr/lib/linux-tools/4.15.0-22-generic/hv_kvp_daemon -n

Jun 07 11:43:22 bionic systemd[1]: Started Hyper-V KVP Protocol Daemon.
Jun 07 11:43:22 bionic KVP[1363]: KVP starting; pid is:1363
Jun 07 11:43:22 bionic KVP[1363]: KVP LIC Version: 3.1

2. After approximately 2 minutes the KVP daemon enters in the failed state (issue 1).
3. Manually starting the daemon with the command "systemctl start hv-kvp-daemon" works perfectly fine.
4. "systemctl status hv-kvp-daemon" will now show the second issue:

hv-kvp-daemon.service - Hyper-V KVP Protocol Daemon
   Loaded: loaded (/lib/systemd/system/hv-kvp-daemon.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-06-07 11:47:29 UTC; 8s ago
 Main PID: 1995 (hv_kvp_daemon)
    Tasks: 1 (limit: 4496)
   CGroup: /system.slice/hv-kvp-daemon.service
           └─1995 /usr/lib/linux-tools/4.15.0-22-generic/hv_kvp_daemon -n

Jun 07 11:47:29 bionic systemd[1]: Started Hyper-V KVP Protocol Daemon.
Jun 07 11:47:29 bionic KVP[1995]: KVP starting; pid is:1995
Jun 07 11:47:29 bionic KVP[1995]: KVP LIC Version: 3.1
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dns_info: not found
Jun 07 11:47:36 bionic hv_kvp_daemon[1995]: sh: 1: /usr/libexec/hypervkvpd/hv_get_dhcp_info: not found

5. Rebooting the system after this point will no longer trigger the first issue, only the second one. Even stopping the VM and turning it back ON does NOT trigger the first issue again.

Could it be that while the hv-kvp-daemon service shows the two files (hv_get_dns_info and hv_get_dhcp_info) are not found, it will not enter in the failed state? Meaning that the two apparently separate issues, are actually related. I am not 100% sure about this.

In order to trigger the first issue again, the Integration Service corresponding to the KVP daemon has to be disabled and re-enabled, and the VM has to be rebooted. This is just the way I managed to reproduce the first issue again, I am not sure if there are other ways to trigger it again.

Joseph Salisbury (jsalisbury) wrote :

Since they might be related, lets focus on the second issue first, which is cause by the missing files.

I'll set up a test system to try and reproduce.

It seems like this is a regression, but it would be good to confirm. Can you test these two kernels?

Last 4.14 Bionic kernel:
https://launchpad.net/~canonical-kernel-security-team/+archive/ubuntu/ppa2/+build/14223014

First 4.15 Bionic kernel:
https://launchpad.net/ubuntu/+source/linux/4.15.0-9.10/+build/14326410

Joseph Salisbury (jsalisbury) wrote :

I built a test kernel which can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1766857/

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

Ionut Lenghel (ilenghel) wrote :

Regarding comment #11:
I have tested both kernels provided with the following results:

- 4.14 Bionic kernel (4.14.0-16-generic) does NOT exhibit issue #2 (missing files), but it does exhibit issue #1 (KVP daemon failure after boot). Linux-cloud-tools version installed is 4.14.0-16.19.

- 4.15 Bionic kernel (4.15.0-9-generic) behaves the same as 4.14, meaning that issue #2 (missing files) is NOT present, but issue #1 (KVP daemon failure after boot) shows up. Linux-cloud-tools version is 4.15.0-9.10.

Regarding comment #12:
I have tested the kernel from the link and it seems that this kernel does NOT exhibit issue #2 (but it does exhibit issue #1).

Joseph Salisbury (jsalisbury) wrote :

At this point, we should probably have two separate bugs to address each problem. Can we keep this bug for the second issue and create a new bug for the first issue?

Ionut Lenghel (ilenghel) wrote :

I have opened a different bug for the first issue: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1779640

Regarding the second issue, as mentioned in comment #13, the tested kernels did not exhibit issue #2.

description: updated
Joseph Salisbury (jsalisbury) wrote :

Did you notice any other issues that occur when testing the test kernel posted in comment #12. The second issue posted in the bug is resolved by changing the definition of KVP_CONFIG_LOC in hv_kvp_daemon.c:

---
 tools/hv/hv_kvp_daemon.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
index 4c99c57..e969057 100644
--- a/tools/hv/hv_kvp_daemon.c
+++ b/tools/hv/hv_kvp_daemon.c
@@ -95,7 +95,7 @@ static struct utsname uts_buf;
 #define KVP_CONFIG_LOC "/var/lib/hyperv"

 #ifndef KVP_SCRIPTS_PATH
-#define KVP_SCRIPTS_PATH "/usr/libexec/hypervkvpd/"
+#define KVP_SCRIPTS_PATH "/usr/sbin/"
 #endif

 #define KVP_NET_DIR "/sys/class/net/"
--
2.7.4

Chris Valean (cvalean) wrote :

Joe, for the 2nd issue initially reported here, about "After the KVP daemon is being started the following messages appear".
We've noticed a regression for this in 18.10 - daily's and even GA.

Can you please add Cosmic 18.10 to the bug here and re-open or should I send a new bug report for it?
Thank you!

Joseph Salisbury (jsalisbury) wrote :

Hi Chris, I added a Cosmic 18.10 bug task. Does this bug only happen in Cosmic, or is it still happening in Bionic?

Chris Valean (cvalean) wrote :

Issue with 'Invalid kvp dependency scripts path /usr/libexec/hypervkvpd' remains in latest kernel-image bionic 4.15.0-38.41.

Verified azure-kernel 4.15.0-1030.31 for bionic, same VM and environment, and the issue does *not* appear.

Joseph Salisbury (jsalisbury) wrote :

Hmm, I wonder if the linux-azure kernel has and extra patch(s) that the generic kernel does not. I'll do some comparisons.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers