DHCPv6 does not work in Hyper-V

[John Davis]

When using Ubuntu (in this case the live 16.04.1 LTS ISO) as a Generation 2 (Gen1 untested) Hyper-V VM, an IPv6 address can't be gotten through DHCPv6. dhclient continues to send out solicitations but they never result in a response. I've looked on the router end (running AdvancedTomato and dnsmasq) and the router is getting the requests and is sending out a advertisement, but the VM never gets it. Using the exact same live ISO works in VirtualBox so I've convinced it is an issue with the Hyper-V network driver or something along those lines.

The only IPv6 address that is configured automatically is the fe80 link-local address.

Windows VMs and older Linux VMs (Debian jessie with 3.16 kernel) work just fine and receive an IPv6 via DHCPv6.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.4.0-31-generic 4.4.0-31.50
ProcVersionSignature: Ubuntu 4.4.0-31.50-generic 4.4.13
Uname: Linux 4.4.0-31-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperVersion: 1.376
CurrentDesktop: Unity
Date: Sat Sep 17 19:56:42 2016
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
LiveMediaBuild: Ubuntu 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Microsoft Corporation Virtual Machine
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/casper/vmlinuz.efi file=/cdrom/preseed/username.seed boot=casper quiet splash ---
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-31-generic N/A
 linux-backports-modules-4.4.0-31-generic N/A
 linux-firmware 1.157.2
RfKill:

SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 11/26/2012
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v1.0
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v1.0
dmi.chassis.asset.tag: 5808-9631-9266-5991-2472-7695-55
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v1.0
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev1.0:bd11/26/2012:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev1.0:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev1.0:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev1.0:
dmi.product.name: Virtual Machine
dmi.product.version: Hyper-V UEFI Release v1.0
dmi.sys.vendor: Microsoft Corporation
---
ApportVersion: 2.20.1-0ubuntu2.1
Architecture: amd64
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperVersion: 1.376
CurrentDesktop: Unity
DistroRelease: Ubuntu 16.04
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
LiveMediaBuild: Ubuntu 16.04.1 LTS "Xenial Xerus" - Release amd64 (20160719)
Lspci:

Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Microsoft Corporation Virtual Machine
Package: linux (not installed)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 hyperv_fb
ProcKernelCmdLine: BOOT_IMAGE=/casper/vmlinuz.efi file=/cdrom/preseed/username.seed boot=casper quiet splash ---
ProcVersionSignature: Ubuntu 4.4.0-31.50-generic 4.4.13
RelatedPackageVersions:
 linux-restricted-modules-4.4.0-31-generic N/A
 linux-backports-modules-4.4.0-31-generic N/A
 linux-firmware 1.157.2
RfKill:

Tags: xenial
Uname: Linux 4.4.0-31-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm cdrom dip lpadmin plugdev sambashare sudo
_MarkForUpload: True
dmi.bios.date: 11/26/2012
dmi.bios.vendor: Microsoft Corporation
dmi.bios.version: Hyper-V UEFI Release v1.0
dmi.board.asset.tag: None
dmi.board.name: Virtual Machine
dmi.board.vendor: Microsoft Corporation
dmi.board.version: Hyper-V UEFI Release v1.0
dmi.chassis.asset.tag: 5808-9631-9266-5991-2472-7695-55
dmi.chassis.type: 3
dmi.chassis.vendor: Microsoft Corporation
dmi.chassis.version: Hyper-V UEFI Release v1.0
dmi.modalias: dmi:bvnMicrosoftCorporation:bvrHyper-VUEFIReleasev1.0:bd11/26/2012:svnMicrosoftCorporation:pnVirtualMachine:pvrHyper-VUEFIReleasev1.0:rvnMicrosoftCorporation:rnVirtualMachine:rvrHyper-VUEFIReleasev1.0:cvnMicrosoftCorporation:ct3:cvrHyper-VUEFIReleasev1.0:
dmi.product.name: Virtual Machine
dmi.product.version: Hyper-V UEFI Release v1.0
dmi.sys.vendor: Microsoft Corporation

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1624722

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[John Davis (goldfish64)]

apport information

[John Davis (goldfish64)]

apport information

[John Davis (goldfish64)]

apport information

[John Davis (goldfish64)]

apport information

[John Davis (goldfish64)]

apport information

[John Davis (goldfish64)]

apport information

[John Davis (goldfish64)]

apport information

[John Davis (goldfish64)]

apport information

[John Davis (goldfish64)]

apport information

[John Davis (goldfish64)]

apport information

[VF (fviktor)]

Same happens in Ubuntu 14.04 with kernel 4.4.0-36-generic. Also reproduced on the above 16.04 distribution.

DHCPv6 server: isc-dhcp-server in IPv6 mode (isc-dhcp-server6)

Wireshark captures indicate that the solicit is sent from the client, arrives to the server's network interface, then lost in the kernel before arriving to dhcpd.

Cleared ip6tables and tc completely, so no firewall nor QoS could block/redirect the packet.

Running dhcpd under strace in the foreground confirms that no broadcast packet is received in user space.

Traced kernel function calls:

#!/bin/bash
mkdir -p /debug
mount -t debugfs nodev /debug
mount -t debugfs nodev /sys/kernel/debug
echo '*' >/debug/tracing/set_ftrace_filter
echo function_graph >/debug/tracing/current_tracer
echo 1 >/debug/tracing/tracing_on
sleep 10
echo 0 >/debug/tracing/tracing_on
cat /debug/tracing/trace > tracing.$$.out

Found the following fragment, where the packet is dropped:

ipv6_rcv() {
  ...
  ip6_rcv_finish() {
    ...
    ip6_mc_input() {
      ipv6_chk_mcast_addr() {
        _raw_read_lock_bh();
        _raw_read_unlock_bh() {
          __local_bh_enable_ip();
        }
      }
      kfree_skb() {
        ...
      }
    }
  }
}

(Timing information removed and irrelevant fragments replaced by ellipsis.)

Relevant kernel source code:

Function discards the packet: ip6_mc_input
http://lxr.free-electrons.com/source/net/ipv6/ip6_input.c?v=4.4#L285

Decision is based on false value returned here:
deliver = ipv6_chk_mcast_addr(skb->dev, &hdr->daddr, NULL);

I verified that deliver is false by instrumenting the kernel with printk here.

(The long conditional block after the above code line does nothing, since it is not an MLD packet.)

Packet is discarded at the end of function by calling the kfree_skb function.

So the ultimate decision is made by this function: ipv6_chk_mcast_addr
http://lxr.free-electrons.com/source/net/ipv6/mcast.c?v=4.4#L955

Please note that src_addr = NULL. (See the function call above.)

I verified by kernel logging that idev is not NULL, so the device is found here.

Since src_addr is NULL rv can only be true if the for loop finds the destination address of the packet (variable: group) in the list of registered broadcast addresses.

Listing registered broadcast groups: netstat -g

I verified that dhcpd is indeed registered to ff02::1:2

Registration is also visible in the strace output.

I can get an address with the server (from itself) by adding a new virtual network adapter and connecting it to the same network. So it works if the packet remains inside the VM and does not go through the virtual switch.

I have spent days on this problem so far. Any ideas?

[VF (fviktor)]

Additions to my previous comment:
- I have Ubuntu 14.04 LTS (and 16.04 LTS) 64 bit version set up as a router with DHCPv6 support.
- Client is vanilla Ubuntu 14.04 64 bit desktop.

The problems I described above apply to the server VM, where the solicit packet appears to be eaten by the kernel.

[VF (fviktor)]

In my case this is not caused by Hyper-V's DhcpGuard feature either, because that's turned off for both VMs involved. (Same for RouterGuard.) See: https://social.technet.microsoft.com/Forums/en-US/f03c0c8e-8924-4071-b86e-18fb141e4710/how-to-configure-dhcp-guard-in-hyperv-2012?forum=winserverhyperv

[John Davis (goldfish64)]

I did a quick capture with Wireshark (this time using Hyper-V on Windows 10) and indeed the VM sends out a solicit, but I don't see any evidence of it reaching the router (nothing in the logs). DHCP Guard is switched off but I think that would affect a DHCP server, not a client. Plain SLAAC with my custom DNS server set appears to work correctly, so RA's are getting to the VM just fine.

I also did a capture with the same ISO in VirtualBox, and it exchanges DHCPv6 messages with the router right away.

[John Davis (goldfish64)]

Adding on to my previous comment, I had tried a fresh Debian jessie install with v3.16 of the Linux kernel. DHCPv6 works just fine until I switch to the one in the jessie-backports repo, which currently is 4.6. So somewhere between 3.16 and one of the 4.x kernels a regression was introduced.

[VF (fviktor)]

I've done the following test today on two physical machines, so no virtualization is involved.

How to reproduce:

What I have tried today: Installed Ubuntu 14.04 64bit (kernel 4.4.0.-38-generic) on two physical machines (no virtualization involved). Upgraded them and installed radvd and isc-dhcp-server on one of them (server). Then connected the two machines directly with a network cable, no other infrastructure involved.

Server:
- Install Ubuntu 14.04 64bit (kernel 4.4.0.-38-generic)
- Apply all upgrades
- Disconnect the machine from the network
- Enable IPv6 forwarding in /etc/sysctl.conf, this is required by radvd
- Reboot
- Add IPv6 address fc00:0:0:1::1/64 (gateway) to eth0
- Configure radvd (/etc/radvd.conf):

interface eth0
{
  AdvSendAdvert on;
  AdvManagedFlag on;
  prefix fec0:0:0:1::/64
  {
    AdvOnLink on;
    AdvRouterAddr on;
    AdvAutonomous off;
    AdvValidLifetime 7200;
    AdvPreferredLifetime 3600;
  };
};

- Configure isc-dhcp-server6:
- In /etc/isc-dhcp-server6 set the INTERFACE to eth0
- Contents of /etc/dhcp/dhcpd6.conf:

authoritative;
ddns-update-style none;
option domain-name "domain";
option dhcp6.name-servers fec0:0:0:0::1;
option dhcp6.domain-search "domain.com";
default-lease-time 3600;
max-lease-time 3600;
shared-network lan {
  subnet6 fec0:0:0:1::/64 {
    range6 fec0:0:0:1::2 fec0:0:0:1::2;
  }
}

- Start both radvd and isc-dhcp-server6 services, there should be no errors in syslog.

Client:

- Install Ubuntu 14.04 64bit (kernel 4.4.0.-38-generic)
- Apply all upgrades
- Disconnect the machine from the network
- Connect the machine with a direct network cable (or through a desk switch) to the server machine.

Expected behavior: IPv6 address fc00:0:0:1::2 is assigned to the client

Actual behavior: dhclient is sending solicit requests to the server, it arrives to the network interface, then eaten by the kernel.

Additional information:

DHCPv6 server does not log anything and does not seem to receive the packet in user space at all. According to netstat -g the dhcpd process is correctly registered to broadcast group ff02::1:2 and according to netstat -nlp it is listening on UDP6 port 547. Running dhcpd in the foreground with strace confirms that the packet does not arrive to the process. Same behavior was observed with latest Ubuntu 16.04 64bit.

[VF (fviktor)]

Since I cannot edit my above comment I post an update:

After writing the above post I checked the client again, just to be sure. To my greatest surprise I found that the fec0::0:0:1::2 address was correctly assigned to it!

Checked syslog on the server and found that the address was assigned 3 minutes a 43 seconds after the dhcp server was started and about 3 minutes after my manual attempts on the client to get an address.

Looks like we have a heisenbug here. Its probability depends on the hardware/virtualization environment used.

I also verified (using dumpcap -xx and Wireshark) that Hyper-V does not change the contents of the solicit packet.

[VF (fviktor)]

> I also verified (using dumpcap -xx and Wireshark) that Hyper-V does not change the contents of the solicit packet.

It was tcpdump -xx and then dumpcap + Wireshark, certainly.

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.8 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8

[VF (fviktor)]

It was a problem with that Hyper-V version, some DHCPv6 related packet was dropped somehow. Microsoft fixed it in a later version and it started working. You can close this bug.