virtual machine cannot get DHCP (DHCPv6) lease for IPV6 due to packet has no checksum.

[Nikhil Singh]

if virtual machine are using virtio driver and switch vhost_net on,
then virtual machine cannot get DHCPv6 lease because the DHCPv6 packet
has no checksum and the kernel of virtual machine will drop those packet.
So we should fill checksum before we pass the DHCPv6 packet to virtual machine.
Similar bug for IPV4 (https://bugs.launchpad.net/neutron/+bug/1244589)is raised and fixed as part of commit:

https://review.openstack.org/#/c/148718/8/neutron/agent/linux/dhcp.py

This functionality need to support for IPV6 as well and should cover both IPV6 stateless and state full address
management cases.

[Bence Romsics (bence-romsics)]

To my understanding this bug was first observed in a downstream modified (Pike-based) neutron. The following is my best reproduction on upstream master:

# create a network with a v6 subnet in dhcp stateless mode
openstack network create net1
openstack subnet create subnet1 --network net1 --subnet-range 10.0.4.0/24
openstack subnet create subnet1 --network net1 --ip-version 6 --ipv6-ra-mode dhcpv6-stateless --ipv6-address-mode dhcpv6-stateless --subnet-range fda6:dd28:1656::/64

I need a vm in which I could control dhclient, therefore I create an image with root password pre-set and I access it via 'libvirt console' from the compute host after openstack booted it.

openstack image create u1804 --container-format bare --disk-format qcow2 --public --file ~/u1804-with-root-password.img
openstack server create vm0 --flavor ds512M --image u1804 --nic net-id=net1 --wait

# on the compute host
source ~/src/os/openstack/devstack/openrc admin admin
sudo virsh console $( openstack server show vm0 -f value -c OS-EXT-SRV-ATTR:instance_name )
# login with pre-set root password
# trigger v6 dhclient as needed
dhclient -v -6 ens2

The client cannot acquire an address, but it seems we have multiple issues:

root@vm0:~# dhclient -v -6 ens2
Internet Systems Consortium DHCP Client 4.3.5
Copyright 2004-2016 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on Socket/ens2
Sending on Socket/ens2
PRC: Soliciting for leases (INIT).
XMT: Forming Solicit, 0 ms elapsed.
XMT: X-- IA_NA 3e:e4:10:e4
XMT: | X-- Request renew in +3600
XMT: | X-- Request rebind in +5400
XMT: Solicit on ens2, interval 1010ms.
RCV: Advertise message on ens2 from fe80::f816:3eff:fe4a:e696.
RCV: X-- Server ID: 00:01:00:01:23:cf:43:eb:fa:16:3e:4a:e6:96
PRC: Lease failed to satisfy.

The first issue is that (likely a configuration error due to my insufficient IPv6 knowledge) the dhcpv6 server thinks there are no v6 addresses available to advertise:

# captured while dhclient was running
$ sudo tcpdump -vvv -n -i $( sudo virsh dumpxml $( openstack server show vm0 -f value -c OS-EXT-SRV-ATTR:instance_name ) | egrep tap | cut -d\' -f2 )
tcpdump: listening on tapdead586f-16, link-type EN10MB (Ethernet), capture size 262144 bytes
14:02:39.194410 IP6 (flowlabel 0x449a9, hlim 1, next-header UDP (17) payload length: 64) fe80::f816:3eff:fee4:10e4.546 > ff02::1:2.547: [bad udp cksum 0x44b7 -> 0x2b6b!] dhcp6 solicit (xid=4961ec (client-ID hwaddr/time type 1 time 600786647 fa163ee410e4) (option-request DNS-server DNS-search-list Client-FQDN SNTP-servers) (elapsed-time 0) (IA_NA IAID:1055133924 T1:3600 T2:5400))
14:02:39.195553 IP6 (class 0xc0, flowlabel 0x27db9, hlim 64, next-header UDP (17) payload length: 76) fe80::f816:3eff:fe4a:e696.547 > fe80::f816:3eff:fee4:10e4.546: [udp sum ok] dhcp6 advertise (xid=4961ec (client-ID hwaddr/time type 1 time 600786647 fa163ee410e4) (server-ID hwaddr/time type 1 time 600785899 fa163e4ae696) (status-code NoAddrsAvail))
14:02:40.325718 IP6 (flowlabel 0x449a9, hlim 1, next-header UDP (17) payload length: 64) fe80::f816:3eff:fee4:10e4.546 > ff02::1:2.547: [bad udp cksum 0x44b7 -> 0x2afb!] dhcp6 solic...

[Bence Romsics (bence-romsics)]

And here's a fix started: https://review.openstack.org/630297

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/631795

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/631798

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/630297
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=26eb2509fea632e67ffabcc15195cc13ee02bf68
Submitter: Zuul
Branch: master

commit 26eb2509fea632e67ffabcc15195cc13ee02bf68
Author: Bence Romsics <email address hidden>
Date: Fri Jan 11 16:08:53 2019 +0100

    Always fill UDP checksums in DHCPv6 replies

    Bug #1244589 re-appeared for IPv6.

    This change adds an ip6tables rule to fix the checksum of DHCPv6
    response packets. Those checksums were left unfilled by virtio (as a
    hypervisor internal optimization), but some picky dhcp clients (AFAIU
    particularly ISC dhclient) try verifying the checksums, so they fail
    to acquire an address if the checksums are left incorrect.

    Change-Id: I4a045e0dcfcbd3c7959a78f1460d5bf7da0252ff
    Closes-Bug: #1811639
    Related-Bug: #1244589

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/634512

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/634514

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/634515

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/631795
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=ac1d034904935235af94564154834ded77db41d9
Submitter: Zuul
Branch: master

commit ac1d034904935235af94564154834ded77db41d9
Author: Bence Romsics <email address hidden>
Date: Fri Jan 18 14:44:00 2019 +0100

    Rehome constant: DHCPV6_CLIENT_PORT

    This constant has its natural place in neutron-lib, but it was first
    merged to neutron for a quick bugfix.

    Change-Id: I1684e04ba1b01c44d47a6cc80d9589f20c48ced4
    Depends-On: https://review.openstack.org/630297
    Related-Bug: #1811639

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634512
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=50a7a74e97abe92430a835f7e440c50989c4e4df
Submitter: Zuul
Branch: stable/rocky

commit 50a7a74e97abe92430a835f7e440c50989c4e4df
Author: Bence Romsics <email address hidden>
Date: Fri Jan 11 16:08:53 2019 +0100

    Always fill UDP checksums in DHCPv6 replies

    Bug #1244589 re-appeared for IPv6.

    This change adds an ip6tables rule to fix the checksum of DHCPv6
    response packets. Those checksums were left unfilled by virtio (as a
    hypervisor internal optimization), but some picky dhcp clients (AFAIU
    particularly ISC dhclient) try verifying the checksums, so they fail
    to acquire an address if the checksums are left incorrect.

    Change-Id: I4a045e0dcfcbd3c7959a78f1460d5bf7da0252ff
    Closes-Bug: #1811639
    Related-Bug: #1244589
    (cherry picked from commit 26eb2509fea632e67ffabcc15195cc13ee02bf68)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634514
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f920dfea8c3ab2dba3b455056336cd95bb3336c1
Submitter: Zuul
Branch: stable/queens

commit f920dfea8c3ab2dba3b455056336cd95bb3336c1
Author: Bence Romsics <email address hidden>
Date: Fri Jan 11 16:08:53 2019 +0100

    Always fill UDP checksums in DHCPv6 replies

    Bug #1244589 re-appeared for IPv6.

    This change adds an ip6tables rule to fix the checksum of DHCPv6
    response packets. Those checksums were left unfilled by virtio (as a
    hypervisor internal optimization), but some picky dhcp clients (AFAIU
    particularly ISC dhclient) try verifying the checksums, so they fail
    to acquire an address if the checksums are left incorrect.

    Change-Id: I4a045e0dcfcbd3c7959a78f1460d5bf7da0252ff
    Closes-Bug: #1811639
    Related-Bug: #1244589
    (cherry picked from commit 26eb2509fea632e67ffabcc15195cc13ee02bf68)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634515
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=399f1c1b65b5aad5a810d9986da35d07216fa707
Submitter: Zuul
Branch: stable/pike

commit 399f1c1b65b5aad5a810d9986da35d07216fa707
Author: Bence Romsics <email address hidden>
Date: Fri Jan 11 16:08:53 2019 +0100

    Always fill UDP checksums in DHCPv6 replies

    Bug #1244589 re-appeared for IPv6.

    This change adds an ip6tables rule to fix the checksum of DHCPv6
    response packets. Those checksums were left unfilled by virtio (as a
    hypervisor internal optimization), but some picky dhcp clients (AFAIU
    particularly ISC dhclient) try verifying the checksums, so they fail
    to acquire an address if the checksums are left incorrect.

    Change-Id: I4a045e0dcfcbd3c7959a78f1460d5bf7da0252ff
    Closes-Bug: #1811639
    Related-Bug: #1244589
    (cherry picked from commit 26eb2509fea632e67ffabcc15195cc13ee02bf68)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/631798
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=01073caf525bad8a8118ed43ed653baaa91837d8
Submitter: Zuul
Branch: master

commit 01073caf525bad8a8118ed43ed653baaa91837d8
Author: Bence Romsics <email address hidden>
Date: Fri Jan 18 14:57:17 2019 +0100

    Consume constant from neutron-lib: DHCPV6_CLIENT_PORT

    Consume the rehomed constant. Remove the constant from neutron.

    Change-Id: Ia5d6ec8b66344c0c0c2d1588f8c1215c6c2b1cbe
    Depends-On: https://review.openstack.org/631795
    Related-Bug: #1811639

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.6 release.