virtual machine can not get DHCP lease due packet has no checksum

[Jiajun Liu]

if virtual machie are using virtio driver and swith vhost_net on, then virutal machine can not get DHCP lease bacause the DHCP packet has no checksum and the kernel of virtual machine will drop those packet. So we should fill checksum before we pass the DHCP packet to virtual machine.

[Jiajun Liu (ljjjustin)]

we can add the following iptable rules to fix this problem.

# iptables -A POSTROUTING -t mangle -p udp --dport 68 -j CHECKSUM --checksum-fill

[Salvatore Orlando (salvatore-orlando)]

Which os are you running in the guest VM?
This is perhaps a bug in the dhcp software on the guest, I recall a similar bug in fedora dhcp client from about 2 years ago.

I agree the workaround works, but if possible I would avoid having iptables fiddling with DHCP traffic for performance reasons.

[Jiajun Liu (ljjjustin)]

Hi Salvatore,

Thanks for your reply. My guest VM is cirros-0.3.1. I noticed that nova-network will add this iptable rule by default.

[Peter Feiner (pete5)]

Please note that this was fixed in nova-network last year by https://review.openstack.org/#/c/18336/4. See https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1029430 for an explanation of why this iptables mangling is necessary. The TL;DR is that some older, albeit widely used, DHCP clients are fickle about packet checksums. For example, this affects the DHCP clients in Ubuntu 12.04 and CirrOS.

I suggest making this a bug of high importance and further suggest that the fix is backported. Without the checksums filled in, VMs with affected DHCP clients are basically stillborn.

[Ian Wells (ijw-ubuntu)]

I've just encountered this too (Ubuntu host kernel). This fixed it for me:

$ sudo ip netns exec qdhcp-90bcac39-7d49-4bcb-a212-865c9c55bce6 ethtool --offload ns-2306c732-8c tx off rx off

(substitute your DHCP namespace accordingly)

Note that offloading checksums on the (virtual) interface in the namespace is clearly not going to work, and should be a no-op in the kernel. Indications I'm getting is that in a 3.2.0 kernel the packet gets checksummed in the host kernel correctly, and in 3.8.0 it doesn't (both stock kernels for Ubuntu 12.04). In my case it seems that Linux guests are fine with the dodgy packets but other more sensitive OSes drop them.

I've done basic due diligence here but it could do with a bit more investigation to confirm.

I'm seeing this on Grizzly devstack with Neutron using linuxbridge, but I don't think that the bug itself is an Openstack one and the version of Openstack concerned won't matter if that's the case.

[Ian Wells (ijw-ubuntu)]

Reading Peter's comment above, I wonder at why what I've done works - I guess dnsmasq is injecting IP, not UDP, packets...

[Phil Hopkins (phil-hopkins-a)]

Is anyone actively working on this bug? It needs some priority.. It is a show stopper when using Linux bridge in some cases. It prevents Mulitcast VXLAN on Linux bridge from working.

[Darragh O'Reilly (darragh-oreilly)]

I'll have a look at it - I saw it in the lab a while ago with linuxbridge and vlan and meant to look into it.

[Ian Wells (ijw-ubuntu)]

Checksums are generally *not* generated for virtual networking. QEMU participates in this - so:

- when a VM has hardware checksum offload turned on, the 'hardware' in QEMU will neither create nor check checksums. This works fine - checksums are wrong as they go over the brdge, but QEMU doesn't bother to check them at all and it's not a problem. You'll see a problem with this if you have software in the VM that sets offload *and* checks the checksum - the hardware validates the packet but the checksum is wrong when the software double-checks.

- when a VM disables hardware checksum offload then QEMU corrects the checksum on receipt

The (host) Linux kernel will fix checksums on packets that leave the machine.

This typically means that if your DHCP server is running on VLAN networking from a control or network node independent of your VMs it will have fixed checksums - and if it's running locally, the checksums will be wrong, but QEMU should either fix or ignore them and the DHCP agent software shouldn't care to look.

I think the problem is that your DHCP server or agent is taking it upon itself to check the checksum (note: nothing to do with the checksum being wrong, more to do with the checksum not being *ignored*), and you might want to find out what precisely is happening before you go further. (And, for what it's worth, there's no reason I know of why this behaviour would differ between LB and OVS.) That said, the targetted netns fix might well be the one we want - it's quite specific if it can truly be said to only be a DHCP issue.

[Ian Wells (ijw-ubuntu)]

(Which means apparently I've learned a bit since my last comment on here. Also, Peter's comment is relevant to what I'm saying, where he talks about 'fickle' clients.)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/148718

[Darragh O'Reilly (darragh-oreilly)]

I did some testing that confirms what Ian is saying.
If the dhcp agent is running on the compute-node, then the dhcp tap and vm tap will be in the same linux bridge and there is a bad checksum:
    172.16.242.5.67 > 172.16.242.10.68: [bad udp cksum 0x3d93 -> 0xd1a6!] BOOTP/DHCP, Reply, length 328, xid 0x8ff2ab64, Flags [none] (0x0000)

Some dhcp clients don't care e.g. the one in trusty-server-cloudimg-amd64-disk1.img, but some do e.g. cirros-0.3.2-x86_64-uec.

If the dhcp agent is on a different machine than the compute node and the network type is vlan, then the bad udp checksum gets fixed by something (probably the physical nic on the sending side).

dhcp-node# tcpdump -vvv -enli em3 port 68
...
4:18:22.658000 fa:16:3e:20:4f:82 > fa:16:3e:4d:0b:89, ethertype 802.1Q (0x8100), length 374: vlan 242, p 0, ethertype IPv4, (tos 0xc0, ttl 64, id 13739, offset 0, flags [none], proto UDP (17), length 356)
    172.16.242.5.67 > 172.16.242.11.68: [bad udp cksum 0x3d94 -> 0x6780!] BOOTP/DHCP, Reply, length 328, xid 0x1b08771d, Flags [none] (0x0000)

compute-node# tcpdump -vvv -enli em3 port 68
...
14:17:54.834967 fa:16:3e:20:4f:82 > fa:16:3e:4d:0b:89, ethertype 802.1Q (0x8100), length 374: vlan 242, p 0, ethertype IPv4, (tos 0xc0, ttl 64, id 13739, offset 0, flags [none], proto UDP (17), length 356)
    172.16.242.5.67 > 172.16.242.11.68: [udp sum ok] BOOTP/DHCP, Reply, length 328, xid 0x1b08771d, Flags [none] (0x0000)

I don't know if a vxlan tunnel would do this fixing - doubt it.

[Phil Hopkins (phil-hopkins-a)]

As a side note this problem will also occur when running OpenStack on an "all-in-one" implementation. Since the DHCP packets never go through a NIC card the checksum never gets filled.

[Darragh O'Reilly (darragh-oreilly)]

one guest OS that this happens on is Debian 7 (wheezy)
root@debian:/home/debian# dpkg -l \*dhcp-client\*

ii isc-dhcp-client 4.2.2.dfsg.1-5+ amd64 ISC DHCP client

[Darragh O'Reilly (darragh-oreilly)]

Did some testing to see if there is a problem when using OVS instead of linuxbridge. Again with DHCP agent and nova-compute on the same node - KVM on baremetal. Now the cirros and debian7 images get their IP on boot, so there does not seem to be a problem. But in VNC of cirros instance, if you do ifdown eth0; ifup eth0, the first DHCP reply has a bad UDP sum, and the client waits a minute and retries, and for some reason the second reply has a good sum and the client gets its address. So there is some problem, but not fatal like linuxbridge. Attached tcpdump taken on linuxbridge between VM and br-int.

[Darragh O'Reilly (darragh-oreilly)]

Tested with ipv6 to see if there is a similar problem. Used the same mix that fail on ipv4 - debian7 image, linuxbridge-agent, kvm(vhost on), dhcp-agent and kvm on same host. Dhcp v6 stateful worked without any iptables rule. Attached log - all udp checksums are good.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/148718
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f5419d90791f0e258801b453c239c187302a3554
Submitter: Jenkins
Branch: master

commit f5419d90791f0e258801b453c239c187302a3554
Author: Darragh O'Reilly <email address hidden>
Date: Tue Feb 3 17:03:23 2015 +0000

    Always fill UDP checksums in DHCP replies

    In some cases the UDP checksums in packets from DHCP servers are
    incorrect. This is a problem for some DHCP clients that ignore
    packets with bad checksums. This patch inserts an iptables rule
    to ensure DHCP servers always send packets with correct checksums.

    Change-Id: I130fe0f2389bdc42eb8c858ea35dd840abecc2e7
    Closes-Bug: 1244589

[Thiago Martins (martinx)]

And what if you entirely disable iptables/ipset out from your OpenStack "all-in-one" environment?

That rule will not appear anywhere...

Don't we have a better solution for this? I mean, why the bad checksum occur in first place?

My environment looks like this:

---
ml2_conf.ini:

[securitygroup]
enable_security_group = False
enable_ipset = False
firewall_driver = neutron.agent.firewall.NoopFirewallDriver
---

---
No "security_group_api" at nova.conf.
---

There are almost no iptables rules in my environment... No Security Groups as well...

So, I am unable to use LinuxBridge in a "all-in-one" setup. :-(

I'm glad that only cirros seems to be affected by this problem, but, this is still a problem when we don't use iptables.

Best!
Thiago

[Thiago Martins (martinx)]

BTW, I'm using Juno with Trusty...

[Darragh O'Reilly (darragh-oreilly)]

Thiago, have you tried? The patch should work even if security groups are disabled. The iptables rule is in the dhcp namespace.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/630297

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/630297
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=26eb2509fea632e67ffabcc15195cc13ee02bf68
Submitter: Zuul
Branch: master

commit 26eb2509fea632e67ffabcc15195cc13ee02bf68
Author: Bence Romsics <email address hidden>
Date: Fri Jan 11 16:08:53 2019 +0100

    Always fill UDP checksums in DHCPv6 replies

    Bug #1244589 re-appeared for IPv6.

    This change adds an ip6tables rule to fix the checksum of DHCPv6
    response packets. Those checksums were left unfilled by virtio (as a
    hypervisor internal optimization), but some picky dhcp clients (AFAIU
    particularly ISC dhclient) try verifying the checksums, so they fail
    to acquire an address if the checksums are left incorrect.

    Change-Id: I4a045e0dcfcbd3c7959a78f1460d5bf7da0252ff
    Closes-Bug: #1811639
    Related-Bug: #1244589

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/634512

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/634514

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/634515

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634512
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=50a7a74e97abe92430a835f7e440c50989c4e4df
Submitter: Zuul
Branch: stable/rocky

commit 50a7a74e97abe92430a835f7e440c50989c4e4df
Author: Bence Romsics <email address hidden>
Date: Fri Jan 11 16:08:53 2019 +0100

    Always fill UDP checksums in DHCPv6 replies

    Bug #1244589 re-appeared for IPv6.

    This change adds an ip6tables rule to fix the checksum of DHCPv6
    response packets. Those checksums were left unfilled by virtio (as a
    hypervisor internal optimization), but some picky dhcp clients (AFAIU
    particularly ISC dhclient) try verifying the checksums, so they fail
    to acquire an address if the checksums are left incorrect.

    Change-Id: I4a045e0dcfcbd3c7959a78f1460d5bf7da0252ff
    Closes-Bug: #1811639
    Related-Bug: #1244589
    (cherry picked from commit 26eb2509fea632e67ffabcc15195cc13ee02bf68)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634514
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f920dfea8c3ab2dba3b455056336cd95bb3336c1
Submitter: Zuul
Branch: stable/queens

commit f920dfea8c3ab2dba3b455056336cd95bb3336c1
Author: Bence Romsics <email address hidden>
Date: Fri Jan 11 16:08:53 2019 +0100

    Always fill UDP checksums in DHCPv6 replies

    Bug #1244589 re-appeared for IPv6.

    This change adds an ip6tables rule to fix the checksum of DHCPv6
    response packets. Those checksums were left unfilled by virtio (as a
    hypervisor internal optimization), but some picky dhcp clients (AFAIU
    particularly ISC dhclient) try verifying the checksums, so they fail
    to acquire an address if the checksums are left incorrect.

    Change-Id: I4a045e0dcfcbd3c7959a78f1460d5bf7da0252ff
    Closes-Bug: #1811639
    Related-Bug: #1244589
    (cherry picked from commit 26eb2509fea632e67ffabcc15195cc13ee02bf68)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/634515
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=399f1c1b65b5aad5a810d9986da35d07216fa707
Submitter: Zuul
Branch: stable/pike

commit 399f1c1b65b5aad5a810d9986da35d07216fa707
Author: Bence Romsics <email address hidden>
Date: Fri Jan 11 16:08:53 2019 +0100

    Always fill UDP checksums in DHCPv6 replies

    Bug #1244589 re-appeared for IPv6.

    This change adds an ip6tables rule to fix the checksum of DHCPv6
    response packets. Those checksums were left unfilled by virtio (as a
    hypervisor internal optimization), but some picky dhcp clients (AFAIU
    particularly ISC dhclient) try verifying the checksums, so they fail
    to acquire an address if the checksums are left incorrect.

    Change-Id: I4a045e0dcfcbd3c7959a78f1460d5bf7da0252ff
    Closes-Bug: #1811639
    Related-Bug: #1244589
    (cherry picked from commit 26eb2509fea632e67ffabcc15195cc13ee02bf68)