iptables checksum-fill causing kernel warning error stack

[Satish Patel]

Last 24 hours i am chasing following error in my "dmesg" logs. I am using QUEENS 17.0.7

[ 633.129619] ------------[ cut here ]------------
[ 633.129624] bond0.28: caps=(0x0000000040005028, 0x0000000000000000) len=7292 data_len=7002 gso_size=1448 gso_type=1 ip_summed=3
[ 633.129651] WARNING: CPU: 6 PID: 9489 at net/core/dev.c:2662 skb_warn_bad_offload+0xd8/0xef
[ 633.129652] Modules linked in: tcp_diag udp_diag inet_diag unix_diag ebtable_filter ebtable_nat iptable_raw veth bonding ip6table_mangle xt_CHECKSUM rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd cryptd glue_helper intel_cstate intel_rapl_perf iTCO_wdt iTCO_vendor_support ocrdma ib_core ipmi_si ipmi_devintf lpc_ich pcspkr sg be2iscsi hpwdt hpilo mfd_core ipmi_msghandler iscsi_boot_sysfs video acpi_power_meter ioatdma shpchp dca pcc_cpufreq xfs sd_mod mgag200 i2c_algo_bit serio_raw drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
[ 633.129704] ttm drm hpsa scsi_transport_sas be2net dm_mirror dm_region_hash dm_log vhost_net tun tap vhost iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_vs iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_mangle iptable_filter ipt_REJECT nf_reject_ipv4 ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_nat nf_conntrack libcrc32c crc32c_intel ip6table_filter ip6_tables ebtables dm_snapshot dm_bufio dm_multipath dm_mod 8021q garp mrp ip_tables
[ 633.129736] CPU: 6 PID: 9489 Comm: httpd Tainted: G W 4.17.12-1.el7.elrepo.x86_64 #1
[ 633.129738] Hardware name: HP ProLiant BL460c Gen8, BIOS I31 06/01/2015
[ 633.129741] RIP: 0010:skb_warn_bad_offload+0xd8/0xef
[ 633.129742] RSP: 0018:ffff88042f783698 EFLAGS: 00010282

You know what i found these error directly related to iptables checksum-fill rules.

[root@ostack-infra-02 etc]# iptables -S -t mangle
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM --checksum-fill
-A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM --checksum-fill
-A POSTROUTING -s 10.0.3.0/24 -o lxcbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill

Can someone explain me what CHECKSUM doing in ports 80/8000? I think this rule make no sense to me, i am curious to know why it is there? doing checksum on tcp port 80 will just hurt performing and won't gain anything.

-A POSTROUTING -p tcp -m tcp --sport 80 -j CHECKSUM --checksum-fill
-A POSTROUTING -p tcp -m tcp --sport 8000 -j CHECKSUM --checksum-fill

I can understand we need DHCP checksum rule because virtual nic doesn't perform checksum so we have to do checksum-fill.

[Kevin Carter (kevin-carter)]

neutron meta-data services required this fix for some time however that can be disabled with the option `neutron_metadata_checksum_fix` set to false [ https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/queens/defaults/main.yml#L351-L356 ].

If you're seeing this on your systems then we either had an issue with the conditions surrounding that option or the nodes started life as an AIO resulting in the iptables rules being created. The script that runs these commands can be found here [ https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/queens/files/post-up-metadata-checksum ].

[Satish Patel (satish-txt)]

Thanks kevin,

I am going to set "neutron_metadata_checksum_fix: false" just curious what kind of issue we have seen with meta-data related checksum? I thought checksum is only use in UDP scenario not in TCP. I may be wrong but love to find out.

[Jean-Philippe Evrard (jean-philippe-evrard)]

The checksum issue is to be investigated, but I think we can consider this case closed with this patch: https://review.openstack.org/#/c/589463/ .