mlx5_core reports hardware checksum error for padded packets on Mellanox NICs

[Mohammad Heib]

BugLink: https://bugs.launchpad.net/bugs/1854842

[Impact]

On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, there is a kernel splat when sending large IP packets which have padding at the end.

enp6s0f0: hw csum failure
CPU: 19 PID: 0 Comm: swapper/19 Not tainted 4.15.0-72-generic
Call Trace:
<IRQ>
dump_stack+0x63/0x8e
netdev_rx_csum_fault+0x38/0x40
__skb_checksum_complete+0xbc/0xd0
nf_ip_checksum+0xc3/0xf0
icmp_error+0x27d/0x310 [nf_conntrack_ipv4]
nf_conntrack_in+0x15a/0x510 [nf_conntrack]
? __skb_checksum+0x68/0x330
ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4]
nf_hook_slow+0x48/0xc0
? skb_send_sock+0x50/0x50
ip_rcv+0x301/0x360
? inet_del_offload+0x40/0x40
__netif_receive_skb_core+0x432/0xb80
__netif_receive_skb+0x18/0x60
? __netif_receive_skb+0x18/0x60
netif_receive_skb_internal+0x45/0xe0
napi_gro_receive+0xc5/0xf0
mlx5e_handle_rx_cqe+0x48d/0x5e0 [mlx5_core]
? enqueue_task_rt+0x1b4/0x2e0
mlx5e_poll_rx_cq+0xd1/0x8c0 [mlx5_core]
mlx5e_napi_poll+0x9d/0x290 [mlx5_core]
net_rx_action+0x140/0x3a0
__do_softirq+0xe4/0x2d4
irq_exit+0xc5/0xd0
do_IRQ+0x86/0xe0
common_interrupt+0x8c/0x8c
</IRQ>

This bug is a further attempt to fix these splats, as there has been previous fixes in LP #1840854 and a series of commits which landed in 4.15.0-67 (LP #1847155) as a part of upstream -stable patches.

This bug will also fix the same problems on the new Mellanox CX6 and Bluefield hardware, which has been enabled already via previous upstream -stable patches which landed in LP #1847155.

[Fix]

This particular issue was fixed for Mellanox series 5 drivers in the following commits:

commit 0aa1d18615c163f92935b806dcaff9157645233a
Author: Saeed Mahameed <email address hidden>
Date: Tue Mar 12 00:24:52 2019 -0700
Subject: net/mlx5e: Rx, Fixup skb checksum for packets with tail padding

This commit required a minor backport.

This commit was selected for upstream -stable in 4.19.76 and 5.0.10.
This commit appears to be omitted from "Bionic update: upstream stable patchset 2019-10-07", which is LP #1847155, probably due to requiring a backport.

commit db849faa9bef993a1379dc510623f750a72fa7ce
Author: Saeed Mahameed <email address hidden>
Date: Fri May 3 13:14:59 2019 -0700
Subject: net/mlx5e: Rx, Fix checksum calculation for new hardware

This commit required a minor backport.

This commit was selected for upstream -stable in 5.1.21 and 5.2.4.
This commit has already been applied to the disco kernel, as part of stable updates.

[Testcase]

The following scapy script will reproduce this issue. Run from the machine with the Mellanox series 5 NIC:

1) a=Ether(dst='ff:ff:ff:ff:ff:ff')/IP(dst='127.0.0.1')/ICMP()/Padding(load='\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe\xfe')

2) sendp(a, iface='enp6s0f0')

3) Check dmesg on the reciever side. The example uses localhost, so check dmesg.

I have built some test kernels, which are available here:

https://launchpad.net/~mruffell/+archive/ubuntu/lp1854842-test
This kernel contains 0aa1d18615c163f92935b806dcaff9157645233a.

and

https://launchpad.net/~mruffell/+archive/ubuntu/lp1854842-test-2
This kernel contains db849faa9bef993a1379dc510623f750a72fa7ce.

If you install the test kernels the issue is resolved.

[Regression Potential]

The changes are limited to the mlx5_core driver, and only modify how packet checksums are calculated when padding is involved.

Both patches have been accepted and published by upstream -stable, and are widely accepted by the community.

Because of this, I believe the risk of regression is low.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1854842/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Mohammad Heib (mohamadh)]

Hi,
any updates?
Have you been able to reproduce the case?

[Matthew Ruffell (mruffell)]

Hi Mohammad,

I had a look at the git repo for 4.15.0-72, which is the current kernel in -updates, to see what commits you mentioned in the description are already present.

Subject: net/mlx5e: Rx, Fix checksum calculation for new hardware
Commit: db849faa9bef993a1379dc510623f750a72fa7ce
In 4.15.0-72? No.

Subject: net/mlx5e: Rx, Check ip headers sanity
Commit: 0318a7b7fcad9765931146efa7ca3a034194737c
In 4.15.0-72? Yes. Arrived in 4.15.0-67.

Subject: net/mlx5e: Rx, Fixup skb checksum for packets with tail padding
Commit: 0aa1d18615c163f92935b806dcaff9157645233a
In 4.15.0-72? No.

Subject: net/mlx5e: XDP, Avoid checksum complete when XDP prog is loaded
Commit: 5d0bb3bac4b9f6c22280b04545626fdfd99edc6b
In 4.15.0-72? Yes. Arrived in 4.15.0-67.

Subject: mlx5: fix get_ip_proto()
Commit: ef6fcd455278c2be3032a346cc66d9dd9866b787
In 4.15.0-72? Yes. Arrived in 4.15.0-67.

Subject: net/mlx5e: Allow reporting of checksum unnecessary
Commit: b856df28f9230a47669efbdd57896084caadb2b3
In 4.15.0-72? Yes. Arrived in 4.15.0-67.

Subject: net/mlx5e: don't set CHECKSUM_COMPLETE on SCTP packets
Commit: fe1dc069990c1f290ef6b99adb46332c03258f38
In 4.15.0-72? Yes. Arrived in 4.15.0-67.

Subject: net/mlx5e: Set ECN for received packets using CQE indication
Commit: f007c13d4ad62f494c83897eda96437005df4a91
In 4.15.0-72? Yes. Arrived in 4.15.0-67.

Subject: net/mlx5e: Add likely to the common RX checksum flow
Commit: 63a612f984a1fae040ab6f1c6a0f1fdcdf1954b8
In 4.15.0-72? No.

Subject: net/mlx5e: CHECKSUM_COMPLETE offload for VLAN/QinQ packets
Commit: f938daeee95eb36ef6b431bf054a5cc6cdada112
In 4.15.0-72? Yes. Arrived in 4.15-rc1.

So it appears that most of these commits are already present in 4.15.0-72.

I had a look at the code for:

Subject: net/mlx5e: Add likely to the common RX checksum flow
Commit: 63a612f984a1fae040ab6f1c6a0f1fdcdf1954b8

This commit does not really fix anything, it is an optimisation. So we won't
worry about this commit any more.

Subject: net/mlx5e: Rx, Fix checksum calculation for new hardware
Commit: db849faa9bef993a1379dc510623f750a72fa7ce

This commit was selected for -upstream stable in 5.1.21 and 5.2.4.

Subject: net/mlx5e: Rx, Fixup skb checksum for packets with tail padding
Commit: 0aa1d18615c163f92935b806dcaff9157645233a

This commit was selected for -upstream stable in 4.19.76 and 5.0.10.

Those two commits really should have been applied to the bionic 4.15 kernel, but it seems they have been missed, probably due to not cleanly applying to the 4.15 tree.

I will try and backport these commits now, and build you a test kernel if I manage to successfully backport these patches.

I will write back shortly.

Matthew

[Matthew Ruffell (mruffell)]

Hi Mohammad,

I have backported:

Subject: net/mlx5e: Rx, Fixup skb checksum for packets with tail padding
Commit: 0aa1d18615c163f92935b806dcaff9157645233a

to the bionic 4.15 kernel: https://paste.ubuntu.com/p/Svfp8DQGRP/

The test kernel is currently building in:
https://launchpad.net/~mruffell/+archive/ubuntu/lp1854842-test

Once the kernel is built and I have tested it to ensure it built correctly, I will provide instructions for you to test the kernel, since I cannot reproduce due to my lab not having any Mellanox NICs.

I have started looking into

Subject: net/mlx5e: Rx, Fix checksum calculation for new hardware
Commit: db849faa9bef993a1379dc510623f750a72fa7ce

and will likely start backporting this and doing a test kernel when I get in on Monday.

I will let you know when the test kernel is ready.

Thanks,
Matthew

[Mohammad Heib (mohamadh)]

Hi Matthew,

Thank you for your effort :).
please let me know if you need any help with the backporting.

Thanks,
Mohammad

[Matthew Ruffell (mruffell)]

Hi Mohammad,

I have completed backporting both commits and I have built test kernels for each.

Can you install the test kernels and run the reproducer with Mellanox hardware?
My lab doesn't have any Mellanox NICs, so I can't test myself.

Please note these test kernels are NOT SUPPORTED by Canonical, and are for TESTING PURPOSES ONLY. ONLY Install in a dedicated test environment.

The first test kernel is for:

Subject: net/mlx5e: Rx, Fixup skb checksum for packets with tail padding
Commit: 0aa1d18615c163f92935b806dcaff9157645233a
Backport: https://paste.ubuntu.com/p/Svfp8DQGRP/

Instructions to install (On a bionic system)
1) sudo add-apt-repository ppa:mruffell/lp1854842-test
2) sudo apt-get update
3) sudo apt install linux-image-unsigned-4.15.0-72-generic linux-modules-4.15.0-72-generic linux-modules-extra-4.15.0-72-generic linux-headers-4.15.0-72 linux-headers-4.15.0-72-generic
4) sudo reboot
5) uname -rv
4.15.0-72-generic #81+hf1854842v20191207b2-Ubuntu SMP Sat Dec 7 09:51:02 UTC 2019

Run the reproducer and see if the problem is fixed or not.

The second test kernel is for:

Subject: net/mlx5e: Rx, Fix checksum calculation for new hardware
Commit: db849faa9bef993a1379dc510623f750a72fa7ce
Backport: https://paste.ubuntu.com/p/QNPPSDf5Tm/

Instructions to install (On a bionic system)
1) sudo add-apt-repository ppa:mruffell/lp1854842-test-2
2) sudo apt-get update
3) sudo apt install linux-image-unsigned-4.15.0-72-generic linux-modules-4.15.0-72-generic linux-modules-extra-4.15.0-72-generic linux-headers-4.15.0-72 linux-headers-4.15.0-72-generic
4) sudo reboot
5) uname -rv
4.15.0-72-generic #81+hf1854842v20191209b1-Ubuntu SMP Sun Dec 8 23:37:16 UTC 2019

Run the reproducer and see if the problem is fixed or not.

Hopefully one of these kernels / commits fixes the problem. If it does, I will submit the commit to the Ubuntu kernel mailing list for SRU.

If neither of the kernels fix the problem, we will have to continue debugging.

Can you please test the kernels for me and let me know how they go?

Thanks,
Matthew

[Mohammad Heib (mohamadh)]

Hi Matthew,

it seems that the issue fixed by the first kernel :
' 1) sudo add-apt-repository ppa:mruffell/lp1854842-test'

Regarding the second kernel, i still see the call trace.

just to confirm you will add both patches to the next SRU since the second patch will fix the issue for our new HW cx6 and Bluefield.

[Matthew Ruffell (mruffell)]

Hi Mohammad,

Thanks for testing those kernels.

The second one did not include 0aa1d18615c163f92935b806dcaff9157645233a, which is why you still see the splat. Re-reading it, db849faa9bef993a1379dc510623f750a72fa7ce depends on 0aa1d18615c163f92935b806dcaff9157645233a to function, and I probably should have included it in the kernel for you to test.

I fixed the backport of db849faa9bef993a1379dc510623f750a72fa7ce slightly to integrate with 0aa1d18615c163f92935b806dcaff9157645233a better, and fixed the offset calculation in include/linux/mlx5/mlx5_ifc.h which I had previously missed on my initial backport. 0x1b - 0x1 = 0x1a instead of the 0xc from the upstream commit, since the 4.15 kernel does not have the max_vxlan_udp_ports[0x8] or reserved_at_38[0x6] offsets.

I have submitted both commits for SRU into the 4.15 bionic kernel, and they are now on the Ubuntu kernel mailing list:

https://lists.ubuntu.com/archives/kernel-team/2019-December/106293.html
https://lists.ubuntu.com/archives/kernel-team/2019-December/106294.html
https://lists.ubuntu.com/archives/kernel-team/2019-December/106295.html

The next step is for the kernel team to review the patches, and ack them. To be accepted into the next SRU cycle, they need a minimum of 2 acks. When they do get 2 acks, they will be applied to master-next and be a part of the next SRU cycle.

Once the patches have been acked and applied, the kernel will be built and available in -proposed. I will ask you to verify that the kernel in -proposed fixes the problem. I will be sure to message you when the time comes for this to happen.

Looking at https://kernel.ubuntu.com/, we see the last day for commits is the 1st of Jan, with eventual release somewhere around the 27th of Jan, give or take a few days.

I'll keep you informed about the progress of these patches.

Thanks,
Matthew

[Mohammad Heib (mohamadh)]

Hi Matthew,

Great Thanks Matthew,
waiting for updates from you :).

[Matthew Ruffell (mruffell)]

Hello Mohammad,

The kernel team has reviewed the patches and they have received two acks from senior kernel developers:

https://lists.ubuntu.com/archives/kernel-team/2019-December/106516.html
https://lists.ubuntu.com/archives/kernel-team/2020-January/106624.html

From there, the patch was applied to bionic master-next branch, which means it will be included in this current SRU cycle:

https://lists.ubuntu.com/archives/kernel-team/2020-January/106643.html

The next steps is for the kernel team to build the kernel update and push it to -proposed. This will likely happen at the end of this week / early next week. Once this happens, I will write again and ask you to test the kernel in -proposed to make sure that it fixes the problem.

If it does, we can tag this bug as verified, and wait for the kernel to be released.

I'll keep you informed of all progress, and I'll write back soon when it is time to test the kernel in -proposed.

Thanks,
Matthew

[Matthew Ruffell (mruffell)]

Hi Mohammad,

I have some bad news. The patches are not present in the current SRU cycle and have been delayed until the next SRU cycle.

This is not unique to us and this LP bug, everyone has had the same thing happen to them.

The reason is that the kernel team are currently spinning kernels for the Ubuntu 18.04.4 ISO image release, which is due in the coming weeks, and wished for a quiet SRU cycle, so it appears that they only applied reverts to known regressions (1x USB regression) and one CVE (a CVE related to intel 915 graphics drivers) to ensure things go smoothly with no regressions for the 18.04.4 release.

The problem with this, is that it has delayed all queued up patches for the bionic and the bionic 5.3 HWE kernel, and has delayed fixes being delivered for another 3 weeks.

I know that this is frustrating, and I sincerely apologise for the delay. I am also annoyed, since I only found out recently when I manually examined the commits present in the current SRU cycle, and only found 4, when there would normally be several hundred or so.

I believe things will go back to normal for the next SRU cycle, and all queued up patches will be incorporated then. As for the timeframe, if we look at: https://kernel.ubuntu.com/, we see a kernel update being built by the kernel team to happen sometime next week, and the update placed into -proposed for us to test between the 3rd and 14th of February. This would put a release around the 17th of February, given no more delays.

Again, I apologise for the delay, and I will make sure to let you know when the next 4.15 bionic kernel with the patches is ready to test in -proposed.

Thanks,
Matthew

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Matthew Ruffell (mruffell)]

Hi Mohammad,

It seems things have returned back to normal for this current SRU cycle, and the two commits you requested:

net/mlx5e: Rx, Fix checksum calculation for new hardware
net/mlx5e: Rx, Fixup skb checksum for packets with tail padding

Have been tagged and built into the 4.15.0-87-generic bionic kernel, which is currently sitting in -proposed awaiting validation.

Can you please install the kernel in -proposed, and run the reproducer and check that no kernel splat is generated when you send large IP packets with padding at the end?

Instructions to install (on a bionic system):
1) Add the -proposed repository, by adding the following line to /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu/ bionic-proposed restricted main multiverse universe
2) sudo apt update
3) sudo apt install linux-image-4.15.0-87-generic linux-modules-4.15.0-87-generic \
linux-modules-extra-4.15.0-87-generic linux-headers-4.15.0-87 linux-headers-4.15.0-87-generic
4) sudo reboot
5) uname -rv
4.15.0-87-generic #87-Ubuntu SMP Fri Jan 31 19:32:37 UTC 2020

Hopefully the reproducer shows everything has been fixed.

I apologise again for the delay, the kernel team were really adamant about having no regressions in the previous SRU cycle, but things should be back to normal now.

Let me know how the kernel in -proposed goes.

Thanks,
Matthew

[Mohammad Heib (mohamadh)]

Hi Matthew,
Sorry for the delay in responding,

i tried your kernel but it seems that something missing on the kernel that you provide above, i still see the Call Trace after running traffic with padding, can you please check if the kernel contains the needed patches, i tried to check the changelog but seems that the changelog not updated because i couldn't find the patches.

[Matthew Ruffell (mruffell)]

Hi Mohammad,

Can you double check that you tested with the 4.15.0-87-generic kernel from -proposed with the following uname string?

4.15.0-87-generic #87-Ubuntu SMP Fri Jan 31 19:32:37 UTC 2020

You can verify that the patches have landed in the kernel by looking at the git tree:

Checkout the code, switch to 4.15.0-87-generic tag:

$ git clone git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git
$ git checkout Ubuntu-4.15.0-87.87

Look at the git log history for the commits:

$ git log --oneline --grep "net/mlx5e: Rx, Fixup skb checksum for packets with tail padding"
d1a32cb24dfe net/mlx5e: Rx, Fixup skb checksum for packets with tail padding
$ git log --oneline --grep "net/mlx5e: Rx, Fix checksum calculation for new hardware"
786fb87b8f55 net/mlx5e: Rx, Fix checksum calculation for new hardware

Look at what version they are tagged in:

$ git describe --contains d1a32cb24dfe
Ubuntu-4.15.0-87.87~459

$ git describe --contains 786fb87b8f55
Ubuntu-4.15.0-87.87~458

Looking at the commits themselves:

$ git show d1a32cb24dfe
https://paste.ubuntu.com/p/yWs3tBqywK/
$ git show 786fb87b8f55
https://paste.ubuntu.com/p/byj34bPFjZ/

Note that:
- d1a32cb24dfe in the Ubuntu Bionic tree is 0aa1d18615c163f92935b806dcaff9157645233a upstream.
- 786fb87b8f55 in the Ubuntu Bionic tree is db849faa9bef993a1379dc510623f750a72fa7ce upstream.

I cross checked the backported commits with the upstream ones again, and they appear correct.

Can you please re-rest and double check you are running the 4.15.0-87-generic kernel from -proposed?

Thanks,
Matthew

[Matthew Ruffell (mruffell)]

Hi Mohammad,

Can you please test the 4.15.0-88-generic kernel in -proposed again? The SRU cycle will end in the next few days, and this bug needs to be verified if at all possible.

To the kernel team:

The two commits:
net/mlx5e: Rx, Fixup skb checksum for packets with tail padding
net/mlx5e: Rx, Fix checksum calculation for new hardware

Were submitted and accepted into upstream -stable versions 4.19.76 and 5.0.10 for the first, and 5.1.21 and 5.2.4 for the second commit. They were dropped from bionic stable updates because each commit required backporting, which I have completed for this SRU.

If Mohammad does not reply before tomorrow, I will mark this bug as verified, on the basis that these commits were marked for upstream -stable and I believe they have been backported correctly. I have done a basic smoke test on 4.15.0-87-generic, but I do not have access to Mellanox NICs to test myself.

Thanks,
Matthew

[Mohammad Heib (mohamadh)]

Hi Matthew,
first of all, I would like to apologize for the late reply
i was OOO for a while, sorry :P).

I re-tested the 4.15.0-88-generic and it seems that the issue resolved So it's verified by Mellanox.

Thank you very much guys for your efforts.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-88.88

---------------
linux (4.15.0-88.88) bionic; urgency=medium

  * bionic/linux: 4.15.0-88.88 -proposed tracker (LP: #1862824)

  * Segmentation fault (kernel oops) with memory-hotplug in
    ubuntu_kernel_selftests on Bionic kernel (LP: #1862312)
    - Revert "mm/memory_hotplug: fix online/offline_pages called w.o.
      mem_hotplug_lock"
    - mm/memory_hotplug: fix online/offline_pages called w.o. mem_hotplug_lock

linux (4.15.0-87.87) bionic; urgency=medium

  * bionic/linux: 4.15.0-87.87 -proposed tracker (LP: #1861165)

  * Bionic update: upstream stable patchset 2020-01-22 (LP: #1860602)
    - scsi: lpfc: Fix discovery failures when target device connectivity bounces
    - scsi: mpt3sas: Fix clear pending bit in ioctl status
    - scsi: lpfc: Fix locking on mailbox command completion
    - Input: atmel_mxt_ts - disable IRQ across suspend
    - iommu/tegra-smmu: Fix page tables in > 4 GiB memory
    - scsi: target: compare full CHAP_A Algorithm strings
    - scsi: lpfc: Fix SLI3 hba in loop mode not discovering devices
    - scsi: csiostor: Don't enable IRQs too early
    - powerpc/pseries: Mark accumulate_stolen_time() as notrace
    - powerpc/pseries: Don't fail hash page table insert for bolted mapping
    - powerpc/tools: Don't quote $objdump in scripts
    - dma-debug: add a schedule point in debug_dma_dump_mappings()
    - clocksource/drivers/asm9260: Add a check for of_clk_get
    - powerpc/security/book3s64: Report L1TF status in sysfs
    - powerpc/book3s64/hash: Add cond_resched to avoid soft lockup warning
    - ext4: update direct I/O read lock pattern for IOCB_NOWAIT
    - jbd2: Fix statistics for the number of logged blocks
    - scsi: tracing: Fix handling of TRANSFER LENGTH == 0 for READ(6) and WRITE(6)
    - scsi: lpfc: Fix duplicate unreg_rpi error in port offline flow
    - f2fs: fix to update dir's i_pino during cross_rename
    - clk: qcom: Allow constant ratio freq tables for rcg
    - irqchip/irq-bcm7038-l1: Enable parent IRQ if necessary
    - irqchip: ingenic: Error out if IRQ domain creation failed
    - fs/quota: handle overflows of sysctl fs.quota.* and report as unsigned long
    - scsi: lpfc: fix: Coverity: lpfc_cmpl_els_rsp(): Null pointer dereferences
    - scsi: ufs: fix potential bug which ends in system hang
    - powerpc/pseries/cmm: Implement release() function for sysfs device
    - powerpc/security: Fix wrong message when RFI Flush is disable
    - scsi: atari_scsi: sun3_scsi: Set sg_tablesize to 1 instead of SG_NONE
    - clk: pxa: fix one of the pxa RTC clocks
    - bcache: at least try to shrink 1 node in bch_mca_scan()
    - HID: logitech-hidpp: Silence intermittent get_battery_capacity errors
    - libnvdimm/btt: fix variable 'rc' set but not used
    - HID: Improve Windows Precision Touchpad detection.
    - scsi: pm80xx: Fix for SATA device discovery
    - scsi: ufs: Fix error handing during hibern8 enter
    - scsi: scsi_debug: num_tgts must be >= 0
    - scsi: NCR5380: Add disconnect_mask module parameter
    - scsi: iscsi: Don't send data to unbound connection
    - scsi: target: iscsi: Wait for all commands to finish before freeing a
...

[Matthew Ruffell (mruffell)]

Hi Mohammad,

Thanks for verifying the kernel again, and good to see that things have been fixed when you run padded traffic over your NICs.

The SRU cycle has completed, and the 4.15.0-88-generic kernel has been released to -updates. You can go ahead and tell your affected customers to upgrade to this kernel to fix their checksum problems.

That's all from me now. Let me know if you need anything else.

Thanks,
Matthew