kernel panic - null pointer dereference on ipset operations

[Laurent Sesquès]

== SRU Justification ==
A regression was introduced in Xenial, even prior to v4.4 Final. I did
not test prior to this kernel once I found the bug was fixed in
mainline. The bug reporter experienced crashes on machines running
iptables using ipsets. He could get a trace from the console on one of
them which is attached to the bug report.

On these machines, some ipset commands are automatically run to update the
sets, and/or to dump them (ipset restore, swap, delete ... / ipset save).

I was able to reproduce this bug as was cking. This bug was found to be
fixed by mainline commits 596cf3fe5854 and e5173418ac59.

== Fixes ==
596cf3fe5854 ("netfilter: ipset: fix race condition in ipset save, swap and delete")
e5173418ac59 ("netfilter: ipset: Fix race between dump and swap")

== Regression Potential ==
Low. This fixes a regression and is limited to netfilter.

== Test Case ==
A test kernel was built with these patches and tested by myself and cking.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Joel Sing (jsing)]

I've been able to reproduce this panic by running Ubuntu Xenial under qemu, with a script that effectively does ipset restore/swap/destroy in a loop, while also running ipset save in a separate loop.

[Haw Loeung (hloeung)]

Maybe it's fixed in this commit which landed in Linux 4.6-rc2:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=596cf3fe5854fe2b1703b0466ed6bf9cfb83c91e

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.19 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19-rc5

[Joseph Salisbury (jsalisbury)]

I'll try to reproduce this bug. @joel Sing do you happen to have the reproducer script available?

In parallel, I'll also build a Xenial test kernel with the commit posted by @Haw Loeung in comment 4

[Joseph Salisbury (jsalisbury)]

I built a Xenial test kernel with commit 596cf3fe5854fe2b1703b0466ed6bf9cfb83c91e. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1793753

Can you test this kernel and see if it resolves this bug?

If you are unable to test, could you provide the commands that you found to reproduce this bug?

[Joseph Salisbury (jsalisbury)]

Note, for the test kernel, you need to install both the linux-image and linux-image-extra .deb packages.

[Joseph Salisbury (jsalisbury)]

Just to confirm, this is a regression? This bug did not happen with 4.4.0-116, and started with 4.4.0-135?

[Joseph Salisbury (jsalisbury)]

I've been trying variations of this in a loop but have been unable to reproduce:

ipset create hash_ip1 hash:ip family inet hashsize 1024 maxelem 500000 counters
ipset create hash_ip2 hash:ip family inet hashsize 300000 maxelem 500000 counters
ipset create hash_ip3 hash:ip family inet hashsize 1024 maxelem 500000 counters

ipset save &

ipset swap hash_ip3 hash_ip2
ipset destroy hash_ip3

@joel sing, is this similar to how you were able to reproduce the bug?

[Joseph Salisbury (jsalisbury)]

I can reproduce this bug now as well as cking. Thanks for the assistance cking!

This bug has actually been around prior 4.4.0, but it fixed in 4.15.0 or newer. Commit 596cf3fe5854fe does not actually fix this bug, but it does provide a test case to reproduce this.

I will now perform a "Reverse" bisect and narrow down the commit(s) needed to resolve this bug.

[Joseph Salisbury (jsalisbury)]

I believe this is the fix that landed in mainline as of v4.14-rc5:
e517341 netfilter: ipset: Fix race between dump and swap

I'm testing now and will post a test kernel for others to test as well.

[Joseph Salisbury (jsalisbury)]

A combination of two commits seems to resolve this bug:
596cf3fe5854 ("netfilter: ipset: fix race condition in ipset save, swap and delete")
e5173418ac59 ("netfilter: ipset: Fix race between dump and swap")

I built a Xenial test kernel with these two commits. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1793753

Can you test this kernel and see if it resolves this bug?

[Joseph Salisbury (jsalisbury)]

SRU request submitted:
https://lists.ubuntu.com/archives/kernel-team/2018-September/095673.html

[Brad Figg (brad-figg)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Laurent Sesquès (sajoupa)]

I could reproduce the panic with a loop of ipset destroys / swaps / restores and then saves in another. Standard xenial machines consistently panic within a few minutes max.
With the same loops and the proposed kernel, I got them running overnight without any issues.
I'll check with Joel and if he's also ok, we'll add the verification-done-xenial tag.

Laurent

[Joel Sing (jsing)]

I've tested the proposed kernel under the previous test environment - after two hours the host was still up, where as it would have previously crashed within 30 minutes or so. As such, this appears to prevent the panic.