kernel panics when executing "conntrackd -c"

[Rainer Sabelka]

I'm using Ubuntu 8.04 server for a pair of redundant firewalls with keepalived and conntackd (conntrack-tools).
The systems run the current ubuntu 2.6.24-18.32 server kernel.
Whenever I execute "conntrackd -c" i get either a kernel oops or a kernel panic.

I've attached a scenshot of the kernel panic.

I've already reported this bug upstream to the netfilter-devel list (see this thread: http://marc.info/?l=netfilter-devel&m=121310368207731&w=2)

The underlying problem has already been fixed in 2.6.26-rc.
Krzysztof Oledzki provided a backport for the fix consisting of 4 patches (see http://marc.info/?l=netfilter-devel&m=121316653716813&w=2)
which I applied to the current Ubuntu linux-source.
These patches indeed fixed the kernel panic and oopses I got on my system.

It would be nice, if this could also fixed in the kernel of Ubuntu 8.04. Especially since this is a LTS distribution which people may use as a basis for building network appliances like iptable firewalls, routers, load balancers etc. this fix seems rater important.

[Leann Ogasawara (leannogasawara)]

Thanks Rainer,

I'm going to include the 4 upstream git commit id's for the kernel team to reference.

Also, just in case you are interested, the upcoming Intrepid Ibex 8.10 kernel already has these patches applied as it was most recently sync'd with the upstream 2.6.26 kernel. If you'd be interested in running the latest Alpha for the upcoming Intrepid release more information can be found at http://www.ubuntu.com/testing. Thanks.

commit 86577c661bc01d5c4e477d74567df4470d6c5138
Author: Patrick McHardy <email address hidden>
Date: Thu Feb 7 17:56:34 2008 -0800

    [NETFILTER]: nf_conntrack: fix ct_extend ->move operation

commit 019f692ea719a2da17606511d2648b8cc1762268
Author: Pekka Enberg <email address hidden>
Date: Mon Mar 10 16:43:41 2008 -0700

    [NETFILTER]: nf_conntrack: replace horrible hack with ksize()

commit ceeff7541e5a4ba8e8d97ffbae32b3f283cb7a3f
Author: Patrick McHardy <email address hidden>
Date: Wed Jun 11 17:51:10 2008 -0700

    netfilter: nf_conntrack: fix ctnetlink related crash in nf_nat_setup_info()

commit 68b80f11380889996aa7eadba29dbbb5c29a5864
Author: Patrick McHardy <email address hidden>
Date: Tue Jun 17 15:51:47 2008 -0700

    netfilter: nf_nat: fix RCU races

[Colin Ian King (colin-king)]

Hi,

I've added the patches and built a new linux-kernel package for you to test - you can download the package from my PPA at: https://launchpad.net/~colin-king/+archive

Add the following lines to your apt sources.list:

deb http://ppa.launchpad.net/colin-king/ubuntu hardy main
deb-src http://ppa.launchpad.net/colin-king/ubuntu hardy main

Then run the command:

sudo apt-get update

Alternatively, follow the instructions at: https://help.ubuntu.com/8.04/add-applications/C/extra-repositories-adding.html

If you can test this package and let me know if it works, then I can add these patches into the hardy as a Stable Release Update (SRU).

Thanks, Colin

[Rainer Sabelka (sabelka)]

Colin,

I've installed the new kernel from you PPA (linux-image-2.6.24-22-virtual).
Conntrackd seems to work now - no more oopses!

Thanks,
-Rainer

[Colin Ian King (colin-king)]

Hi Rainer,

Thanks for testing this. I had some difficulty trying to reproduce this bug on the original Hardy kernel - can you supply me some details on all the necessary configuration to help me reproduce this for testing purposes? Thanks

Colin

[Rainer Sabelka (sabelka)]

Colin,
this is my setup where this bug first occured:
* Ubuntu Linux with kernel 2.6.24-18-server
* libnfnetlink 0.0.38 (compiled from sources)
* libnetfilter-conntrack 0.0.94 (compiled from sources)
* conntrack-tools 0.9.7 (compiled from sources)
* Keepalived v1.1.15

Both machines have a rather identical configuation running a redundant iptables firewall. Conntrackd is running tpo replicate the connection state of the active firewall to the backup (I'll attach conntrackd.conf).
The firewalls are in an active/standby configuration managed by keepalived (thoug I don't think you need keepalived to reproduce the bug).

So to trigger this bug you'll need to:
* make some connections over the active firewall. When I do "conntrackd -i" on the active firewall I see typically a few hundred lines.
* Then execute "conntackd -c" on the other machine. This does not trigger the bug everytime, especially since I've turned on logging in conntrackd.conf the bug occured rather infrequent. But executing "while sleep 1 ; do conntrackd -c ; done" crashed the machine almost always within a few seconds.

Here I've two references which probably describe the same bug:
http://lists.netfilter.org/pipermail/netfilter-failover/2007-January/000710.html
http://www.spinics.net/lists/netfilter-devel/msg04170.html

[Steve Langasek (vorlon)]

Hi Colin,

Is there any more information needed before this can be committed for Ubuntu 8.04.3?

[Andy Whitcroft (apw)]

@steve -- although it has been tested, the patch is complex and some concerns have been raised about the backported patches. We are reevaluating those currently and likely will produce a further test kernel before being happy to recommend these for SRU.

[Andy Whitcroft (apw)]

@Rainer -- we have some new test kernels containing updated patches for this issue. it would help if you could test these kernels and let us know if they also resolve the issue. Please test and report back here. These kernels can be found at the URL below:

    http://people.ubuntu.com/~apw/lp239215-hardy/

[Colin Ian King (colin-king)]

No response from request to test apw's test kernels back in June, as per bug team policy, marking this as Won't Fix.