Repeatable kernel oops on container delete

[Alex Bligh]

== SRU Justification ==
Destroying a container causes a kernel Oops and will hang the system. The issue is reproducible. The user has successfully tested the patch against Oneiric and can confirm the Oops no longer occurs when using a patched Oneiric kernel. The patch has been submitted upstream (CC'd upstream stable) and is currently queued in the -mm tree. It also appears it will hit the 3.2 merge window. Please consider for SRU against Oneiric and Natty.

== Impact ==
The commit message of the patch notes that this will likely affect 2.6.26 and newer kernels, ie affects Lucid, Maverick, Natty, Oneiric. However, due to the nature of our SRU process, the bug reporter is likely only able to readily test Natty and Oneiric. Thus I'm only submitting this for SRU against Oneiric and Natty.

== Test Case ==
See reproducer in comment #6

== Fix ==
http://marc.info/?l=linux-mm-commits&m=131603308900694&w=2

-----

On linux-image-2.6.38-11-generic and linux-image-3.0.0-10-server, destroying a container causes a kernel OOPS and hang. This is totally repeatable.

Procedure to repeat:

Use the attached perl program.

The perl program:
a) sets up a veth device
b) forks
c) does clone(NS_NEWNET) on the child
d) moves one end of the veth device into the child's network namespace
e) pings between the parent and the child and runs conntrack -L
f) kills the child after a while.

[NB: this section used to mention lxc - this is a red herring caused by some surprising semantics of lxc, and in fact is nothing to do with the bug]

The oops is in general not possible to catch save via the console as the reboot/hang is immediate. However, I have attached an Oops from a marginally different kernel (2.6.38-10-server on Lucid) which is created in a marginally different way, but has the same call stack.

Bug information as required

1. System information.

lsb_release -rd gives:

Description: Ubuntu 11.04
Release: 11.04

or on another machine showing the same issue

$ lsb_release -rd
Description: Ubuntu oneiric (development branch)
Release: 11.10

2. apt-cache policy linux-image-2.6.38-11-generic

linux-image-2.6.38-11-generic:
 Installed: 2.6.38-11.49
 Candidate: 2.6.38-11.49
 Version table:
*** 2.6.38-11.49 0
       500 http://gb.archive.ubuntu.com/ubuntu/ natty-proposed/main amd64 Packages
       100 /var/lib/dpkg/status
    2.6.38-11.48 0
       500 http://gb.archive.ubuntu.com/ubuntu/ natty-updates/main amd64 Packages
       500 http://security.ubuntu.com/ubuntu/ natty-security/main amd64 Packages

or on the second machine:

$ apt-cache policy linux-image-3.0.0-10-server
linux-image-3.0.0-10-server:
  Installed: 3.0.0-10.16
  Candidate: 3.0.0-10.16
  Version table:
 *** 3.0.0-10.16 0
        500 http://gb.archive.ubuntu.com/ubuntu/ oneiric/main amd64 Packages
        100 /var/lib/dpkg/status

3) What I expected to happen:

Test program continues to run, showing ICMP traffic moving periodically

4) What actually happened:

Kernel hang within 10-20 seconds, Oops on console, data lost

5) We currently do not believe this to be a security vulnerability as containers cannot be created as non-root.
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 2011-09-10 19:18 seq
 crw-rw---- 1 root audio 116, 33 2011-09-10 19:18 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 1.22.1-0ubuntu2
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 11.10
HibernationDevice: RESUME=UUID=49b12664-6859-4b83-b861-2354c9c23c26
InstallationMedia: Ubuntu-Server 11.04 "Natty Narwhal" - Alpha amd64 (20110301.4)
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.
Lsusb:
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
MachineType: Bochs Bochs
Package: linux-lts-backport-natty
PciMultimedia:

ProcEnviron:
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.0.0-11-server root=/dev/mapper/hostname-root ro crashkernel=384M-2G:64M,2G-:128M quiet
ProcVersionSignature: Ubuntu 3.0.0-11.17-server 3.0.4
RelatedPackageVersions:
 linux-restricted-modules-3.0.0-11-server N/A
 linux-backports-modules-3.0.0-11-server N/A
 linux-firmware 1.60
RfKill: Error: [Errno 2] No such file or directory
Tags: oneiric
Uname: Linux 3.0.0-11-server x86_64
UpgradeStatus: Upgraded to oneiric on 2011-09-10 (0 days ago)
UserGroups: adm admin cdrom dialout lpadmin plugdev sambashare
dmi.bios.date: 01/01/2007
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2007:svnBochs:pnBochs:pvr:cvnBochs:ct1:cvr:
dmi.product.name: Bochs
dmi.sys.vendor: Bochs

[Alex Bligh (ubuntu-alex-org)]

Note that this bug affects Lucid (10.04 LTS) where 2.6.38-10 is a shipped kernel, as well as 11.04.

[Serge Hallyn (serge-hallyn)]

I can't reproduce this. I installed a new natty server VM, and created a natty container.

You might be doing different networking setup than I am. Can you describe your network setup and the lxc.conf you used (for -f argument for lxc-create)

[Alex Bligh (ubuntu-alex-org)]

I think this is two separate problems.

The kernel bug is one thing, but we now think it may not be recreated using the method specified.

Our method of attempting to recreate the oops (which appears reliably in an application we have written) does indeed cause an immediate reboot, but we now think for a different reason; the exact commands are set out above and the network setup was a desktop with eth0 on DHCP (i.e. nothing very exciting). We were not passing a -f argument to lxc-create, and we are now guessing this running /sbin/init (perhaps inside the container). ^C then kills the container's /sbin/init, which causes the system to reboot. It would probably be a good thing if in the absence of a config file (which the manpage describes as optional) lxc-create did not run /sbin/init by default (if that's what it's doing), but perhaps ran /bin/sh or something, if only on the principle of least surprise.

From my point of view the bug is really the oops though. It looks like we need to find another way to duplicate that.

[Alex Bligh (ubuntu-alex-org)]

OK, we are much closer to repeating this oops. Ignore the original instructions - that seems to be a surprising lxc feature rather than the oops.

In order to repeat, you need to have
a) a container
b) which is forwarding IPv4 traffic from one interface in the container to another (2 veth interfaces in this case) - one ping packet per second will do
c) iptables with an IP conntrack rule.
d) delete the container (it doesn't matter if you delete the iptables rule first and sleep for a couple of seconds).

You then get the oops above. This also occurs on

root@node-10-157-128-101:~# uname -a
Linux node-10-157-128-101 3.0.0-10-server #16-Ubuntu SMP Fri Sep 2 18:51:05 UTC 2011 x86_64 GNU/Linux

I have attached a double oops from that kernel too.

[Alex Bligh (ubuntu-alex-org)]

OK I can now replicate this at will with a perl script on vanilla Oneiric. This locks the machine hard. You will need a console to the virtual machine to see the Oops - it doesn't get written to disk as the lockup is immediate.

[Alex Bligh (ubuntu-alex-org)]

The attach patch fixes the issue.

The oops is called from cleanup_net when the namespace is destroyed. conntrack iterates through outstanding events and calls death_by_timeout on each of them, which in turn produces a call to ctnetlink_conntrack_event. This calls nf_netlink_has_listeners, which oopses because net->nfnl is NULL.

I made the container through (essentially) 'unshare -n'; I didn't explicitly set up netlink, but I presume it was set up else net->nfnl would have been NULL earlier (i.e. when an earlier connection timed out). This would thus suggest that net->nfnl is made NULL during the destruction of the container, which I think is done by nfnetlink_net_exit_batch.

I can see that the various subsystems are deinitialised in the opposite order to which the relevant register_pernet_subsys calls are called, and both nf_conntrack and nfnetlink_net_ops register their relevant subsystems. If nfnetlink_net_ops registered later than nfconntrack, then its exit routine would have been called first, which would cause the oops described. I am not sure there is anything to prevent this happening in a container environment.

Whilst there's perhaps a more complex problem revolving around ordering of subsystem deinit, it seems to me that missing a netlink event on a container that is dying is not a disaster. An early check for net->nfnl being non-NULL in ctnetlink_conntrack_event appears to fix this. There may remain a potential race condition if it becomes NULL immediately after being checked (I am not sure any lock is held at this point or how synchronisation for subsystem deinitialization works).

This patch should apply on everything from 2.6.26 (if not before) onwards; it appears to be a problem on all kernels. This was taken against Ubuntu-3.0.0-11.17. I have torture-tested it with the above perl script for 15 minutes or so; the perl script hung the machine within 20 seconds without this patch.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 843892

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Alex Bligh (ubuntu-alex-org)]

apport information

[Alex Bligh (ubuntu-alex-org)]

apport information

[Alex Bligh (ubuntu-alex-org)]

apport information

[Alex Bligh (ubuntu-alex-org)]

apport information

[Alex Bligh (ubuntu-alex-org)]

apport information

[Alex Bligh (ubuntu-alex-org)]

apport information

[Alex Bligh (ubuntu-alex-org)]

apport information

[Alex Bligh (ubuntu-alex-org)]

apport information

[Alex Bligh (ubuntu-alex-org)]

apport information

[Alex Bligh (ubuntu-alex-org)]

apport information

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Patch to fix oops" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Brad Figg (brad-figg)]

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel currently in the release pocket than the one you tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

Thank you for your help.

[Alex Bligh (ubuntu-alex-org)]

This patch has been added to the -mm tree. See the mail below.

---------- Forwarded Message ----------
Date: 14 September 2011 13:43:36 -0700
From: <email address hidden>
To: <email address hidden>
CC: <email address hidden>, <email address hidden>, <email address hidden>, <email address hidden>
Subject: + net-netfilter-nf_conntrack_netlinkc-fix-oops-on-container-destroy.patch added to -mm tree

The patch titled
     net/netfilter/nf_conntrack_netlink.c: fix Oops on container destroy
has been added to the -mm tree. Its filename is
     net-netfilter-nf_conntrack_netlinkc-fix-oops-on-container-destroy.patch

Before you just go and hit "reply", please:
   a) Consider who else should be cc'ed
   b) Prefer to cc a suitable mailing list as well
   c) Ideally: find the original patch on the mailing list and do a
      reply-to-all to that, adding suitable additional cc's

*** Remember to use Documentation/SubmitChecklist when testing your code ***

See http://userweb.kernel.org/~akpm/stuff/added-to-mm.txt to find
out what to do about this

The current -mm tree may be found at http://userweb.kernel.org/~akpm/mmotm/

------------------------------------------------------
Subject: net/netfilter/nf_conntrack_netlink.c: fix Oops on container destroy
From: Alex Bligh <email address hidden>

Problem:

A repeatable Oops can be caused if a container with networking
unshared is destroyed when it has nf_conntrack entries yet to expire.

A copy of the oops follows below. A perl program generating the oops
repeatably is attached inline below.

Analysis:

The oops is called from cleanup_net when the namespace is
destroyed. conntrack iterates through outstanding events and calls
death_by_timeout on each of them, which in turn produces a call to
ctnetlink_conntrack_event. This calls nf_netlink_has_listeners, which
oopses because net->nfnl is NULL.

The perl program generates the container through fork() then
clone(NS_NEWNET). I does not explicitly set up netlink
explicitly set up netlink, but I presume it was set up else net->nfnl
would have been NULL earlier (i.e. when an earlier connection
timed out). This would thus suggest that net->nfnl is made NULL
during the destruction of the container, which I think is done by
nfnetlink_net_exit_batch.

I can see that the various subsystems are deinitialised in the opposite
order to which the relevant register_pernet_subsys calls are called,
and both nf_conntrack and nfnetlink_net_ops register their relevant
subsystems. If nfnetlink_net_ops registered later than nfconntrack,
then its exit routine would have been called first, which would cause
the oops described. I am not sure there is anything to prevent this
happening in a container environment.

Whilst there's perhaps a more complex problem revolving around ordering
of subsystem deinit, it seems to me that missing a netlink event on a
container that is dying is not a disaster. An early check for net->nfnl
being non-NULL in ctnetlink_conntrack_event appears to fix this. There
may remain a potential race condition if it becomes NULL immediately
after being checked (I am not sure any lock is held at this point or
how synchronisation for subsystem deinitialization works).

Patch:

The ...

[Leann Ogasawara (leannogasawara)]

Hi Alex,

Thanks for working this upstream to get this submitted. I've built an Oneiric test kernel with the patch applied. Could you please test and let me know your results. I can then get this submitted for SRU. Thanks.

http://people.canonical.com/~ogasawara/lp843892/oneiric/

Also just posting a reference to the mm-commits mailing list thread for this patch:
http://marc.info/?l=linux-mm-commits&m=131603308900694&w=2

[Alex Bligh (ubuntu-alex-org)]

Leanne,

Will do. In the absence of kernel.org, please also see the message below, suggesting this is queued for 3.2.

Alex

---------- Forwarded Message ----------
Date: 20 September 2011 13:47:55 -0700
From: <email address hidden>
To: <email address hidden>
CC: <email address hidden>, <email address hidden>, <email address hidden>, <email address hidden>, <email address hidden>
Subject: [patch for 3.2 1/1] net/netfilter/nf_conntrack_netlink.c: fix Oops on container destroy

From: Alex Bligh <email address hidden>
Subject: net/netfilter/nf_conntrack_netlink.c: fix Oops on container destroy

Problem:

A repeatable Oops can be caused if a container with networking
unshared is destroyed when it has nf_conntrack entries yet to expire.

A copy of the oops follows below. A perl program generating the oops
repeatably is attached inline below.

Analysis:

The oops is called from cleanup_net when the namespace is
destroyed. conntrack iterates through outstanding events and calls
death_by_timeout on each of them, which in turn produces a call to
ctnetlink_conntrack_event. This calls nf_netlink_has_listeners, which
oopses because net->nfnl is NULL.

The perl program generates the container through fork() then
clone(NS_NEWNET). I does not explicitly set up netlink
explicitly set up netlink, but I presume it was set up else net->nfnl
would have been NULL earlier (i.e. when an earlier connection
timed out). This would thus suggest that net->nfnl is made NULL
during the destruction of the container, which I think is done by
nfnetlink_net_exit_batch.

I can see that the various subsystems are deinitialised in the opposite
order to which the relevant register_pernet_subsys calls are called,
and both nf_conntrack and nfnetlink_net_ops register their relevant
subsystems. If nfnetlink_net_ops registered later than nfconntrack,
then its exit routine would have been called first, which would cause
the oops described. I am not sure there is anything to prevent this
happening in a container environment.

Whilst there's perhaps a more complex problem revolving around ordering
of subsystem deinit, it seems to me that missing a netlink event on a
container that is dying is not a disaster. An early check for net->nfnl
being non-NULL in ctnetlink_conntrack_event appears to fix this. There
may remain a potential race condition if it becomes NULL immediately
after being checked (I am not sure any lock is held at this point or
how synchronisation for subsystem deinitialization works).

Patch:

The patch attached should apply on everything from 2.6.26 (if not before)
onwards; it appears to be a problem on all kernels. This was taken against
Ubuntu-3.0.0-11.17 which is very close to 3.0.4. I have torture-tested it
with the above perl script for 15 minutes or so; the perl script hung the
machine within 20 seconds without this patch.

Applicability:

If this is the right solution, it should be applied to all stable kernels
as well as head. Apart from the minor overhead of checking one variable
against NULL, it can never 'do the wrong thing', because if net->nfnl
is NULL, an oops will inevitably result. Therefore, checking is a reasonable
thing to do unless it can be proven than net->nfnl will never be NULL.

C...

[Alex Bligh (ubuntu-alex-org)]

Leann,

The kernel at
  http://people.canonical.com/~ogasawara/lp843892/oneiric/

seems to work fine.

Thanks for your work on this! (& apologies for spelling your name wrong last time)

Alex

[Leann Ogasawara (leannogasawara)]

Thanks for the quick testing turn around. I'll get this submitted to the Ubuntu kernel-team mailing list.

[Leann Ogasawara (leannogasawara)]

https://lists.ubuntu.com/archives/kernel-team/2011-September/017129.html

[Leann Ogasawara (leannogasawara)]

Patch has been applied to both the Oneiric and Natty git repo's. Marking this Fix Committed. Thanks.

[Leann Ogasawara (leannogasawara)]

Hi Alex,

I just want to give you a heads up that once this is uploaded to the natty-proposed pocket, a comment will be posted here (by a bot) to request testing (instructions will be supplied). There is a strict SRU policy reinforcing test confirmation of all patches which are uploaded, if there is no test feedback from the original bug reporter (or another affected user), the patches will be dropped. Thanks.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 3.0.0-12.19

---------------
linux (3.0.0-12.19) oneiric; urgency=low

  [ Alex Bligh ]

  * SAUCE: (drop after v3.1) net/netfilter/nf_conntrack_netlink.c: fix Oops
    on container destroy
    - LP: #843892

  [ Andy Whitcroft ]

  * [Config] standardise on HZ=250
  * SAUCE: headers_install: fix #include "..." usage for userspace
    - LP: #824377
  * make module-inclusion selection retain the left overs
  * add a new linux-image-extras package for virtual

  [ edwin_rong ]

  * SAUCE: Staging: add driver for Realtek RTS5139 cardreader
    - LP: #824273

  [ Greg Kroah-Hartman ]

  * SAUCE: staging: rts5139: add vmalloc.h to some files to fix the build.
    - LP: #824273

  [ Jesse Sung ]

  * SAUCE: Unregister input device only if it is registered
    - LP: #839238

  [ Keng-Yu Lin ]

  * [Config] Enable CONFIG_RTS5139=m on i386/amd64
    - LP: #824273

  [ Leann Ogasawara ]

  * SAUCE: x86: reboot: Make Dell Optiplex 990 use reboot=pci
    - LP: #768039
  * SAUCE: x86: reboot: Make Dell Latitude E6220 use reboot=pci
    - LP: #838402

  [ Ming Lei ]

  * SAUCE: ata: make DVD drive recognisable on systems with Sandybridge CPT
    chipset
    - LP: #794642

  [ Paolo Pisati ]

  * [Config] Compile-in vfat support for armel
    - LP: #853783

  [ Randy Dunlap ]

  * SAUCE: staging: fix rts5139 depends & build
    - LP: #824273

  [ Tim Gardner ]

  * [Config] Fix binary-% build target
  * SAUCE: (drop after 3.0.0) OMAP3 and 4 hwmod I2C units only allow 16 bit
    access
    - LP: #852225

  [ Upstream Kernel Changes ]

  * hfsplus: Fix kfree of wrong pointers in hfsplus_fill_super() error path
    - LP: #854987
  * rt2x00: Serialize TX operations on a queue.
    - LP: #855239
 -- Leann Ogasawara <email address hidden> Wed, 14 Sep 2011 06:14:30 -0700

[Herton R. Krzesinski (herton)]

This bug is awaiting verification that the kernel for Natty in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-natty' to 'verification-done-natty'.

If verification is not done by one week from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[HeinMueck (cperz)]

What should be the version of the proposed kernel? I see linux-image-2.6.38-9-generic in natty-proposed packages, whereas in natty I see linux-image-2.6.38-10-generic. Are you sure its available?

[HeinMueck (cperz)]

K, packages don't seem to contain it, but at least when I upgrade I get 2.6.38-11.50. Is that the kernel that should be tested?

[Leann Ogasawara (leannogasawara)]

It should be the 3.6.38-12.51 kernel in the natty-proposed pocket that contains the fix for this issue. Are you sure you enabled natty-proposed?

https://launchpad.net/ubuntu/+source/linux/2.6.38-12.51

[HeinMueck (cperz)]

Found it - looked into main instead of universe. Surprise surprise :) Its working for me, although once I saw a reboot instead of a crash. But I was not able to reproduce that. Anyone else?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Herton R. Krzesinski (herton)]

@HeinMueck: perhaps your reboot was the issue Alex described in comment #4 ? (not sure if you tested with containers or the testcase in comment #6)

Can you or Alex confirm the Natty 2.6.38-12.51 kernel fixes the oops, and the reboot is just the containers/init issue? If you are safe with it (oops is solved, testcase from comment #6), then change the tag verification-needed-natty to verification-done-natty, thanks.

[Alex Bligh (ubuntu-alex-org)]

I have tested this on the supplied Natty kernel

root@test:/home/amb# uname -a
Linux test 2.6.38-12-server #51-Ubuntu SMP Wed Sep 28 16:07:08 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
root@test:/home/amb# dpkg --list | fgrep 2.6.38-12
ii linux-image-2.6.38-12-server 2.6.38-12.51 Linux kernel image for version 2.6.38 on x86_64

with the perl test program supplied, and it works fine.

I think I have changed the tag appropriately.

[Chaskiel Grundman (cg2v)]

When I run the perl script on 2.6.38-12.51, the system does not crash, but the script doesn't behave the same as with 2.6.38-11.51

The pings fail, and every other run through startcontainer doesn't even set up the veth devices correctly. in addition, my lxc setup still triggers an double-oops-and-hang involving cleanup_net (though it's not the same oops, and I don't think the conntrack module is in the backtrace anymore. it certainly isn't in the second backtrace.)

[Alex Bligh (ubuntu-alex-org)]

Chaskiel,

Your logs show the pings going through in every other run. That's to be expected. My script has poor error handling, and relies on various sleep statements to set things up in order - it's meant to reliably trigger the bug, rather than reliably transmit data. Feel free to fix it up :-)

When you say "When I run the perl script on 2.6.38-12.51, the system does not crash, but the script doesn't behave the same as with 2.6.38-11.51", I would expect you to get oops on 2.6.38-11.51, as that's without my script. So I'm not sure what the word "but" means here, as opposed to "and".

I think your last sentence starting "In addition" you are saying that even with 2.6.8-12.51 you can still trigger a container related oops but with a different non-conntrack related traceback (so probably a different oops), and triggered by a different test case. That should probably be handled separately.

However, when I was looking for this bug, I noticed there were other places (including one in the same file) where nfnetlink_has_listeners() was being called without checking net->nfnl for NULL; in many, net->nfnl can't be NULL no doubt. If it's an oops at the same place (I don't think you attached the oops) you might try fixing it py putting in the obvious check.

[Chaskiel Grundman (cg2v)]

> the script doesn't behave the same as with 2.6.38-11
specifically, in -11, before it crashed, I didn't see the "destination host unreachable" before the successful ping responses. I thought that was strange, and that perhaps some other change in -12 changed some timing.

>That should probably be handled separately.
ok. if I can reduce my testcase, I'll open a new report. I don't have the oops, in part because there was a recursive fault which possiby hides the real issue and in part becaus the stacktrace was long and scrolled important bits off the screen. I suppose I should test inside vmware with a serial console..... I can tell you that the second oops did not have any modules in the backtrace. I _think_ that cleanup_net and do_error were in adjacent stack frames.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 2.6.38-12.51

---------------
linux (2.6.38-12.51) natty-proposed; urgency=low

  [Herton R. Krzesinski]

  * Release Tracking Bug
    - LP: #860832

  [ Alex Bligh ]

  * SAUCE: net/netfilter/nf_conntrack_netlink.c: fix Oops on container
    destroy
    - LP: #843892

  [ Jesse Sung ]

  * SAUCE: Unregister input device only if it is registered
    - LP: #839238

  [ Leann Ogasawara ]

  * SAUCE: x86: reboot: Make Dell Latitude E6220 use reboot=pci
    - LP: #838402
  * SAUCE: x86: reboot: Make Dell Latitude E6520 use reboot=pci
    - LP: #833705

  [ Ming Lei ]

  * SAUCE: fireware: add NO_MSI quirks for o2micro controller
    - LP: #801719

  [ Stefan Bader ]

  * [Config] Include all filesystem modules for virtual
    - LP: #761809

  [ Tim Gardner ]

  * [Config] kernel preparation cannot be parallelized
  * [Config] Linearize module/abi checks
  * [Config] Linearize and simplify tree preparation rules
  * [Config] Build kernel image in parallel with modules
  * [Config] Set concurrency for kmake invocations
  * [Config] Improve install-arch-headers speed
  * [Config] Fix binary-perarch dependencies
  * [Config] Removed stamp-flavours target
  * [Config] Serialize binary indep targets
  * [Config] Use build stamp directly
  * [Config] Restore prepare-% target
  * [Config] Fix binary-% build target

  [ Upstream Kernel Changes ]

  * Revert "drm/i915: disable PCH ports if needed when disabling a CRTC"
    - LP: #814325, #838181
  * drm/i915: restore only the mode of this driver on lastclose (v2)
    - LP: #848687
  * cifs: fix possible memory corruption in CIFSFindNext, CVE-2011-3191
    - LP: #834135
    - CVE-2011-3191
  * befs: Validate length of long symbolic links, CVE-2011-2928
    - LP: #834124
    - CVE-2011-2928
  * gro: Only reset frag0 when skb can be pulled, CVE-2011-2723
    - LP: #844371
    - CVE-2011-2723
  * inet_diag: fix inet_diag_bc_audit(), CVE-2011-2213
    - LP: #838421
    - CVE-2011-2213
  * si4713-i2c: avoid potential buffer overflow on si4713, CVE-2011-2700
    - LP: #844370
    - CVE-2011-2700
  * Bluetooth: Prevent buffer overflow in l2cap config request,
    CVE-2011-2497
    - LP: #838423
    - CVE-2011-2497
  * crypto: Move md5_transform to lib/md5.c, CVE-2011-3188
    - LP: #834129
    - CVE-2011-3188
  * net: Compute protocol sequence numbers and fragment IDs using MD5,
    CVE-2011-3188
    - LP: #834129
    - CVE-2011-3188
  * x86, intel, power: Initialize MSR_IA32_ENERGY_PERF_BIAS
    - LP: #760131
  * x86, intel, power: Correct the MSR_IA32_ENERGY_PERF_BIAS message
    - LP: #760131
  * rt2x00: Serialize TX operations on a queue.
    - LP: #855239
  * ext4: Fix max file size and logical block counting of extent format
    file, CVE-2011-2695
    - LP: #819574
    - CVE-2011-2695
 -- Herton Ronaldo Krzesinski <email address hidden> Tue, 27 Sep 2011 16:19:57 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-lts-backport-natty - 2.6.38-12.51~lucid1

---------------
linux-lts-backport-natty (2.6.38-12.51~lucid1) lucid-proposed; urgency=low

  [Herton R. Krzesinski]

  * Release Tracking Bug
    - LP: #862556

  [ Alex Bligh ]

  * SAUCE: net/netfilter/nf_conntrack_netlink.c: fix Oops on container
    destroy
    - LP: #843892

  [ Jesse Sung ]

  * SAUCE: Unregister input device only if it is registered
    - LP: #839238

  [ Leann Ogasawara ]

  * SAUCE: x86: reboot: Make Dell Latitude E6220 use reboot=pci
    - LP: #838402
  * SAUCE: x86: reboot: Make Dell Latitude E6520 use reboot=pci
    - LP: #833705

  [ Ming Lei ]

  * SAUCE: fireware: add NO_MSI quirks for o2micro controller
    - LP: #801719

  [ Stefan Bader ]

  * [Config] Include all filesystem modules for virtual
    - LP: #761809

  [ Tim Gardner ]

  * [Config] kernel preparation cannot be parallelized
  * [Config] Linearize module/abi checks
  * [Config] Linearize and simplify tree preparation rules
  * [Config] Build kernel image in parallel with modules
  * [Config] Set concurrency for kmake invocations
  * [Config] Improve install-arch-headers speed
  * [Config] Fix binary-perarch dependencies
  * [Config] Removed stamp-flavours target
  * [Config] Serialize binary indep targets
  * [Config] Use build stamp directly
  * [Config] Restore prepare-% target
  * [Config] Fix binary-% build target

  [ Upstream Kernel Changes ]

  * Revert "drm/i915: disable PCH ports if needed when disabling a CRTC"
    - LP: #814325, #838181
  * drm/i915: restore only the mode of this driver on lastclose (v2)
    - LP: #848687
  * cifs: fix possible memory corruption in CIFSFindNext, CVE-2011-3191
    - LP: #834135
    - CVE-2011-3191
  * befs: Validate length of long symbolic links, CVE-2011-2928
    - LP: #834124
    - CVE-2011-2928
  * gro: Only reset frag0 when skb can be pulled, CVE-2011-2723
    - LP: #844371
    - CVE-2011-2723
  * inet_diag: fix inet_diag_bc_audit(), CVE-2011-2213
    - LP: #838421
    - CVE-2011-2213
  * si4713-i2c: avoid potential buffer overflow on si4713, CVE-2011-2700
    - LP: #844370
    - CVE-2011-2700
  * Bluetooth: Prevent buffer overflow in l2cap config request,
    CVE-2011-2497
    - LP: #838423
    - CVE-2011-2497
  * crypto: Move md5_transform to lib/md5.c, CVE-2011-3188
    - LP: #834129
    - CVE-2011-3188
  * net: Compute protocol sequence numbers and fragment IDs using MD5,
    CVE-2011-3188
    - LP: #834129
    - CVE-2011-3188
  * x86, intel, power: Initialize MSR_IA32_ENERGY_PERF_BIAS
    - LP: #760131
  * x86, intel, power: Correct the MSR_IA32_ENERGY_PERF_BIAS message
    - LP: #760131
  * rt2x00: Serialize TX operations on a queue.
    - LP: #855239
  * ext4: Fix max file size and logical block counting of extent format
    file, CVE-2011-2695
    - LP: #819574
    - CVE-2011-2695

linux (2.6.38-11.50) natty-proposed; urgency=low

  [Herton R. Krzesinski]

  * Release Tracking Bug
    - LP: #848246

  [ Upstream Kernel Changes ]

  * Revert "eCryptfs: Handle failed metadata read in lookup"
  * Revert "KVM: fix kvmclock regression due to missing clock update"
  * Revert "ath9k: use split rx buffers to get rid of...