kernel NULL pointer in -virtual (-server) kernel

[Scott Moser]

When stress testing eucalyptus we have run into this oops inside VMs
[ 82.907577] BUG: unable to handle kernel NULL pointer dereference at 0000000000000358^M
[ 82.908842] IP: [<ffffffff813982e8>] sym_int_sir+0x2a8/0x750^M
[ 82.909773] PGD 0 ^M
[ 82.910110] Thread overran stack, or stack corrupted^M
[ 82.910870] Oops: 0000 [#1] SMP ^M
[ 82.911407] last sysfs file: /sys/devices/virtual/block/ram9/uevent^M

We launched 18 instances, 2 of them failed this way. The instances run with 192M of memory. With 6 VM launches on a single node all at the same time the host is under heavy load.

This occurred in 20100323 lucid x86_64 uec-image instance.

ProblemType: Bug
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
CurrentDmesg:

Date: Wed Mar 24 22:06:32 2010
DistroRelease: Ubuntu 10.04
Frequency: Once a day.
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Bochs Bochs
Package: linux-image-2.6.32-16-virtual 2.6.32-16.25
PciMultimedia:

ProcCmdLine: root=/dev/sda1 console=ttyS0
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: User Name 2.6.32-16.25-server
Regression: No
Reproducible: No
SourcePackage: linux
TestedUpstream: No
Uname: Linux 2.6.32-16-server x86_64
dmi.bios.date: 01/01/2007
dmi.bios.vendor: Bochs
dmi.bios.version: Bochs
dmi.chassis.type: 1
dmi.chassis.vendor: Bochs
dmi.modalias: dmi:bvnBochs:bvrBochs:bd01/01/2007:svnBochs:pnBochs:pvr:cvnBochs:ct1:cvr:
dmi.product.name: Bochs
dmi.sys.vendor: Bochs

[James (james-redhat-bugs)]

Description of problem:

During an VNC installation on a virt guest, the installer appeared to hang while installing the bootloader. Closer inspection on the console shows a kernel NULL pointer dereference.

Version-Release number of selected component (if applicable):

 * anaconda-13.23
 * kernel-2.6.33-0.23.rc5.git1.fc13.x86_64

How reproducible:
 * This is the first time out of 8 or more installs

Steps to Reproduce:
1. Perform an installation against http://alt.fedoraproject.org/pub/alt/stage/rawhide-testing/

Actual results:

Running anaconda 13.23, the Fedora system installer - please wait.
21:09:16 Starting VNC...
21:09:17 The VNC server is now running.
21:09:17

WARNING!!! VNC server running with NO PASSWORD!
You can use the vncpassword=<password> boot option
if you would like to secure the server.

21:09:17 Please manually connect your vnc client to test1200.test.redhat.com:1 (10.10.10.200) to begin the install.
Press <enter> for a shell
21:09:18 Starting graphical installation.

sh-4.1# BUG: unable to handle kernel NULL pointer dereference at 0000000000000358
IP: [<ffffffffa010846b>] sym_int_sir+0x646/0x1549 [sym53c8xx]
PGD 2b2ef067 PUD 3eea2067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:04.0/host2/target2:0:0/2:0:0:0/block/sda/removable
CPU 0
Pid: 0, comm: swapper Not tainted 2.6.33-0.23.rc5.git1.fc13.x86_64 #1 /
RIP: 0010:[<ffffffffa010846b>] [<ffffffffa010846b>] sym_int_sir+0x646/0x1549 [sym53c8xx]
RSP: 0018:ffff880003c039b0 EFLAGS: 00010087
RAX: 000000000000000a RBX: ffff88003e41c000 RCX: 0000000000000070
RDX: 0000000000000000 RSI: ffffffffa0103c22 RDI: ffffc90000a5a006
RBP: ffff880003c03a30 R08: ffffffff81a4b830 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff88003e41c090 R14: ffff88003e6a800b R15: 0000000000000084
FS: 0000000000000000(0000) GS:ffff880003c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000358 CR3: 000000002b2ec000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a4b020)
Stack:
 0000000000000000 0000000000000002 ffffffff81a4b020 0000000000000000
<0> 0000000000000000 0000000000000000 0000000000000000 0000000000000046
<0> 0000000000000000 0000000000000000 ffff880003c03a60 ffff88003e41c000
Call Trace:
 <IRQ>
 [<ffffffffa010980c>] sym_interrupt+0x49e/0x6d2 [sym53c8xx]
 [<ffffffffa0103c2a>] sym53c8xx_intr+0x4d/0x7b [sym53c8xx]
 [<ffffffff810abec8>] handle_IRQ_event+0x53/0x119
 [<ffffffff810add95>] handle_fasteoi_irq+0x90/0xd0
 [<ffffffff8100c437>] handle_irq+0x88/0x91
 [<ffffffff8147ca74>] do_IRQ+0x5c/0xc3
 [<ffffffff81477b13>] ret_from_intr+0x0/0x16
 [<ffffffff814778d6>] ? _raw_spin_unlock_irqrestore+0x4c/0x56
 [<ffffffff812fb08f>] ? spin_unlock_irqrestore+0xe/0x10
 [<ffffffff812fbb16>] ? scsi_dispatch_cmd+0x1c1/0x234
 [<ffffffff81301b81>] ? scsi_request_fn+0x476/0x4a3
 [<ffffffff812152ed>] ? __blk_run_queue+0x45/0x74
 [<ffffffff812153d1>] ? blk_run_queue+0x26/0x3a
 [<ffffffff81300ff4>] ? s...

[Ariel (ariel-redhat-bugs)]

This seems to be the same kvm scsi emulation issue reported and discussed on the LKML,
   http://<email address hidden>/msg08927.html

and also Debian bug #511914
   http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=511914

Rgds, Ariel

[Amit (amit-redhat-bugs)]

Which qemu-kvm version are you using? Could you try the one from virt-preview?

qemu's SCSI support isn't the one that gets most developer attention; can you please try ide or virtio?

[James (james-redhat-bugs)]

(In reply to comment #2)
> Which qemu-kvm version are you using? Could you try the one from virt-preview?

 * qemu-kvm-0.11.0-12.fc12.x86_64

I'm running into other issues when trying to use the F-12 virt-preview repository. I'll take those to <email address hidden> for feedback.

> qemu's SCSI support isn't the one that gets most developer attention; can you
> please try ide or virtio?

I can work around this by using ide or virtio. This is certainly specific to SCSI KVM installs.

[James (james-redhat-bugs)]

(In reply to comment #3)
> (In reply to comment #2)
> > Which qemu-kvm version are you using? Could you try the one from virt-preview?
>
> * qemu-kvm-0.11.0-12.fc12.x86_64
>
> I'm running into other issues when trying to use the F-12 virt-preview
> repository. I'll take those to <email address hidden> for feedback.

I'm still not able to start a guest using virt-preview repository. I've filed bug#566425 to address that issue.

[Fedora (fedora-redhat-bugs)]

This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.

[Bug (bug-redhat-bugs)]

This bug appears to have been reported against 'rawhide' during the Fedora 13 development cycle.
Changing version to '13'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Scott Moser (smoser)]

I saw this https://bugzilla.redhat.com/show_bug.cgi?id=560114 which links to http://<email address hidden>/msg08927.html

[Scott Moser (smoser)]

just for the record, ubuntu-bug was run on a different instance (same kernel/filesystem, but not the one that failed)

[Justin (justin-redhat-bugs)]

The issue which blocks use of virt-preview/F-13 qemu has been resolved. Would it be possible to retest with qemu-0.12.x on the host?

[Jeremy Foshee (jeremyfoshee)]

Hi Scott,

This bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? Can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 546458

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[C de-Avillez (hggdh2)]

This is still a concern, and not easily reproducible. I would rather keep it open, unless you have reason to consider the OOPS resolved on current 2.6.32-21. I am stress-testing (by extension) kvm, and hopefully either I can state it has not happened on the last (few) thousands tries, and recornfirm it.

[Jeremy Foshee (jeremyfoshee)]

then let's set the linux task to Triaged. i only like to use 'new' for actual new bugs. :-)

~JFo

[Anthony Liguori (anthony-codemonkey)]

I see no indication that this is actually a qemu issue. If there's any evidence that it is, please reopen.

[C de-Avillez (hggdh2)]

Just to confirm -- still present, 10.04 LTS up-to-date, UEC images also 10.04 up-to-date. A 2,000 run creating KVM instances under Eucalyptus shows 6 occurences of this OOPS:

WARNING:INSTANCE i-3EDE078A:[ 129.998256] BUG: unable to handle kernel NULL pointer dereference at 00000358
WARNING:INSTANCE i-406C06CE:[ 89.245841] BUG: unable to handle kernel NULL pointer dereference at 0000000000000358
WARNING:INSTANCE i-411D0851:[ 158.375444] BUG: unable to handle kernel NULL pointer dereference at 00000358
WARNING:INSTANCE i-4E1C08D4:[ 196.089623] BUG: unable to handle kernel NULL pointer dereference at 0000000000000358
WARNING:INSTANCE i-54800A8D:[ 67.825483] BUG: unable to handle kernel NULL pointer dereference at 0000000000000358
WARNING:INSTANCE i-5E970AA3:[ 87.610866] BUG: unable to handle kernel NULL pointer dereference at 00000358

[Bug (bug-redhat-bugs)]

This message is a reminder that Fedora 13 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 13. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as WONTFIX if it remains open with a Fedora
'version' of '13'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version'
to a later Fedora version prior to Fedora 13's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we may not be able to fix it before Fedora 13 is end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora please change the 'version' of this
bug to the applicable version. If you are unable to change the version,
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Bug (bug-redhat-bugs)]

Fedora 13 changed to end-of-life (EOL) status on 2011-06-25. Fedora 13 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.

[Serge Hallyn (serge-hallyn)]

Per comment #7, marking this invalid for qemu-kvm.

[Po-Hsu Lin (cypressyew)]

Closing this bug with Won't fix as this kernel / release is no longer supported.
Please feel free to open a new bug report if you're still experiencing this on a newer release (Bionic 18.04.3 / Disco 19.04)
Thanks!