kernel stacktrace on volume detach in kvm guest

Bug #458201 reported by Scott Moser on 2009-10-22
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
QEMU
Undecided
Unassigned
linux (Ubuntu)
Medium
Stefan Bader
qemu-kvm (Ubuntu)
Low
Unassigned

Bug Description

Binary package hint: qemu-kvm

I'm using eucalyptus, an install post beta

I do (cc$ indicates cloud controller, instance$ indicates in instance)

- cc$ euca-create-volume -s 1 -z sm-zone-1a
- cc$ euca-attach-volume -i i-4AA8096D -d sdb vol-32F804B0
- instance$ sudo mkfs.ext4 -F /dev/sdb
- instance$ sudo mount /dev/sdb /mnt
- instance$ echo "hello world" | tee /mnt/foo
- instance$ sudo umount /mnt
- instance$ sudo mount /dev/sdb /mnt
- instance$ cat /mnt/foo
- instance$ sudo umount /mnt
- cc$ euca-detach-volume vol-32F804B0

And the instance sees a kernel stack trace in dmesg.

$ dpkg-query --show qemu-kvm "*euca*"
euca2ools 1.0+bzr20091007-0ubuntu1
eucalyptus-common 1.6~bzr931-0ubuntu2
eucalyptus-gl 1.6~bzr931-0ubuntu2
eucalyptus-nc 1.6~bzr931-0ubuntu2
qemu-kvm 0.11.0-0ubuntu5

the UEC image was 20091022 (RC)

Scott Moser (smoser) wrote :
Scott Moser (smoser) wrote :

For the record, IRC snippit:

<smoser> i think i might have been missing some step to explicitly release /dev/sdb from the guest (maybe 'eject /dev/sdb' or something)
<kirkland> smoser: the stack trace on detach?
<kirkland> smoser: i've seen that, clearly wrong, but I haven't noticed any mal-effects

Subsequently, I did:
cc$ euca-attach-volume -i i-4AA8096D -d sdb vol-32F804B0
instance$ sudo mount /dev/sdb /mnt
instance$ # verifiy file is there
instance$ sudo umount /mnt
cc$ euca-detach-volume vol-32F804B0
instance$ grep sdb /proc/partitions || echo "IT IS STILL THERE"
# it wasn't still there.

So, this does indeed not seem to affect re-attaching the volume with the same device name.

Can you please reproduce outside of a eucalyptus environment

Changed in qemu:
status: New → Incomplete
Thierry Carrez (ttx) on 2009-10-28
Changed in qemu-kvm (Ubuntu):
importance: Undecided → Low
status: New → Confirmed
Scott Moser (smoser) wrote :

We need to test this on lucid Eucalyptus.

Dustin Kirkland  (kirkland) wrote :

Scott- Can you please confirm this as present or fixed in Lucid?

Changed in qemu-kvm (Ubuntu):
status: Confirmed → Incomplete
Scott Moser (smoser) wrote :

Ok, I've verified this is still present in lucid.
On lucid beta1 kvm, with lucid beta-1 server install

# create a disk image
truncate 10M /tmp/my.img

# launch kvm, watch it boot
kvm -drive file=sda.img,if=scsi,index=0,boot=on -drive file=sdb.img,if=scsi,index=1 -m 512

# in kvm guest, do
$ sudo modprobe acpi_php
# dmesg | tail will now show evidence of something (slot...)

# in kvm monitor, do
% pci_add auto storage file=/tmp/my.img,if=scsi
OK domain 0, bus 0, slot 5, function 0

# in kvm guest, /proc/partitions should now have a new entity (/dev/sdc)
# make a filesystem there, mount, write, unmount
$ sudo mke2fs /dev/sdc
$ sudo mount /dev/sdc /mnt
$ sudo sh -c 'echo hi mom > /mnt/foo'
$ sudo umount /mnt

# in the kvm monitor, detach the device
% pci_del 0:5

# in the kvm guest, now dmesg will show traceback failure

Changed in qemu-kvm (Ubuntu):
status: Incomplete → Confirmed
Changed in qemu:
status: Incomplete → Confirmed
Stefan Bader (smb) wrote :

It seems this is a problem in the symbios scsi driver which looks like being present in the same form upstream.

[ 271.732918] [<ffffffff8139a109>] __sym_mfree_dma+0x69/0x100

This function disables interrupts to take a spinlock. Then calls functions to free resources which will emit the warning when called with interrupts disabled.

void __sym_mfree_dma(m_pool_ident_t dev_dmat, void *m, int size, char *name)
{
        unsigned long flags;
        m_pool_p mp;

        spin_lock_irqsave(&sym53c8xx_lock, flags);
        mp = ___get_dma_pool(dev_dmat);
        if (!mp)
                goto out;
        __sym_mfree(mp, m, size, name);
#ifdef SYM_MEM_FREE_UNUSED
        if (!mp->nump)
                ___del_dma_pool(mp);
#endif
 out:
        spin_unlock_irqrestore(&sym53c8xx_lock, flags);
}

Not sure whether the spinunlock might be simply moved up a bit. That might need some discussion with upstream and or testing. But the bug task could be moved to kernel. I guess the QEMU task can only be set to invalid.

Changed in linux (Ubuntu):
importance: Undecided → Low
status: New → Triaged
Stefan Bader (smb) wrote :

SILOGIC/SYMBIOS/NCR 53C8XX and 53C1010 PCI-SCSI drivers
M: Matthew Wilcox <email address hidden>
L: <email address hidden>
S: Maintained
F: drivers/scsi/sym53c8xx_2/

Changed in qemu-kvm (Ubuntu):
status: Confirmed → Invalid
Scott Moser (smoser) wrote :

I'm moving this to 'Importance' medium, but might want it to be 'high'. This is the code path that Eucalyptus uses for EBS. EBS is suposed to be reliable storage for cloud instances. With that dmesg output you don't get warm fuzzy feelings about how reliable your data is.

Changed in linux (Ubuntu):
importance: Low → Medium
tags: added: uec-images
Stefan Bader (smb) wrote :

Would one of the kernels at http://people.canonical.com/~smb/lp458201/ make things better or worse?

Scott Moser (smoser) wrote :

I tested the linux-image-2.6.32-17-virtual_2.6.32-17.26+lp458201v1_amd64.deb . It appears to fix the problem. Thanks.

Stefan Bader (smb) wrote :

Update to this. I sent the quick patch for upstream review and it seems this needs a more complex approach. Regarding the warning the response was that it is bogus for any platform beside ARM. But is there because ARM might sleep in there.
I will try to get a better approach together but that needs a bit of thinking.

Changed in linux (Ubuntu):
assignee: nobody → Stefan Bader (stefan-bader-canonical)
status: Triaged → In Progress
Andy Whitcroft (apw) on 2010-04-13
Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 2.6.32-21.31

---------------
linux (2.6.32-21.31) lucid; urgency=low

  [ Andy Whitcroft ]

  * allow modules.builtin to be optional
  * d-i: add mpt2sas to the message-modules udeb
    - LP: #530361

  [ Christopher James Halse Rogers ]

  * SAUCE: Nouveau: Add quirk framework to disable acceleration
    - LP: #544088, #546393
  * SAUCE: Nouveau: Disable acceleration on MacBook Pros
    - LP: #546393
  * SAUCE: Nouveau: Disable acceleration on GeForce3 cards
    - LP: #544088
  * SAUCE: Nouveau: Disable acceleration on 6100 cards
    - LP: #542950

  [ Stefan Bader ]

  * SAUCE: dma-mapping: Remove WARN_ON in dma_free_coherent
    - LP: #458201

  [ Surbhi Palande ]

  * SAUCE: sync before umount to reduce time taken by ext4 umount
    - LP: #543617

  [ Upstream Kernel Changes ]

  * tipc: Fix oops on send prior to entering networked mode (v3)
    - CVE-2010-1187
  * KVM: x86 emulator: Add Virtual-8086 mode of emulation
    - LP: #561425
  * KVM: x86 emulator: fix memory access during x86 emulation
    - LP: #561425
  * KVM: x86 emulator: Check IOPL level during io instruction emulation
    - LP: #561425
  * KVM: x86 emulator: Fix popf emulation
    - LP: #561425
  * KVM: Fix segment descriptor loading
    - LP: #561425
  * KVM: VMX: Update instruction length on intercepted BP
    - LP: #561425
  * KVM: VMX: Use macros instead of hex value on cr0 initialization
    - LP: #561425
  * KVM: SVM: Reset cr0 properly on vcpu reset
    - LP: #561425
  * KVM: VMX: Disable unrestricted guest when EPT disabled
    - LP: #561425
  * KVM: x86: disable paravirt mmu reporting
    - LP: #561425
  * AppArmor: Fix put of unassigned ns if aa_unpack fails
  * AppArmor: Fix refcount bug when exec fails
    - LP: #562063
  * AppArmor: Take refcount on cxt->profile to ensure it remains a valid
    reference
    - LP: #367499
  * AppArmor: fix typo in scrubbing environment variable warning
    - LP: #562060
  * AppArmor: fix regression by setting default to mediate deleted files
    - LP: #562056
  * AppArmor: fix refcount order bug that can trigger during replacement
    - LP: #367499
  * AppArmor: Make sure to unmap aliases for vmalloced dfas before they are
    live
    - LP: #529288
  * AppArmor: address performance regression of replaced profile
    - LP: #549428
  * AppArmor: make the global side the correct type
    - LP: #562047
  * AppArmor: use the kernel shared workqueue to free vmalloc'ed dfas
  * sky2: add register definitions for new chips
    - LP: #537168
  * sky2: 88E8059 support
    - LP: #537168
  * net: Fix Yukon-2 Optima TCP offload setup
    - LP: #537168
  * net: Add missing TST_CFG_WRITE bits around sky2_pci_write
    - LP: #537168
  * sky2: print Optima chip name
    - LP: #537168
  * (Upstream) dell-laptop: defer dell_rfkill_update to worker thread
    - LP: #555261
  * drm/nv40: add LVDS table quirk for Dell Latitude D620
    - LP: #539730
 -- Andy Whitcroft <email address hidden> Tue, 13 Apr 2010 18:50:58 +0100

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Changed in qemu:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers