Xen dom0 kernel corrupts software raid (2.6.24-19)

[spbike]

Binary package hint: linux-xen

Hard heron (fully patched) installed with kickstart file:
http://shell.cse.ucdavis.edu/~sbeards/xen.cfg

I build 4 directories on 4 raids:
mdadm -C /dev/md0 -l 5 -n 4 /dev/sd[abcd]3
mdadm -C /dev/md1 -l 5 -n 4 /dev/sd[efgh]3
mdadm -C /dev/md2 -l 5 -n 4 /dev/sd[ijkl]3
mdadm -C /dev/md3 -l 5 -n 4 /dev/sd[mnop]3

for i in `seq 0 3`; do
    mkfs.ext3 /dev/md$i;
    mount /dev/md$i /disk/$i;
    touch /disk/$i/f;
done

Then run:
 iozone -s 16g -r 1024 -t 4 -F /disk/[0123]/f

When I run iozone the disk system gets corrupted:

dmesg reports:
[ 2435.753683] 3w-9xxx: scsi8: ERROR: (0x06:0x001C): Failed to map scatter gather list.
[ 2435.753745] PCI-DMA: Out of SW-IOMMU space for 32768 bytes at device 0000:06:00.0
[ 2435.753811] 3w-9xxx: scsi8: ERROR: (0x06:0x001C): Failed to map scatter gather list.
[ 2435.753864] PCI-DMA: Out of SW-IOMMU space for 32768 bytes at device 0000:06:00.0
[ 2435.753913] 3w-9xxx: scsi8: ERROR: (0x06:0x001C): Failed to map scatter gather list.
[ 2435.754003] PCI-DMA: Out of SW-IOMMU space for 32768 bytes at device 0000:06:00.0
[ 2435.754068] 3w-9xxx: scsi8: ERROR: (0x06:0x001C): Failed to map scatter gather list.
[ 2435.754121] PCI-DMA: Out of SW-IOMMU space for 32768 bytes at device 0000:06:00.0
[ 2435.754170] 3w-9xxx: scsi8: ERROR: (0x06:0x001C): Failed to map scatter gather list.
[ 2435.754176] sd 8:0:3:0: [sdd] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
[ 2435.754181] end_request: I/O error, dev sdd, sector 37085198
[ 2435.754191] sd 8:0:3:0: [sdd] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
[ 2435.754194] end_request: I/O error, dev sdd, sector 37085582
[ 2435.854168] raid5:md0: read error corrected (8 sectors at 1902976 on sdd3)
[ 2435.854544] raid5:md0: read error corrected (8 sectors at 1902984 on sdd3)
[ 2435.854552] raid5:md0: read error corrected (8 sectors at 1902992 on sdd3)
[ 2435.854556] raid5:md0: read error corrected (8 sectors at 1903000 on sdd3)
[ 2435.854559] raid5:md0: read error corrected (8 sectors at 1903008 on sdd3)
[ 2435.854566] raid5:md0: read error corrected (8 sectors at 1903016 on sdd3)
...

/proc/mdstat reports:
md0 : active raid5 sda3[0] sdd3[4](F) sdc3[5](F) sdb3[6](F)
      187526400 blocks level 5, 64k chunk, algorithm 2 [4/1] [U___]
md1 : active raid5 sde3[0] sdh3[3] sdg3[2] sdf3[1]
      187526400 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
md2 : active raid5 sdi3[4](F) sdl3[5](F) sdk3[2] sdj3[1]
      187526400 blocks level 5, 64k chunk, algorithm 2 [4/2] [_UU_]
md3 : active raid5 sdm3[0] sdp3[3] sdo3[2] sdn3[1]
      187526400 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

When I replace the xen dom0 kernel:
Linux 43-246-120-128 2.6.24-19-xen #1 SMP Wed Jun 18 16:08:38 UTC 2008 x86_64 GNU/Linux

With:
Linux 43-246-120-128 2.6.24-19-generic #1 SMP Wed Jun 18 14:15:37 UTC 2008 x86_64 GNU/Linux

Everything works. I've reproduced this several times and every time the xen kernel causes multiple disks to drop out of raid, and the generic kernel works perfectly (no drops, no dmesg, no errors).

[spbike (bill-broadley)]

found the package name

[Amit Vainsencher (asvain)]

Hello,
This is my first time posting in this system, so if I've overlooked some form of etiquette or done something wrong, I apologize in advance.

There was a discussion of this issue in the xen mailing list, culminating in a patch: http://lists.xensource.com/archives/html/xen-changelog/2008-04/msg00008.html

I've adapted the patch to the Ubuntu kernel build structure. The attached file simply needs to be dropped into debian/binary-custom.d/xen/patchset/ in the kernel source tree. After a rebuild of the xen-flavored kernel, I haven't been able to reproduce this bug no matter how hard I try. It was quite easy to trigger before via a few minutes of bonnie++ execution. I'll continue my stress testing and report back here if this doesn't fix the issue, but for now, I'm pretty sure that it does.

[Amit Vainsencher (asvain)]

To be clear, I was testing this with linux-image-2.6.24-19-xen and both xen-3.2 and xen-3.1. According to posts on the xen mailing list, the issue is caused because the kernel will perform unnecessary bounce buffering with the sw-iommu when under heavy load from the 3w-9xxx driver.

[Amit Vainsencher (asvain)]

I've been using the system heavily with linux-image-2.6.24-19-xen, ver 2.6.24-19.41 w/patch applied and xen-hypervisor-3.2 ver 3.2.0-0ubuntu10. The issue seems to be fixed by the upstream patch.

[bJXjLjEHIaWT0tFd (bjxjljehiawt0tfd-deactivatedaccount)]

Thank you for your efforts, Amit!

Is anybody else running a kernel with the mentioned patch? I will try it myself in the coming days and report back here.

Why though is this seemingly ignored by the Ubuntu kernel team? 2.6.24 is the hardy kernel and hardy being a LTS release, shouldn't its kernels get fixed even if the bugged feature (e.g. xen dom0 support) has been dropped in the next release?

[bJXjLjEHIaWT0tFd (bjxjljehiawt0tfd-deactivatedaccount)]

I just went ahead assigning this to the kernel team in the hopes of getting the fix into hardy-updates. My apologies if community members are not supposed to escalate issues in this way

[bJXjLjEHIaWT0tFd (bjxjljehiawt0tfd-deactivatedaccount)]

Sorry for spamming your inboxes, just wanted to confirm that Amit's patch fixed the issue for me.

Previously I constantly ran into "Out of SW-IOMMU space" errors with my SATA controller ("ATI Technologies Inc SB600 Non-Raid-5 SATA"). Since applying the attached patch to the 2.6.24-21-xen kernel of my up-to-date hardy installation, I can not reproduce this error!

Hopefully this can be included into hardy-updates.

Thanks again, Amit!

[Stefan Bader (smb)]

This patch would have to go into the xen custom binaries. As there are no custom binaries for Intrepid and later that task could be closed.

[Stefan Bader (smb)]

IMO this is not linux-meta.

[Stefan Bader (smb)]

SRU justification:

Impact: Using SWIOMMU unnecessarily causes "out of space" errors and subsequently causes corruptions in software RAID (and probably also caused the problems mentioned in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/238118/comments/9).

Fix: (taken from the Xen changelog) This patch adds a new check, check_pages_physically_contiguous(),
to the test for pages stradding page boundaries both in swiotlb_map_sg() and dma_map_sg(), to capture these ranges and map them directly via virt_to_bus() mapping rather than through the swiotlb.

Testcase: See above.

[Stefan Bader (smb)]

http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-hardy.git;a=commitdiff;h=eb657c7a4d123b94b6b92d18650ee0ae3f87cdbb

[Martin Pitt (pitti)]

Accepted linux into hardy-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 2.6.24-23.46

---------------
linux (2.6.24-23.46) hardy-proposed; urgency=low

  [Alessio Igor Bogani]

  * rt: Updated PREEMPT_RT support to rt21
    - LP: #302138

  [Amit Kucheria]

  * SAUCE: Update lpia patches from moblin tree
    - LP: #291457

  [Andy Whitcroft]

  * SAUCE: replace gfs2_bitfit with upstream version to prevent oops
    - LP: #276641

  [Colin Ian King]

  * isdn: Do not validate ISDN net device address prior to interface-up
    - LP: #237306
  * hwmon: (coretemp) Add Penryn CPU to coretemp
    - LP: #235119
  * USB: add support for Motorola ROKR Z6 cellphone in mass storage mode
    - LP: #263217
  * md: fix an occasional deadlock in raid5
    - LP: #208551

  [Stefan Bader]

  * SAUCE: buildenv: Show CVE entries in printchanges
  * SAUCE: buildenv: Send git-ubuntu-log informational message to stderr
  * Xen: dma: avoid unnecessarily SWIOTLB bounce buffering
    - LP: #247148
  * Update openvz patchset to apply to latest stable tree.
    - LP: #301634
  * XEN: Fix FTBS with stable updates
    - LP: #301634

  [Steve Conklin]

  * Add HID quirk for dual USB gamepad
    - LP: #140608

  [Tim Gardner]

  * Enable CONFIG_AX25_DAMA_SLAVE=y
    - LP: #257684
  * SAUCE: Correctly blacklist Thinkpad r40e in ACPI
    - LP: #278794
  * SAUCE: ALPS touchpad for Dell Latitude E6500/E6400
    - LP: #270643

  [Upstream Kernel Changes]

  * Revert "[Bluetooth] Eliminate checks for impossible conditions in IRQ
    handler"
    - LP: #217659
  * KVM: VMX: Clear CR4.VMXE in hardware_disable
    - LP: #268981
  * iov_iter_advance() fix
    - LP: #231746
  * Fix off-by-one error in iov_iter_advance()
    - LP: #231746
  * USB: serial: ch341: New VID/PID for CH341 USB-serial
    - LP: #272485
  * x86: Fix 32-bit x86 MSI-X allocation leakage
    - LP: #273103
  * b43legacy: Fix failure in rate-adjustment mechanism
    - LP: #273143
  * x86: Reserve FIRST_DEVICE_VECTOR in used_vectors bitmap.
    - LP: #276334
  * openvz: merge missed fixes from vanilla 2.6.24 openvz branch
    - LP: #298059
  * openvz: some autofs related fixes
    - LP: #298059
  * openvz: fix ve stop deadlock after nfs connect
    - LP: #298059
  * openvz: fix netlink and rtnl inside container
    - LP: #298059
  * openvz: fix wrong size of ub0_percpu
    - LP: #298059
  * openvz: fix OOPS while stopping VE started before binfmt_misc.ko loaded
    - LP: #298059
  * x86-64: Fix "bytes left to copy" return value for copy_from_user()
  * NET: Fix race in dev_close(). (Bug 9750)
    - LP: #301608
  * IPV6: Fix IPsec datagram fragmentation
    - LP: #301608
  * IPV6: dst_entry leak in ip4ip6_err.
    - LP: #301608
  * IPV4: Remove IP_TOS setting privilege checks.
    - LP: #301608
  * IPCONFIG: The kernel gets no IP from some DHCP servers
    - LP: #301608
  * IPCOMP: Disable BH on output when using shared tfm
    - LP: #301608
  * IRQ_NOPROBE helper functions
    - LP: #301608
  * MIPS: Mark all but i8259 interrupts as no-probe.
    - LP: #301608
  * ub: fix up the conversion to sg_init_table()
    - LP: #301608
  * x86: adjust enable_NMI_through_LVT0()
    - LP: #301608
  * SCSI ips: handle scsi_add_host() failure, and other err cl...

[Leann Ogasawara (leannogasawara)]

It would be great to get confirmation from the original bug reporter, spbike, that this bug is indeed fixed with this newer kernel. Thanks.

[Alvin Cura (alvinc)]

I'm very excited for this to be fixed in Hardy, as I have had quite a bit of trouble with it.

I am perhaps not up-to-speed on things. But although I know that KVM is now the preferred virtualization solution, due in part to ease-of-use at the desktop, Xen still has a lot of usefulness at the server.

I would, of course, be happy to run my data center on Ubuntu, if all the requirements can be met. At the data center level, support for Xen would be one of them.

Is there any chance of keeping builds of Xen binaries in the repositories of intrepid and forward, without changing the direction of KVM as the preferred solution? I don't believe that what I am requesting here is a change. Merely that we continue to build and support what we already used to build and support.

Thanks,
- alvinc

[spbike (bill-broadley)]

Leann Ogasawara wrote:
> It would be great to get confirmation from the original bug reporter,
> spbike, that this bug is indeed fixed with this newer kernel. Thanks.
>

I switched to a fedora derived kernel for a good while, but when I installed a
few more dom0's I used the ubuntu kernel without problem. Now it seems harder
(again), but Xen seems to be no longer supported. Our newest installs are now
switching to Enomaly and kvm.