qemu crashes with iscsi initiator (libiscsi) when using virtio

[Klaus Hochlehnert]

I just tested qemu (with kvm) with an iscsi disk using libiscsi.

I tried to install Ubuntu 12.04 as guest and when it comes to the disk configuration during the installation qemu crashes without any notice. No log, no dump.

In the log of the iscsi-target I found this:
iscsi-scst: ***ERROR***: Connection with initiator iqn.2013-06.de.test:testsrv01 unexpectedly closed!

qemu version 1.5.0
libiscsi version 1.9.0
Host OS: Ubuntu 12.04 with 13.04-Kernel (3.8)
iSCSI-target: SCST 2.2.x

qemu command line:
/usr/bin/kvm ... -drive file=iscsi://192.168.x.x/iqn.2006-10.de.test:test/0,if=virtio,bus=0,unit=0,cache=none,media=disk,aio=native -iscsi initiator-name=iqn.2013-06.de.test:testsrv01,header-digest=CRC32C-NONE ...

When choosing IDE instead of VIRTIO it doesn't crash when scanning the disks...

[Stefan Hajnoczi (stefanha)]

On Sun, Jun 16, 2013 at 11:48:27PM -0000, Klaus Hochlehnert wrote:
> Public bug reported:
>
> I just tested qemu (with kvm) with an iscsi disk using libiscsi.
>
> I tried to install Ubuntu 12.04 as guest and when it comes to the disk
> configuration during the installation qemu crashes without any notice.
> No log, no dump.
>
> In the log of the iscsi-target I found this:
> iscsi-scst: ***ERROR***: Connection with initiator iqn.2013-06.de.test:testsrv01 unexpectedly closed!
>
> qemu version 1.5.0
> libiscsi version 1.9.0
> Host OS: Ubuntu 12.04 with 13.04-Kernel (3.8)
> iSCSI-target: SCST 2.2.x
>
> qemu command line:
> /usr/bin/kvm ... -drive file=iscsi://192.168.x.x/iqn.2006-10.de.test:test/0,if=virtio,bus=0,unit=0,cache=none,media=disk,aio=native -iscsi initiator-name=iqn.2013-06.de.test:testsrv01,header-digest=CRC32C-NONE ...
>
> When choosing IDE instead of VIRTIO it doesn't crash when scanning the
> disks...

CCing Ronnie Sahlberg (libiscsi)

Can you run /usr/bin/kvm under gdb and post the backtrace?

Stefan

[Klaus Hochlehnert (klaus-kh-dev)]

Without debug information I just can provide this (on that server I can't recompile qemu with debugging information):

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffe67fe000
0x00007f1bca857313 in poll () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) c
Continuing.
[New Thread 0x7f1bb9a64700 (LWP 36180)]
[Thread 0x7f1bb9a64700 (LWP 36180) exited]
[New Thread 0x7f1bb9a64700 (LWP 36181)]
[Thread 0x7f1bb9a64700 (LWP 36181) exited]
[New Thread 0x7f1bb9a64700 (LWP 36207)]

Program received signal SIGABRT, Aborted.
0x00007f1bca7a5425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) c
Continuing.
[Thread 0x7f1bd023b940 (LWP 36119) exited]
[Thread 0x7f1bb9a64700 (LWP 36207) exited]
[Thread 0x7f1bc1e3c700 (LWP 36132) exited]
[Thread 0x7f1bbafff700 (LWP 36135) exited]

Program terminated with signal SIGABRT, Aborted.
The program no longer exists.

[Stefan Hajnoczi (stefanha)]

On Mon, Jun 17, 2013 at 05:37:57PM -0000, Klaus Hochlehnert wrote:
> Without debug information I just can provide this (on that server I
> can't recompile qemu with debugging information):
>
> warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffe67fe000
> 0x00007f1bca857313 in poll () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) c
> Continuing.
> [New Thread 0x7f1bb9a64700 (LWP 36180)]
> [Thread 0x7f1bb9a64700 (LWP 36180) exited]
> [New Thread 0x7f1bb9a64700 (LWP 36181)]
> [Thread 0x7f1bb9a64700 (LWP 36181) exited]
> [New Thread 0x7f1bb9a64700 (LWP 36207)]
>
> Program received signal SIGABRT, Aborted.
> 0x00007f1bca7a5425 in raise () from /lib/x86_64-linux-gnu/libc.so.6

The program invoked abort(3). This is a deliberate fatal error case.

If you're using distro packages, please do the backtrace without symbols
("bt" and "info proc mappings") when gdb stops with SIGABRT and post the
package version and distro you are using.

Then I'll download the debuginfo packages from the distro and look up
the symbols manually.

If you're using a QEMU built from source then I would need the symbols
from you.

Stefan

[Stefan Hajnoczi (stefanha)]

On Tue, Jun 18, 2013 at 02:26:19PM +0200, Laszlo Ersek wrote:
> On 06/18/13 11:38, Stefan Hajnoczi wrote:
> > On Mon, Jun 17, 2013 at 05:37:57PM -0000, Klaus Hochlehnert wrote:
> >> Without debug information I just can provide this (on that server I
> >> can't recompile qemu with debugging information):
> >>
> >> warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffe67fe000
> >> 0x00007f1bca857313 in poll () from /lib/x86_64-linux-gnu/libc.so.6
> >> (gdb) c
> >> Continuing.
> >> [New Thread 0x7f1bb9a64700 (LWP 36180)]
> >> [Thread 0x7f1bb9a64700 (LWP 36180) exited]
> >> [New Thread 0x7f1bb9a64700 (LWP 36181)]
> >> [Thread 0x7f1bb9a64700 (LWP 36181) exited]
> >> [New Thread 0x7f1bb9a64700 (LWP 36207)]
> >>
> >> Program received signal SIGABRT, Aborted.
> >> 0x00007f1bca7a5425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> >
> > The program invoked abort(3). This is a deliberate fatal error case.
> >
> > If you're using distro packages, please do the backtrace without symbols
> > ("bt" and "info proc mappings") when gdb stops with SIGABRT and post the
> > package version and distro you are using.
> >
> > Then I'll download the debuginfo packages from the distro and look up
> > the symbols manually.
> >
> > If you're using a QEMU built from source then I would need the symbols
> > from you.
>
> Yes. Yesterday I started responding to this report along the same lines
> -- SIGABRT + raise() is assert() or abort(), but there was no error
> message, hence abort().
>
> Also, the reporter said in the opening comment that he used
>
> Host OS: Ubuntu 12.04 with 13.04-Kernel (3.8)
>
> which does ship a debuginfo package for qemu-kvm ("qemu-kvm-dbgsym"),
> just some extra repos are needed:
>
> https://wiki.ubuntu.com/DebuggingProgramCrash#Debug_Symbol_Packages
>
> Thus I wanted to recommend the reporter to install the debuginfo package
> and reproduce the crash (as this would not require him to rebuild anything).
>
> And that's when I noticed
>
> qemu version 1.5.0
>
> (which is not Ubuntu's pristine version in the 12.04 release, ie. it's
> probably a manual build), *with*
>
> on that server I can't recompile qemu with debugging information
>
> and decided it was a lost cause and deleted my email.

I don't think it's a lost cause, we can figure it out but effort will be
required on all sides.

Stefan

[Klaus Hochlehnert (klaus-kh-dev)]

I'll see what I can do to recompile qemu with debugging information.
Maybe tomorrow.

But one other question. I thought this is the "normal" qemu bug reporting or is it Ubuntu only?
I tried with the latest release and followed the "Report a bug"-link from the qemu web site.

[Stefan Hajnoczi (stefanha)]

On Tue, Jun 18, 2013 at 09:34:39PM -0700, ronnie sahlberg wrote:
> I can reproduce with current QEMU.
>
> Ubuntu 13 crashes with if=virtio but if=ide is fine.
>
>
> But it seems dependent on the guest/kernel.
>
> For example Fedora-18-x86_64-Live-Desktop.iso installs and runs just
> fine, even with virtio
> But both ubuntu-12.04-desktop-amd64.iso or
> ubuntu-13.04-desktop-amd64.iso crash with if=virtio
>
>
> Stack backtrace I got is

The issue is not obvious to me yet but here some comments on the stack
trace:

> #0 0x00007f7a9e22d037 in __GI_raise (sig=sig@entry=6)
> at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1 0x00007f7a9e230698 in __GI_abort () at abort.c:90
> #2 0x00007f7aa0a93ec8 in qemu_ram_addr_from_host_nofail (
> ptr=ptr@entry=0x2020202024008000) at /DATA/SRC/qemu-kvm/qemu/exec.c:1399
> #3 0x00007f7aa0a94a50 in address_space_unmap (as=<optimised out>,
> buffer=0x2020202024008000, len=<optimised out>, is_write=<optimised out>,

Junk buffer address.

> access_len=1) at /DATA/SRC/qemu-kvm/qemu/exec.c:2155
> #4 0x00007f7aa0a94bef in cpu_physical_memory_unmap (buffer=<optimised out>,
> len=<optimised out>, is_write=<optimised out>, access_len=<optimised out>)
> at /DATA/SRC/qemu-kvm/qemu/exec.c:2189
> #5 0x00007f7aa0ad7867 in virtqueue_fill (vq=vq@entry=0x7f7aa34277f0,
> elem=elem@entry=0x7f7aa37ca328, len=1, idx=idx@entry=0)
> at /DATA/SRC/qemu-kvm/qemu/hw/virtio/virtio.c:243

Unmapping req->elem.in_sg[0] (serial number buffer).

> #6 0x00007f7aa0ad79cf in virtqueue_push (vq=0x7f7aa34277f0,
> elem=elem@entry=0x7f7aa37ca328, len=<optimised out>)
> at /DATA/SRC/qemu-kvm/qemu/hw/virtio/virtio.c:279
> #7 0x00007f7aa0aa9989 in virtio_blk_req_complete (
> req=req@entry=0x7f7aa37ca320, status=status@entry=0)
> at /DATA/SRC/qemu-kvm/qemu/hw/block/virtio-blk.c:49
> #8 0x00007f7aa0aa9ffb in virtio_blk_handle_request (
> req=req@entry=0x7f7aa37ca320, mrb=mrb@entry=0x7fff7a7b2060)
> at /DATA/SRC/qemu-kvm/qemu/hw/block/virtio-blk.c:376

VIRTIO_BLK_T_GET_ID - the guest is querying the device's serial number.

Stefan

[Laszlo Ersek (Red Hat) (lersek)]

http://lists.nongnu.org/archive/html/qemu-devel/2013-06/msg03987.html

[Stefan Hajnoczi (stefanha)]

On Fri, Jun 21, 2013 at 06:37:18PM -0700, Ronnie Sahlberg wrote:

Looks fine except these whitespace issues:

> @@ -727,14 +730,36 @@ static BlockDriverAIOCB *iscsi_aio_ioctl(BlockDriverState *bs,
> memcpy(&acb->task->cdb[0], acb->ioh->cmdp, acb->ioh->cmd_len);
> acb->task->expxferlen = acb->ioh->dxfer_len;
>
> + data.size = 0;
> if (acb->task->xfer_dir == SCSI_XFER_WRITE) {
> - data.data = acb->ioh->dxferp;
> - data.size = acb->ioh->dxfer_len;
> + if (acb->ioh->iovec_count == 0) {
> + data.data = acb->ioh->dxferp;
> + data.size = acb->ioh->dxfer_len;
> + } else {
> +#if defined(LIBISCSI_FEATURE_IOVECTOR)
> + scsi_task_set_iov_out(acb->task,

Indentation should be 4 spaces.

> + (struct scsi_iovec *) acb->ioh->dxferp,
> + acb->ioh->iovec_count);
> + #else

Space before #else?

[Stefan Hajnoczi (stefanha)]

On Mon, Jun 24, 2013 at 04:40:11PM +0200, Paolo Bonzini wrote:
> Il 23/06/2013 17:07, Ronnie Sahlberg ha scritto:
> > Don't assume that SG_IO is always invoked with a simple buffer,
> > check the iovec_count and if it is >= 1 then we need to pass an array
> > of iovectors to libiscsi instead of just a plain buffer.
> >
> > Signed-off-by: Ronnie Sahlberg <email address hidden>
> > ---
> > block/iscsi.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++++-------
> > 1 files changed, 49 insertions(+), 7 deletions(-)
>
> Cc: <email address hidden>
>
> Will apply to scsi-next in the next few days.

Paolo, there are small whitespace issues that you might like to fix when
merging:

> > + } else {
> > +#if defined(LIBISCSI_FEATURE_IOVECTOR)
> > + scsi_task_set_iov_out(acb->task,
> > + (struct scsi_iovec *) acb->ioh->dxferp,
> > + acb->ioh->iovec_count);

Should be 4-space indentation.

> > + #else

Space before #else?

[Klaus Hochlehnert (klaus-kh-dev)]

I had a chance to test it again.

I used qemu 1.6.0 now for this.
And the installation works now for virtio and ide.

Thanks for fixing this,
Klaus