aarch64: virtio disks don't show up in guest (neither blk nor scsi)

[Richard Jones]

kernel-3.18.0-0.rc1.git0.1.rwmj5.fc22.aarch64 (3.18 rc1 + some hardware enablement)
qemu from git today

When I create a guest with virtio-scsi disks, they don't show up inside the guest.
Literally after the virtio_mmio.ko and virtio_scsi.ko modules are loaded, there are
no messages about disks, and of course nothing else works.

Really long command line (generated by libvirt):

HOME=/home/rjones USER=rjones LOGNAME=rjones QEMU_AUDIO_DRV=none TMPDIR=/home/rjones/d/libguestfs/tmp /home/rjones/d/qemu/aarch64-softmmu/qemu-system-aarch64 -name guestfs-oqv29um3jp03kpjf -S -machine virt,accel=tcg,usb=off -cpu cortex-a57 -m 500 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid a5f1a15d-2bc7-46df-9974-1d1f643b2449 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/home/rjones/.config/libvirt/qemu/lib/guestfs-oqv29um3jp03kpjf.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -no-reboot -boot strict=on -kernel /home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel -initrd /home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/initrd -append panic=1 console=ttyAMA0 earlyprintk=pl011,0x9000000 ignore_loglevel efi-rtc=noprobe udevtimeout=6000 udev.event-timeout=6000 no_timer_check lpj=500000 acpi=off printk.time=1 cgroup_disable=memory root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color -device virtio-scsi-device,id=scsi0 -device virtio-serial-device,id=virtio-serial0 -usb -drive file=/home/rjones/d/libguestfs/tmp/libguestfs4GxfQ9/scratch.1,if=none,id=drive-scsi0-0-0-0,format=raw,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive file=/home/rjones/d/libguestfs/tmp/libguestfs4GxfQ9/overlay2,if=none,id=drive-scsi0-0-1-0,format=qcow2,cache=unsafe -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-scsi0-0-1-0,id=scsi0-0-1-0 -serial unix:/home/rjones/d/libguestfs/tmp/libguestfs4GxfQ9/console.sock -chardev socket,id=charchannel0,path=/home/rjones/d/libguestfs/tmp/libguestfs4GxfQ9/guestfsd.sock -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0 -msg timestamp=on

There are no kernel messages about the disks, they just are not seen.

Worked with kernel 3.16 so I suspect this could be a kernel bug rather than a
qemu bug, but I've no idea where to report those.

[Peter Maydell (pmaydell)]

On 21 October 2014 20:07, Richard Jones <email address hidden> wrote:
> Public bug reported:
>
> kernel-3.18.0-0.rc1.git0.1.rwmj5.fc22.aarch64 (3.18 rc1 + some hardware enablement)
> qemu from git today
>
> When I create a guest with virtio-scsi disks, they don't show up inside the guest.
> Literally after the virtio_mmio.ko and virtio_scsi.ko modules are loaded, there are
> no messages about disks, and of course nothing else works.

> There are no kernel messages about the disks, they just are not seen.
>
> Worked with kernel 3.16 so I suspect this could be a kernel bug rather than a
> qemu bug, but I've no idea where to report those.

Yeah, "regressed with this newer kernel" sounds more like a kernel
bug than a QEMU bug to me, especially if all the other virt devices
still work.

-- PMM

[Richard Jones (rjones-redhat)]

I have now also tried virtio-blk and that doesn't work either. Same symptoms: no log messages at all (even with ignore_loglevel), and no disks appear.

> Yeah, "regressed with this newer kernel" sounds more like a kernel
> bug than a QEMU bug to me, especially if all the other virt devices
> still work.

It seems like no virtio drivers work, but it's very hard to tell when your guest has no disks and therefore no operating system. How can I debug this further?

[Peter Maydell (pmaydell)]

On 21 October 2014 22:15, Richard Jones <email address hidden> wrote:
> I have now also tried virtio-blk and that doesn't work either. Same
> symptoms: no log messages at all (even with ignore_loglevel), and no
> disks appear.
>
>> Yeah, "regressed with this newer kernel" sounds more like a kernel
>> bug than a QEMU bug to me, especially if all the other virt devices
>> still work.
>
> It seems like no virtio drivers work, but it's very hard to tell when
> your guest has no disks and therefore no operating system. How can I
> debug this further?

Oh. I assumed from the fact your commandline had other virtio
devices and you were only complaining about virtio-scsi that it
was a single-backend issue.

Suggestions:
 * bisect the kernel to find out when it broke
 * add a bunch of debug printks to the kernel to find out why
   it's not finding the disks. The logging here is terrible IMHO:
   the kernel usually detects a specific problem and then throws
   all this info away in favour of just silently deciding there's
   no hardware there

thanks
-- PMM

[Peter Maydell (pmaydell)]

On 21 October 2014 22:33, Peter Maydell <email address hidden> wrote:
> Suggestions:

...you might also try asking on <email address hidden>, which
is where the KVM/ARM kernel devs hang out, to see if somebody
else has seen this.

-- PMM

[Richard Jones (rjones-redhat)]

Finally finished the git bisect (of the guest kernel, not qemu):

421520ba98290a73b35b7644e877a48f18e06004 is the first bad commit
commit 421520ba98290a73b35b7644e877a48f18e06004
Author: Yalin Wang <email address hidden>
Date: Fri Sep 26 03:07:09 2014 +0100

    ARM: 8167/1: extend the reserved memory for initrd to be page aligned

    This patch extends the start and end address of initrd to be page aligned,
    so that we can free all memory including the un-page aligned head or tail
    page of initrd, if the start or end address of initrd are not page
    aligned, the page can't be freed by free_initrd_mem() function.

    Signed-off-by: Yalin Wang <email address hidden>
    Acked-by: Catalin Marinas <email address hidden>
    Signed-off-by: Russell King <email address hidden>

:040000 040000 23bd54d302533c173a4ae592969dd2868794e9ed f1833b44ee7a389902f6f9d2fb55f4b89ba0de16 M arch

Now might be a good time to mention that Fedora has very recently switched to using 64k pages.

I'll continue this on the mailing list you suggested.

[Richard Jones (rjones-redhat)]

Still happening with latest upstream kernel. It seems to involve using the -initrd option at all, with any cpio file, even a tiny one. More results posted here:

https://lists.cs.columbia.edu/pipermail/kvmarm/2014-December/012557.html

[Richard Jones (rjones-redhat)]

Finally found the problem, patch posted:
https://lists.gnu.org/archive/html/qemu-devel/2014-12/msg00034.html

[Christoffer Dall (cdall)]

I was just about to get to testing this stuff, but thanks for working
through it on your own, and apologies I didn't get to it before.

Cc'ing Marc so he's aware of the progress.

-Christoffer

On Mon, Dec 1, 2014 at 3:46 PM, Richard Jones <email address hidden> wrote:
> Finally found the problem, patch posted:
> https://lists.gnu.org/archive/html/qemu-devel/2014-12/msg00034.html
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1383857
>
> Title:
> aarch64: virtio disks don't show up in guest (neither blk nor scsi)
>
> Status in QEMU:
> New
>
> Bug description:
> kernel-3.18.0-0.rc1.git0.1.rwmj5.fc22.aarch64 (3.18 rc1 + some hardware enablement)
> qemu from git today
>
> When I create a guest with virtio-scsi disks, they don't show up inside the guest.
> Literally after the virtio_mmio.ko and virtio_scsi.ko modules are loaded, there are
> no messages about disks, and of course nothing else works.
>
> Really long command line (generated by libvirt):
>
> HOME=/home/rjones USER=rjones LOGNAME=rjones QEMU_AUDIO_DRV=none
> TMPDIR=/home/rjones/d/libguestfs/tmp
> /home/rjones/d/qemu/aarch64-softmmu/qemu-system-aarch64 -name guestfs-
> oqv29um3jp03kpjf -S -machine virt,accel=tcg,usb=off -cpu cortex-a57 -m
> 500 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid
> a5f1a15d-2bc7-46df-9974-1d1f643b2449 -nographic -no-user-config
> -nodefaults -chardev
> socket,id=charmonitor,path=/home/rjones/.config/libvirt/qemu/lib
> /guestfs-oqv29um3jp03kpjf.monitor,server,nowait -mon
> chardev=charmonitor,id=monitor,mode=control -rtc
> base=utc,driftfix=slew -no-reboot -boot strict=on -kernel
> /home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel -initrd
> /home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/initrd -append
> panic=1 console=ttyAMA0 earlyprintk=pl011,0x9000000 ignore_loglevel
> efi-rtc=noprobe udevtimeout=6000 udev.event-timeout=6000
> no_timer_check lpj=500000 acpi=off printk.time=1 cgroup_disable=memory
> root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm-256color -device
> virtio-scsi-device,id=scsi0 -device virtio-serial-device,id=virtio-
> serial0 -usb -drive
> file=/home/rjones/d/libguestfs/tmp/libguestfs4GxfQ9/scratch.1,if=none,id
> =drive-scsi0-0-0-0,format=raw,cache=unsafe -device scsi-
> hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-
> scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -drive
> file=/home/rjones/d/libguestfs/tmp/libguestfs4GxfQ9/overlay2,if=none,id
> =drive-scsi0-0-1-0,format=qcow2,cache=unsafe -device scsi-
> hd,bus=scsi0.0,channel=0,scsi-id=1,lun=0,drive=drive-
> scsi0-0-1-0,id=scsi0-0-1-0 -serial
> unix:/home/rjones/d/libguestfs/tmp/libguestfs4GxfQ9/console.sock
> -chardev
> socket,id=charchannel0,path=/home/rjones/d/libguestfs/tmp/libguestfs4GxfQ9/guestfsd.sock
> -device virtserialport,bus=virtio-
> serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0
> -msg timestamp=on
>
> There are no kernel messages about the disks, they just are not seen.
>
> Worked with kernel 3.16 so I suspect this could be a kernel bug rather t...

[Peter Maydell (pmaydell)]

Richard -- I assume we fixed this one way or another, right?

[Richard Jones (rjones-redhat)]

Oh yes, long fixed.