Only one NVMe device is usable in Windows (10) guest

[Tom Yan]

Full command: qemu-system-x86_64 -enable-kvm -cpu host -smp cores=4 -m 4G -net bridge -net nic -full-screen -drive file=ovmf_x64.bin,format=raw,if=pflash -drive file=disks/win16_ide.img,format=raw,cache=none,aio=native -drive file=disks/one.img,if=none,format=qcow2,id=one -drive file=disks/two.img,if=none,format=qcow2,id=two -device nvme,drive=one,serial=E86C3CFC43518D6F -device nvme,drive=two,serial=2BDAC262CF831698

QEMU version: 2.5.0

Kernel: 4.5.1 (Arch Linux)

When there are two NVMe devices specified, only the second one will be usable in Windows. The following error is shown under "Device status" of the failed NVMe controller in Device Manager:

"This device cannot start. (Code 10)

The I/O device is configured incorrectly or the configuration parameters to the driver are incorrect."

The only thing seems suspicious to me is that the nvme emulation in qemu does not have WWN/EUI-64 set for the devices, though I have no idea at all whether that is mandatory:

"C:\Windows\system32>sg_vpd -i PD1
Device Identification VPD page:
  Addressed logical unit:
    designator type: SCSI name string, code set: UTF-8
      SCSI name string:
      8086QEMU NVMe Ctrl 00012BDAC262CF831698

C:\Windows\system32>sg_vpd -p sn PD1
Unit serial number VPD page:
  Unit serial number: 0000_0000_0000_0000."

[Tom Yan (tom-ty89)]

You can see from the "SCSI name string" that the working NVMe device is the second one specified in the command.

[Tom Yan (tom-ty89)]

"if=virtio" (which similarly has one controller per drive and has each controller occupies a pci slot as the nvme emulation) works fine:

qemu-system-x86_64 -enable-kvm -cpu host -smp cores=4 -m 4G -net bridge -net nic -full-screen -drive file=ovmf_x64.bin,format=raw,if=pflash -drive file=disks/win16_ide.img,format=raw,cache=none,aio=native -drive file=disks/one.img,if=virtio,format=qcow2,id=one -drive file=disks/two.img,if=virtio,format=qcow2,id=two

[Tom Yan (tom-ty89)]

Apparently it works fine in Linux though:

qemu-system-x86_64 -enable-kvm -cpu host -smp cores=4 -m 4G -net bridge -net nic -full-screen -drive file=ovmf_x64.bin,format=raw,if=pflash -drive file=Downloads/archlinux-2016.04.01-dual.iso,media=cdrom -drive file=disks/one.img,if=none,format=qcow2,id=one -drive file=disks/two.img,if=none,format=qcow2,id=two -device nvme,drive=one,serial=E86C3CFC43518D6F -device nvme,drive=two,serial=2BDAC262CF831698

[Tom Yan (tom-ty89)]

I also tested WIndows installation ISOs instead as well.

Windows 10 Enterprise 10586:

qemu-system-x86_64 -enable-kvm -cpu host -smp cores=4 -m 4G -net bridge -net nic -full-screen -drive file=ovmf_x64.bin,format=raw,if=pflash -drive file=Downloads/10586.0.151029-1700.TH2_RELEASE_CLIENTENTERPRISEEVAL_OEMRET_X64FRE_EN-US.ISO,media=cdrom -drive file=disks/one.img,if=none,format=qcow2,id=one -drive file=disks/two.img,if=none,format=qcow2,id=two -device nvme,drive=one,serial=E86C3CFC43518D6F -device nvme,drive=two,serial=2BDAC262CF831698

Windows Server 2016 Technical Preview 5:

qemu-system-x86_64 -enable-kvm -cpu host -smp cores=4 -m 4G -net bridge -net nic -full-screen -drive file=ovmf_x64.bin,format=raw,if=pflash -drive file=Downloads/14300.1000.160324-1723.RS1_RELEASE_SVC_SERVER_OEMRET_X64FRE_EN-US.ISO,media=cdrom -drive file=disks/one.img,if=none,format=qcow2,id=one -drive file=disks/two.img,if=none,format=qcow2,id=two -device nvme,drive=one,serial=E86C3CFC43518D6F -device nvme,drive=two,serial=2BDAC262CF831698

Both of them shows only one of the NVMe drives.

I also tested with two versions of OVMF, which are "18419" (https://github.com/tianocore/edk2/commit/ddd097e33f6e6829dc0413820e9971f3bf025f87) and "18455" (https://github.com/tianocore/edk2/commit/59844e126614fc8275aab083fafa5818cde0901c).

[Stefan Hajnoczi (stefanha)]

On Thu, Apr 28, 2016 at 05:44:21PM -0000, Tom Yan wrote:

CCing Keith Busch <email address hidden>, maintainer of QEMU NVMe.
Maybe he has an idea.

> Public bug reported:
>
> Full command: qemu-system-x86_64 -enable-kvm -cpu host -smp cores=4 -m
> 4G -net bridge -net nic -full-screen -drive
> file=ovmf_x64.bin,format=raw,if=pflash -drive
> file=disks/win16_ide.img,format=raw,cache=none,aio=native -drive
> file=disks/one.img,if=none,format=qcow2,id=one -drive
> file=disks/two.img,if=none,format=qcow2,id=two -device
> nvme,drive=one,serial=E86C3CFC43518D6F -device
> nvme,drive=two,serial=2BDAC262CF831698
>
> QEMU version: 2.5.0
>
> Kernel: 4.5.1 (Arch Linux)
>
> When there are two NVMe devices specified, only the second one will be
> usable in Windows. The following error is shown under "Device status" of
> the failed NVMe controller in Device Manager:
>
> "This device cannot start. (Code 10)
>
> The I/O device is configured incorrectly or the configuration parameters
> to the driver are incorrect."
>
> The only thing seems suspicious to me is that the nvme emulation in qemu
> does not have WWN/EUI-64 set for the devices, though I have no idea at
> all whether that is mandatory:
>
> "C:\Windows\system32>sg_vpd -i PD1
> Device Identification VPD page:
> Addressed logical unit:
> designator type: SCSI name string, code set: UTF-8
> SCSI name string:
> 8086QEMU NVMe Ctrl 00012BDAC262CF831698
>
> C:\Windows\system32>sg_vpd -p sn PD1
> Unit serial number VPD page:
> Unit serial number: 0000_0000_0000_0000."
>
> ** Affects: qemu
> Importance: Undecided
> Status: New
>
> ** Attachment added: "Screenshot of Device Manager and the error."
> https://bugs.launchpad.net/bugs/1576347/+attachment/4650548/+files/01.PNG
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1576347
>
> Title:
> Only one NVMe device is usable in Windows (10) guest
>
> Status in QEMU:
> New
>
> Bug description:
> Full command: qemu-system-x86_64 -enable-kvm -cpu host -smp cores=4 -m
> 4G -net bridge -net nic -full-screen -drive
> file=ovmf_x64.bin,format=raw,if=pflash -drive
> file=disks/win16_ide.img,format=raw,cache=none,aio=native -drive
> file=disks/one.img,if=none,format=qcow2,id=one -drive
> file=disks/two.img,if=none,format=qcow2,id=two -device
> nvme,drive=one,serial=E86C3CFC43518D6F -device
> nvme,drive=two,serial=2BDAC262CF831698
>
> QEMU version: 2.5.0
>
> Kernel: 4.5.1 (Arch Linux)
>
> When there are two NVMe devices specified, only the second one will be
> usable in Windows. The following error is shown under "Device status"
> of the failed NVMe controller in Device Manager:
>
> "This device cannot start. (Code 10)
>
> The I/O device is configured incorrectly or the configuration
> parameters to the driver are incorrect."
>
> The only thing seems suspicious to me is that the nvme emulation in
> qemu does not have WWN/EUI-64 set for the devices, though I have no
> idea at all whether that is mandatory:
>
> "C:\Windows\system32>sg_vpd -i PD1
>...

[Keith Busch (keith-busch)]

On Fri, Apr 29, 2016 at 10:10:39AM +0100, Stefan Hajnoczi wrote:
> On Thu, Apr 28, 2016 at 05:44:21PM -0000, Tom Yan wrote:
>
> CCing Keith Busch <email address hidden>, maintainer of QEMU NVMe.
> Maybe he has an idea.

Thanks for the report. Sounds like a Windows specific issue as I have no
problem with multiple nvme drives on my dev machines:

[Host]
# uname -r
4.6.0-rc5+

# qemu-system-x86_64 --version
QEMU emulator version 2.5.50, Copyright (c) 2003-2008 Fabrice Bellard

# qemu-system-x86_64 -m 4096 -smp 4 -enable-kvm debian.img \
    -drive file=nvme.1.img,if=none,id=one -device nvme,drive=one,serial=foo \
    -drive file=nvme.2.img,if=none,id=two -device nvme,drive=two,serial=bar

[Guest]
# uname -r
4.5.0

# ls /dev/nvme*
/dev/nvme0 /dev/nvme0n1 /dev/nvme1 /dev/nvme1n1

# nvme id-ctrl /dev/nvme0 | grep sn
sn : foo

# nvme id-ctrl /dev/nvme1 | grep sn
sn : bar

> > When there are two NVMe devices specified, only the second one will be
> > usable in Windows. The following error is shown under "Device status" of
> > the failed NVMe controller in Device Manager:
> >
> > "This device cannot start. (Code 10)
> >
> > The I/O device is configured incorrectly or the configuration parameters
> > to the driver are incorrect."
> >
> > The only thing seems suspicious to me is that the nvme emulation in qemu
> > does not have WWN/EUI-64 set for the devices, though I have no idea at
> > all whether that is mandatory:

These are not mandatory. They were only introduced in the 1.1 and 1.2 versions
of the NVMe spec, though we only cared to emulate the 1.0 portions rather than
provide a full featured NVMe controller.

That said, there needs to be care in the host OS to provide an appropriate
translation IF it is using a SCSI stack to talk to NVMe. Linux doesn't care,
but Windows does.

> > "C:\Windows\system32>sg_vpd -i PD1
> > Device Identification VPD page:
> > Addressed logical unit:
> > designator type: SCSI name string, code set: UTF-8
> > SCSI name string:
> > 8086QEMU NVMe Ctrl 00012BDAC262CF831698

The above looks reasonable for your second controller that had serial
2BDAC262CF831698.

> > C:\Windows\system32>sg_vpd -p sn PD1
> > Unit serial number VPD page:
> > Unit serial number: 0000_0000_0000_0000."

This doesn't look like a very good SCSI-NVMe translation and possibly
suspicious. But I don't know the first thing about windows; does it care about
unique unit serial numbers in order to surface a "SCSI" disk?

[Keith Busch (keith-busch)]

> > C:\Windows\system32>sg_vpd -p sn PD1
> > Unit serial number VPD page:
> > Unit serial number: 0000_0000_0000_0000."

I checked your serial number against the SNT refernce on nvmexpress.org and
it's definitely the wrong translation, so that has to be a guest OS driver bug
(Linux has the right translation if interested, but it's use is deprecated).

I pinged some Windows comrades to see if a potential serial conflict prevents
both disks from surfacing.

I'm surprised to see this bad translation as I know of folks successfully
testing multiple nvme drives in various versions of Windows with both the OFA
and Microsoft drivers. An emulated NVMe is no different than real h/w for
namespace identification from the host's perspective.

[Tom Yan (tom-ty89)]

The problem still exists with qemu 2.7.0 and Windows 10 RS1. Btw I just did one more test, that I disable the working nvme controller in Device Manager to see if it makes the non-working one comes alive, but it does not, even if I reboot the virtual machine.

For the record, the serial number translation is not fixed in RS1 either.

[Sergey Blagodurov (blagodurov)]

Trying it on Windows 10 guest and QEMU 2.8.0 has the same issue. However, I noticed that:
  Supplying 1 NVMe drive -> Win10 sees it.
  Supplying 2 NVMe drives -> Win10 sees only one of them.
  Supplying 3 NVMe drives -> Win10 sees only two of them.
So I still have been able to create a ReFS mirrored storage space with two NVMe disks under QEMU, I just had to pass three drives instead of two:

qemu-system-x86_64 -enable-kvm <...> -drive file=/media/ssd/NVMe_drvA.qcow2,id=diskNVMeA,format=qcow2,if=none -drive file=/media/ssd/NVMe_drvB.qcow2,id=diskNVMeB,format=qcow2,if=none -drive file=/media/ssd/NVMe_drvC.qcow2,id=diskNVMeC,format=qcow2,if=none -device nvme,drive=diskNVMeA,serial=foo -device nvme,drive=diskNVMeB,serial=foo -device nvme,drive=diskNVMeC,serial=foo

Hope this helps,
--Sergey

[Marshall Porter (meoporter)]

Hello, I can also confirm that if you have a series of NVMe devices, at first when just the OVMF boot menu has loaded, using qemu with a -monitor and typing 'info pci', it appears that all are given valid bus addresses.

However, once the Windows 10 x64 installer iso loads, it seems that every other NVMe device stays initialized correctly, but the ones in-between switch to a 0xfffff... bus address. So, for example, with 6 devices, numbers 1,3,5 will get initialized and recognized by the Windows installer, and 2,4,6 will not. Not sure if this clue helps identify this a qemu or Windows issue or not, but I thought it was worth noting.

Also, I found that an alternative way to getting two or more NVMe devices successfully recognized by Windows is to use multiple root PCIe devices like so:

qemu-system-x86_64 \
-enable-kvm \
-machine pc-q35-2.11,accel=kvm \
-nodefaults \
...
-device pcie-root-port,id=pcie_r0,bus=pcie.0,chassis=0,slot=0 \
-device nvme,drive=disk0,serial=serial0,bus=pcie_r0 \
-drive id=disk0,if=none,media=disk,cache=none,aio=native,format=raw,discard=unmap,file=/opt/os.raw \
-device pcie-root-port,id=pcie_r1,bus=pcie.0,chassis=0,slot=1 \
-device nvme,drive=disk1,serial=serial1,bus=pcie_r1 \
-drive id=disk1,if=none,media=disk,cache=none,aio=native,format=raw,discard=unmap,file=/opt/data.raw \

[Thomas Huth (th-huth)]

The QEMU project is currently considering to move its bug tracking to another system. For this we need to know which bugs are still valid and which could be closed already. Thus we are setting older bugs to "Incomplete" now.
If you still think this bug report here is valid, then please switch the state back to "New" within the next 60 days, otherwise this report will be marked as "Expired". Or mark it as "Fix Released" if the problem has been solved with a newer version of QEMU already. Thank you and sorry for the inconvenience.

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]