We noticed that Ubuntu 16.04 guests running on Nutanix AHV stopped booting after they were upgraded to the latest kernel (4.4.0-127). Only guests with scsi mq enabled suffered from this problem. AHV is one of the few hypervisor products to offer multiqueue for virtio-scsi devices.
Upon further investigation, we could see that the kernel would hang during the scanning of scsi targets. More specifically, immediately after coming across a target without any luns present. That's the first time the kernel destroys a target (given it doesn't have luns). This could be confirmed with gdb (attached to qemu's gdbserver):
#0 0xffffffffc0045039 in ?? ()
#1 0xffff88022c753c98 in ?? ()
#2 0xffffffff815d1de6 in scsi_target_destroy (starget=0xffff88022ad62400)
at /build/linux-E14mqW/linux-4.4.0/drivers/scsi/scsi_scan.c:322
This shows the guest vCPU stuck on virtio-scsi's implementation of target_destroy. Despite lacking symbols, we managed to examine the virtio_scsi_target_state to see that the 'reqs' counter was invalid:
This drew our attention to the following patch which is exclusive to the Ubuntu kernel:
commit f1f609d8015e1d34d39458924dcd9524fccd4307
Author: Jay Vosburgh <email address hidden>
Date: Thu Apr 19 21:40:00 2018 +0200
In a nutshell, the patch spins on the target's 'reqs' counter waiting for the target to quiesce:
--- a/drivers/scsi/virtio_scsi.c
+++ b/drivers/scsi/virtio_scsi.c
@@ -785,6 +785,10 @@ static int virtscsi_target_alloc(struct scsi_target *starget)
static void virtscsi_target_destroy(struct scsi_target *starget)
{
struct virtio_scsi_target_state *tgt = starget->hostdata;
+
+ /* we can race with concurrent virtscsi_complete_cmd */
+ while (atomic_read(&tgt->reqs))
+ cpu_relax();
kfree(tgt);
}
Personally, I think this is a catastrophic way of waiting for a target to quiesce since virtscsi_target_destroy() is called with IRQs disabled from scsi_scan.c:scsi_target_destroy(). Devices which take a long time to quiesce during a target_destroy() could hog the CPU for relatively long periods of time.
Nevertheless, further study revealed that virtio-scsi itself is broken in a way that it doesn't increment the 'reqs' counter when submitting requests on MQ in certain conditions. That caused the counter to go to -1 (on the completion of the first request) and the CPU to hang indefinitely.
Signed-off-by: Felipe Franciosi <email address hidden>
Please consider this a urgent fix as all of our customers which use Ubuntu 16.04 and have MQ enabled for better performance will be affected by your latest update. Our workaround is to recommend that they disable SCSI MQ while you work on the issue.
We noticed that Ubuntu 16.04 guests running on Nutanix AHV stopped booting after they were upgraded to the latest kernel (4.4.0-127). Only guests with scsi mq enabled suffered from this problem. AHV is one of the few hypervisor products to offer multiqueue for virtio-scsi devices.
Upon further investigation, we could see that the kernel would hang during the scanning of scsi targets. More specifically, immediately after coming across a target without any luns present. That's the first time the kernel destroys a target (given it doesn't have luns). This could be confirmed with gdb (attached to qemu's gdbserver):
#0 0xffffffffc0045039 in ?? () 0xffff88022ad62 400) linux-E14mqW/ linux-4. 4.0/drivers/ scsi/scsi_ scan.c: 322
#1 0xffff88022c753c98 in ?? ()
#2 0xffffffff815d1de6 in scsi_target_destroy (starget=
at /build/
This shows the guest vCPU stuck on virtio-scsi's implementation of target_destroy. Despite lacking symbols, we managed to examine the virtio_ scsi_target_ state to see that the 'reqs' counter was invalid:
(gdb) p *(struct virtio_ scsi_target_ state *)starget->hostdata
$6 = {tgt_seq = {sequence = 0}, reqs = {counter = -1}, req_vq = 0xffff88022cbdd9e8}
(gdb)
This drew our attention to the following patch which is exclusive to the Ubuntu kernel: 4d39458924dcd95 24fccd4307
commit f1f609d8015e1d3
Author: Jay Vosburgh <email address hidden>
Date: Thu Apr 19 21:40:00 2018 +0200
In a nutshell, the patch spins on the target's 'reqs' counter waiting for the target to quiesce: scsi/virtio_ scsi.c scsi/virtio_ scsi.c target_ alloc(struct scsi_target *starget) target_ destroy( struct scsi_target *starget) scsi_target_ state *tgt = starget->hostdata; complete_ cmd */ read(&tgt- >reqs))
--- a/drivers/
+++ b/drivers/
@@ -785,6 +785,10 @@ static int virtscsi_
static void virtscsi_
{
struct virtio_
+
+ /* we can race with concurrent virtscsi_
+ while (atomic_
+ cpu_relax();
kfree(tgt);
}
Personally, I think this is a catastrophic way of waiting for a target to quiesce since virtscsi_ target_ destroy( ) is called with IRQs disabled from scsi_scan. c:scsi_ target_ destroy( ). Devices which take a long time to quiesce during a target_destroy() could hog the CPU for relatively long periods of time.
Nevertheless, further study revealed that virtio-scsi itself is broken in a way that it doesn't increment the 'reqs' counter when submitting requests on MQ in certain conditions. That caused the counter to go to -1 (on the completion of the first request) and the CPU to hang indefinitely.
The following patch fixes the issue:
--- new/linux- 4.4.0/drivers/ scsi/virtio_ scsi.c 2018-06-05 10:03:29.083428545 -0700 4.4.0/drivers/ scsi/virtio_ scsi.c 2018-06-04 10:23:07.000000000 -0700
scsi_ target( sc->device) ->hostdata;
+++ old/linux-
@@ -641,10 +641,9 @@
struct virtio_scsi_vq *req_vq;
- if (shost_ use_blk_ mq(sh)) { use_blk_ mq(sh))
req_vq = virtscsi_ pick_vq_ mq(vscsi, sc); inc(&tgt- >reqs);
req_vq = virtscsi_ pick_vq( vscsi, tgt);
+ if (shost_
- atomic_
- } else
+ else
return virtscsi_ queuecommand( vscsi, req_vq, sc);
Signed-off-by: Felipe Franciosi <email address hidden>
Please consider this a urgent fix as all of our customers which use Ubuntu 16.04 and have MQ enabled for better performance will be affected by your latest update. Our workaround is to recommend that they disable SCSI MQ while you work on the issue.
Best regards,
Felipe