[gate] devstack-plugin-nfs-tempest-full-ussuri job failing

[Rajat Dhasmana]

Currently the test test_snapshot_create_delete_with_volume_in_use is failing on devstack-plugin-nfs gate with the following nova error[1]. n-cpu logs[2]

[1] ERROR nova.virt.libvirt.driver [req-9e847496-e25f-475d-8cdf-0f12ae2bbfe2 req-b8e1cde4-b38f-4c75-98d3-f976153794c3 service nova] [instance: 6aff2151-c8a3-42aa-95a0-39e10267e782] Error occurred during volume_snapshot_delete, sending error status to Cinder.: libvirt.libvirtError: internal error: qemu block name 'json:{"backing": {"driver": "raw", "file": {"driver": "file", "filename": "/opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-3c98d95e-a839-4eaf-ba76-325ed3a3a946"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "/opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-3c98d95e-a839-4eaf-ba76-325ed3a3a946.adb8bb67-dfb5-4ac7-8e95-3643f8a0e8ed"}}' doesn't match expected '/opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-3c98d95e-a839-4eaf-ba76-325ed3a3a946.adb8bb67-dfb5-4ac7-8e95-3643f8a0e8ed'

[2] https://c21a2e86899d4e33d85a-8ff305a045ea105f4906486adfb49654.ssl.cf1.rackcdn.com/763516/2/check/devstack-plugin-nfs-tempest-full-ussuri/e9e69b9/controller/logs/screen-n-cpu.txt

[Luigi Toscano (ltoscano)]

Additional note: the same test generally passes on train, master and victoria.

[Lee Yarwood (lyarwood)]

A better log of the error is:

Feb 23 10:53:27.741085 ubuntu-bionic-limestone-regionone-0023148232 nova-compute[25050]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 707, in blockCommit
Feb 23 10:53:27.741085 ubuntu-bionic-limestone-regionone-0023148232 nova-compute[25050]: ERROR oslo_messaging.rpc.server if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self)
Feb 23 10:53:27.747609 ubuntu-bionic-limestone-regionone-0023148232 nova-compute[25050]: ERROR oslo_messaging.rpc.server libvirt.libvirtError: internal error: qemu block name 'json:{"backing": {"driver": "raw", "file": {"driver": "file", "filename": "/opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-3c98d95e-a839-4eaf-ba76-325ed3a3a946"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "/opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-3c98d95e-a839-4eaf-ba76-325ed3a3a946.adb8bb67-dfb5-4ac7-8e95-3643f8a0e8ed"}}' doesn't match expected '/opt/stack/data/nova/mnt/896fb15da6036b68a917322e72ebfe57/volume-3c98d95e-a839-4eaf-ba76-325ed3a3a946.adb8bb67-dfb5-4ac7-8e95-3643f8a0e8ed'

[Lee Yarwood (lyarwood)]

Apologies that's the same I just couldn't parse it in launchpad. I've never seen this before tbh so I'm not sure why it's only impacting ussuri.

[Luigi Toscano (ltoscano)]

Ping - any news on this? It prevents any changes from being merged in devstack-plugin-nfs.