volume related tests sometimes fail with "Could not open '/dev/sdb': Permission denied" on stable/(ussuri|train|stein)

[melanie witt]

Seen twice today on changes I've reviewed and rechecking on stable/train and stable/stein.

Tests working with volumes sometimes fail because of "Could not open '/dev/sdb': Permission denied" from libvirt. Here's a couple of examples.

tempest.api.compute.admin.test_volume_swap.TestVolumeSwap.test_volume_swap "Failure rebasing volume /dev/sdb on vdb.: libvirt.libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/sdb': Permission denied"

https://zuul.opendev.org/t/openstack/build/c9515f1edc774e74a84e91a7dee50074/log/compute1/logs/screen-n-cpu.txt#2138

and

2021-03-17 23:24:26.792 | [ERROR] /opt/stack/new/grenade/projects/70_cinder/resources.sh:172 Volume failed to become in-use [1]

Mar 17 23:22:54.242764 ubuntu-bionic-ovh-gra1-0023547430 nova-compute[27357]: ERROR nova.virt.block_device [None req-bdf9e703-73b4-49b5-8755-4b6798749af0 cinder_grenade cinder_grenade] [instance: 3496ab76-6fd2-477d-87fc-f0c945687d29] Driver failed to attach volume d73f3349-c514-4082-8bd1-17b64ad2970a at /dev/vdb: libvirt.libvirtError: operation failed: 'serial' is deprecated, please use the corresponding option of '-device' instead
Mar 17 23:22:54.242764 ubuntu-bionic-ovh-gra1-0023547430 nova-compute[27357]: Could not open '/dev/sdb': Permission denied [2]

Logstash query:

all failures, check and gate, stable/ussuri stable/train stable/stein only

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Could%20not%20open%20'%2Fdev%2Fsdb'%3A%20Permission%20denied%5C%22%20AND%20(tags%3Ascreen-n-cpu.txt%20OR%20tags%3Aconsole%20OR%20tags%3Ascreen-c-vol.txt)&from=7d

[1] https://zuul.opendev.org/t/openstack/build/e1050a661fec47b49d6d7babc89d71d3/log/logs/grenade.sh.txt#41025
[2] https://zuul.opendev.org/t/openstack/build/e1050a661fec47b49d6d7babc89d71d3/log/logs/screen-n-cpu.txt#3189