tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume failed when the self.device(vdb)value was in use(config_drive vfat in my case)

[Paras Babbar]

The issue is basically in this line https://github.com/openstack/tempest/blob/master/tempest/api/compute/volumes/test_attach_volume.py#L107
So if self.drive i.e volume device name in tempest.conf is not mentioned in tempest.conf then this function as per Note below takes the auto assign value which is independent of checking of it's in use or not. My suggestion is to fetch the value of disk from the json response generated after created/attached volume instance and use this device value to check if the device is properly removed or not rather than relying on self.drive which is based on assumptions.

More Granular details:

def test_attach_detach_volume(self):

''''''''''''

        # NOTE: As of the 12.0.0 Liberty release, the Nova libvirt driver
        # no longer honors a user-supplied device name, in that case
        # CONF.compute.volume_device_name must be set the equal value as
        # the libvirt auto-assigned one
        attachment = self.attach_volume(server, volume,
                                        device=('/dev/%s' % self.device))
=> This self.device is randomly coming may be from libvirt(as per the comment) if we don't specify the volume_device_name in tempest.conf and in my case it picked vdb and vdb was already in use by config drive (vfat) thus my test got failed in assertNotIn function.

        self.servers_client.stop_server(server['id'])
        waiters.wait_for_server_status(self.servers_client, server['id'],
                                       'SHUTOFF')

        self.servers_client.start_server(server['id'])
        waiters.wait_for_server_status(self.servers_client, server['id'],
                                       'ACTIVE')

        if CONF.validation.run_validation:
            disks = linux_client.get_disks()
            device_name_to_match = '\n' + self.device + ' '
            self.assertIn(device_name_to_match, disks)

        self.servers_client.detach_volume(server['id'], attachment['volumeId'])(This will detach
        waiters.wait_for_volume_resource_status(
            self.volumes_client, attachment['volumeId'], 'available')

        self.servers_client.stop_server(server['id'])
        waiters.wait_for_server_status(self.servers_client, server['id'],
                                       'SHUTOFF')

        self.servers_client.start_server(server['id'])
        waiters.wait_for_server_status(self.servers_client, server['id'],
                                       'ACTIVE')

        if CONF.validation.run_validation:
            disks = linux_client.get_disks()
            self.assertNotIn(device_name_to_match, disks) => This function check vdb value is in the instance or not but vdb was used in config drive which has not been detached , this we see the issue

********output*****************************************

>>print(device_name_to_match)
vdb => self.device value took from libvirt
try to attach the volume and use vdb but actually vdb was already in use by meta data volume so vdc got attached as below
>>print(disks) (disk of instance)
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 1073741824 0 disk #root volume
vdb 253:16 0 67108864 0 disk #config drive(vfat)
vdc 253:32 0 1073741824 0 disk #new volume attached

>>print(disks) #this one is after the vdc got detached (disk that got attached)
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 1073741824 0 disk
vdb 253:16 0 67108864 0 disk
#self.assertNotIn(device_name_to_match, disks) this will face as it is matching the wrong volume attached

Steps to reproduce
1. deploy an instance with one already added vdb volume attached(Like I have used config drive format vfat that already took vdb volume)
2. run this testcase tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume
3. This will fail in assertNotIn mismatch error like below as it tries to validate that vdb is detached or not which is set by libvirt assumption

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/volumes/test_attach_volume.py", line 124, in test_attach_detach_volume
    self.assertNotIn(device_name_to_match, disks)
  File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 455, in assertNotIn
    self.assertThat(haystack, matcher, message)
  File "/usr/lib/python3.6/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT\nvda 253:0 0 1073741824 0 disk \nvdb 253:16 0 67108864 0 disk ' matches Contains('\nvdb ')

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/694636

[Ghanshyam Mann (ghanshyammann)]

do you get the failure on other tests also which use CONF.compute.volume_device_name for mounting the volume [1] or checking the disk name [2]

[1] https://github.com/openstack/tempest/blob/887f99a288c1d2bef45356c916b317d37828e6ac/tempest/scenario/test_stamp_pattern.py#L101

[2] https://github.com/openstack/tempest/blob/67e07786189661151c73263a13a523865770bdde/tempest/scenario/test_minimum_basic.py#L71

[Paras Babbar (pbabbar)]

I just executed them and yes we have issues here too:

[1]https://github.com/openstack/tempest/blob/887f99a288c1d2bef45356c916b317d37828e6ac/tempest/scenario/test_stamp_pattern.py#L101
[2]https://github.com/openstack/tempest/blob/887f99a288c1d2bef45356c916b317d37828e6ac/tempest/scenario/manager.py#L756

So In the env when the instance got deployed it used vdb as config_Drive and the testcase tries to create FS on vdb and hence it failed . We tried this on without config_Drive as well and it worked fine.
(overcloud) [stack@undercloud-0 tempest-dir]$ tempest run --regex tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern
/usr/lib/python2.7/site-packages/paramiko/rsakey.py:119: DeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
  algorithm=hashes.SHA1(),
{0} tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern [97.043124s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 88, in wrapper
        return f(*func_args, **func_kwargs)
      File "/usr/lib/python2.7/site-packages/tempest/scenario/test_stamp_pattern.py", line 104, in test_stamp_pattern
        server=server)
      File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 623, in create_timestamp
        ssh_client.make_fs(dev_name)
      File "/usr/lib/python2.7/site-packages/tempest/common/utils/linux/remote_client.py", line 161, in make_fs
        self.exec_command(cmd_mkfs)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 33, in wrapper
        return function(self, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 108, in exec_command
        return self.ssh_client.exec_command(cmd)
      File "/usr/lib/python2.7/site-packages/tempest/lib/common/ssh.py", line 202, in exec_command
        stderr=err_data, stdout=out_data)
    tempest.lib.exceptions.SSHExecCommandFailed: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; sudo mke2fs -t ext4 /dev/vdb', exit status: 1, stderr:
    sudo: mke2fs: command not found

    stdout:

Captured pythonlogging:
~~~~~~~~~~~~~~~~~~~~~~~
    2019-11-22 14:46:06,381 31966 INFO [tempest.lib.common.rest_client] Request (TestStampPattern:test_stamp_pattern): 200 POST http://10.0.0.106:8774/v2.1/os-keypairs 0.578s
    2019-11-22 14:46:06,381 31966 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
            Body: {"keypair": {"name": "tempest-TestStampPattern-974168698"}}
        Response - Headers: {'status': '200', u'content-length': '2313', 'content-location': 'http://10.0.0.106:8774/v2.1/os-keypairs', u'x-compute-request-id': 'req-bb27cd2f-c339-418e-ba59-86c87eb34a05', u'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', u'server': 'Apache', u'openstack-api-version': 'compute 2.1', u'connection': 'close', u'x-openstack-nova-api-version': '2.1', u'date': 'Fri, 22 Nov ...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694636
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=f11856505fcfb85f1ad4fde4bf68078c63be787a
Submitter: Zuul
Branch: master

commit f11856505fcfb85f1ad4fde4bf68078c63be787a
Author: Paras Babbar <email address hidden>
Date: Fri Nov 15 16:55:45 2019 -0500

    Fix test_attach_detach_volume test case to not rely on user/libvirt supplied disk name

    This patch removes user provided and libvirt assigned
    value as that was based on assumption rather count the
    disk name before and after attach/detach based on output
    generated using 'lsblk' command in the running instance
    with the help of new method list_disks in client and
    existing util remote client for remote ssh.

    we observer this test case failing when the instance boot
    with config drive format "vfat" enabled with two volume
    vda and vdb volume disk name, so whatever test case boot with
    config drive as well as ephemeral disk was not be able to
    use this testcase.

    Change-Id: Id3e537bc55089ca5c35fa1f973dd47f2b11147ae
    Partial-Bug: #1852807

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/698598

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698598
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=4b45f9e54c8bbb89e55e9d7dc8542c074b63245d
Submitter: Zuul
Branch: master

commit 4b45f9e54c8bbb89e55e9d7dc8542c074b63245d
Author: Paras Babbar <email address hidden>
Date: Wed Dec 11 16:51:57 2019 -0500

    Fix test_stamp_pattern to remove dependency of user supplied volume name

    This patch removes CONF.compute.volume_device_name dependency and
    use modified list_disks method to get the newly attached
    device name and use that for mkfs rather than the user supplied
    device name which is based on assumtion.

    Also modfied test_attach_detach testcase and unit test
    for remote_client accordingly.

    Change-Id: I862d1ba55f45b09385057a51b0b52a851c19287e
    Closes-Bug: #1852807