Tempest test test_server_basic_ops is failed on rhel based compute nodes

[Sofiia Andriichenko]

Configuration:
Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation.
Storage Backends - Ceph RBD for volumes (Cinder), Ceph RBD for ephemeral volumes (Nova), Ceph RBD for images (Glance), Ceph RadosGW for objects (Swift API)

In tab Settings->Compute check Nova quotas
In tab Settings->OpenStack Services check enable Install Ceilometer and Aodh
In tab Networks->Other check enable Neutron DVR

Nodes: controller, 2 RHEL_compute_nodes, Ceph

Steps to reproduce:
    1. Deploy ISO in configuration see (Detailed bug description)
    2. Navigate to controller node
    3. Install git (use apt-get install git)
    4. Clone script to deploy rally + tempest
       # git clone https://github.com/obutenko/mos-rally-verify.git
    5. Navigate to the https://github.com/obutenko/mos-rally-verify
    6. Execute necessary steps to deploy tempest
    7. Tun test in debug mode
        #rally --debug verify start --regex tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops

Expected results:
Test is passed

Actual result:
Test is Failed
(see comments)

Reproducibility:
See attachment

Workaround:
Traceback (most recent call last):

  File "tempest/test.py", line 113, in wrapper

    return f(self, *func_args, **func_kwargs)

  File "tempest/scenario/test_server_basic_ops.py", line 137, in test_server_basic_ops

    self.verify_metadata_on_config_drive()

  File "tempest/scenario/test_server_basic_ops.py", line 107, in verify_metadata_on_config_drive

    dev_name = self.ssh_client.exec_command(cmd_blkid)

  File "tempest/common/utils/linux/remote_client.py", line 44, in exec_command

    return self.ssh_client.exec_command(cmd)

  File "tempest/lib/common/ssh.py", line 168, in exec_command

    stderr=err_data, stdout=out_data)

tempest.lib.exceptions.SSHExecCommandFailed: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; blkid -t LABEL=config-2 -o device', exit status: 2, stderr:

stdout:

snapshot: https://drive.google.com/a/mirantis.com/file/d/0B8hkiEm94sEtZjJLWk8xblNGN3c/view?usp=sharing

[Sofiia Andriichenko (sandriichenko)]

mos iso 495
Reproduce for:
Configuration:
Settings:
Compute - QEMU.
Network - Neutron with tunneling segmentation
Storage Backends - Ceph RBD for volumes (Cinder), Ceph RBD for ephemeral volumes (Nova), Ceph RBD for images (Glance), Ceph RadosGW for objects (Swift API)

In tab Settings->Compute check Nova quotas
In tab Settings->OpenStack Services check enable Install Ceilometer and Aodh
In tab Networks->Other check enable Neutron DVR

Nodes: controller, 2 RHEL_compute_nodes, Ceph

Trace:

Traceback (most recent call last):

  File "tempest/test.py", line 113, in wrapper

    return f(self, *func_args, **func_kwargs)

  File "tempest/scenario/test_server_basic_ops.py", line 137, in test_server_basic_ops

    self.verify_metadata_on_config_drive()

  File "tempest/scenario/test_server_basic_ops.py", line 107, in verify_metadata_on_config_drive

    dev_name = self.ssh_client.exec_command(cmd_blkid)

  File "tempest/common/utils/linux/remote_client.py", line 44, in exec_command

    return self.ssh_client.exec_command(cmd)

  File "tempest/lib/common/ssh.py", line 168, in exec_command

    stderr=err_data, stdout=out_data)

tempest.lib.exceptions.SSHExecCommandFailed: Command 'set -eu -o pipefail; PATH=$PATH:/sbin; blkid -t LABEL=config-2 -o device', exit status: 2, stderr:

stdout:

snapshot https://drive.google.com/a/mirantis.com/file/d/0B8hkiEm94sEtSkxnQkhQa2lET3M/view?usp=sharing

[Alexander Gubanov (ogubanov)]

Hi!
Pavel Kholkin and I have been digging into this issue and found that test failed in line
https://github.com/openstack/tempest/blob/master/tempest/scenario/test_server_basic_ops.py#L86
because of config_drive label in uppercase - "CONFIG-2" insted of "config-2".

Details: http://pastebin.com/wiPiBMyv

Seems, the case of lable name depends on "mkfs" tool in RHEL.
Here is similar bug https://bugs.launchpad.net/mos/+bug/1587960

All descriptions of config_drive format mention "config-2", not "CONFIG-2"

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/4/html/End_User_Guide/config-drive.html

So, it is tempest test problem, it should support both cases of lable.
I'll provide patchset to tempest.

[Alexander Gubanov (ogubanov)]

Here is patchset in tempest https://review.openstack.org/#/c/333933/
It should fix this issue.

[Timur Nurlygayanov (tnurlygayanov)]

Assigned to Aleksandr because he prepared the fix. This is an issue with Tempest code, added tag "non-release".

[Alexander Gubanov (ogubanov)]

Fix is merged.
Here is upstream bug https://bugs.launchpad.net/tempest/+bug/1596868