tempest run --regex '\[.*\bsmoke\b.*\]' --blacklist-file=/home/jenkins/tempest/skip.txt not able to run smoke tests

[chandan kumar]

While using Tempest-15.0 on a Tripleo setup created using triple-quickstart. I tried to run smoke using following command it failed to execute.
Below is the output.

bash-4.2$ /usr/bin/tempest run --regex '\[.*\bsmoke\b.*\]' --blacklist-file=/home/jenkins/skip_file
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-500} \
${PYTHON:-python} -m subunit.run discover -t /usr/lib/python2.7/site-packages/tempest /usr/lib/python2.7/site-packages/tempest/test_discover --list
The test run didn't actually run any tests

bash-4.2$ cat /home/jenkins/skip_file
# rhbz1304933
tempest.api.telemetry.test_telemetry_notification_api
# rhbz1266947
tempest.api.identity.*v3
# rhbz1357667
tempest.api.data_processing
# lpbug 1605654
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps
# rhbz1378989
tempest.api.network.test_ports.PortsIpV6TestJSON
tempest.api.network.test_ports.PortsTestJSON
# rhbz1379829
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
#rhbz1380146
tempest.api.orchestration.stacks.test_resource_types.ResourceTypesTest
#rhbz1380145
tempest.api.identity.admin.v2.test_services.ServicesTestJSON
#rhbz1380187
tempest.api.network.test_routers.RoutersIpV6Test.test_add_multiple_router_interfaces
# TripleO does not deploy Barbican by default, which is needed for this test to pass
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup
tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks
#rhbz1411690
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic
tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_multiple_security_groups
# Cinder backup is not enabled by default in tripleo only in particular
# scenarios
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV1Test
tempest.api.volume.admin.test_volumes_backup.VolumesBackupsAdminV2Test
tempest.api.volume.test_volumes_backup.VolumesBackupsV1Test
tempest.api.volume.test_volumes_backup.VolumesBackupsV2Test
# Glance isn't populating metadata definition database LP1664995
tempest.api.image.v2.test_images_metadefs_namespaces.MetadataNamespacesTest.test_basic_metadata_definition_namespaces
tempest.api.image.v2.test_images_metadefs_resource_types.MetadataResourceTypesTest.test_basic_meta_def_resource_type_association
tempest.api.image.v2.test_images_metadefs_namespace_properties.MetadataNamespacePropertiesTest.test_basic_meta_def_namespace_property

bash-4.2$ tempest list-plugins
+------------------------+------------------------------------------------------------------------------+
| Name | EntryPoint |
+------------------------+------------------------------------------------------------------------------+
| neutron_tests | neutron.tests.tempest.plugin:NeutronTempestPlugin |
| zaqar_tests | zaqar.tests.tempest_plugin.plugin:ZaqarTempestPlugin |
| mistral_test | mistral_tempest_tests.plugin:MistralTempestPlugin |
| aodh_tests | aodh.tests.tempest.plugin:AodhTempestPlugin |
| cinder_tests | cinder.tests.tempest.plugin:CinderTempestPlugin |
| ceilometer_tests | ceilometer.tests.tempest.plugin:CeilometerTempestPlugin |
| ironic_tests | ironic_tempest_plugin.plugin:IronicTempestPlugin |
| keystone_tests | keystone_tempest_plugin.plugin:KeystoneTempestPlugin |
| heat_tests | heat_integrationtests.plugin:HeatTempestPlugin |
| gnocchi_tests | gnocchi.tempest.plugin:GnocchiTempestPlugin |
| ironic_inspector_tests | ironic_inspector.test.inspector_tempest_plugin.plugin:InspectorTempestPlugin |
+------------------------+------------------------------------------------------------------------------+
bash-4.2$

When i invoked the tests using "ostestr --regex '\[.*\bsmoke\b.*\]' --blacklist_file=/home/jenkins/skip_file " " /usr/bin/tempest run --regex '\[.*\bsmoke\b.*\]'" it works fine.

ostestr -l is listing the tests. I think there is some problem with --blacklist-file regex parsing.
Please have a look.

[chandan kumar (chkumar246)]

https://review.openstack.org/#/c/373428/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/440771

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ken'ichi Ohmichi (<email address hidden>) on branch: master
Review: https://review.openstack.org/440771

[Szymon Datko (sdatko)]

I proposed a patchset that fixes the issue in os-testr library: https://review.openstack.org/#/c/522768/

The problem is related to incorrect regex contructor present in os-testr and used when executing `tempest run` command.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-testr 1.1.0 release.

[Martin Kopec (mkopec)]

The issue got fixed in os-testr