Cinder tests don't exit due to dependency conflicts

[Bas de Bruijne]

Rally tests with cinder run, but do not exit:

------------------------------------------------------------------------------
2022-02-03-21:47:27 root DEBUG 2022-02-03 21:47:27.170 9020 INFO rally.task.engine [-] Running workload:
2022-02-03-21:47:27 root DEBUG position = 0
2022-02-03-21:47:27 root DEBUG config = {
2022-02-03-21:47:27 root DEBUG "version": 2,
2022-02-03-21:47:27 root DEBUG "title": "A cropped version of a bigger task.",
2022-02-03-21:47:27 root DEBUG "description": "Auto-generated task from a single workload (uuid=461e59f1-4034-4bc3-910b-3f86134d8895)",
2022-02-03-21:47:27 root DEBUG "subtasks": [
2022-02-03-21:47:27 root DEBUG {
2022-02-03-21:47:27 root DEBUG "title": "Authenticate.validate_cinder",
2022-02-03-21:47:27 root DEBUG "description": "Check Cinder Client to ensure validation of token.",
2022-02-03-21:47:27 root DEBUG "scenario": {
2022-02-03-21:47:27 root DEBUG "Authenticate.validate_cinder": {
2022-02-03-21:47:27 root DEBUG "repetitions": 2
2022-02-03-21:47:27 root DEBUG }
2022-02-03-21:47:27 root DEBUG },
2022-02-03-21:47:27 root DEBUG "contexts": {
2022-02-03-21:47:27 root DEBUG "users": {
2022-02-03-21:47:27 root DEBUG "tenants": 3,
2022-02-03-21:47:27 root DEBUG "users_per_tenant": 5
2022-02-03-21:47:27 root DEBUG },
2022-02-03-21:47:27 root DEBUG "api_versions": {
2022-02-03-21:47:27 root DEBUG "cinder": {
2022-02-03-21:47:27 root DEBUG "version": 3,
2022-02-03-21:47:27 root DEBUG "service_type": "volumev3"
2022-02-03-21:47:27 root DEBUG }
2022-02-03-21:47:27 root DEBUG }
2022-02-03-21:47:27 root DEBUG },
2022-02-03-21:47:27 root DEBUG "runner": {
2022-02-03-21:47:27 root DEBUG "constant": {
2022-02-03-21:47:27 root DEBUG "times": 10,
2022-02-03-21:47:27 root DEBUG "concurrency": 5
2022-02-03-21:47:27 root DEBUG }
2022-02-03-21:47:27 root DEBUG },
2022-02-03-21:47:27 root DEBUG "hooks": [],
2022-02-03-21:47:27 root DEBUG "sla": {
2022-02-03-21:47:27 root DEBUG "failure_rate": {
2022-02-03-21:47:27 root DEBUG "max": 0
2022-02-03-21:47:27 root DEBUG }
2022-02-03-21:47:27 root DEBUG }
2022-02-03-21:47:27 root DEBUG }
2022-02-03-21:47:27 root DEBUG ]
2022-02-03-21:47:27 root DEBUG }
2022-02-03-21:47:30 root DEBUG 2022-02-03 21:47:30.764 9020 INFO rally.task.context [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | Context users@openstack setup() finished in 3.58 sec
2022-02-03-21:47:31 root DEBUG 2022-02-03 21:47:31.420 9020 INFO rally.task.context [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | Context api_versions@openstack setup() finished in 0.66 sec
2022-02-03-21:47:31 root DEBUG 2022-02-03 21:47:31.454 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 1 START
2022-02-03-21:47:31 root DEBUG 2022-02-03 21:47:31.457 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 3 START
2022-02-03-21:47:31 root DEBUG 2022-02-03 21:47:31.457 9312 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 2 START
2022-02-03-21:47:31 root DEBUG 2022-02-03 21:47:31.460 9318 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 4 START
2022-02-03-21:47:31 root DEBUG 2022-02-03 21:47:31.469 9313 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 5 START
2022-02-03-21:47:35 root DEBUG 2022-02-03 21:47:35.757 9312 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 2 END: OK
2022-02-03-21:47:36 root DEBUG 2022-02-03 21:47:36.039 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 3 END: OK
2022-02-03-21:47:36 root DEBUG 2022-02-03 21:47:36.041 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 6 START
2022-02-03-21:47:36 root DEBUG 2022-02-03 21:47:36.047 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 1 END: OK
2022-02-03-21:47:36 root DEBUG 2022-02-03 21:47:36.091 9313 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 5 END: OK
2022-02-03-21:47:36 root DEBUG 2022-02-03 21:47:36.227 9318 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 4 END: OK
2022-02-03-21:47:36 root DEBUG 2022-02-03 21:47:36.514 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 6 END: OK
2022-02-03-21:47:36 root DEBUG 2022-02-03 21:47:36.517 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 7 START
2022-02-03-21:47:37 root DEBUG 2022-02-03 21:47:37.296 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 7 END: OK
2022-02-03-21:47:37 root DEBUG 2022-02-03 21:47:37.298 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 8 START
2022-02-03-21:47:37 root DEBUG 2022-02-03 21:47:37.714 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 8 END: OK
2022-02-03-21:47:37 root DEBUG 2022-02-03 21:47:37.716 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 9 START
2022-02-03-21:47:38 root DEBUG 2022-02-03 21:47:38.634 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 9 END: OK
2022-02-03-21:47:38 root DEBUG 2022-02-03 21:47:38.636 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 10 START
2022-02-03-21:47:39 root DEBUG 2022-02-03 21:47:39.356 9310 INFO rally.task.runner [-] Task fa5ab0f5-96fc-4a37-b532-9c9af156f717 | ITER: 10 END: OK
------------------------------------------------------------------------------

After this, nothing happens. It doesn't raise an error and it doesn't continue to the next test.

CinderVolumes.create_and_delete_volume has the same problem.

I tried setting the cinder api version to V1 to see what happens. It gives 10 errors, instead of END: OK, but it again does not exit.

[Bas de Bruijne (basdbruijne)]

After some more investigation, this turned out to be a dependency issue with:

keystone-tempest-plugin
neutron-tempest-plugin
heat-tempest-plugin

I have not been able to pinpoint the exact package that causes this, but I suspect it is lxml or argparse.

[Derek Robertson (rober546)]

Downgrading fcbtest snap from --channel=latest/beta to latest/candidate fixed this for me at least.

[Bas de Bruijne (basdbruijne)]

Hi Derek. This bug is indeed for fcbtest beta, we are trying to push it to stable but this bug is holding us back.

[Andriy Kurilin (andreykurilin)]

try to execute `rally --debug plugin show existing --platform openstack` to see whether rally framework can detect OpenStack plugins. `--debug` option should display dependency issues if something occures.

[Graeme Moss (graememoss)]

I had the same problem no output errors just stales on anything cinder.
after following dereks advise going from latest/stable to latest/candidate it worked.

FCE 2.9 running bionic/ussuri