verify-config KeyError with tempest plugins

[Martin Kopec]

Let's say I install several tempest plugins, for example:

$ tempest list-plugins
+-----------------+--------------------------------------------------------+
| Name | EntryPoint |
+-----------------+--------------------------------------------------------+
| heat | heat_tempest_plugin.plugin:HeatTempestPlugin |
| telemetry_tests | telemetry_tempest_plugin.plugin:TelemetryTempestPlugin |
| cinder_tests | cinder_tempest_plugin.plugin:CinderTempestPlugin |
| designate | designate_tempest_plugin.plugin:DesignateTempestPlugin |
| horizon | tempest_horizon.plugin:HorizonTempestPlugin |
| ironic_tests | ironic_tempest_plugin.plugin:IronicTempestPlugin |
+-----------------+--------------------------------------------------------+

Then I generate tempest.conf (manually or by python-tempestconf, doesn't really matter) and I want to verify tempest.conf by `tempest verify-config`:

$ tempest verify-config -o new_tempest.conf
Running config verification...
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/cmd/verify_tempest_config.py", line 484, in take_action
    main(parsed_args)
  File "/usr/lib/python2.7/site-packages/tempest/cmd/verify_tempest_config.py", line 451, in main
    services = check_service_availability(os, update)
  File "/usr/lib/python2.7/site-packages/tempest/cmd/verify_tempest_config.py", line 382, in check_service_availability
    codename_match[cfgname]):
KeyError: 'heat_plugin'

and the same with :baremetal_introspection" or "metric" and so on ...

It seems verify-config doesn't count with a situation, when there is a plugin available in the system belonging to a different service than these [1], because it fails here [2].

Either we can put all known plugins to that list [1], which I believe is unrealistic and also it doesn't count with situations when a user has his own custom plugin, or verify-config should be able to create/detect such a list dynamically.

Meanwhile we should at least make verify-config not fail and inform a user that there is a situation where verify-config can't verify such plugin, f.e.: "verify config can't verify heat_plugin, baremetal_introspection, metric - manual verification needed."

[1] https://github.com/openstack/tempest/blob/98c0aca141f8decf5768003fd3d80e1ddc735420/tempest/cmd/verify_tempest_config.py#L345
[2] https://github.com/openstack/tempest/blob/98c0aca141f8decf5768003fd3d80e1ddc735420/tempest/cmd/verify_tempest_config.py#L373

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/631801

[Ghanshyam Mann (ghanshyammann)]

Yes, with current code verify-config does not handle the Tempest plugin. We should extend that for plugin but i agree as of now we can catch that error and print appropriate msg.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/631801
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=07a572ca175d10fb7366c5015aa28419b76af3c8
Submitter: Zuul
Branch: master

commit 07a572ca175d10fb7366c5015aa28419b76af3c8
Author: Martin Kopec <email address hidden>
Date: Fri Jan 18 14:23:06 2019 +0000

    Fix verify-config KeyError

    When a user runs verify-config while custom, third
    party plugins are available in the system, it fails
    with KeyError.
    The patch fixes that by informing a user about the
    plugins and suggesting their setup.

    Related-Bug: #1812385
    Change-Id: I8019a82717d9463bdc1a6e5025758e1f917358bb

[Martin Kopec (mkopec)]

KeyError got fixed by the linked review so I'm gonna mark this as Fixed .. although providing a way how to verify plugin's configuration within tempest.conf is still missing - for that I created a new bug: https://bugs.launchpad.net/tempest/+bug/1868683