Tempest plugins does not set the corresponding service in service_available config option

Bug #1743688 reported by Anusha on 2018-01-17
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Aodh
Fix Released
Undecided
Lingxian Kong
Ceilometer
Fix Released
Undecided
Ghanshyam Mann
OpenStack DBaaS (Trove)
Undecided
Ghanshyam Mann
Panko
New
Undecided
Unassigned
congress
Medium
Anusha
devstack
Undecided
Ghanshyam Mann
tempest
Medium
Ghanshyam Mann

Bug Description

Tempest identifies service availability based on just tempest-plugin availability, not actual service is enabled or not.

For example, eventhough devstack plugin of aodh is disabled, and telemetry-tempest-plugin is added to TEMPEST_PLUGINS, still tempest identifies aodh service is enabled, which is wrong.

This behavior is causing congress tempest tests to fail, in actual they have to skip as the service not available

Anusha (anusha-iiitm) on 2018-01-17
Changed in congress:
importance: Undecided → Medium
assignee: nobody → Anusha (anusha-iiitm)
status: New → Confirmed

Fix proposed to branch: master
Review: https://review.openstack.org/534624

Changed in congress:
status: Confirmed → In Progress

Yea, this is issue at least for congress plugin tests. let me describe from where issue started and what can be potential fix:

Congress issue:
aodh and ceilometer tempest plugins are merged into single repo now - telemetry-tempest-plugin. telemetry-tempest-plugin set both ceilomter and aodh_plugin config option in service_available group with default value as True.

Now what happened, congress tempest plugin which is user of aodh and ceilometer tempest plugin interfaces to gets both service available in service_available config group (because of default value). Even Aodh service has not been installed in congress's job setup. This is issue and causing false service availability to congress tests.

General issue:
This issue can considered as general case also where service is being installed and setup bu different repo/pckg (project) and service_available config options is being registered with default value by separate repo (tempest plugin tests).

There can be case where actual service is not installed but you install the plugin for interface usage etc. This is what happening in gate for congress case now.

As solution we can give ownership of setting the service_available options on devstack side which should check actual service availability before setting it to true/false.

Now question is whether it can be done on devstack side[1] or devstack plugin of each service. Devstack plugin can give same issue as it would not run if service is not installed.

..1 https://github.com/openstack-dev/devstack/blob/52237ee5482ff8c614dc68088ecaaf7d59a85915/lib/tempest#L529

Ghanshyam Mann (ghanshyammann) wrote :

thanks for adding discussion.

based on above discussion on IRC, we can check to expand the devstack setting the service availability or if any issue on telemetry-tempest-plugin side.

Changed in tempest:
assignee: nobody → Ghanshyam Mann (ghanshyammann)
importance: Undecided → Medium
Changed in tempest:
status: New → Triaged
Ghanshyam Mann (ghanshyammann) wrote :

In Devstack TEMPEST_SERVICES had been made local variable (https://review.openstack.org/#/c/281788) to move all tempest supported services in devstack setting only. - I9c24705e494689f09a885eb0a640efd50db33fcf

Based on local tempest_services devstack register the available service on tempest side
- https://github.com/openstack-dev/devstack/blob/614cab33c40159f0bc10d92c9f8dc3f9783708d9/lib/tempest#L528

Almost all tempest plugin still using unsupported global variable TEMPEST_SERVICES in impressions that they have registered them as service available on tempest side which is not the case.

- http://codesearch.openstack.org/?q=TEMPEST_SERVICES&i=nope&files=&repos=

Below is plan to solve this issue which exist for almost all the plugins
1. make all tempest plugin services set by devstack with local variable
2. remove unsupported TEMPEST_SERVICES setting for all plugins.
3. any new tempest plugin want their testing service to available in tempest should register that in devstack itself.

Changed in devstack:
assignee: nobody → Ghanshyam Mann (ghanshyammann)

Fix proposed to branch: master
Review: https://review.openstack.org/536723

Changed in devstack:
status: New → In Progress

Seems like devstack patch would not be enough as aodh tempest plugin register aodh as "aodh_plugin" in service_available which is being used same way in congress tempest plugin.

affects: aodh → ceilometer
Changed in ceilometer:
assignee: nobody → Ghanshyam Mann (ghanshyammann)

Change abandoned by Anusha Ramineni (<email address hidden>) on branch: master
Review: https://review.openstack.org/534624

Fix proposed to branch: master
Review: https://review.openstack.org/536741

Changed in ceilometer:
status: New → In Progress

Change abandoned by Ghanshyam Mann (<email address hidden>) on branch: master
Review: https://review.openstack.org/536745
Reason: not needed as fixed in https://review.openstack.org/#/c/536753/

Reviewed: https://review.openstack.org/536830
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=b3539b9fd58b4dd34759bc913e50d95561e451e4
Submitter: Zuul
Branch: master

commit b3539b9fd58b4dd34759bc913e50d95561e451e4
Author: ghanshyam <email address hidden>
Date: Tue Jan 23 16:48:09 2018 +0300

    Remove use of unsupported TEMPEST_SERVICES variable

    TEMPEST_SERVICES global variable is not supported
    by devstack since long back.
    - I380dd20e5ed716a0bdf92aa02c3730359b8136e4
    - I9c24705e494689f09a885eb0a640efd50db33fcf

    Service availability of tempest known services will be
    set by devstack with local check.
    - I02be777bf93143d946ccbb8e9eff637bfd1928d4

    This commit removes the unused TEMPEST_SERVICES setting
    Related-Bug: #1743688

    Change-Id: I50bd32fc2103af0170ee8d350f31cf589d85422f

Reviewed: https://review.openstack.org/536741
Committed: https://git.openstack.org/cgit/openstack/telemetry-tempest-plugin/commit/?id=8bccc06c2bc71798f532f4ba6bc873c5e5dbb896
Submitter: Zuul
Branch: master

commit 8bccc06c2bc71798f532f4ba6bc873c5e5dbb896
Author: ghanshyam <email address hidden>
Date: Tue Jan 23 12:35:06 2018 +0300

    Fix aodh service config option name

    aodh service config option name registered in
    Tempest's service_available group is 'aodh_plugin' which
    is not consistent with all other service_available config
    option.

    After below patch Devstack will automatically set all service
    availability on tempest side. For that all services name should
    going to be registered on Tempest must be consistent with
    service name registered on devstack side.
    - I02be777bf93143d946ccbb8e9eff637bfd1928d4

    This patch rename the config option name to 'aodh'
    and keep the old name with deprecation warning.
    Closes-Bug: #1743688

    Change-Id: I7739e0b672a7211a1d643543fdc79031affa0a0b

Changed in ceilometer:
status: In Progress → Fix Released

Change abandoned by Ghanshyam Mann (<email address hidden>) on branch: master
Review: https://review.openstack.org/536723
Reason: doing it on plugin side.

As per discussion on IRC and review, below is the agreed plan

- Devstack should set only tempest own services
- each plugin to set their service as service_availability in their devstack plugin
- plugin using other plugins can set service_availability of required plugins in devstack plugin as per job definition

As per comment above, not a Tempest bug. Marking as "incomplete" on Tempest

Changed in tempest:
status: Triaged → Incomplete

Reviewed: https://review.openstack.org/536753
Committed: https://git.openstack.org/cgit/openstack/congress-tempest-plugin/commit/?id=e58b0eb52d6c3794820e33e60bd208328c3a1380
Submitter: Zuul
Branch: master

commit e58b0eb52d6c3794820e33e60bd208328c3a1380
Author: ghanshyam <email address hidden>
Date: Tue Jan 23 13:25:35 2018 +0300

    Use 'aodh' for aodh service availability

    below patch rename the aodh service_available
    config option from 'aodh_plugin' to 'aodh'.
    - I7739e0b672a7211a1d643543fdc79031affa0a0b

    This patch make congress tempest test to use the
    new name so that it can be fetched correctly from
    devstack setting on tempest config file.

    Depends-On: I7739e0b672a7211a1d643543fdc79031affa0a0b

    Change-Id: Iccfc40cc738994d6c7d08cbea845e0c2f888b995
    Related-Bug: #1743688

Reviewed: https://review.openstack.org/533133
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=aa779c5bd46649d9b672e1112c9e65ec857cde66
Submitter: Zuul
Branch: master

commit aa779c5bd46649d9b672e1112c9e65ec857cde66
Author: Anusha Ramineni <email address hidden>
Date: Mon Feb 5 10:03:26 2018 +0530

    Configure tempest service_availability correctly

    To be able to correctly identify services are enabled or not to
    run respective tempest tests, configure the same in devstack plugin
    in proper way, instead of deciding the service availability based on
    import of plugins.

    Closes-Bug: #1743688
    Change-Id: I9cbe3f1307b78d0e703069bc6c011b62c0d3c51f

Changed in congress:
status: In Progress → Fix Released

This issue was fixed in the openstack/congress 7.0.0.0rc1 release candidate.

Patch for devstack has been abandoned, please update if there is still an issue with devstack.

Changed in devstack:
status: In Progress → Incomplete
Changed in tempest:
status: Incomplete → Invalid
Ghanshyam Mann (ghanshyammann) wrote :

marked invalid for Tempest.

Ghanshyam Mann (ghanshyammann) wrote :

For Devstack - We need to move the non tempest owned service availability setting on devtsack plugins side and remove those from devstack.

Fix proposed to branch: master
Review: https://review.openstack.org/619973

Changed in devstack:
status: Incomplete → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/619984

Changed in trove:
assignee: nobody → Ghanshyam Mann (ghanshyammann)
status: New → In Progress
summary: - Tempest unable to detect service availability properly, causing congress
- tests to fail
+ Tempest plugins does not set the corresponding service in
+ service_available config option

Fix proposed to branch: master
Review: https://review.openstack.org/620002

Changed in aodh:
assignee: nobody → Ghanshyam Mann (ghanshyammann)
status: New → In Progress

Reviewed: https://review.openstack.org/620255
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=6611df0e2da0abc96979c52da0013535abf79428
Submitter: Zuul
Branch: master

commit 6611df0e2da0abc96979c52da0013535abf79428
Author: ghanshyam <email address hidden>
Date: Tue Nov 27 08:21:35 2018 +0000

    Set Tempest's service_available setting for congress

    Tempest's service_available config option includes all the service
    availability which is further used by tests to take decision of skip
    or run the test.

    For example, [service_available].congress is true then, congress test will run
    or if [service_available].congress is false then, all congress related tests either
    in telemetry tempest plugin or any other plugins will be skipped.

    This commit add the setting of congress service[1] on congress devstack plugin.

    Closes-Bug: #1743688
    [1] http://git.openstack.org/cgit/openstack/congress-tempest-plugin/tree/congress_tempest_plugin/config.py#n24

    Change-Id: I57de34d2b185b8e9838dd288d3673149391f0024

Reviewed: https://review.openstack.org/619984
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=d5d9342beeff5c8ac655073b8986feb6b91a9f25
Submitter: Zuul
Branch: master

commit d5d9342beeff5c8ac655073b8986feb6b91a9f25
Author: ghanshyam <email address hidden>
Date: Mon Nov 26 08:25:23 2018 +0000

    Set Tempest's service_availability setting for Trove

    Tempest's service_availability config option includes all the service
    availability which is further used by tests to take decision of skip
    or run the test.

    For example, [service_availability].trove is true then, trove test will run
    or if [service_availability].trove is false then, all trove related tests either
    in trove tempest plugin or any other plugins will be skipped.

    Currently this setting for trove service[1] is in devstack lib/tempest
    which is being removed by - https://review.openstack.org/#/c/619973/

    For better maintenance, we are moving all tempest non-owned service setting
    to service devstack plugin side.

    This commit add the setting of trove service on trove devstack plugin.

    Closes-Bug: #1743688
    [1] https://git.openstack.org/cgit/openstack/trove-tempest-plugin/tree/trove_tempest_plugin/config.py#n17

    Needed-By: https://review.openstack.org/#/c/619973/
    Change-Id: Ie3c6bab6e0be9dfcfe29ec3f412e704f3ad79d61

Changed in trove:
status: In Progress → Fix Released
Ghanshyam Mann (ghanshyammann) wrote :

Octavia and other tempest plugins are also facing same issue bug#1817154.

Ghanshyam Mann (ghanshyammann) wrote :

Note: we are facing these issue because service_available config option is registered by tempest plugin repo which is separate repo from service. So registration of config option and setting value is happening via separate repo.

These are the final steps to fix the service_availabilty config option.

1. Devstack should set only tempest own services - https://review.openstack.org/#/c/619973/
2. each plugin to set their service as service_availability in their devstack plugin - In progress
3. Change the default value of service_available.<service> config option default value to false.
4. Add used tempest plugins in tempest plugins's requirements.txt after adding it in g-r.

That way we will solve all the issues currently faced by cross tempest plugin:
1. Required tempest plugin's service_availability config option is registered:
- Any tempest plugin can install required tempest plugins as a dependency in requirement.txt.
2. Service_available value is set correctly:
- Each service will set its service_available config option correctly while installation itself via devstack plugin.

This issue was fixed in the openstack/congress 9.0.0.0rc1 release candidate.

This issue was fixed in the openstack/trove 11.0.0.0rc1 release candidate.

Changed in aodh:
assignee: Ghanshyam Mann (ghanshyammann) → Lingxian Kong (kong)

Reviewed: https://review.openstack.org/620002
Committed: https://git.openstack.org/cgit/openstack/aodh/commit/?id=0e40e57c1ab2e8468016d4d877a91ce3a06df40a
Submitter: Zuul
Branch: master

commit 0e40e57c1ab2e8468016d4d877a91ce3a06df40a
Author: ghanshyam <email address hidden>
Date: Mon Nov 26 10:19:39 2018 +0000

    Set Tempest's service_availability setting for Aodh

    Tempest's service_available config option includes all the service
    availability which is further used by tests to take decision of skip
    or run the test.

    For example, [service_available].aodh is true then, aodh test will run
    or if [service_available].aodh is false then, all aodh related tests either
    in telemetry tempest plugin or any other plugins will be skipped.

    This commit add the setting of aodh service[1] on aodh devstack plugin.

    Closes-Bug: #1743688
    [1] https://github.com/openstack/telemetry-tempest-plugin/blob/master/telemetry_tempest_plugin/config.py#L27

    Change-Id: Idac9f38aac6964ce4d40b6afccbd9dfbf6e16a41

Changed in aodh:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers