Destroy DB services in service.stop()

[kiran pawar]

Issue
=====
Manila services are created(in DB) in the start() method. DB consider unique service, based on service-name and host-name combination. However when services are stopped or restarted, the DB entries are not deleted, they are destroyed only in kill() method.

scenario
========
1. In big cluster where multiple instances of manila-scheduler are launched, unique host-name (we use node-name of k8s) and service-name the entries are created in DB.
2. However when deployed again and launched on new hosts, the old entries of manila-scheduler remains as it is.

Solution
========
To overcome this situation, we should remove the DB entry each time service is stopped or killed and create new entry again in start() method.

Exception
========
1. This issue does not reproduce on devstack where single host and single instance of manila-scheduler is used.
2. This issue is not observed for manila-share services since hostname is host@backend which is unique across deployments.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/859301

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/875002

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/859301
Committed: https://opendev.org/openstack/manila/commit/98be6376b2ec74343e0b51eab4e5f36927cfc88e
Submitter: "Zuul (22348)"
Branch: master

commit 98be6376b2ec74343e0b51eab4e5f36927cfc88e
Author: Kiran Pawar <email address hidden>
Date: Mon Sep 26 14:24:29 2022 +0000

    Add 'state' column in 'services' table.

    When manila services are stopped or restarted via stop(), the DB
    entries are not deleted, they are destroyed only in kill() method. In
    cluster deployments, where multiple instances of manila services are
    deployed via PODs, unique hostname is derived from node name. However
    if pods are deployed again and launched on new hosts/nodes, the old
    entries of manila service remains as it is.
    Fix it by adding 'state' column in 'services' table and introducing
    per service cleanup function. On service stop, state is changed to
    'stopped' and cleanup function will delete 'stopped' services unless
    they are 'up' again before cleanup periodic interval.

    Closes-bug: #1990839
    Change-Id: I8b71c4c27ff8fcb25616a95a5ed8362a7f4ffc61

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/875002
Committed: https://opendev.org/openstack/manila/commit/ce42bd9e1a3c2df14362328b13e6e3985edc1227
Submitter: "Zuul (22348)"
Branch: master

commit ce42bd9e1a3c2df14362328b13e6e3985edc1227
Author: Kiran Pawar <email address hidden>
Date: Fri Feb 24 11:10:23 2023 +0000

    Fix db query for service cleanup.

    Instead of getting all services, fetch topic related entries from db
    during service cleanup.

    Related-bug: #1990839
    Depends-on: I8b71c4c27ff8fcb25616a95a5ed8362a7f4ffc61
    Change-Id: I9e911021bd144b76d39193e5480d5ca88973973e

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 16.0.0.0rc1 release candidate.