There is a possibility that 'running' notification will remain

[takahara.kengo]

[Impact]
masakari-engine has two periodic tasks, one for processing 'new' notifications and the other for processing 'error' notifications But it doesn't have a periodic task for processing 'running' notifications.

Looking at the code of masakari-engine, if the process of it goes down immediately after it changes notification status from 'new' to 'running', then the notification which status is 'running' will remain will not be processed by periodic tasks.

So, should masakari-engine's periodic task process the 'running' notification?
(Although it need to make such a logic that main process doesn't compete with periodic tasks.)
Or should the 'running' notification be handled by the operator?

[Test Case]
lxc launch ubuntu-daily:groovy g1 (or other corresponding release combination)
lxc exec g1 /bin/bash
sudo apt install masakari-engine

== expect test failure with old code ==
setup:
* copy new test code from patch to /usr/lib/python3/dist-packages/masakari/tests/unit/engine/test_engine_mgr.py
* modify /usr/lib/python3/dist-packages/masakari/tests/unit/engine/test_engine_mgr.py to set EXPIRED_TIME = NOW, and comment out call to test_check_expired_notifications since it doesn't exist without new patch applied.
test:
* cd /usr/lib/python3/dist-packages
* python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications

== expect test success with patched code ==
setup: enable corresponding -proposed pocket
test:
* cd /usr/lib/python3/dist-packages
* python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications

[Regression Potential]
A regression in this code could occur if either of the time intervals were calculated incorrectly which means a notification could be marked as failed perhaps long before the expiration interval. The defaults can be changed for check_expired_notifications_interval and notifications_expired_interval via masakari-engine config options which helps mitigate that risk. The defaults seem reasonable with a 10 minute periodic check and setting expired notifications to failed status only after 24 hours.

[Peter Jose De Sousa (pjds)]

Hit this bug yesterday, I was unable to reset my compute node status using

`openstack segment host update $SEGMENT_UUID node01 --on_maintenance False`.

After much digging I noticed that there was still a notification with "running".

I worked around this by logging into the DB and deleting the running record from the notifications table.

[Peter Jose De Sousa (pjds)]

Logs from this deployment: https://drive.google.com/open?id=1Qir71XFHsjuumT9ApjbYKsaDoCp3jqoS

[suzhengwei (sue.sam)]

I met the same problem recently.

I think there are two solutions.
1.Add one Notification API, which allow the operator to change one notification's state.
2.As suggested by takahara.kengo, in masakari-engine, we can add one periodic task to check runing notification. If process time of the notification longer than config time, force set it state to 'failed'.

[Nobuto Murata (nobuto)]

Linking to the review:
https://review.opendev.org/#/c/720623/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720623
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=3a4f782441f6bdcfb8ee49a393937267fc246c56
Submitter: Zuul
Branch: master

commit 3a4f782441f6bdcfb8ee49a393937267fc246c56
Author: suzhengwei <[e-mail]>
Date: Fri Apr 17 08:59:35 2020 +0800

    check expired notifications

    Occasionally, there would be notifications which will remain 'new',
    'error' or 'running' status all times, and not to be processed again.
    Due to this, operator can not update the segment or host.

    This patch add one task to periodically check unfinished notifications.
    If one unfinished notification is expired, just set its status to
    'failed'.

    Close-Bug: #1773765
    Change-Id: If49635639dd976aeec3ea73e702ad2636fcf1e0a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/753921

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/753926

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/753927

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/753927
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=cd05dc31bb8c4ce529dac4698dd7bb80866a4139
Submitter: Zuul
Branch: stable/stein

commit cd05dc31bb8c4ce529dac4698dd7bb80866a4139
Author: suzhengwei <[e-mail]>
Date: Fri Apr 17 08:59:35 2020 +0800

    check expired notifications

    Occasionally, there would be notifications which will remain 'new',
    'error' or 'running' status all times, and not to be processed again.
    Due to this, operator can not update the segment or host.

    This patch add one task to periodically check unfinished notifications.
    If one unfinished notification is expired, just set its status to
    'failed'.

    Conflicts:
            masakari/tests/unit/engine/test_engine_mgr.py
            - difference of import mock and import unittest.mock

    Closes-Bug: #1773765
    Change-Id: If49635639dd976aeec3ea73e702ad2636fcf1e0a
    (cherry picked from commit 3a4f782441f6bdcfb8ee49a393937267fc246c56)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/753926
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=e0423906a7aabb4ea02b754e041f725c16535be5
Submitter: Zuul
Branch: stable/train

commit e0423906a7aabb4ea02b754e041f725c16535be5
Author: suzhengwei <[e-mail]>
Date: Fri Apr 17 08:59:35 2020 +0800

    check expired notifications

    Occasionally, there would be notifications which will remain 'new',
    'error' or 'running' status all times, and not to be processed again.
    Due to this, operator can not update the segment or host.

    This patch add one task to periodically check unfinished notifications.
    If one unfinished notification is expired, just set its status to
    'failed'.

    Conflicts:
            masakari/tests/unit/engine/test_engine_mgr.py
            - Update test for difference of unittest.mock and mock

    Closes-Bug: #1773765
    Change-Id: If49635639dd976aeec3ea73e702ad2636fcf1e0a
    (cherry picked from commit 3a4f782441f6bdcfb8ee49a393937267fc246c56)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/753921
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=b53155b314068a46cd097d3788add1b6d4fc0aff
Submitter: Zuul
Branch: stable/ussuri

commit b53155b314068a46cd097d3788add1b6d4fc0aff
Author: suzhengwei <[e-mail]>
Date: Fri Apr 17 08:59:35 2020 +0800

    check expired notifications

    Occasionally, there would be notifications which will remain 'new',
    'error' or 'running' status all times, and not to be processed again.
    Due to this, operator can not update the segment or host.

    This patch add one task to periodically check unfinished notifications.
    If one unfinished notification is expired, just set its status to
    'failed'.

    Close-Bug: #1773765
    Change-Id: If49635639dd976aeec3ea73e702ad2636fcf1e0a
    (cherry picked from commit 3a4f782441f6bdcfb8ee49a393937267fc246c56)

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "groovy victoria patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Radosław Piliszek (yoctozepto)]

Groovy should really just use the newly-released RC1 of Victoria.

[Billy Olsen (billy-olsen)]

Agreed that Groovy should just use the newly-released RC1.

[Corey Bryant (corey.bryant)]

@Billy, Thanks for your work on this. I've uploaded new versions of masakari to groovy, focal, train and stein [1] (note: minor changes to changelogs and I re-picked the stein patch from upstream stable/stein).

[1]
https://launchpad.net/ubuntu/groovy/+queue?queue_state=1&queue_text=masakari
https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=masakari
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/train-staging/+packages?field.name_filter=masakari&field.status_filter=published&field.series_filter=
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/stein-staging/+packages?field.name_filter=masakari&field.status_filter=published&field.series_filter=

[Launchpad Janitor (janitor)]

This bug was fixed in the package masakari - 10.0.0~rc1-0ubuntu1

---------------
masakari (10.0.0~rc1-0ubuntu1) groovy; urgency=medium

  [ Chris MacNaughton ]
  * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

  [ Corey Bryant ]
  * d/watch: Scope to 10.x series.
  * New upstream release candidate for OpenStack Victoria (LP: #1773765).
  * d/p/monkey-patch-original-current-thread.patch: Dropped. Fixed in rc1.
  * d/p/allow-bare-hostnames.patch: Dropped. Alternative patch merged in rc1.

 -- Corey Bryant <email address hidden> Thu, 08 Oct 2020 13:20:37 -0400

[Robie Basak (racb)]

Hello takahara.kengo, or anyone else affected,

Accepted masakari into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/masakari/9.0.0-0ubuntu0.20.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello takahara.kengo, or anyone else affected,

Accepted masakari into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello takahara.kengo, or anyone else affected,

Accepted masakari into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello takahara.kengo, or anyone else affected,

Accepted masakari into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

This bug was fixed in the package masakari - 10.0.0~rc1-0ubuntu1~cloud0
---------------

 masakari (10.0.0~rc1-0ubuntu1~cloud0) focal-victoria; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 masakari (10.0.0~rc1-0ubuntu1) groovy; urgency=medium
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
 .
   [ Corey Bryant ]
   * d/watch: Scope to 10.x series.
   * New upstream release candidate for OpenStack Victoria (LP: #1773765).
   * d/p/monkey-patch-original-current-thread.patch: Dropped. Fixed in rc1.
   * d/p/allow-bare-hostnames.patch: Dropped. Alternative patch merged in rc1.

[Corey Bryant (corey.bryant)]

Verified successfully for all corresponding proposed package versions:

== focal-proposed ==
root@f1:/usr/lib/python3/dist-packages# apt policy masakari-engine
masakari-engine:
  Installed: 9.0.0-0ubuntu0.20.04.3
  Candidate: 9.0.0-0ubuntu0.20.04.3
  Version table:
 *** 9.0.0-0ubuntu0.20.04.3 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
root@f1:~# cd /usr/lib/python3/dist-packages
root@f1:/usr/lib/python3/dist-packages# python3 -m unittest masakari.tests.unit.engine.test_engine_mgr.EngineManagerUnitTestCase.test_check_expired_notifications
2020-10-26 18:39:39.505 4505 INFO masakari.engine.driver [-] Loading masakari notification driver 'taskflow_driver'
/usr/lib/python3/dist-packages/taskflow/atom.py:31: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  _sequence_types = (list, tuple, collections.Sequence)
/usr/lib/python3/dist-packages/taskflow/atom.py:32: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
  _set_types = (set, collections.Set)
/usr/lib/python3/dist-packages/masakari/context.py:92: DeprecationWarning: Using the 'user' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_id' argument instead
  super(RequestContext, self).__init__(
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'user_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'user_domain_id' argument instead
  return wrapped(*args, **kwargs)
/usr/lib/python3/dist-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'project_domain' argument is deprecated in version '2.18' and will be removed in version '3.0', please use the 'project_domain_id' argument instead
  return wrapped(*args, **kwargs)
2020-10-26 18:39:40.122 4505 ERROR masakari.engine.manager [req-03b9518f-a271-4d8c-aeee-f6c3d3252a09 - - - - -] Periodic task 'check_expired_notifications': Notification a03ac64b-a29e-474f-b94c-77767454b323 is expired.
.
----------------------------------------------------------------------
Ran 1 test in 0.917s

OK

== ussuri-proposed ==
root@b1:~# apt policy masakari-engine
masakari-engine:
  Installed: 9.0.0-0ubuntu0.20.04.3~cloud0
  Candidate: 9.0.0-0ubuntu0.20.04.3~cloud0
  Version table:
 *** 9.0.0-0ubuntu0.20.04.3~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/ussuri/main amd64 Pa...

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for masakari has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package masakari - 9.0.0-0ubuntu0.20.04.3

---------------
masakari (9.0.0-0ubuntu0.20.04.3) focal; urgency=medium

  [ Chris MacNaughton ]
  * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

  [ Billy Olsen ]
  * Check expired notifications and clean running notifications past an expiration
    to allow for a host to become manageable again (LP: #1773765).
    - d/p/check-expired-notifications.patch: Adds a periodic task to check for running
      notifications that have exceeded a threshold and removes them if necessary.

 -- Billy Olsen <email address hidden> Wed, 23 Sep 2020 20:19:36 -0700

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for masakari has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package masakari - 9.0.0-0ubuntu0.20.04.3~cloud0
---------------

 masakari (9.0.0-0ubuntu0.20.04.3~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 masakari (9.0.0-0ubuntu0.20.04.3) focal; urgency=medium
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
 .
   [ Billy Olsen ]
   * Check expired notifications and clean running notifications past an expiration
     to allow for a host to become manageable again (LP: #1773765).
     - d/p/check-expired-notifications.patch: Adds a periodic task to check for running
       notifications that have exceeded a threshold and removes them if necessary.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for masakari has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package masakari - 8.0.0-0ubuntu1~cloud1
---------------

 masakari (8.0.0-0ubuntu1~cloud1) bionic-train; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/train branch.
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
 .
   [ Billy Olsen ]
   * Check expired notifications and clean running notifications past an expiration
     to allow for a host to become manageable again (LP: #1773765).
     - d/p/check-expired-notifications.patch: Adds a periodic task to check for running
       notifications that have exceeded a threshold and removes them if necessary.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for masakari has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package masakari - 7.0.0-0ubuntu1~cloud1
---------------

 masakari (7.0.0-0ubuntu1~cloud1) bionic-stein; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/stein branch.
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
 .
   [ Billy Olsen ]
   * Check expired notifications and clean running notifications past an expiration
     to allow for a host to become manageable again (LP: #1773765).
     - d/p/check-expired-notifications.patch: Adds a periodic task to check for running
       notifications that have exceeded a threshold and removes them if necessary.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/758899
Committed: https://git.openstack.org/cgit/openstack/charm-masakari/commit/?id=62bc9c05b9cd438d45067e0efafe6fa75ef5b500
Submitter: Zuul
Branch: master

commit 62bc9c05b9cd438d45067e0efafe6fa75ef5b500
Author: Billy Olsen <email address hidden>
Date: Mon Oct 19 19:48:26 2020 -0700

    Add config options for expired notifications

    Adds config options to configure the notification expiration time
    as well as how frequent to check for expired notifications.

    Change-Id: I9607f01a800baded37cfc23b6bc9f8e6aecc2d74
    Closes-Bug: #1773765

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/20.10
Review: https://review.opendev.org/762216

[Giuseppe Petralia (peppepetra)]

We are still seeing this happening in bionic ussuri

$ os notification list | grep running
| a5a79aba-c3e3-464e-9b0b-3cedd272fd4d | 2021-02-18T04:57:01.000000 | running | COMPUTE_HOST | 88eae267-fabc-48d2-b6cc-4acdb6f9b5e2 | {'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'} |
| e1b2b27f-7a7a-4b33-948e-26d3fb06d846 | 2021-02-18T04:56:59.000000 | running | COMPUTE_HOST | fcbc74ad-264c-48c8-8070-6a032c1e17ce | {'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'} |
| 5990a030-dc86-4860-92b6-5c95bb14f081 | 2021-02-18T04:56:58.000000 | running | COMPUTE_HOST | f975d04f-a345-4ea0-994b-5f014fbd146b | {'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'} |
| 4b1b4989-0ac5-406b-a71c-3f1e0890a2f4 | 2021-02-18T04:56:57.000000 | running | COMPUTE_HOST | 7573d928-5a07-4ea2-9d1c-96c7c6fabb19 | {'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'} |
| b0aedfe2-4c9c-428b-afdc-84f47609d378 | 2021-02-18T04:56:56.000000 | running | COMPUTE_HOST | a41da0a6-b41e-4e1d-872d-14517fa6865d | {'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'} |
| 554c7e55-49a8-46e5-b4c8-e6a0cfbd4628 | 2021-02-18T04:56:55.000000 | running | COMPUTE_HOST | 1a6350ae-f4b2-4c3c-99a0-20a4801b34ff | {'event': 'STOPPED', 'cluster_status': 'OFFLINE', 'host_status': 'NORMAL'} |

$ os segment host list $seg
+--------------------------------------+-------------------------------------------------------------------+---------+--------------------+----------+----------------+--------------------------------------+
| uuid | name | type | control_attributes | reserved | on_maintenance | failover_segment_id |
+--------------------------------------+-------------------------------------------------------------------+---------+--------------------+----------+----------------+--------------------------------------+
| fcbc74ad-264c-48c8-8070-6a032c1e17ce | compute-server-5 | COMPUTE | SSH | False | True | 92cd125b-daff-4c13-b6a6-584a92528808 |
| f975d04f-a345-4ea0-994b-5f014fbd146b | compute-server-4| COMPUTE | SSH | False | True | 92cd125b-daff-4c13-b6a6-584a92528808 |
| a41da0a6-b41e-4e1d-872d-14517fa6865d | compute-server-2 | COMPUTE | SSH | False | True | 92cd125b-daff-4c13-b6a6-584a92528808 |
| 1a6350ae-f4b2-4c3c-99a0-20a4801b34ff | compute-server-1 | COMPUTE | SSH | False | True | 92cd125b-daff-4c13-b6a6-584a92528808 |
| 88eae267-fabc-48d2-b6cc-4acdb6f9b5e2 | compute-server-6 | COMPUTE | SSH | False | True | 92cd125b-daff-4c13-b6a6-584a92528808 |
| 7573d928-5a07-4ea2-9d1c-96c7c6fabb19 | compute-server-3 | COMPUTE | SSH | False | True | 92cd1...

[Giuseppe Petralia (peppepetra)]

Version in use:
dpkg -l | grep masaka
ii masakari-api 9.0.0-0ubuntu0.20.04.3~cloud0 all Virtual Machine High Availability (VMHA) for OpenStack - API Server
ii masakari-common 9.0.0-0ubuntu0.20.04.3~cloud0 all Virtual Machine High Availability (VMHA) for OpenStack - Common Files
ii masakari-engine 9.0.0-0ubuntu0.20.04.3~cloud0 all Virtual Machine High Availability (VMHA) for OpenStack - Engine
ii python3-masakari 9.0.0-0ubuntu0.20.04.3~cloud0 all Virtual Machine High Availability (VMHA) for OpenStack - Python 3

[Radosław Piliszek (yoctozepto)]

This is fixed in a later Train release. The current is 9.1.1.

Ubuntu has to build the package.

[Radosław Piliszek (yoctozepto)]

Ussuri* release. Sorry for the confusion.