Masakari hostmonitor fails beacuse of missing config

Bug #1959669 reported by Yusuf Güngör
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
kolla-ansible
New
Undecided
Unassigned

Bug Description

**Bug Report**

What happened:

Hi,

We are trying to implement masakari with hacluster.

After installation and testing the Compute Node HA, there exist ERROR logs on controllers:

"/var/log/kolla/masakari/masakari-wsgi.log":
--------------------------------------------------------
2022-01-31 17:21:51.079 954 INFO masakari.api.openstack.wsgi [req-f9909168-e668-4a97-9752-efe352b3d07e 8e4170ab9f534a2abd04a48b230cf514 0cd27cc3628a4fbeb74a816a11857547 - - -] HTTP excepti
on thrown: Notification received from host r2-osp-test-compute-05.mycompany.dmz of type 'COMPUTE_HOST' is ignored as the host is already under maintenance.
2022-01-31 17:22:46.351 953 ERROR masakari.api.openstack.extensions [req-95361f54-dbb7-4768-8783-e4cccab61968 18c347f36cd74daa9f020b12aee085d2 ba1fc36f5c9c4bbc9fb9647ae0a81b04 - - -] Unexp
ected exception in API method: masakari.exception_Remote.MasakariException_Remote: Failed to fetch notification recovery workflow details for 7e0eec81-e782-4ee7-ad01-fe5f7c4d0fc5
Traceback (most recent call last):

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/masakari/engine/manager.py", line 422, in get_notification_recovery_workflow_details
    self.driver.get_notification_recovery_workflow_details(

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/masakari/engine/drivers/taskflow/driver.py", line 275, in get_notification_recovery_workflow_details
    backend = backends.fetch(PERSISTENCE_BACKEND)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/taskflow/persistence/backends/__init__.py", line 53, in fetch
    backend, conf = misc.extract_driver_and_conf(conf, 'connection')

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/taskflow/utils/misc.py", line 123, in extract_driver_and_conf
    maybe_uri = conf[conf_key]

TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
    res = self.dispatcher.dispatch(message)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/var/lib/kolla/venv/lib/python3.8/site-packages/masakari/engine/manager.py", line 429, in get_notification_recovery_workflow_details
    raise exception.MasakariException(msg)
--------------------------------------------------------

This issue has similarities with https://bugs.launchpad.net/charm-masakari/+bug/1882656

We have followed that case and resolved this issue after overriding the "masakari.conf" like below:

/etc/kolla/config/masakari.conf
--------------------------------------------------------
[DEFAULT]
notification_driver = taskflow_driver

[taskflow]
connection = mysql+pymysql://masakari:*********@osp-r2-test-internal.mycompany.com:3306/masakari
--------------------------------------------------------

We had to add "notification_driver" to "DEFAULT" section and also added "taskflow" section with DB connection string.

We also had to remove all masakari tables from DB and re-run the "$ masakari-manage db sync" command from the "masakari_api" container.

After that we got the "flowdetails" and other missing DB tables and everything worked.

What you expected to happen: Masakari Installation should work out of the box.

How to reproduce it (minimal and precise): Tried to install hacluster and masakari.

**Environment**:
* OS (e.g. from /etc/os-release): Ubuntu 20.04.2 LTS
* Kernel (e.g. `uname -a`): Linux 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
* Docker version if applicable (e.g. `docker version`): 20.10.7
* Kolla-Ansible version (e.g. `git head or tag or stable branch` or pip package version if using release): pip installation, kolla-ansible==12.3.0
* Docker image Install type (source/binary): source
* Docker image distribution: DockerHub Images

Revision history for this message
Yusuf Güngör (yusuf2) wrote :

There also exist a bug on Masakari: https://bugs.launchpad.net/masakari/+bug/1882656

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.