hosts cannot be taken out of maintenance mode even if there are no running notifications

[Syed Mohammad Adnan Karim]

I ran into a situation where some hosts in a failover segment entered on_maintenance=True but the same hosts had no notifications in a 'running' state.
When I attempted to set the on_maintenance=False for these hosts, I received the following error:

skarim-macbook:~ skarim$ openstack segment host update segment1 0971c8ff-b3a7-40df-8be3-6e878e1e0257 --on_maintenance False
ConflictException: 409: Client Error for url: http://api-masakari:15868/v1/a192df6550d74a7a9d42e8f4afca0804/segments/9958dfe4-ce48-4b6f-a60c-9494998d0a55/hosts/0971c8ff-b3a7-40df-8be3-6e878e1e0257, Host 0971c8ff-b3a7-40df-8be3-6e878e1e0257 can't be updated as it is in-use to process notifications.

segment and notification list outputs: https://paste.ubuntu.com/p/PfqvqdMDkt/

The only workaround for this was to delete all the notifications for these hosts.

[Liam Young (gnuoy)]

Are you able to reliably reproduce this ? If so what steps does it take to reproduce it ? When you say that the only way to work-around it is to delete notifications do you mean by running SQL against the database ?

[Liam Young (gnuoy)]

I am going to mark this as invalid wrt the masakari charm as it seems unlikely this is related to the charm.

[Syed Mohammad Adnan Karim (karimsye)]

> Are you able to reliably reproduce this ? If so what steps does it take to reproduce it ?

I was not able to reliably reproduce this but I have seen it happen a few times.

1. The first time it was observed was when the masakari charms/bundle were installed and then a failover segment was created and hosts were added for the first time (https://bugs.launchpad.net/charm-masakari-monitors/+bug/1884284)

2. The second time it was observed was when new host aggregates were created and then hosts were added to the new host aggregates (https://bugs.launchpad.net/masakari/+bug/1884291)

> When you say that the only way to work-around it is to delete notifications do you mean by running SQL against the database ?

Yes

[suzhengwei (sue.sam)]

Seems it is the same problem with this one:
https://bugs.launchpad.net/masakari/train/+bug/1773765

If there are running notifications in one segment, the user can't update the segment and hosts which belong to this segment.