| 2020-05-14 07:07:33 |
Daisuke Suzuki |
bug |
|
|
added bug |
| 2020-05-14 07:09:06 |
Daisuke Suzuki |
masakari-monitors: assignee |
|
Daisuke Suzuki (suzuki-di) |
|
| 2020-05-14 07:09:19 |
Daisuke Suzuki |
masakari-monitors: status |
New |
In Progress |
|
| 2020-09-12 16:31:53 |
Radosław Piliszek |
bug |
|
|
added subscriber Radosław Piliszek |
| 2020-09-26 13:12:26 |
Radosław Piliszek |
masakari-monitors: status |
In Progress |
Incomplete |
|
| 2021-03-14 19:44:25 |
Radosław Piliszek |
masakari-monitors: status |
Incomplete |
Invalid |
|
| 2021-03-21 17:57:41 |
Radosław Piliszek |
masakari-monitors: status |
Invalid |
Triaged |
|
| 2021-03-21 17:57:47 |
Radosław Piliszek |
masakari-monitors: importance |
Undecided |
High |
|
| 2021-03-21 18:02:04 |
Radosław Piliszek |
summary |
There are cases when masakari-hostmonitor will recognize online ComputeNodes as offline if ComputeNodes are managed by pacemaker_remote |
There are cases when masakari-hostmonitor will recognize online nodes as offline and send (in)appropriate notifications to Masakari |
|
| 2021-03-21 18:06:15 |
Radosław Piliszek |
description |
[Issue]
ComputeNodes are managed by pacemaker_remote in my environment.
When one ComputeNode is isolated in the network, masakari-hostmonitors on the other ComputeNodes will send failure notification about the isolated ComputeNode to masakari-api.
At that time, the isolated masakari-hostomonitor will recognize other ComputeNodes as offline. So it sends failure notification about online ComputeNodes.
As a result, masakari-engine runs the recovery procedure to online ComputeNodes.
[Cause]
The current masakari-hostmonitor can't determine whether or not it is isolated in the network if ComputeNodes are managed by pacemaker_remote.
masakari-hostmonitor with pacemaker(not remote) will wait until it is killed if it is isolated in the network. It is implemented in the following code.
<https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/hostmonitor/host_handler/handle_host.py#L398-L402>
But masakari-hostmonitor with pacemaker_remote won't determine if it is isolated.
<https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/hostmonitor/host_handler/handle_host.py#L93-L95>
[Solution]
The ComputeNode managed by pacemaker_remote should determine recognize itself as offline when it is isolated.
The state monitoring process should be skipped in that case. |
[Issue]
ComputeNodes are managed by pacemaker_remote in my environment.
When one ComputeNode is isolated in the network, masakari-hostmonitors on the other ComputeNodes will send failure notification about the isolated ComputeNode to masakari-api.
At that time, the isolated masakari-hostomonitor will recognize other ComputeNodes as offline. So it sends failure notification about online ComputeNodes.
As a result, masakari-engine runs the recovery procedure to online ComputeNodes.
[Cause]
The current masakari-hostmonitor can't determine whether or not it is isolated in the network if ComputeNodes are managed by pacemaker_remote.
masakari-hostmonitor with pacemaker(not remote) will wait until it is killed if it is isolated in the network. It is implemented in the following code.
<https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/hostmonitor/host_handler/handle_host.py#L398-L402>
But masakari-hostmonitor with pacemaker_remote won't determine if it is isolated.
<https://github.com/openstack/masakari-monitors/blob/master/masakarimonitors/hostmonitor/host_handler/handle_host.py#L93-L95>
[Solution]
The ComputeNode managed by pacemaker_remote should determine recognize itself as offline when it is isolated.
The state monitoring process should be skipped in that case.
See comment #11 for how yoctozepto managed to reproduce something similar to the described. |
|
| 2021-09-13 19:15:42 |
Radosław Piliszek |
masakari-monitors: assignee |
Daisuke Suzuki (suzuki-di) |
Radosław Piliszek (yoctozepto) |
|
| 2021-09-13 19:34:35 |
OpenStack Infra |
masakari-monitors: status |
Triaged |
In Progress |
|
| 2021-09-14 10:48:16 |
OpenStack Infra |
masakari-monitors: status |
In Progress |
Fix Released |
|
| 2021-09-15 18:16:08 |
OpenStack Infra |
tags |
|
in-stable-wallaby |
|
| 2021-09-15 18:19:23 |
OpenStack Infra |
tags |
in-stable-wallaby |
in-stable-victoria in-stable-wallaby |
|
| 2021-09-15 18:22:15 |
OpenStack Infra |
tags |
in-stable-victoria in-stable-wallaby |
in-stable-ussuri in-stable-victoria in-stable-wallaby |
|
| 2021-09-18 18:47:15 |
Radosław Piliszek |
nominated for series |
|
masakari-monitors/victoria |
|
| 2021-09-18 18:47:15 |
Radosław Piliszek |
bug task added |
|
masakari-monitors/victoria |
|
| 2021-09-18 18:47:15 |
Radosław Piliszek |
nominated for series |
|
masakari-monitors/xena |
|
| 2021-09-18 18:47:15 |
Radosław Piliszek |
bug task added |
|
masakari-monitors/xena |
|
| 2021-09-18 18:47:15 |
Radosław Piliszek |
nominated for series |
|
masakari-monitors/ussuri |
|
| 2021-09-18 18:47:15 |
Radosław Piliszek |
bug task added |
|
masakari-monitors/ussuri |
|
| 2021-09-18 18:47:15 |
Radosław Piliszek |
nominated for series |
|
masakari-monitors/wallaby |
|
| 2021-09-18 18:47:15 |
Radosław Piliszek |
bug task added |
|
masakari-monitors/wallaby |
|
| 2021-09-18 18:47:22 |
Radosław Piliszek |
masakari-monitors/xena: status |
New |
Fix Released |
|
| 2021-09-18 18:47:24 |
Radosław Piliszek |
masakari-monitors/wallaby: status |
New |
Fix Released |
|
| 2021-09-18 18:47:27 |
Radosław Piliszek |
masakari-monitors/ussuri: status |
New |
Fix Released |
|
| 2021-09-18 18:47:31 |
Radosław Piliszek |
masakari-monitors/victoria: status |
New |
Fix Released |
|
| 2021-09-18 18:47:51 |
Radosław Piliszek |
masakari-monitors/xena: importance |
Undecided |
High |
|
| 2021-09-18 18:47:53 |
Radosław Piliszek |
masakari-monitors/wallaby: importance |
Undecided |
High |
|
| 2021-09-18 18:47:54 |
Radosław Piliszek |
masakari-monitors/victoria: importance |
Undecided |
High |
|
| 2021-09-18 18:47:56 |
Radosław Piliszek |
masakari-monitors/ussuri: importance |
Undecided |
High |
|
| 2021-09-18 18:47:58 |
Radosław Piliszek |
masakari-monitors/xena: assignee |
|
Radosław Piliszek (yoctozepto) |
|
| 2021-09-18 18:48:00 |
Radosław Piliszek |
masakari-monitors/wallaby: assignee |
|
Radosław Piliszek (yoctozepto) |
|
| 2021-09-18 18:48:01 |
Radosław Piliszek |
masakari-monitors/victoria: assignee |
|
Radosław Piliszek (yoctozepto) |
|
| 2021-09-18 18:48:04 |
Radosław Piliszek |
masakari-monitors/ussuri: assignee |
|
Radosław Piliszek (yoctozepto) |
|
| 2022-11-22 18:18:15 |
Corey Bryant |
bug task added |
|
masakari-monitors (Ubuntu) |
|
| 2022-11-22 18:18:22 |
Corey Bryant |
nominated for series |
|
Ubuntu Focal |
|
| 2022-11-22 18:18:22 |
Corey Bryant |
bug task added |
|
masakari-monitors (Ubuntu Focal) |
|
| 2022-11-22 18:18:30 |
Corey Bryant |
masakari-monitors (Ubuntu): status |
New |
Fix Released |
|
| 2022-11-22 18:18:32 |
Corey Bryant |
masakari-monitors (Ubuntu Focal): status |
New |
Triaged |
|
| 2022-11-22 18:18:34 |
Corey Bryant |
masakari-monitors (Ubuntu Focal): importance |
Undecided |
High |
|
| 2022-11-22 18:18:40 |
Corey Bryant |
bug task added |
|
cloud-archive |
|
| 2022-11-22 18:18:50 |
Corey Bryant |
nominated for series |
|
cloud-archive/wallaby |
|
| 2022-11-22 18:18:50 |
Corey Bryant |
bug task added |
|
cloud-archive/wallaby |
|
| 2022-11-22 18:18:50 |
Corey Bryant |
nominated for series |
|
cloud-archive/ussuri |
|
| 2022-11-22 18:18:50 |
Corey Bryant |
bug task added |
|
cloud-archive/ussuri |
|
| 2022-11-22 18:18:50 |
Corey Bryant |
nominated for series |
|
cloud-archive/victoria |
|
| 2022-11-22 18:18:50 |
Corey Bryant |
bug task added |
|
cloud-archive/victoria |
|
| 2022-11-22 18:18:59 |
Corey Bryant |
cloud-archive: status |
New |
Fix Released |
|
| 2022-11-22 18:19:11 |
Corey Bryant |
cloud-archive/ussuri: importance |
Undecided |
High |
|
| 2022-11-22 18:19:11 |
Corey Bryant |
cloud-archive/ussuri: status |
New |
Triaged |
|
| 2022-11-22 18:19:20 |
Corey Bryant |
cloud-archive/victoria: importance |
Undecided |
High |
|
| 2022-11-22 18:19:20 |
Corey Bryant |
cloud-archive/victoria: status |
New |
Triaged |
|
| 2022-11-22 18:19:30 |
Corey Bryant |
cloud-archive/wallaby: importance |
Undecided |
High |
|
| 2022-11-22 18:19:30 |
Corey Bryant |
cloud-archive/wallaby: status |
New |
Triaged |
|
| 2022-11-22 18:34:08 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2022-11-22 20:52:48 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2022-11-22 20:53:56 |
Corey Bryant |
cloud-archive/wallaby: status |
Triaged |
Fix Committed |
|
| 2022-11-22 20:53:58 |
Corey Bryant |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-wallaby-needed |
|
| 2022-11-22 20:54:10 |
Corey Bryant |
cloud-archive/victoria: status |
Triaged |
Fix Committed |
|
| 2022-11-22 20:54:12 |
Corey Bryant |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-wallaby-needed |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-victoria-needed verification-wallaby-needed |
|
| 2022-12-06 22:12:22 |
Brian Murray |
masakari-monitors (Ubuntu Focal): status |
Triaged |
Fix Committed |
|
| 2022-12-06 22:12:30 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2022-12-06 22:12:34 |
Brian Murray |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-victoria-needed verification-wallaby-needed |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-needed verification-needed-focal verification-victoria-needed verification-wallaby-needed |
|
| 2023-02-16 13:36:17 |
Corey Bryant |
cloud-archive/ussuri: status |
Triaged |
Fix Committed |
|
| 2023-02-16 13:36:19 |
Corey Bryant |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-needed verification-needed-focal verification-victoria-needed verification-wallaby-needed |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-needed verification-wallaby-needed |
|
| 2023-11-29 12:34:37 |
Edward Hope-Morley |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-needed verification-wallaby-needed |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-needed verification-wallaby-done |
|
| 2023-11-29 14:30:18 |
Corey Bryant |
cloud-archive/wallaby: status |
Fix Committed |
Fix Released |
|
| 2023-12-02 14:16:17 |
Edward Hope-Morley |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-needed verification-wallaby-done |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-done verification-wallaby-done |
|
| 2023-12-04 13:20:07 |
Corey Bryant |
cloud-archive/victoria: status |
Fix Committed |
Fix Released |
|
| 2023-12-06 12:30:01 |
Edward Hope-Morley |
tags |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-needed verification-needed-focal verification-ussuri-needed verification-victoria-done verification-wallaby-done |
in-stable-ussuri in-stable-victoria in-stable-wallaby verification-done verification-done-focal verification-ussuri-needed verification-victoria-done verification-wallaby-done |
|
| 2023-12-07 20:45:44 |
Launchpad Janitor |
masakari-monitors (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2023-12-11 15:02:49 |
Corey Bryant |
cloud-archive/ussuri: status |
Fix Committed |
Fix Released |
|
