Bug #1943254 “ovn-dbs update tasks can take place when the clust...” : Bugs : tripleo

Michele Baldessari (michele) on 2021-09-10

tags:	added: train-backport-potential ussuri-backport-potential victoria-backport-potential wallaby-backport-potential removed: wall
Changed in tripleo:
assignee:	nobody → Michele Baldessari (michele)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-10: Fix proposed to tripleo-heat-templates (master)

#1

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808205

Changed in tripleo:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-13: Fix merged to tripleo-heat-templates (master)

#2

Download full text (4.1 KiB)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808205
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ff3173bdcf304dc726f5b03d813fa2194e2c63cc
Submitter: "Zuul (22348)"
Branch: master

commit ff3173bdcf304dc726f5b03d813fa2194e2c63cc
Author: Michele Baldessari <email address hidden>
Date: Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up

    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.

    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-zero retu...

Reviewed:  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808205
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ff3173bdcf304dc726f5b03d813fa2194e2c63cc
Submitter: "Zuul (22348)"
Branch:    master

commit ff3173bdcf304dc726f5b03d813fa2194e2c63cc
Author: Michele Baldessari <michele@acksyn.org>
Date:   Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up
    
    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.
    
    So we can observe something like the following:
    2021-09-10 10:05:13.370339 | 001c2891-506d-f833-ff5a-000000000954 | TASK | Change the bundle operation timeout
    2021-09-10 10:05:14.136798 | 001c2891-506d-f833-ff5a-000000000954 | CHANGED | Change the bundle operation timeout | ovn-db-01
    2021-09-10 10:05:14.137982 | 001c2891-506d-f833-ff5a-000000000954 | TIMING | Change the bundle operation timeout | ovn-db-01 | 0:00:54.808754 | 0.77s
    2021-09-10 10:05:14.146853 | 001c2891-506d-f833-ff5a-000000000956 | TASK | Acquire the cluster shutdown lock to stop pacemaker cluster
    2021-09-10 10:05:14.508085 | 001c2891-506d-f833-ff5a-000000000956 | CHANGED | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01
    2021-09-10 10:05:14.509257 | 001c2891-506d-f833-ff5a-000000000956 | TIMING | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01 | 0:00:55.180032 | 0.36s
    2021-09-10 10:05:14.518668 | 001c2891-506d-f833-ff5a-000000000957 | TASK | Stop pacemaker cluster
    2021-09-10 10:05:18.559627 | 001c2891-506d-f833-ff5a-000000000957 | CHANGED | Stop pacemaker cluster | ovn-db-01
    
    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-zero return code", "rc": 1, "start": "2021-09-10 10:05:19.645901", "stderr": "Error: Unable to forget failed operations of resource: ovn-dbs-bundle\nError connecting to the CIB manager: Transport endpoint is not connected\nError perf
    orming operation: Transport endpoint is not connected", "stderr_lines": ["Error: Unable to forget failed operations of resource: ovn-dbs-bundle", "Error connecting to the CIB manager: Transport endpoint is not connected", "Error performing operation: Tran
    sport endpoint is not connected"], "stdout": "", "stdout_lines": []}
    
    We cannot call pcs resource cleanup at step1, we must call it at step0 so we're
    guaranteed that the cluster is up, no matter how heat/ansible decide to order
    the update_tasks.
    
    Note: This is the short-term less-invasive fix. The mid-long term fix
    should be around verifying that we can now remove those workarounds
    that were implemented for OVN bugs.
    
    Closes-Bug: #1943254
    Change-Id: Idd827f72c0033978db7b9a8ea6acec2086cda961

Changed in tripleo:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-13: Fix proposed to tripleo-heat-templates (stable/wallaby)

#3

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808664

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-14: Fix merged to tripleo-heat-templates (stable/wallaby)

#4

Download full text (4.2 KiB)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808664
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/9cab0d721a572d23243c83c8b6eba9e946f376dd
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 9cab0d721a572d23243c83c8b6eba9e946f376dd
Author: Michele Baldessari <email address hidden>
Date: Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up

    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.

    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-z...

Reviewed:  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808664
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/9cab0d721a572d23243c83c8b6eba9e946f376dd
Submitter: "Zuul (22348)"
Branch:    stable/wallaby

commit 9cab0d721a572d23243c83c8b6eba9e946f376dd
Author: Michele Baldessari <michele@acksyn.org>
Date:   Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up
    
    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.
    
    So we can observe something like the following:
    2021-09-10 10:05:13.370339 | 001c2891-506d-f833-ff5a-000000000954 | TASK | Change the bundle operation timeout
    2021-09-10 10:05:14.136798 | 001c2891-506d-f833-ff5a-000000000954 | CHANGED | Change the bundle operation timeout | ovn-db-01
    2021-09-10 10:05:14.137982 | 001c2891-506d-f833-ff5a-000000000954 | TIMING | Change the bundle operation timeout | ovn-db-01 | 0:00:54.808754 | 0.77s
    2021-09-10 10:05:14.146853 | 001c2891-506d-f833-ff5a-000000000956 | TASK | Acquire the cluster shutdown lock to stop pacemaker cluster
    2021-09-10 10:05:14.508085 | 001c2891-506d-f833-ff5a-000000000956 | CHANGED | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01
    2021-09-10 10:05:14.509257 | 001c2891-506d-f833-ff5a-000000000956 | TIMING | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01 | 0:00:55.180032 | 0.36s
    2021-09-10 10:05:14.518668 | 001c2891-506d-f833-ff5a-000000000957 | TASK | Stop pacemaker cluster
    2021-09-10 10:05:18.559627 | 001c2891-506d-f833-ff5a-000000000957 | CHANGED | Stop pacemaker cluster | ovn-db-01
    
    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-zero return code", "rc": 1, "start": "2021-09-10 10:05:19.645901", "stderr": "Error: Unable to forget failed operations of resource: ovn-dbs-bundle\nError connecting to the CIB manager: Transport endpoint is not connected\nError perf
    orming operation: Transport endpoint is not connected", "stderr_lines": ["Error: Unable to forget failed operations of resource: ovn-dbs-bundle", "Error connecting to the CIB manager: Transport endpoint is not connected", "Error performing operation: Tran
    sport endpoint is not connected"], "stdout": "", "stdout_lines": []}
    
    We cannot call pcs resource cleanup at step1, we must call it at step0 so we're
    guaranteed that the cluster is up, no matter how heat/ansible decide to order
    the update_tasks.
    
    Note: This is the short-term less-invasive fix. The mid-long term fix
    should be around verifying that we can now remove those workarounds
    that were implemented for OVN bugs.
    
    Closes-Bug: #1943254
    Change-Id: Idd827f72c0033978db7b9a8ea6acec2086cda961
    (cherry picked from commit ff3173bdcf304dc726f5b03d813fa2194e2c63cc)

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-14: Fix proposed to tripleo-heat-templates (stable/victoria)

#5

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808855

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-15: Fix merged to tripleo-heat-templates (stable/victoria)

#6

Download full text (4.3 KiB)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808855
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/fd508d1cab9574c2f5ea55dbed295dde08f8f8d8
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit fd508d1cab9574c2f5ea55dbed295dde08f8f8d8
Author: Michele Baldessari <email address hidden>
Date: Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up

    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.

    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-...

Reviewed:  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/808855
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/fd508d1cab9574c2f5ea55dbed295dde08f8f8d8
Submitter: "Zuul (22348)"
Branch:    stable/victoria

commit fd508d1cab9574c2f5ea55dbed295dde08f8f8d8
Author: Michele Baldessari <michele@acksyn.org>
Date:   Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up
    
    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.
    
    So we can observe something like the following:
    2021-09-10 10:05:13.370339 | 001c2891-506d-f833-ff5a-000000000954 | TASK | Change the bundle operation timeout
    2021-09-10 10:05:14.136798 | 001c2891-506d-f833-ff5a-000000000954 | CHANGED | Change the bundle operation timeout | ovn-db-01
    2021-09-10 10:05:14.137982 | 001c2891-506d-f833-ff5a-000000000954 | TIMING | Change the bundle operation timeout | ovn-db-01 | 0:00:54.808754 | 0.77s
    2021-09-10 10:05:14.146853 | 001c2891-506d-f833-ff5a-000000000956 | TASK | Acquire the cluster shutdown lock to stop pacemaker cluster
    2021-09-10 10:05:14.508085 | 001c2891-506d-f833-ff5a-000000000956 | CHANGED | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01
    2021-09-10 10:05:14.509257 | 001c2891-506d-f833-ff5a-000000000956 | TIMING | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01 | 0:00:55.180032 | 0.36s
    2021-09-10 10:05:14.518668 | 001c2891-506d-f833-ff5a-000000000957 | TASK | Stop pacemaker cluster
    2021-09-10 10:05:18.559627 | 001c2891-506d-f833-ff5a-000000000957 | CHANGED | Stop pacemaker cluster | ovn-db-01
    
    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-zero return code", "rc": 1, "start": "2021-09-10 10:05:19.645901", "stderr": "Error: Unable to forget failed operations of resource: ovn-dbs-bundle\nError connecting to the CIB manager: Transport endpoint is not connected\nError perf
    orming operation: Transport endpoint is not connected", "stderr_lines": ["Error: Unable to forget failed operations of resource: ovn-dbs-bundle", "Error connecting to the CIB manager: Transport endpoint is not connected", "Error performing operation: Tran
    sport endpoint is not connected"], "stdout": "", "stdout_lines": []}
    
    We cannot call pcs resource cleanup at step1, we must call it at step0 so we're
    guaranteed that the cluster is up, no matter how heat/ansible decide to order
    the update_tasks.
    
    Note: This is the short-term less-invasive fix. The mid-long term fix
    should be around verifying that we can now remove those workarounds
    that were implemented for OVN bugs.
    
    Closes-Bug: #1943254
    Change-Id: Idd827f72c0033978db7b9a8ea6acec2086cda961
    (cherry picked from commit ff3173bdcf304dc726f5b03d813fa2194e2c63cc)
    (cherry picked from commit 9cab0d721a572d23243c83c8b6eba9e946f376dd)

tags:

added: in-stable-victoria

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-15: Fix proposed to tripleo-heat-templates (stable/ussuri)

#7

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809133

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-15: Fix merged to tripleo-heat-templates (stable/ussuri)

#8

Download full text (4.4 KiB)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809133
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/76226ff2abc5ee9cbe1a2f3c7dc5cceec89abf0d
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 76226ff2abc5ee9cbe1a2f3c7dc5cceec89abf0d
Author: Michele Baldessari <email address hidden>
Date: Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up

    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.

    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-ze...

Reviewed:  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809133
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/76226ff2abc5ee9cbe1a2f3c7dc5cceec89abf0d
Submitter: "Zuul (22348)"
Branch:    stable/ussuri

commit 76226ff2abc5ee9cbe1a2f3c7dc5cceec89abf0d
Author: Michele Baldessari <michele@acksyn.org>
Date:   Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up
    
    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.
    
    So we can observe something like the following:
    2021-09-10 10:05:13.370339 | 001c2891-506d-f833-ff5a-000000000954 | TASK | Change the bundle operation timeout
    2021-09-10 10:05:14.136798 | 001c2891-506d-f833-ff5a-000000000954 | CHANGED | Change the bundle operation timeout | ovn-db-01
    2021-09-10 10:05:14.137982 | 001c2891-506d-f833-ff5a-000000000954 | TIMING | Change the bundle operation timeout | ovn-db-01 | 0:00:54.808754 | 0.77s
    2021-09-10 10:05:14.146853 | 001c2891-506d-f833-ff5a-000000000956 | TASK | Acquire the cluster shutdown lock to stop pacemaker cluster
    2021-09-10 10:05:14.508085 | 001c2891-506d-f833-ff5a-000000000956 | CHANGED | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01
    2021-09-10 10:05:14.509257 | 001c2891-506d-f833-ff5a-000000000956 | TIMING | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01 | 0:00:55.180032 | 0.36s
    2021-09-10 10:05:14.518668 | 001c2891-506d-f833-ff5a-000000000957 | TASK | Stop pacemaker cluster
    2021-09-10 10:05:18.559627 | 001c2891-506d-f833-ff5a-000000000957 | CHANGED | Stop pacemaker cluster | ovn-db-01
    
    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-zero return code", "rc": 1, "start": "2021-09-10 10:05:19.645901", "stderr": "Error: Unable to forget failed operations of resource: ovn-dbs-bundle\nError connecting to the CIB manager: Transport endpoint is not connected\nError perf
    orming operation: Transport endpoint is not connected", "stderr_lines": ["Error: Unable to forget failed operations of resource: ovn-dbs-bundle", "Error connecting to the CIB manager: Transport endpoint is not connected", "Error performing operation: Tran
    sport endpoint is not connected"], "stdout": "", "stdout_lines": []}
    
    We cannot call pcs resource cleanup at step1, we must call it at step0 so we're
    guaranteed that the cluster is up, no matter how heat/ansible decide to order
    the update_tasks.
    
    Note: This is the short-term less-invasive fix. The mid-long term fix
    should be around verifying that we can now remove those workarounds
    that were implemented for OVN bugs.
    
    Closes-Bug: #1943254
    Change-Id: Idd827f72c0033978db7b9a8ea6acec2086cda961
    (cherry picked from commit ff3173bdcf304dc726f5b03d813fa2194e2c63cc)
    (cherry picked from commit 9cab0d721a572d23243c83c8b6eba9e946f376dd)
    (cherry picked from commit fd508d1cab9574c2f5ea55dbed295dde08f8f8d8)

tags:

added: in-stable-ussuri

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-15: Fix proposed to tripleo-heat-templates (stable/train)

#9

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809273

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-09-16: Fix merged to tripleo-heat-templates (stable/train)

#10

Download full text (4.6 KiB)

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809273
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/5382da5957408a3c020ecd8cb24d0556eb13b680
Submitter: "Zuul (22348)"
Branch: stable/train

commit 5382da5957408a3c020ecd8cb24d0556eb13b680
Author: Michele Baldessari <email address hidden>
Date: Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up

    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.

    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-zer...

Reviewed:  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/809273
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/5382da5957408a3c020ecd8cb24d0556eb13b680
Submitter: "Zuul (22348)"
Branch:    stable/train

commit 5382da5957408a3c020ecd8cb24d0556eb13b680
Author: Michele Baldessari <michele@acksyn.org>
Date:   Fri Sep 10 14:34:34 2021 +0200

Guarantee that ovn-dbs-pcmk update_tasks are run when the cluster is up
    
    On particular role compositions, the code joining the update_tasks might order
    things differently then on a typical 3ctrl control plane and the ovn-dbs tasks
    at step1 (which require the cluster to be up) will happen after the pacemaker
    task at step1 which stops the cluster.
    
    So we can observe something like the following:
    2021-09-10 10:05:13.370339 | 001c2891-506d-f833-ff5a-000000000954 | TASK | Change the bundle operation timeout
    2021-09-10 10:05:14.136798 | 001c2891-506d-f833-ff5a-000000000954 | CHANGED | Change the bundle operation timeout | ovn-db-01
    2021-09-10 10:05:14.137982 | 001c2891-506d-f833-ff5a-000000000954 | TIMING | Change the bundle operation timeout | ovn-db-01 | 0:00:54.808754 | 0.77s
    2021-09-10 10:05:14.146853 | 001c2891-506d-f833-ff5a-000000000956 | TASK | Acquire the cluster shutdown lock to stop pacemaker cluster
    2021-09-10 10:05:14.508085 | 001c2891-506d-f833-ff5a-000000000956 | CHANGED | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01
    2021-09-10 10:05:14.509257 | 001c2891-506d-f833-ff5a-000000000956 | TIMING | Acquire the cluster shutdown lock to stop pacemaker cluster | ovn-db-01 | 0:00:55.180032 | 0.36s
    2021-09-10 10:05:14.518668 | 001c2891-506d-f833-ff5a-000000000957 | TASK | Stop pacemaker cluster
    2021-09-10 10:05:18.559627 | 001c2891-506d-f833-ff5a-000000000957 | CHANGED | Stop pacemaker cluster | ovn-db-01
    
    2021-09-10 10:05:18.560561 | 001c2891-506d-f833-ff5a-000000000957 | TIMING | Stop pacemaker cluster | ovn-db-01 | 0:00:59.231336 | 4.04s
    2021-09-10 10:05:18.569161 | 001c2891-506d-f833-ff5a-000000000958 | TASK | Start pacemaker cluster
    2021-09-10 10:05:18.627924 | 001c2891-506d-f833-ff5a-000000000958 | SKIPPED | Start pacemaker cluster | ovn-db-01
    2021-09-10 10:05:18.628678 | 001c2891-506d-f833-ff5a-000000000958 | TIMING | Start pacemaker cluster | ovn-db-01 | 0:00:59.299453 | 0.06s
    2021-09-10 10:05:18.637292 | 001c2891-506d-f833-ff5a-000000000959 | TASK | Release the cluster shutdown lock
    2021-09-10 10:05:18.694945 | 001c2891-506d-f833-ff5a-000000000959 | SKIPPED | Release the cluster shutdown lock | ovn-db-01
    2021-09-10 10:05:18.695717 | 001c2891-506d-f833-ff5a-000000000959 | TIMING | Release the cluster shutdown lock | ovn-db-01 | 0:00:59.366493 | 0.06s
    2021-09-10 10:05:18.704368 | 001c2891-506d-f833-ff5a-00000000095a | TASK | Clear ovndb cluster pacemaker error
    2021-09-10 10:05:19.368816 | 001c2891-506d-f833-ff5a-00000000095a | FATAL | Clear ovndb cluster pacemaker error | ovn-db-01 | error={"changed": true, "cmd": "pcs resource cleanup ovn-dbs-bundle", "delta": "0:00:00.399084", "end": "2021-09-10 10:05:20
    .044985", "msg": "non-zero return code", "rc": 1, "start": "2021-09-10 10:05:19.645901", "stderr": "Error: Unable to forget failed operations of resource: ovn-dbs-bundle\nError connecting to the CIB manager: Transport endpoint is not connected\nError perf
    orming operation: Transport endpoint is not connected", "stderr_lines": ["Error: Unable to forget failed operations of resource: ovn-dbs-bundle", "Error connecting to the CIB manager: Transport endpoint is not connected", "Error performing operation: Tran
    sport endpoint is not connected"], "stdout": "", "stdout_lines": []}
    
    We cannot call pcs resource cleanup at step1, we must call it at step0 so we're
    guaranteed that the cluster is up, no matter how heat/ansible decide to order
    the update_tasks.
    
    Note: This is the short-term less-invasive fix. The mid-long term fix
    should be around verifying that we can now remove those workarounds
    that were implemented for OVN bugs.
    
    NB: The train backport has a slight change because the code to move to
    HA by default which has the ' Tear-down non-HA ovn-dbs containers' tasks
    is not present here so no blocks moving around are needed.
    
    Closes-Bug: #1943254
    Change-Id: Idd827f72c0033978db7b9a8ea6acec2086cda961
    (cherry picked from commit ff3173bdcf304dc726f5b03d813fa2194e2c63cc)
    (cherry picked from commit 9cab0d721a572d23243c83c8b6eba9e946f376dd)
    (cherry picked from commit fd508d1cab9574c2f5ea55dbed295dde08f8f8d8)

tags:

added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-18: Fix included in openstack/tripleo-heat-templates 15.1.0

#11

This issue was fixed in the openstack/tripleo-heat-templates 15.1.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-03-10: Fix included in openstack/tripleo-heat-templates 12.4.6

#12

This issue was fixed in the openstack/tripleo-heat-templates 12.4.6 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-21: Fix included in openstack/tripleo-heat-templates 13.6.0

#13

This issue was fixed in the openstack/tripleo-heat-templates 13.6.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-02-06: Fix included in openstack/tripleo-heat-templates train-eol

#14

This issue was fixed in the openstack/tripleo-heat-templates train-eol release.

tripleo

ovn-dbs update tasks can take place when the cluster is down

Bug Description

Other bug subscribers

Remote bug watches