ovn-dbs update tasks can take place when the cluster is down
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
High
|
Michele Baldessari |
Bug Description
On particular role compositions, the code joining the update_tasks might order things differently then on a typical 3ctrl control plane and the ovn-dbs tasks at step1 (which require the cluster to be up) will happen after the pacemaker task at step1 which stops the cluster.
So we can observe something like the following:
2021-09-10 10:05:13.370339 | 001c2891-
2021-09-10 10:05:14.136798 | 001c2891-
2021-09-10 10:05:14.137982 | 001c2891-
2021-09-10 10:05:14.146853 | 001c2891-
2021-09-10 10:05:14.508085 | 001c2891-
2021-09-10 10:05:14.509257 | 001c2891-
2021-09-10 10:05:14.518668 | 001c2891-
2021-09-10 10:05:18.559627 | 001c2891-
2021-09-10 10:05:18.560561 | 001c2891-
2021-09-10 10:05:18.569161 | 001c2891-
2021-09-10 10:05:18.627924 | 001c2891-
2021-09-10 10:05:18.628678 | 001c2891-
2021-09-10 10:05:18.637292 | 001c2891-
2021-09-10 10:05:18.694945 | 001c2891-
2021-09-10 10:05:18.695717 | 001c2891-
2021-09-10 10:05:18.704368 | 001c2891-
2021-09-10 10:05:19.368816 | 001c2891-
.044985", "msg": "non-zero return code", "rc": 1, "start": "2021-09-10 10:05:19.645901", "stderr": "Error: Unable to forget failed operations of resource: ovn-dbs-
orming operation: Transport endpoint is not connected", "stderr_lines": ["Error: Unable to forget failed operations of resource: ovn-dbs-bundle", "Error connecting to the CIB manager: Transport endpoint is not connected", "Error performing operation: Tran
sport endpoint is not connected"], "stdout": "", "stdout_lines": []}
We cannot call pcs resource cleanup at step1, we must call it at step0 so we're guaranteed that the cluster is up, no matter how heat/ansible decide to order the update_tasks
tags: |
added: train-backport-potential ussuri-backport-potential victoria-backport-potential wallaby-backport-potential removed: wall |
Changed in tripleo: | |
assignee: | nobody → Michele Baldessari (michele) |
Fix proposed to branch: master /review. opendev. org/c/openstack /tripleo- heat-templates/ +/808205
Review: https:/