Pacemaker resource upgrade tasks compatible with staged upgrade
Add better idempotency checks on editing the pacemaker resources and
fetching and re-tagging new images, which prevents the upgrade from
failing. The latest status after staged upgrade looks like this:
podman container set: galera-bundle [brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-mariadb:pcmklatest] galera-bundle-0 (ocf::heartbeat:galera): Master controller-0 galera-bundle-1 (ocf::heartbeat:galera): Master controller-1 galera-bundle-2 (ocf::heartbeat:galera): Master controller-2
podman container set: rabbitmq-bundle [brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-rabbitmq:pcmklatest] rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Started controller-0 rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Started controller-1
podman container set: redis-bundle [brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-redis:pcmklatest] redis-bundle-0 (ocf::heartbeat:redis): Master controller-0 redis-bundle-1 (ocf::heartbeat:redis): Slave controller-1
ip-192.168.24.8 (ocf::heartbeat:IPaddr2): Started controller-0
ip-10.0.0.106 (ocf::heartbeat:IPaddr2): Started controller-0
ip-172.17.1.16 (ocf::heartbeat:IPaddr2): Started controller-0
ip-172.17.1.23 (ocf::heartbeat:IPaddr2): Started controller-0
ip-172.17.3.11 (ocf::heartbeat:IPaddr2): Started controller-0
ip-172.17.4.25 (ocf::heartbeat:IPaddr2): Started controller-0
podman container set: haproxy-bundle [brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-haproxy:pcmklatest] haproxy-bundle-podman-0 (ocf::heartbeat:podman): Started controller-0 haproxy-bundle-podman-1 (ocf::heartbeat:podman): Started controller-1 haproxy-bundle-podman-2 (ocf::heartbeat:podman): Stopped
podman container: openstack-cinder-volume [brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/rhosp15/openstack-cinder-volume:pcmklatest] openstack-cinder-volume-podman-0 (ocf::heartbeat:podman): Started controller-1
Failed Resource Actions:
* rabbitmq_monitor_10000 on rabbitmq-bundle-0 'unknown error' (1): call=4861, status=Timed Out, exitreason='', last-rc-change='Mon Aug 5 10:37:51 2019', queued=0ms, exec=0ms
* rabbitmq_monitor_10000 on rabbitmq-bundle-1 'unknown error' (1): call=42, status=Timed Out, exitreason='', last-rc-change='Mon Aug 5 10:15:55 2019', queued=0ms, exec=0ms
This indicates that there are still issues we'll need to solve, but at
least the upgrade passes now and we can keep solving the follow-up
issues while the critical upgrade path is unblocked.
Reviewed: https:/ /review. opendev. org/673456 /git.openstack. org/cgit/ openstack/ tripleo- heat-templates/ commit/ ?id=8827e4f7f10 d71027d9be4fbe6 50f9d1beccedeb
Committed: https:/
Submitter: Zuul
Branch: master
commit 8827e4f7f10d710 27d9be4fbe650f9 d1beccedeb
Author: Jiri Stransky <email address hidden>
Date: Tue Jul 30 10:57:54 2019 +0200
Pacemaker resource upgrade tasks compatible with staged upgrade
Add better idempotency checks on editing the pacemaker resources and
fetching and re-tagging new images, which prevents the upgrade from
failing. The latest status after staged upgrade looks like this:
Online: [ controller-0 controller-1 controller-2 ] bundle- 0@controller- 0 galera- bundle- 1@controller- 1 galera- bundle- 2@controller- 2 rabbitmq- bundle- 0@controller- 0 rabbitmq- bundle- 1@controller- 1 redis-bundle- 0@controller- 0 redis-bundle- 1@controller- 1 ]
GuestOnline: [ galera-
Full list of resources:
podman container set: galera-bundle [brew-pulp- docker01. web.prod. ext.phx2. redhat. com:8888/ rhosp15/ openstack- mariadb: pcmklatest]
galera- bundle- 0 (ocf::heartbeat :galera) : Master controller-0
galera- bundle- 1 (ocf::heartbeat :galera) : Master controller-1
galera- bundle- 2 (ocf::heartbeat :galera) : Master controller-2 docker01. web.prod. ext.phx2. redhat. com:8888/ rhosp15/ openstack- rabbitmq: pcmklatest]
rabbitmq- bundle- 0 (ocf::heartbeat :rabbitmq- cluster) : Started controller-0
rabbitmq- bundle- 1 (ocf::heartbeat :rabbitmq- cluster) : Started controller-1 docker01. web.prod. ext.phx2. redhat. com:8888/ rhosp15/ openstack- redis:pcmklates t]
redis-bundle- 0 (ocf::heartbeat :redis) : Master controller-0
redis-bundle- 1 (ocf::heartbeat :redis) : Slave controller-1 192.168. 24.8 (ocf::heartbeat :IPaddr2) : Started controller-0 :IPaddr2) : Started controller-0 :IPaddr2) : Started controller-0 :IPaddr2) : Started controller-0 :IPaddr2) : Started controller-0 :IPaddr2) : Started controller-0 docker01. web.prod. ext.phx2. redhat. com:8888/ rhosp15/ openstack- haproxy: pcmklatest]
haproxy- bundle- podman- 0 (ocf::heartbeat :podman) : Started controller-0
haproxy- bundle- podman- 1 (ocf::heartbeat :podman) : Started controller-1
haproxy- bundle- podman- 2 (ocf::heartbeat :podman) : Stopped cinder- volume [brew-pulp- docker01. web.prod. ext.phx2. redhat. com:8888/ rhosp15/ openstack- cinder- volume: pcmklatest]
openstack- cinder- volume- podman- 0 (ocf::heartbeat :podman) : Started controller-1
podman container set: rabbitmq-bundle [brew-pulp-
podman container set: redis-bundle [brew-pulp-
ip-
ip-10.0.0.106 (ocf::heartbeat
ip-172.17.1.16 (ocf::heartbeat
ip-172.17.1.23 (ocf::heartbeat
ip-172.17.3.11 (ocf::heartbeat
ip-172.17.4.25 (ocf::heartbeat
podman container set: haproxy-bundle [brew-pulp-
podman container: openstack-
Failed Resource Actions: monitor_ 10000 on rabbitmq-bundle-0 'unknown error' (1): call=4861, status=Timed Out, exitreason='',
last-rc- change= 'Mon Aug 5 10:37:51 2019', queued=0ms, exec=0ms monitor_ 10000 on rabbitmq-bundle-1 'unknown error' (1): call=42, status=Timed Out, exitreason='',
last-rc- change= 'Mon Aug 5 10:15:55 2019', queued=0ms, exec=0ms
* rabbitmq_
* rabbitmq_
This indicates that there are still issues we'll need to solve, but at
least the upgrade passes now and we can keep solving the follow-up
issues while the critical upgrade path is unblocked.
Closes-Bug: #1838971 4523de4c52a1873 438c78e972f
Change-Id: I2e88dc34fa5962