Failover of loadbalancer fails when Amphora master is missing

[Hemanth Nakkina]

[Impact]
(from storyboard desciption) Currently if taskflow process is interrupted (during create/update/failover - node is rebooted or service is restarted) - loadbalancer will stuck in PENDING state.
Taskflow provides persistence module which allows to save flows state for recovery https://docs.openstack.org/taskflow/latest/user/persistence.html
Otherwise partially created/updated/deleted resources should be moved to ERROR state when service is up again. (like it is done in Cinder)

[Test Case]

* deploy Openstack with Octavia and 2 compute hosts e.g. ./generate-bundle.sh --use-stable-charms --release train --octavia --num-compute 3
* juju config octavia loadbalancer-topology=ACTIVE_STANDBY
* create ubuntu vm and install apache2 (i.e. listen port 80)
* create loadbalancer with vm as member and floating ip for LB vip
* test connection with: nc -vz LB_FIP 80
* openstack loadbalancer amphora list
* get amphora master vm uuid: openstack loadbalancer amphora show -c compute_id -f value <master>
* openstack server show -c "OS-EXT-SRV-ATTR:host" -f value <master uuid>
* poweroff compute host from previous step
* openstack loadbalancer failover LB_UUID
* wait a few seconds
* openstack loadbalancer amphora list
* Wait until you have one BACKUP and one MASTER
* Test connection with: nc -vz LB_FIP 80

[Regression Potential]
While new failovers have been proven to work properly, this will resolve existing failed failovers which will require setting the LB state from PENDING_UPDATE to ERROR in the database prior to triggering a new failover.

------------------------------------------------------------------------

Tried to failover a loadbalancer that has missing entries of amphora master.
The loadbalancer went to ERROR state.

OpenStack version: Train

The fix is available in upstream as part of the Octavia Failover refactor patches in Train
https://review.opendev.org/#/q/status:merged+project:openstack/octavia+branch:stable/train+topic:failover-refactor

Verified with the upstream patches and it worked.

[Edward Hope-Morley (hopem)]

Task originally tracked in storyboard:

https://storyboard.openstack.org/#!/story/2005072
https://storyboard.openstack.org/#!/story/2007847

Victoria patches are already included in Ubuntu packages for 7.0.0

Ussuri patches to be SRUd: https://review.opendev.org/#/q/status:merged+project:openstack/octavia+branch:stable/ussuri+topic:failover-refactor

Train patches to be SRUd: https://review.opendev.org/#/q/status:merged+project:openstack/octavia+branch:stable/train+topic:failover-refactor

[Nicolas Bock (nicolasbock)]

Verified with the following steps:

1. Create Train deployment with Octavia

./generate-bundle.sh --defaults --use-stable-charms --release train --octavia --num-compute 2

2. Configure Octavia

juju config octavia loadbalancer-topology=ACTIVE_STANDBY
juju config octavia spare-pool-size=2

2. Create Cirros VMs

./tools/instance_launch.sh 2 cirros

3. Create fake webserver on VMs (See https://code.launchpad.net/~nicolasbock/stsstack-bundles/+git/stsstack-bundles/+merge/392344)

./tools/run_fake_webserver.sh

4. Create load balancer

./tools/create_octavia_lb.sh

5. Test load balancer with

curl LB_FIP

6. Shut down a nova-compute that is hosting one of the amphorae. This
   will break the load balancer due to https://storyboard.openstack.org/#!/story/2003084

7. Install SRU in octavia unit

8. Check load balancer with

openstack loadbalancer list

9. Verify operation with step 5.

[Corey Bryant (corey.bryant)]

I'd like to see if we can pick these up in an upstream stable release so I've proposed new upstream releases at https://review.opendev.org/#/c/758606/.

[Corey Bryant (corey.bryant)]

octavia 6.1.0 and 5.0.3 will be included in the following SRUs for Ubuntu:
https:pad.lv/1900477
https:pad.lv/1900476

[Edward Hope-Morley (hopem)]

^^ links don't work, these do:
https://bugs.launchpad.net/cloud-archive/+bug/1900476
https://bugs.launchpad.net/cloud-archive/+bug/1900477

[Corey Bryant (corey.bryant)]

Marked as fix released since the point releases are released.