Comment 0 for bug 1961088

Octavia Amphora LBs get stuck at OFFLINE although the actual data plane is working. There seems an issue in updating the LB status in the control plane somewhere. However, after running config-changed hook explicitly everything starts working.

Might be related:
https://bugs.launchpad.net/charm-octavia/+bug/1953639

Restarting octavia related services didn't do the trick but the whole config-changed hook was required.

$ date; openstack loadbalancer list --format yaml
Wed Feb 16 15:15:16 UTC 2022
- id: 092cfccd-cb9f-46b1-961f-e92215c72447
  name: openstack-integrator-6b38f382a2fc-kubernetes-master
  operating_status: OFFLINE
  project_id: ef8e573ad99d45a2b9cf1725a355c0f3
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 10.5.5.14

^^^ operating_status: OFFLINE

But actual traffic to the backend through the floating IP(192.168.151.61) in front of LB works:

$ openstack floating ip list --fixed-ip 10.5.5.14 --format yaml
- Fixed IP Address: 10.5.5.14
  Floating IP Address: 192.168.151.61
  Floating Network: b81a6386-d544-4502-8087-901f3012576b
  ID: 86189424-7cfc-48b9-9a1f-ff864b24ef49
  Port: 872da267-4a7d-42a6-b0b3-bd35c14701e6
  Project: ef8e573ad99d45a2b9cf1725a355c0f3

$ kubectl -v6 get nodes
I0216 15:21:03.347246 94802 loader.go:372] Config loaded from file: /home/ubuntu/.kube/config
I0216 15:21:03.376692 94802 round_trippers.go:553] GET https://192.168.151.61:443/api/v1/nodes?limit=500 200 OK in 21 milliseconds
NAME STATUS ROLES AGE VERSION
juju-82a2fc-k8s-on-openstack-1 Ready <none> 37m v1.23.3

Tried to restart some services:
# date; systemctl restart octavia-worker.service
Wed Feb 16 15:29:19 UTC 2022
# date; systemctl restart octavia-health-manager.service
Wed Feb 16 15:31:37 UTC 2022
# date; systemctl restart octavia-housekeeping.service
Wed Feb 16 15:33:48 UTC 2022
# date; systemctl restart apache2.service
Wed Feb 16 15:35:48 UTC 2022
-> no luck

Config-changed hook explicitly:

$ juju run --unit octavia/0 -- hooks/config-changed
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Fetched 336 kB in 2s (157 kB/s)
Reading package lists...
Adding user systemd-network to group octavia
inactive
octavia-api (enabled by site administrator)
active
active
active
active
Synchronizing state of octavia-api.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable octavia-api
Unit /etc/systemd/system/octavia-api.service is masked, ignoring.

Then, LB status gets ONLINE.

$ date; openstack loadbalancer list --format yaml
Wed Feb 16 15:43:15 UTC 2022
- id: 092cfccd-cb9f-46b1-961f-e92215c72447
  name: openstack-integrator-6b38f382a2fc-kubernetes-master
  operating_status: ONLINE
  project_id: ef8e573ad99d45a2b9cf1725a355c0f3
  provider: amphora
  provisioning_status: ACTIVE
  vip_address: 10.5.5.14