| 2022-02-16 15:48:07 |
Nobuto Murata |
bug |
|
|
added bug |
| 2022-02-16 15:49:24 |
Nobuto Murata |
attachment added |
|
bundle_testbed.yaml https://bugs.launchpad.net/charm-octavia/+bug/1961088/+attachment/5561439/+files/bundle_testbed.yaml |
|
| 2022-02-16 15:51:03 |
Nobuto Murata |
attachment added |
|
unit-octavia-0.log https://bugs.launchpad.net/charm-octavia/+bug/1961088/+attachment/5561440/+files/unit-octavia-0.log |
|
| 2022-02-16 15:56:01 |
Nobuto Murata |
attachment added |
|
juju-crashdump-57529923-1796-4bc4-af3a-cdb420579f5e.tar.xz https://bugs.launchpad.net/charm-octavia/+bug/1961088/+attachment/5561443/+files/juju-crashdump-57529923-1796-4bc4-af3a-cdb420579f5e.tar.xz |
|
| 2022-02-16 15:56:22 |
Nobuto Murata |
bug |
|
|
added subscriber Canonical Field High |
| 2022-02-17 12:31:12 |
Nobuto Murata |
description |
Octavia Amphora LBs get stuck at OFFLINE although the actual data plane is working. There seems an issue in updating the LB status in the control plane somewhere. However, after running config-changed hook explicitly everything starts working.
Might be related:
https://bugs.launchpad.net/charm-octavia/+bug/1953639
Restarting octavia related services didn't do the trick but the whole config-changed hook was required.
$ date; openstack loadbalancer list --format yaml
Wed Feb 16 15:15:16 UTC 2022
- id: 092cfccd-cb9f-46b1-961f-e92215c72447
name: openstack-integrator-6b38f382a2fc-kubernetes-master
operating_status: OFFLINE
project_id: ef8e573ad99d45a2b9cf1725a355c0f3
provider: amphora
provisioning_status: ACTIVE
vip_address: 10.5.5.14
^^^ operating_status: OFFLINE
But actual traffic to the backend through the floating IP(192.168.151.61) in front of LB works:
$ openstack floating ip list --fixed-ip 10.5.5.14 --format yaml
- Fixed IP Address: 10.5.5.14
Floating IP Address: 192.168.151.61
Floating Network: b81a6386-d544-4502-8087-901f3012576b
ID: 86189424-7cfc-48b9-9a1f-ff864b24ef49
Port: 872da267-4a7d-42a6-b0b3-bd35c14701e6
Project: ef8e573ad99d45a2b9cf1725a355c0f3
$ kubectl -v6 get nodes
I0216 15:21:03.347246 94802 loader.go:372] Config loaded from file: /home/ubuntu/.kube/config
I0216 15:21:03.376692 94802 round_trippers.go:553] GET https://192.168.151.61:443/api/v1/nodes?limit=500 200 OK in 21 milliseconds
NAME STATUS ROLES AGE VERSION
juju-82a2fc-k8s-on-openstack-1 Ready <none> 37m v1.23.3
Tried to restart some services:
# date; systemctl restart octavia-worker.service
Wed Feb 16 15:29:19 UTC 2022
# date; systemctl restart octavia-health-manager.service
Wed Feb 16 15:31:37 UTC 2022
# date; systemctl restart octavia-housekeeping.service
Wed Feb 16 15:33:48 UTC 2022
# date; systemctl restart apache2.service
Wed Feb 16 15:35:48 UTC 2022
-> no luck
Config-changed hook explicitly:
$ juju run --unit octavia/0 -- hooks/config-changed
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Fetched 336 kB in 2s (157 kB/s)
Reading package lists...
Adding user systemd-network to group octavia
inactive
octavia-api (enabled by site administrator)
active
active
active
active
Synchronizing state of octavia-api.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable octavia-api
Unit /etc/systemd/system/octavia-api.service is masked, ignoring.
Then, LB status gets ONLINE.
$ date; openstack loadbalancer list --format yaml
Wed Feb 16 15:43:15 UTC 2022
- id: 092cfccd-cb9f-46b1-961f-e92215c72447
name: openstack-integrator-6b38f382a2fc-kubernetes-master
operating_status: ONLINE
project_id: ef8e573ad99d45a2b9cf1725a355c0f3
provider: amphora
provisioning_status: ACTIVE
vip_address: 10.5.5.14 |
focal-ussuri
Octavia Amphora LBs get stuck at OFFLINE although the actual data plane is working. There seems an issue in updating the LB status in the control plane somewhere. However, after running config-changed hook explicitly everything starts working.
Might be related:
https://bugs.launchpad.net/charm-octavia/+bug/1953639
Restarting octavia related services didn't do the trick but the whole config-changed hook was required.
$ date; openstack loadbalancer list --format yaml
Wed Feb 16 15:15:16 UTC 2022
- id: 092cfccd-cb9f-46b1-961f-e92215c72447
name: openstack-integrator-6b38f382a2fc-kubernetes-master
operating_status: OFFLINE
project_id: ef8e573ad99d45a2b9cf1725a355c0f3
provider: amphora
provisioning_status: ACTIVE
vip_address: 10.5.5.14
^^^ operating_status: OFFLINE
But actual traffic to the backend through the floating IP(192.168.151.61) in front of LB works:
$ openstack floating ip list --fixed-ip 10.5.5.14 --format yaml
- Fixed IP Address: 10.5.5.14
Floating IP Address: 192.168.151.61
Floating Network: b81a6386-d544-4502-8087-901f3012576b
ID: 86189424-7cfc-48b9-9a1f-ff864b24ef49
Port: 872da267-4a7d-42a6-b0b3-bd35c14701e6
Project: ef8e573ad99d45a2b9cf1725a355c0f3
$ kubectl -v6 get nodes
I0216 15:21:03.347246 94802 loader.go:372] Config loaded from file: /home/ubuntu/.kube/config
I0216 15:21:03.376692 94802 round_trippers.go:553] GET https://192.168.151.61:443/api/v1/nodes?limit=500 200 OK in 21 milliseconds
NAME STATUS ROLES AGE VERSION
juju-82a2fc-k8s-on-openstack-1 Ready <none> 37m v1.23.3
Tried to restart some services:
# date; systemctl restart octavia-worker.service
Wed Feb 16 15:29:19 UTC 2022
# date; systemctl restart octavia-health-manager.service
Wed Feb 16 15:31:37 UTC 2022
# date; systemctl restart octavia-housekeeping.service
Wed Feb 16 15:33:48 UTC 2022
# date; systemctl restart apache2.service
Wed Feb 16 15:35:48 UTC 2022
-> no luck
Config-changed hook explicitly:
$ juju run --unit octavia/0 -- hooks/config-changed
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Fetched 336 kB in 2s (157 kB/s)
Reading package lists...
Adding user systemd-network to group octavia
inactive
octavia-api (enabled by site administrator)
active
active
active
active
Synchronizing state of octavia-api.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable octavia-api
Unit /etc/systemd/system/octavia-api.service is masked, ignoring.
Then, LB status gets ONLINE.
$ date; openstack loadbalancer list --format yaml
Wed Feb 16 15:43:15 UTC 2022
- id: 092cfccd-cb9f-46b1-961f-e92215c72447
name: openstack-integrator-6b38f382a2fc-kubernetes-master
operating_status: ONLINE
project_id: ef8e573ad99d45a2b9cf1725a355c0f3
provider: amphora
provisioning_status: ACTIVE
vip_address: 10.5.5.14 |
|
| 2022-03-09 01:39:52 |
Billy Olsen |
charm-octavia: assignee |
|
Aurelien Lourot (aurelien-lourot) |
|
| 2022-03-14 17:27:21 |
Billy Olsen |
charm-octavia: assignee |
Aurelien Lourot (aurelien-lourot) |
Dmitrii Shcherbakov (dmitriis) |
|
| 2022-05-09 11:40:30 |
Alex Kavanagh |
charm-octavia: status |
New |
Triaged |
|
| 2022-05-09 11:40:42 |
Alex Kavanagh |
charm-octavia: importance |
Undecided |
High |
|
| 2022-06-17 06:59:59 |
Nobuto Murata |
charm-octavia: assignee |
Dmitrii Shcherbakov (dmitriis) |
|
|
| 2022-07-11 13:46:44 |
Alex Kavanagh |
tags |
|
cdo-qa cdo-tempest |
|
| 2022-07-29 06:08:58 |
Vern Hart |
bug |
|
|
added subscriber Vern Hart |
| 2022-09-21 06:49:21 |
Samuel Allan |
charm-octavia: assignee |
|
Samuel Walladge (swalladge) |
|
| 2022-10-11 04:29:10 |
OpenStack Infra |
charm-octavia: status |
Triaged |
In Progress |
|
| 2023-10-03 02:29:56 |
Felipe Reyes |
nominated for series |
|
charm-octavia/yoga |
|
| 2023-10-03 02:29:56 |
Felipe Reyes |
bug task added |
|
charm-octavia/yoga |
|
| 2023-10-03 02:29:56 |
Felipe Reyes |
nominated for series |
|
charm-octavia/zed |
|
| 2023-10-03 02:29:56 |
Felipe Reyes |
bug task added |
|
charm-octavia/zed |
|
| 2023-10-03 02:44:18 |
OpenStack Infra |
charm-octavia: status |
In Progress |
Fix Committed |
|
| 2023-10-03 04:16:51 |
OpenStack Infra |
charm-octavia/yoga: status |
New |
In Progress |
|
| 2023-10-03 04:18:16 |
OpenStack Infra |
charm-octavia/zed: status |
New |
In Progress |
|
| 2023-11-15 02:09:46 |
OpenStack Infra |
charm-octavia/zed: status |
In Progress |
Fix Committed |
|
| 2023-11-15 02:09:50 |
OpenStack Infra |
charm-octavia/yoga: status |
In Progress |
Fix Committed |
|
