octavia

2023.2 Code Patched with: https://opendev.org/openstack/octavia/commit/7bb6096eccc7966bd963a0529dc7b352246dfdbb - can not create a new LB

Bug #2061268 reported by Noel Ashford
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
octavia
New
Undecided
Unassigned

Bug Description

root@5net:~# tail -f /var/log/kolla/octavia/octavia-*.log
==> /var/log/kolla/octavia/octavia-api-access.log <==
192.168.5.99 - - [14/Apr/2024:02:11:05 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 14403 "-" "gophercloud/2.0.0"
- - - [14/Apr/2024:02:11:05 -0400] "GET / HTTP/1.1" 200 3978 2490 "-" "curl-healthcheck"
192.168.5.99 - - [14/Apr/2024:02:11:09 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 35714 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:14 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 34457 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:15 -0400] "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 61885 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:15 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 15925 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:19 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 80599 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:24 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 80288 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:25 -0400] "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 122145 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:25 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 13787 "-" "gophercloud/2.0.0"

==> /var/log/kolla/octavia/octavia-api-error.log <==
2024-04-14 02:02:45.880226 warnings.warn(deprecated_msg)
2024-04-14 02:02:45.880255 /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_policy/policy.py:810: UserWarning: Policy "load-balancer:member_and_owner":"role:load-balancer_member and rule:load-balancer:owner" was deprecated in W in favor of "load-balancer:member_and_owner":"role:load-balancer_member and rule:project-member". Reason: The Octavia API now requires the OpenStack default roles and scoped tokens. See https://docs.openstack.org/octavia/latest/configuration/policy.html and https://docs.openstack.org/keystone/latest/contributor/services.html#reusable-default-roles for more information.. Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually.
2024-04-14 02:02:45.880260 warnings.warn(deprecated_msg)
2024-04-14 02:02:59.447984 Deprecated: Option "ca_certificates_file" from group "neutron" is deprecated for removal (The cafile option defined by keystoneauth1 is the new name for this option.). Its value may be silently ignored in the future.
2024-04-14 02:03:00.082530 /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_policy/policy.py:810: UserWarning: Policy "context_is_admin":"role:admin or role:load-balancer_admin" was deprecated in W in favor of "context_is_admin":"role:load-balancer_admin or rule:system-admin or role:admin". Reason: The Octavia API now requires the OpenStack default roles and scoped tokens. See https://docs.openstack.org/octavia/latest/configuration/policy.html and https://docs.openstack.org/keystone/latest/contributor/services.html#reusable-default-roles for more information.. Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually.
2024-04-14 02:03:00.082554 warnings.warn(deprecated_msg)
2024-04-14 02:03:00.082595 /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_policy/policy.py:810: UserWarning: Policy "load-balancer:observer_and_owner":"role:load-balancer_observer and rule:load-balancer:owner" was deprecated in W in favor of "load-balancer:observer_and_owner":"role:load-balancer_observer and rule:project-reader". Reason: The Octavia API now requires the OpenStack default roles and scoped tokens. See https://docs.openstack.org/octavia/latest/configuration/policy.html and https://docs.openstack.org/keystone/latest/contributor/services.html#reusable-default-roles for more information.. Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually.
2024-04-14 02:03:00.082614 warnings.warn(deprecated_msg)
2024-04-14 02:03:00.082643 /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_policy/policy.py:810: UserWarning: Policy "load-balancer:member_and_owner":"role:load-balancer_member and rule:load-balancer:owner" was deprecated in W in favor of "load-balancer:member_and_owner":"role:load-balancer_member and rule:project-member". Reason: The Octavia API now requires the OpenStack default roles and scoped tokens. See https://docs.openstack.org/octavia/latest/configuration/policy.html and https://docs.openstack.org/keystone/latest/contributor/services.html#reusable-default-roles for more information.. Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually.
2024-04-14 02:03:00.082649 warnings.warn(deprecated_msg)

==> /var/log/kolla/octavia/octavia-api.log <==
2024-04-14 02:02:59.558 733 INFO octavia.api.app [-] oslo_messaging_notifications.topics = ['notifications']
2024-04-14 02:02:59.558 733 INFO octavia.api.app [-] oslo_messaging_notifications.transport_url = ****
2024-04-14 02:02:59.558 733 INFO octavia.api.app [-] ********************************************************************************
2024-04-14 02:02:59.759 733 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
2024-04-14 02:09:18.979 735 WARNING openstack [None req-dd0763f2-f57d-49ef-b956-aab069076211 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'block-storage': Encountered an exception attempting to process config for project 'cinder' (service type 'block-storage'): no such option valid_interfaces in group [cinder]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [cinder]
2024-04-14 02:09:18.979 735 WARNING openstack [None req-dd0763f2-f57d-49ef-b956-aab069076211 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'compute': Encountered an exception attempting to process config for project 'nova' (service type 'compute'): no such option valid_interfaces in group [nova]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [nova]
2024-04-14 02:09:18.980 735 WARNING openstack [None req-dd0763f2-f57d-49ef-b956-aab069076211 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'image': Encountered an exception attempting to process config for project 'glance' (service type 'image'): no such option valid_interfaces in group [glance]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [glance]
2024-04-14 02:10:11.649 735 WARNING openstack [None req-3a997194-acf1-4f44-9916-ea04569c7b6f - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'block-storage': Encountered an exception attempting to process config for project 'cinder' (service type 'block-storage'): no such option valid_interfaces in group [cinder]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [cinder]
2024-04-14 02:10:11.649 735 WARNING openstack [None req-3a997194-acf1-4f44-9916-ea04569c7b6f - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'compute': Encountered an exception attempting to process config for project 'nova' (service type 'compute'): no such option valid_interfaces in group [nova]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [nova]
2024-04-14 02:10:11.650 735 WARNING openstack [None req-3a997194-acf1-4f44-9916-ea04569c7b6f - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'image': Encountered an exception attempting to process config for project 'glance' (service type 'image'): no such option valid_interfaces in group [glance]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [glance]

==> /var/log/kolla/octavia/octavia-health-manager.log <==
2024-04-14 01:44:51.574 7 INFO octavia.cmd.health_manager [-] Health manager check process starts:
2024-04-14 01:44:51.576 1008 INFO octavia.amphorae.drivers.health.heartbeat_udp [-] attempting to listen on 192.168.5.1 port 5555
2024-04-14 01:59:43.557 7 INFO octavia.cmd.health_manager [-] Health Manager exiting due to signal
2024-04-14 01:59:44.406 1008 INFO octavia.cmd.health_manager [-] Waiting for executor to shutdown...
2024-04-14 01:59:44.407 1008 INFO octavia.cmd.health_manager [-] Executor shutdown finished.
2024-04-14 02:02:43.810 7 INFO octavia.common.config [-] Logging enabled!
2024-04-14 02:02:43.810 7 INFO octavia.common.config [-] /var/lib/kolla/venv/bin/octavia-health-manager version 13.0.1.dev20
2024-04-14 02:02:43.856 7 INFO octavia.cmd.health_manager [-] Health Manager listener process starts:
2024-04-14 02:02:43.860 7 INFO octavia.cmd.health_manager [-] Health manager check process starts:
2024-04-14 02:02:43.861 731 INFO octavia.amphorae.drivers.health.heartbeat_udp [-] attempting to listen on 192.168.5.1 port 5555

==> /var/log/kolla/octavia/octavia-housekeeping.log <==
2024-04-14 01:44:52.907 7 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2024-04-14 02:02:40.900 7 INFO octavia.cmd.house_keeping [-] Attempting to gracefully terminate House-Keeping
2024-04-14 02:02:40.901 7 INFO octavia.cmd.house_keeping [-] House-Keeping process terminated
2024-04-14 02:02:45.736 7 INFO octavia.common.config [-] Logging enabled!
2024-04-14 02:02:45.736 7 INFO octavia.common.config [-] /var/lib/kolla/venv/bin/octavia-housekeeping version 13.0.1.dev20
2024-04-14 02:02:45.839 7 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2024-04-14 06:02:45.839614
2024-04-14 02:02:45.840 7 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec
2024-04-14 02:02:45.840 7 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds
2024-04-14 02:02:45.840 7 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2024-04-14 02:02:45.840 7 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec

==> /var/log/kolla/octavia/octavia-worker.log <==
2024-04-14 01:45:01.876 731 INFO octavia.controller.queue.v2.consumer [-] Starting V2 consumer...
2024-04-14 02:02:47.138 7 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process
2024-04-14 02:02:47.239 731 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [731]
2024-04-14 02:02:47.239 731 INFO octavia.controller.queue.v2.consumer [-] Stopping V2 consumer...
2024-04-14 02:02:51.980 731 INFO octavia.controller.queue.v2.consumer [-] V2 Consumer successfully stopped. Waiting for final messages to be processed...
2024-04-14 02:02:51.985 731 WARNING amqp [-] Received method (60, 30) during closing channel 1. This method will be ignored
2024-04-14 02:02:51.986 731 INFO octavia.controller.queue.v2.consumer [-] Shutting down V2 endpoint worker executors...
2024-04-14 02:02:56.757 7 INFO octavia.common.config [-] Logging enabled!
2024-04-14 02:02:56.757 7 INFO octavia.common.config [-] /var/lib/kolla/venv/bin/octavia-worker version 13.0.1.dev20
2024-04-14 02:02:56.843 731 INFO octavia.controller.queue.v2.consumer [-] Starting V2 consumer...

==> /var/log/kolla/octavia/octavia-api-access.log <==
192.168.5.99 - - [14/Apr/2024:02:11:29 -0400] "GET /v2.0/lbaas/availabilityzones HTTP/1.1" 200 58 39504 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:29 -0400] "GET /v2.0/lbaas/flavors HTTP/1.1" 200 36 96963 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:29 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 37895 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:34 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 38753 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:35 -0400] "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 149652 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:35 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 12304 "-" "gophercloud/2.0.0"
- - - [14/Apr/2024:02:11:35 -0400] "GET / HTTP/1.1" 200 3978 2702 "-" "curl-healthcheck"
192.168.5.99 - - [14/Apr/2024:02:11:36 -0400] "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 58485 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:36 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 17650 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:39 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 36406 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"

==> /var/log/kolla/octavia/octavia-api.log <==
2024-04-14 02:11:42.134 736 WARNING openstack [None req-f4cdd978-d9ce-4cd3-ba91-6fff2d85ea99 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'block-storage': Encountered an exception attempting to process config for project 'cinder' (service type 'block-storage'): no such option valid_interfaces in group [cinder]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [cinder]
2024-04-14 02:11:42.134 736 WARNING openstack [None req-f4cdd978-d9ce-4cd3-ba91-6fff2d85ea99 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'compute': Encountered an exception attempting to process config for project 'nova' (service type 'compute'): no such option valid_interfaces in group [nova]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [nova]
2024-04-14 02:11:42.134 736 WARNING openstack [None req-f4cdd978-d9ce-4cd3-ba91-6fff2d85ea99 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'image': Encountered an exception attempting to process config for project 'glance' (service type 'image'): no such option valid_interfaces in group [glance]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [glance]

Also, separate issue - prior to this, i had additional patches from that had it up but when i rebooted, it totally lost state and was not recoverable. It sat in a pending state forever.

Revision history for this message
Gregory Thiemonge (gthiemonge) wrote :

Can you share the [nova] [cinder] [glance] sections of your octavia configuration?

Revision history for this message
Noel Ashford (nashford77) wrote : Re: [Bug 2061268] Re: 2023.2 Code Patched with: https://opendev.org/openstack/octavia/commit/7bb6096eccc7966bd963a0529dc7b352246dfdbb - can not create a new LB
Download full text (16.6 KiB)

I have tried to switch it to valid_interfaces prior, but it did not work. I
also tried to patch code myself, but did not succeed as I don't understand
how Octavia works fully.

[glance]
region_name = 5Net
endpoint_type = internal
ca_certificates_file = /etc/ssl/certs/ca-certificates.crt

[neutron]
region_name = 5Net
valid_interfaces = internal
ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
endpoint_override = https://cloud.5net.lan.tunninet.com:9696

[nova]
region_name = 5Net
endpoint_type = internal
ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
availability_zone = 5Net

[cinder]
availability_zone = 5Net
endpoint_type = internal
ᐧ

On Mon, Apr 15, 2024 at 3:25 AM Gregory Thiemonge <
<email address hidden>> wrote:

> Can you share the [nova] [cinder] [glance] sections of your octavia
> configuration?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2061268
>
> Title:
> 2023.2 Code Patched with:
>
> https://opendev.org/openstack/octavia/commit/7bb6096eccc7966bd963a0529dc7b352246dfdbb
> - can not create a new LB
>
> Status in octavia:
> New
>
> Bug description:
> root@5net:~# tail -f /var/log/kolla/octavia/octavia-*.log
> ==> /var/log/kolla/octavia/octavia-api-access.log <==
> 192.168.5.99 - - [14/Apr/2024:02:11:05 -0400] "GET
> /v2.0/octavia/amphorae HTTP/1.1" 200 38 14403 "-" "gophercloud/2.0.0"
> - - - [14/Apr/2024:02:11:05 -0400] "GET / HTTP/1.1" 200 3978 2490 "-"
> "curl-healthcheck"
> 192.168.5.99 - - [14/Apr/2024:02:11:09 -0400] "GET
> /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b
> HTTP/1.1" 200 48 35714 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0
> python-requests/2.28.2 CPython/3.10.12"
> 192.168.5.99 - - [14/Apr/2024:02:11:14 -0400] "GET
> /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b
> HTTP/1.1" 200 48 34457 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0
> python-requests/2.28.2 CPython/3.10.12"
> 192.168.5.99 - - [14/Apr/2024:02:11:15 -0400] "GET
> /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 61885 "-" "gophercloud/2.0.0"
> 192.168.5.99 - - [14/Apr/2024:02:11:15 -0400] "GET
> /v2.0/octavia/amphorae HTTP/1.1" 200 38 15925 "-" "gophercloud/2.0.0"
> 192.168.5.99 - - [14/Apr/2024:02:11:19 -0400] "GET
> /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b
> HTTP/1.1" 200 48 80599 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0
> python-requests/2.28.2 CPython/3.10.12"
> 192.168.5.99 - - [14/Apr/2024:02:11:24 -0400] "GET
> /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b
> HTTP/1.1" 200 48 80288 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0
> python-requests/2.28.2 CPython/3.10.12"
> 192.168.5.99 - - [14/Apr/2024:02:11:25 -0400] "GET
> /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 122145 "-" "gophercloud/2.0.0"
> 192.168.5.99 - - [14/Apr/2024:02:11:25 -0400] "GET
> /v2.0/octavia/amphorae HTTP/1.1" 200 38 13787 "-" "gophercloud/2.0.0"
>
> ==> /var/log/kolla/octavia/octavia-api-error.log <==
> 2024-04-14 02:02:45.880226 warnings.warn(deprecated_msg)
> 2024-04-14 02:02:45.880255
> /var/lib/kolla/venv/lib/python3.10/site-packages/os...

Revision history for this message
Noel Ashford (nashford77) wrote :
Download full text (17.5 KiB)

Separate from this, when i use magnum to create a LB for master etc., i
reboot that node and it gets stuck in pending after it was up prior and i
can not get it "un-stuck" I saw nothing in the logs, and yes jobboard is
setup.

[task_flow]
persistence_connection = mysql+pymysql://octavia_persistence:<hidden>@
cloud.5net.lan.tunninet.com:3306/octavia_persistence
jobboard_enabled = true
jobboard_backend_password = <hidden>
jobboard_backend_port = 6379
jobboard_backend_hosts = 192.168.5.1
ᐧ

On Tue, Apr 16, 2024 at 11:57 AM Noel Ashford <email address hidden> wrote:

> I have tried to switch it to valid_interfaces prior, but it did not work.
> I also tried to patch code myself, but did not succeed as I
> don't understand how Octavia works fully.
>
> [glance]
> region_name = 5Net
> endpoint_type = internal
> ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
>
> [neutron]
> region_name = 5Net
> valid_interfaces = internal
> ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
> endpoint_override = https://cloud.5net.lan.tunninet.com:9696
>
> [nova]
> region_name = 5Net
> endpoint_type = internal
> ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
> availability_zone = 5Net
>
> [cinder]
> availability_zone = 5Net
> endpoint_type = internal
> ᐧ
>
> On Mon, Apr 15, 2024 at 3:25 AM Gregory Thiemonge <
> <email address hidden>> wrote:
>
>> Can you share the [nova] [cinder] [glance] sections of your octavia
>> configuration?
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/2061268
>>
>> Title:
>> 2023.2 Code Patched with:
>>
>> https://opendev.org/openstack/octavia/commit/7bb6096eccc7966bd963a0529dc7b352246dfdbb
>> - can not create a new LB
>>
>> Status in octavia:
>> New
>>
>> Bug description:
>> root@5net:~# tail -f /var/log/kolla/octavia/octavia-*.log
>> ==> /var/log/kolla/octavia/octavia-api-access.log <==
>> 192.168.5.99 - - [14/Apr/2024:02:11:05 -0400] "GET
>> /v2.0/octavia/amphorae HTTP/1.1" 200 38 14403 "-" "gophercloud/2.0.0"
>> - - - [14/Apr/2024:02:11:05 -0400] "GET / HTTP/1.1" 200 3978 2490 "-"
>> "curl-healthcheck"
>> 192.168.5.99 - - [14/Apr/2024:02:11:09 -0400] "GET
>> /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b
>> HTTP/1.1" 200 48 35714 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0
>> python-requests/2.28.2 CPython/3.10.12"
>> 192.168.5.99 - - [14/Apr/2024:02:11:14 -0400] "GET
>> /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b
>> HTTP/1.1" 200 48 34457 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0
>> python-requests/2.28.2 CPython/3.10.12"
>> 192.168.5.99 - - [14/Apr/2024:02:11:15 -0400] "GET
>> /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 61885 "-" "gophercloud/2.0.0"
>> 192.168.5.99 - - [14/Apr/2024:02:11:15 -0400] "GET
>> /v2.0/octavia/amphorae HTTP/1.1" 200 38 15925 "-" "gophercloud/2.0.0"
>> 192.168.5.99 - - [14/Apr/2024:02:11:19 -0400] "GET
>> /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b
>> HTTP/1.1" 200 48 80599 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0
>> python-requests/2.28.2 CPython/3.10.12"
>> 192.168.5.99 - - [14/Apr/2024:02:11...

Revision history for this message
Noel Ashford (nashford77) wrote :
Download full text (6.5 KiB)

Also, this was all up and perfect prior, i rebooted .... Dead as a doornail (See attachment)

2024-04-18 02:35:38.888 743 INFO octavia.controller.worker.v2.tasks.database_tasks [-] Mark ALLOCATED in DB for amphora: 90ca8204-23d5-466b-a203-22717e2a429d with compute id: bd31af56-96f0-47a3-8168-e970148c5dd6 for load balancer: 17e60f7c-4b12-439c-824e-674ec4561e97
2024-04-18 02:35:39.545 743 INFO octavia.controller.worker.v2.tasks.network_tasks [-] Created port octavia-lb-vrrp-90ca8204-23d5-466b-a203-22717e2a429d with ID 2a11b813-3961-4373-a25f-add98bd8e58c for amphora 90ca8204-23d5-466b-a203-22717e2a429d
2024-04-18 02:35:56.873 743 INFO octavia.controller.worker.v2.tasks.database_tasks [-] Mark ACTIVE in DB for load balancer id: 17e60f7c-4b12-439c-824e-674ec4561e97
2024-04-18 02:35:56.964 743 INFO octavia.controller.worker.v2.flows.amphora_flows [-] Performing failover for amphora: {'id': '25e017ff-fb56-4467-8eda-a76605f39174', 'load_balancer_id': '9c2979fb-ed59-4164-8d95-f0e3ed2369dc', 'lb_network_ip': '10.5.1.201', 'compute_id': 'bfa596e6-ea72-49ac-8afa-1a32cfb7629f', 'role': 'standalone'}
2024-04-18 02:35:57.414 743 INFO octavia.controller.worker.v2.tasks.database_tasks [-] Created Amphora 8d5feed9-1ff5-4fd3-ae4f-7f050bda1555 in DB for load balancer 9c2979fb-ed59-4164-8d95-f0e3ed2369dc
2024-04-18 02:35:57.570 743 INFO octavia.certificates.generator.local [-] Signing a certificate request using OpenSSL locally.
2024-04-18 02:35:57.571 743 INFO octavia.certificates.generator.local [-] Using CA Certificate from config.
2024-04-18 02:35:57.571 743 INFO octavia.certificates.generator.local [-] Using CA Private Key from config.
2024-04-18 02:35:57.571 743 INFO octavia.certificates.generator.local [-] Using CA Private Key Passphrase from config.
2024-04-18 02:35:58.972 743 INFO octavia.controller.worker.v2.tasks.compute_tasks [-] Server created with id: 952051df-26a6-49c5-bf33-cba192f819ad for amphora id: 8d5feed9-1ff5-4fd3-ae4f-7f050bda1555
2024-04-18 02:36:02.011 743 WARNING octavia.common.base_taskflow [-] Task 'octavia-failover-loadbalancer-flow-octavia-failover-loadbalancer-flow-octavia-create-amp-for-lb-subflow-octavia-compute-wait' (253ab4a3-a188-4f13-b154-704ab57273f8) transitioned into state 'REVERTED' from state 'REVERTING'
2024-04-18 02:36:04.727 743 WARNING octavia.common.base_taskflow [-] Task 'octavia-failover-loadbalancer-flow-octavia-failover-loadbalancer-flow-octavia-create-amp-for-lb-subflow-octavia-compute-wait' (253ab4a3-a188-4f13-b154-704ab57273f8) transitioned into state 'REVERTED' from state 'REVERTING'
2024-04-18 02:36:08.986 743 WARNING octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect to instance. Retrying.: requests.exceptions.ConnectionError: HTTPSConnectionPool(host='10.5.1.126', port=9443): Max retries exceeded with url: // (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x73fd3c70d150>: Failed to establish a new connection: [Errno 113] No route to host'))
2024-04-18 02:36:14.263 743 WARNING octavia.common.base_taskflow [-] Task 'octavia-failover-loadbalancer-flow-octavia-failover-loadbalancer-flow-octavia-create-amp-for-lb-subflow-octavia-amp-compute-connectivity-wait' ...

Read more...

Revision history for this message
Noel Ashford (nashford77) wrote :

To me, it looks like the failover occurs, but never undoes the error state or moves IP's etc over ? I tried a manual failover which se it to this:

(kolla-2023.2) root@slurm-primary-controller:~/ansible/openstack/5Net# openstack loadbalancer amphora list
+--------------------------------------+--------------------------------------+-----------+------------+---------------+------------+
| id | loadbalancer_id | status | role | lb_network_ip | ha_ip |
 +--------------------------------------+--------------------------------------+-----------+------------+---------------+------------+
| c8122c47-1848-459e-a28d-38016aabd7e1 | 17e60f7c-4b12-439c-824e-674ec4561e97 | ALLOCATED | STANDALONE | 10.5.1.168 | 10.5.1.101 |
| 6e6435b4-a66b-422e-945a-03b9b6a189e1 | 9c2979fb-ed59-4164-8d95-f0e3ed2369dc | ALLOCATED | STANDALONE | 10.5.1.238 | 10.5.1.250 |
 +--------------------------------------+--------------------------------------+-----------+------------+---------------+------------+

but it still is stuck in error. Either way, this can't occur on every reboot? Something has to be wrong on my side or perhaps a bug ?

Revision history for this message
Noel Ashford (nashford77) wrote :

The HA IP the floating IP maps to is not pingable notably ? the lb_network one is tho for both. ?

Revision history for this message
Noel Ashford (nashford77) wrote :
Download full text (6.2 KiB)

(kolla-2023.2) root@slurm-primary-controller:~/ansible/openstack/5Net# openstack loadbalancer list
+--------------------------------------+----------------------------------------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
| id | name | project_id | vip_address | provisioning_status | operating_status | provider |
 +--------------------------------------+----------------------------------------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
| 17e60f7c-4b12-439c-824e-674ec4561e97 | k8s-5net-7apn27y4uejl-api_lb-mvh2gcryrzgw-loadbalancer-pv6mrc5vjwkm | e95c8c84b62e416daf03b91a9648962b | 10.5.1.101 | ACTIVE | ERROR | amphora |
| 9c2979fb-ed59-4164-8d95-f0e3ed2369dc | k8s-5net-7apn27y4uejl-etcd_lb-qst6cawfxwlb-loadbalancer-cyc3hvuvnwjo | e95c8c84b62e416daf03b91a9648962b | 10.5.1.250 | ACTIVE | ERROR | amphora |
 +--------------------------------------+----------------------------------------------------------------------+----------------------------------+-------------+---------------------+------------------+----------+
(kolla-2023.2) root@slurm-primary-controller:~/ansible/openstack/5Net# openstack loadbalancer show 17e60f7c-4b12-439c-824e-674ec4561e97
+---------------------+---------------------------------------------------------------------+
| Field | Value |
 +---------------------+---------------------------------------------------------------------+
| admin_state_up | True |
| availability_zone | None |
| created_at | 2024-04-18T05:44:39 |
| description | |
| flavor_id | None |
| id | 17e60f7c-4b12-439c-824e-674ec4561e97 |
| listeners | a9b18744-034b-49ab-9b7a-9e5af77c9069 |
| name | k8s-5net-7apn27y4uejl-api_lb-mvh2gcryrzgw-loadbalancer-pv6mrc5vjwkm |
| operating_status | ERROR |
| pools | 0b0bc9d6-3e33-4637-9034-ba4ddf0109d4 |
| project_id | e95c8c84b62e416daf03b91a9648962b |
| provider | amphora |
| provisioning_status | ACTIVE |
| updated_at | 2024-04-18T06:48:23 |
| vip_address | 10.5.1.101 ...

Read more...

Revision history for this message
Gregory Thiemonge (gthiemonge) wrote :

> | 17e60f7c-4b12-439c-824e-674ec4561e97 | k8s-5net-7apn27y4uejl-api_lb-mvh2gcryrzgw-loadbalancer-pv6mrc5vjwkm | e95c8c84b62e416daf03b91a9648962b | 10.5.1.101 | ACTIVE | ERROR | amphora |
> | 9c2979fb-ed59-4164-8d95-f0e3ed2369dc | k8s-5net-7apn27y4uejl-etcd_lb-qst6cawfxwlb-loadbalancer-cyc3hvuvnwjo | e95c8c84b62e416daf03b91a9648962b | 10.5.1.250 | ACTIVE | ERROR | amphora |

for both LBs:
provisioning_status is ACTIVE
operating_status is ERROR

it indicates that the LBs are provisioned correctly (ACTIVE) but octavia detects an error with the health-monitors, you can check the status of the all members of a LB with:

openstack loadbalancer status show <lb-uuid>

in your case I guess that all the members are in ERROR, it means that the applications or the servers are not reachable from the amphora VMs.

Revision history for this message
Noel Ashford (nashford77) wrote :

Ok, ref the error state, it was due to the host (single host only, no HA) shutting down OS wise and then Nova broke seemingly. I could recover 5/6 vm's w the state fix, consistently one was lost. very odd behavior why kolla ansible NOVA wont shut down vm's cleanly on a host reboot ... I may have to make a systemd service to fix this before target state shutdown.

lets focus on the other legit bug for now.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.