root@5net:~# tail -f /var/log/kolla/octavia/octavia-*.log
==> /var/log/kolla/octavia/octavia-api-access.log <==
192.168.5.99 - - [14/Apr/2024:02:11:05 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 14403 "-" "gophercloud/2.0.0"
- - - [14/Apr/2024:02:11:05 -0400] "GET / HTTP/1.1" 200 3978 2490 "-" "curl-healthcheck"
192.168.5.99 - - [14/Apr/2024:02:11:09 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 35714 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:14 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 34457 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:15 -0400] "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 61885 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:15 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 15925 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:19 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 80599 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:24 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 80288 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:25 -0400] "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 122145 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:25 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 13787 "-" "gophercloud/2.0.0"
==> /var/log/kolla/octavia/octavia-api-error.log <==
2024-04-14 02:02:45.880226 warnings.warn(deprecated_msg)
2024-04-14 02:02:45.880255 /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_policy/policy.py:810: UserWarning: Policy "load-balancer:member_and_owner":"role:load-balancer_member and rule:load-balancer:owner" was deprecated in W in favor of "load-balancer:member_and_owner":"role:load-balancer_member and rule:project-member". Reason: The Octavia API now requires the OpenStack default roles and scoped tokens. See https://docs.openstack.org/octavia/latest/configuration/policy.html and https://docs.openstack.org/keystone/latest/contributor/services.html#reusable-default-roles for more information.. Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually.
2024-04-14 02:02:45.880260 warnings.warn(deprecated_msg)
2024-04-14 02:02:59.447984 Deprecated: Option "ca_certificates_file" from group "neutron" is deprecated for removal (The cafile option defined by keystoneauth1 is the new name for this option.). Its value may be silently ignored in the future.
2024-04-14 02:03:00.082530 /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_policy/policy.py:810: UserWarning: Policy "context_is_admin":"role:admin or role:load-balancer_admin" was deprecated in W in favor of "context_is_admin":"role:load-balancer_admin or rule:system-admin or role:admin". Reason: The Octavia API now requires the OpenStack default roles and scoped tokens. See https://docs.openstack.org/octavia/latest/configuration/policy.html and https://docs.openstack.org/keystone/latest/contributor/services.html#reusable-default-roles for more information.. Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually.
2024-04-14 02:03:00.082554 warnings.warn(deprecated_msg)
2024-04-14 02:03:00.082595 /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_policy/policy.py:810: UserWarning: Policy "load-balancer:observer_and_owner":"role:load-balancer_observer and rule:load-balancer:owner" was deprecated in W in favor of "load-balancer:observer_and_owner":"role:load-balancer_observer and rule:project-reader". Reason: The Octavia API now requires the OpenStack default roles and scoped tokens. See https://docs.openstack.org/octavia/latest/configuration/policy.html and https://docs.openstack.org/keystone/latest/contributor/services.html#reusable-default-roles for more information.. Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually.
2024-04-14 02:03:00.082614 warnings.warn(deprecated_msg)
2024-04-14 02:03:00.082643 /var/lib/kolla/venv/lib/python3.10/site-packages/oslo_policy/policy.py:810: UserWarning: Policy "load-balancer:member_and_owner":"role:load-balancer_member and rule:load-balancer:owner" was deprecated in W in favor of "load-balancer:member_and_owner":"role:load-balancer_member and rule:project-member". Reason: The Octavia API now requires the OpenStack default roles and scoped tokens. See https://docs.openstack.org/octavia/latest/configuration/policy.html and https://docs.openstack.org/keystone/latest/contributor/services.html#reusable-default-roles for more information.. Either ensure your deployment is ready for the new default or copy/paste the deprecated policy into your policy file and maintain it manually.
2024-04-14 02:03:00.082649 warnings.warn(deprecated_msg)
==> /var/log/kolla/octavia/octavia-api.log <==
2024-04-14 02:02:59.558 733 INFO octavia.api.app [-] oslo_messaging_notifications.topics = ['notifications']
2024-04-14 02:02:59.558 733 INFO octavia.api.app [-] oslo_messaging_notifications.transport_url = ****
2024-04-14 02:02:59.558 733 INFO octavia.api.app [-] ********************************************************************************
2024-04-14 02:02:59.759 733 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
2024-04-14 02:09:18.979 735 WARNING openstack [None req-dd0763f2-f57d-49ef-b956-aab069076211 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'block-storage': Encountered an exception attempting to process config for project 'cinder' (service type 'block-storage'): no such option valid_interfaces in group [cinder]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [cinder]
2024-04-14 02:09:18.979 735 WARNING openstack [None req-dd0763f2-f57d-49ef-b956-aab069076211 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'compute': Encountered an exception attempting to process config for project 'nova' (service type 'compute'): no such option valid_interfaces in group [nova]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [nova]
2024-04-14 02:09:18.980 735 WARNING openstack [None req-dd0763f2-f57d-49ef-b956-aab069076211 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'image': Encountered an exception attempting to process config for project 'glance' (service type 'image'): no such option valid_interfaces in group [glance]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [glance]
2024-04-14 02:10:11.649 735 WARNING openstack [None req-3a997194-acf1-4f44-9916-ea04569c7b6f - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'block-storage': Encountered an exception attempting to process config for project 'cinder' (service type 'block-storage'): no such option valid_interfaces in group [cinder]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [cinder]
2024-04-14 02:10:11.649 735 WARNING openstack [None req-3a997194-acf1-4f44-9916-ea04569c7b6f - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'compute': Encountered an exception attempting to process config for project 'nova' (service type 'compute'): no such option valid_interfaces in group [nova]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [nova]
2024-04-14 02:10:11.650 735 WARNING openstack [None req-3a997194-acf1-4f44-9916-ea04569c7b6f - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'image': Encountered an exception attempting to process config for project 'glance' (service type 'image'): no such option valid_interfaces in group [glance]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [glance]
==> /var/log/kolla/octavia/octavia-health-manager.log <==
2024-04-14 01:44:51.574 7 INFO octavia.cmd.health_manager [-] Health manager check process starts:
2024-04-14 01:44:51.576 1008 INFO octavia.amphorae.drivers.health.heartbeat_udp [-] attempting to listen on 192.168.5.1 port 5555
2024-04-14 01:59:43.557 7 INFO octavia.cmd.health_manager [-] Health Manager exiting due to signal
2024-04-14 01:59:44.406 1008 INFO octavia.cmd.health_manager [-] Waiting for executor to shutdown...
2024-04-14 01:59:44.407 1008 INFO octavia.cmd.health_manager [-] Executor shutdown finished.
2024-04-14 02:02:43.810 7 INFO octavia.common.config [-] Logging enabled!
2024-04-14 02:02:43.810 7 INFO octavia.common.config [-] /var/lib/kolla/venv/bin/octavia-health-manager version 13.0.1.dev20
2024-04-14 02:02:43.856 7 INFO octavia.cmd.health_manager [-] Health Manager listener process starts:
2024-04-14 02:02:43.860 7 INFO octavia.cmd.health_manager [-] Health manager check process starts:
2024-04-14 02:02:43.861 731 INFO octavia.amphorae.drivers.health.heartbeat_udp [-] attempting to listen on 192.168.5.1 port 5555
==> /var/log/kolla/octavia/octavia-housekeeping.log <==
2024-04-14 01:44:52.907 7 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2024-04-14 02:02:40.900 7 INFO octavia.cmd.house_keeping [-] Attempting to gracefully terminate House-Keeping
2024-04-14 02:02:40.901 7 INFO octavia.cmd.house_keeping [-] House-Keeping process terminated
2024-04-14 02:02:45.736 7 INFO octavia.common.config [-] Logging enabled!
2024-04-14 02:02:45.736 7 INFO octavia.common.config [-] /var/lib/kolla/venv/bin/octavia-housekeeping version 13.0.1.dev20
2024-04-14 02:02:45.839 7 INFO octavia.cmd.house_keeping [-] Starting house keeping at 2024-04-14 06:02:45.839614
2024-04-14 02:02:45.840 7 INFO octavia.cmd.house_keeping [-] DB cleanup interval is set to 30 sec
2024-04-14 02:02:45.840 7 INFO octavia.cmd.house_keeping [-] Amphora expiry age is 604800 seconds
2024-04-14 02:02:45.840 7 INFO octavia.cmd.house_keeping [-] Load balancer expiry age is 604800 seconds
2024-04-14 02:02:45.840 7 INFO octavia.cmd.house_keeping [-] Expiring certificate check interval is set to 3600 sec
==> /var/log/kolla/octavia/octavia-worker.log <==
2024-04-14 01:45:01.876 731 INFO octavia.controller.queue.v2.consumer [-] Starting V2 consumer...
2024-04-14 02:02:47.138 7 INFO cotyledon._service_manager [-] Caught SIGTERM signal, graceful exiting of master process
2024-04-14 02:02:47.239 731 INFO cotyledon._service [-] Caught SIGTERM signal, graceful exiting of service ConsumerService(0) [731]
2024-04-14 02:02:47.239 731 INFO octavia.controller.queue.v2.consumer [-] Stopping V2 consumer...
2024-04-14 02:02:51.980 731 INFO octavia.controller.queue.v2.consumer [-] V2 Consumer successfully stopped. Waiting for final messages to be processed...
2024-04-14 02:02:51.985 731 WARNING amqp [-] Received method (60, 30) during closing channel 1. This method will be ignored
2024-04-14 02:02:51.986 731 INFO octavia.controller.queue.v2.consumer [-] Shutting down V2 endpoint worker executors...
2024-04-14 02:02:56.757 7 INFO octavia.common.config [-] Logging enabled!
2024-04-14 02:02:56.757 7 INFO octavia.common.config [-] /var/lib/kolla/venv/bin/octavia-worker version 13.0.1.dev20
2024-04-14 02:02:56.843 731 INFO octavia.controller.queue.v2.consumer [-] Starting V2 consumer...
==> /var/log/kolla/octavia/octavia-api-access.log <==
192.168.5.99 - - [14/Apr/2024:02:11:29 -0400] "GET /v2.0/lbaas/availabilityzones HTTP/1.1" 200 58 39504 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:29 -0400] "GET /v2.0/lbaas/flavors HTTP/1.1" 200 36 96963 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:29 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 37895 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:34 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 38753 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
192.168.5.99 - - [14/Apr/2024:02:11:35 -0400] "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 149652 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:35 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 12304 "-" "gophercloud/2.0.0"
- - - [14/Apr/2024:02:11:35 -0400] "GET / HTTP/1.1" 200 3978 2702 "-" "curl-healthcheck"
192.168.5.99 - - [14/Apr/2024:02:11:36 -0400] "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 48 58485 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:36 -0400] "GET /v2.0/octavia/amphorae HTTP/1.1" 200 38 17650 "-" "gophercloud/2.0.0"
192.168.5.99 - - [14/Apr/2024:02:11:39 -0400] "GET /v2.0/lbaas/loadbalancers?project_id=e95c8c84b62e416daf03b91a9648962b HTTP/1.1" 200 48 36406 "-" "openstacksdk/1.5.0 keystoneauth1/5.3.0 python-requests/2.28.2 CPython/3.10.12"
==> /var/log/kolla/octavia/octavia-api.log <==
2024-04-14 02:11:42.134 736 WARNING openstack [None req-f4cdd978-d9ce-4cd3-ba91-6fff2d85ea99 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'block-storage': Encountered an exception attempting to process config for project 'cinder' (service type 'block-storage'): no such option valid_interfaces in group [cinder]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [cinder]
2024-04-14 02:11:42.134 736 WARNING openstack [None req-f4cdd978-d9ce-4cd3-ba91-6fff2d85ea99 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'compute': Encountered an exception attempting to process config for project 'nova' (service type 'compute'): no such option valid_interfaces in group [nova]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [nova]
2024-04-14 02:11:42.134 736 WARNING openstack [None req-f4cdd978-d9ce-4cd3-ba91-6fff2d85ea99 - e95c8c84b62e416daf03b91a9648962b - - default default] Disabling service 'image': Encountered an exception attempting to process config for project 'glance' (service type 'image'): no such option valid_interfaces in group [glance]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [glance]
Also, separate issue - prior to this, i had additional patches from that had it up but when i rebooted, it totally lost state and was not recoverable. It sat in a pending state forever.
Can you share the [nova] [cinder] [glance] sections of your octavia configuration?