[bobcat/jammy] Kubernetes loadbalancer creation fails: An auth plugin is required to determine endpoint URL

[Bas de Bruijne]

Related to LP: #2041268

In test run https://solutions.qa.canonical.com/testruns/0ffd9f4d-f033-4bde-9d8a-2b9cda9bfb5c, which is testing bobcat/candidate on jammy and charmed kubernetes deployed on OpenStack, the cloud validation fails because kubernetes is unable to create loadbalancers through OpenStack:

========
2023-11-09-04:33:31 keystoneauth.session DEBUG RESP BODY: {"loadbalancers": [{"id": "5c7671ad-a2e1-41ba-9fda-e767fe2a8578", "name": "kube_service_kubernetes-9xp4i6ke3rwlt9o0cdjkgvcfkzircfpx_default_octavia-validator-service", "description": "Kubernetes external service default/octavia-validator-service from cluster kubernetes-9xp4i6ke3rwlt9o0cdjkgvcfkzircfpx", "provisioning_status": "ERROR", "operating_status": "OFFLINE", "admin_state_up": true, "project_id": "c1a1822482a74e74859644fd1efb79ed", "created_at": "2023-11-09T04:23:58", "updated_at": "2023-11-09T04:24:05", "vip_address": "172.16.0.102", "vip_port_id": "49382be4-9d76-4a89-b0d6-b617765ae8d9", "vip_subnet_id": "275891ea-8462-4f84-9cc0-9ae553d12389", "vip_network_id": "b99742a0-0fba-479a-8f6d-511fc1bb336e", "additional_vips": [], "listeners": [{"id": "55c3d018-8881-45b2-a3c9-04555ebbae8e"}], "pools": [{"id": "e17c5276-e0ea-485f-8b32-38af6aa0e8e3"}], "provider": "amphora", "flavor_id": null, "vip_qos_policy_id": null, "tags": ["kube_service_kubernetes-9xp4i6ke3rwlt9o0cdjkgvcfkzircfpx_default_octavia-validator-service"], "availability_zone": null, "tenant_id": "c1a1822482a74e74859644fd1efb79ed"}], "loadbalancers_links": []}
2023-11-09-04:33:31 keystoneauth.session DEBUG GET call to load-balancer for https://octavia.silo5.lab1.solutionsqa:9876/v2.0/lbaas/loadbalancers used request id req-d0ce7e0d-a8e0-4627-91e7-c514ccaf63ba
2023-11-09-04:33:31 fce.kubernetes.octavia DEBUG LoadBalancer: kube_service_kubernetes-9xp4i6ke3rwlt9o0cdjkgvcfkzircfpx_default_octavia-validator-service Status: ERROR
2023-11-09-04:33:31 root DEBUG [localhost]: kubectl --kubeconfig /home/ubuntu/project/generated/kubernetes-openstack/kube.conf get svc octavia-validator-service -o yaml
2023-11-09-04:34:01 keystoneauth.session DEBUG REQ: curl -g -i --cacert "/home/ubuntu/sqa-labs/ssl/root.pem" -X GET https://octavia.silo5.lab1.solutionsqa:9876/v2.0/lbaas/loadbalancers -H "Accept: application/json" -H "User-Agent: openstacksdk/0.46.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10" -H "X-Auth-Token: {SHA256}d997f4666f07462c057fd25f9f3d79af3785260b43ae862f36ad69230765c3b9"
2023-11-09-04:34:01 urllib3.connectionpool DEBUG https://octavia.silo5.lab1.solutionsqa:9876 "GET /v2.0/lbaas/loadbalancers HTTP/1.1" 200 1139
2023-11-09-04:34:01 keystoneauth.session DEBUG RESP: [200] Connection: Keep-Alive Content-Length: 1139 Content-Type: application/json Date: Thu, 09 Nov 2023 04:34:01 GMT Keep-Alive: timeout=75, max=981 Server: Apache/2.4.52 (Ubuntu) x-openstack-request-id: req-56a6cd2a-02e2-4c93-a9ee-e2c83c8bac64
2023-11-09-04:34:01 keystoneauth.session DEBUG RESP BODY: {"loadbalancers": [{"id": "5c7671ad-a2e1-41ba-9fda-e767fe2a8578", "name": "kube_service_kubernetes-9xp4i6ke3rwlt9o0cdjkgvcfkzircfpx_default_octavia-validator-service", "description": "Kubernetes external service default/octavia-validator-service from cluster kubernetes-9xp4i6ke3rwlt9o0cdjkgvcfkzircfpx", "provisioning_status": "ERROR", "operating_status": "OFFLINE", "admin_state_up": true, "project_id": "c1a1822482a74e74859644fd1efb79ed", "created_at": "2023-11-09T04:23:58", "updated_at": "2023-11-09T04:24:05", "vip_address": "172.16.0.102", "vip_port_id": "49382be4-9d76-4a89-b0d6-b617765ae8d9", "vip_subnet_id": "275891ea-8462-4f84-9cc0-9ae553d12389", "vip_network_id": "b99742a0-0fba-479a-8f6d-511fc1bb336e", "additional_vips": [], "listeners": [{"id": "55c3d018-8881-45b2-a3c9-04555ebbae8e"}], "pools": [{"id": "e17c5276-e0ea-485f-8b32-38af6aa0e8e3"}], "provider": "amphora", "flavor_id": null, "vip_qos_policy_id": null, "tags": ["kube_service_kubernetes-9xp4i6ke3rwlt9o0cdjkgvcfkzircfpx_default_octavia-validator-service"], "availability_zone": null, "tenant_id": "c1a1822482a74e74859644fd1efb79ed"}], "loadbalancers_links": []}
2023-11-09-04:34:01 keystoneauth.session DEBUG GET call to load-balancer for https://octavia.silo5.lab1.solutionsqa:9876/v2.0/lbaas/loadbalancers used request id req-56a6cd2a-02e2-4c93-a9ee-e2c83c8bac64
2023-11-09-04:34:01 fce.kubernetes.octavia DEBUG LoadBalancer: kube_service_kubernetes-9xp4i6ke3rwlt9o0cdjkgvcfkzircfpx_default_octavia-validator-service Status: ERROR
2023-11-09-04:34:01 root DEBUG [localhost]: kubectl --kubeconfig /home/ubuntu/project/generated/kubernetes-openstack/kube.conf get svc octavia-validator-service -o yaml
2023-11-09-04:34:01 root DEBUG [localhost]: kubectl --kubeconfig /home/ubuntu/project/generated/kubernetes-openstack/kube.conf delete svc octavia-validator-service
2023-11-09-04:34:20 fce.kubernetes.octavia DEBUG service "octavia-validator-service" deleted
2023-11-09-04:34:20 root DEBUG [localhost]: kubectl --kubeconfig /home/ubuntu/project/generated/kubernetes-openstack/kube.conf delete deploy octavia-validator-deployment
2023-11-09-04:34:20 fce.kubernetes.octavia DEBUG deployment.apps "octavia-validator-deployment" deleted
Traceback (most recent call last):
  File "/usr/local/bin/fce", line 11, in <module>
    load_entry_point('foundationcloudengine', 'console_scripts', 'fce')()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 217, in entry_point
    sys.exit(main(sys.argv[1:]))
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/main.py", line 208, in main
    opts.func(opts)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/command.py", line 94, in run
    self.run_per_layer(layer, args)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 46, in run_per_layer
    build_and_validate_if_needed(
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/commands/build.py", line 109, in build_and_validate_if_needed
    layer.validate()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 280, in validate
    ran_validator = self.validate_inner(*args, **kwargs)
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 265, in validate_inner
    validator.run()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/baselayer.py", line 524, in run
    self.run_inner()
  File "/home/ubuntu/cpe/foundation/foundationcloudengine/foundationcloudengine/layers/kubernetes.py", line 654, in run_inner
    raise Exception("Service failed.")
Exception: Service failed.
========

In the octavia logs found in the OpenStack crashdump, we see the following message:

========
5/lxd/10/var/log/octavia/octavia-worker.log:2023-11-09 04:23:59.329 302416 INFO octavia.controller.queue.v2.endpoints [-] Creating load balancer '5c7671ad-a2e1-41ba-9fda-e767fe2a8578'...
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:01.769 302416 INFO octavia.network.drivers.neutron.allowed_address_pairs [-] Port 49382be4-9d76-4a89-b0d6-b617765ae8d9 already exists. Nothing to be done.
5/lxd/10/var/log/octavia/octavia-worker.log:2023-11-09 04:24:01.769 302416 INFO octavia.controller.worker.v2.tasks.network_tasks [-] Allocated vip with port id 49382be4-9d76-4a89-b0d6-b617765ae8d9, subnet id 275891ea-8462-4f84-9cc0-9ae553d12389, ip address 172.16.0.102 for load balancer 5c7671ad-a2e1-41ba-9fda-e767fe2a8578
5/lxd/10/var/log/octavia/octavia-worker.log:2023-11-09 04:24:01.971 302416 INFO octavia.controller.worker.v2.tasks.database_tasks [-] Updated vip with port id 49382be4-9d76-4a89-b0d6-b617765ae8d9, subnet id 275891ea-8462-4f84-9cc0-9ae553d12389, ip address 172.16.0.102 for load balancer 5c7671ad-a2e1-41ba-9fda-e767fe2a8578
5/lxd/10/var/log/octavia/octavia-worker.log:2023-11-09 04:24:03.859 302416 INFO octavia.controller.worker.v2.tasks.network_tasks [-] Set up VIP SG edf81e83-4ccb-42a0-a4ef-696d3c4a9ef6 for load balancer 5c7671ad-a2e1-41ba-9fda-e767fe2a8578 complete
5/lxd/10/var/log/octavia/octavia-worker.log:2023-11-09 04:24:04.052 302416 INFO octavia.controller.worker.v2.tasks.network_tasks [-] Got subnet 275891ea-8462-4f84-9cc0-9ae553d12389 for load balancer 5c7671ad-a2e1-41ba-9fda-e767fe2a8578
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver [-] Error create server group instance.: keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver Traceback (most recent call last):
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python3/dist-packages/octavia/compute/drivers/nova_driver.py", line 288, in create_server_group
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver server_group_obj = self.server_groups.create(**kwargs)
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python3/dist-packages/novaclient/api_versions.py", line 393, in substitution
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver return methods[-1].func(obj, *args, **kwargs)
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python3/dist-packages/novaclient/v2/server_groups.py", line 101, in create
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver return self._create('/os-server-groups', body, 'server_group')
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python3/dist-packages/novaclient/base.py", line 363, in _create
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver resp, body = self.api.client.post(url, body=body)
--
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 803, in request
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver base_url = self.get_endpoint(auth, allow=allow,
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1233, in get_endpoint
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver auth = self._auth_required(auth, 'determine endpoint URL')
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1173, in _auth_required
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver raise exceptions.MissingAuthPlugin(msg_fmt % msg)
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin: An auth plugin is required to determine endpoint URL
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.059 302416 ERROR octavia.compute.drivers.nova_driver
5/lxd/10/var/log/octavia/octavia-worker.log-2023-11-09 04:24:04.062 302416 WARNING octavia.controller.worker.v2.controller_worker [-] Task 'octavia-create-loadbalancer-flow-octavia-create-server-group-flow' (2af00921-6d42-4758-a76d-6a8581801be7) transitioned into state 'FAILURE' from state 'RUNNING'
========

It looks like Octavia started the load balancer creation process, but then ran into an issue similar to reported in LP: #2041268

Additional configs and crashdumps can be found here: https://oil-jenkins.canonical.com/artifacts/0ffd9f4d-f033-4bde-9d8a-2b9cda9bfb5c/index.html

[Corey Bryant (corey.bryant)]

I have a fix for this. There are a few changes, but mainly it seems the [service_auth] section requires more config than it did in the past.

In the fix that I'm proposing, I've validated the config options vs upstream ('tox -e genconfig' generates etc/octavia/octavia.conf.sample). Note, however the [neutron] section doesn't have the auth config that is mentioned in the referenced commit. I've opened a bug for that at: https://bugs.launchpad.net/octavia/+bug/2043129

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-octavia/+/900547

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/900547
Committed: https://opendev.org/openstack/charm-octavia/commit/f76d066df5330d8b8dd47ddc54abbb41e6f78bc2
Submitter: "Zuul (22348)"
Branch: master

commit f76d066df5330d8b8dd47ddc54abbb41e6f78bc2
Author: Corey Bryant <email address hidden>
Date: Thu Nov 9 11:00:01 2023 -0500

    Update octavia.conf auth config for bobcat

    As noticed in bobcat testing, the [service_auth] section requires
    more configuration than it did in the past. Additionally, as noted
    in I686cfdef78de927fa4bc1921c15e8d5853fd2ef9, Octavia will no longer
    take the authentication settings for Neutron from the [service_auth]
    as a fallback. It will instead require them to be in the [neutron]
    section. However, [service_auth] settings will still be used for
    other services like Nova and Glance.

    This change adds auth config options to the [service_auth] and
    [neutron] sections, moves the auth_section config option to the
    [keystone_authtoken] section, and sets the auth_section config
    option to point to the [service_auth] section.

    Closes-Bug: #2043095
    Related-Bug: #2043129
    Change-Id: I290f543827d63bb685209d615c9f448c2ff9d31e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-octavia/+/900887

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-octavia/+/900887
Committed: https://opendev.org/openstack/charm-octavia/commit/85b2b31512ef449732c298ae39269bcf21e6dc27
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 85b2b31512ef449732c298ae39269bcf21e6dc27
Author: Corey Bryant <email address hidden>
Date: Thu Nov 9 11:00:01 2023 -0500

    Update octavia.conf auth config for bobcat

    As noticed in bobcat testing, the [service_auth] section requires
    more configuration than it did in the past. Additionally, as noted
    in I686cfdef78de927fa4bc1921c15e8d5853fd2ef9, Octavia will no longer
    take the authentication settings for Neutron from the [service_auth]
    as a fallback. It will instead require them to be in the [neutron]
    section. However, [service_auth] settings will still be used for
    other services like Nova and Glance.

    This change adds auth config options to the [service_auth] and
    [neutron] sections, moves the auth_section config option to the
    [keystone_authtoken] section, and sets the auth_section config
    option to point to the [service_auth] section.

    Closes-Bug: #2043095
    Related-Bug: #2043129
    Change-Id: I290f543827d63bb685209d615c9f448c2ff9d31e
    (cherry picked from commit f76d066df5330d8b8dd47ddc54abbb41e6f78bc2)