Unexpected API Error. <class 'nova.exception.GlanceConnectionFailed'>

[Ananya Banerjee]

tempest tests are failing in featureset001 master with :

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 493, in test_connectivity_between_vms_on_different_networks
    self._setup_network_and_servers()
  File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 120, in _setup_network_and_servers
    server = self._create_server(self.network, port_id)
  File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 169, in _create_server
    server = self.create_server(
  File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 309, in create_server
    body, _ = compute.create_test_server(
  File "/usr/lib/python3.9/site-packages/tempest/common/compute.py", line 285, in create_test_server
    body = clients.servers_client.create_server(name=name, imageRef=image_id,
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/servers_client.py", line 115, in create_server
    resp, body = self.post('servers', post_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/base_compute_client.py", line 47, in request
    resp, resp_body = super(BaseComputeClient, self).request(
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 720, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 896, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.GlanceConnectionFailed'>

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/f1ac0bd/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/19/828019/9/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/244ce75/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/10/859110/8/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/830c28c/logs/undercloud/var/log/tempest/stestr_results.html.gz

We see this in tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001 and periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master

[Ananya Banerjee (frenzyfriday)]

Recent logs: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/0d49709/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Ananya Banerjee (frenzyfriday)]

Also seeing this in
periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master,
periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866099

[Takashi Kajinami (kajinamit)]

(Ignore what I wrote. I was looking at uc instead of oc)

[Rabi Mishra (rabi)]

Looks like the haproxy backend is configured with ctlplane ip

backend glance_api_internal_be
  mode http
  option httpchk GET /healthcheck
  server overcloud-controller-0.ctlplane.localdomain 192.168.24.7:9293 check fall 5 inter 2000 rise 2
  server overcloud-controller-1.ctlplane.localdomain 192.168.24.25:9293 check fall 5 inter 2000 rise 2
  server overcloud-controller-2.ctlplane.localdomain 192.168.24.15:9293 check fall 5 inter 2000 rise 2

https://github.com/openstack/tripleo-heat-templates/blob/master/overcloud-resource-registry-puppet.j2.yaml#L118

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/866100

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/puppet-tripleo/+/866100
Reason: this breaks glance-api-edge https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/haproxy/haproxy-edge-container-puppet.yaml#L101-L102

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866165

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866165
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/d24d6121c01514d89c5bef73b6ed5ed969a85661
Submitter: "Zuul (22348)"
Branch: master

commit d24d6121c01514d89c5bef73b6ed5ed969a85661
Author: Alan Bishop <email address hidden>
Date: Wed Nov 30 05:39:32 2022 -0800

    Fix GlanceApiInternal network definition

    Id093613f9d410eb3fe5564a724c0f75275eeb4e8 defines a new internal
    glance-api service, but the patch contains an error in defining the
    network associated with the service. This patch corrects the error.

    Closes-Bug: #1998227
    Change-Id: I456b4235242cae125f5ad4cd9cc7415f2699462c

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 18.0.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rabi Mishra <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/866099