[dcn] instance boot fails on DistributedComputeHCIScaleOut node: Error contacting glance server 'http://172.25.2.182:9292' for 'get', done trying.: glanceclient.exc.HTTPServiceUnavailable: HTTP 503 Service Unavailable

[John Fulton]

When deploying HCI nodes on the edge with the following node count per DCN site:

  DistributedComputeHCICount: 3
  DistributedComputeHCIScaleOutCount: 1

instances booted on DistributedComputeHCI like this work fine:

openstack server create instance2 --image 42af42c6-c663-49b1-a800-feb6f2547371 --flavor m1.nano --nic net-id=afb1c6dd-b425-4cec-b0d3-c70cc837df01 --availability-zone az-dcn1 --hypervisor-hostname dcn1-computehci1-0.redhat.local

However instances fail to boot on DistributedComputeHCIScaleOut nodes when running the following:

openstack server create instance3 --image 42af42c6-c663-49b1-a800-feb6f2547371 --flavor m1.nano --nic net-id=afb1c6dd-b425-4cec-b0d3-c70cc837df01 --availability-zone az-dcn1 --hypervisor-hostname dcn1-computehciscaleout1-0.redhat.local

The following is seen in the nova-compute log:

/var/log/containers/nova/nova-compute.log:60:2020-06-16 11:05:26.822 8 ERROR nova.image.glance [req-05906d7b-ca3e-4977-bbb3-3d400c85f9ea efb7deed85904588a504616e95035276 699ce3c09f914440ae9408375f5af143 - default default] Error contacting glance server 'http://172.25.2.182:9292' for 'get', done trying.: glanceclient.exc.HTTPServiceUnavailable: HTTP 503 Service Unavailable: No server is available to handle this request.
/var/log/containers/nova/nova-compute.log:61:2020-06-16 11:05:26.822 8 ERROR nova.image.glance Traceback (most recent call last):
/var/log/containers/nova/nova-compute.log:62:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 193, in call
/var/log/containers/nova/nova-compute.log:63:2020-06-16 11:05:26.822 8 ERROR nova.image.glance result = getattr(controller, method)(*args, **kwargs)
/var/log/containers/nova/nova-compute.log:64:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/v2/images.py", line 198, in get
/var/log/containers/nova/nova-compute.log:65:2020-06-16 11:05:26.822 8 ERROR nova.image.glance return self._get(image_id)
/var/log/containers/nova/nova-compute.log:66:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/common/utils.py", line 598, in inner
/var/log/containers/nova/nova-compute.log:67:2020-06-16 11:05:26.822 8 ERROR nova.image.glance return RequestIdProxy(wrapped(*args, **kwargs))
/var/log/containers/nova/nova-compute.log:68:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/v2/images.py", line 191, in _get
/var/log/containers/nova/nova-compute.log:69:2020-06-16 11:05:26.822 8 ERROR nova.image.glance resp, body = self.http_client.get(url, headers=header)
/var/log/containers/nova/nova-compute.log:70:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get
/var/log/containers/nova/nova-compute.log:71:2020-06-16 11:05:26.822 8 ERROR nova.image.glance return self.request(url, 'GET', **kwargs)
/var/log/containers/nova/nova-compute.log:72:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/common/http.py", line 387, in request
/var/log/containers/nova/nova-compute.log:73:2020-06-16 11:05:26.822 8 ERROR nova.image.glance return self._handle_response(resp)
/var/log/containers/nova/nova-compute.log:74:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/common/http.py", line 126, in _handle_response
/var/log/containers/nova/nova-compute.log:75:2020-06-16 11:05:26.822 8 ERROR nova.image.glance raise exc.from_response(resp, resp.content)
/var/log/containers/nova/nova-compute.log:76:2020-06-16 11:05:26.822 8 ERROR nova.image.glance glanceclient.exc.HTTPServiceUnavailable: HTTP 503 Service Unavailable: No server is available to handle this

[John Fulton (jfulton-org)]

WORKAROUND:

Open port 9292 on the DistributedComputeHCI nodes

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/736080

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/737188

[OpenStack Infra (hudson-openstack)]

Change abandoned by John Fulton (<email address hidden>) on branch: master
Review: https://review.opendev.org/736080
Reason: Doing this instead https://review.opendev.org/#/c/737188/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/737188
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=975e47f2194c319a1191a927b26c67e0787d3b89
Submitter: Zuul
Branch: stable/train

commit 975e47f2194c319a1191a927b26c67e0787d3b89
Author: Alan Bishop <email address hidden>
Date: Sun Jun 21 09:55:43 2020 -0700

    [TRAIN ONLY] Fix the glance-api-edge firewall configuration

    Add a firewall rule for the glance-api-edge service that opens glance's
    port 9292. This allows "scale out" nodes running at the edge to access
    a glance endpoint at the edge site.

    The glance-api-edge service tries to inherit most of its configuration
    from the base glance-api service. However, the puppet code that applies
    firewall rules requires the hiera key that defines the rule contain the
    actual service name [1]. Thus, a rule defined for the glance-api service
    will not be added when the service name is glance-api-edge.

    [1] https://opendev.org/openstack/puppet-tripleo/src/branch/stable/train/manifests/firewall/service_rules.pp#L35

    Closes-Bug: #1883807
    Change-Id: I6a3c77be03d5ad6b20c1eb11291340217544bff8

[John Fulton (jfulton-org)]

You should not hit his bug in master or ussuri. A patch has been added to train.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.