[dcn] instance boot fails on DistributedComputeHCIScaleOut node: Error contacting glance server 'http://172.25.2.182:9292' for 'get', done trying.: glanceclient.exc.HTTPServiceUnavailable: HTTP 503 Service Unavailable

Bug #1883807 reported by John Fulton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
John Fulton

Bug Description

When deploying HCI nodes on the edge with the following node count per DCN site:

  DistributedComputeHCICount: 3
  DistributedComputeHCIScaleOutCount: 1

instances booted on DistributedComputeHCI like this work fine:

openstack server create instance2 --image 42af42c6-c663-49b1-a800-feb6f2547371 --flavor m1.nano --nic net-id=afb1c6dd-b425-4cec-b0d3-c70cc837df01 --availability-zone az-dcn1 --hypervisor-hostname dcn1-computehci1-0.redhat.local

However instances fail to boot on DistributedComputeHCIScaleOut nodes when running the following:

openstack server create instance3 --image 42af42c6-c663-49b1-a800-feb6f2547371 --flavor m1.nano --nic net-id=afb1c6dd-b425-4cec-b0d3-c70cc837df01 --availability-zone az-dcn1 --hypervisor-hostname dcn1-computehciscaleout1-0.redhat.local

The following is seen in the nova-compute log:

/var/log/containers/nova/nova-compute.log:60:2020-06-16 11:05:26.822 8 ERROR nova.image.glance [req-05906d7b-ca3e-4977-bbb3-3d400c85f9ea efb7deed85904588a504616e95035276 699ce3c09f914440ae9408375f5af143 - default default] Error contacting glance server 'http://172.25.2.182:9292' for 'get', done trying.: glanceclient.exc.HTTPServiceUnavailable: HTTP 503 Service Unavailable: No server is available to handle this request.
/var/log/containers/nova/nova-compute.log:61:2020-06-16 11:05:26.822 8 ERROR nova.image.glance Traceback (most recent call last):
/var/log/containers/nova/nova-compute.log:62:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/nova/image/glance.py", line 193, in call
/var/log/containers/nova/nova-compute.log:63:2020-06-16 11:05:26.822 8 ERROR nova.image.glance result = getattr(controller, method)(*args, **kwargs)
/var/log/containers/nova/nova-compute.log:64:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/v2/images.py", line 198, in get
/var/log/containers/nova/nova-compute.log:65:2020-06-16 11:05:26.822 8 ERROR nova.image.glance return self._get(image_id)
/var/log/containers/nova/nova-compute.log:66:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/common/utils.py", line 598, in inner
/var/log/containers/nova/nova-compute.log:67:2020-06-16 11:05:26.822 8 ERROR nova.image.glance return RequestIdProxy(wrapped(*args, **kwargs))
/var/log/containers/nova/nova-compute.log:68:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/v2/images.py", line 191, in _get
/var/log/containers/nova/nova-compute.log:69:2020-06-16 11:05:26.822 8 ERROR nova.image.glance resp, body = self.http_client.get(url, headers=header)
/var/log/containers/nova/nova-compute.log:70:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get
/var/log/containers/nova/nova-compute.log:71:2020-06-16 11:05:26.822 8 ERROR nova.image.glance return self.request(url, 'GET', **kwargs)
/var/log/containers/nova/nova-compute.log:72:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/common/http.py", line 387, in request
/var/log/containers/nova/nova-compute.log:73:2020-06-16 11:05:26.822 8 ERROR nova.image.glance return self._handle_response(resp)
/var/log/containers/nova/nova-compute.log:74:2020-06-16 11:05:26.822 8 ERROR nova.image.glance File "/usr/lib/python3.6/site-packages/glanceclient/common/http.py", line 126, in _handle_response
/var/log/containers/nova/nova-compute.log:75:2020-06-16 11:05:26.822 8 ERROR nova.image.glance raise exc.from_response(resp, resp.content)
/var/log/containers/nova/nova-compute.log:76:2020-06-16 11:05:26.822 8 ERROR nova.image.glance glanceclient.exc.HTTPServiceUnavailable: HTTP 503 Service Unavailable: No server is available to handle this

Revision history for this message
John Fulton (jfulton-org) wrote :

WORKAROUND:

Open port 9292 on the DistributedComputeHCI nodes

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/736080

Changed in tripleo:
status: Triaged → In Progress
tags: added: edge ussuri-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/737188

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (master)

Change abandoned by John Fulton (<email address hidden>) on branch: master
Review: https://review.opendev.org/736080
Reason: Doing this instead https://review.opendev.org/#/c/737188/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/737188
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=975e47f2194c319a1191a927b26c67e0787d3b89
Submitter: Zuul
Branch: stable/train

commit 975e47f2194c319a1191a927b26c67e0787d3b89
Author: Alan Bishop <email address hidden>
Date: Sun Jun 21 09:55:43 2020 -0700

    [TRAIN ONLY] Fix the glance-api-edge firewall configuration

    Add a firewall rule for the glance-api-edge service that opens glance's
    port 9292. This allows "scale out" nodes running at the edge to access
    a glance endpoint at the edge site.

    The glance-api-edge service tries to inherit most of its configuration
    from the base glance-api service. However, the puppet code that applies
    firewall rules requires the hiera key that defines the rule contain the
    actual service name [1]. Thus, a rule defined for the glance-api service
    will not be added when the service name is glance-api-edge.

    [1] https://opendev.org/openstack/puppet-tripleo/src/branch/stable/train/manifests/firewall/service_rules.pp#L35

    Closes-Bug: #1883807
    Change-Id: I6a3c77be03d5ad6b20c1eb11291340217544bff8

tags: added: in-stable-train
Revision history for this message
John Fulton (jfulton-org) wrote :

You should not hit his bug in master or ussuri. A patch has been added to train.

Changed in tripleo:
status: In Progress → Fix Committed
Changed in tripleo:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.4.0

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.