Frequent deployment failure due to attempting to talk HTTP to a HTTPS service

[Frode Nordahl]

Not quite sure where the root of the issue is, but we frequently see this during deployment:
Traceback (most recent call last):
  File "/snap/simplestreams/27/bin/sstream-mirror-glance", line 185, in <module>
    main()
  File "/snap/simplestreams/27/bin/sstream-mirror-glance", line 181, in main
    tmirror.sync(smirror, args.path)
  File "/snap/simplestreams/27/lib/python3.6/site-packages/simplestreams/mirrors/__init__.py", line 91, in sync
    return self.sync_index(reader, path, data, content)
  File "/snap/simplestreams/27/lib/python3.6/site-packages/simplestreams/mirrors/__init__.py", line 254, in sync_index
    self.sync(reader, path=epath)
  File "/snap/simplestreams/27/lib/python3.6/site-packages/simplestreams/mirrors/__init__.py", line 89, in sync
    return self.sync_products(reader, path, data, content)
  File "/snap/simplestreams/27/lib/python3.6/site-packages/simplestreams/mirrors/__init__.py", line 269, in sync_products
    target = self.load_products(path, content_id)
  File "/snap/simplestreams/27/lib/python3.6/site-packages/simplestreams/mirrors/glance.py", line 204, in load_products
    for image in images:
  File "/snap/simplestreams/27/lib/python3.6/site-packages/glanceclient/common/utils.py", line 581, in __next__
    return self._next()
  File "/snap/simplestreams/27/lib/python3.6/site-packages/glanceclient/common/utils.py", line 570, in _next
    obj, resp = next(self._self_wrapped)
  File "/snap/simplestreams/27/lib/python3.6/site-packages/glanceclient/v2/images.py", line 184, in list
    for image, resp in paginate(url, page_size, limit):
  File "/snap/simplestreams/27/lib/python3.6/site-packages/glanceclient/v2/images.py", line 111, in paginate
    resp, body = self.http_client.get(next_url, headers=req_id_hdr)
  File "/snap/simplestreams/27/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get
    return self.request(url, 'GET', **kwargs)
  File "/snap/simplestreams/27/lib/python3.6/site-packages/glanceclient/common/http.py", line 387, in request
    return self._handle_response(resp)
  File "/snap/simplestreams/27/lib/python3.6/site-packages/glanceclient/common/http.py", line 126, in _handle_response
    raise exc.from_response(resp, resp.content)
glanceclient.exc.HTTPBadRequest: HTTP 400 Bad Request: Bad Request: Your browser sent a request that this server could not understand.: Reason: You're speaking plain HTTP to an SSL-enabled server port.: Instead use the HTTPS scheme to access this URL, please.: Apache/2.4.29 (Ubuntu) Server at 172.16.0.87 Port 443

[Frode Nordahl (fnordahl)]

https://openstack-ci-reports.ubuntu.com/artifacts/8df/790554/12/check/bionic-train-ha/8df4e02/index.html

simplestreams uses the Keystone catalog to find endpoints.

The offending requests do come in through the haproxy running on the Glance unit so it's almost as if there is a mis-configuration there?

glance_0/var/log/haproxy.log:
Jun 2 22:53:27 juju-f2b745-zaza-531e4489bf5e-0 haproxy[15467]: 172.16.3.36:60054 [02/Jun/2021:22:53:27.815] tcp-in_glance_api glance_api_172.16.0.87/glance-0 1/0/7 622 -- 1/1/0/0/0 0/0
Jun 2 22:53:38 juju-f2b745-zaza-531e4489bf5e-0 haproxy[15467]: 172.16.3.36:60068 [02/Jun/2021:22:53:38.994] tcp-in_glance_api glance_api_172.16.0.87/glance-0 1/0/3 622 -- 1/1/0/0/0 0/0
Jun 2 22:53:49 juju-f2b745-zaza-531e4489bf5e-0 haproxy[15467]: 172.16.3.36:60082 [02/Jun/2021:22:53:49.689] tcp-in_glance_api glance_api_172.16.0.87/glance-0 1/0/7 622 -- 1/1/0/0/0 0/0

glance_0/var/log/apache2/other_vhosts_access.log:
172.16.0.87:443 172.16.0.87 - - [02/Jun/2021:22:53:27 +0000] "GET / HTTP/1.0" 400 0 "-" "-"
172.16.0.87:443 172.16.0.87 - - [02/Jun/2021:22:53:38 +0000] "GET / HTTP/1.0" 400 0 "-" "-"
172.16.0.87:443 172.16.0.87 - - [02/Jun/2021:22:53:49 +0000] "GET / HTTP/1.0" 400 0 "-" "-"

The crashdump does not contain the relevant configuration files, so it's hard to tell.