retrofit-image can't communicate with glance endpoint

[Corey Bryant]

Run of action "retrofit-image" with parameters "{'force': False, 'source-image': ''}" on "octavia-diskimage-retrofit/0" failed with "Error finding address for https://10.5.2.161:9292/v2/images/22de3089-d13c
-4285-9f3e-8ee7dc488818/file: Unable to establish connection to https://10.5.2.161:9292/v2/images/22de3089-d13c-4285-9f3e-8ee7dc488818/file: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'
))" (id=44 status=failed enqueued=2023-09-14T19:35:53Z started=2023-09-14T19:35:53Z completed=2023-09-14T19:37:28Z output={'Code': '0'})

[Corey Bryant (corey.bryant)]

ubuntu@coreycb-bastion:/mnt/charmed-openstack-tester$ openstack image list
Failed to contact the endpoint at https://10.5.1.162:9292 for discovery. Fallback to using that endpoint as the base url.
Failed to contact the endpoint at https://10.5.1.162:9292 for discovery. Fallback to using that endpoint as the base url.
The image service for :RegionOne exists but does not have any supported versions.

ubuntu@coreycb-bastion:/mnt/charmed-openstack-tester$ openstack image list --debug ...
Instantiating identity client: <class 'keystoneclient.v3.client.Client'>
REQ: curl -g -i --cacert "/home/ubuntu/.cache/openstack-root-ca.crt" -X GET https://10.5.1.162:9292 -H "Accept: application/json" -H "User-Agent: openstacksdk/1.0.1 keystone
auth1/5.1.2 python-requests/2.25.1 CPython/3.10.12"
Starting new HTTPS connection (1): 10.5.1.162:9292
No version document at https://10.5.1.162:9292: SSL exception connecting to https://10.5.1.162:9292: HTTPSConnectionPool(host='10.5.1.162', port=9292): Max retries exceeded
with url: / (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
Failed to contact the endpoint at https://10.5.1.162:9292 for discovery. Fallback to using that endpoint as the base url.
REQ: curl -g -i --cacert "/home/ubuntu/.cache/openstack-root-ca.crt" -X GET https://10.5.1.162:9292 -H "Accept: application/json" -H "User-Agent: openstacksdk/1.0.1 keystone
auth1/5.1.2 python-requests/2.25.1 CPython/3.10.12"
Starting new HTTPS connection (2): 10.5.1.162:9292
No version document at https://10.5.1.162:9292: SSL exception connecting to https://10.5.1.162:9292: HTTPSConnectionPool(host='10.5.1.162', port=9292): Max retries exceeded
with url: / (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1007)')))
Failed to contact the endpoint at https://10.5.1.162:9292 for discovery. Fallback to using that endpoint as the base url.
The image service for :RegionOne exists but does not have any supported versions.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cliff/app.py", line 410, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3/dist-packages/cliff/display.py", line 117, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python3/dist-packages/openstackclient/image/v2/image.py", line 832, in take_action
    image_client = self.app.client_manager.image
  File "/usr/lib/python3/dist-packages/osc_lib/clientmanager.py", line 45, in __get__
    self._handle = self.factory(instance)
  File "/usr/lib/python3/dist-packages/openstackclient/image/client.py", line 38, in make_client
    instance.sdk_connection.image,
  File "/usr/lib/python3/dist-packages/openstack/service_description.py", line 87, in __get__
    proxy = self._make_proxy(instance)
  File "/usr/lib/python3/dist-packages/openstack/service_description.py", line 266...

[Corey Bryant (corey.bryant)]

Bumping to 20G fixed this and resulted in the following after diskimage-retrofit ran:

ubuntu@juju-e6cd06-zaza-b65ddad607b5-2:~$ sudo ceph status
  cluster:
    id: 952e3ee0-5622-11ee-aca3-05a57e326cb4
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum juju-e6cd06-zaza-b65ddad607b5-2,juju-e6cd06-zaza-b65ddad607b5-1,juju-e6cd06-zaza-b65ddad607b5-0 (age 2h)
    mgr: juju-e6cd06-zaza-b65ddad607b5-1(active, since 2h), standbys: juju-e6cd06-zaza-b65ddad607b5-2, juju-e6cd06-zaza-b65ddad607b5-0
    osd: 3 osds: 3 up (since 2h), 3 in (since 2h)

  data:
    pools: 3 pools, 65 pgs
    objects: 1.26k objects, 9.5 GiB
    usage: 29 GiB used, 31 GiB / 60 GiB avail
    pgs: 65 active+clean

An increase to 12G-15G should be good enough. I'm leaning toward 15G to allow additional room for tempest tests to run.

[Corey Bryant (corey.bryant)]

Marked as invalid. This is a resource issue in tests, fixed in: https://github.com/openstack-charmers/charmed-openstack-tester/pull/115