Image sync failed, retrying soon

Bug #1931611 reported by Marcelo Subtil Marcal on 2021-06-10
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack glance-simplestreams-sync charm
Undecided
Unassigned
juju
Undecided
Unassigned

Bug Description

When deploying cs:glance-simplestreams-sync-36, the status stucks on "Image sync failed, retrying soon."

The log shows:

---
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py", line 432, in main
    do_sync(charm_conf)
  File "/usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py", line 289, in do_sync
    subprocess.check_call(sync_command)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/snap/bin/simplestreams.sstream-mirror-glance', '-vv', '--keep', '--max', '1', '--content-id', 'auto.sync', '--cloud-name', 'glance-simplestreams-sync-openstack', '--path', 'streams/v1/index.sjson', '--name-prefix', 'auto-sync/', '--keyring', '/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg', '--log-file', '/var/snap/simplestreams/common/sstream-mirror-glance.log', '--output-swift', 'simplestreams/data/', 'http://cloud-images.ubuntu.com/releases/', 'release~(trusty|xenial|bionic|focal)', 'arch~(x86_64|amd64)', 'ftype~(disk1.img|disk.img)']' returned non-zero exit status 120.
----

Running manually the command, the following error occurs:

----
Traceback (most recent call last):
  File "/snap/simplestreams/27/bin/sstream-mirror-glance", line 185, in <module>
    main()
  File "/snap/simplestreams/27/bin/sstream-mirror-glance", line 154, in main
    regions = openstack.get_regions(services=['image'])
  File "/snap/simplestreams/27/lib/python3.6/site-packages/simplestreams/openstack.py", line 133, in get_regions
    creds = load_keystone_creds()
  File "/snap/simplestreams/27/lib/python3.6/site-packages/simplestreams/openstack.py", line 120, in load_keystone_creds
    raise ValueError("Need values for: %s" % missing)
ValueError: Need values for: ['username', 'auth_url', '(auth_token or password)', '(tenant_id or tenant_name)']
----

Revision history for this message
Marcelo Subtil Marcal (msmarcal) wrote :

Subscribed field-high

Revision history for this message
Billy Olsen (billy-olsen) wrote :

@msmarcal - is there a juju status that you can provide? The glance simplestreams sync charm should get credentials from keystone and place those in the config file and at first blush I'm curious if that relation is complete and the config files are updated with appropriate info

Revision history for this message
Marcelo Subtil Marcal (msmarcal) wrote :
Revision history for this message
Marcelo Subtil Marcal (msmarcal) wrote :

The identity.yaml from the unit:

https://pastebin.canonical.com/p/DsKkh43nQx/

Revision history for this message
Billy Olsen (billy-olsen) wrote :

@msmarcal - when you say manually running the command, what command did you manually run?

Revision history for this message
Billy Olsen (billy-olsen) wrote :

I spent a fair amount of time working with @msmarcal on this particular problem. When revisiting this, the sync was reporting the following:

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "/snap/simplestreams/27/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
          timeout=timeout
        File "/snap/simplestreams/27/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
          method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
        File "/snap/simplestreams/27/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
          raise MaxRetryError(_pool, url, error or ResponseError(cause))
      urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='cloud-images.ubuntu.com', port=80): Max retries exceeded with url: /releases/streams/v1/index.sjson (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffb0470a0b8>: Failed to establish a new connection: [Errno 101] Network is unreachable',))

Which felt like a proxy configuration issue. The manual sync performed above was run using the simplestreams.sstream-mirror-glance command directly, which depends on the OS_* environment variables being set.

There is a proxy in play for the configuration and the script to sync the images in charm tries *really* hard to configure the proxy environment settings [0] by sourcing the various proxy config files. The /etc/profile.d/juju-proxy.sh file only had logic to source the contents of /etc/juju-proxy.conf but that was empty. The /home/ubuntu/.juju-proxy file does not exist.

Workaround was to manually specify proxy variables on the commandline and run `python3 /usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py` or to add the environment settings manually to /etc/juju-proxy.conf (each line is `export <variable>=<value>`) and the sync images runs just fine.

[0] - https://opendev.org/openstack/charm-glance-simplestreams-sync/src/branch/master/files/glance-simplestreams-sync.sh

Revision history for this message
Billy Olsen (billy-olsen) wrote :

Adding juju - the charm provides a script that runs the glance-simplestream-sync logic and attempts to source the proxy information from known options. However, the information was not present within the set of known files that are used.

Revision history for this message
Ian Booth (wallyworld) wrote :

juju will set env vars like JUJU_CHARM_HTTPS_PROXY for the hook to be able to use what the various model https/ftp/no-proxy config has been set to.

Are you saying these are not set? These are intended for use in a hook.

As far as I am aware, ~/.juju-proxy is only set for when juju run is used to run a script manually.

The /etc/juju-proxy.conf is only used for legacy proxy values, not the newer juju ones. ie if you use "juju-https-proxy" etc this file is not used.

Revision history for this message
Ian Booth (wallyworld) wrote :

I'll mark as incomplete pending further clarification about whether legacy or newer style proxy values are being used. The observed behaviour seems consistent with new proxy values being used and perhaps the charm needs to account for that.

Changed in juju:
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers