[SRU] glance-simplestreams-sync charm doesn't support keystone v3

[Brad Marshall]

[Impact]

simplestreams can't sync images when keystone is configured to use v3, keystone v2 is deprecated since mitaka[0] (the version shipped with xenial)

The OpenStack Keystone charm supports v3 only since Queens and later[1]

[Test Case]

* deploy a openstack environment with keystone v3 enabled
  - get a copy of the bundle available at http://paste.ubuntu.com/p/hkhsHKqt4h/ , this bundle deploys a minimal version of xenial-mitaka.

Expected result:

- "glance image-list" lists trusty and xenial images
- the file glance-simplestreams-sync/0:/var/log/glance-simplestreams-sync.log contains details of the images pulled from cloud-images.u.c (example: https://pastebin.ubuntu.com/p/RWG8QrkVDz/ )

Actual result:

- "glance image-list" is empty
- the file glance-simplestreams-sync/0:/var/log/glance-simplestreams-sync.log contains the following stacktrace
INFO * 04-09 22:04:06 [PID:14571] * root * Calling DryRun mirror to get item list
ERROR * 04-09 22:04:06 [PID:14571] * root * Exception during syncing:
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 471, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 232, in do_sync
    objectstore=store)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 374, in __init__
    super(ItemInfoDryRunMirror, self).__init__(config, objectstore)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 126, in __init__
    self.keystone_creds = openstack.load_keystone_creds()
  File "/usr/lib/python2.7/dist-packages/simplestreams/openstack.py", line 61, in load_keystone_creds
    raise ValueError("(tenant_id or tenant_name)")
ValueError: (tenant_id or tenant_name)

[Regression Potential]

* A possible regression will manifest itself figuring out if v2 or v3 should be used, after the connection is made there are no further changes introduced by this SRU

[Other Info]

I was deploying a Mitaka Trusty 16.04 charm based Openstack cloud (using the cloud archives), including glance-simplestreams-sync, using keystone v3.

Once I had everything deployed, the glance-simplestreams-sync service couldn't authenticate because it's using keystone v2, not v3, as you can see from the following:

INFO * 08-10 23:16:01 [PID:33554] * root * glance-simplestreams-sync started.
DEBUG * 08-10 23:16:01 [PID:33554] * keystoneclient.session * REQ: curl -i -X POST http://x.y.z.240:5000/v2.0/tokens -H "Content-Type: application/json" -H "User-Agent: python-keystoneclient" -d '{"auth": {"passwordCredentials": {"username": "image-stream", "password": "thisisnotapassword"}, "tenantId": "blahblahtenantidblahblah"}}'
INFO * 08-10 23:16:01 [PID:33554] * urllib3.connectionpool * Starting new HTTP connection (1): x.y.z.240
DEBUG * 08-10 23:16:01 [PID:33554] * urllib3.connectionpool * Setting read timeout to None
DEBUG * 08-10 23:16:01 [PID:33554] * urllib3.connectionpool * "POST /v2.0/tokens HTTP/1.1" 401 114
DEBUG * 08-10 23:16:01 [PID:33554] * keystoneclient.session * RESP: [401] CaseInsensitiveDict({'content-length': '114', 'vary': 'X-Auth-Token', 'server': 'Apache/2.4.7 (Ubuntu)', 'date': 'Wed, 10 Aug 2016 23:16:01 GMT', 'www-authenticate': 'Keystone uri="http://x.y.z.240:5000"', 'x-openstack-request-id': 'req-f8aaf12d-01ea-46be-869a-6948ab38361b', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

DEBUG * 08-10 23:16:01 [PID:33554] * keystoneclient.session * Request returned failure status: 401
DEBUG * 08-10 23:16:01 [PID:33554] * keystoneclient.v2_0.client * Authorization Failed.

Please update the charm to understand when its using keystone v3 and use the right auth methods.

Related bugs:
 * bug 1686437: glance sync: need keystone v3 auth support

[Liam Young (gnuoy)]

So this is fun.

Swift proxy v3 integration appears to be broken: Bug #1624304
The simplestreams package does not support v3 integration: Bug #1624306

[Liam Young (gnuoy)]

Swift Proxy Fix: https://review.openstack.org/#/c/371554/
Simplestreams Fix: https://code.launchpad.net/~gnuoy/simplestreams/ksv3/+merge/305963

[Junien F (axino)]

Hi,

What's the status on this ?

This is absolutely required on queens, which dropped keystone v2 support.

Also, https://jujucharms.com/glance-simplestreams-sync/11 "submit a bug" links to https://bugs.launchpad.net/charms/+source/glance-simplestreams-sync/+filebug, should it link to https://bugs.launchpad.net/charm-glance-simplestreams-sync/+filebug instead ?

Thanks

[Tom Haddon (mthaddon)]

Subscribed ~field-high. This bug affects any cloud running Queens, which dropped keystone v2 support.

[Ryan Beisner (1chb1n)]

This charm has no Queens integration tests, and it has no existing functional test suite to confirm upstream KSV3 commits, KSV2, or anything else for that matter.

I'm adding a basic functional test, based on the official glance charm, and augmenting that so that we can quickly exercise and assess this charm for the following:

 - Existing combos and older releases (@ KSV2)
 - Queens and later (KSV3)

To be clear, this effort does not add an automated test suite to the charm, but it does declare a reproducible test environment in line with other openstack charms. That test environment can be used to consistently develop and verify new and existing code.

It should be noted, also, that this charm lacks support for juju workload status. That means that this charm will not affirm a ready state for those waiting for a ready state through tools like juju-wait, leading to anticipated race conditions and certain automation blocking conditions. It is also not on the roadmap to provide that functionality in this charm.

Ultimately, the official glance openstack charm is slated to receive an image sync feature, and this GSS charm should be deprecated in favor of that when it is available.

We'll post our outcome of exercising the charm's git master against KSV3 combos as soon as possible.

Thank you.

[Ryan Beisner (1chb1n)]

I'm finding that the gss charm at master does not result in images being sync'd, and I believe this is why:

https://bugs.launchpad.net/simplestreams/+bug/1686437

[David Ames (thedac)]

The git version of glance-simplestreams-sync [0] is ready for keystone v3, however, we are blocked on the released version of simplestreams on xenial in Bug#1686437.

As a work around and to see the gsss charm work, on the gsss node run the following:

sudo apt-get install bzr
bzr branch lp:simplestreams
cd simplestreams
sudo python setup.py install

After this the sync will succeed.

We could discuss the possibility of putting simplestreams into the cloud archive and setting the gsss source accordingly.

[0] https://github.com/openstack-charmers/charm-glance-simplestreams-sync/
[1] https://bugs.launchpad.net/simplestreams/+bug/1686437

[Scott Moser (smoser)]

Hi,

Can someone test the PPA upload I've done at
 https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3

apt-add-repository -y ppa:smoser/sstream-ks3

[Ryan Beisner (1chb1n)]

@smoser, i'll exercise that now. Thank you.

[Ryan Beisner (1chb1n)]

@smoser, feedback on that posted to https://bugs.launchpad.net/simplestreams/+bug/1686437

[David Ames (thedac)]

@smoser

When using ppa:smoser/sstream-ks3 version 0.1.0~bzr426-0ubuntu1.3~ppa0

And a quick fix:
https://code.launchpad.net/~thedac/simplestreams/lp1611987/+merge/342971

I get an error:
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 473, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 246, in do_sync
    tmirror.sync(smirror, path=initial_path)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 91, in sync
    return self.sync_index(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 254, in sync_index
    self.sync(reader, path=epath)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 89, in sync
    return self.sync_products(reader, path, data, content)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/__init__.py", line 346, in sync_products
    (prodname, vername))
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 368, in insert_version
    self._insert_item(*iargs)

Sure enough _insert_item does not exist in this version although it does exist in tip of lp:simplestreams.

[Ryan Beisner (1chb1n)]

As of today, the charm @ git master can deploy Xenial-Queens, using ksv3, and sync images, if the dev ppa of simplestreams is used. There are SRUs of simplestreams packages underway.

We've added a test suite to the charm as well, which is what has allowed us to confirm the combos which are ok, and raise simplestreams regression reports against those that are not. We will be adding that test suite to a charm test gate in gerrit ASAP, to keep it all healthy going forward.

The updated charm will not be released to the charm store until we can verify that it actually works where expected and that it does not regress functionality on the existing releases. This validation is in tandem with the simplestreams SRU.

#### Workaround and Verification:
Use the charm at current master and set the 'source' charm config at deploy-time to 'ppa:simplestreams-dev/trunk' to pick up the version of simplestreams which supports ksv3.

We have not yet tested a charm-upgrade and post-deploy package upgrade scenario, but that should also be possible. If anyone validates that before we do, please report back on this bug with your findings.

#### Repo:
https://github.com/openstack-charmers/charm-glance-simplestreams-sync

https://github.com/openstack-charmers/charm-glance-simplestreams-sync/commit/d8355af37c1b8683b33b39c873a672abdfa3eb2c

#### Workload status correction:
Earlier in this bug, I observed that this charm had not implemented workload status updates, but it has, with some caveats.

The charm will show in an unknown state until the first image sync begins. If the sync never begins, the workload status is never updated. That condition exists today by default with the released version of simplestreams on Xenial, when used with ksv3, hence no workload status in my initial test case.

Ex. when things go well, workload status is updated and helpful:

glance-simplestreams-sync/1* maintenance idle 6 10.5.0.3 Synchronising images

We will keep this bug updated as it all progresses.

[Launchpad Janitor (janitor)]

This bug was fixed in the package simplestreams - 0.1.0~bzr460-0ubuntu1

---------------
simplestreams (0.1.0~bzr460-0ubuntu1) bionic; urgency=medium

  * New upstream snapshot.
    - Glance: handle Keystone v2 session based authentication.
      [David Ames] (LP: #1611987)

 -- Scott Moser <email address hidden> Thu, 12 Apr 2018 12:50:58 -0400

[Junien F (axino)]

I can confirm that the workaround proposed in #13 works. Small twist though : updating the "source" of an already installed glance-simplestream-sync application won't do anything, since the "source" option is only used at install time. You'll want to manually add the PPA and upgrade the python-simplestreams package.

[Chris Gregan (cgregan)]

Field High SLA now requires that a estimated date for a fix is listed in the comments. Please provide this estimate for the open tasks.

[Alvaro Uria (aluria)]

When could we expect a simplestreams backport to xenial? Thank you.

[Scott Moser (smoser)]

Hi,
There is more information on a SRU of simplestreams in bug 1686437.

https://bugs.launchpad.net/simplestreams/+bug/1686437/comments/11

[Alex Kavanagh (ajkavanagh)]

Marking the simplestreams bug as fix released as keystone v3 was added to it in 2017-09-14 with this commit: https://git.launchpad.net/simplestreams/commit/?id=1dab951f16541ee29d72a80296e4b4f413761067

[Rafael David Tinoco (rafaeldtinoco)]

For Xenial SRU, please read summary at:

https://bugs.launchpad.net/ubuntu/+source/simplestreams/+bug/1686437/comments/19

[Robie Basak (racb)]

> simplestreams can't sync images when keystone is configured to use v3, keystone v2 is deprecated since mitaka[0] (the version shipped with xenial)

I don't think this feature addition qualifies for an SRU under our usual policies. If an exception is needed we can discuss that, but we concluded on IRC in #ubuntu-devel that the expected way to fix this is via the Cloud Archive, rather than an SRU, so I'm rejecting this from the SRU upload queue.

[Robie Basak (racb)]

An upload of simplestreams to xenial-proposed has been rejected from the upload queue for the following reason: "See https://bugs.launchpad.net/charms/+source/glance-simplestreams-sync/+bug/1611987/comments/21".

[Rafael David Tinoco (rafaeldtinoco)]

TL;DR version:

We are going to fix Bionic simplestreams package and provide this Bionic simplestreams package in Ubuntu Cloud Archive for Xenial. With that, we are fixing Xenial simplestreams behavior (to work with keystone v3) only if end-user enables Ubuntu Cloud Archive.

Conversation about this topic:

https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/simplestreams/+git/simplestreams/+merge/373030/comments/980111