[SRU] glance sync: need keystone v3 auth support

Bug #1686437 reported by Scott Moser
34
This bug affects 3 people
Affects Status Importance Assigned to Milestone
simplestreams
High
Unassigned
simplestreams (Ubuntu)
Medium
Unassigned
Xenial
Medium
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned
Disco
Medium
Unassigned

Bug Description

[Impact]

simplestreams can't sync images when keystone is configured to use v3, keystone v2 is deprecated since mitaka[0] (the version shipped with xenial)

The OpenStack Keystone charm supports v3 only since Queens and later[1]

[Test Case]

* deploy a openstack environment with keystone v3 enabled
  - get a copy of the bundle available at http://paste.ubuntu.com/p/hkhsHKqt4h/ , this bundle deploys a minimal version of xenial-mitaka.

Expected result:

- "glance image-list" lists trusty and xenial images
- the file glance-simplestreams-sync/0:/var/log/glance-simplestreams-sync.log contains details of the images pulled from cloud-images.u.c (example: https://pastebin.ubuntu.com/p/RWG8QrkVDz/ )

Actual result:

- "glance image-list" is empty
- the file glance-simplestreams-sync/0:/var/log/glance-simplestreams-sync.log contains the following stacktrace
INFO * 04-09 22:04:06 [PID:14571] * root * Calling DryRun mirror to get item list
ERROR * 04-09 22:04:06 [PID:14571] * root * Exception during syncing:
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 471, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 232, in do_sync
    objectstore=store)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 374, in __init__
    super(ItemInfoDryRunMirror, self).__init__(config, objectstore)
  File "/usr/lib/python2.7/dist-packages/simplestreams/mirrors/glance.py", line 126, in __init__
    self.keystone_creds = openstack.load_keystone_creds()
  File "/usr/lib/python2.7/dist-packages/simplestreams/openstack.py", line 61, in load_keystone_creds
    raise ValueError("(tenant_id or tenant_name)")
ValueError: (tenant_id or tenant_name)

[Regression Potential]

* A possible regression will manifest itself figuring out if v2 or v3 should be used, after the connection is made there are no further changes introduced by this SRU

[Other Info]

When trying to test my changes for bug 1686086, I was unable to auth
to keystone, which means glance image sync just doesn't work with
a v3 keystone.

Related bugs:
 * bug 1719879: swift client needs to use v1 auth prior to ocata
 * bug 1728982: openstack mirror with keystone v3 always imports new images
 * bug 1611987: glance-simplestreams-sync charm doesn't support keystone v3

[0] https://docs.openstack.org/releasenotes/keystone/mitaka.html#deprecation-notes
[1] https://docs.openstack.org/charm-guide/latest/1802.html#keystone-support-is-v3-only-for-queens-and-later

Related branches

Scott Moser (smoser)
Changed in simplestreams:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Scott Moser (smoser) wrote :

i've linked a work-in-progress branch.
https://code.launchpad.net/~smoser/simplestreams/trunk.openstack-v3-auth
there i had keystone auth working.

Scott Moser (smoser)
Changed in simplestreams (Ubuntu):
importance: Undecided → Medium
status: New → In Progress
assignee: nobody → Scott Moser (smoser)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package simplestreams - 0.1.0~bzr450-0ubuntu1

---------------
simplestreams (0.1.0~bzr450-0ubuntu1) artful; urgency=medium

  * New upstream snapshot.
    - Keystone v3 Support [David Ames] (LP: #1686437)
    - flake8/pycodestyle updates.
    - tests: change to having http server select its own port
    - Support filters that contain a '-' in the tag name
    - Improvements for running flake8 in different Ubuntu release
      environments.
    - add running of tox.
    - json2streams: Accept items with no size.
    - tools changes (not related to package functionality)
      - tools/ubuntu_versions.py: Exclude old versions by version not name
      - Update default LTS alias to point to Xenial (LP: #1606606)
      - Create chksum for LXD metadata+root for squashfs (LP: #1577922)

 -- Scott Moser <email address hidden> Thu, 14 Sep 2017 10:38:07 -0400

Changed in simplestreams (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Felipe Reyes (freyes) wrote :

Scott, will 0.1.0~bzr450-0ubuntu1 be backported to xenial?

Felipe Reyes (freyes)
tags: added: sts
Scott Moser (smoser)
Changed in simplestreams (Ubuntu Xenial):
status: New → Confirmed
Changed in simplestreams (Ubuntu Zesty):
status: New → Confirmed
Changed in simplestreams (Ubuntu Xenial):
importance: Undecided → Medium
Changed in simplestreams (Ubuntu Zesty):
importance: Undecided → Medium
Changed in simplestreams:
status: Confirmed → Fix Committed
Revision history for this message
Eric Desrochers (slashd) wrote :

Felipe,

I just talked to smoser this morning and he agreed on backporting this into supported stable release.
We already set the release nominations accordingly.

There is a few details that smoser and thedac need to review together and they will keep us posted.

- Eric

Scott Moser (smoser)
description: updated
Eric Desrochers (slashd)
tags: added: sts-sru-needed
Eric Desrochers (slashd)
tags: removed: sts-sru-needed
Scott Moser (smoser)
description: updated
Scott Moser (smoser)
description: updated
Scott Moser (smoser)
Changed in simplestreams (Ubuntu Zesty):
status: Confirmed → Won't Fix
Scott Moser (smoser)
description: updated
Felipe Reyes (freyes)
description: updated
Felipe Reyes (freyes)
summary: - glance sync: need keystone v3 auth support
+ [SRU] glance sync: need keystone v3 auth support
Revision history for this message
David Ames (thedac) wrote :

Noting here the released version on xenial does not currently support Keystone v3 and blocks Bug #1611987.

For the record, we have been running a bzr branch @455 on serverstack (a Keystone v3 cloud) for months now. So the code in simplestreams works, it just needs to get to xenial.

David Britton (dpb)
Changed in simplestreams (Ubuntu):
assignee: Scott Moser (smoser) → nobody
Changed in simplestreams (Ubuntu Xenial):
assignee: nobody → Eric Desrochers (slashd)
Eric Desrochers (slashd)
Changed in simplestreams (Ubuntu Xenial):
assignee: Eric Desrochers (slashd) → Felipe Reyes (freyes)
Revision history for this message
Ryan Beisner (1chb1n) wrote :

I believe there is a ksv2 regression in the dev ppa of simplestreams, but I can also confirm that the ksv3 scenario is resolved with the dev ppa.

#### PASS:
ksv2 + 0.1.0~bzr426-0ubuntu1.2 (distro) + xenial-pike

ksv3 + 0.1.0~bzr459~trunk-0ubuntu1~ubuntu16.04.1 (dev ppa) + xenial-queens

#### FAIL:
ksv2 + 0.1.0~bzr459~trunk-0ubuntu1~ubuntu16.04.1 (dev ppa) + xenial-pike

ksv3 + 0.1.0~bzr426-0ubuntu1.2 (distro) + xenial-queens

https://paste.ubuntu.com/p/HxttSNpfc8/

Revision history for this message
Ryan Beisner (1chb1n) wrote :

Also, exercising the SRU ppa, I see xenial-pike fail similarly with ksv2:

https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3

https://paste.ubuntu.com/p/kVnFqsj8Gj/

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Scott, or anyone else affected,

Accepted simplestreams into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/simplestreams/0.1.0~bzr426-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in simplestreams (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Scott Moser (smoser) wrote :

Hi,
I've marked this verification-failed as we had found some issues and forgotten that there was an upload in the xenial queue.

tags: added: verification-failed verification-failed-xenial
removed: verification-needed verification-needed-xenial
Revision history for this message
Scott Moser (smoser) wrote : Fixed in simplestreams version 0.1.0.

This bug is believed to be fixed in simplestreams in version 0.1.0. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in simplestreams:
status: Fix Committed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

Hi,
The current state of this bug:
a.) 0.1.0~bzr426-0ubuntu1.3 was uploaded. It contained a fix for
bug 1686437. That change contained a regression (bug 1728982, bug 1719879),
so the bug was marked xenial-verification-failed.

b.) We have subsequently done work to fix that regression
A merge proposal is made for xenial at [1]. We have a PPA at [2] that
contains a fix for the regression caused by 1.3 and others.

Someone can/should test the PPA [1] and then upload a
0.1.0~bzr426-0ubuntu1.4 to xenial.

--
[1] https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214
[2] https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3/+packages

Revision history for this message
Chris Newcomer (cnewcomer) wrote :

Hi,

I've had a chance to test the fix for this. I was able to confirm that the 1.2 version of the package had an issue with the v3 keystone API. I then installed the 1.4 version of the package and it was able to successfully sync the images.

glance-simplestreams-sync log file included

Revision history for this message
Chris Newcomer (cnewcomer) wrote :

I forgot to include my test cloud info:
Openstack Queens running on Xenial

| 458939d4dc404a47a8a496b7d513a5ab | RegionOne | keystone | identity | True | internal | http://10.5.0.18:5000/v3 |
| 930b5cf107724f87845a645fc625fc07 | RegionOne | keystone | identity | True | admin | http://10.5.0.18:35357/v3 |
| b7fec4dbcddf42e8867d2750f95378fc | RegionOne | keystone | identity | True | public | http://10.5.0.18:5000/v3 |

juju status output is attached

Revision history for this message
Edward Hope-Morley (hopem) wrote :

I've tested this as well on xenial. This is what I tested:

root@juju-97b42b-0-lxd-4:~# apt-cache policy simplestreams
simplestreams:
  Installed: (none)
  Candidate: 0.1.0-482-g409fdc1-0ubuntu1~ubuntu16.04.1
  Version table:
     0.1.0-482-g409fdc1-0ubuntu1~ubuntu16.04.1 500
        500 http://ppa.launchpad.net/simplestreams-dev/trunk/ubuntu xenial/main amd64 Packages
     0.1.0~bzr426-0ubuntu1.2 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
     0.1.0~bzr426-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages

The 0.1.0-482-g409fdc1-0ubuntu1~ubuntu16.04.1 package was successfully able to sync images with glance and swift.

Revision history for this message
Edward Hope-Morley (hopem) wrote :

Also, it seems that the xenial-proposed (1.3) package has been deleted so we need to get the PPA version resubmitted as an sru and test that.

tags: removed: sts verification-failed verification-failed-xenial
Changed in simplestreams (Ubuntu Xenial):
status: Fix Committed → New
no longer affects: simplestreams (Ubuntu Zesty)
Changed in simplestreams (Ubuntu Xenial):
assignee: Felipe Reyes (freyes) → nobody
Revision history for this message
Edward Hope-Morley (hopem) wrote :

The adjoining bug 1611987 to this one says there is a fix landed in Bionic but it is not clear from the changelog whether that actually fixes v3 support so im going to leave this targeted at X/B/C/D under the assumption that none is fixed yet.

Revision history for this message
David Ames (thedac) wrote :

For the keystone v3 fixes revno 454 is the minimum we need SRU'd back to xenial.
Bionic 0.1.0~bzr460-0ubuntu1 has these changes.

These two merges are the pertinent changes:

https://code.launchpad.net/~thedac/simplestreams/keystone-v3-support/+merge/325781
https://code.launchpad.net/~thedac/simplestreams/lp1719879/+merge/333011

A package 0.1.0~bzr454-0ubuntu1 existed in xenial-proposed at one time. Still trying to figure out what happened to that package.

It would seem the SRU process needs to occur on:
https://bugs.launchpad.net/simplestreams/+bug/1719879

Junien Fridrick (axino)
Changed in simplestreams (Ubuntu Disco):
status: Fix Released → New
Revision history for this message
Junien Fridrick (axino) wrote :

Version 0.1.0~bzr460-0ubuntu1 from bionic on fixes the problem, so I'm marking B/C/D as "Fix Released".

Changed in simplestreams (Ubuntu Bionic):
status: New → Fix Released
Changed in simplestreams (Ubuntu Cosmic):
status: New → Fix Released
Changed in simplestreams (Ubuntu Disco):
status: New → Fix Released
Changed in simplestreams (Ubuntu Xenial):
assignee: nobody → Rafael David Tinoco (rafaeldtinoco)
status: New → Confirmed
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

SUMMARY FOR SIMPLESTREAMS SRU TO XENIAL

After analyzing the following bugs:

• LP: #1611987 - simplestreams - [SRU] glance-simplestreams-sync charm doesn't support keystone v3
• LP: #1686437 - simplestreams - can't sync images for keystone v3
• LP: #1719879 - simplestreams - [SRU] swift client needs to use v1 auth prior to ocata
• LP: #1728982 - simplestreams - [SRU] openstack mirror with keystone v3 always imports new images

For the keystone v3 fixes revno 454 is the minimum we need SRU'd back to xenial. Bionic 0.1.0~bzr460-0ubuntu1 has these changes. These two merges are the pertinent changes:

 1. Keystone v3 Support - https://is.gd/wq7r6g
 2. Fix KSv3 Bugs - https://is.gd/OOEo3G

0.1.0~bzr426-0ubuntu1.3 was uploaded. It contained a fix for LP: #1686437 (can't sync images for keystone v3). That change contained a regression (LP: #1728982 - openstack mirror with keystone v3 always imports new images - and LP: #1719879 - swift client needs to use v1 auth prior to ocata) and was marked as verification needed.

Work was done to fix that regression. A merge proposal is made for Xenial at https://is.gd/7ixQbO. We have a PPA at https://is.gd/Boda8J that contains a fix for the regression caused by 0.1.0~bzr426-0ubuntu1.3 and others.

Feedback from that PPA was asked and was given by Ed, Chris, Felipe and Billy. Billy found an issue about squashfs and that was fixed into 0.1.0~bzr426-0ubuntu1.4~ppa0, also uploaded to PPA at https://is.gd/Boda8J.

SRU template is needed in all referenced bugs:

 • 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
 • 433-glance-ignore-inactive-images.patch (LP: #1583276)
 • 436-glance-fix-race-conditions.patch (LP: #1584938)
 • 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
 • 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)

And version 0.1.0~bzr426-0ubuntu1.4 is good for a SRU and already tested by multiple people.

Please confirm this summary in order for me to prepare the SRU template and merge for Xenial SRU proposal.

Revision history for this message
Scott Moser (smoser) wrote :

Rafael's comment matches my memory, and also my comment in the merge proposal at https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214

Changed in simplestreams (Ubuntu Xenial):
status: Confirmed → In Progress
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

Thank you, Scott, for confirming my summary. I also got IRC confirmation from Billy and Ed.

With that, and because I had to "re-wrap" understanding, I am doing a similar merge request, fixing only commit comments, but keeping changelogs and credits, and asking for Ed and Felipe's review, and then Ubuntu Server Team review (for the SRU to happen).

I'll do a -1 in the existing Xenial SRU merge request, so I can keep a centralized place for the reviews, tests and approvals for everything.

Revision history for this message
Robie Basak (racb) wrote : Proposed package upload rejected

An upload of simplestreams to xenial-proposed has been rejected from the upload queue for the following reason: "See https://bugs.launchpad.net/charms/+source/glance-simplestreams-sync/+bug/1611987/comments/21".

Changed in simplestreams (Ubuntu Bionic):
status: Fix Released → In Progress
Changed in simplestreams (Ubuntu Xenial):
status: In Progress → Won't Fix
assignee: Rafael David Tinoco (rafaeldtinoco) → nobody
Changed in simplestreams (Ubuntu Bionic):
assignee: nobody → Rafael David Tinoco (rafaeldtinoco)
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

TL;DR version:

We are going to fix Bionic simplestreams package and provide this Bionic simplestreams package in Ubuntu Cloud Archive for Xenial. With that, we are fixing Xenial simplestreams behavior (to work with keystone v3) only if end-user enables Ubuntu Cloud Archive.

Conversation about this topic:

https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/simplestreams/+git/simplestreams/+merge/373030/comments/980111

Changed in simplestreams (Ubuntu Bionic):
status: In Progress → Fix Released
assignee: Rafael David Tinoco (rafaeldtinoco) → nobody
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers