Glance deployment with python3 + "keystone" paste_deploy flavor Fails

Bug #1789351 reported by yatin on 2018-08-28
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Undecided
Unassigned
keystonemiddleware
Undecided
wangxiyuan
oslo.config
Undecided
Unassigned
python-keystonemiddleware (Ubuntu)
High
James Page

Bug Description

This happens with oslo.config >= 6.3.0([1]) + python3 + "keystone" paste_deploy + current glance(before https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 it works)
Testing in devstack: https://review.openstack.org/#/c/596380/

The glance api service fails to start with below Error, reproducing here: https://review.openstack.org/#/c/596380/:-
ERROR: dictionary changed size during iteration , see logs below

Failure logs from job:- http://logs.openstack.org/80/596380/2/check/tempest-full-py3/514fa29/controller/logs/screen-g-api.txt.gz#_Aug_27_07_26_10_698243

The Runtime Error is returned at keystonemiddleware:- https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L551
Adding code snippet here:-
if self._conf.oslo_conf_obj != cfg.CONF: <-- Fails here
    oslo_cache.configure(self._conf.oslo_conf_obj)

So with pdb found that an additional key(fatal_deprecations) was added to cfg.CONF at ^^, so Error is returned in python3. With python2 same key is added but no Error.

There are multiple ways to avoid it, like use the paste_deploy configuration that works(ex: keystone+cachemanagement), use oslo.config <= 6.2.0, Use python2 or update glance(https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 as use_user_token is deprecated since long)
with keystone+cachemanagement, all the config items were added before reaching the Failure point in keystonemiddleware and self._conf.oslo_conf_obj != cfg.CONF didn't raised an error and returned Boolean. Don't know why.

But it seems a real issue to me as it may happen in python3 at different places. So it would be good if Teams from affected projects(oslo.config, keystonemiddleware, glance) can look at it and fix(not avoid) at the best place.
To me it looks like keystonemiddleware is not handling(comparing the dict) it properly for python3, as the conf is dynamically updated(how ? and when ?).

- so can oslo.config Team check if glance and keystonmiddleware are handling/using oslo.config properly.
- i checked keystone+cachemanagement is default in devstack from last 6 years, is "keystone" flavor supported? if yes it should be fixed. Also it would be good to cleanup the deprecated options those are deprecated since Mitaka.
- If it's wrongly used in keystonemiddleware/glance, it would be good to fix there.

Initially detected while testing with Fedora[2], but later digged on why it's working in CI with Ubuntu and started [3].

[1] https://review.openstack.org/#/c/560094/
[2] https://review.rdoproject.org/r/#/c/14921/
[3] https://review.openstack.org/#/c/596380/

Doug Hellmann (doug-hellmann) wrote :

I suspect that != line should be changed to "is not" for an object identity comparison instead of equality comparison.

As commented by Doug in comment #1 changing != to "is not" is fixing the issue, but I wonder why it occurs only for python3.

wangxiyuan (wangxiyuan) wrote :

In a recent oslo.config change[1] , it will call "versionutils.report_deprecated_feature" if the option is deprecated when getting the option.

And then in oslo.log "versionutils.report_deprecated_feature"[2], it will register a new option, so in this bug, when compare the CONF(there is a deprecated options "registry_host") in keystonemiddleware, the CONF's context is changed, that's the reason why the Error is "dictionary changed size during iteration" happened.

I'm not sure change != to "is not" in keystonemiddleware is the best way, because in keystonemiddleware here, it want to compare the content. If change to "is not", it'll not compare the content any more. (i.e. we should double check that it will not introduce other bugs)

I think the best way may be fixing it in oslo.log. When register oslo.log's options[3], it should register the "fatal_deprecations" as well.

Or a minimum change is in Glance, just add two lines in glance.cmd.api:

from oslo_log import versionutils
versionutils.register_options()

[1]: https://review.openstack.org/#/c/560094/4/oslo_config/cfg.py
[2]: https://git.openstack.org/cgit/openstack/oslo.log/tree/oslo_log/versionutils.py#n286
[3]: https://git.openstack.org/cgit/openstack/oslo.log/tree/oslo_log/log.py#n241

Doug Hellmann (doug-hellmann) wrote :

Applications should not need to worry about registering options used by libraries, so don't make that change, please.

Why does keystonemiddleware want to compare the contents of the configuration objects?

wangxiyuan (wangxiyuan) wrote :

Compare the contents is not very needed. After thinking again, I'm OK with "if not" now. It won't break the logic IMO.

Fix proposed to branch: master
Review: https://review.openstack.org/599936

Changed in keystonemiddleware:
assignee: nobody → wangxiyuan (wangxiyuan)
status: New → In Progress

Fixed in keystonemiddleware.

Changed in glance:
status: New → Invalid
yatin (yatinkarel) wrote :

@Abhishek good to track the issue(mentioned in the report and setting registry_host in glance api v2) found in glance in a separate bug so those are worked upon :)

Reviewed: https://review.openstack.org/599936
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=4fb7fef1eab2a981b8e366091073b6ee2d80ee6a
Submitter: Zuul
Branch: master

commit 4fb7fef1eab2a981b8e366091073b6ee2d80ee6a
Author: wangxiyuan <email address hidden>
Date: Wed Sep 5 10:21:51 2018 +0800

    No need to compare CONF content

    When setup AuthProtocol class, if the CONF object contains
    deprecated options, An Error "dictionary changed size during
    iteration" will raise when comparing the CONF content.

    Changing "!=" to "is not" here to avoid compare the CONF
    content anymore.

    Change-Id: I820aa244160db4f81149d2576386c86b46de0084
    Closes-bug: #1789351

Changed in keystonemiddleware:
status: In Progress → Fix Released
James Page (james-page) on 2018-09-07
Changed in python-keystonemiddleware (Ubuntu):
status: New → In Progress
assignee: nobody → James Page (james-page)
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-keystonemiddleware - 5.2.0-0ubuntu2

---------------
python-keystonemiddleware (5.2.0-0ubuntu2) cosmic; urgency=medium

  * d/p/bug1789351.patch: Cherry pick fix for dict iteration issue under
    Python 3 (LP: #1789351).

 -- James Page <email address hidden> Fri, 07 Sep 2018 14:24:11 +0100

Changed in python-keystonemiddleware (Ubuntu):
status: In Progress → Fix Released

This issue was fixed in the openstack/keystonemiddleware 5.3.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers