DuplicateOptError intermittently breaks UT

Bug #1494327 reported by Matthew Edmonds on 2015-09-10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Brant Knudson

Bug Description

keystonemiddleware UTs sometimes (pretty often, but not consistently) fail with a DuplicateOptError as seen here: http://logs.openstack.org/73/217373/6/check/gate-keystonemiddleware-python34/03e20d0/console.html

2015-09-09 23:26:24.494 | FAIL: keystonemiddleware.tests.unit.auth_token.test_user_auth_plugin.V2UserPluginTests.test_with_service_information
2015-09-09 23:26:24.494 | tags: worker-3
2015-09-09 23:26:24.495 | ----------------------------------------------------------------------
2015-09-09 23:26:24.495 | Empty attachments:
2015-09-09 23:26:24.495 | pythonlogging:''
2015-09-09 23:26:24.495 | stderr
2015-09-09 23:26:24.495 | stdout
2015-09-09 23:26:24.495 |
2015-09-09 23:26:24.495 | Traceback (most recent call last):
2015-09-09 23:26:24.496 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/keystonemiddleware/tests/unit/auth_token/test_user_auth_plugin.py", line 95, in setUp
2015-09-09 23:26:24.496 | tenant_id=self.service_token.tenant_id)
2015-09-09 23:26:24.497 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/keystonemiddleware/tests/unit/auth_token/test_user_auth_plugin.py", line 32, in configure_middleware
2015-09-09 23:26:24.497 | self.cfg.register_opts(opts, group=group)
2015-09-09 23:26:24.497 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/fixture.py", line 105, in register_opts
2015-09-09 23:26:24.497 | self.register_opt(opt, group=group)
2015-09-09 23:26:24.497 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/fixture.py", line 93, in register_opt
2015-09-09 23:26:24.497 | self.conf.register_opt(opt, group=group)
2015-09-09 23:26:24.497 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/cfg.py", line 1824, in __inner
2015-09-09 23:26:24.498 | result = f(self, *args, **kwargs)
2015-09-09 23:26:24.498 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/cfg.py", line 1967, in register_opt
2015-09-09 23:26:24.498 | return group._register_opt(opt, cli)
2015-09-09 23:26:24.498 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/cfg.py", line 1345, in _register_opt
2015-09-09 23:26:24.498 | if _is_opt_registered(self._opts, opt):
2015-09-09 23:26:24.498 | File "/home/jenkins/workspace/gate-keystonemiddleware-python34/.tox/py34/lib/python3.4/site-packages/oslo_config/cfg.py", line 574, in _is_opt_registered
2015-09-09 23:26:24.499 | raise DuplicateOptError(opt.name)
2015-09-09 23:26:24.501 | oslo_config.cfg.DuplicateOptError: duplicate option: user-name

How to reproduce:
Just run `tox -e py27` in the keystonemiddleware code base serially, you will hit the failure as this.

This is referenced with this commit (https://review.openstack.org/#/c/216579) but it is not the root cause, we will still see the DuplicateOptError with user-id option but it's relatively not such frequently.

Dolph Mathews (dolph) on 2015-09-10
Changed in keystonemiddleware:
status: New → Confirmed
importance: Undecided → High
Changed in keystonemiddleware:
assignee: nobody → Lin Hua Cheng (lin-hua-cheng)
Dave Chen (wei-d-chen) wrote :

yep, I found the failure is caused by these testcase too, and asked in the oslo channel but got no answer.

Dave Chen (wei-d-chen) wrote :

And this is primarly caused by the `keystone_authtoken` group is not cleared after the testcase, then when `user-name` options is registered again in other testcase, the issue happened, there is no way in the oslo-config that could clear the group options. I am wondering whether this is a bug in oslo-config.

Changed in keystonemiddleware:
assignee: Lin Hua Cheng (lin-hua-cheng) → Dave Chen (wei-d-chen)
status: Confirmed → In Progress
Dave Chen (wei-d-chen) on 2015-09-14
description: updated
Brant Knudson (blk-u) wrote :

I'm able to recreate this consistently:

$ echo "keystonemiddleware.tests.unit.auth_token.test_auth_token_middleware.GeneralAuthTokenMiddlewareTest.test_auth_region_name" > my-list
$ echo "keystonemiddleware.tests.unit.auth_token.test_user_auth_plugin.V2UserPluginTests.test_user_information" >> my-list
$ .tox/py27/bin/python -m testtools.run discover --load-list my-list

These aren't the only 2 tests that will cause the 2nd test to fail. Could also use keystonemiddleware.tests.unit.auth_token.test_auth_token_middleware.GeneralAuthTokenMiddlewareTest.test_auth_region_name . I think it's actually any test that creates and AuthProtocol.

Fix proposed to branch: master
Review: https://review.openstack.org/223338

Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)
Brant Knudson (blk-u) on 2015-09-14
Changed in keystonemiddleware:
assignee: Brant Knudson (blk-u) → Dave Chen (wei-d-chen)
Matthew Edmonds (edmondsw) wrote :

Dave has also proposed https://review.openstack.org/#/c/222919/ that I don't see linked here.

Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)
Brant Knudson (blk-u) on 2015-09-15
Changed in keystonemiddleware:
assignee: Brant Knudson (blk-u) → Dave Chen (wei-d-chen)
Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)
Brant Knudson (blk-u) on 2015-09-15
Changed in keystonemiddleware:
assignee: Brant Knudson (blk-u) → Dave Chen (wei-d-chen)
Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)
Brant Knudson (blk-u) wrote :

This is preventing us from merging changes (such as requirements updates) in keystonemiddleware, and we've got to make these changes for the liberty freeze, so marking as critical.

Changed in keystonemiddleware:
importance: High → Critical
assignee: Brant Knudson (blk-u) → Dave Chen (wei-d-chen)
Dave Chen (wei-d-chen) wrote :

Change the assignee to Brant Knudson since he has the patch will properly the right fix.

Changed in keystonemiddleware:
assignee: Dave Chen (wei-d-chen) → Brant Knudson (blk-u)

Reviewed: https://review.openstack.org/223338
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=5352ff068dc0466b3a376e802512c2065c130455
Submitter: Jenkins
Branch: master

commit 5352ff068dc0466b3a376e802512c2065c130455
Author: Brant Knudson <email address hidden>
Date: Mon Sep 14 18:18:32 2015 -0500

    auth_token tests use clean config

    The auth_token tests were failing randomly (depending on which tests
    were run together) because the global config had options left in it
    by other tests (options were added as part of auth plugin
    initialization and oslo.config provides no way to remove registered

    This change makes it so that auth_token gets a fresh config for
    every test (other than one test that requires using the global CONF

    Closes-Bug: 1494327
    Change-Id: I647f7fade01e2a619f4a5d12a4e71897423469c1

Changed in keystonemiddleware:
status: In Progress → Fix Committed

Change abandoned by Brant Knudson (<email address hidden>) on branch: master
Review: https://review.openstack.org/222919
Reason: With https://review.openstack.org/#/c/223338/ , all the options are reset after each test, so no need for this.

Changed in keystonemiddleware:
milestone: none → 2.3.0
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers