Keystone rises not a NotFound exception

[Alexander Kurenyshev]

Found on CI: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.services_reconfiguration_thread_1/168/console

Test name: Check scalability of configured environment
Test steps:
1. Revert snapshot "reconfigure_nova_ephemeral_disk"
2. Upload a new openstack configuration for keystone
3. Wait for configuration applying
4. Verify keystone settings
5. >>>fail>>>Keystone actions<<<

What keystone action does:
On step 5 [0] we get the token expiration time from config and check that keystone throws the NotFound exception if the token is expired [1]

For some reason we have an exception in test:

Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.services_reconfiguration_thread_1/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.services_reconfiguration_thread_1/fuelweb_test/tests/test_services_reconfiguration.py", line 814, in reconfiguration_scalability
    self.check_token_expiration(os_conn, time_expiration)
  File "/home/jenkins/workspace/9.x.system_test.ubuntu.services_reconfiguration_thread_1/fuelweb_test/tests/test_services_reconfiguration.py", line 311, in check_token_expiration
    (token.id, )
AssertionError: Expected an exception of type <class 'keystoneauth1.exceptions.http.NotFound'> to be raised.

Need to investigate what is happened. Is it wrong exception from keystone or something else
[0] https://github.com/openstack/fuel-qa/blob/9399434e626e88af4302ed7d24c40a4c1946480a/fuelweb_test/tests/test_services_reconfiguration.py#L808
[1] https://github.com/openstack/fuel-qa/blob/9399434e626e88af4302ed7d24c40a4c1946480a/fuelweb_test/tests/test_services_reconfiguration.py#L299

[Vladimir Khlyunev (vkhlyunev)]

After applying config apache2 was not restarted during api request:
300 sec timeout:

before restart:
'issued_at': u'2017-01-17T13:21:16.000000Z', u'expires': u'2017-01-17T14:21:16Z',

after restart:
'issued_at': u'2017-01-17T13:30:46.000000Z', u'expires': u'2017-01-17T13:35:46Z'

[Vladimir Khlyunev (vkhlyunev)]

Dev team, please confirm/give us correct data about config: https://github.com/openstack/fuel-qa/blob/stable/mitaka/fuelweb_test/config_templates/keystone.yaml

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/421307

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/421308

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/421309

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/421887

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/422148

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/422151

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421887
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=63805ab80a362d04795952110424c44d1eeae0cf
Submitter: Jenkins
Branch: master

commit 63805ab80a362d04795952110424c44d1eeae0cf
Author: Dmitry Ukov <email address hidden>
Date: Wed Jan 18 17:03:38 2017 +0400

    Resources shoud be created in a separate task

    We shoud not create resources in all tasks. Only resouce parameters
    update should be available for user. Separate task will be used
    in order to create new resources (e.g. to introduce new
    configuration file parameters).

    Change-Id: I89ff7d9b6434519107f88d26fceb564a3bda2edc
    Closes-Bug: 1644656
    Closes-Bug: 1655928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422151
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=24318bea019fd2a8312b280853f1e7fa34f2edff
Submitter: Jenkins
Branch: stable/mitaka

commit 24318bea019fd2a8312b280853f1e7fa34f2edff
Author: Dmitry Ukov <email address hidden>
Date: Wed Jan 18 17:03:38 2017 +0400

    Resources shoud be created in a separate task

    We shoud not create resources in all tasks. Only resouce parameters
    update should be available for user. Separate task will be used
    in order to create new resources (e.g. to introduce new
    configuration file parameters).

    Change-Id: I89ff7d9b6434519107f88d26fceb564a3bda2edc
    Closes-Bug: 1644656
    Closes-Bug: 1655928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422148
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=62e6754e7f30ded4d57357ab576f046edc37fe8b
Submitter: Jenkins
Branch: stable/newton

commit 62e6754e7f30ded4d57357ab576f046edc37fe8b
Author: Dmitry Ukov <email address hidden>
Date: Wed Jan 18 17:03:38 2017 +0400

    Resources shoud be created in a separate task

    We shoud not create resources in all tasks. Only resouce parameters
    update should be available for user. Separate task will be used
    in order to create new resources (e.g. to introduce new
    configuration file parameters).

    Change-Id: I89ff7d9b6434519107f88d26fceb564a3bda2edc
    Closes-Bug: 1644656
    Closes-Bug: 1655928

[Ekaterina Shutova (eshutova)]

Verified on 9.2 snapshot 804, test passed:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.services_reconfiguration_thread_1/181/testReport/(root)/reconfiguration_scalability/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421307
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=538cb116f070b065e45f6ac1e95f64389cfb8624
Submitter: Jenkins
Branch: master

commit 538cb116f070b065e45f6ac1e95f64389cfb8624
Author: Dmitry Ukov <email address hidden>
Date: Tue Jan 17 19:03:58 2017 +0400

    Restart keystone after token update

    Change-Id: Ibe9e7cbe7f34e6b32a5aa4c1a614c051080f2dbd
    Closes-Bug: 1655928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421309
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=698e57cc5586ff75c7292cc89fd4d8ed3855329c
Submitter: Jenkins
Branch: stable/mitaka

commit 698e57cc5586ff75c7292cc89fd4d8ed3855329c
Author: Dmitry Ukov <email address hidden>
Date: Tue Jan 17 19:03:58 2017 +0400

    Restart keystone after token update

    Change-Id: Ibe9e7cbe7f34e6b32a5aa4c1a614c051080f2dbd
    Closes-Bug: 1655928

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/421308
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=63a234d5354ac2f589e7d5ebc0c6210677b44f05
Submitter: Jenkins
Branch: stable/newton

commit 63a234d5354ac2f589e7d5ebc0c6210677b44f05
Author: Dmitry Ukov <email address hidden>
Date: Tue Jan 17 19:03:58 2017 +0400

    Restart keystone after token update

    Change-Id: Ibe9e7cbe7f34e6b32a5aa4c1a614c051080f2dbd
    Closes-Bug: 1655928

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-qa 11.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.