[sys-tests] Changing of keystone password fails: 'Bad Gateway (HTTP 502)'

[Artem Panchenko]

Fuel version: http://paste.openstack.org/show/149606/

Changing of keystone admin password fails in system tests when we revert environment from snapshot and then reset Fuel master node (because it isn't accessible after resume):

http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0_fuelmain.system_test.centos.thread_5/53/testReport/junit/%28root%29/ha_delete_vips/ha_delete_vips/
http://paste.openstack.org/show/149607/

As you can see after master node reset it became to respond on 8000/tcp port and then test got Fuel version using API. This should mean that all services are started and work fine on master node but it doesn't, because access to '/api/version/' url isn't protected and doesn't require valid keystone token in request. So actually keystone service was starting at that moment and changing of admin tenant password at the next step failed:

http://paste.openstack.org/show/149608/

We should wait until all services are started and running on master node after its reset before proceed with tests.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/141194

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/141195

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141195
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=01a807a40012f26c5041d9144a351ac8b0dbbc87
Submitter: Jenkins
Branch: stable/6.0

commit 01a807a40012f26c5041d9144a351ac8b0dbbc87
Author: Artem Panchenko <email address hidden>
Date: Fri Dec 12 00:37:39 2014 +0200

    Wait for Keystone start after master node reset

    Use protected Nailgun API URL to check that Keystone
    is up and works fine before running tests.

    Change-Id: I28a84f8e2a304e18cf54eb7d08b24f0ef67503f5
    Closes-bug: #1401692

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141194
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=4a00e4565b879f110ff56a3c941c0bd81efa7086
Submitter: Jenkins
Branch: master

commit 4a00e4565b879f110ff56a3c941c0bd81efa7086
Author: Artem Panchenko <email address hidden>
Date: Fri Dec 12 00:37:39 2014 +0200

    Wait for Keystone start after master node reset

    Use protected Nailgun API URL to check that Keystone
    is up and works fine before running tests.

    Change-Id: I28a84f8e2a304e18cf54eb7d08b24f0ef67503f5
    Closes-bug: #1401692

[Dennis Dmitriev (ddmitriev)]

The same issue faced in the upgrade job 'ceph_multinode_compact.upgrade' , when environment 5.1.1 was reverted:

http://jenkins-product.srt.mirantis.net:8080/view/6.0_swarm/job/6.0.system_test.Ubuntu.ceph_multinode_compact.upgrade/20/console

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/143264

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/143264
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=bb837820f27c00cf54620e65934184d65f7a898a
Submitter: Jenkins
Branch: stable/5.1

commit bb837820f27c00cf54620e65934184d65f7a898a
Author: Artem Panchenko <email address hidden>
Date: Fri Dec 12 00:37:39 2014 +0200

    Wait for Keystone start after master node reset

    Use protected Nailgun API URL to check that Keystone
    is up and works fine before running tests.

    Change-Id: I28a84f8e2a304e18cf54eb7d08b24f0ef67503f5
    Closes-bug: #1401692
    (cherry picked from commit 4a00e4565b879f110ff56a3c941c0bd81efa7086)