Master-node deploy failed due removed admin_token_auth pipeline in keystone

[Igor Yozhikov]

Fuel iso:
fuel-10.0-397-2016-06-24_05-00-00.iso

Example of failed jobs:
https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-centos/862/console
https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-centos/863/console

Detailed info:
Environments were reverted and analyzed and found miss-configuration:
 keystone-paste.ini file has no required admin_token_auth option in pipeline.
 Keystone.conf has admin_token

https://github.com/openstack/keystone/blob/stable/mitaka/etc/keystone-paste.ini#L54

With with options keystone service is unable to authenticate user using admin_token.
Since that required to add checks for proper keystone configuration before launching of keystone task on fuel master node.

This problem is affecting further work on packages update

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/334576

[Ivan Berezovskiy (iberezovskiy)]

We've analyzed https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-centos/862/console failure.

Problem is next:
puppet modular tasks for master node were applied two times (or even bootstrap_admin_node.sh script was run two times).

First run was ok: keystone was successfully deployed - you can check puppet report at /var/lib/puppet/reports/nailgun.test.domain.local/201606271452.yaml

As a result master node was ready at 15.07. You can also check keystone logs, they are ok at 15.07 and then they are interrupted:

2016-06-27 15:07:30.639 23663 INFO eventlet.wsgi.server [req-32116659-8d3f-4e60-a657-98046ebb7ecb ac865ae3550c45fc91a6468c0eaa6189 4533009dfe7a441fa949e3782edcb1ea - 004b44ef6c5947a38c0be59aa5bc49a0 004b44ef6c5947a38c0be59aa5bc49a0] 10.109.25.2 - - [27/Jun/2016 15:07:30] "GET /v2.0/tokens/df2ab78c4ab240c8b9a6f3bccd8ad3fb HTTP/1.1" 200 1606 0.037590

Then some activity was started at 15:58:

2016-06-27 15:58:23.325 23665 INFO keystone.common.wsgi [req-c8962249-f4b3-4c72-a3f2-146cfb4b5d0f - - - - -] POST http://10.109.25.2:5000/v2.0/tokens
2016-06-27 15:58:23.491 23665 INFO eventlet.wsgi.server [req-c8962249-f4b3-4c72-a3f2-146cfb4b5d0f - - - - -] 10.109.25.2 - - [27/Jun/2016 15:58:23] "POST /v2.0/tokens HTTP/1.0" 200 1575 0.174539
2016-06-27 15:58:23.509 23665 INFO keystone.common.wsgi [req-d8594f1d-df87-4f76-9f3f-b9ff600ac020 3940ed22549949cdb63e0242d858f4d7 10e62a41261745929d6d164fdbbca107 - 004b44ef6c5947a38c0be59aa5bc49a0 004b44ef6c5947a38c0be59aa5bc49a0] POST http://10.109.25.2:5000/v2.0/tokens
2016-06-27 15:58:23.611 23665 INFO eventlet.wsgi.server [req-d8594f1d-df87-4f76-9f3f-b9ff600ac020 3940ed22549949cdb63e0242d858f4d7 10e62a41261745929d6d164fdbbca107 - 004b44ef6c5947a38c0be59aa5bc49a0 004b44ef6c5947a38c0be59aa5bc49a0] 10.109.25.2 - - [27/Jun/2016 15:58:23] "POST /v2.0/tokens HTTP/1.0" 200 1601 0.113422
2016-06-27 16:00:37.990 23662 INFO eventlet.wsgi.server [req-f2235ced-04b5-4211-9ad8-4687b7adca87 - - - - -] (23662) wsgi exited, is_accepting=True
2016-06-27 16:00:37.991 23655 INFO oslo_service.service [req-b646f863-8aae-46ae-ac34-40464556e07f - - - - -] Caught SIGTERM, stopping children
2016-06-27 16:00:37.996 23664 INFO eventlet.wsgi.server [req-6ecdb753-fa06-4994-908b-a06fe65206e5 3940ed22549949cdb63e0242d858f4d7 10e62a41261745929d6d164fdbbca107 - 004b44ef6c5947a38c0be59aa5bc49a0 004b44ef6c5947a38c0be59aa5bc49a0] (23664) wsgi exited, is_accepting=True
2016-06-27 16:00:37.997 23663 INFO eventlet.wsgi.server [req-32116659-8d3f-4e60-a657-98046ebb7ecb ac865ae3550c45fc91a6468c0eaa6189 4533009dfe7a441fa949e3782edcb1ea - 004b44ef6c5947a38c0be59aa5bc49a0 004b44ef6c5947a38c0be59aa5bc49a0] (23663) wsgi exited, is_accepting=True
2016-06-27 16:00:38.014 23655 INFO oslo_service.service [req-b646f863-8aae-46ae-ac34-40464556e07f - - - - -] Waiting on 4 children to exit
2016-06-27 16:00:38.016 23655 INFO oslo_service.service [req-b646f863-8aae-46ae-ac34-40464556e07f - - - - -] Child 23665 killed by signal 15
2016-06-27 16:00:38.029 23655 INFO oslo_service.service [req-b646f863-8aae-46ae-ac34-40464556e07f - - - - -] Child 23662 exited with status 0
2016-06-27 16:00:38.030 23655 INFO oslo_service.service [req-b646f863-8aae-46ae...

[Ivan Berezovskiy (iberezovskiy)]

Fuel CI and Fuel QA teams, please investigate why puppet (or even bootstrap_admin_node.sh sctipt) was run two times in this test scenario.

[Nastya Urlapova (aurlapova)]

@Ivan, @Igor, the job with failure was using the same fuel-qa sha 838238138c82a00f1fd29213d6be4fa6d9613711 as job which was green https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-centos/859/console

[Nastya Urlapova (aurlapova)]

May be there is some change in repo, you can try find.

[Ivan Berezovskiy (iberezovskiy)]

I've created separate bug to fix fuel-qa test scenario (bug - https://bugs.launchpad.net/fuel/+bug/1596953). We need to perform master node updates in another way (not like now https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/models/environment.py#L609-L650).

But we still need to fix puppet tasks for keystone. We should be able to re-run puppet tasks on master node and keep master node working.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/334576
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b4799980088c4d44880784080319be8eb1e1d645
Submitter: Jenkins
Branch: master

commit b4799980088c4d44880784080319be8eb1e1d645
Author: Dmitry Ilyin <email address hidden>
Date: Mon Jun 27 20:17:39 2016 +0300

    Ensure that keystone has admin token enabled

    * Ensure that admin_token_auth filter is enabled
      in the paste file before working with keystone
      Puppet types.

    Change-Id: I2baad2f6f775f3aa9c6c9624ec53f08fed479fb1
    Closes-Bug: 1596622

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/338940

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/338940
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f3e844581a01535036ef73e2ea53b496e804fa5e
Submitter: Jenkins
Branch: stable/mitaka

commit f3e844581a01535036ef73e2ea53b496e804fa5e
Author: Dmitry Ilyin <email address hidden>
Date: Mon Jun 27 20:17:39 2016 +0300

    Ensure that keystone has admin token enabled

    * Ensure that admin_token_auth filter is enabled
      in the paste file before working with keystone
      Puppet types.

    cherry picked from b4799980088c4d44880784080319be8eb1e1d645
    Change-Id: I2baad2f6f775f3aa9c6c9624ec53f08fed479fb1
    Closes-Bug: 1596622

[Kristina Berezovskaia (kkuznetsova)]

Verified on
CUSTOM_VERSION=snapshot #116
MAGNET_LINK=magnet:?xt=urn:btih:bfec808dd71ff42c5613a3527733d9012bb1fabc&dn=MirantisOpenStack-9.0.iso&tr=http%3A%2F%2Ftracker01-bud.infra.mirantis.net%3A8080%2Fannounce&tr=http%3A%2F%2Ftracker01-scc.infra.mirantis.net%3A8080%2Fannounce&tr=http%3A%2F%2Ftracker01-msk.infra.mirantis.net%3A8080%2Fannounce&ws=http%3A%2F%2Fvault.infra.mirantis.net%2FMirantisOpenStack-9.0.iso
FUEL_QA_COMMIT=5279ce17271bc0ac6cefc8c0ac4b9482260531ce
UBUNTU_MIRROR_ID=ubuntu-2016-08-03-174238
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-08-09-160321
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-08-09-170321
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-07-18-162958
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

Ran /usr/sbin/bootstrap_admin_node.sh, everything worked well