dockerctl check of keystone container is failing

[Alexey Stupnikov]

fuel-library patches [1] were pushed to stable/6.1, stable/7.0 and stable/8.0 branches to fix bug #1582893. The reason is admin_token_auth was deleted from keystone's pipelines.

[1] https://review.openstack.org/#/q/status:merged+project:openstack/fuel-library+topic:bug/1582893

Steps to reproduce:

1. Connect to fuel CLI

2. Start shell in keystone container (dockerctl shell keystone)

3. Run primary keystone puppets in debug mode:
puppet apply --debug --verbose --color false --detailed-exitcodes /etc/puppet/modules/nailgun/examples/keystone-only.pp

Expected: success

Actual result: puppet can't authenticate using admin token and can't apply all openstack commands

[Alexey Stupnikov (astupnikov)]

This issue is a locker for 8.0 MU3 swarm tests.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/341371

[Alexey Stupnikov (astupnikov)]

Matt said that we should check every version to confirm that this bug is not there.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/341371
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e03d04d23db7eb8fee04a874b17975cd207f3442
Submitter: Jenkins
Branch: stable/8.0

commit e03d04d23db7eb8fee04a874b17975cd207f3442
Author: Alexey Stupnikov <email address hidden>
Date: Wed Jul 13 11:54:30 2016 +0300

    Fix admin_token issue in keystone-only.pp manifest

    To fix bug #1582893 we have removed admin_token_auth middleware
    from public/admin/v3 pipelines. It turns out that this change made
    it impossible to use dockerctl check command in swarm tests, since
    openstack util were unable to authorize itself using admin_token.

    I have modified keystone-only.pp to temporary add admin_token_auth
    middleware during puppet execution.

    Change-Id: I00ad873cda454fcf062db3a383057ad12511922c
    Closes-bug: #1602573

[Dmitry Klenov (dklenov)]

Setting priority to high for other releases as there is no direct evidences that bug is there.

[Alexey Stupnikov (astupnikov)]

This bug is not reproducable for MOS6. Diag is attached.

[Alexey Stupnikov (astupnikov)]

This bug is not reproducable for MOS7.

[Alexey Stupnikov (astupnikov)]

I have to update my statements about this bug in MOS 6 and MOS7 distribution. This bug is reproducible there, but it doesn't break dockerctl check process. The only problem we got there is that we don't check some keystone DB settings because of missing pipeline's entries. As a result, user will not be able to fix some broken installations.

IMO we should fix this bug, but it is not critical issue and can wait for next MU.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/351115

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/351116

[Alexey Stupnikov (astupnikov)]

Interesting news is that problems in MOS 6 and MOS7 are not caused by the patch from bug #1582893. Closing both series as Invalid and abandoning patches for stable/6.1 and stable/7.0 branches.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexey Stupnikov (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/351116
Reason: No need for this patch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexey Stupnikov (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/351115
Reason: No need for this patch.

[Alexey Stupnikov (astupnikov)]

Check works well in MOS9:
[root@localhost puppet]# fuel-utils check_service keystone
checking with command "keystone --os-auth-url "http://10.88.0.50:35357/v2.0" --os-username "nailgun" --os-password "PASSWORD" token-get &>/dev/null"
checking with command "! pgrep puppet"
keystone is ready.

[Alexey Stupnikov (astupnikov)]

Looks like this is not an issue for MOS9. Closing as invalid, but feel free to re-open it if needed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/351115
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=8a7036751631b85249afe0295ca60d9630ed8a40
Submitter: Jenkins
Branch: stable/7.0

commit 8a7036751631b85249afe0295ca60d9630ed8a40
Author: Alexey Stupnikov <email address hidden>
Date: Wed Jul 13 11:54:30 2016 +0300

    Fix admin_token issue in keystone-only.pp manifest

    To fix bug #1582893 we have removed admin_token_auth middleware
    from public/admin/v3 pipelines. It turns out that this change made
    it impossible to use dockerctl check command in swarm tests, since
    openstack util were unable to authorize itself using admin_token.

    I have modified keystone-only.pp to temporary add admin_token_auth
    middleware during puppet execution.

    Change-Id: I00ad873cda454fcf062db3a383057ad12511922c
    Closes-bug: #1602573
    (cherry picked from commit e03d04d23db7eb8fee04a874b17975cd207f3442)

[Ekaterina Shutova (eshutova)]

Verified in scope MOS7.0 + mu5 updates.

Checked swarm tests that was failed due to this bug:
fuel_master_migrate: https://patching-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.fuel_master_migrate/
thread_non_func_1: https://patching-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.thread_non_func_1/
With fix both are passed.