Keystone container infinitely restarts due to puppet error

[Ihor Kalnytskyi]

The issue occurs only after upgrading from 5.1.1 to 6.0.

Keystone's puppet fails with error:

2014-12-09 13:04:40,134 DEBG 'docker-keystone' stdout output:
Error: /Stage[main]/Main/Keystone_user[admin]: Could not evaluate: Execution of '/usr/bin/keystone --os-auth-url http://127.0.0.1:35357/v2.0/ token-get' returned 1: The request you have made requires authentication. (HTTP 401)

2014-12-09 13:04:40,134 DEBG 'docker-keystone' stdout output:

2014-12-09 13:04:40,135 DEBG 'docker-keystone' stdout output:
Notice: /Stage[main]/Main/Keystone_user_role[admin@admin]: Dependency Keystone_user[admin] has failures: true
2014-12-09 13:04:40,135 DEBG 'docker-keystone' stdout output:

2014-12-09 13:04:40,136 DEBG 'docker-keystone' stdout output:
Warning: /Stage[main]/Main/Keystone_user_role[admin@admin]: Skipping because of failed dependencies
2014-12-09 13:04:40,136 DEBG 'docker-keystone' stdout output:

Full log: http://xsnippet.org/360356/raw/

Situation:

1. start.sh exits with error -> container stoped
2. supervisor starts it again
3. go to #1

Result:

Keystone on master node is not always available

[Matthew Mosesohn (raytrac3r)]

Please post the logs in /var/log/docker-logs/keystone/
There might be a DB issue.

Also, was the admin password changed in this env?

[Ihor Kalnytskyi (ikalnytskyi)]

Ok, the password was changed by means fuelcli but we haven't changed it in /etc/fuel/astute.yaml. So, the puppet reads old password and obviously fails to retrieve token.

[Matthew Mosesohn (raytrac3r)]

The issue is that the keystone command returns different output than the ruby provider expects.
Related bug here: https://bugs.launchpad.net/puppet-keystone/+bug/1340447
Related patch: https://review.openstack.org/#/c/106250/1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/140407

[Matthew Mosesohn (raytrac3r)]

Okay, so we should support the functionality where we create admin user, but the user resets his password (which is completely normal). We can add a "manage_password" parameter to keystone_user provider so we don't reset the password if it gets changed.

[Matthew Mosesohn (raytrac3r)]

Upstream bug for disabling management of passwords: https://bugs.launchpad.net/puppet-keystone/+bug/1400798
I'll address this half of the bug tomorrow.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/140648

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/140649

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/140652

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/140653

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/140654

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140407
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4a2980191869d3854bddd101093eaa5b02038d11
Submitter: Jenkins
Branch: master

commit 4a2980191869d3854bddd101093eaa5b02038d11
Author: Matthew Mosesohn <email address hidden>
Date: Tue Dec 9 20:04:09 2014 +0400

    Fix keystone password update for users

    keystone-client changed its error message when an invalid
    password is specified. The logic is updated so that puppet
    can update passwords for users.

    Backport of I090f7e2ee62ee189f37921c091fe51b6d587cd74

    Change-Id: I049bd99b9474204ff498863095beb8eecc641aee
    Partial-Bug: #1400701

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140648
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=fca8ef014c8daecb7a122f66c6a92e194cd57405
Submitter: Jenkins
Branch: master

commit fca8ef014c8daecb7a122f66c6a92e194cd57405
Author: Matthew Mosesohn <email address hidden>
Date: Wed Dec 10 14:33:16 2014 +0400

    new option manage_password for keystone_user

    Adds a new option manage_password, which defaults
    to 'True' for all keystone_user objects, that
    has the effect of enforcing the password of a given
    keystone_user. If this is disabled, the user may change
    his or her password at a later time and not be reset
    by the keystone Puppet module.

    Change-Id: I4bd59f233273374545953f16c7843488148096d6
    Closes-Bug: #1400701

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140649
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=b51da0684f4f0c2ac7f6f343dc3e603c1d8e708b
Submitter: Jenkins
Branch: master

commit b51da0684f4f0c2ac7f6f343dc3e603c1d8e708b
Author: Matthew Mosesohn <email address hidden>
Date: Wed Dec 10 14:36:07 2014 +0400

    unmanage password for admin keystone user in nailgun

    Setting keystone_user 'admin' to be unmanaged allows
    a user to change his or her password to another password
    and expect it to remain in tact after reapplying puppet
    during a Fuel Master upgrade scenario.

    Change-Id: I16a2c1fae2cc7916a9afc9847fcd03b20f431258
    Closes-Bug: #1400701

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140654
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=1f401c323b565886d13e7d1a1bc0be479e2a42bc
Submitter: Jenkins
Branch: stable/6.0

commit 1f401c323b565886d13e7d1a1bc0be479e2a42bc
Author: Matthew Mosesohn <email address hidden>
Date: Wed Dec 10 14:33:16 2014 +0400

    new option manage_password for keystone_user

    Adds a new option manage_password, which defaults
    to 'True' for all keystone_user objects, that
    has the effect of enforcing the password of a given
    keystone_user. If this is disabled, the user may change
    his or her password at a later time and not be reset
    by the keystone Puppet module.

    Change-Id: I4bd59f233273374545953f16c7843488148096d6
    Closes-Bug: #1400701

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140653
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f30886db9c2ae9d47596cbd6907bc36ccc530f6a
Submitter: Jenkins
Branch: stable/6.0

commit f30886db9c2ae9d47596cbd6907bc36ccc530f6a
Author: Matthew Mosesohn <email address hidden>
Date: Wed Dec 10 14:36:07 2014 +0400

    unmanage password for admin keystone user in nailgun

    Setting keystone_user 'admin' to be unmanaged allows
    a user to change his or her password to another password
    and expect it to remain in tact after reapplying puppet
    during a Fuel Master upgrade scenario.

    Change-Id: I16a2c1fae2cc7916a9afc9847fcd03b20f431258
    Closes-Bug: #1400701

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/140652
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=2841834b7edbc18afeb9bf604c38b7bc0a58f231
Submitter: Jenkins
Branch: stable/6.0

commit 2841834b7edbc18afeb9bf604c38b7bc0a58f231
Author: Matthew Mosesohn <email address hidden>
Date: Tue Dec 9 20:04:09 2014 +0400

    Fix keystone password update for users

    keystone-client changed its error message when an invalid
    password is specified. The logic is updated so that puppet
    can update passwords for users.

    Backport of I090f7e2ee62ee189f37921c091fe51b6d587cd74

    Change-Id: I049bd99b9474204ff498863095beb8eecc641aee
    Partial-Bug: #1400701

[Matthew Mosesohn (raytrac3r)]

Just as a closing summary for this bug, the patches applied fixed the following:
1 - keystone puppet module no longer errors when the password is changed from its initial state
2 - keystone_user now has manage_password option to define keystone users that may change their password
3 - Fuel Master keystone now creates "admin" user with an unmanaged password.

[Ihor Kalnytskyi (ikalnytskyi)]

The issue doesn't affect 5.1.1. In 5.1.1we still has old ruby keystone client, so the manifest can handle its response correctly.

Moving it to Invalid.

[Ihor Kalnytskyi (ikalnytskyi)]

Ok, in the comment above I've noticed that we don't have this issue in 5.1.1 and why.

Still, there's another related issue in 6.0 which was fixed as part of this bug - password resetting.

What that means? That means:

1. User changes his password through UI/CLI and it works.
2. By some reason the keystone container was restarted.
3. User's changed password is not working, since it was resetted by keystone's puppet and defaulted to one from /etc/fuel/astute.yaml

Thanks to @Matthew, we fixed in 6.0.

Fortunately, the issue doesn't occurs in 5.1.1 since we have commented code (which updates user's password) in 5.1.1 manifests: https://github.com/stackforge/fuel-library/blob/stable/5.1/deployment/puppet/keystone/lib/puppet/provider/keystone_user/keystone.rb#L83-L96

[Andrey Sledzinskiy (asledzinskiy)]

verified on {

    "build_id": "2014-12-18_01-32-01",
    "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
    "build_number": "56",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90",
    "production": "docker",
    "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9",
    "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.0",
    "release_versions": {
        "2014.1.3-5.1.1": {
            "VERSION": {
                "build_id": "2014-12-03_01-07-36",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "48",
                "api": "1.0",
                "nailgun_sha": "500e36d08a45dbb389bf2bd97673d9bff48ee84d",
                "production": "docker",
                "fuelmain_sha": "7626c5aeedcde77ad22fc081c25768944697d404",
                "astute_sha": "ef8aa0fd0e3ce20709612906f1f0551b5682a6ce",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1.1",
                "fuellib_sha": "a3043477337b4a0a8fd166dc83d6cd5d504f5da8"
            }
        },
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2014-12-18_01-32-01",
                "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4",
                "build_number": "56",
                "api": "1.0",
                "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90",
                "production": "docker",
                "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9",
                "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.0",
                "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"
            }
        }
    },
    "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"

}