MOS 5.1.1: keystone container can't start

[Timur Nurlygayanov]

This issue was found by BVT tests for checks new packages for stable repository.

Durring the automated tests we can see errors with connectivity to keystone endpoint:

2014-11-10 12:15:31,142 - WARNING http.py:47 -- Cant establish connection to keystone with url http://10.108.67.2:5000/v2.0
2014-11-10 12:15:31,157 - WARNING http.py:95 -- Authorization failure: Authentication required
2014-11-10 12:15:31,158 - INFO http.py:37 -- Initialize keystoneclient with url http://10.108.67.2:5000/v2.0
2014-11-10 12:15:31,161 - WARNING http.py:47 -- Cant establish connection to keystone with url http://10.108.67.2:5000/v2.0

example:
http://jenkins-product.srt.mirantis.net:8080/job/5.1.1.staging.centos.bvt_1/16/console

Steps To Reproduce:
1. Deploy Fuel main node.
2. Check keystone container.

Observed Result:
Keystone container can't start:

___________________
[root@nailgun ~]# dockerctl start keystone
Error: Cannot start container fuel-core-5.1.1-keystone: (exit status 1)
2014/11/10 12:24:12 Error: failed to start one or more containers
checking container keystone
No defined test for determining if keystone is ready.
keystone is ready.
___________________

[root@nailgun ~]# dockerctl restart keystone
Error: Cannot restart container fuel-core-5.1.1-keystone: (exit status 1)
2014/11/10 12:24:37 Error: failed to restart one or more containers

___________________

In log file /var/log/docker-keystone.log we can see the following errors:
- - - - - - - - - -

No defined test for determining if keystone is ready.
keystone is ready.
Error: Cannot start container fuel-core-5.1.1-keystone: (exit status 1)
2014/11/10 12:24:40 Error: failed to start one or more containers
Attaching to container fuel-core-5.1.1-keystone...
2014/11/10 12:24:40 You cannot attach to a stopped container, start it first
checking container keystone
No defined test for determining if keystone is ready.
keystone is ready.
Error: Cannot start container fuel-core-5.1.1-keystone: (exit status 1)
2014/11/10 12:24:51 Error: failed to start one or more containers
Attaching to container fuel-core-5.1.1-keystone...
2014/11/10 12:24:51 You cannot attach to a stopped container, start it first
checking container keystone
No defined test for determining if keystone is ready.
keystone is ready.
Error: Cannot start container fuel-core-5.1.1-keystone: (exit status 1)
2014/11/10 12:25:02 Error: failed to start one or more containers
Attaching to container fuel-core-5.1.1-keystone...
2014/11/10 12:25:02 You cannot attach to a stopped container, start it first

- - - - - - - - - -

[Matthew Mosesohn (raytrac3r)]

It looks like keystone is missing pbr.version in requirements:
Notice: /Stage[main]/Keystone/Exec[keystone-manage db_sync]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Keystone/Exec[keystone-manage db_sync]/returns: File "/usr/bin/keystone-manage", line 37, in <module>
Notice: /Stage[main]/Keystone/Exec[keystone-manage db_sync]/returns: from keystone import cli
Notice: /Stage[main]/Keystone/Exec[keystone-manage db_sync]/returns: File "/usr/lib/python2.6/site-packages/keystone/cli.py", line 20, in <module>
Notice: /Stage[main]/Keystone/Exec[keystone-manage db_sync]/returns: import pbr.version
Notice: /Stage[main]/Keystone/Exec[keystone-manage db_sync]/returns: ImportError: No module named pbr.version
Error: keystone-manage db_sync returned 1 instead of one of [0]

Passing to mos-keystone to fix spec for python-pbr dependency

[Matthew Mosesohn (raytrac3r)]

This patch is related for Juno: https://review.fuel-infra.org/gitweb?p=openstack-build/keystone-build.git;a=commit;h=35893a3eb0b2e4d569d050f4ac43b5369122c512

[Matthew Mosesohn (raytrac3r)]

I will commit a related patch that fails keystone check correctly when obtaining a token fails.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/133474

[OSCI Robot (oscirobot)]

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1.mira1.git.ef943bf.c4c9c06

Changeset: https://review.fuel-infra.org/693
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: patchset-created

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.1.mira1.git.ef943bf.c4c9c06.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.1.mira1.git.ef943bf.c4c9c06.noarch.rpm
python-keystone-2014.1.3-fuel5.1.1.mira1.git.ef943bf.c4c9c06.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-693/centos

[OSCI Robot (oscirobot)]

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1.mira1.git.ef943bf.a9b7e24

Changeset: https://review.fuel-infra.org/693
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: patchset-created

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.1.mira1.git.ef943bf.a9b7e24.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.1.mira1.git.ef943bf.a9b7e24.noarch.rpm
python-keystone-2014.1.3-fuel5.1.1.mira1.git.ef943bf.a9b7e24.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-693/centos

[OSCI Robot (oscirobot)]

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1.mira1.git.ef943bf.c4c9c06

Changeset: https://review.fuel-infra.org/693
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: patchset-created

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.1.mira1.git.ef943bf.a9b7e24.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.1.mira1.git.ef943bf.a9b7e24.noarch.rpm
python-keystone-2014.1.3-fuel5.1.1.mira1.git.ef943bf.a9b7e24.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-693/centos

[OSCI Robot (oscirobot)]

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1~mira1+git.ef943bf.a9b7e24

Changeset: https://review.fuel-infra.org/693
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: patchset-created

Files placed on repository:
keystone-doc_2014.1.3-fuel5.1.1~mira1+git.ef943bf.a9b7e24_all.deb
keystone_2014.1.3-fuel5.1.1~mira1+git.ef943bf.a9b7e24_all.deb
python-keystone_2014.1.3-fuel5.1.1~mira1+git.ef943bf.a9b7e24_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-693/ubuntu

[OSCI Robot (oscirobot)]

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1.mira1.git.ef943bf.2ca6836

Changeset: https://review.fuel-infra.org/693
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: patchset-created

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.1.mira1.git.ef943bf.2ca6836.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.1.mira1.git.ef943bf.2ca6836.noarch.rpm
python-keystone-2014.1.3-fuel5.1.1.mira1.git.ef943bf.2ca6836.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable-693/centos

[OSCI Robot (oscirobot)]

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.1, package release == fuel5.0.3.mira9.git.48c4000.12ce61a

Changeset: https://review.fuel-infra.org/694
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.0.3/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: patchset-created

Files placed on repository:
openstack-keystone-2014.1.1-fuel5.0.3.mira9.git.48c4000.12ce61a.noarch.rpm
openstack-keystone-doc-2014.1.1-fuel5.0.3.mira9.git.48c4000.12ce61a.noarch.rpm
python-keystone-2014.1.1-fuel5.0.3.mira9.git.48c4000.12ce61a.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.3-stable-694/centos

[OSCI Robot (oscirobot)]

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1~mira1+git.ef943bf.2ca6836

Changeset: https://review.fuel-infra.org/693
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: patchset-created

Files placed on repository:
keystone-doc_2014.1.3-fuel5.1.1~mira1+git.ef943bf.2ca6836_all.deb
keystone_2014.1.3-fuel5.1.1~mira1+git.ef943bf.2ca6836_all.deb
python-keystone_2014.1.3-fuel5.1.1~mira1+git.ef943bf.2ca6836_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable-693/ubuntu

[OSCI Robot (oscirobot)]

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.1, package release == fuel5.0.3~mira8+git.48c4000.12ce61a

Changeset: https://review.fuel-infra.org/694
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.0.3/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: patchset-created

Files placed on repository:
keystone-doc_2014.1.1-fuel5.0.3~mira8+git.48c4000.12ce61a_all.deb
keystone_2014.1.1-fuel5.0.3~mira8+git.48c4000.12ce61a_all.deb
python-keystone_2014.1.1-fuel5.0.3~mira8+git.48c4000.12ce61a_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.3-stable-694/ubuntu

[OSCI Robot (oscirobot)]

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1.mira1

Changeset: https://review.fuel-infra.org/693
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: change-merged

Files placed on repository:
openstack-keystone-2014.1.3-fuel5.1.1.mira1.noarch.rpm
openstack-keystone-doc-2014.1.3-fuel5.1.1.mira1.noarch.rpm
python-keystone-2014.1.3-fuel5.1.1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.1-stable/centos

[OSCI Robot (oscirobot)]

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.3, package release == fuel5.1.1~mira1

Changeset: https://review.fuel-infra.org/693
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.1.1/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: change-merged

Files placed on repository:
keystone-doc_2014.1.3-fuel5.1.1~mira1_all.deb
keystone_2014.1.3-fuel5.1.1~mira1_all.deb
python-keystone_2014.1.3-fuel5.1.1~mira1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.1-stable/ubuntu

[OSCI Robot (oscirobot)]

RPM package keystone has been built for project openstack/keystone
Package version == 2014.1.1, package release == fuel5.0.3.mira9

Changeset: https://review.fuel-infra.org/694
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.0.3/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: change-merged

Files placed on repository:
openstack-keystone-2014.1.1-fuel5.0.3.mira9.noarch.rpm
openstack-keystone-doc-2014.1.1-fuel5.0.3.mira9.noarch.rpm
python-keystone-2014.1.1-fuel5.0.3.mira9.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0.3-stable/centos

[OSCI Robot (oscirobot)]

DEB package keystone has been built for project openstack/keystone
Package version == 2014.1.1, package release == fuel5.0.3~mira8

Changeset: https://review.fuel-infra.org/694
project: openstack-build/keystone-build
branch: openstack-ci/fuel-5.0.3/2014.1.1
author: Alexander Makarov
committer: Alexander Makarov
subject: Add RPM reqirement for pbr
status: change-merged

Files placed on repository:
keystone-doc_2014.1.1-fuel5.0.3~mira8_all.deb
keystone_2014.1.1-fuel5.0.3~mira8_all.deb
python-keystone_2014.1.1-fuel5.0.3~mira8_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0.3-stable/ubuntu

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/133474
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5b48790720b859ed8b49f4892e6b9fc3acc27eea
Submitter: Jenkins
Branch: master

commit 5b48790720b859ed8b49f4892e6b9fc3acc27eea
Author: Matthew Mosesohn <email address hidden>
Date: Mon Nov 10 16:35:32 2014 +0400

    Add check for keystone container

    keystone token-get is now run from within
    the keystone container to verify that
    keystone is correctly configured.

    Change-Id: I3657059951e87d505647b6c3991dc37f1d76975d
    Related-Bug: #1391145
    Closes-Bug: #1390137

[Andrey Sledzinskiy (asledzinskiy)]

verified on
{

    "build_id": "2014-11-18_21-51-41",
    "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
    "build_number": "22",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1",
    "production": "docker",
    "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9",
    "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1.1",
    "release_versions": {
        "2014.1.3-5.1.1": {
            "VERSION": {
                "build_id": "2014-11-18_21-51-41",
                "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346",
                "build_number": "22",
                "api": "1.0",
                "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1",
                "production": "docker",
                "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9",
                "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "5.1.1",
                "fuellib_sha": "b928ba300daedfc56b906b7c208641cb27698072"
            }
        }
    },
    "fuellib_sha": "b928ba300daedfc56b906b7c208641cb27698072"

}