Master node setup fails due to keystone

[Roman Podoliaka]

http://jenkins-product.srt.mirantis.net:8080/view/Staging%20Mirrors/job/6.0.1.staging.ubuntu.bvt_2/65/console

[root@nailgun log]# grep ERROR ./puppet/bootstrap_admin_node.log
Notice: /Stage[main]/Docker/Exec[build docker containers]/returns: ERROR: keystone failed to start.
ERROR: keystone failed to start.
ERROR: Fuel node deployment FAILED! Check /var/log/puppet/bootstrap_admin_node.log for details

stdout/stderr of keystone container: http://paste.openstack.org/show/184813/

[Matthew Mosesohn (raytrac3r)]

Error: /Stage[main]/Main/Keystone_tenant[admin]: Could not evaluate: Expected 3 columns for tenant row, found 0. Line +----------------------------------+-------+---------+

Full log in pastebin: http://paste.openstack.org/show/WNjUnzNs2uCDz1C5KU35/

Looks like new keystone client package broke BVT.

[Matthew Mosesohn (raytrac3r)]

/usr/lib/python2.6/site-packages/keystoneclient/session.py:19: DeprecationWarning: The oslo namespace package is deprecated. Please use oslo_config instead.
  from oslo.config import cfg
^ This error causes all parsing to break.

[Matthew Mosesohn (raytrac3r)]

Update: This affects all client libraries:
python-ceilometerclient-1.0.11-fuel6.0.mira14.noarch
python-cinderclient-1.1.1-fuel6.0.mira10.noarch
python-heatclient-0.2.12-fuel6.0.mira9.noarch
python-keystoneclient-0.11.1-fuel6.0.mira12.noarch
python-muranoclient-0.5.5-fuel6.0.mira23.noarch
python-neutronclient-2.3.9-fuel6.0.mira21.noarch
python-novaclient-2.20.0-fuel6.0.mira16.noarch
python-saharaclient-0.7.6-fuel6.0.1.mira1.noarch

[OSCI Robot (oscirobot)]

RPM package oslo.config has been built for project openstack/oslo.config
Package version == 1.6.0, package release == fuel6.0.1.mira1.git.8ed8269.7257205

Changeset: https://review.fuel-infra.org/4189
project: openstack/oslo.config
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo-config-1.6.0-fuel6.0.1.mira1.git.8ed8269.7257205.noarch.rpm
python-oslo-config-doc-1.6.0-fuel6.0.1.mira1.git.8ed8269.7257205.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4189/centos

[OSCI Robot (oscirobot)]

DEB package oslo.config has been built for project openstack/oslo.config
Package version == 1.6.0, package release == fuel6.0.1~mira1+git.8ed8269.7257205

Changeset: https://review.fuel-infra.org/4189
project: openstack/oslo.config
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo.config_1.6.0-fuel6.0.1~mira1+git.8ed8269.7257205_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4189/ubuntu

[OSCI Robot (oscirobot)]

RPM package oslo.config has been built for project openstack/oslo.config
Package version == 1.6.0, package release == fuel6.0.1.mira1.git.3c26e5e.7257205

Changeset: https://review.fuel-infra.org/4189
project: openstack/oslo.config
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo-config-1.6.0-fuel6.0.1.mira1.git.3c26e5e.7257205.noarch.rpm
python-oslo-config-doc-1.6.0-fuel6.0.1.mira1.git.3c26e5e.7257205.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4189/centos

[OSCI Robot (oscirobot)]

DEB package oslo.config has been built for project openstack/oslo.config
Package version == 1.6.0, package release == fuel6.0.1~mira1+git.3c26e5e.7257205

Changeset: https://review.fuel-infra.org/4189
project: openstack/oslo.config
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo.config_1.6.0-fuel6.0.1~mira1+git.3c26e5e.7257205_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4189/ubuntu

[OSCI Robot (oscirobot)]

RPM package oslo.i18n has been built for project openstack/oslo.i18n
Package version == 1.3.1, package release == fuel6.0.1.mira1.git.a2428f6.82b70ea

Changeset: https://review.fuel-infra.org/4191
project: openstack/oslo.i18n
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo-i18n-1.3.1-fuel6.0.1.mira1.git.a2428f6.82b70ea.noarch.rpm
python-oslo-i18n-doc-1.3.1-fuel6.0.1.mira1.git.a2428f6.82b70ea.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4191/centos

[OSCI Robot (oscirobot)]

DEB package oslo.i18n has been built for project openstack/oslo.i18n
Package version == 1.3.1, package release == fuel6.0.1~mira1+git.a2428f6.82b70ea

Changeset: https://review.fuel-infra.org/4191
project: openstack/oslo.i18n
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo.i18n-doc_1.3.1-fuel6.0.1~mira1+git.a2428f6.82b70ea_all.deb
python-oslo.i18n_1.3.1-fuel6.0.1~mira1+git.a2428f6.82b70ea_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4191/ubuntu

[OSCI Robot (oscirobot)]

RPM package oslo.i18n has been built for project openstack/oslo.i18n
Package version == 1.3.1, package release == fuel6.0.1.mira1.git.2d1c30d.82b70ea

Changeset: https://review.fuel-infra.org/4191
project: openstack/oslo.i18n
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo-i18n-1.3.1-fuel6.0.1.mira1.git.2d1c30d.82b70ea.noarch.rpm
python-oslo-i18n-doc-1.3.1-fuel6.0.1.mira1.git.2d1c30d.82b70ea.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4191/centos

[OSCI Robot (oscirobot)]

DEB package oslo.i18n has been built for project openstack/oslo.i18n
Package version == 1.3.1, package release == fuel6.0.1~mira1+git.2d1c30d.82b70ea

Changeset: https://review.fuel-infra.org/4191
project: openstack/oslo.i18n
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo.i18n-doc_1.3.1-fuel6.0.1~mira1+git.2d1c30d.82b70ea_all.deb
python-oslo.i18n_1.3.1-fuel6.0.1~mira1+git.2d1c30d.82b70ea_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4191/ubuntu

[OSCI Robot (oscirobot)]

RPM package oslo.utils has been built for project openstack/oslo.utils
Package version == 1.2.1, package release == fuel6.0.1.mira1.git.90de4e1.9919191

Changeset: https://review.fuel-infra.org/4194
project: openstack/oslo.utils
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo-utils-1.2.1-fuel6.0.1.mira1.git.90de4e1.9919191.noarch.rpm
python-oslo-utils-doc-1.2.1-fuel6.0.1.mira1.git.90de4e1.9919191.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4194/centos

[OSCI Robot (oscirobot)]

DEB package oslo.utils has been built for project openstack/oslo.utils
Package version == 1.2.1, package release == fuel6.0.1~mira1+git.90de4e1.9919191

Changeset: https://review.fuel-infra.org/4194
project: openstack/oslo.utils
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo.utils-doc_1.2.1-fuel6.0.1~mira1+git.90de4e1.9919191_all.deb
python-oslo.utils_1.2.1-fuel6.0.1~mira1+git.90de4e1.9919191_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4194/ubuntu

[OSCI Robot (oscirobot)]

RPM package oslo.config has been built for project openstack/oslo.config
Package version == 1.6.0, package release == fuel6.0.1.mira1.git.031fe53.7257205

Changeset: https://review.fuel-infra.org/4189
project: openstack/oslo.config
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning abour the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo-config-1.6.0-fuel6.0.1.mira1.git.031fe53.7257205.noarch.rpm
python-oslo-config-doc-1.6.0-fuel6.0.1.mira1.git.031fe53.7257205.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4189/centos

[OSCI Robot (oscirobot)]

DEB package oslo.config has been built for project openstack/oslo.config
Package version == 1.6.0, package release == fuel6.0.1~mira1+git.031fe53.7257205

Changeset: https://review.fuel-infra.org/4189
project: openstack/oslo.config
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning abour the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo.config_1.6.0-fuel6.0.1~mira1+git.031fe53.7257205_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4189/ubuntu

[OSCI Robot (oscirobot)]

RPM package oslo.concurrency has been built for project openstack/oslo.concurrency
Package version == 1.5.0, package release == fuel6.0.1.mira1.git.8b2d9b0.27c90ec

Changeset: https://review.fuel-infra.org/4200
project: openstack/oslo.concurrency
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning abour the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo-concurrency-1.5.0-fuel6.0.1.mira1.git.8b2d9b0.27c90ec.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4200/centos

[OSCI Robot (oscirobot)]

DEB package oslo.concurrency has been built for project openstack/oslo.concurrency
Package version == 1.5.0, package release == fuel6.0.1~mira1+git.8b2d9b0.27c90ec

Changeset: https://review.fuel-infra.org/4200
project: openstack/oslo.concurrency
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning abour the oslo.* namespace
status: patchset-created

Files placed on repository:
python-oslo.concurrency-doc_1.5.0-fuel6.0.1~mira1+git.8b2d9b0.27c90ec_all.deb
python-oslo.concurrency_1.5.0-fuel6.0.1~mira1+git.8b2d9b0.27c90ec_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4200/ubuntu

[Viktor Serhieiev (vsergeyev)]

ok, I can agree, that developers warning messages should be hidden from deployers, so I'm going to remove these warnings from oslo.libraries, which were updated recently.
Anyway, I still have a doubts, that this is the proper way (and only place) to fix this bug, because if puppet manifests can be broken by warning messages, it something wrong with puppet manifests, and they should be fixed as well.
Also if we use python 2.7 or higher, we can set env variable PYTHONWARNINGS to "ignore" to hide these warnings. As for me, this solution looks more flexible and allows us to leave source code unchanged. But, afaik, we still use python 2.6 on CentOS...

[OSCI Robot (oscirobot)]

RPM package oslo.concurrency has been built for project openstack/oslo.concurrency
Package version == 1.5.0, package release == fuel6.0.1.mira1

Changeset: https://review.fuel-infra.org/4200
project: openstack/oslo.concurrency
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning abour the oslo.* namespace
status: change-merged

Files placed on repository:
python-oslo-concurrency-1.5.0-fuel6.0.1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

[OSCI Robot (oscirobot)]

DEB package oslo.concurrency has been built for project openstack/oslo.concurrency
Package version == 1.5.0, package release == fuel6.0.1~mira1

Changeset: https://review.fuel-infra.org/4200
project: openstack/oslo.concurrency
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning abour the oslo.* namespace
status: change-merged

Files placed on repository:
python-oslo.concurrency-doc_1.5.0-fuel6.0.1~mira1_all.deb
python-oslo.concurrency_1.5.0-fuel6.0.1~mira1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[OSCI Robot (oscirobot)]

RPM package oslo.i18n has been built for project openstack/oslo.i18n
Package version == 1.3.1, package release == fuel6.0.1.mira1

Changeset: https://review.fuel-infra.org/4191
project: openstack/oslo.i18n
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: change-merged

Files placed on repository:
python-oslo-i18n-1.3.1-fuel6.0.1.mira1.noarch.rpm
python-oslo-i18n-doc-1.3.1-fuel6.0.1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

[OSCI Robot (oscirobot)]

DEB package oslo.i18n has been built for project openstack/oslo.i18n
Package version == 1.3.1, package release == fuel6.0.1~mira1

Changeset: https://review.fuel-infra.org/4191
project: openstack/oslo.i18n
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: change-merged

Files placed on repository:
python-oslo.i18n-doc_1.3.1-fuel6.0.1~mira1_all.deb
python-oslo.i18n_1.3.1-fuel6.0.1~mira1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[OSCI Robot (oscirobot)]

DEB package oslo.utils has been built for project openstack/oslo.utils
Package version == 1.2.1, package release == fuel6.0.1~mira1

Changeset: https://review.fuel-infra.org/4194
project: openstack/oslo.utils
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: change-merged

Files placed on repository:
python-oslo.utils-doc_1.2.1-fuel6.0.1~mira1_all.deb
python-oslo.utils_1.2.1-fuel6.0.1~mira1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[OSCI Robot (oscirobot)]

RPM package oslo.config has been built for project openstack/oslo.config
Package version == 1.6.0, package release == fuel6.0.1.mira1

Changeset: https://review.fuel-infra.org/4189
project: openstack/oslo.config
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning abour the oslo.* namespace
status: change-merged

Files placed on repository:
python-oslo-config-1.6.0-fuel6.0.1.mira1.noarch.rpm
python-oslo-config-doc-1.6.0-fuel6.0.1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

[OSCI Robot (oscirobot)]

RPM package oslo.utils has been built for project openstack/oslo.utils
Package version == 1.2.1, package release == fuel6.0.1.mira1

Changeset: https://review.fuel-infra.org/4194
project: openstack/oslo.utils
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning about the oslo.* namespace
status: change-merged

Files placed on repository:
python-oslo-utils-1.2.1-fuel6.0.1.mira1.noarch.rpm
python-oslo-utils-doc-1.2.1-fuel6.0.1.mira1.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

[OSCI Robot (oscirobot)]

DEB package oslo.config has been built for project openstack/oslo.config
Package version == 1.6.0, package release == fuel6.0.1~mira1

Changeset: https://review.fuel-infra.org/4189
project: openstack/oslo.config
branch: openstack-ci/fuel-6.0.1/2014.2
author: Victor Sergeyev
committer: Victor Sergeyev
subject: Remove warning abour the oslo.* namespace
status: change-merged

Files placed on repository:
python-oslo.config_1.6.0-fuel6.0.1~mira1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[Viktor Serhieiev (vsergeyev)]

ok, after the fix on oslo side staging tests doesn't show this bug anymore. Anyway, I insist, that manifests must be fixed to avoid of the similar bug in the future

[Matthew Mosesohn (raytrac3r)]

Re-removing Fuel. The source of this issue is on MOS side, not Fuel

[Viktor Serhieiev (vsergeyev)]

ok, Matthew, who should fix puppet manifests?

[Dmitry Savenkov (dsavenkov)]

Hi, Matthew,

The thing is that the bug has already been fixed on the MOS side, so it doesn't really matter what the source of the bug was. However, in order to prevent such bugs from occurring in the future, the manifests MUST be fixed on the Fuel side, which is the responsibility of FUEL-LIB (According to Timur Nurlygayanov).

This bug has a long history and should be fixed by the Fuel-lib now, so I am reassigning it back to the aforementioned team.

If you/the team cannot handle the bug now, I will close the bug, since it has been fixed in MOS. But this is not the fixing of the root cause (we were just dealing with symptoms).

Let's discuss this separately if you feel like it.

Thank you for your understanding.

Regards,
Dmitry,

[Matthew Mosesohn (raytrac3r)]

We discussed this previously outside of this bug. Any deprecation warnings will look negatively if our users see them. We shouldn't release any client packages that display deprecation warnings because these messages are for developers, not end users. Yes, it has the side effect of breaking fragile puppet modules, but the impact on user experience is just as worse. We should treat all these deprecation issues as critical and MOS side is responsible.

[Kyrylo Romanenko (kromanenko)]

Build http://jenkins-product.srt.mirantis.net:8080/view/6.0.1/job/6.0.1.staging.ubuntu.bvt_2/66/
passed successfully after changes on March 3.