[10.0] bvt-2 Bootstrap node deployment has failed because of auth issue

[Kairat Kushaev]

Links to Jenkins jobs:
https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.smoke_neutron/576/
https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.bvt_2/580/

Scenario:
Create environment and set up master node
Snapshot: empty

Waiting for admin node to start up
Proceed with installation
Switching NTPD on the Fuel admin node to use 10.109.10.1 as the time source.
Replace default Ubuntu mirror URL for bootstrap image in Fuel settings
Waiting while bootstrapping is in progress
Running bootstrap (timeout: 3600.0)
Initiate Nailgun client with url http://10.109.10.2:8000
Completed fuel node deployment
...
Expected result:
fuel node deployment has been completed.
Actual result:
fuel node deployment has failed with the following error:
2016-08-22 11:31:39,661 - ERROR decorators.py:123 -- Traceback (most recent call last):
  File "/home/jenkins/workspace/10.0.main.ubuntu.smoke_neutron/fuelweb_test/helpers/decorators.py", line 117, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0.main.ubuntu.smoke_neutron/fuelweb_test/tests/base_test_case.py", line 362, in setup_master
    self.env.setup_environment()
  File "/home/jenkins/workspace/10.0.main.ubuntu.smoke_neutron/fuelweb_test/models/environment.py", line 441, in setup_environment
    self.wait_bootstrap()
  File "/home/jenkins/workspace/10.0.main.ubuntu.smoke_neutron/fuelweb_test/models/environment.py", line 557, in wait_bootstrap
    raise Exception('Fuel node deployment failed.')
Exception: Fuel node deployment failed.

Environment:
Ubuntu Xenial, MOS 10.0, Job #580,
Neutron VXLAN. See more details in failed jobs.

Additional information:
The admin bootstrap node shows the following:
The were failures while running task: keystone with exit code: 6
ERROR: Fuel node deployment FAILED! Check /var/log/puppet/bootstrap_admin_node.log for details.

If I look into it I can find a lot of messages like this:
Debug: Executing '/bin/openstack project list --quiet --format csv --long'
Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: _validate_auth() takes exactly 3 arguments (4 given) (tried 48, for a total of 170 seconds)

After that you can see that _validate_auth for openstack client requires 3 arguments:
https://github.com/openstack/python-openstackclient/blob/34f5515a329d7bdd9b945e256729cd4c97595201/openstackclient/common/client_config.py#L155
but _validate_auth for os-client-config requires 4 arguments:
https://github.com/openstack/os-client-config/blob/9d757b3a9a89bdd63f56ce171b7c878ded9a4cd8/os_client_config/config.py#L909

I am wondering how this was used in Fuel.

At first glance QA fixes that has been merged today seem not related to the bug.

[Alex Schultz (alex-schultz)]

openstack client was updated from previous runs and is now breaking things.

From BVT#576:
python-openstackclient-2.6.0-1.el7~mos101.noarch
python-os-client-config-1.19.1-1.el7~mos2.noarch

From BVT#580:
python-openstackclient-2.6.0-2.el7~mos214.noarch
python-os-client-config-1.19.1-1.el7~mos2.noarch

[Igor Yozhikov (iyozhikov)]

Fix was proposed to upstream by - https://github.com/openstack/python-openstackclient/commit/684412ca4cc0abad2c2a800d8247d12992b994e5

We've got this code 6 hr ago from now.
Working on packages update.

[Igor Yozhikov (iyozhikov)]

It seams like version of openstackclient >= 3.0.0 has a lot of changes which are prevents from expected using of token based authentication/authorization.

Using python-openstackclient >= 3.0.0 lead to this result(token auth is broken):
# pip freeze | grep openstackclient
python-openstackclient==3.0.1
# openstack --os-token=ADMIN --os-url=http://127.0.0.1:35357/v3 user list
Missing parameter(s):
Set a username with --os-username, OS_USERNAME, or auth.username
Set an authentication URL, with --os-auth-url, OS_AUTH_URL or auth.auth_url

After switching back to v2.6.0 token auth works again!
# pip freeze | grep openstackclient
python-openstackclient==2.6.0
# openstack --os-token=ADMIN --os-url=http://127.0.0.1:35357/v3 user list
+----------------------------------+-------+
| ID | Name |
+----------------------------------+-------+
| e1e325738ed849dfa78fc93257207c57 | admin |
+----------------------------------+-------+

Found and commented similar issue - https://bugs.launchpad.net/python-openstackclient/+bug/1615110

Also requested help on openstack-keystone irc channel.

Since that, while we are investigating how to solve appeared issue with token based authorization, I'm suggesting to revert v2.6.0 from package repositories snapshots:

DEB - http://mirror.fuel-infra.org/mos-repos/xenial/snapshots/master-2016-08-22-082021/pool/main/p/python-openstackclient/
* http://mirror.fuel-infra.org/mos-repos/xenial/snapshots/master-2016-08-22-082021/pool/main/p/python-openstackclient/python-openstackclient_2.6.0-1~u16.04%2bmos101_all.deb
* http://mirror.fuel-infra.org/mos-repos/xenial/snapshots/master-2016-08-22-082021/pool/main/p/python-openstackclient/python-openstackclient-doc_2.6.0-1~u16.04%2bmos101_all.deb
* http://mirror.fuel-infra.org/mos-repos/xenial/snapshots/master-2016-08-22-082021/pool/main/p/python-openstackclient/python3-openstackclient_2.6.0-1~u16.04%2bmos101_all.deb

RPM - http://mirror.fuel-infra.org/mos-repos/centos/mos-master-centos7/snapshots/os-2016-08-19-150024/x86_64/Packages/
* http://mirror.fuel-infra.org/mos-repos/centos/mos-master-centos7/snapshots/os-2016-08-19-150024/x86_64/Packages/python-openstackclient-2.6.0-1.el7~mos101.noarch.rpm
* http://mirror.fuel-infra.org/mos-repos/centos/mos-master-centos7/snapshots/os-2016-08-19-150024/x86_64/Packages/python-openstackclient-doc-2.6.0-1.el7~mos101.noarch.rpm

[Kairat Kushaev (kkushaev)]

The issue reproduced again so I'll raise bug priority because it is going to block the gate.

[Igor Yozhikov (iyozhikov)]

After discussion on IRC #openstack-keystone was found workaround, it was verified and failed new bug - https://bugs.launchpad.net/python-openstackclient/+bug/1615988.
We need to get back to using v2.6.0 until upstream is broken.

[Dmitry Burmistrov (dburmistrov)]

All the packages have been reverted at 2.6.0-mos101 revision. Waiting to update it on the public mirrors

[Igor Yozhikov (iyozhikov)]

Packages with v2.6.0 and mos101 suffix appeared in mirrors.

[Alexandr Kostrikov (akostrikov-mirantis)]

It is failing again: https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.bvt_2/

Error: Could not prefetch keystone_tenant provider 'openstack': Execution of '/bin/openstack project list --quiet --format csv --long' returned 1: __init__() got an unexpected keyword argument 'project_domain_id' (tried 46, for a total of 170 seconds)

https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.bvt_2/891/

[Ivan Udovichenko (iudovichenko)]

Packages are still being sourced from master repository [1].

More information from deployed master node [10.0.main.ubuntu.bvt_2.891_admin]:
http://paste.openstack.org/show/589035/

[1] http://perestroika-repo-tst.infra.mirantis.net/mos-repos/centos/mos-master-centos7/os/x86_64/

[Ivan (iremizov)]

already fixed with https://product-ci.infra.mirantis.net/job/10.0.all/jobConfigHistory/showDiffFiles?timestamp1=2016-11-11_10-45-06&timestamp2=2016-11-11_13-44-58

Fuel-main should be patched to build newton packages from newton branches.

[Roman Vyalov (r0mikiam)]

https://github.com/openstack/fuel-main/blob/stable/newton/config.mk

FUELLIB_COMMIT?=master
NAILGUN_COMMIT?=master
PYTHON_FUELCLIENT_COMMIT?=master
FUEL_AGENT_COMMIT?=master
FUEL_NAILGUN_AGENT_COMMIT?=master
ASTUTE_COMMIT?=master
OSTF_COMMIT?=master
FUEL_MIRROR_COMMIT?=master
FUELMENU_COMMIT?=master
SHOTGUN_COMMIT?=master
NETWORKCHECKER_COMMIT?=master
FUELUPGRADE_COMMIT?=master
FUEL_UI_COMMIT?=master

please change the fuel version from amster to stable/newton in upstream

[Roman Vyalov (r0mikiam)]

Related change request https://review.openstack.org/#/c/397130/

[Dmitry Burmistrov (dburmistrov)]

merged