devstack fails while running Ironic grenade job: init__() got an unexpected keyword argument 'token'

[John L. Villalovos]

When running the Ironic grenade job see the following error in the devstacklog.txt

The error occurs twice in the stable/liberty portion of the devstack:

__init__() got an unexpected keyword argument 'token'

This causes errors later in the process of the Ironic grenade.

<snip>
2016-02-18 05:42:29.999 | +++ grep ' id '
2016-02-18 05:42:30.000 | +++ get_field 2
2016-02-18 05:42:30.001 | +++ local data field
2016-02-18 05:42:30.001 | +++ read data
2016-02-18 05:42:30.002 | ++++ basename /opt/stack/old/devstack/files/ir-deploy-pxe_ssh.kernel
2016-02-18 05:42:30.005 | +++ openstack --os-token 5f1f7e15106f47cfae0f90792817a915 --os-url http://127.0.0.1:9292 image create ir-deploy-pxe_ssh.kernel --public --disk-format=aki --container-format=aki
2016-02-18 05:42:30.551 | __init__() got an unexpected keyword argument 'token'
2016-02-18 05:42:30.578 | ++ IRONIC_DEPLOY_KERNEL_ID=
2016-02-18 05:42:30.580 | +++ grep ' id '
2016-02-18 05:42:30.580 | ++++ basename /opt/stack/old/devstack/files/ir-deploy-pxe_ssh.initramfs
2016-02-18 05:42:30.581 | +++ get_field 2
2016-02-18 05:42:30.581 | +++ local data field
2016-02-18 05:42:30.581 | +++ read data
2016-02-18 05:42:30.583 | +++ openstack --os-token 5f1f7e15106f47cfae0f90792817a915 --os-url http://127.0.0.1:9292 image create ir-deploy-pxe_ssh.initramfs --public --disk-format=ari --container-format=ari
2016-02-18 05:42:31.110 | __init__() got an unexpected keyword argument 'token'
2016-02-18 05:42:31.132 | ++ IRONIC_DEPLOY_RAMDISK_ID=
<snip>

http://logs.openstack.org/24/269924/3/experimental/gate-grenade-dsvm-ironic/0b213b6/logs/old/devstacklog.txt.gz#_2016-02-18_05_42_31_110

[John L. Villalovos (happycamp)]

My dummy test patch to run the Ironic grenade job:
https://review.openstack.org/#/c/269924/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/284442

[Dean Troyer (dtroyer)]

This fails in the 'old' (stable/liberty) ironic code that is still in the devstack repo. The admin auth env vars are set up by this point so a hacky work-around would be to just remove --os-token and --os-url, but we should sort out what actually broke here.

[John L. Villalovos (happycamp)]

Updated the script to do a '--debug' and here is what I got

2016-02-25 00:24:09.717 | +++ openstack --debug --os-token 69b0eb5a9c0b4c499dec1003303bdf42 --os-url http://127.0.0.1:9292 image create ir-deploy-pxe_ssh.kernel --public --disk-format=aki --container-format=aki
2016-02-25 00:24:10.329 | START with options: ['--debug', '--os-token', '69b0eb5a9c0b4c499dec1003303bdf42', '--os-url', 'http://127.0.0.1:9292', 'image', 'create', 'ir-deploy-pxe_ssh.kernel', '--public', '--disk-format=aki', '--container-format=aki']
2016-02-25 00:24:10.331 | options: Namespace(access_token_endpoint='', auth_type='v2password', auth_url='http://127.0.0.1:35357/v2.0', cacert='', client_id='', client_secret='***', cloud='', debug=True, default_domain='default', deferred_help=False, domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, interface='', log_file=None, os_baremetal_api_version='1.6', os_compute_api_version='', os_data_processing_api_version='1.1', os_identity_api_version='', os_image_api_version='', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_volume_api_version='', password='***', project_domain_id='', project_domain_name='', project_id='', project_name='admin', protocol='', region_name='RegionOne', scope='', service_provider_endpoint='', timing=False, token='***', trust_id='', url='http://127.0.0.1:9292', user_domain_id='', user_domain_name='', user_id='', username='admin', verbose_level=3, verify=None)
2016-02-25 00:24:10.343 | Deferring keystone exception: __init__() got an unexpected keyword argument 'token'
2016-02-25 00:24:10.344 | defaults: {u'auth_type': 'token_endpoint', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', 'cacert': None, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', u'interface': None, u'network_api_version': u'2', u'image_format': u'qcow2', u'key_manager_api_version': u'v1', u'metering_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'disable_vendor_agent': {}}
2016-02-25 00:24:10.344 | cloud cfg: {'auth_type': 'v2password', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'data_processing_api_version': '1.1', u'network_api_version': u'2', u'image_format': u'qcow2', u'image_api_version': u'2', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', 'verbose_level': 3, 'region_name': 'RegionOne', 'api_timeout': None, u'baremetal_api_version': '1.6', 'auth': {'username': 'admin', 'tenant_name': 'admin', 'token': '***', 'password': '***', 'auth_url': 'http://127.0.0.1:35357/v2.0'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'cacert': None, u'key_manager_api_version': u'v1', u'metering_api_version': ...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/279270
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=e6b3c4217f27818acedb15c925641f9d1aa5ab34
Submitter: Jenkins
Branch: master

commit e6b3c4217f27818acedb15c925641f9d1aa5ab34
Author: John L. Villalovos <email address hidden>
Date: Thu Feb 11 13:24:53 2016 -0800

    Don't use token for glance & check for some unset vars

    There is a regression issue in stable/liberty where using a token for
    auth is failing for glance. We don't need to use a token as we already
    have our credentials. So stop using a token to upload to glance.

    We call various commands to set certain variables. Add some checks to
    make sure those variables are set and fail if they are not.

    Partial-Bug: #1549095
    Change-Id: Ic941d061edf3fe68c18b3eda198b1ed03f4a18fc

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/290858

[Steve Martinelli (stevemar)]

I marked bug 1555366 as a duplicate and am copying the contents of the description here for completeness:

tl;dr: please check if OS_TOKEN is set when someone types "openstack token issue" and print a nice error message

Longer version:

Playing with "openstack token issue", I tried to do:
export OS_TOKEN=SOME-VALUE

to avoid passing the token on the command line, which worked very well.

Though, as I was playing, I attempted to get another token, with the OS_TOKEN env var set (which I didn't even remember I did earlier), and then the result is surprising:

$ openstack token issue
__init__() got an unexpected keyword argument 'token'

While I do understand that it may be normal that it fails, the error message is kind of surprising and not explaining anything. So, it'd be nice if OSC was carefully checking for the input and printing a nicer error message, as this typically will happen to someone else, who will, just like me, not understand what's going on.

[Steve Martinelli (stevemar)]

so it sounds like for OSC the only task is to make the error message more helpful if OS_TOKEN is used with `openstack token issue`

[Steve Martinelli (stevemar)]

for here: https://github.com/openstack/python-openstackclient/blob/master/openstackclient/identity/v3/token.py#L175

we should check to see if there is a valid 'auth_ref':

  token = self.app.client_manager.auth_ref.service_catalog.get_token()

because in the case of using the ADMIN_TOKEN, it carries no authorization or authentication data, it's a bypass mechanism, so auth_ref isn't set

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/290858
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=49b97f733dc89502e1a57077cc8c7b12eea53f23
Submitter: Jenkins
Branch: stable/liberty

commit 49b97f733dc89502e1a57077cc8c7b12eea53f23
Author: John L. Villalovos <email address hidden>
Date: Wed Mar 9 13:09:38 2016 -0800

    Ironic: Don't use token for glance & check for some unset vars

    There is a regression issue in stable/liberty where using a token for
    auth is failing for glance. We don't need to use a token as we already
    have our credentials. So stop using a token to upload to glance.

    We call various commands to set certain variables. Add some checks to
    make sure those variables are set and fail if they are not.

    Partial-Bug: #1549095
    (cherry-picked from openstack/ironic commit e6b3c4217f27818acedb15c925641f9d1aa5ab34)
    Change-Id: I398cd61dd731c03346cf10e5b3102d8d809d3f87

[OpenStack Infra (hudson-openstack)]

Change abandoned by John L. Villalovos (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/284442
Reason: stable/mitaka is out so don't need this anymore.

[Stephen Finucane (stephenfinucane)]

Added python-keystoneclient as #1552615 appears to be a duplicate

[Stephen Finucane (stephenfinucane)]

As stevemar states above, you can cause this error in the openstack client by exporting OS_TOKEN. Even if this is valid (it works with cURL), the client breaks:

    $ export OS_TOKEN=xyz
    $ openstack server list
    __init__() got an unexpected keyword argument 'token'
    $ unset OS_TOKEN
    $ openstack server list
    +--------------------------------------+-------+--------+--------------------------------------------------------+
    | ID | Name | Status | Networks |
    +--------------------------------------+-------+--------+--------------------------------------------------------+
    | c3ee781d-bda8-4bfb-95e9-50d6e5b8820d | test1 | ACTIVE | private=10.0.0.3, fd47:d64b:f3e1:0:f816:3eff:fe41:a88a |
    +--------------------------------------+-------+--------+--------------------------------------------------------+

[Thomas Goirand (thomas-goirand)]

I had the issue as well, and I got the same problem too. IMO, one should never use "export OS_TOKEN=", but instead, something like this, which I use for bootstrapping keystone without leaking passwords in /proc:

OS_TOKEN=`openstack token issue -c id -f value` openstack server list

[lipeng (lipjiang)]

hit the error with openstack cloud-archive:mitaka,

thanks #13, unset OS_TOKEN fix the error

[Dave Walker (davewalker)]

It should be valid in most circumstances to use OS_TOKEN... I'm not sure if this is the same bug, or a different one.. but OS_AUTH_TYPE=token also fails weirdly..

$ env | grep ^OS
OS_USER_DOMAIN_NAME=TEST
OS_PROJECT_NAME=Test
OS_IDENTITY_API_VERSION=3
OS_AUTH_TYPE=token
OS_AUTH_URL=http://<REDACTED>:35357/v3
OS_TOKEN=eca36e2f30f84a8d82bbbf499d77ccb0
OS_USERNAME=walkerd

$ openstack image list
__init__() got an unexpected keyword argument 'user_domain_name'

$ unset OS_USER_DOMAIN_NAME

$ openstack image list
Expecting to find domain in project - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-3d65a287-2b40-47c2-a5c4-7eb4c43d8066)

(Ugh! Can't win.)

[Steve Martinelli (stevemar)]

As zigo said, don't export OS_TOKEN. OSC gets its options from KSA and os-client-config and the combination of OS_TOKEN and other options will produce funky results. The original issue in this post has long since been resolved.

[Dmitry Tantsur (divius)]

I think our grenade job runs just fine now, so it can be fixed. Please reopen if I'm wrong.

[Stanislav Kolenkin (skolenkin)]

I have the same problem in mitaka.
openstack --version
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils. This warning is caused by an out-of-date import in /usr/lib/python2.7/site-packages/saharaclient/osc/plugin.py
openstack 3.5.0

openstack service list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils. This warning is caused by an out-of-date import in /usr/lib/python2.7/site-packages/saharaclient/osc/plugin.py
argument of type 'OSC_Config' is not iterable
__init__() got an unexpected keyword argument 'token'
__init__() got an unexpected keyword argument 'token'