openstackclient plugins using private attributes

[wes hayutin]

openstack baremetal import --json instackenv.json fails

Still investigating whether or not this is an auth issue, but I wanted to provide a trace.

https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal-389/undercloud/home/stack/undercloud_post_install.log.gz

openstack --debug baremetal import --json instackenv.json
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
START with options: ['--debug', 'baremetal', 'import', '--json', 'instackenv.json']
options: Namespace(access_key='', access_secret='***', access_token='***', access_token_endpoint='', access_token_type='', auth_type='', auth_url='http://192.0.2.1:5000/v2.0', authorization_code='', cacert='', cert='', client_id='', client_secret='***', cloud='', consumer_key='', consumer_secret='***', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', identity_provider_url='', insecure=None, inspector_api_version='1', inspector_url=None, interface='', key='', log_file=None, murano_url='', old_profile=None, openid_scope='', os_application_catalog_api_version='1', os_baremetal_api_version='1.11', os_beta_command=False, os_compute_api_version='', os_data_processing_api_version='1.1', os_data_processing_url='', os_dns_api_version='2', os_identity_api_version='', os_image_api_version='1', os_key_manager_api_version='1', os_metrics_api_version='1', os_network_api_version='', os_object_api_version='', os_orchestration_api_version='1', os_project_id=None, os_project_name=None, os_queues_api_version='2', os_tripleoclient_api_version='1', os_volume_api_version='', os_workflow_api_version='2', passcode='', password='***', profile=None, project_domain_id='', project_domain_name='', project_id='', project_name='admin', protocol='', redirect_uri='', region_name='', roles='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='', user_id='', username='admin', verbose_level=3, verify=None)
Auth plugin password selected
auth_config_hook(): {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', u'orchestration_api_version': '1', u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', 'auth_url': 'http://192.0.2.1:5000/v2.0', u'network_api_version': u'2', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'username': 'admin', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.11', 'queues_api_version': '2', 'auth': {'project_name': 'admin'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', 'key': None, 'timing': False, 'password': '0b4835fba169e7eb72fb627ed37420dd4b0bf9e3', 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
defaults: {u'auth_type': 'password', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'api_timeout': None, u'baremetal_api_version': u'1', u'image_api_version': u'2', 'cacert': None, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', u'interface': None, u'network_api_version': u'2', u'image_format': u'qcow2', u'key_manager_api_version': u'v1', u'metering_api_version': u'2', 'verify': True, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'dns_api_version': u'2', u'object_store_api_version': u'1', u'disable_vendor_agent': {}}
cloud cfg: {'auth_type': 'password', 'beta_command': False, 'tripleoclient_api_version': '1', u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', 'metrics_api_version': '1', 'data_processing_api_version': '1.1', 'inspector_api_version': '1', u'network_api_version': u'2', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '1', 'verify': True, u'dns_api_version': '2', u'object_store_api_version': u'1', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': '1.11', 'queues_api_version': '2', 'auth': {'username': 'admin', 'project_name': 'admin', 'password': '***', 'auth_url': 'http://192.0.2.1:5000/v2.0'}, 'default_domain': 'default', u'container_api_version': u'1', u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': '1', 'timing': False, 'application_catalog_api_version': '1', 'cacert': None, u'key_manager_api_version': '1', u'metering_api_version': u'2', 'deferred_help': False, u'identity_api_version': u'2.0', 'workflow_api_version': '2', u'volume_api_version': u'2', 'cert': None, u'secgroup_source': u'neutron', 'debug': True, u'interface': None, u'disable_vendor_agent': {}}
compute API version 2, cmd group openstack.compute.v2
network API version 2, cmd group openstack.network.v2
image API version 1, cmd group openstack.image.v1
volume API version 2, cmd group openstack.volume.v2
identity API version 2.0, cmd group openstack.identity.v2
object_store API version 1, cmd group openstack.object_store.v1
orchestration API version 1, cmd group openstack.orchestration.v1
key_manager API version 1, cmd group openstack.key_manager.v1
dns API version 2, cmd group openstack.dns.v2
data_processing API version 1.1, cmd group openstack.data_processing.v1
workflow_engine API version 2, cmd group openstack.workflow_engine.v2
messaging API version 2, cmd group openstack.messaging.v2
neutronclient API version 2, cmd group openstack.neutronclient.v2
metric API version 1, cmd group openstack.metric.v1
application_catalog API version 1, cmd group openstack.application_catalog.v1
baremetal_introspection API version 1, cmd group openstack.baremetal_introspection.v1
baremetal API version 1.11, cmd group openstack.baremetal.v1
tripleoclient API version 1, cmd group openstack.tripleoclient.v1
command: baremetal import -> tripleoclient.v1.baremetal.ImportBaremetal
Using auth plugin: password
Using parameters {'username': 'admin', 'project_name': 'admin', 'password': '***', 'auth_url': 'http://192.0.2.1:5000/v2.0'}
Get auth_ref
REQ: curl -g -i -X GET http://192.0.2.1:5000/v2.0 -H "Accept: application/json" -H "User-Agent: osc-lib keystoneauth1/2.9.1 python-requests/2.10.0 CPython/2.7.5"
Starting new HTTP connection (1): 192.0.2.1
"GET /v2.0 HTTP/1.1" 200 223
RESP: [200] Date: Sat, 23 Jul 2016 13:25:32 GMT Server: Apache/2.4.6 (CentOS) Vary: X-Auth-Token,Accept-Encoding x-openstack-request-id: req-08295268-1f96-4440-a57a-b2d24f80f3f4 Content-Encoding: gzip Content-Length: 223 Connection: close Content-Type: application/json
RESP BODY: {"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}], "id": "v2.0", "links": [{"href": "http://192.0.2.1:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}]}}

Making authentication request to http://192.0.2.1:5000/v2.0/tokens
Resetting dropped connection: 192.0.2.1
"POST /v2.0/tokens HTTP/1.1" 200 1013
take_action(Namespace(csv=False, deploy_kernel='bm-deploy-kernel', deploy_ramdisk='bm-deploy-ramdisk', file_in=<open file 'instackenv.json', mode 'r' at 0x3baf5d0>, initial_state='available', instance_boot_option='local', json=True, no_deploy_image=False, service_host=None))
Instantiating baremetal client: <class 'ironicclient.v1.client.Client'>
Instantiating workflow engine client: <class 'mistralclient.api.v2.client.Client'>
_auth_params
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 346, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/baremetal.py", line 233, in take_action
    ramdisk_name=deploy_ramdisk
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 24, in register_or_update
    workflow_client = clients.workflow_engine
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 53, in __get__
    six.reraise(new_err.__class__, new_err, sys.exc_info()[2])
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 48, in __get__
    self._handle = self.factory(instance)
  File "/usr/lib/python2.7/site-packages/mistralclient/osc/plugin.py", line 38, in make_client
    if not instance._url:
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 64, in __getattr__
    return self._auth_params[name[1:]]
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 66, in __getattr__
    raise AttributeError(name)
PluginAttributeError: _auth_params
clean_up ImportBaremetal: _auth_params
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 118, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 226, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 163, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 346, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/baremetal.py", line 233, in take_action
    ramdisk_name=deploy_ramdisk
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 24, in register_or_update
    workflow_client = clients.workflow_engine
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 53, in __get__
    six.reraise(new_err.__class__, new_err, sys.exc_info()[2])
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 48, in __get__
    self._handle = self.factory(instance)
  File "/usr/lib/python2.7/site-packages/mistralclient/osc/plugin.py", line 38, in make_client
    if not instance._url:
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 64, in __getattr__
    return self._auth_params[name[1:]]
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 66, in __getattr__
    raise AttributeError(name)
PluginAttributeError: _auth_params

END return value: 1

[Alan Pevec (apevec)]

Stack trace makes latest refactoring in OSC prime suspect, last known good to bad OSC: https://github.com/openstack/python-openstackclient/compare/719c5d7...a8880e8
Does it work with OSC revert back to 2.6.0 ?

[James Slagle (james-slagle)]

From what I can tell it is due to this commit in python-openstackclient:
https://review.openstack.org/#/c/329189/

that commit switched to using the ClientManager class from osc-lib and removed the one from openstackclient. The one in openstackclient used to set self._auth_params in setup_auth by calling the build_auth_params function. That no longer happens in the class from osc-lib, so there is no attribute _auth_params.

I'm not honestly sure if the fix should be with openstackclient or python-mistralclient or possibly python-tripleoclient.

[Steve Martinelli (stevemar)]

temporarily we could revert the osc patch that was just merged

[Alan Pevec (apevec)]

OSC revert https://review.openstack.org/346469 has trouble in the gate
so I've included this as RPM patch in https://review.rdoproject.org/r/1726

[Alan Pevec (apevec)]

In RDO Trunk master build: python-openstackclient-2.6.1-0.20160724225155.a8880e8.el7

[Dean Troyer (dtroyer)]

The client plugins need to be fixed. In the mean time a comapt attribute can be added similar to the handful that are already present.

[Dean Troyer (dtroyer)]

It looks like these clients are using private data they never should have. Yes I am well aware that 'private' has not really been enforced until now, that is why a bunch of attributes are no longer private.

But the base plugins have never used _auth_params and nobody needs to. Those are cli args before auth has happened. The change to using only os-client-config to handle the options removed our ability to even have that data available in the same form. Besides, anything anyone could possible want auth-related should come directly from the actual auth plugin in ClientManager.auth. Even better yet, plugins should be using ClientManager.session and they should not need to think about auth again.

[Dean Troyer (dtroyer)]

I should amend that last comment to say that it isn't direct use of _auth_params that was the problem, it is how that was being called via ClientManager.__getattr__(). In the above traceback, the attempt was made to get ClientManager._url, which then was looked up in _auth_params. This is being totally removed in osc-lib as everything related to that dict is now handled directly or indirectly by os-client-config.

The auth plugin interface at ClientManager.auth is the public, supported way to get information about the current authentication plugin.

[James Slagle (james-slagle)]

i think the problem might actually be in mistralclient. That is what is trying to access the private attribute _url:

   File "/usr/lib/python2.7/site-packages/mistralclient/osc/plugin.py", line 38, in make_client
    if not instance._url:

[James Slagle (james-slagle)]

proposed fix: https://review.openstack.org/#/c/347354/

[Alan Pevec (apevec)]

So patch is only in mistralclient and no changes required in tripleoclient?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/348460

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/348460
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=4f10fe4080809fcec9d4bfdeac4633ddd4683499
Submitter: Jenkins
Branch: master

commit 4f10fe4080809fcec9d4bfdeac4633ddd4683499
Author: Brad P. Crochet <email address hidden>
Date: Thu Jul 28 12:02:40 2016 -0400

    Remove another private OSC attribute usage

    tripleoclient WebsocketClient used a private auth attribute. This
    switches it to a more public api.

    Change-Id: Ib6a3ad21b9d770dad943b6e567cfbea8402cc84f
    Closes-Bug: #1605876

[John Trowbridge (trown)]

We found another place that a private attribute from OSC was being used... this time in heatclient:

Instantiating orchestration client: <class 'heatclient.v1.client.Client'>
'ClientManager' object has no attribute '_auth_url'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 346, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 929, in take_action
    orchestration_client = clients.orchestration
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 53, in __get__
    six.reraise(new_err.__class__, new_err, sys.exc_info()[2])
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 48, in __get__
    self._handle = self.factory(instance)
  File "/usr/lib/python2.7/site-packages/heatclient/osc/plugin.py", line 43, in make_client
    'auth_url': instance._auth_url,
PluginAttributeError: 'ClientManager' object has no attribute '_auth_url'
clean_up DeployOvercloud: 'ClientManager' object has no attribute '_auth_url'
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 118, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 226, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 163, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 346, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 59, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 929, in take_action
    orchestration_client = clients.orchestration
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 53, in __get__
    six.reraise(new_err.__class__, new_err, sys.exc_info()[2])
  File "/usr/lib/python2.7/site-packages/osc_lib/clientmanager.py", line 48, in __get__
    self._handle = self.factory(instance)
  File "/usr/lib/python2.7/site-packages/heatclient/osc/plugin.py", line 43, in make_client
    'auth_url': instance._auth_url,
PluginAttributeError: 'ClientManager' object has no attribute '_auth_url'

[Alan Pevec (apevec)]

Anyone working on heatclient fix?

[Rabi Mishra (rabi)]

requirements upper_constraints is still 'python-openstackclient===2.6.0'[1] and this seems like a breaking change and not backward compatible with some of the attributes like 'region_name'.

[1] https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L291

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/349138

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/349138
Committed: https://git.openstack.org/cgit/openstack/python-heatclient/commit/?id=4d7ab61187c3d6cdf2682df1eeabe9d96121a5e2
Submitter: Jenkins
Branch: master

commit 4d7ab61187c3d6cdf2682df1eeabe9d96121a5e2
Author: Rabi Mishra <email address hidden>
Date: Sat Jul 30 08:04:45 2016 +0530

    Change usage of _auth_url private attribute

    python-openstackclient-2.6.1 has been changed
    to use osc-lib ClientManager which removes
    some private attributes.

    We can change '_interface' and '_region_name'
    once the requirements change to 2.6.1.

    Change-Id: I17fbe2250fdeb60ffb9b2f94df26d9edef00bea6
    Closes-Bug: #1605876

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/python-mistralclient 2.1.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/python-heatclient 1.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 5.0.0 release.