Having a '{' or '}' in password causes formatting errors

[Jon Gjengset]

On Arch Linux, all commands exit with the error "unmatched '{' in format", and nothing else. I'm using the following package versions:

  python2-cliff 2.2.0-1
  python2-openstackclient 3.3.0-1
  python2-osc-lib 1.2.0-1

Running "openstack2 server list --debug" yields:

    START with options: [u'server', u'list', u'--debug']
    options: Namespace(access_token_endpoint='', access_token_type='', auth_type='', auth_url='https://nimbus.csail.mit.edu:5001/v2.0', authorization_code='', cacert=None, cert='', client_id='', client_secret='***', cloud='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, domain_id='', domain_name='', endpoint='', identity_provider='', insecure=None, interface='', key='', log_file=None, openid_scope='', os_beta_command=False, os_compute_api_version='', os_identity_api_version='', os_image_api_version='1', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_volume_api_version='', password='***', project_domain_id='', project_domain_name='', project_id='', project_name='pdos', protocol='', redirect_uri='', region_name='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='', user_id='', user_name='', <email address hidden>', verbose_level=3, verify=None)
    Auth plugin password selected
    auth_config_hook(): {'auth_type': 'password', 'beta_command': False, u'compute_api_version': u'2', 'key': None, u'database_api_version': u'1.0', u'metering_api_version': u'2', 'auth_url': 'https://nimbus.csail.mit.edu:5001/v2.0', u'network_api_version': u'2', u'message': u'', u'image_format': u'qcow2', 'networks': [], u'image_api_version': '1', 'verify': True, u'dns_api_version': u'2', u'object_store_api_version': u'1', u'status': u'active', 'verbose_level': 3, 'region_name': '', 'api_timeout': None, u'baremetal_api_version': u'1', 'auth': {'project_name': 'pdos'}, 'default_domain': 'default', 'debug': True, u'image_api_use_tasks': False, u'floating_ip_source': u'neutron', u'orchestration_api_version': u'1', 'timing': False, 'password': '***', 'cacert': None, u'key_manager_api_version': u'v1', 'deferred_help': False, u'identity_api_version': u'2.0', u'volume_api_version': u'2', 'username': '<email address hidden>', 'cert': None, u'secgroup_source': u'neutron', u'container_api_version': u'1', u'interface': None, u'disable_vendor_agent': {}}
    Deferring keystone exception: argument of type 'OSC_Config' is not iterable
    unmatched '{' in format
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/cliff/app.py", line 250, in run
        self.initialize_app(remainder)
      File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 135, in initialize_app
        super(OpenStackShell, self).initialize_app(argv)
      File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 395, in initialize_app
        argparse=self.options,
      File "/usr/lib/python2.7/site-packages/osc_lib/cli/client_config.py", line 168, in get_one_cloud
        **kwargs
      File "/usr/lib/python2.7/site-packages/os_client_config/config.py", line 1116, in get_one_cloud
        config[key] = value.format(**config)
    ValueError: unmatched '{' in format
    Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
        ret_val = super(OpenStackShell, self).run(argv)
      File "/usr/lib/python2.7/site-packages/cliff/app.py", line 250, in run
        self.initialize_app(remainder)
      File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 135, in initialize_app
        super(OpenStackShell, self).initialize_app(argv)
      File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 395, in initialize_app
        argparse=self.options,
      File "/usr/lib/python2.7/site-packages/osc_lib/cli/client_config.py", line 168, in get_one_cloud
        **kwargs
      File "/usr/lib/python2.7/site-packages/os_client_config/config.py", line 1116, in get_one_cloud
        config[key] = value.format(**config)
    ValueError: unmatched '{' in format

    END return value: 1

I get a similar error with the Python 3 version: "openstack server list --debug"

    START with options: ['server', 'list', '--debug']
    options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', auth_type='', auth_url='https://nimbus.csail.mit.edu:5001/v2.0', authorization_code='', cacert=None, cert='', client_id='', client_secret='***', cloud='', debug=True, default_domain='default', default_domain_id='', default_domain_name='', deferred_help=False, discovery_endpoint='', domain_id='', domain_name='', endpoint='', identity_provider='', insecure=None, interface='', key='', log_file=None, openid_scope='', os_beta_command=False, os_compute_api_version='', os_identity_api_version='', os_image_api_version='1', os_network_api_version='', os_object_api_version='', os_project_id=None, os_project_name=None, os_volume_api_version='', passcode='', password='***', project_domain_id='', project_domain_name='', project_id='', project_name='pdos', protocol='', redirect_uri='', region_name='', timing=False, token='***', trust_id='', url='', user_domain_id='', user_domain_name='', user_id='', <email address hidden>', verbose_level=3, verify=None)
    Auth plugin password selected
    auth_config_hook(): {'compute_api_version': '2', 'timing': False, 'disable_vendor_agent': {}, 'cacert': None, 'auth_type': 'password', 'auth_url': 'https://nimbus.csail.mit.edu:5001/v2.0', 'image_api_version': '1', 'beta_command': False, 'volume_api_version': '2', 'orchestration_api_version': '1', 'key_manager_api_version': 'v1', 'api_timeout': None, 'verbose_level': 3, 'secgroup_source': 'neutron', 'metering_api_version': '2', 'interface': None, 'status': 'active', 'password': '***', 'deferred_help': False, 'auth': {'project_name': 'pdos'}, 'container_api_version': '1', 'message': '', 'verify': True, 'debug': True, 'database_api_version': '1.0', 'default_domain': 'default', 'key': None, 'cert': None, 'networks': [], 'baremetal_api_version': '1', 'floating_ip_source': 'neutron', 'image_api_use_tasks': False, 'object_store_api_version': '1', 'username': '<email address hidden>', 'identity_api_version': '2.0', 'network_api_version': '2', 'region_name': '', 'image_format': 'qcow2', 'dns_api_version': '2'}
    Deferring keystone exception: argument of type 'OSC_Config' is not iterable
    unmatched '{' in format spec
    Traceback (most recent call last):
      File "/usr/lib/python3.5/site-packages/cliff/app.py", line 250, in run
        self.initialize_app(remainder)
      File "/usr/lib/python3.5/site-packages/openstackclient/shell.py", line 135, in initialize_app
        super(OpenStackShell, self).initialize_app(argv)
      File "/usr/lib/python3.5/site-packages/osc_lib/shell.py", line 395, in initialize_app
        argparse=self.options,
      File "/usr/lib/python3.5/site-packages/osc_lib/cli/client_config.py", line 168, in get_one_cloud
        **kwargs
      File "/usr/lib/python3.5/site-packages/os_client_config/config.py", line 1116, in get_one_cloud
        config[key] = value.format(**config)
    ValueError: unmatched '{' in format spec
    Traceback (most recent call last):
      File "/usr/lib/python3.5/site-packages/osc_lib/shell.py", line 135, in run
        ret_val = super(OpenStackShell, self).run(argv)
      File "/usr/lib/python3.5/site-packages/cliff/app.py", line 250, in run
        self.initialize_app(remainder)
      File "/usr/lib/python3.5/site-packages/openstackclient/shell.py", line 135, in initialize_app
        super(OpenStackShell, self).initialize_app(argv)
      File "/usr/lib/python3.5/site-packages/osc_lib/shell.py", line 395, in initialize_app
        argparse=self.options,
      File "/usr/lib/python3.5/site-packages/osc_lib/cli/client_config.py", line 168, in get_one_cloud
        **kwargs
      File "/usr/lib/python3.5/site-packages/os_client_config/config.py", line 1116, in get_one_cloud
        config[key] = value.format(**config)
    ValueError: unmatched '{' in format spec

    END return value: 1

It appears that this might be related to an incorrect regex: https://github.com/SublimeLinter/SublimeLinter-for-ST2/issues/23

[Brian Davidson (bjdavidson)]

In my case, I am hitting this bug when I have a '{' in my openstack user password.

[Jon Gjengset (jonhoo)]

Ah, I didn't even think of that. Yes, that is the case for me as well!

[Steve Martinelli (stevemar)]

Ah, that actually helps out a lot. I was having trouble reproducing this issue.

[Ray Harris (lrh1)]

It appears that any "{" or "}" in a password must be duplicated to avoid those characters being treated as part of a format spec.

It's unacceptable that a password has to be modified before being entered to avoid errors.

[Steve Martinelli (stevemar)]

@Ray, totally agree. I'll see if I can investigate this issue further.

[Dean Troyer (dtroyer)]

This appears to be a result of os-client-config attempting to do nested expansion of config values. It will probably happen with any config value containing a '{' char. Yay new format syntax!

I think we're going to have to escape '{" and '}' chars now.

[Christian Schlotter (christi-schlotter)]

I alsoe have this bug using "{" in a password.
if I', going to escape it using "{{" in my environment variable I still get back that I'm not authorized.
I added a debug output to "keystoneauth1/identity/v2.py" and printed the password and it still contains "{{" instead of "{".

[Ben Ritter (britter)]

If you need a temporary fix, this worked for me:

in /usr/lib/python2.7/site-packages/os_client_config/config.py

1079 for (key, value) in config.items():
1080 if hasattr(value, 'format'):
1081 # config[key] = value.format(**config)
1082 config[key] = value

[John Dennis (jdennis-a)]

I have a proposed fix for this I'll be submitting momentarily.

At first glance Dean's suggestion in comment #6 seems like the right approach, escape the config values but that does not work because:

1) Escaping all braces breaks valid use of the format string syntax.

2) Trying to determine exactly which braces should be escaped and which should be preserved is a daunting task and likely would not be robust.

3) Some strings might look like valid format syntax but still should be escaped (e.g. "foo{bar}", if this appeared in a password we wouldn't escape it and there would be a key error on the 'bar' key.

4) In general passwords should never be modified, you never want to apply formatting to them.

I believe the right approach is to maintain a list of config values which are excluded from having formatting applied to them. At the moment that list just includes 'password' but perhaps down the road other exceptions might crop up. The proposed patch follows this approach, the list of excluded values can easily be updated if others are discovered.

[Christian Schlotter (christi-schlotter)]

As quick workaround I am using the attached patch.

In fact I agree that a password should never be modified.

[John Dennis (jdennis-a)]

For some reason the gerrit review with the proposed fix was not automatically linked to this bug.

The proposed fix is: https://review.openstack.org/#/c/525744/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/525744
Committed: https://git.openstack.org/cgit/openstack/os-client-config/commit/?id=c7243f1747b679c6bc4df8aead8133ec25898182
Submitter: Zuul
Branch: master

commit c7243f1747b679c6bc4df8aead8133ec25898182
Author: John Dennis <email address hidden>
Date: Tue Dec 5 15:19:56 2017 -0500

    Do not apply format expansions to passwords

    get_one_cloud() and get_one_cloud_osc() iterate over config
    values and try to expand any variables in those values by
    calling value.format(), however some config values
    (e.g. password) should never have format() applied to them, not
    only might that change the password but it will also cause the
    format() function to raise an exception if it can not parse the
    format string. Examples would be single brace (e.g. 'foo{')
    which raises an ValueError because it's looking for a matching
    end brace or a brace pair with a key value that cannot be found
    (e.g. 'foo{bar}') which raises a KeyError.

    It is not reasonsable to try to escape any braces because:

    1) Escaping all braces breaks valid use of the format string syntax.

    2) Trying to determine exactly which braces should be escaped and
    which should be preserved is a daunting task and likely would not be
    robust.

    3) Some strings might look like valid format syntax but still should
    be escaped (e.g. "foo{bar}", if this appeared in a password we
    wouldn't escape it and there would be a key error on the 'bar' key.

    4) In general passwords should never be modified, you never want to
    apply formatting to them.

    The right approach is to maintain a list of config values which are
    excluded from having formatting applied to them. At the moment that
    list just includes 'password' but perhaps down the road other
    exceptions might crop up. This patch follows this approach,
    the list of excluded values can easily be updated if others are
    discovered.

    Change-Id: I187bdec582d4c2cc6c7fda47a1538194137c616b
    Closes-Bug: 1635696
    Signed-off-by: John Dennis <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-openstacksdk 0.11.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-client-config 1.29.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-client-config 1.28.1 release.