it should be possible to disable interpolation for some option definitions

[Dr. Jens Harbott]

After installing shaker on an Ubuntu Trusty cloud instance according to http://pyshaker.readthedocs.io/en/latest/installation.html#installation-on-ubuntu-cloud-image I get:

$ shaker-image-builder -d
2016-05-03 11:05:08.507 2882 INFO shaker.engine.utils [-] Logging enabled
2016-05-03 11:05:08.507 2882 DEBUG shaker.engine.utils [-] ******************************************************************************** log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2503
2016-05-03 11:05:08.507 2882 DEBUG shaker.engine.utils [-] Configuration options gathered from: log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2504
2016-05-03 11:05:08.507 2882 DEBUG shaker.engine.utils [-] command line args: ['-d'] log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2505
2016-05-03 11:05:08.508 2882 DEBUG shaker.engine.utils [-] config files: [] log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2507
2016-05-03 11:05:08.508 2882 DEBUG shaker.engine.utils [-] ================================================================================ log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2508
2016-05-03 11:05:08.508 2882 DEBUG shaker.engine.utils [-] cleanup_on_error = True log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.508 2882 DEBUG shaker.engine.utils [-] config_dir = None log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.508 2882 DEBUG shaker.engine.utils [-] config_file = [] log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.508 2882 DEBUG shaker.engine.utils [-] debug = True log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.509 2882 DEBUG shaker.engine.utils [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urll
ib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'dogpile.core.dogpile=INFO',
'pykwalify=INFO'] log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.509 2882 DEBUG shaker.engine.utils [-] external_net = None log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.509 2882 DEBUG shaker.engine.utils [-] flavor_name = shaker-flavor log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.509 2882 DEBUG shaker.engine.utils [-] image_builder_template = ubuntu log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.509 2882 DEBUG shaker.engine.utils [-] image_name = shaker-image log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.509 2882 DEBUG shaker.engine.utils [-] instance_format = [instance: %(uuid)s] log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.510 2882 DEBUG shaker.engine.utils [-] instance_uuid_format = [instance: %(uuid)s] log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.510 2882 DEBUG shaker.engine.utils [-] log_config_append = None log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.510 2882 DEBUG shaker.engine.utils [-] log_date_format = %Y-%m-%d %H:%M:%S log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.510 2882 DEBUG shaker.engine.utils [-] log_dir = None log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.510 2882 DEBUG shaker.engine.utils [-] log_file = None log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.510 2882 DEBUG shaker.engine.utils [-] logging_context_format_string = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [%(request_id)s %(user_identity)s] %(instance)s%(message)s log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_conf
ig/cfg.py:2517
2016-05-03 11:05:08.511 2882 DEBUG shaker.engine.utils [-] logging_debug_format_suffix = %(funcName)s %(pathname)s:%(lineno)d log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.511 2882 DEBUG shaker.engine.utils [-] logging_default_format_string = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.511 2882 DEBUG shaker.engine.utils [-] logging_exception_prefix = %(asctime)s.%(msecs)03d %(process)d ERROR %(name)s %(instance)s log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.511 2882 DEBUG shaker.engine.utils [-] logging_user_identity_format = %(user)s %(tenant)s %(domain)s %(user_domain)s %(project_domain)s log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.511 2882 DEBUG shaker.engine.utils [-] os_auth_url = https://beta.hamburg.iaas.xws.x-ion.de:5000/v2.0 log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.511 2882 DEBUG shaker.engine.utils [-] os_cacert = None log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.512 2882 DEBUG shaker.engine.utils [-] os_insecure = False log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2517
2016-05-03 11:05:08.512 2882 CRITICAL shaker [-] NoSuchOptError: no such option in group DEFAULT: os_password
2016-05-03 11:05:08.512 2882 ERROR shaker Traceback (most recent call last):
2016-05-03 11:05:08.512 2882 ERROR shaker File "/usr/local/bin/shaker-image-builder", line 11, in <module>
2016-05-03 11:05:08.512 2882 ERROR shaker sys.exit(build_image())
2016-05-03 11:05:08.512 2882 ERROR shaker File "/usr/local/lib/python2.7/dist-packages/shaker/engine/image_builder.py", line 56, in build_image
2016-05-03 11:05:08.512 2882 ERROR shaker openstack_client = init()
2016-05-03 11:05:08.512 2882 ERROR shaker File "/usr/local/lib/python2.7/dist-packages/shaker/engine/image_builder.py", line 35, in init
2016-05-03 11:05:08.512 2882 ERROR shaker config.OPENSTACK_OPTS + config.IMAGE_BUILDER_OPTS)
2016-05-03 11:05:08.512 2882 ERROR shaker File "/usr/local/lib/python2.7/dist-packages/shaker/engine/utils.py", line 73, in init_config_and_logging
2016-05-03 11:05:08.512 2882 ERROR shaker conf.log_opt_values(LOG, std_logging.DEBUG)
2016-05-03 11:05:08.512 2882 ERROR shaker File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2517, in log_opt_values
2016-05-03 11:05:08.512 2882 ERROR shaker _sanitize(opt, getattr(self, opt_name)))
2016-05-03 11:05:08.512 2882 ERROR shaker File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2183, in __getattr__
2016-05-03 11:05:08.512 2882 ERROR shaker raise NoSuchOptError(name)
2016-05-03 11:05:08.512 2882 ERROR shaker NoSuchOptError: no such option in group DEFAULT: os_password
2016-05-03 11:05:08.512 2882 ERROR shaker

[Dr. Jens Harbott (j-harbott)]

Looks like this is really a bug in oslo_config, only happens when OS_PASSWORD environment variable is set to a value containing special characters, will try to debug this further.

[Dr. Jens Harbott (j-harbott)]

The trigger is using a password that contains a "$", but I'm uncertain as of yet whether the issue is within oslo_config or in the way shaker uses it.

[Dr. Jens Harbott (j-harbott)]

Attached is a stripped down sample of what shaker does:

(venv) ubuntu@x1:~$ OS_PASSWORD=test python test.py
Pass is "test"

(venv) ubuntu@x1:~$ OS_PASSWORD=te\$st python test.py
Traceback (most recent call last):
  File "test.py", line 17, in <module>
    conf.log_opt_values(LOG, std_logging.DEBUG)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2517, in log_opt_values
    _sanitize(opt, getattr(self, opt_name)))
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2183, in __getattr__
    raise NoSuchOptError(name)
oslo_config.cfg.NoSuchOptError: no such option in group DEFAULT: os_password
(venv) ubuntu@x1:~$ OS_PASSWORD=test python test.py --os-password=over
Pass is "over"

(venv) ubuntu@x1:~$ OS_PASSWORD=test python test.py --os-password=ov\$er
Traceback (most recent call last):
  File "test.py", line 16, in <module>
    conf(project='test')
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2162, in __call__
    else sys.argv[1:])
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2754, in _parse_cli_opts
    return self._parse_config_files()
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2773, in _parse_config_files
    self._validate_cli_options(namespace)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2786, in _validate_cli_options
    value = self._substitute(value, group=group, namespace=namespace)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2652, in _substitute
    self.StrSubWrapper(self, group=group, namespace=namespace))
  File "/usr/lib/python2.7/string.py", line 207, in safe_substitute
    return self.pattern.sub(convert, self.template)
  File "/usr/lib/python2.7/string.py", line 198, in convert
    return '%s' % (mapping[named],)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 3035, in __getitem__
    value = self.conf._get(key, namespace=self.namespace)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2567, in _get
    value = self._do_get(name, group, namespace)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2584, in _do_get
    info = self._get_opt_info(name, group)
  File "/home/ubuntu/venv/local/lib/python2.7/site-packages/oslo_config/cfg.py", line 2718, in _get_opt_info
    raise NoSuchOptError(opt_name, group)
oslo_config.cfg.NoSuchOptError: no such option in group DEFAULT: er

[Dr. Jens Harbott (j-harbott)]

Seems to be an unexpected case of value interpolation at work, see http://docs.openstack.org/developer/oslo.config/cfg.html#option-value-interpolation

But still, I think a) oslo.config should do a better error message in the first case, b) shaker should somehow make sure that these variables work the same way they do with other tools like OSC.

[Ilya Shakhat (shakhat)]

The issue occurs only if the value starts with "$" character. The workaround is to escape it.

From Shaker perspective it's possible to fix this by escaping "$" before setting it as default to config option (http://git.openstack.org/cgit/openstack/shaker/tree/shaker/engine/config.py#n92). Personally, I'd prefer oslo-config to be responsible for reading option values from environment variables and thus doing all magic under the hood.

[Dr. Jens Harbott (j-harbott)]

@Ilya: The expansion is not restricted to a starting "$", it can happen anywhere within the value, at least for the tests that I did.

Escaping by using "$$" is possible as a workaround for shaker but will break all other clients that reference OS_PASSWORD like e.g. openstackclient.

[ChangBo Guo(gcb) (glongwave)]

I think oslo.config can provide a way to forbid this case explicitly by some parameter like allow_interpolation=False.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/338668

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ilya Shakhat (<email address hidden>) on branch: master
Review: https://review.openstack.org/320537
Reason: postponed until https://review.openstack.org/338668 comes into oslo.config

[OpenStack Infra (hudson-openstack)]

Change abandoned by ChangBo Guo(gcb) (<email address hidden>) on branch: master
Review: https://review.openstack.org/338668