Cannot load complex configuration options

[Gorka Eguileor]

Cinderlib cannot properly load configuration options of the type ListOpt, or MultiOpt where each item is a dictionary, or DictOpt...

This used to work before we started using the oslo config parser and StringIO as a file (commit 85776225cb8a971e6e26227dc469c48df08ca7c8).

Later on we fixed the basic MultiOpt case (commit c8fc95260c46004ab5ddd16cd26bbfda2f4802c6), but all other complex option types are still broken.

For example, when we try to load a ListOpt option we will see:

  oslo_config.cfg.ConfigFileValueError: Value for option XYZ from LocationInfo(location=<Locations.user: (4, True)>, detail='/home/me/in_memory_file') is not valid: Value should start with "["

This is because cinderlib is treating this option as a MultiOpt.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/696375

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696375
Committed: https://git.openstack.org/cgit/openstack/cinderlib/commit/?id=085cb4e4327624dd76f00f2d34aa2bf28a470d8c
Submitter: Zuul
Branch: master

commit 085cb4e4327624dd76f00f2d34aa2bf28a470d8c
Author: Gorka Eguileor <email address hidden>
Date: Wed Nov 27 18:32:17 2019 +0100

    Fix complex configuration options

    Cinderlib cannot properly load configuration options of the type
    ListOpt, or MultiOpt where each item is a dictionary, or DictOpt...

    For example, when we try to load a ListOpt option we will see:

      oslo_config.cfg.ConfigFileValueError: Value for option XYZ from
      LocationInfo(location=<Locations.user: (4, True)>,
      detail='/home/me/in_memory_file') is not valid: Value should start
      with "["

    This is because cinderlib is treating this option as a MultiOpt.

    Cinderlib used to just set the values of the options in the instance and
    pass it to the driver, and back then we had no problems with complex
    types.

    The problem with that approach is that there are some drivers (like
    NetApp's) that dynamically add configuration options and then force a
    reload of the file, and they couldn't work with our old approach.

    So we changed to used oslo.config's normal parsing of files, but to do
    that we had to convert the driver parameters passed to cinderlib to a
    config file and make the parser use a StringIO instance instead of a
    real file. We also had to prevent it from looking at the CLI options
    (since those belong to the program that imports the library).

    This new approach is more complex, because cinderlib receives the
    parameters as Python primitives (list, dicts, integers, etc.), but the
    Oslo config parser expects an INI file conforming to the non standard
    types it provides.

    Some of these non standar types are:

    - DictOpt is in the form:
         option = key1:value1,key2:value2

    - MultiOpt is in the form:
         option = value1
         option = value2

    - ListOpt is in the form:
         option = [value1,value2]

    So cinderlib would receive a list [value1, value2] or a tuple (value1,
    value2) and needs to generate a string with the right configuration
    option which can be in the form of a MultiOpt or a ListOpt, depending on
    how the configuration option was defined.

    This patch adds more logic to the conversion of cinder driver
    configuration parameters to oslo config file and uses the configuration
    options definitions to decide what conversion needs to be done.

    With this change we now do the right converstion for ListOpt, DictOpt,
    and we can even support MultiOpt options that have dictionaries as
    items.

    Closes-Bug: #1854188
    Change-Id: I62e992804a3ae6aa0b4aa4f883807783197d4b33

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinderlib 1.0.0 release.