Adding options (+=) fail with extended sections (macros)

[Kai Lautaportti]

Adding new options using the += notation to an option in a section that is an extended section (using another section as a macro with <= notation) fails if the option in question is defined only in the macro section.

For example, the following doctest

    >>> write(sample_buildout, 'base.cfg',
    ... """
    ... [buildout]
    ... develop = recipes
    ... parts = mypart
    ... log-level = INFO
    ...
    ... [macro]
    ... recipe = recipes:debug
    ... foo =
    ... original_value_1
    ... original_value_2
    ... original_value_3
    ...
    ... [mypart]
    ... <= macro
    ... """)

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... extends = base.cfg
    ...
    ... [mypart]
    ... foo +=
    ... extended_option_1
    ... extended_option_2
    ... """)

Gives

    >>> print system(buildout),
    Develop: '/sample-buildout/recipes'
    Installing mypart.
    foo
    <BLANKLINE>
    extended_option_1
    extended_option_2
    recipe recipes:debug

Losing the original values from the ``foo`` option. It should print.

    >>> print system(buildout),
    Develop: '/sample-buildout/recipes'
    Installing mypart.
    foo
    extended_option_1
    extended_option_2
    original_value_1
    original_value_2
    original_value_3
    recipe recipes:debug

It appears that the +=/-= processing takes place before the macro substitution which should be vice versa.

[Kai Lautaportti (kai-lautaportti)]

After taking a better look at the zc.buildout code I can see that the +=/-= semantics are implemented in _update_section() function which is called by _open() when configurations are read in. However, the macro expansion is currently done much later by Options._do_extend_raw().

I prototyped a different implementation that expands macros as part of the _open() function before it calls _update_section() which seems to partially solve the issue and make. However, the prototype introduces a regression in the ``tests.increment_on_command_line`` test because the command line overrides are only applied after all the config files have been read (and therefore _update_section() calls have been made and expanding macros is too late).

This has lead me to believe that at the moment it is not possible to fix this issue by simply moving the macro expansion to an earlier stage in the initialization of the buildout.

I would be willing to work on this but would appreciate some guidance as how to best approach this as some restructuring of the configuration reading logic may be needed.

[Kai Lautaportti (kai-lautaportti)]

I have attached a patch (bug-435837.patch) with proposed changes to make
macro expansion work with option modifiers and extended configs.

The patch makes the following major changes:

 1. Macros are expanded earlier during the parsing of configuration files as
    part of the _open() function.

 2. Command-line overrides are also applied during the parsing process, not
    afterwards.

 3. Removes the previous macro expansion implementation from the Options class.

 4. Refactors some of the internal helper functions to facilitate the other
    changes.

The patch is against the current SVN trunk (r104648) and all tests pass on OSX
10.6 and Debian Linux running Python 2.4, 2.5 and 2.6. I am currently not set
up to test on Windows.

[Kai Lautaportti (kai-lautaportti)]

I've attached a revised patch that fixes an issue with recursive macro expansion under certain circumstances.

[Kai Lautaportti (kai-lautaportti)]

Latest patch with all the test cases included (some were missing from the previous attachment).