Multi extends which set the same option, and += behaviour

Bug #1060236 reported by Benoît Bryon on 2012-10-02
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Buildout
Undecided
Unassigned

Bug Description

In buildout, we can extend multiple files,
i.e. ``[buildout] extends = base1 base2``.

Implementation starts here:
https://github.com/buildout/buildout/blob/f45bcbf1ae9bae74954cc61ace7bbec8bbd51f00/src/zc/buildout/buildout.py#L1393

But what is the typical usage or expected behaviour of extending multiple
configuration files? Depending on the answer, this ticket could be a bug
report, a documentation request, or a feature request.

Here is the story...

Basically, I'd like to construct a ``[buildout] parts = base1 base2 foo`` where:

* ``parts = base1`` is inherited from some base1.cfg,
* ``parts = base2`` is inherited from some base2.cfg,
* ``parts = foo`` is set in root configuration file.

Is that the expected behaviour of multiple extends? Is that a possible use case for multiple extends?
Else, could I get this particular result by another mean? How?

First example: the way I'd like it to work
==========================================

The following pattern doesn't work:

.. code-block:: cfg
   :filename: buildout.cfg

   [buildout]
   extends = base1.cfg base2.cfg
   parts += foo

.. code-block:: cfg
   :filename: base1.cfg

   [buildout]
   parts = base1

.. code-block:: cfg
   :filename: base2.cfg

   [buildout]
   parts = base2

=> KO, we get ``parts = base2 foo``.

This syntax is the one I'd prefer, because I feel it is easy to understand and use:

* every base declares its own directives
* in most cases, bases declare distinct directives and sections
* but if there are clashes between bases, values are concatenated
* it makes bases independent from each other
* it makes it easy to reuse and combine existing configuration files
* if necessary, we still get the opportunity to override some inherited values with =, += or -=
* bases don't have to use ``+=`` everywhere

... but, as told before, it doesn't give the result I expect. Let's try something else.

Second example: using ``+=``
============================

I tried to use ``+=`` everywhere, which "almost works":

.. code-block:: cfg
   :filename: buildout.cfg

   [buildout]
   extends = base1.cfg base2.cfg
   parts += foo

.. code-block:: cfg
   :filename: base1.cfg

   [buildout]
   parts += base1

.. code-block:: cfg
   :filename: base2.cfg

   [buildout]
   parts += base2

=> KO, we get ``parts = base2 foo base1``. It's OK if order doesn't matter.
In my current use case, I don't care about order, so I could accept it. But...

A more complex example. Fails
=============================

Then, even with ``+=`` everywhere the following example fails:

.. code-block:: cfg
   :filename: buildout.cfg

   [buildout]
   extends = a.cfg b.cfg

.. code-block:: cfg
   :filename: a.cfg

   [buildout]
   extends = base-a1.cfg base-a2.cfg

.. code-block:: cfg
   :filename: base-a1.cfg

   [buildout]
   # Doesn't affect "parts" directive.

.. code-block:: cfg
   :filename: base-a2.cfg

   [buildout]
   parts += base-a2

.. code-block:: cfg
   :filename: b.cfg

   [buildout]
   extends = base-b1.cfg
   parts += b

.. code-block:: cfg
   :filename: base-b1.cfg

   [buildout]
   parts += base-b1

=> I expect ``parts = base-a2 base-b1 b`` but I get ``parts = base-b1 b``,
i.e. the parts directive in base-a2.cfg is ignored!

.. note::

   if base-a1.cfg does ``parts += base-a1``, then I get
   ``parts = base-a1 base-b1 b``, i.e. base-a2 is still ignored.

Implementation
==============

Looking at the code:

* Multiple extends handling starts here:
  https://github.com/buildout/buildout/blob/f45bcbf1ae9bae74954cc61ace7bbec8bbd51f00/src/zc/buildout/buildout.py#L1397

* If we'd like to implement the feature, I suppose we could use distinct
  functions to "update" buildout values depending whether we are "merging
  multiple bases with each other" or "applying result of bases on current
  file".

  I mean, I'm suggesting a change at
  https://github.com/buildout/buildout/blob/f45bcbf1ae9bae74954cc61ace7bbec8bbd51f00/src/zc/buildout/buildout.py#L1398
  replace _update(base1, base2) by some _merge_bases(base1, base2).

  Notice it wouldn't affect the "apply bases on current file" at
  https://github.com/buildout/buildout/blob/f45bcbf1ae9bae74954cc61ace7bbec8bbd51f00/src/zc/buildout/buildout.py#L1400

Conclusion
==========

The previous examples and the implementation make me wonder whether it is a
bug or a feature: is the implementation wrong? is my usage?

In bases, I try to define unique sections, i.e. each base implements distinct
features. But, there are still some cases where conflicts can appear. Values
in ``[buildout]`` section are subject to conflicts.

As a conclusion, I'm proposing to:

* if you feel the feature describe above is valuable:

  * document another way to "combine bases", maybe with some explicit
    declarations that avoid the clashes.

  * or change the implementation of "merging bases", then make it clear in
    documentation too.

* or, if that definitely not a wanted-feature or definitely not a recommended
  practice:

  * make it clear in the documentation.
  * or display some warning when assigning the same directive in several bases.
  * or maybe disallow multiple inheritance.

Thanks for reading this long report ;)

Benoît Bryon (benoit.bryon) wrote :

I did a pull request about this ticket: https://github.com/buildout/buildout/pull/22

It makes the 3 examples provided in the ticket description above pass.
On the other side, it changes the behaviour of the multiple inheritance.

The motivation for the behaviour change is:

* develop bases as independant components
* inherit all properties of bases
* override some properties if necessary

Another note.... in http://pypi.python.org/pypi/zc.buildout/1.6.3#multiple-configuration-files:

``extends = b1.cfg b2.cfg b3.cfg`` gives ``${debug:op2} == b2 2``
but ``extends = b2.cfg b1.cfg b3.cfg`` (order changed) gives ``${debug:op2} == b1 2``

With the pull request, we'd get respectively "b1 2\nb2 2" and "b2 2\nb1 2".

Benoît Bryon (benoit.bryon) wrote :

.. note::

   found 2 other open tickets related to inheritance and ``+=``:

   * https://bugs.launchpad.net/zc.buildout/+bug/435837
   * https://bugs.launchpad.net/zc.buildout/+bug/458363

Encolpe Degoute (encolpe) wrote :

The documentation is not clear about that but there's no bug there.

You can see how to you can use extends and += look at this package:
https://github.com/collective/zopeskel.unis

Benoît Bryon (benoit.bryon) wrote :

Ok, let's focus first on current feature, implementation and documentation.
I opened https://bugs.launchpad.net/zc.buildout/+bug/1067259

I suspect the real problem here is not so much multiple inheritance but +=.
If you have parts+= without having parts= in a base then it gets ignore. But then you have multiple inheritance you don't want to have parts= in multiple bases.

I think the solution is to make += work the same as = if you don't inherit from something that includes a =. That is intuitive to me.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers