maas does not combine kernel_opts when nodes have multiple tags with kernel options

[Ryan Harper]

1. will attach if needed

2. $ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-================================-=============================-============-===============================================================================
ii maas 1.5.1+bzr2269-0ubuntu0.1 all MAAS server all-in-one metapackage
ii maas-cli 1.5.1+bzr2269-0ubuntu0.1 all MAAS command line API tool
ii maas-cluster-controller 1.5.1+bzr2269-0ubuntu0.1 all MAAS server cluster controller
ii maas-common 1.5.1+bzr2269-0ubuntu0.1 all MAAS server common files
ii maas-dhcp 1.5.1+bzr2269-0ubuntu0.1 all MAAS DHCP server
ii maas-dns 1.5.1+bzr2269-0ubuntu0.1 all MAAS DNS server
ii maas-region-controller 1.5.1+bzr2269-0ubuntu0.1 all MAAS server complete region controller
ii maas-region-controller-min 1.5.1+bzr2269-0ubuntu0.1 all MAAS Server minimum region controller
ii python-django-maas 1.5.1+bzr2269-0ubuntu0.1 all MAAS server Django web framework
ii python-maas-client 1.5.1+bzr2269-0ubuntu0.1 all MAAS python API client
ii python-maas-provisioningserver 1.5.1+bzr2269-0ubuntu0.1 all MAAS server provisioning libraries

3.
a) maas $user tags new name=disable_biosdevname kernel_opts="net.ifnames=1 biosdevname=0"
b) maas $user tags new name=enable_console kernel_options="console=ttyS0,115200n8"
c) maas $user tag update-nodes disable_biosdevname add=$node1_system_id
d) maas $user tag update-nodes enable_console add=$node1_system_id
e) maas $user node start $node1_system_id
f) check /proc/cmdline on $node1_system_id

Expected result:

$node1_system_id boots with kernel params: console=ttyS0,115200n8 net.ifnames=1 biosdevname=0

Actual result:

$node1_system_id boots with kernel params: net.ifnames=1 biosdevname=0

[Raphaël Badin (rvb)]

Right now, the first tag (lexicographical order) with a non-empty kernel_opts field wins. Concatenating all the kernel_opts of the node's tags makes more sense indeed.

[Gavin Panella (allenap)]

The rule that the first tag with options wins (and also overrides global options) is arbitrary, there to provide some predictability, but if we switch to combining tags we then need to make an arbitrary choice of which order in which to combine them (assuming it matters).

Another problem arises if two or more tags have conflicting kernel options. You've got to be strict then to avoid assigning multiple of these tags to a single node. Only using the options from a single tag avoids this problem.

A concept like radio-groups for tags might help, where a subset of tags can be said to be in a group, and only one of the group can be selected at a time. These could be used for conflicting kernel options. However, how would these play with auto tags? Also, we kind of have these already with physical zones.

That's not to say that we shouldn't combine options, but we do need to give it more thought before we do because it's a subtle problem.

[Ryan Harper (raharper)]

An alternative is to allow specifying kernel_opts on a per-node basis independent of tags.

[Raphaël Badin (rvb)]

> The rule that the first tag with options wins (and also overrides global options) is arbitrary, there to provide some predictability,
> but if we switch to combining tags we then need to make an arbitrary choice of which order in which to combine them
> (assuming it matters).

I agree; we should order the tags before we concatenate the kernel_opts so that the end result can be fine-tuned by the user.

> Another problem arises if two or more tags have conflicting kernel options

Right, but this is all under the user's control so I don't see that as a problem. As long as the resulting kernel_opt line is predictable and exposed in the UI/API (for verification), I think it's fine to concatenate the kernel_opts as it gives the user more leeway.

[Raphaël Badin (rvb)]

At least the current behavior is documented: http://maas.ubuntu.com/docs/kernel-options.html.

[Julian Edwards (julian-edwards)]

Why don't we just check for overlap when adding a tag to a node, and
reject it?

[Gavin Panella (allenap)]

> Why don't we just check for overlap when adding a tag to a node, and
> reject it?

Do you mean checking for overlap option by option? Or prevent nodes from
being used where multiple tags with kernel options apply?

[Julian Edwards (julian-edwards)]

On Tuesday 01 Jul 2014 11:56:23 you wrote:
> > Why don't we just check for overlap when adding a tag to a node, and
> > reject it?
>
> Do you mean checking for overlap option by option? Or prevent nodes from
> being used where multiple tags with kernel options apply?

The former, although failing that the latter would be a good thing anyway.

It would have to be a pretty naive algorithm though.

[Ryan Harper (raharper)]

My preference would be to not bother checking for overlap and let the user control. If there is simple concat, then the user can accept the overlap, or modify the kernel_opts in the tag. Having maas do more things to the options than just joining could have more side-effects.

I'd really not like tag creation/update to fail if some other user happened to define tag with one of the kernel opts I wanted to us.

[Julian Edwards (julian-edwards)]

On 01/07/14 23:34, Ryan Harper wrote:
> My preference would be to not bother checking for overlap and let the
> user control. If there is simple concat, then the user can accept the
> overlap, or modify the kernel_opts in the tag. Having maas do more
> things to the options than just joining could have more side-effects.

But in which order should it concat the options? Because this will
affect way in which the kernel sees them.

IoW I don't think this is as simple as a straight concat.

> I'd really not like tag creation/update to fail if some other user
> happened to define tag with one of the kernel opts I wanted to us.

Remember that only admins can tag nodes. So restricting to one tag that
has kernel options is not that big a deal.

[Ryan Harper (raharper)]

Let me explain my use case.

I have hundreds of systems in my maas, very heterogeneous in make and model. Some of them require various console=ttySX,YY values which vary depending on the chassis. With in the mix of machines, I have some number N of each of these. I tag each group of machines hardware-VENDOR_X-MODEL_Y; and I'd like to associate a specific set of kernel options for these tagged machines. I may find that some of the machines require one-off, or special tags for debugging that apply kernel parameters (for example the biosdevname workaround). So, for any given machine, I need to be able to joined multiple tags of kernel_opts to build the command line via tag assignment. Some systems also make use of third-party drivers (which is currently appended to tag-based kernel parameters).

Certainly a single tag would be sufficient, but instead of doing updating a single tag's kernel options when I wanted to change the parameters for systems with one-off tags (maas $user tag update kernel_opts="foo"), I need to apply tag update kernel_opts="foo" for a systems that need it, which means i need to track what tag needs applied somewhere else.

>> My preference would be to not bother checking for overlap and let the
>> user control. If there is simple concat, then the user can accept the
>> overlap, or modify the kernel_opts in the tag. Having maas do more
>> things to the options than just joining could have more side-effects.
>
>But in which order should it concat the options? Because this will
>affect way in which the kernel sees them.
>
>IoW I don't think this is as simple as a straight concat.

I think we can extend the current behavior of sorted tags with kernel_opts and append based on that order.

tag name="foobar" kernel_opts="wark"
tag name="disable_foo" kernel_opts="foo=0"
tag name="zomg_need_foo" kernel_opts="foo=1"
third_party_params="ahci=0"

would generate:
kernel_opts: foo=0 wark foo=1 ahci=0

[Adam Collard (adam-collard)]

This bug has not seen any activity in the last 6 months, so it is being automatically closed.

If you are still experiencing this issue, please feel free to re-open.

MAAS Team

[Felipe Reyes (freyes)]

> If you are still experiencing this issue, please feel free to re-open.

I'm re-opening this bug since we are having a hard time when trying to configure our lab's maas where we would like to have different kernel options depending on the peculiarities of the hardware, for example some machines might need to be deployed with console=ttyS1 while others don't, this is a multi-user MAAS environment, so sometimes users add new tags with kernel_opts and it's difficult to determine the kernel options that the machine will since even when this is documented it's counterintuitive.

this command line is used as example in the docs:

maas $PROFILE tags create name='nomodeset' \
    comment='nomodeset kernel option' kernel_opts='nomodeset vga'

This approach suggests that a user could come up with a set of tags that later could be used to compose the desired kernel options.