Bionic netplan render invalid yaml duplicate anchor declaration for nameserver entries

Bug #1815051 reported by Márton Kiss on 2019-02-07
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
cloud-init
High
Unassigned
cloud-init (Ubuntu)
High
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned

Bug Description

The netplan configuration redeclares the nameservers anchor for every single section (vlans, bonds), and use the same id for similar entries (id001).

In this specific case the network configuration in maas have a bond0 with two vlans, bond0.3502 and bond0.3503, and an untagged bond1 without vlans. The rendered 50-cloud-init.yaml looks like this:

network:
    version: 2
    ethernets:
...
    bonds:
...
        bond1:
...
            nameservers: &id001 <- anchor declaration here
                addresses:
                - 255.255.255.1
                - 255.255.255.2
                - 255.255.255.3
                - 255.255.255.5
                search:
                - customer.domain
                - maas
...
        bondM:
...
            nameservers: *id001

   vlans:
        bond0.3502:
...
            nameservers: &id001 <- anchor redeclaration here
                addresses:
                - 255.255.255.1
                - 255.255.255.2
                - 255.255.255.3
                - 255.255.255.5
                search:
                - customer.domain
                - maas
        bond0.3503:
...
            nameservers: *id001

As the cloudinit renders an invalid yaml file, the netplan apply produces the following error: (due to the anchor redeclaration in the vlans section):

   Invalid YAML at /etc/netplan/50-cloud-init.yaml line 118 column 25: second occurence

This render bug prevents us using the untagged bond and the bond with the vlans in the same configuration.

Related branches

Ryan Harper (raharper) wrote :

Can you please run 'sudo cloud-init collect-logs' and attach the log file and the contents of /etc/netplan/

Changed in cloud-init:
importance: Undecided → High
status: New → Incomplete
Márton Kiss (marton-kiss) wrote :

cloud init logs.

Márton Kiss (marton-kiss) wrote :

netplan yaml file.

Dmitrii Shcherbakov (dmitriis) wrote :

From what I see, aliases come from the base library (pyyaml) which uses aliases if object references are used:

https://git.launchpad.net/~usd-import-team/ubuntu/+source/pyyaml/tree/lib3/yaml/serializer.py?h=ubuntu/bionic#n11
    ANCHOR_TEMPLATE = 'id%03d'
# ...
    def generate_anchor(self, node):
        self.last_anchor_id += 1
        return self.ANCHOR_TEMPLATE % self.last_anchor_id

In [2]: import yaml

In [6]: a = {'name': 'foo', 'mtu': '1500'}
   ...: b = {'name': 'bar', 'mtu': '9000'}
   ...: c = {'ref1': a, 'ref2': a, 'ref3': b, 'ref4': b}
   ...: print(yaml.dump(c))
   ...:
   ...:
ref1: &id001 {mtu: '1500', name: foo}
ref2: *id001
ref3: &id002 {mtu: '9000', name: bar}
ref4: *id002

So there must be something wrong with how cloud-init produces the aggregated yaml file.

I think this happens because it renders yaml for each section and bond interfaces and VLAN interfaces belong to different sections while nameservers object is identical and is reused in both sections (e.g. 2 bonds, 2 VLAN interfaces both use the same nameservers section but give you an anchor *per section* as the second entry in each section uses the same nameservers key which results in id001 being present).

https://github.com/cloud-init/cloud-init/blob/ubuntu/18.4-0ubuntu1_18.04.1/cloudinit/net/netplan.py#L359-L367

        def _render_section(name, section):
            if section:
                dump = util.yaml_dumps({name: section},
                                       explicit_start=False,
                                       explicit_end=False)
                txt = util.indent(dump, ' ' * 4)
                return [txt]
            return []

        content.append("network:\n version: 2\n")
        content += _render_section('ethernets', ethernets)
        content += _render_section('wifis', wifis)
        content += _render_section('bonds', bonds) # <--- first id001 gets generated
        content += _render_section('bridges', bridges)
        content += _render_section('vlans', vlans) # <--- second id001 gets generated

Changed in cloud-init:
status: Incomplete → Confirmed
Ryan Harper (raharper) wrote :

Yes, the per-section rendering means there are duplicate anchors.

I'll have a branch up that will remove the use of aliases.

Ryan Harper (raharper) on 2019-02-07
Changed in cloud-init:
status: Confirmed → In Progress
Ryan Harper (raharper) on 2019-02-07
Changed in cloud-init (Ubuntu):
importance: Undecided → High
status: New → In Progress

This bug is fixed with commit cf308366 to cloud-init on branch master.
To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=cf308366

Changed in cloud-init:
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cloud-init (Ubuntu Bionic):
status: New → Confirmed
Changed in cloud-init (Ubuntu Cosmic):
status: New → Confirmed
Changed in cloud-init (Ubuntu Xenial):
status: New → Confirmed
Nicolas Pochet (npochet) wrote :

Subscribing field-critical as it blocks a deployment

Joshua Powers (powersj) wrote :

This bug is fixed in master and as such is available for testing or usage in the cloud-init daily PPA. This is available at the URL below:

https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily

We have an SRU of cloud-init that is in progress and wrapping up in the next couple of days. The plan is to start a new SRU containing this fix as soon as the current one is done. We will do whatever we can to help expedite that SRU to release this fix.

Nicolas Pochet (npochet) wrote :

@Joshua Thanks a lot for your help!

Joshua Powers (powersj) wrote :

Our first SRU is done, the next SRU is started and it will contain the fix. This can be tracked in the following bug report:

https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1819067

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 18.5-45-g3554ffe8-0ubuntu1

---------------
cloud-init (18.5-45-g3554ffe8-0ubuntu1) disco; urgency=medium

  * New upstream snapshot.
    - cloud-init-per: POSIX sh does not support string subst, use sed
      (LP: #1819222)

 -- Daniel Watkins <email address hidden> Fri, 08 Mar 2019 17:42:34 -0500

Changed in cloud-init (Ubuntu):
status: In Progress → Fix Released
Joshua Powers (powersj) wrote :

SRU is complete, fix is released. Daily images should pick this up on Mar 20 or Mar 21

Joshua Powers (powersj) wrote :

Marking all fixed-released and unsubscribing field-critical.

Changed in cloud-init (Ubuntu Bionic):
status: Confirmed → Fix Released
Changed in cloud-init:
status: Fix Committed → Fix Released
Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Released
Changed in cloud-init (Ubuntu Cosmic):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers