Track origin file for each individual YAML setting/stanza

[Lukas Märdian]

In order to properly update the YAML hierarchy or (over-)write new settings, e.g. via "netplan set", Netplan needs to know the origin file of each individual setting.

Storing the "filepath" on a per NetDef level is not enough, as a higher level file could modify/extend an existing NetDef, such that its final definition would be generated from two different YAML files.

The "private_data->dirty_fields" data-structure might be extended to support such usecase: https://github.com/canonical/netplan/pull/299#discussion_r1032248428

[Danilo Egea Gondolfo (danilogondolfo)]

"netplan set" is not working properly when you have openvswitch configuration. After running netplan set the resulting configuration will contain many duplicated lines. This was tested after https://github.com/canonical/netplan/pull/299 be merged.

How to reproduce the issue

apt install -y openvswitch-switch

wget https://raw.githubusercontent.com/canonical/netplan/main/examples/openvswitch.yaml -O /etc/netplan/openvswitch.yaml

netplan set network.bonds.bond0.interfaces=null
netplan set network.bonds.bond0.interfaces=[patch1-0]
netplan set network.ethernets.eth1=null

and check the files generated/changed.

In a simpler case where you have the openvswitch.yaml from netplan/examples and the file below, the result when using --origin-hint contains openvswitch configuration.

# cat 50-cloud-init.yaml
network:
  version: 2
  ethernets:
    enp5s0:
      dhcp4: false

root@deleteme-lunar:/etc/netplan# netplan set --origin-hint 90-snapd-config network.ethernets.enp5s0.dhcp4=true

# cat 90-snapd-config.yaml
network:
  version: 2
  ethernets:
    enp5s0:
      dhcp4: true
  openvswitch:
    ports:
    - - patch0-1
      - patch1-0
    external-ids:
      somekey: "somevalue"
...

When not using --origin-hint, the files 50-cloud-init.yaml and 70-netplan-set.yaml will both contain openvswitch configuration.

[Danilo Egea Gondolfo (danilogondolfo)]

Found another issue with the netplan.io currently in Lunar:

# apt policy netplan.io
netplan.io:
  Installed: 0.105-0ubuntu4
  Candidate: 0.105-0ubuntu4

(the same problem happens with 0.106)

After calling "netplan set" with --origin-hint and an existing interface and then calling it again for a non-existing interface, the original file where the first interface was defined is removed (and the renderer is not preserved):

root@lunar:/etc/netplan# ls
0-defaults.yaml

root@lunar:/etc/netplan# cat 0-defaults.yaml
network:
  version: 2
  renderer: NetworkManager
  bridges:
    br54:
      dhcp4: true

root@lunar:/etc/netplan# netplan set --origin-hint 90-snapd-config bridges.br54.dhcp4=false
root@lunar:/etc/netplan# ls
0-defaults.yaml 90-snapd-config.yaml

root@lunar:/etc/netplan# cat 0-defaults.yaml
network:
  version: 2
  renderer: NetworkManager
  bridges:
    br54:
      dhcp4: true

root@lunar:/etc/netplan# cat 90-snapd-config.yaml
network:
  version: 2
  bridges:
    br54:
      dhcp4: false

root@lunar:/etc/netplan# netplan set ethernets.eth2.dhcp4=false
root@deleteme-lunar:/etc/netplan# ls
70-netplan-set.yaml 90-snapd-config.yaml

root@deleteme-lunar:/etc/netplan# cat 70-netplan-set.yaml
network:
  version: 2
  ethernets:
    eth2:
      dhcp4: false
root@deleteme-lunar:/etc/netplan# cat 90-snapd-config.yaml
network:
  version: 2
  bridges:
    br54:
      dhcp4: false

The same steps on Focal (0.104-0ubuntu2~20.04.2) seem to work fine, except the renderer from 0-defaults.yaml is not preserved in the origin-hint file.

[Danilo Egea Gondolfo (danilogondolfo)]

One related to the renderer now

I have the files below:

root@lunar:/etc/netplan# cat 0-defaults.yaml
network:
  version: 2
  renderer: NetworkManager
  bridges:
    br54:
      dhcp4: false

root@lunar:/etc/netplan# cat 90-snapd-config.yaml
network:
  version: 2
  bridges:
    br54:
      dhcp4: true

root@lunar:/etc/netplan# cat main.yaml
network:
  ethernets:
    enp5s0:
      dhcp4: true

Netplan generate will emit all the configuration for NetworkManager even though I don't have the renderer defined in the last 2 files, which should default to networkd.

# ls /run/NetworkManager/system-connections/
netplan-br54.nmconnection netplan-enp5s0.nmconnection

[Danilo Egea Gondolfo (danilogondolfo)]

Apparently the renderer is still being dropped from the origin-hint file

root@lunar:/etc/netplan# ls
0-defaults.yaml

root@lunar:/etc/netplan# cat 0-defaults.yaml
network:
  version: 2
  renderer: NetworkManager
  bridges:
    br54:
      dhcp4: false

root@lunar:/etc/netplan# netplan set --origin-hint 90-snapd-config bridges.br54.dhcp4=true

root@lunar:/etc/netplan# cat 90-snapd-config.yaml
network:
  version: 2
  bridges:
    br54:
      dhcp4: true

[Lukas Märdian (slyon)]

Wrt. comment #4:
> Apparently the renderer is still being dropped from the origin-hint file

This is the correct and expected behavior. We don't want the global "renderer: NetworkManager" stanza from 0-defaults.yaml to leak into 90-snapd-config.yaml, if not defined explicitly in the "netplan set" command. This global stanza applies to the full Netplan configuration, not just the given YAML file. As visible from the "netplan get" command.

I think the same is true for comment #3 (a non-issue): The global renderer stanza is set to "NetworkManager" so any configuration should be generated on the NM backend, if not overwritten on a per netdef basis in other files. As visible from the state that "netplan get" prints.