Order in which files are parsed and interfaces dependencies

[Danilo Egea Gondolfo]

When parsing a file, if netplan finds an interface A that depends on an interface B (for example a bond that depends on the interfaces found in the "interfaces" list), it will look for B in the table of parsed definitions. If it can't find it there, B will be added to the missing_id table and the parser will keep looking for it while the file is parsed.

When it finishes parsing all the file and there are still missing IDs, it will fail with "interface B is not defined".

Consider the example below:

file 10-vlan.yaml:
---
network:
  renderer: NetworkManager
  vlans:
    vlan100:
      id: 100
      link: "eth0"
---

file 99-zzz.yaml:
---
network:
  renderer: NetworkManager
  ethernets:
    eth0: {}
---

In this case, the VLAN "vlan100" depends on the definition of "eth0". "eth0" is defined in a separate YAML.
But because we sort the files in ASCIIbetical order, the file "10-vlan.yaml" will be parsed first and "99-zzz.yaml" will be parsed after that.

The netplan's parser controls the missing IDs on a per-file basis and resets it before parsing the next file. In the end, if "eth0" is not already in the parsed definitions table OR in the file that's being parsed at the moment, it will not bother looking for it in the next files and will fail with:

/etc/netplan/10-vlan.yaml:6:13: Error in network definition: vlan100: interface 'eth0' is not defined
      link: "eth0"
            ^

Renaming the file "99-zzz.yaml" to "01-zzz.yaml" is enough to get netplan working for this example.

Although, when we create netplan YAML configuration through our integration with Network Manager, we don't have control of the file names, they will be called 90-NM-<connection UUID>.yaml. So the order in which YAML will parsed is unpredictable. But it's not a big problem right now because we create standalone YAML from Network Manager, the things that would create a dependency issue are injected in the passthrough section of the YAML so we don't bother tying them together in the netplan state.

But if we want to start creating better netplan YAML from NM, we'll need to address this issue.

One solution for that is to make the missing_ids table survive across different files and keep looking for the missing interfaces until we don't have any more files to look at. Only then we can say the interface B is nowhere to be found in the netplan configuration.

Possible solution

The missing IDs handling could be done automatically if the caller is using the "netplan_parser_load_yaml_hierarchy()" API. So the logic implement in "process_document()" could be moved there. This would make the parser use the missing IDs data for every file it's parsing.

One problem with that approach is that we destroy the YAML document after parsing it in "_netplan_parser_load_single_file()", and the missing ID object has a reference to the YAML node where the ID was referenced.

If the consumer is calling "netplan_parser_load_yaml()" for files individually, they will be responsible for checking if the missing_ids table has any item when they finish parsing their files.

Reasons for fixing it

Netplan users will not have to bother anymore about where they should defined their interfaces.

It will enable us to do a better job when loading NM keyfiles into Netplan's state. Due to this issue, we can't link interface inside netplan, because if we do that, and because of the order netplan will parse NM-generated files with their random names, eventually it will fail to find dependencies.