2.3 - Network preseed misconfigures static routes

Bug #1752332 reported by Scott Hussey
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Triaged
High
Unassigned

Bug Description

In 2.3, with cloud-init configuring networking, the ENI renderer produces a valid, but suspect interfaces file when using static routes. MaaS is producing a network config that puts all the static routes in the 'network.routes' stanza instead of including each route in the 'subnets.[].routes' section for the interface that would be the source for that route.

The static routes are translated into post-up/pre-down that appear at the tail end of the interfaces file. ifup interprets this by attaching these post-up/pre-down actions to the last interface. We have seen a lot of indeterminate behavior with routes not getting configured becauses the last interface defined in the interfaces will come up before the source interface for some of the routes. The particular configuration this has shown up on is a bonded interface with multiple VLAN tagged subinterfaces on it. Each subinterface has one or more static routes, but all the route statements are attributed to the last VLAN tagged interface. Deployed nodes on reboot can have a random set of the routes not configured with the fix being to ifdown/ifup the last defined VLAN tagged interface.

Attached is a patch we are using to correct this until it is fixed in the upstream MaaS. It is rough and is likely not a final fix.

diff --git a/src/maasserver/preseed_network.py b/src/maasserver/preseed_network.py
index 1be72d6..96ee6a3 100644
--- a/src/maasserver/preseed_network.py
+++ b/src/maasserver/preseed_network.py
@@ -192,7 +192,7 @@ class InterfaceConfiguration:
         return {
             route
             for route in self.routes
- if route.source == source
+ if str(route.source.cidr) == str(source.cidr)
         }

     def _generate_addresses(self, version=1):
@@ -251,8 +251,12 @@ class InterfaceConfiguration:
                                 v2_nameservers["addresses"] = []
                         v2_nameservers["addresses"].extend(
                             [server for server in subnet.dns_servers])
- self.matching_routes.update(
- self._get_matching_routes(subnet))
+ net_routes = self._get_matching_routes(subnet)
+ rl = [_generate_route_operation(r, version=version)
+ for r
+ in net_routes]
+ v1_subnet_operation['routes'] = rl
+ v2_config['routes'] = rl
             if dhcp_type:
                 v1_config.append(
                     {"type": dhcp_type}

Revision history for this message
Scott Hussey (sh8121) wrote :
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi Scott,

MAAS doesn’t pass network configuration to cloud-init. MAAS passes the network configuration to Curtin. Curtin decided to pass through this configuration to cloud-init instead of doing it in Curtin itself.

If cloud-init is not interpreting the configuration correctly as expected by Curtin, then this is a bug in cloud-init. Could you please attach the present that’s been sent so we can triage debug this issue.

Changed in maas:
status: New → Incomplete
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Btw, you can get the preseed with “maas <user> machine get-curtin-config <systemd_id>

Feel free to attach the relevant information.

Revision history for this message
Scott Hussey (sh8121) wrote :
Download full text (5.7 KiB)

cloud-init is interpreting it fine. The issue is the configuration generated by MaaS is really not rendered correctly to get a stable network configuration on a deployed node.

Stock MaaS 2.3:
Success.
Machine-readable output follows:
apt:
  preserve_sources_list: false
  primary:
  - arches:
    - default
    uri: http://archive.ubuntu.com/ubuntu
  security:
  - arches:
    - default
    uri: http://archive.ubuntu.com/ubuntu
cloudconfig:
  maas-cloud-config:
    content: "#cloud-config\ndatasource:\n MAAS: {consumer_key: rzjLYaaUtzNnNb8kFT,\
      \ metadata_url: 'http://172.24.1.100:31900/MAAS/metadata/',\n token_key:\
      \ HgLwAXPcmyUzQQpqsj, token_secret: defLwJ5V63EXwK5TEU2W6h3C4aGgk8Ls}\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_config.cfg
  maas-datasource:
    content: 'datasource_list: [ MAAS ]'
    path: /etc/cloud/cloud.cfg.d/90_maas_datasource.cfg
  maas-reporting:
    content: "#cloud-config\nreporting:\n maas: {consumer_key: rzjLYaaUtzNnNb8kFT,\
      \ endpoint: 'http://172.24.1.100:31900/MAAS/metadata/status/fndqc4',\n token_key:\
      \ HgLwAXPcmyUzQQpqsj, token_secret: defLwJ5V63EXwK5TEU2W6h3C4aGgk8Ls,\n type:\
      \ webhook}\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_init_reporting.cfg
  maas-ubuntu-sso:
    content: '#cloud-config

      snappy: {email: none@none}

      '
    path: /etc/cloud/cloud.cfg.d/90_maas_ubuntu_sso.cfg
debconf_selections:
  grub2: grub2 grub2/update_nvram boolean false
  maas: 'cloud-init cloud-init/datasources multiselect MAAS

    cloud-init cloud-init/maas-metadata-url string http://172.24.1.100:31900/MAAS/metadata/

    cloud-init cloud-init/maas-metadata-credentials string oauth_token_key=HgLwAXPcmyUzQQpqsj&oauth_token_secret=defLwJ5V63EXwK5TEU2W6h3C4aGgk8Ls&oauth_consumer_key=rzjLYaaUtzNnNb8kFT

    cloud-init cloud-init/local-cloud-config string apt:\n preserve_sources_list:
    false\n primary:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\n security:\n -
    arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\napt_preserve_sources_list:
    true\nmanage_etc_hosts: false\nmanual_cache_clean: true\nreporting:\n maas: {consumer_key:
    rzjLYaaUtzNnNb8kFT, endpoint: ''http://172.24.1.100:31900/MAAS/metadata/status/fndqc4'',\n token_key:
    HgLwAXPcmyUzQQpqsj, token_secret: defLwJ5V63EXwK5TEU2W6h3C4aGgk8Ls,\n type:
    webhook}\nsystem_info:\n package_mirrors:\n - arches: [i386, amd64]\n failsafe:
    {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''http://security.ubuntu.com/ubuntu''}\n search:\n primary:
    [''http://archive.ubuntu.com/ubuntu'']\n security: [''http://archive.ubuntu.com/ubuntu'']\n -
    arches: [default]\n failsafe: {primary: ''http://ports.ubuntu.com/ubuntu-ports'',
    security: ''http://ports.ubuntu.com/ubuntu-ports''}\n search:\n primary:
    [''http://ports.ubuntu.com/ubuntu-ports'']\n security: [''http://ports.ubuntu.com/ubuntu-ports'']\n

    '
early_commands:
  driver_00:
  - sh
  - -c
  - echo third party drivers not installed or necessary.
install:
  log_file: /tmp/install.log
  post_files:
  - /tmp/install.log
kernel:
  mapping: {}...

Read more...

Revision history for this message
Scott Hussey (sh8121) wrote :
Download full text (5.5 KiB)

With the patch applied:
apt:
  preserve_sources_list: false
  primary:
  - arches:
    - default
    uri: http://archive.ubuntu.com/ubuntu
  security:
  - arches:
    - default
    uri: http://archive.ubuntu.com/ubuntu
cloudconfig:
  maas-cloud-config:
    content: "#cloud-config\ndatasource:\n MAAS: {consumer_key: rzjLYaaUtzNnNb8kFT,\
      \ metadata_url: 'http://172.24.1.100:31900/MAAS/metadata/',\n token_key:\
      \ HgLwAXPcmyUzQQpqsj, token_secret: defLwJ5V63EXwK5TEU2W6h3C4aGgk8Ls}\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_config.cfg
  maas-datasource:
    content: 'datasource_list: [ MAAS ]'
    path: /etc/cloud/cloud.cfg.d/90_maas_datasource.cfg
  maas-reporting:
    content: "#cloud-config\nreporting:\n maas: {consumer_key: rzjLYaaUtzNnNb8kFT,\
      \ endpoint: 'http://172.24.1.100:31900/MAAS/metadata/status/fndqc4',\n token_key:\
      \ HgLwAXPcmyUzQQpqsj, token_secret: defLwJ5V63EXwK5TEU2W6h3C4aGgk8Ls,\n type:\
      \ webhook}\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_init_reporting.cfg
  maas-ubuntu-sso:
    content: '#cloud-config

      snappy: {email: none@none}

      '
    path: /etc/cloud/cloud.cfg.d/90_maas_ubuntu_sso.cfg
debconf_selections:
  grub2: grub2 grub2/update_nvram boolean false
  maas: 'cloud-init cloud-init/datasources multiselect MAAS

    cloud-init cloud-init/maas-metadata-url string http://172.24.1.100:31900/MAAS/metadata/

    cloud-init cloud-init/maas-metadata-credentials string oauth_token_key=HgLwAXPcmyUzQQpqsj&oauth_consumer_key=rzjLYaaUtzNnNb8kFT&oauth_token_secret=defLwJ5V63EXwK5TEU2W6h3C4aGgk8Ls

    cloud-init cloud-init/local-cloud-config string apt:\n preserve_sources_list:
    false\n primary:\n - arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\n security:\n -
    arches: [default]\n uri: http://archive.ubuntu.com/ubuntu\napt_preserve_sources_list:
    true\nmanage_etc_hosts: false\nmanual_cache_clean: true\nreporting:\n maas: {consumer_key:
    rzjLYaaUtzNnNb8kFT, endpoint: ''http://172.24.1.100:31900/MAAS/metadata/status/fndqc4'',\n token_key:
    HgLwAXPcmyUzQQpqsj, token_secret: defLwJ5V63EXwK5TEU2W6h3C4aGgk8Ls,\n type:
    webhook}\nsystem_info:\n package_mirrors:\n - arches: [i386, amd64]\n failsafe:
    {primary: ''http://archive.ubuntu.com/ubuntu'', security: ''http://security.ubuntu.com/ubuntu''}\n search:\n primary:
    [''http://archive.ubuntu.com/ubuntu'']\n security: [''http://archive.ubuntu.com/ubuntu'']\n -
    arches: [default]\n failsafe: {primary: ''http://ports.ubuntu.com/ubuntu-ports'',
    security: ''http://ports.ubuntu.com/ubuntu-ports''}\n search:\n primary:
    [''http://ports.ubuntu.com/ubuntu-ports'']\n security: [''http://ports.ubuntu.com/ubuntu-ports'']\n

    '
early_commands:
  driver_00:
  - sh
  - -c
  - echo third party drivers not installed or necessary.
install:
  log_file: /tmp/install.log
  post_files:
  - /tmp/install.log
kernel:
  mapping: {}
  package: linux-signed-generic-hwe-16.04
late_commands:
  maas:
  - wget
  - --no-proxy
  - http://172.24.1.100:31900/MAAS/metadata/latest/by-id/fndqc4/
  - --post-data
  - op=netboot_off
  - -O
  - /dev/null
...

Read more...

Revision history for this message
Scott Hussey (sh8121) wrote :

Contrasting the above you can see that in stock MaaS the bond0 definition has no routes defined in its subnets section. The static route is instead at the top-level of the network configuration and not tied to its source subnet. This ends up with the cloud-init ENI renderer adding the post-up/pre-down commands to the end of the interfaces file.

With the patch, the static route is instead included in the bond0 subnets section and so tied to the interface that must be up for the route to have a valid next-hop. This ends up with the cloud-init ENI renderer adding the post-up/pre-down commands to the stanza for the interface.

Revision history for this message
Mike Pontillo (mpontillo) wrote :

When we originally implemented the v1 YAML, routes were not allowed to be associated with interfaces. I think this is a good change, but we'll need to make sure we depend on a version of curtin that supports it. (It isn't clear from the changelog when curtin gained that support.)

Changed in maas:
status: Incomplete → Triaged
importance: Undecided → High
Revision history for this message
Mike Pontillo (mpontillo) wrote :

Talking to Ryan Harper on IRC, It looks like the current version of curtin that MAAS depends on allows per-interface routes. So it should be safe to make this change.

Revision history for this message
Scott Hussey (sh8121) wrote :

How is this a duplicate of a bug in Launchpad?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.