MAAS

Bug #1879933
Comment #2

Comment 2 for bug 1879933

Revision history for this message

Ryan Harper (raharper) wrote on 2020-05-21:

> netplan is non-blocking and I've seen cloud-init output incomplete network information even though netplan hasn't finished applying network config

cloud-init calls netplan generate which reads the config passed in from MAAS, and writes out all of the networkd files per the config; this happens before network-online.target is reached, so systemd-networkd runs and cloud-init will not proceed until systemd-networkd-wait-online.service is complete;

systemd-networkd-wait-online.service will wait for all interfaces which have configuration on them.

From the config posted, there's not config for eno1, so this appears to be a
output from one config and input from a different system. can you provide
the failing out, and the /etc/netplan/50-cloud-init.yaml and
/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg files?

> cloud-init log on the machine does not have a default gateway

0 | 0.0.0.0 | 10.245.168.1 | 0.0.0.0 | eno1 | UG |

Is this not the default gateway?

And lastly, if your config is using non-standard routing tables like the
paste you supplied, ip route will only show routes in the default table,
and the default route appears to be in table 1.

routes:
  - table: 1
  to: 0.0.0.0/0
  via: 10.245.168.1

I took the config from your paste andput it in a container, then ran netplan apply

root@g1:~# netplan --debug apply
** (generate:5092): DEBUG: 21:55:25.895: Processing input file /etc/netplan/50-cloud-init.yaml..
** (generate:5092): DEBUG: 21:55:25.895: starting new processing pass
** (generate:5092): DEBUG: 21:55:25.895: We have some netdefs, pass them through a final round of validation
** (generate:5092): DEBUG: 21:55:25.895: eth0: setting default backend to 1
** (generate:5092): DEBUG: 21:55:25.895: Configuration is valid
** (generate:5092): DEBUG: 21:55:25.895: Generating output files..
** (generate:5092): DEBUG: 21:55:25.895: NetworkManager: definition eth0 is not for us (backend 1)
(generate:5092): GLib-DEBUG: 21:55:25.895: posix_spawn avoided (fd close requested)
DEBUG:netplan generated networkd configuration changed, restarting networkd
DEBUG:no netplan generated NM configuration exists
DEBUG:eth0 not found in {}
DEBUG:Merged config:
network:
  bonds: {}
  bridges: {}
  ethernets:
    eth0:
      addresses:
      - 10.245.168.63/21
      match:
        macaddress: 00:16:3e:39:6c:f7
      mtu: 1500
      routes:
      - table: 1
        to: 0.0.0.0/0
        via: 10.245.168.1
      routing-policy:
      - from: 10.245.168.0/21
        priority: 100
        table: 1
      - from: 10.245.168.0/21
        table: 254
        to: 10.245.168.0/21
  vlans: {}
  wifis: {}

DEBUG:Skipping non-physical interface: lo
DEBUG:{}
DEBUG:netplan triggering .link rules for lo
DEBUG:netplan triggering .link rules for eth0
root@g1:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:39:6c:f7 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.245.168.63/21 brd 10.245.175.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe39:6cf7/64 scope link tentative
       valid_lft forever preferred_lft forever
root@g1:~# ip route
10.245.168.0/21 dev eth0 proto kernel scope link src 10.245.168.63

root@g1:~# ip route show table 1
default via 10.245.168.1 dev eth0 proto static

root@g1:~# ip route show table 254
10.245.168.0/21 dev eth0 proto kernel scope link src 10.245.168.63

And to replicate the cloud-init output:

root@g1:~# python3 -c 'import sys; from cloudinit import netinfo; sys.stderr.write("%s\n" % (netinfo.debug_info()))'
ci-info: +++++++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-------------------------------------------+---------------+--------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-------------------------------------------+---------------+--------+-------------------+
ci-info: | eth0 | True | 10.245.168.63 | 255.255.248.0 | global | 00:16:3e:39:6c:f7 |
ci-info: | eth0 | True | fd42:f890:56f5:dcfb:216:3eff:fe39:6cf7/64 | . | global | 00:16:3e:39:6c:f7 |
ci-info: | eth0 | True | fe80::216:3eff:fe39:6cf7/64 | . | link | 00:16:3e:39:6c:f7 |
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: +--------+------+-------------------------------------------+---------------+--------+-------------------+
ci-info: +++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++
ci-info: +-------+--------------+---------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+--------------+---------+---------------+-----------+-------+
ci-info: | 0 | 10.245.168.0 | 0.0.0.0 | 255.255.248.0 | eth0 | U |
ci-info: +-------+--------------+---------+---------------+-----------+-------+
ci-info: ++++++++++++++++++++++++++++++++++Route IPv6 info+++++++++++++++++++++++++++++++++++
ci-info: +-------+--------------------------+---------------------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+--------------------------+---------------------------+-----------+-------+
ci-info: | 0 | fd42:f890:56f5:dcfb::/64 | :: | eth0 | Ue |
ci-info: | 1 | fe80::/64 | :: | eth0 | U |
ci-info: | 2 | ::/0 | fe80::b42b:1cff:fed1:3998 | eth0 | UGe |
ci-info: | 4 | local | :: | eth0 | U |
ci-info: | 5 | local | :: | eth0 | U |
ci-info: | 6 | ff00::/8 | :: | eth0 | U |
ci-info: +-------+--------------------------+---------------------------+-----------+-------+

I think this matches up to your output, one does not "see" the default route
as it's in table 1. So this is expected behavior as cloud-init's netinfo
dumps information from 'ip route' output.

> netplan is non-blocking and I've seen cloud-init output incomplete network information even though netplan hasn't finished applying network config

systemd-networkd-wait-online.service will wait for all interfaces which have configuration on them.

From the config posted, there's not config for eno1, so this appears to be a
output from one config and input from a different system.  can you provide
the failing out, and the /etc/netplan/50-cloud-init.yaml and
/etc/cloud/cloud.cfg.d/50-curtin-networking.cfg files?

> cloud-init log on the machine does not have a default gateway

0   |   0.0.0.0    | 10.245.168.1 |    0.0.0.0    |    eno1   |   UG  |

Is this not the default gateway?

And lastly, if your config is using non-standard routing tables like the
paste you supplied, ip route will only show routes in the default table,
and the default route appears to be in table 1.

routes:
  - table: 1
  to: 0.0.0.0/0
  via: 10.245.168.1

I took the config from your paste andput it in a container, then ran netplan apply

root@g1:~# ip route show table 1
default via 10.245.168.1 dev eth0 proto static

root@g1:~# ip route show table 254
10.245.168.0/21 dev eth0 proto kernel scope link src 10.245.168.63

And to replicate the cloud-init output:

root@g1:~# python3 -c 'import sys; from cloudinit import netinfo; sys.stderr.write("%s\n" % (netinfo.debug_info()))'
ci-info: +++++++++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-------------------------------------------+---------------+--------+-------------------+
ci-info: | Device |  Up  |                  Address                  |      Mask     | Scope  |     Hw-Address    |
ci-info: +--------+------+-------------------------------------------+---------------+--------+-------------------+
ci-info: |  eth0  | True |               10.245.168.63               | 255.255.248.0 | global | 00:16:3e:39:6c:f7 |
ci-info: |  eth0  | True | fd42:f890:56f5:dcfb:216:3eff:fe39:6cf7/64 |       .       | global | 00:16:3e:39:6c:f7 |
ci-info: |  eth0  | True |        fe80::216:3eff:fe39:6cf7/64        |       .       |  link  | 00:16:3e:39:6c:f7 |
ci-info: |   lo   | True |                 127.0.0.1                 |   255.0.0.0   |  host  |         .         |
ci-info: |   lo   | True |                  ::1/128                  |       .       |  host  |         .         |
ci-info: +--------+------+-------------------------------------------+---------------+--------+-------------------+
ci-info: +++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++
ci-info: +-------+--------------+---------+---------------+-----------+-------+
ci-info: | Route | Destination  | Gateway |    Genmask    | Interface | Flags |
ci-info: +-------+--------------+---------+---------------+-----------+-------+
ci-info: |   0   | 10.245.168.0 | 0.0.0.0 | 255.255.248.0 |    eth0   |   U   |
ci-info: +-------+--------------+---------+---------------+-----------+-------+
ci-info: ++++++++++++++++++++++++++++++++++Route IPv6 info+++++++++++++++++++++++++++++++++++
ci-info: +-------+--------------------------+---------------------------+-----------+-------+
ci-info: | Route |       Destination        |          Gateway          | Interface | Flags |
ci-info: +-------+--------------------------+---------------------------+-----------+-------+
ci-info: |   0   | fd42:f890:56f5:dcfb::/64 |             ::            |    eth0   |   Ue  |
ci-info: |   1   |        fe80::/64         |             ::            |    eth0   |   U   |
ci-info: |   2   |           ::/0           | fe80::b42b:1cff:fed1:3998 |    eth0   |  UGe  |
ci-info: |   4   |          local           |             ::            |    eth0   |   U   |
ci-info: |   5   |          local           |             ::            |    eth0   |   U   |
ci-info: |   6   |         ff00::/8         |             ::            |    eth0   |   U   |
ci-info: +-------+--------------------------+---------------------------+-----------+-------+

I think this matches up to your output, one does not "see" the default route
as it's in table 1.  So this is expected behavior as cloud-init's netinfo
dumps information from 'ip route' output.