artful network vmtests fail now that resovlconf is missing (and ifupdown is still present)

[Ryan Harper]

https://jenkins.ubuntu.com/server/job/curtin-vmtest-devel-amd64/33/consoleFull

FAIL: test_etc_resolvconf (vmtests.test_network_ipv6.ArtfulTestNetworkIPV6)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/jenkins/slaves/torkoal/workspace/curtin-vmtest-devel-amd64/curtin-33/tests/vmtests/test_network.py", line 232, in test_etc_resolvconf
    self.assertTrue(dns_line in resolv_lines)
AssertionError: False is not true
-------------------- >> begin captured stdout << ---------------------
Network Renderer: ifupdown
k=nameservers v=['72.3.128.240', '72.3.128.241']

I believe this maybe related to Bug 1713803 -> please explain / ellaborate specifically what. As clearly integrating android and sendmail should be irrelevant for curtin/maas operation, no?

Related bugs:
 * bug 1698181: Switch to netplan renderer in Artful -> all done, apart from MAAS task, which is incomplete.
 * bug 1714308: dns does not work in initramfs after configure_networking -> done
 * bug 1714028: artful network vmtests fail now that ifdown command is removed -> this is this bug itself?! is this a typo?

[Ryan Harper (raharper)]

Running an Artful image from 2017-08-30 I observe the following issues:

1) This image contains ifupdown package (which means cloud-init will render eni); that's the default cloud-init policy
2) systemd-networkd-wait-online is enabled in addition to the 'networking' service; this are somewhat equivalent w.r.t waiting till networking is up

Networkd runs (but has no config, so does nothing)
Networking runs (which hooks the ifup commands on interfaces)
dhclient runs and performs dhcp on the specified interfaces
however, the aliased interfaces which include dns-* values
get skipped due to 'resolvconf' package not being present in the image

Systemd-resolved --status output shows that none of the eni related dns-* entries
are present:

Global
         DNS Servers: 10.0.2.3
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

In particular, the eni configuration includes this bit:

# control-alias interface1
iface interface1 inet static
    address 10.0.2.200/24
    dns-nameservers 8.8.8.8
    dns-search barley.maas

which should mean we get 8.8.8.8 in our DNS list, but it's not present.

[Ryan Harper (raharper)]

If I force cloud-init to render netplan (via cloud-config) then things work out better. A small adjustment is needed to the network unittest for parsing ip route output.

=== modified file 'tests/vmtests/test_network.py'
--- tests/vmtests/test_network.py 2017-08-02 15:46:35 +0000
+++ tests/vmtests/test_network.py 2017-08-31 20:42:12 +0000
@@ -30,6 +30,7 @@
         cp -av /etc/network/interfaces .
         cp -av /etc/network/interfaces.d .
         cp /etc/resolv.conf .
+ systemd-resolve --status 2>/dev/null |tee systemd-resolvd.status
         cp -av /etc/udev/rules.d/70-persistent-net.rules . ||:
         ip -o route show | tee ip_route_show
         ip -6 -o route show | tee ip_6_route_show
@@ -48,6 +49,7 @@
         mkdir -p etc_netplan
         cp -av /etc/netplan/* ./etc_netplan/ ||:
         networkctl |tee networkctl
+ ls -1 /sys/class/net/ | xargs -i networkctl status {} | tee nctl.status
         mkdir -p run_systemd_network
         cp -a /run/systemd/network/* ./run_systemd_network/ ||:
         cp -a /run/systemd/netif ./run_systemd_netif ||:
@@ -285,13 +287,16 @@
         ip_route_show = self.load_collect_file("ip_route_show")
         logger.debug("ip route show:\n{}".format(ip_route_show))
         for line in [line for line in ip_route_show.split('\n')
- if 'src' in line]:
+ if 'src' in line and 'default' not in line]:
             m = re.search(r'^(?P<network>\S+)\sdev\s' +
                           r'(?P<devname>\S+)\s+' +
- r'proto kernel\s+scope link' +
+ r'proto\s(?P<proto>\S+)\s+scope link' +
                           r'\s+src\s(?P<src_ip>\S+)',
                           line)
- route_info = m.groupdict('')
+ try:
+ route_info = m.groupdict('')
+ except Exception:
+ print('Failed to match ip_route_show line:\n%s' % line)
             logger.debug(route_info)

[Dimitri John Ledkov (xnox)]

Image 8-30 had re-introduced ifupdown due to $reasons. Thus any image before or after would have been better.

Current daily images do not have neither ifupdown or resolvconf on them.

Are there specific netplan/networkd/resolved bugs i should look into trianging/debugging/fixing to have a working curtin for artful release?

[Dimitri John Ledkov (xnox)]

Updated the description, as it is not clear if there are any blockers for this any more. As the mentioned bug numbers are either wrong, fixed released, or are orthogonal to curtin/maas (not a dep, not a reverse dep)

[Ryan Harper (raharper)]

On Tue, Oct 3, 2017 at 9:48 AM, Dimitri John Ledkov <email address hidden>
wrote:

> Image 8-30 had re-introduced ifupdown due to $reasons. Thus any image
> before or after would have been better.
>

We are currently still failing in maas images:

artful amd64/hwe-a 20170928 root-image.gz

> Current daily images do not have neither ifupdown or resolvconf on them.
>
> Are there specific netplan/networkd/resolved bugs i should look into
> trianging/debugging/fixing to have a working curtin for artful release?
>

Can you confirm when we will have an artful maas image with no ifupdown?
We'll likely see it when it's there at our daily vmtests:

https://jenkins.ubuntu.com/server/view/curtin/job/curtin-vmtest-devel-amd64/

The current image (9/28) still has ifupdown in it.

> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1714028
>
> Title:
> artful network vmtests fail now that resovlconf is missing (and
> ifupdown is still present)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1714028/+subscriptions
>

[Ryan Harper (raharper)]

The artful image is ifupdown free (for some time) the failing cases we've found we're related to a few things:

1) curtin currently installs ifenslave if it gets a network config with bonds in it; this in turn installs ifupdown (it's a proper dependency since it provides ifupdown hooks); however on artful there is no need. Curtin will now filter out ifenslave dep if the target image is artful.

2) serveral curtin tests set IPV6 MTU values, this fails to render in netplan since netplan/networkd dont currently support configuring IPV6 mtu (https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1671951)

3) bridging was failing in artful due to netplan enables STP by default; this prevents setting forwarding delays < 2 seconds (we currently test disabling forwarding delay) so we set STP off and fd of 0; however cloud-init doesn't support rendering netplan stp configuration (https://bugs.launchpad.net/cloud-init/+bug/1721157) so cloud-init will need a fix to render the correct value for bridge stp settings.

[Scott Moser (smoser)]

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.