vlan configuration/unconfigured interfaces creates slow boot time

[James Page]

maas: 1.9.1+bzr4543-0ubuntu1~trusty1 (from proposed PPA)

Deploying juju bootstrap node on Ubuntu 14.04 with the following network configuration:

eth0
    static assigned IP address, default VLAN (no trunking)

eth1
   static assigned IP address, secondary VLAN

   eth1.2667
       static assigned IP address, VLAN 2667

   eth1.2668
       static assigned IP address, VLAN 2668

   eth1.2669
       static assigned IP address, VLAN 2669

   eth1.2670
       static assigned IP address, VLAN 2670

eth2
  unconfigured

eth3
  unconfigured

MAAS generates a /e/n/i which auto stanzas for the VLAN devices and the unconfigured network interfaces; the upstart process which checks that network configuration is complete waits for /var/run/ifup.XXXX to exists for all auto interfaces; these will never appear for either the VLAN interfaces or the unconfigured network interfaces.

As a result, boot time if very long as cloud-init and networking both take 2 minutes to timeout waiting for network interfaces that will never appear to be configured.

[James Page (james-page)]

Correction; auto is required on the VLAN interfaces.

[James Page (james-page)]

That said, the update if-up.d script still waits for an if-up.XXX file that will never exist for a VLAN interface.

[James Page (james-page)]

/etc/network/if-up.d/upstart is the script that waits for all interfaces to be configured, and then emits the static-network-up event.

[Blake Rouse (blake-rouse)]

Can you provide the created '/etc/network/interface' along with "maas [session] node get-curtin-config [system-id]
" and "maas [session] interfaces read [system-id]".

[Blake Rouse (blake-rouse)]

This might also be related to curtin and cloud-init. Targeting them as well.

[James Page (james-page)]

Blake

requested information added to bug - setting back to New.

[Blake Rouse (blake-rouse)]

So the configuration that MAAS emits and curtin generates looks correct. Cloud-init just waits for the signal so I actually think the issue is with ifupdown. I have targeted that package as well, will leave the others for now just to track.

[Ryan Harper (raharper)]

I can recreate this issue with multiple vlan ifaces over eth2 in a guest.

[Ryan Harper (raharper)]

I concur with the ifupdown issue.

I copied out /run/network from the VM that had very slow networking start (timeout on base vlan device, eth2, and both unconfigured "manual" ifaces).

drwxr-xr-x 3 rharper rharper 4096 Apr 5 13:17 ../
-rw-r--r-- 1 rharper rharper 88 Apr 5 13:13 dynamic-interfaces
-rw-r--r-- 1 rharper rharper 96 Apr 5 13:15 ifstate
-rw-r--r-- 1 rharper rharper 5 Apr 5 13:13 ifstate.eth0
-rw-r--r-- 1 rharper rharper 1 Apr 5 13:15 ifstate.eth1
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifstate.eth2
-rw-r--r-- 1 rharper rharper 10 Apr 5 13:13 ifstate.eth2.2667
-rw-r--r-- 1 rharper rharper 10 Apr 5 13:13 ifstate.eth2.2668
-rw-r--r-- 1 rharper rharper 10 Apr 5 13:13 ifstate.eth2.2669
-rw-r--r-- 1 rharper rharper 10 Apr 5 13:13 ifstate.eth2.2670
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifstate.eth3
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifstate.eth4
-rw-r--r-- 1 rharper rharper 3 Apr 5 13:13 ifstate.lo
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 .ifstate.lock
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth0
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth2.2667
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth2.2668
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth2.2669
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.eth2.2670
-rw-r--r-- 1 rharper rharper 0 Apr 5 13:13 ifup.lo

Note, this same configuration (eni with vlans) passes under Xenial.

[Ryan Harper (raharper)]

It also passes on Vivid and Wily, so looking at changes between ifupdown-0.7.47.2ubuntu4.4 (trusty-updates) and ifupdown-0.7.48.1ubuntu10 (vivid). These looks particularly relevant:

+ifupdown (0.7.48.1ubuntu1) utopic; urgency=medium
+
+ [ Stéphane Graber ]
+ * Merge from Debian. Remaining changes:

+ - Allow setting the MTU and HWADDR on manual interfaces. (LP: #1294807)
+ - The above change also means that manual interfaces will now be
+ brought up and down (ias many users expected until now).
+ - Disable link.defn as it's not covering all the cases supported by
+ the vlan and bridge hooks and so causes more harm than good at this
+ point. (LP: #1295304)

[Ryan Harper (raharper)]

Just as a test, using 0.7.48-1ubuntu10 from vivid works fine. Now to narrow down changes.

[Ryan Harper (raharper)]

Unfortunately it's racy; I've had the exact config attached succeed and fail on trusty.

[Ryan Harper (raharper)]

And now, I can't get it to trigger at all. 300 runs last night with out triggering the fault.

James, is it reproducible for you on physical hardware?

[Ryan Harper (raharper)]

I don't think there is a curtin issue here but we may figure out a way to mitigate this issue if we can't figure out if there's a clear bug in ifupdown.

[Andres Rodriguez (andreserl)]

We believe that this is not longer an issue in the latest releases of MAAS. If you believe this is still an issue, please re-open this bug report and target it accordingly.

[Dan Watkins (oddbloke)]

Similarly, please re-open the cloud-init task if you think anything is required from us.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2639