trusty ifupdown fails to handle new ENIs

[Ross Vandegrift]

I am working on an HA application which relies on dynamically adding a secondary ENI to trusty instances. For some reason (timing?) trusty cannot reliably activate the secondary ENI when it is added.

I can reproduce this almost all the time - but not 100%. Steps:

1) Launch a trusty instance in a VPC. I used ami-fce3c696 in us-east-1 on t2.micro.

2) Populate /etc/network/interfaces.d/eth1.cfg:
allow-hotplug eth1
iface eth1 inet dhcp

2) Create a new ENI in the same availability zone as the instance.

3) Attach the ENI to the instance.

4) dhclient has been started before link is up:
$ ps ax | grep dhclient
  575 ? Ss 0:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
 1401 ? S 0:00 /bin/sh -c dhclient -1 -v -pf /run/dhclient.eth1.pid -lf /var/lib/dhcp/dhclient.eth1.leases eth1 ?
 1402 ? S 0:00 dhclient -1 -v -pf /run/dhclient.eth1.pid -lf /var/lib/dhcp/dhclient.eth1.leases eth1
$ ip link list dev eth1
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 0e:8d:96:a5:5e:47 brd ff:ff:ff:ff:ff:ff

DHCP won't complete in this state. From syslog:
Apr 18 14:47:15 ip-10-209-32-175 dhclient: Internet Systems Consortium DHCP Client 4.2.4
Apr 18 14:47:15 ip-10-209-32-175 dhclient: Copyright 2004-2012 Internet Systems Consortium.
Apr 18 14:47:15 ip-10-209-32-175 dhclient: All rights reserved.
Apr 18 14:47:15 ip-10-209-32-175 dhclient: For info, please visit https://www.isc.org/software/dhcp/
Apr 18 14:47:15 ip-10-209-32-175 dhclient:
Apr 18 14:47:15 ip-10-209-32-175 dhclient: Listening on LPF/eth1/00:00:00:00:00:00
Apr 18 14:47:15 ip-10-209-32-175 dhclient: Sending on LPF/eth1/00:00:00:00:00:00
Apr 18 14:47:15 ip-10-209-32-175 dhclient: Sending on Socket/fallback
Apr 18 14:47:15 ip-10-209-32-175 dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 3 (xid=0x38f03f7c)
Apr 18 14:47:15 ip-10-209-32-175 dhclient: send_packet: Network is down
Apr 18 14:47:15 ip-10-209-32-175 dhclient: dhclient.c:2038: Failed to send 300 byte long packet over eth1 interface.
Apr 18 14:47:15 ip-10-209-32-175 dhclient: receive_packet failed on eth1: Network is down

5) Forcing the link up does not fix the problem, syslog messages change:
$ sudo ip link eth1 link up
$ ip addr list dev eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0e:8d:96:a5:5e:47 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c8d:96ff:fea5:5e47/64 scope link
       valid_lft forever preferred_lft forever
Apr 18 14:50:18 ip-10-209-32-175 dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 15 (xid=0x38f03f7c)
Apr 18 14:50:33 ip-10-209-32-175 dhclient: DHCPDISCOVER on eth1 to 255.255.255.255 port 67 interval 11 (xid=0x38f03f7c)

6) Bouncing the interface with ifupdown does work:
$ sudo ifdown eth1; sudo ifup eth1
...
$ ip addr list dev eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0e:8d:96:a5:5e:47 brd ff:ff:ff:ff:ff:ff
    inet 10.209.32.249/25 brd 10.209.32.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::c8d:96ff:fea5:5e47/64 scope link
       valid_lft forever preferred_lft forever

Other info:
ubuntu@ip-10-209-32-175:/etc/network/interfaces.d$ lsb_release -d
Description: Ubuntu 14.04.3 LTS
ubuntu@ip-10-209-32-175:/etc/network/interfaces.d$ uname -a
Linux ip-10-209-32-175 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux