Multiple Interfaces and IPs not detected in AWS VPC

Bug #1153626 reported by Michael Hale on 2013-03-11
98
This bug affects 16 people
Affects Status Importance Assigned to Milestone
cloud-init
Medium
Unassigned
cloud-init (Ubuntu)
Medium
Unassigned

Bug Description

I booted a c1.medium with ami-3fec7956, 2 interfaces and 2 IPs per interface. I expected the 4 IPs to be configured on the instance when I logged in, however only the primary interface had a single IP assigned.

DHCP does not seem to support multiple IPs per interface so the alias IPs will probably need to be statically assigned based on the EC2 meta-data API (http://169.254.169.254/latest/meta-data/).

Additionally since network interfaces, and their IPs, can be moved between running instances in VPC, any solution should detect hot-plug events and not just setup things at boot time.

Scott Moser (smoser) wrote :

Thanks for taking the time to open a bug and make ubuntu better.

When multiple interfaces and IPs first came out, I looked at this, but yours is the first request for this.

I agree, I'd like to have a udev hook that would fire on interface adaded and then determine if it was in EC2, and if so, hit the EC2 metadata service and configure the device. That'd be smooth.

I'm more than happy to review and help out, but its not something that will be high priority on my plate at the moment.

Changed in cloud-init (Ubuntu):
importance: Undecided → Medium
status: New → Triaged
tags: added: cloud-images
Michael Hale (mikehale) wrote :

Scott, thanks for the reply. This is not something I need at the moment, but it does seem like a good way for me to perhaps learn more about Ubuntu network configuration and udev rules. I did a bit of digging and found the following files which seem relevant:

/lib/udev/write_net_rules
/lib/udev/rules.d/75-net-description.rules
/lib/udev/rules.d/75-persistent-net-generator.rules
/etc/udev/rules.d/70-persistent-net.rules

Are there any other starting points I should be aware of?

Michael Hale (mikehale) wrote :

My current thinking is that a udev rule will need to have a RUN hook that updates /etc/network/interfaces or at a minimum runs the appropriate ip commands based on what is discovered in the EC2 metadata service. It seems like updating /etc/network/interfaces would be good since ifup/ifdown rely on its configuration. According to the VPC docs (http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#detach_eni) it appears that eth0 can *not* be removed from a running instance while eth1 may be removed or added while the machine is running. However, it is unclear if additional VIPs can be added to eth0 while the instance is running. I'm not sure how changes to the available IPs would be detected without polling the meta-data service.

Michael Hale (mikehale) wrote :

Adding notes from IRC:

* use 'manual eth1' to avoid cloud-init blocking on boot waiting for interfaces
* /etc/network/interfaces originates from a hardcoded file in the image
* cloud-init places /etc/cloud/on-ec2 (or similar) flag file
* on interface add/remove have a rule that fires, checks ec2 flag file, then updates /etc/network/interfaces, and ifup/down

Scott Moser (smoser) on 2013-05-15
Changed in cloud-init:
status: New → Triaged
importance: Undecided → Medium
Forest (foresto) wrote :

I'm in a similar situation: An ec2 instance with just one network interface, to which I have assigned a secondary IP address.

This bit from the AWS docs is interesting:

"If you are using Amazon Linux, the ec2-net-utils package can take care of this step for you. It configures additional network interfaces that you attach while the instance is running, refreshes secondary IP addresses during DHCP lease renewal, and updates the related routing rules."

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html#StepTwoConfigOS

Perhaps it would be worthwhile for cloud-init to do the same thing that Amazon's ec2-net-utils package does.

Scott Moser (smoser) wrote :

Forest,
  Thanks for the link. I hadn't seen that utility. It does appear to be doing basically what I was suggesting.

Forest (foresto) wrote :

Sure. The part that stood out to me was "refreshes secondary IP addresses during DHCP lease renewal." I haven't examined Amazon's code, but that description reads like it solves the problem of automatically configuring additional IP addresses without requiring additional network interfaces.

Alle (alessandro-demaria) wrote :

Hi Forest,

looking at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#ec2-net-utils

"DHCP script
Whenever the network interface receives a new DHCP lease, this script queries the instance metadata for Elastic IP addresses. For each Elastic IP address, it adds a rule to the routing policy database to ensure that outbound traffic from that address uses the correct network interface. It also adds each private IP address to the network interface as a secondary address."

I think all they do is run a hook script on DHCP lease that configures the interface. And yes they add multiple ip addresses to the same interface.

I am looking at some sort of solution because I want to use some instances as LXC hosts and would love to give each container its own ip. So I think I will have to go down the route of using subinterfaces.

Will keep tracking this bug

Thank you
Alessandro

Alle (alessandro-demaria) wrote :

Hi,

I have actually adapted the files from the Amazon AMI which are working just fine.
There is also an ansible file which tells where the files should be dropped. You might need to adjust a few things but it should all be working

There is only a problem with this bug (https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1235169) which has actually been solved and I am waiting for the updated package to be available. Otherwise the solution for this is to add the interfaces-eth1 configuration into the main /etc/network/interfaces file.

I am not good at packaging stuff so if you think this is useful it would be great if someone could wrap it up in a nice package. Myself I am ok with using ansible for now :)

Hope this helps
Regards
Alessandro

Alle (alessandro-demaria) wrote :

Re-uploading attachment as solution

Alle (alessandro-demaria) wrote :

Fixed some bugs. New version here
https://github.com/ademaria/ubuntu-ec2net

Scott Moser (smoser) wrote :

I'm attaching ec2-utils-0.4-1.19.amzn1.src.rpm as I retrieved it with:
 get_reference_source -p ec2-net-utils
from inside ami-83e4bcea (amazon/amzn-ami-pv-2013.09.1.x86_64-ebs) today.

The license per spec file and headers is 'Apache License 2.0'.

Scott Moser (smoser) wrote :

adding also as a tarball of the rpmbuild dir after rpm -Uvh (jsut for convenience).

Jason Dusek (solidsnack) wrote :

We're using a variant Alessandro's work to configure out instances.

https://github.com/instacart/ubuntu-ec2net

Reboot doesn't seem to refresh the interfaces list -- maybe there is another event hook that we need? I'm not terribly familiar with udev but would love to see this ironed out and ready for Ubuntu.

Installing the RPMs (via Alien) seemed to have no effect at all.

I'm using cloud-init on CentOS, but would really like this functionality as well. For the time being, I'll be trying to reuse the upstream AWS pieces.

Naren Palepu (naren-j) wrote :

Hi

This is still the same for new instances on AWS ec2 ubuntu 144 . Floating i/p secondary eth0 not coming up automatically. IS there a fix in plan for ubuntu 14,15 ?

Thanks,

NP

tags: added: ec2-images
removed: ec2
Gaëtan Duchaussois (gaetanfl) wrote :

Hi,
I made a small patch to support multiple ips on one interface. Should work on several interfaces too even if I din't test it.
I tested it on ec2 with ubuntu 18.04 and netplan export

The attachment "ec2_multiple_ips.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers