ec2-init hangs boot on non-EC2/Eucalpytus systems

[David Favor]

Installing the ec2-init package on all Karmic versions through Alpha 5 hang with the message...

"Waiting for ec2 meta-data service"

I've tried this with and without EC2_PRIVATE_KEY & EC2_CERT set. Same results.

[arky (arky)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. This bug did not have a package associated with it, which is important for ensuring that it gets looked at by the proper developers. You can learn more about finding the right package at https://wiki.ubuntu.com/Bugs/FindRightPackage. I have classified this bug as a bug in ec2-init.

When reporting bugs in the future please use apport, either via the appropriate application's "Help -> Report a Problem" menu or using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[David Favor (davidfavor)]

Someone's smokin' crack and while under the influence has created this problem.

In /usr/lib/python2.6/dist-packages/ec2init/__init__.py around line 106 this code exists...

    def wait_for_metadata_service(self):
        timeout = 2
        # This gives us about half an hour before we ultimately bail out
        for x in range(10):
            s = socket.socket()
            try:
                address = '169.254.169.254'
                port = 80
                s.connect((address,port))
                s.close()
                return True
            except socket.error, e:
                time.sleep(timeout)
                timeout = timeout * 2
        return False

Thus 'service ec2-init start' hangs forever trying to connect to the hard coded address.

Looking back through the code /etc/ec2-init/ec2-config.cfg exists as a config file, so most likely the developer meant to have this address (perhaps for an AMI instance) live in the cfg file.

Even this seems a poor choice. Better to have this IP address set through an environment variable although this is still cumbersome as it appear the ec2-init service must then be bounced each time interaction with a different instance is required.

[David Favor (davidfavor)]

Ah... http://docs.amazonwebservices.com/AmazonEC2/dg/2006-10-01/AESDG-chapter-instancedata.html states that 169.254.169.254 is always available.

And this IP address is no longer in use.

The docs referenced are from 2006-10-01 so most likely this IP address has changed.

Best to move getting this IP address to the .cfg file then research the current IP address and a way to track if this IP address changes.

[David Favor (davidfavor)]

The 2007 docs state to use this same IP address...

http://docs.amazonwebservices.com/AWSEC2/2007-08-29/DeveloperGuide/AESDG-chapter-instancedata.html

[David Favor (davidfavor)]

The 2009 docs state to use the same IP address...

http://docs.amazonwebservices.com/AWSEC2/2009-07-15/DeveloperGuide/index.html?AESDG-chapter-instancedata.html

http://169.254.169.254/2007-08-29/meta-data/ is suppose to return the toplevel metadata items and it shows the IP address is not responding.

http://status.aws.amazon.com/ shows no outages.

[David Favor (davidfavor)]

Ah... This is a non-routed address like the 192... IP range and appears to only be active once some AMI instance is active in the cloud. Still this is very confusing to someone just starting out exploring cloud technology.

Although this still makes very little sense as a person may have many AMI instances running, thus this probably means there is some sort of Metadata server which runs which serves out Metadata API information.

There should be some check in this package for whatever dependency is required (like the Metadata API server), rather than hanging forever when the package is installed.

Please update this bug with the correct procedure (package install sequence & commands to execute) to allow the ec2-init package to be installed.

[David Favor (davidfavor)]

The problem is ec2-init should only be installed on running cloud instances.

The fix is to have this package throw an error if an attempt is made to install it on a non-cloud instance.

[Eric Hammond (esh)]

The ec2-init package should only be installed on an instance (or image) which is used inside of Amazon EC2 or compatible environment (Eucalyptus). End users should never install the ec2-init package themselves as it is only useful when building images for EC2/Eucalyptus.

Are there any other packages in Ubuntu which should never be installed by normal users? What protection do they take?

Perhaps ec2-init could by default install itself in safety mode with no bootup actions and require an additional step to "activate" it.

[Soren Hansen (soren)]

On Sat, Sep 05, 2009 at 10:23:54PM -0000, David Favor wrote:
> The problem is ec2-init should only be installed on running cloud
> instances.
>
> The fix is to have this package throw an error if an attempt is made to
> install it on a non-cloud instance.

That's really the crux of the problem. The only way to be sure that
you're running on EC2 or a compatible environment (like UEC), is by
looking for the meta- or user-data service.

I happened to discuss this last week, though, and I've come up with an
approach that should make most people happy.

I'm going to add a check to the init script that will look for signs
that we're not running on EC2, such as "crashkernel" or "splash" on the
kernel command line. It's not 100% accurate, but errs on the side of
running the init script if there's doubt, and should solve this problem
for at least Ubuntu users, as I believe they will all have "splash" on
their kernel command line.

[Eric Hammond (esh)]

Soren:

There are a number of packages which do not activate automatically when they installed, but instead require changing a configuration line in /etc/default/xxx to "RUN=yes" or some such.

The vmbuilder EC2 plugin would have no trouble tweaking this after installing the ec2-init package.

What do you think about using this approach to avoid breaking Ubuntu systems?

I suspect a lot of people looking to get started with EC2 will find "ec2-init" in the package lists and install it without any idea that it really isn't what they want. I though there were situations where removing "splash" is sometimes preferable, but I could be mistaken.

[Scott Moser (smoser)]

This is fix-released.

with ec2-init 0.4.999-0ubuntu2 or later, it by default doesn't do anything. The UEC images explicitly turn ec2-init on in
/etc/ec2-init/is-compat-env . You should not see this, unless
a.) you have 'compat=1' in a file /etc/ec2-init/is-compat-env ,
b.) you have 'ec2init=1' in your /proc/cmdline
c.) you're running a UEC image outside of ec2/uec (or with inside UEC with broken metadataservice)