Hostname not set correctly on UEC cloud due to IP address in local-hostname manifest data (DNS)

[Neil Wilson]

UEC images use the local-hostname setting of the manifest data to set the hostname. On Eucalyptus that appears to be set to the local ip address - which doesn't work properly as the hostname obviously stops at the first period it sees.

Hence you have a lot of machines starting up with hostname '172'...

Additionally /etc/hostname isn't altered by the ec2-init processes.

The default hostname in the manifest data should probably be a combination of the availability zone name (clusterid) and the local/public ip address as appropriate.

so local-hostname should be for example: isp-lab-172-19-1-30 and public-hostname should be isp-lab-10-99-67-9

ProblemType: Bug
.etc.eucalyptus.eucalyptus.cc.conf: CC_NAME="isp-lab"
Architecture: amd64
Date: Thu Nov 5 11:25:12 2009
DistroRelease: Ubuntu 9.10
Package: eucalyptus-cc 1.6~bzr931-0ubuntu7
ProcEnviron:
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-14.48-server
SourcePackage: eucalyptus
Uname: Linux 2.6.31-14-server x86_64

=== Begin SRU Information [ec2-init] ===
- impact on users: When running an instance of UEC image under Eucalyptus, hostname is set to '172' or whatever is the first token of the IP address. This means that all your instances have seemingly the same hostname.
- how the bug has been addressed: Instead of blindly calling 'hostname' with the contents of the metadata services 'local-hostname', the code now checks to see if that appears to be a ipv4 address. If it is an ipv4 address, the hostname is set to 'ip-W-X-Y-Z" where WXYZ are values from the IP address.
- patch is available at: http://bazaar.launchpad.net/%7Esmoser/ubuntu/karmic/ec2-init/karmic-updates/revision/37.1.2
- regression pottential:
  The regression potential should be fairly low because the test for "is this an IPV4 address" is conservative. The check makes makes sure that a.) there are 4 pieces of the hostname when split on '.', b.) each of the pieces is an integer greater than 0 and less than 255. If both of those conditions are not met, the old path is taken.

TEST CASE:
  - start karmic ec2/uec instance in UEC
  - ssh to instance, run 'hostname'. Given an IP address of 172.1.2.15, failure will show "172". With the fix in place, it will show 'ip-172-1-2-15'.

=== End SRU Information ===

[Chuck Short (zulcss)]

I think this might be more of a problem with ec2-init.

Regards
chuck

[Scott Moser (smoser)]

> I think this might be more of a problem with ec2-init.

Just to be clear on what is happening here:

- eucalyptus is intending to provide an ec2 compatible meta-data service
  (described at
http://docs.amazonwebservices.com/AWSEC2/2009-04-04/DeveloperGuide/ ).
  that includes a field "local-hostname".
- on ec2, the value of 'local-hostname' is a fully qualified domain name
  that can also be looked up via the provided dns
  (example:ip-10-212-65-188.ec2.internal)
- on eucalyptus, the value of 'local-hostname' is the internal IP address
  (example: 192.168.1.194 or 172.123.1.2).
- ec2-init is utilizing local-hostname to set the hostname. It is doing
  so by taking the first token of a dot-delimited string.

The result, is that on eucalyptus, the hostname is set to the decimal
value of the first octet of the ip address (ie "172").

We could work around this in ec2-init, and be smarter about setting
hostname, but the real issue is that the field 'local-hostname' contains
an IP address, not a "hostname".

[Boris Devouge (bdevouge)]

As suggested, tried editing /etc/ec2-init/templates/hosts.tmpl and replacing the $hostname var with a fixed value (ubuntu), rebundling and restarted the instance still landed me in a shell: ubuntu@172

[Boris Devouge (bdevouge)]

ubuntu@172:~$ cat /etc/ec2-init/templates/hosts.tmpl
127.0.0.1 localhost
127.0.1.1 ubuntu

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
ubuntu@172:~$ cat /etc/hosts
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

ubuntu@172:~$ hostname
172
ubuntu@172:~$

[Dustin Kirkland  (kirkland)]

Talking to Dan, it sounds like this should be solved in ec2-init, rather than Eucalyptus.

[Disconnect (dismc)]

For those of us who aren't Dan, why would you solve an ec2 incompatibility with a distribution-specific workaround? Seems like either a dns lookup (at startup and cached?) or a simple iterator mapping ("Hostname format: cloud-$1-$2-$3-$4.foo.com" for example) would be a proper fix.

[Boris Devouge (bdevouge)]

This also badly breaks JBoss. I am not sure this is specific to Jboss, as other java app servers will also most likely grab the wrong hostname.

[chris grzegorczyk (chris-grze)]

This is not a bug in Eucalyptus: to get the hostname information there DNS needs to be enabled. It is disabled by default for the UEC.

[Daniel Nurmi (nurmi)]

Some more context: if the eucalyptus DNS feature is disabled (as it is in the UEC, by default), then we cannot guarantee that the IP addresses that are specified by the administrator resolve to an actual hostname (as, often, users are choosing private/unroutable IP addresses for VMs). Inventing a hostname here will break other services that rely on this information as being string that can be passed other network services. This is the reason why, when the Eucalyptus DNS service is disabled, we put an IP address in this field.

To us, the first priority of this field is to return 'a string that can be used as a network endpoint' rather than 'a string that the hostname command handles properly'.

If a Eucalyptus user requires a legitimate hostname here, then it is best to enable the Eucalyptus DNS service as detailed here:

http://open.eucalyptus.com/wiki/DynamicDNSGuide

[Boris Devouge (bdevouge)]

Thanks for all comment and explanations for the DNS services of eucalyptus, however it is not accepted/possible in all cases, and not the whole issue.

There is still an issue where we do not get an full IP attributed as a hostname (which would help already assuming the uniqueness of hostnames and make things work), but only the first 3 digit, so '172' in the case as only the private IP space is visible from the instance.

The main issue is that '172' seems to break many things, Jboss running RMI on http://172:8083 but also other basic networking tools... and seems to not comply with what is an allowed hostname by convention. Could we not generate an unique hostname with random bits everytime, and let people the option then to edit / have them in DNS if desired?

Courtesy of Andrew Glen-Young, the following snippet of code, work around for current issue, not sure if it would help:

https://pastebin.canonical.com/27281/

[Boris Devouge (bdevouge)]

To be clear, i understand Eucalytus has a mapping as described in the url Dan Nurmi provided, but i am trying to understand if it is the job of UEC to assign a proper hostname to its instances (even when DNS is missing), or if it sole responsibility is to launch the instances, even if they all end up with the same hostname of "172".

A simple iterator as mentioned by Disconnect will be a great addition.

[Etienne Goyer (etienne-goyer-outlands)]

Dan, would enabling the Eucalyptus DNS feature actually fix this problem? If so, then is there a good reason to disable it in the first place?

[Thierry Carrez (ttx)]

Reopening eucalyptus and closing ec2-init task: looks like enabling DNS on CLC is the right way to fix this.

[Dustin Kirkland  (kirkland)]

Okay, worked on this a bit here today at the Distro Sprint.

We enabled Eucalyptus' DNS with: DISABLE_DNS="N" in /etc/eucalyptus/eucalyptus.conf.

This seems to work well enough to provide dynamically generated host names to vm's. See:

RESERVATION r-43AC08FE admin default
INSTANCE i-2DC5056F emi-DF7014F1 euca-172-19-1-2.eucalyptus.localhost euca-172-19-1-2.eucalyptus.internal running mykey 0 c1.medium 2010-02-03T22:14:40.651Z canyonedge eki-25291A19 eri-058C1996

So this gets us part of the way there.

HOWEVER, there's a major piece of this that cannot be solved without external configuration of your network's DNS.

Even though my instance might have hostname euca-172-19-1-2.eucalyptus.localhost, I cannot ssh to euca-172-19-1-2.eucalyptus.localhost without tweaking my DNS configuration.

For this reason, we cannot actually fix this bug within Ubuntu's Eucalyptus package. Rather, this will need to be solved in some how-to documentation under:
 * https://help.ubuntu.com/community/UEC/DNS

Upstream has some good docs at http://open.eucalyptus.com/wiki/DynamicDNSGuide. We'll base our guide on this, and tailor it for UEC.

[Dustin Kirkland  (kirkland)]

Unassigning myself. Anyone could theoretically document this. Hint hint ;-)

[Thierry Carrez (ttx)]

The cloud images should workaround this by using a ip-x-y-z-w hostname whenever local-hostname returns an IP address.

[Disconnect (dismc)]

Any time you say "the images should work around..", that means that UEC's EC2 compatibility is -broken-. UEC should return a text hostname (ip-a-b-c-d? instancename? no-dns-you-loser.euca-a-b-c-d? whatever.) in the hostname field. Regardless of configuration. The spec you are claiming compatibility with -insists- on that.

The iterator would be nice (that is what we would have used in our installation), or even a sprintf-style configurator, but that is (reasonably) a wishlist request.

"We put numbers where the spec -requires- text" IS A BUG.

[Boris Devouge (bdevouge)]

Dustin,

 Which version of UEC are you using? Not managed to get internal DNS working here (up2date 9.10 UEC). Cheers.

[Dustin Kirkland  (kirkland)]

We're developing/testing 10.04 Lucid right now. 9.10 is closed to
development, for all but the most critical data-loss or security bugs.

:-Dustin

[Scott Moser (smoser)]

this is fixed with the 'ip-uuu.xxx.yyy.zzz' solution that Boris suggested. (ec2-init was renamed to cloud-init)

[chris grzegorczyk (chris-grze)]

A fix (note status change to "In Progress") is in QA. Merge later tonight.

On Thu, Feb 4, 2010 at 4:33 PM, Scott Moser <email address hidden> wrote:
> this is fixed with the 'ip-uuu.xxx.yyy.zzz' solution that Boris
> suggested.  (ec2-init was renamed to cloud-init)
>
> ** Package changed: ec2-init (Ubuntu Lucid) => cloud-init (Ubuntu Lucid)
>
> --
> Hostname not set correctly on UEC cloud due to IP address in local-hostname manifest data (DNS)
> https://bugs.launchpad.net/bugs/475354
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: In Progress
> Status in “cloud-init” package in Ubuntu: Triaged
> Status in “eucalyptus” package in Ubuntu: Triaged
> Status in “cloud-init” source package in Lucid: Triaged
> Status in “eucalyptus” source package in Lucid: Triaged
>
> Bug description:
> UEC images use the local-hostname setting of the manifest data to set the hostname. On Eucalyptus that appears to be set to the local ip address - which doesn't work properly as the hostname obviously stops at the first period it sees.
>
> Hence you have a lot of machines starting up with hostname '172'...
>
> Additionally /etc/hostname isn't altered by the ec2-init processes.
>
> The default hostname in the manifest data should probably be a combination of the availability zone name (clusterid) and the local/public ip address as appropriate.
>
> so local-hostname should be for example: isp-lab-172-19-1-30 and public-hostname should be isp-lab-10-99-67-9
>
> ProblemType: Bug
> .etc.eucalyptus.eucalyptus.cc.conf: CC_NAME="isp-lab"
> Architecture: amd64
> Date: Thu Nov  5 11:25:12 2009
> DistroRelease: Ubuntu 9.10
> Package: eucalyptus-cc 1.6~bzr931-0ubuntu7
> ProcEnviron:
>  LANG=en_GB.UTF-8
>  SHELL=/bin/bash
> ProcVersionSignature: Ubuntu 2.6.31-14.48-server
> SourcePackage: eucalyptus
> Uname: Linux 2.6.31-14-server x86_64
>
>
>

--
Chris Grzegorczyk
Co-Founder and Engineer
Eucalyptus Systems, Inc.

130 Castilian St. | Goleta, CA | 93117
Office: 805-968-1400 x e^1 | Cell: 805-807-8237
Email: <email address hidden>
www.eucalyptus.com
________________________________________

[chris grzegorczyk (chris-grze)]

------------------------------------------------------------
revno: 1172
committer: decker <decker@hawaii>
branch nick: 1.6.2
timestamp: Fri 2010-02-05 04:04:49 -0800
message:
  fix updating of vm state w/ dns info
------------------------------------------------------------

[Launchpad Janitor (janitor)]

This bug was fixed in the package eucalyptus - 1.6.2~bzr1166-0ubuntu6

---------------
eucalyptus (1.6.2~bzr1166-0ubuntu6) lucid; urgency=low

  * Merge from upstream bzr revision, should fix:
    - LP: #475354 - send correct hostname in metadata service
    - LP: #461464 - fix ec2 tools compatibility
 -- Dustin Kirkland <email address hidden> Fri, 05 Feb 2010 09:51:58 -0800

[Scott Moser (smoser)]

fixed in cloud-init 0.5.3-0ubuntu1

[Scott Moser (smoser)]

I'm in need of a sponsor for lp:~smoser/ubuntu/karmic/ec2-init/karmic-updates . That branch contains fixes for bug 520734 and bug 475354 .

[Thierry Carrez (ttx)]

Uploaded to -proposed

[Martin Pitt (pitti)]

Accepted ec2-init into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Scott Moser (smoser)]

I've tested this on karmic released uec image (20100121):

$ hostname
172
$ echo deb http://archive.ubuntu.com/ubuntu/ karmic-proposed restricted main multiverse universe | sudo tee /etc/apt/sources.list.d/proposed.list
$ sudo apt-get update
$ apt-cache policy ec2-init
ec2-init:
  Installed: 0.4.999-0ubuntu7.1
  Candidate: 0.4.999-0ubuntu7.2
$ sudo apt-get install ec2-init
...
$ hostname
ip-172-19-1-3

Then, just for good measure
$ sudo reboot
...
system comes back up, and prompt shows:
ubuntu@ip-172-19-1-3:~$

[Boris Devouge (bdevouge)]

Ok, Verified and confirmed fixing.

I effectively get the right hostname without even a reboot.

Thanks !!

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-init - 0.4.999-0ubuntu7.2

---------------
ec2-init (0.4.999-0ubuntu7.2) karmic-proposed; urgency=low

  * fix install/upgrade failure if python2.4 is installed (LP: #520734)
  * set hostname to a hostname-like value (ip-W-X-Y-Z) if instance
    metadata includes a ip address rather than a hostname in
    local-hostname (LP: #475354)
 -- Scott Moser <email address hidden> Thu, 11 Feb 2010 20:35:41 -0500