Hostname not set correctly on UEC cloud due to IP address in local-hostname manifest data (DNS)

Bug #475354 reported by Neil Wilson
38
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Eucalyptus
Fix Released
Undecided
chris grzegorczyk
cloud-init (Ubuntu)
Fix Released
Medium
Scott Moser
Karmic
Invalid
Undecided
Unassigned
Lucid
Fix Released
Medium
Scott Moser
ec2-init (Ubuntu)
Fix Released
Undecided
Unassigned
Karmic
Fix Released
Medium
Thierry Carrez
Lucid
Fix Released
Undecided
Unassigned
eucalyptus (Ubuntu)
Fix Released
Wishlist
Unassigned
Karmic
Won't Fix
Undecided
Unassigned
Lucid
Fix Released
Wishlist
Unassigned

Bug Description

UEC images use the local-hostname setting of the manifest data to set the hostname. On Eucalyptus that appears to be set to the local ip address - which doesn't work properly as the hostname obviously stops at the first period it sees.

Hence you have a lot of machines starting up with hostname '172'...

Additionally /etc/hostname isn't altered by the ec2-init processes.

The default hostname in the manifest data should probably be a combination of the availability zone name (clusterid) and the local/public ip address as appropriate.

so local-hostname should be for example: isp-lab-172-19-1-30 and public-hostname should be isp-lab-10-99-67-9

ProblemType: Bug
.etc.eucalyptus.eucalyptus.cc.conf: CC_NAME="isp-lab"
Architecture: amd64
Date: Thu Nov 5 11:25:12 2009
DistroRelease: Ubuntu 9.10
Package: eucalyptus-cc 1.6~bzr931-0ubuntu7
ProcEnviron:
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-14.48-server
SourcePackage: eucalyptus
Uname: Linux 2.6.31-14-server x86_64

=== Begin SRU Information [ec2-init] ===
- impact on users: When running an instance of UEC image under Eucalyptus, hostname is set to '172' or whatever is the first token of the IP address. This means that all your instances have seemingly the same hostname.
- how the bug has been addressed: Instead of blindly calling 'hostname' with the contents of the metadata services 'local-hostname', the code now checks to see if that appears to be a ipv4 address. If it is an ipv4 address, the hostname is set to 'ip-W-X-Y-Z" where WXYZ are values from the IP address.
- patch is available at: http://bazaar.launchpad.net/%7Esmoser/ubuntu/karmic/ec2-init/karmic-updates/revision/37.1.2
- regression pottential:
  The regression potential should be fairly low because the test for "is this an IPV4 address" is conservative. The check makes makes sure that a.) there are 4 pieces of the hostname when split on '.', b.) each of the pieces is an integer greater than 0 and less than 255. If both of those conditions are not met, the old path is taken.

TEST CASE:
  - start karmic ec2/uec instance in UEC
  - ssh to instance, run 'hostname'. Given an IP address of 172.1.2.15, failure will show "172". With the fix in place, it will show 'ip-172-1-2-15'.

=== End SRU Information ===

Revision history for this message
Neil Wilson (neil-aldur) wrote :
Revision history for this message
Chuck Short (zulcss) wrote :

I think this might be more of a problem with ec2-init.

Regards
chuck

Changed in eucalyptus (Ubuntu):
importance: Undecided → Medium
status: New → Incomplete
Revision history for this message
Scott Moser (smoser) wrote : Re: [Bug 475354] Re: Hostname not set correctly on UEC cloud due to IP address in local-hostname manifest data

> I think this might be more of a problem with ec2-init.

Just to be clear on what is happening here:

- eucalyptus is intending to provide an ec2 compatible meta-data service
  (described at
http://docs.amazonwebservices.com/AWSEC2/2009-04-04/DeveloperGuide/ ).
  that includes a field "local-hostname".
- on ec2, the value of 'local-hostname' is a fully qualified domain name
  that can also be looked up via the provided dns
  (example:ip-10-212-65-188.ec2.internal)
- on eucalyptus, the value of 'local-hostname' is the internal IP address
  (example: 192.168.1.194 or 172.123.1.2).
- ec2-init is utilizing local-hostname to set the hostname. It is doing
  so by taking the first token of a dot-delimited string.

The result, is that on eucalyptus, the hostname is set to the decimal
value of the first octet of the ip address (ie "172").

We could work around this in ec2-init, and be smarter about setting
hostname, but the real issue is that the field 'local-hostname' contains
an IP address, not a "hostname".

Thierry Carrez (ttx)
Changed in eucalyptus (Ubuntu):
status: Incomplete → Triaged
Revision history for this message
Boris Devouge (bdevouge) wrote : Re: Hostname not set correctly on UEC cloud due to IP address in local-hostname manifest data

As suggested, tried editing /etc/ec2-init/templates/hosts.tmpl and replacing the $hostname var with a fixed value (ubuntu), rebundling and restarted the instance still landed me in a shell: ubuntu@172

Revision history for this message
Boris Devouge (bdevouge) wrote :

ubuntu@172:~$ cat /etc/ec2-init/templates/hosts.tmpl
127.0.0.1 localhost
127.0.1.1 ubuntu

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
ubuntu@172:~$ cat /etc/hosts
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

ubuntu@172:~$ hostname
172
ubuntu@172:~$

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Talking to Dan, it sounds like this should be solved in ec2-init, rather than Eucalyptus.

Changed in ec2-init (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Changed in eucalyptus (Ubuntu):
status: Triaged → Invalid
Changed in ec2-init (Ubuntu):
assignee: nobody → Scott Moser (smoser)
Revision history for this message
Disconnect (dismc) wrote :

For those of us who aren't Dan, why would you solve an ec2 incompatibility with a distribution-specific workaround? Seems like either a dns lookup (at startup and cached?) or a simple iterator mapping ("Hostname format: cloud-$1-$2-$3-$4.foo.com" for example) would be a proper fix.

Revision history for this message
Boris Devouge (bdevouge) wrote :

This also badly breaks JBoss. I am not sure this is specific to Jboss, as other java app servers will also most likely grab the wrong hostname.

Revision history for this message
chris grzegorczyk (chris-grze) wrote :

This is not a bug in Eucalyptus: to get the hostname information there DNS needs to be enabled. It is disabled by default for the UEC.

Changed in eucalyptus:
assignee: nobody → chris grzegorczyk (chris-grze)
status: New → Invalid
Revision history for this message
Daniel Nurmi (nurmi) wrote :

Some more context: if the eucalyptus DNS feature is disabled (as it is in the UEC, by default), then we cannot guarantee that the IP addresses that are specified by the administrator resolve to an actual hostname (as, often, users are choosing private/unroutable IP addresses for VMs). Inventing a hostname here will break other services that rely on this information as being string that can be passed other network services. This is the reason why, when the Eucalyptus DNS service is disabled, we put an IP address in this field.

To us, the first priority of this field is to return 'a string that can be used as a network endpoint' rather than 'a string that the hostname command handles properly'.

If a Eucalyptus user requires a legitimate hostname here, then it is best to enable the Eucalyptus DNS service as detailed here:

http://open.eucalyptus.com/wiki/DynamicDNSGuide

Revision history for this message
Boris Devouge (bdevouge) wrote :

Thanks for all comment and explanations for the DNS services of eucalyptus, however it is not accepted/possible in all cases, and not the whole issue.

There is still an issue where we do not get an full IP attributed as a hostname (which would help already assuming the uniqueness of hostnames and make things work), but only the first 3 digit, so '172' in the case as only the private IP space is visible from the instance.

The main issue is that '172' seems to break many things, Jboss running RMI on http://172:8083 but also other basic networking tools... and seems to not comply with what is an allowed hostname by convention. Could we not generate an unique hostname with random bits everytime, and let people the option then to edit / have them in DNS if desired?

Courtesy of Andrew Glen-Young, the following snippet of code, work around for current issue, not sure if it would help:

https://pastebin.canonical.com/27281/

Revision history for this message
Boris Devouge (bdevouge) wrote :

To be clear, i understand Eucalytus has a mapping as described in the url Dan Nurmi provided, but i am trying to understand if it is the job of UEC to assign a proper hostname to its instances (even when DNS is missing), or if it sole responsibility is to launch the instances, even if they all end up with the same hostname of "172".

A simple iterator as mentioned by Disconnect will be a great addition.

Revision history for this message
Etienne Goyer (etienne-goyer-outlands) wrote :

Dan, would enabling the Eucalyptus DNS feature actually fix this problem? If so, then is there a good reason to disable it in the first place?

Revision history for this message
Thierry Carrez (ttx) wrote :

Reopening eucalyptus and closing ec2-init task: looks like enabling DNS on CLC is the right way to fix this.

Changed in eucalyptus (Ubuntu):
status: Invalid → Triaged
assignee: nobody → Dustin Kirkland (kirkland)
Changed in ec2-init (Ubuntu):
status: Triaged → Invalid
Changed in eucalyptus (Ubuntu):
milestone: none → lucid-alpha-3
Thierry Carrez (ttx)
Changed in eucalyptus (Ubuntu Lucid):
importance: Medium → High
summary: Hostname not set correctly on UEC cloud due to IP address in local-
- hostname manifest data
+ hostname manifest data (DNS)
Thierry Carrez (ttx)
Changed in ec2-init (Ubuntu Lucid):
status: Invalid → Triaged
Changed in eucalyptus (Ubuntu Lucid):
importance: High → Wishlist
milestone: lucid-alpha-3 → ubuntu-10.04
Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Okay, worked on this a bit here today at the Distro Sprint.

We enabled Eucalyptus' DNS with: DISABLE_DNS="N" in /etc/eucalyptus/eucalyptus.conf.

This seems to work well enough to provide dynamically generated host names to vm's. See:

RESERVATION r-43AC08FE admin default
INSTANCE i-2DC5056F emi-DF7014F1 euca-172-19-1-2.eucalyptus.localhost euca-172-19-1-2.eucalyptus.internal running mykey 0 c1.medium 2010-02-03T22:14:40.651Z canyonedge eki-25291A19 eri-058C1996

So this gets us part of the way there.

HOWEVER, there's a major piece of this that cannot be solved without external configuration of your network's DNS.

Even though my instance might have hostname euca-172-19-1-2.eucalyptus.localhost, I cannot ssh to euca-172-19-1-2.eucalyptus.localhost without tweaking my DNS configuration.

For this reason, we cannot actually fix this bug within Ubuntu's Eucalyptus package. Rather, this will need to be solved in some how-to documentation under:
 * https://help.ubuntu.com/community/UEC/DNS

Upstream has some good docs at http://open.eucalyptus.com/wiki/DynamicDNSGuide. We'll base our guide on this, and tailor it for UEC.

Revision history for this message
Dustin Kirkland  (kirkland) wrote :

Unassigning myself. Anyone could theoretically document this. Hint hint ;-)

Changed in eucalyptus (Ubuntu Lucid):
assignee: Dustin Kirkland (kirkland) → nobody
Revision history for this message
Thierry Carrez (ttx) wrote :

The cloud images should workaround this by using a ip-x-y-z-w hostname whenever local-hostname returns an IP address.

Revision history for this message
Disconnect (dismc) wrote :

Any time you say "the images should work around..", that means that UEC's EC2 compatibility is -broken-. UEC should return a text hostname (ip-a-b-c-d? instancename? no-dns-you-loser.euca-a-b-c-d? whatever.) in the hostname field. Regardless of configuration. The spec you are claiming compatibility with -insists- on that.

The iterator would be nice (that is what we would have used in our installation), or even a sprintf-style configurator, but that is (reasonably) a wishlist request.

"We put numbers where the spec -requires- text" IS A BUG.

Revision history for this message
Boris Devouge (bdevouge) wrote :

Dustin,

 Which version of UEC are you using? Not managed to get internal DNS working here (up2date 9.10 UEC). Cheers.

Revision history for this message
Dustin Kirkland  (kirkland) wrote : Re: [Bug 475354] Re: Hostname not set correctly on UEC cloud due to IP address in local-hostname manifest data (DNS)

We're developing/testing 10.04 Lucid right now. 9.10 is closed to
development, for all but the most critical data-loss or security bugs.

:-Dustin

Changed in eucalyptus:
status: Invalid → In Progress
Revision history for this message
Scott Moser (smoser) wrote :

this is fixed with the 'ip-uuu.xxx.yyy.zzz' solution that Boris suggested. (ec2-init was renamed to cloud-init)

affects: ec2-init (Ubuntu Lucid) → cloud-init (Ubuntu Lucid)
Revision history for this message
chris grzegorczyk (chris-grze) wrote :

A fix (note status change to "In Progress") is in QA. Merge later tonight.

On Thu, Feb 4, 2010 at 4:33 PM, Scott Moser <email address hidden> wrote:
> this is fixed with the 'ip-uuu.xxx.yyy.zzz' solution that Boris
> suggested.  (ec2-init was renamed to cloud-init)
>
> ** Package changed: ec2-init (Ubuntu Lucid) => cloud-init (Ubuntu Lucid)
>
> --
> Hostname not set correctly on UEC cloud due to IP address in local-hostname manifest data (DNS)
> https://bugs.launchpad.net/bugs/475354
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: In Progress
> Status in “cloud-init” package in Ubuntu: Triaged
> Status in “eucalyptus” package in Ubuntu: Triaged
> Status in “cloud-init” source package in Lucid: Triaged
> Status in “eucalyptus” source package in Lucid: Triaged
>
> Bug description:
> UEC images use the local-hostname setting of the manifest data to set the hostname. On Eucalyptus that appears to be set to the local ip address - which doesn't work properly as the hostname obviously stops at the first period it sees.
>
> Hence you have a lot of machines starting up with hostname '172'...
>
> Additionally /etc/hostname isn't altered by the ec2-init processes.
>
> The default hostname in the manifest data should probably be a combination of the availability zone name (clusterid) and the local/public ip address as appropriate.
>
> so local-hostname should be for example: isp-lab-172-19-1-30 and public-hostname should be isp-lab-10-99-67-9
>
> ProblemType: Bug
> .etc.eucalyptus.eucalyptus.cc.conf: CC_NAME="isp-lab"
> Architecture: amd64
> Date: Thu Nov  5 11:25:12 2009
> DistroRelease: Ubuntu 9.10
> Package: eucalyptus-cc 1.6~bzr931-0ubuntu7
> ProcEnviron:
>  LANG=en_GB.UTF-8
>  SHELL=/bin/bash
> ProcVersionSignature: Ubuntu 2.6.31-14.48-server
> SourcePackage: eucalyptus
> Uname: Linux 2.6.31-14-server x86_64
>
>
>

--
Chris Grzegorczyk
Co-Founder and Engineer
Eucalyptus Systems, Inc.

130 Castilian St. | Goleta, CA | 93117
Office: 805-968-1400 x e^1 | Cell: 805-807-8237
Email: <email address hidden>
www.eucalyptus.com
________________________________________

Revision history for this message
chris grzegorczyk (chris-grze) wrote :

------------------------------------------------------------
revno: 1172
committer: decker <decker@hawaii>
branch nick: 1.6.2
timestamp: Fri 2010-02-05 04:04:49 -0800
message:
  fix updating of vm state w/ dns info
------------------------------------------------------------

Changed in eucalyptus:
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eucalyptus - 1.6.2~bzr1166-0ubuntu6

---------------
eucalyptus (1.6.2~bzr1166-0ubuntu6) lucid; urgency=low

  * Merge from upstream bzr revision, should fix:
    - LP: #475354 - send correct hostname in metadata service
    - LP: #461464 - fix ec2 tools compatibility
 -- Dustin Kirkland <email address hidden> Fri, 05 Feb 2010 09:51:58 -0800

Changed in eucalyptus (Ubuntu Lucid):
status: Triaged → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

fixed in cloud-init 0.5.3-0ubuntu1

Changed in cloud-init (Ubuntu Lucid):
status: Triaged → Fix Released
Scott Moser (smoser)
Changed in ec2-init (Ubuntu Lucid):
status: New → Fix Released
Chuck Short (zulcss)
Changed in cloud-init (Ubuntu Karmic):
status: New → Won't Fix
Changed in eucalyptus (Ubuntu Karmic):
status: New → Won't Fix
Scott Moser (smoser)
description: updated
Scott Moser (smoser)
Changed in cloud-init (Ubuntu Karmic):
status: Won't Fix → Invalid
Scott Moser (smoser)
description: updated
Revision history for this message
Scott Moser (smoser) wrote :

I'm in need of a sponsor for lp:~smoser/ubuntu/karmic/ec2-init/karmic-updates . That branch contains fixes for bug 520734 and bug 475354 .

Scott Moser (smoser)
Changed in ec2-init (Ubuntu Karmic):
assignee: nobody → Scott Moser (smoser)
importance: Undecided → Medium
milestone: none → karmic-updates
status: New → Fix Committed
Thierry Carrez (ttx)
Changed in ec2-init (Ubuntu Karmic):
assignee: Scott Moser (smoser) → Thierry Carrez (ttx)
status: Fix Committed → In Progress
Revision history for this message
Thierry Carrez (ttx) wrote :

Uploaded to -proposed

Changed in ec2-init (Ubuntu Karmic):
status: In Progress → Triaged
Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Accepted ec2-init into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in ec2-init (Ubuntu Karmic):
status: Triaged → Fix Committed
tags: added: verification-needed
Revision history for this message
Scott Moser (smoser) wrote :

I've tested this on karmic released uec image (20100121):

$ hostname
172
$ echo deb http://archive.ubuntu.com/ubuntu/ karmic-proposed restricted main multiverse universe | sudo tee /etc/apt/sources.list.d/proposed.list
$ sudo apt-get update
$ apt-cache policy ec2-init
ec2-init:
  Installed: 0.4.999-0ubuntu7.1
  Candidate: 0.4.999-0ubuntu7.2
$ sudo apt-get install ec2-init
...
$ hostname
ip-172-19-1-3

Then, just for good measure
$ sudo reboot
...
system comes back up, and prompt shows:
ubuntu@ip-172-19-1-3:~$

tags: added: verification-done
removed: verification-needed
Revision history for this message
Boris Devouge (bdevouge) wrote :

Ok, Verified and confirmed fixing.

I effectively get the right hostname without even a reboot.

Thanks !!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ec2-init - 0.4.999-0ubuntu7.2

---------------
ec2-init (0.4.999-0ubuntu7.2) karmic-proposed; urgency=low

  * fix install/upgrade failure if python2.4 is installed (LP: #520734)
  * set hostname to a hostname-like value (ip-W-X-Y-Z) if instance
    metadata includes a ip address rather than a hostname in
    local-hostname (LP: #475354)
 -- Scott Moser <email address hidden> Thu, 11 Feb 2010 20:35:41 -0500

Changed in ec2-init (Ubuntu Karmic):
status: Fix Committed → Fix Released
Changed in eucalyptus:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.