Can't login with SSH to Lucid UEC image - uec metadata service returns a string instead of a list of string for ssh public keys

Bug #506332 reported by Thierry Carrez on 2010-01-12
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Eucalyptus
Incomplete
Undecided
chris grzegorczyk
ec2-init (Ubuntu)
High
Scott Moser
Lucid
High
Scott Moser
eucalyptus (Ubuntu)
Low
Unassigned
Lucid
Low
Unassigned

Bug Description

Binary package hint: ec2-init

Testing 20100112 UEC image (amd64) on UEC
Instance starts OK, by when trying to connect through SSH, my key is refused:

ttx@odyssey:~$ euca-run-instances -k mykey emi-6F0C1333 -t m1.large
RESERVATION r-3ABF0772 admin admin-default
INSTANCE i-367506EE emi-6F0C1333 0.0.0.0 0.0.0.0 pending mykey 2010-01-12T10:05:12.947Z eki-9E481847 eri-803C17BC
ttx@odyssey:~$ ssh -i mykey.priv ubuntu@192.168.0.230
Permission denied (publickey).

Running the karmic UEC image on the same UEC setup succeeds, so it's something in the lucid image.

Thierry Carrez (ttx) wrote :

Note sure if it also affects the cloud image on EC2.

Changed in ec2-init (Ubuntu):
assignee: nobody → Scott Moser (smoser)
importance: Undecided → High
milestone: none → lucid-alpha-2
Thierry Carrez (ttx) wrote :

The Lucid / 20100108.1 image works alright.

Thierry Carrez (ttx) wrote :

euca-get-console-output stops before "ec2: Generating public/private rsa key pair." on the affected images.

Scott Moser (smoser) wrote :

just a point of reference, I just tried
ami-f73cd19e ubuntu-images-testing-us/ubuntu-lucid-daily-i386-server-20100112.manifest.xml
and was able to ssh to it.

Thierry Carrez (ttx) wrote :

Confirmed regression in UEC:
20100111.1 FAIL
20100111 PASS
So it's more than probably linked to ec2-init 0.5.0

Scott Moser (smoser) wrote :

It looks like this is a difference in the metadata service on eucalyptus and ec2.
python -c 'import boto.utils; import pprint; pprint.pprint(boto.utils.get_instance_metadata())'

get_instance_metadata() is used by ec2-init 0.5.0 to crawl metadata service. The objects look different. For reference, I'll attach them here.

Changed in ec2-init (Ubuntu):
status: New → In Progress
Scott Moser (smoser) wrote :

Please see build 0.5.0-0ubuntu3~smoser2 at https://launchpad.net/~smoser/+archive/ppa/+packages?field.name_filter=ec2-init&field.status_filter=published&field.series_filter=lucid

You can test this by booting your 20100111 instance that boots, and
- installing new ec2-init deb
- rm ~ubuntu/.ssh/authorized_keys
- reboot

Scott Moser (smoser) wrote :
Mathias Gug (mathiaz) wrote :

As outlined in comment 7 and 8:

On ec2:

 'public-keys': {'ec2-keypair.us-east-1': ['ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCD9dlT00vOUC8Ttq6YH8RzUCVqPQl6HaSfWSTKYnZiVCpTBj1CaRZPLRLmkSB9Nziy4aRJa/LZMbBHXytQKnB1psvNknqC2UNlrXXMk+Vx5S4vg21MXYYimK4uZEY0Qz29QUiTyNsx18jpAaF4ocUpTpRhxPEBCcSCDmMbc27MU2XuTbasM2NjW/w0bBF3ZFhdH68dZICXdTxS2jUrtrCnc1D/QXVZ5kQO3jsmSyJg8E0nE+6Onpx2YRoVRSwjpGzVZ+BlXPnN5xBREBG8XxzhNFHJbek+RgK5TfL+k4yD4XhnVZuZu53cBAFhj+xPKhtisSd+YmaEq+Jt9uS0Ekd5 ec2-keypair.us-east-1',
                                           '']},
On UEC:

 'public-keys': {'mykey': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCbw9Pxu9Wq7v32vEYuGlzB4qYX6bnIt+/zIe8mdO0Eyt+smus1PYMic9FGuSrc9tUJyMHeNbXRvqQEqQdfGJb1KlCuSBl7kqUQwnVZj++jYWRCmcgzxhBPtCmZAniqo8+XFIOuigydkiipYPcnQ7DEEcXNmRb3I/4XENG+caIoA5+BwaBx3+si+XiwC8C8IKs0ZQ+IDbu0RySVp9RcSiE/kuuYrdoPaToGA7mf6oQMX2RrY7aAUnVqI+nm7cK+20GP8xoHSUm3Ohg/ioDglETwxZD5kzkdgtzHoNdvD3CBq3iBJM0Nd/o1OL9qK375kLdC0XwnnuXZBCCeumWWhwIX admin@eucalyptus'},

summary: - Can't login with SSH to Lucid UEC image
+ Can't login with SSH to Lucid UEC image - uec metadata service returns a
+ string instead of a list of string for ssh public keys
Scott Moser (smoser) wrote :

Not as relevant here, but to point out, there are other differences in the 2 files:
 - uec has 'ancestor-ami-ids' field (probably ok)
 - ec2 has instance-action
 - uec has 'product-codes' . apparently if there are none, ec2 doesn't present that in the metadata
 - the noted public-keys difference

chris grzegorczyk (chris-grze) wrote :

#12:

ancestor-ami-ids are also in ec2 if the image has ancestors.
instance-action is not supported by Eucalyptus at this time (so it is absent).

#11:
The difference between the two outputs posted seems to be a parser issue which is converting the public-key string into a list. Note the second empty string entry in the example EC2 output.

Changed in eucalyptus:
assignee: nobody → chris grzegorczyk (chris-grze)
status: New → Incomplete
Thierry Carrez (ttx) wrote :

Even if the bug is in boto parsing, there is something different in UEC metadata output, compared to EC2, that makes boto behave differently ?

Changed in eucalyptus (Ubuntu Lucid):
importance: Undecided → Medium
status: New → Confirmed
Thierry Carrez (ttx) wrote :

ec2-init (0.5.0-0ubuntu3) lucid; urgency=low

  * work around difference in uec/ec2 metadata service (LP:506332)
 -- Scott Moser < <email address hidden>> Tue, 12 Jan 2010 11:33:11 -0500

Changed in ec2-init (Ubuntu Lucid):
status: In Progress → Fix Released
Scott Moser (smoser) wrote :

yes, obviously there is something different in the server metadata output that makes the client parse it different. by definition of api compatibility, that is a bug in the server. I agree that the parsed eucalyptus data seems more sane, but thats not something that can be changed.

Thierry Carrez (ttx) on 2010-01-15
Changed in eucalyptus (Ubuntu Lucid):
importance: Medium → Low

Actually, this is not part of any of the EC2 specifications.

If there is a need to change the behaviour of Eucalyptus to remedy
this problem I will need more information about what differs between
the two. That is, the provided output indicates there is a
difference, but neither is the actual output and so I can't tell what
needs to be changed.

c

On Wed, Jan 13, 2010 at 6:05 AM, Scott Moser <email address hidden> wrote:
> yes, obviously there is something different in the server metadata
> output that makes the client parse it different.  by definition of api
> compatibility, that is a bug in the server.  I agree that the parsed
> eucalyptus data seems more sane, but thats not something that can be
> changed.
>
> --
> Can't login with SSH to Lucid UEC image - uec metadata service returns a string instead of a list of string for ssh public keys
> https://bugs.launchpad.net/bugs/506332
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: Incomplete
> Status in “ec2-init” package in Ubuntu: Fix Released
> Status in “eucalyptus” package in Ubuntu: Confirmed
> Status in “ec2-init” source package in Lucid: Fix Released
> Status in “eucalyptus” source package in Lucid: Confirmed
>
> Bug description:
> Binary package hint: ec2-init
>
> Testing 20100112 UEC image (amd64) on UEC
> Instance starts OK, by when trying to connect through SSH, my key is refused:
>
> ttx@odyssey:~$ euca-run-instances -k mykey emi-6F0C1333 -t m1.large
> RESERVATION     r-3ABF0772      admin   admin-default
> INSTANCE        i-367506EE      emi-6F0C1333    0.0.0.0 0.0.0.0 pending mykey   2010-01-12T10:05:12.947Z        eki-9E481847    eri-803C17BC
> ttx@odyssey:~$ ssh -i mykey.priv ubuntu@192.168.0.230
> Permission denied (publickey).
>
> Running the karmic UEC image on the same UEC setup succeeds, so it's something in the lucid image.
>
>
>

--
Chris Grzegorczyk
Co-Founder and Engineer
Eucalyptus Systems, Inc.

130 Castilian St. | Goleta, CA | 93117
Office: 805-968-1400 x e^1 | Cell: 805-807-8237
Email: <email address hidden>
www.eucalyptus.com
________________________________________

Thierry Carrez (ttx) wrote :

We workarounded the issue in ec2-init by supporting both formats. The issue still stands in eucalyptus, but it's no longer blocking for us (so set to importance Low). It's more about other tools that might query the metadata service and could run into the same issue as we did. Even if it's not written into any EC2 spec, it's still something where Eucalyptus output differs from EC2's one.

chris grzegorczyk (chris-grze) wrote :

Just to clarify, I am not objecting to fixing the issue in Eucalyptus.
 Rather, the information provided in this bug report is not enough for
me to fix it (and I would need someone to provide that information for
me as I don't have an EC2 account). The HTTP response itself would be
good enough.

thanks.
chris

On Mon, Jan 18, 2010 at 11:36 PM, Thierry Carrez
<email address hidden> wrote:
> We workarounded the issue in ec2-init by supporting both formats. The
> issue still stands in eucalyptus, but it's no longer blocking for us (so
> set to importance Low). It's more about other tools that might query the
> metadata service and could run into the same issue as we did. Even if
> it's not written into any EC2 spec, it's still something where
> Eucalyptus output differs from EC2's one.
>
> --
> Can't login with SSH to Lucid UEC image - uec metadata service returns a string instead of a list of string for ssh public keys
> https://bugs.launchpad.net/bugs/506332
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: Incomplete
> Status in “ec2-init” package in Ubuntu: Fix Released
> Status in “eucalyptus” package in Ubuntu: Confirmed
> Status in “ec2-init” source package in Lucid: Fix Released
> Status in “eucalyptus” source package in Lucid: Confirmed
>
> Bug description:
> Binary package hint: ec2-init
>
> Testing 20100112 UEC image (amd64) on UEC
> Instance starts OK, by when trying to connect through SSH, my key is refused:
>
> ttx@odyssey:~$ euca-run-instances -k mykey emi-6F0C1333 -t m1.large
> RESERVATION     r-3ABF0772      admin   admin-default
> INSTANCE        i-367506EE      emi-6F0C1333    0.0.0.0 0.0.0.0 pending mykey   2010-01-12T10:05:12.947Z        eki-9E481847    eri-803C17BC
> ttx@odyssey:~$ ssh -i mykey.priv ubuntu@192.168.0.230
> Permission denied (publickey).
>
> Running the karmic UEC image on the same UEC setup succeeds, so it's something in the lucid image.
>
>
>

--
Chris Grzegorczyk
Co-Founder and Engineer
Eucalyptus Systems, Inc.

130 Castilian St. | Goleta, CA | 93117
Office: 805-968-1400 x e^1 | Cell: 805-807-8237
Email: <email address hidden>
www.eucalyptus.com
________________________________________

Dustin Kirkland  (kirkland) wrote :

Okay, so what remains to be fixed in Ubuntu's Eucalyptus package for this bug?

I see a request from Chris for more information such that Eucalyptus could solve this.

I'm marking this bug incomplete, for now, as it doesn't appear there's enough information to solve the Eucalyptus part of this bug (if it exists there at all). Please update the status if you have more or better information than me.

Changed in eucalyptus (Ubuntu Lucid):
status: Confirmed → Incomplete
Thierry Carrez (ttx) on 2010-03-11
Changed in eucalyptus (Ubuntu Lucid):
status: Incomplete → Won't Fix
tags: added: patch
Mackenzie Morgan (maco.m) wrote :

Debdiff looks sane but is for a package in main, so subscribing sponsors.

Dave Walker (davewalker) wrote :

Marking Invalid for the Eucalyptus (ubuntu) task, as it seems to be resolved elsewhere. Please re-open if this is not the case.

Thanks.

Changed in eucalyptus (Ubuntu):
status: Incomplete → Invalid
Andy Grimm (agrimm) wrote :

This issue is now being tracked upstream at http://eucalyptus.atlassian.net/browse/EUCA-2674

Please watch that issue for further updates.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers