User data is not parsed correctly by Eucalyptus in some cases

[Markus Lindenberg]

User data supplied using "euca-run-instances --user-data-file" is not decoded before being presented to the instance.
Inside the instance, "curl http://169.254.169.254/latest/user-data" should fetch the decoded user data, whereas eucalyptus will return a base64 and url encoded string. This breaks ec2-run-user-data from the ec2-init package, rendering instance configuration using the user-data mechanism unusable.

EC2 documentation at http://docs.amazonwebservices.com/AWSEC2/2007-03-01/DeveloperGuide/AESDG-chapter-instancedata.html states that EC2 will return decoded data (i don't have a ec2 account so i can't confirm this):
"The user data must be base64-encoded before being submitted to the API. The API command-line tools perform the base64-encoding for you. The data will be base64 decoded before being presented to the instance."

=================
SRU Report (eucalyptus, euca2ools):

Impact: This bug makes userdata unusable in cloud images used withing UEC. userdata is used for a lot of things, in particular boot-time configuration of our cloud images. This works within EC2 but not within UEC, due to this bug.

This requires a fix in euca2ools (do not b64_encode twice). But fixing it in euca2ools triggers a bug in eucalyptus when certain userdata is received (the previous bug was protecting eucalyptus from this), so this needs a eucalyptus update as well.

Fix in development release: This was fixed in lucid in eucalyptus (1.6.1~bzr1083-0ubuntu1) and in euca2ools (1.0+bzr20091007-0ubuntu2) by applying the same patches.

Minimal patch for eucalyptus:
http://bazaar.launchpad.net/~ubuntu-core-dev/eucalyptus/ubuntu-karmic/revision/726

Minimal patch for euca2ools:
--- euca2ools-1.0+bzr20091007.orig/bin/euca-run-instances
+++ euca2ools-1.0+bzr20091007/bin/euca-run-instances
@@ -170,8 +170,6 @@
                  print 'Invalid user data file path'
                  sys.exit(1)
   user_data = read_user_data(user_data_file)
- if user_data:
- user_data = base64.urlsafe_b64encode(user_data)
         euca_conn = euca.make_connection()
  try:
             reservation = euca_conn.run_instances(image_id = image_id,

TEST CASE:
euca-run-instances -k $MYKEY --user-data " << FOO >" $EMI -t c1.medium
ssh -i $MYKEYPRIV ubuntu@$IP 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
Expected results: should return " << FOO >"
Fails with affected euca2ools and eucalyptus (returns "IDw8IEZPTyA-" instead of " << FOO >")
Succeeds with proposed euca2ools and proposed eucalyptus.

Regression potential:
The regression potential is small, since userdata is not really usable right now. In euca2ools, only someone relying on the bug (and base64_decoding the userdata in the cloud image itself) would be affected. Regression potential is slightly higher on eucalyptus side, since the fix is about escaping special characters in userdata. Careful testing with various userdata strings (to hit the special characters in the urlsafe-base64-encoded string) is necessary.

=================
Karmic release notes:

user-data not usable by guest instances

Starting an instance with euca-run-instances and user-data (either using the -d, --user-data option or the -f, --user-data-file option) will store the user data in base64 encoding. Accessing the user data from the instance at http://169.254.169.254/latest/user-data will return the user data in base64 encoding. Because of this bug ec2-init is unable make use of user-data. In order to use this data it must first be decoded. A fix for this issue is expected to be provided in a post-release update immediately after the Ubuntu 9.10 release.

==================

[Scott Moser (smoser)]

Confirming this:
$ printf "%s\n%s\n" '#!/bin/sh' 'echo hello world" > /tmp/user-data.sh
$ euca-run-instances -k mykey --user-data-file /tmp/user-data.sh emi-220011A6 -t m1.small
$ euca-describe-instances
RESERVATION r-342805FE admin default
INSTANCE i-3BB70743 emi-220011A6 192.168.1.192 172.19.1.2 running mykey 0 m1.small 2009-10-26T15:21:07.472Z sm-zone-1a eki-3EE316B1 eri-225F1633
$ ssh -i mykey.priv ubuntu@192.168.1.192 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
IyEvYmluL3NoCmVjaG8gaGVsbG8gd29ybGQK

The above should show plaintext.
$ dpkg-query --show "euca*"
euca2ools 1.0+bzr20091007-0ubuntu1
eucalyptus-cc 1.6~bzr931-0ubuntu7
eucalyptus-cloud 1.6~bzr931-0ubuntu7
eucalyptus-common 1.6~bzr931-0ubuntu7
eucalyptus-gl 1.6~bzr931-0ubuntu7
eucalyptus-java-common 1.6~bzr931-0ubuntu7
eucalyptus-javadeps
eucalyptus-sc 1.6~bzr931-0ubuntu7
eucalyptus-walrus 1.6~bzr931-0ubuntu7

[Thierry Carrez (ttx)]

Please also think of a workaround in ec2-init if fixing eucalyptus is no longer a possibility.

[Neil Soman (neilsoman)]

------------------------------------------------------------
revno: 942
committer: Neil
branch nick: 1.6
timestamp: Mon 2009-10-26 09:36:30 -0700
message:
  fixes #461156
------------------------------------------------------------

[Thierry Carrez (ttx)]

In my reproduction test I retrieve a string that is base64'd two times:

$ ssh -i mykey.priv ubuntu@192.168.0.230 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
SXlFdlltbHVMM05vQ21WamFHOGdQVDA5UFQwOVBUMDlJRWhGVEV4UElEMDlQVDA50

Which base64-decodes to:
IyEvYmluL3NoCmVjaG8gPT09PT09PT09IEhFTExPID09PT09

Which base-64 decodes to:
#!/bin/sh
echo ========= HELLO =====

[Scott Moser (smoser)]

I believe I've verified that using euca-run-instances base64 encodes data 2 times rather than once.
For example, with euca2ools configured to run against ec2, I do:

$ euca-run-instances --user-data "hello world" --key ec2-keypair ami-ef00e386
$ ssh ${ec2_host} 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
aGVsbG8gd29ybGQ=
$ ssh $ec2_host 'wget -q http://169.254.169.254/latest/user-data -O -' |
   base64 -d; echo
hello world

This is true for both '--user-data' and '--user-data-file'.

[Scott Moser (smoser)]

A bit more data...
I'm fairly sure that the correct thing to do here is just to remove the following two lines from main() in euca-run-instances:
| if user_data:
| user_data = base64.urlsafe_b64encode(user_data)

Also 'import base64' on line 37 could then be removed as this is the only place base64 is used.

The reason we dont need this encoding is that boto is doing it for us.

In euca-run-instances, the following are the lines that matter:

| if user_data:
| user_data = base64.urlsafe_b64encode(user_data)
| euca_conn = euca.make_connection()
| try:
| reservation = euca_conn.run_instances(image_id = image_id,
| min_count = min_count,
| max_count = max_count,
| key_name = keyname,
| security_groups = group_names,
| user_data = user_data,
...

euca.make_connection in euca2ools/euca2ools/__init__.py returns a connection that is created with:
   return boto.connect_ec2()

Then, we call euca_conn.run_instances. run_instances in boto/ec2/connection.py does:
| if user_data:
| params['UserData'] = base64.b64encode(user_data)

So, the user data is getting encoded twice.

[Neil Soman (neilsoman)]

Turns out it is an issue with euca2ools and not eucalyptus. reverting.

------------------------------------------------------------
revno: 943
committer: Neil
branch nick: 1.6
timestamp: Mon 2009-10-26 10:52:44 -0700
message:
  reverting invalid fix in revno 942 (was not a bug).
------------------------------------------------------------

[Neil Soman (neilsoman)]

See #461301

[Thierry Carrez (ttx)]

From duplicate bug 461301:
------------------------------------------------------------
revno: 222
committer: Neil
branch nick: euca2ools-1.0
timestamp: Mon 2009-10-26 11:00:47 -0700
message:
  fixes #461301
------------------------------------------------------------

[Thierry Carrez (ttx)]

From bug 461922 (which discusses a fix that did not make it into Ubuntu yet):
=====================================
After applying the [proposed] fix, euca-run-instances will fail depending on the user-data input. I found this while trying to provide a shell script as user data input. The bug is triggered by the --user-data parameter as well as providing user-data from a file using --user-data-file.

lindenm@dyn-220:~/cloud$ euca-run-instances -k markuskey -t c1.medium --user-data " << FO >" --kernel eki-245816A6 --ramdisk eri-05D01618 emi-092D1197
RESERVATION r-46BB06FB admin admin-default
INSTANCE i-47980954 emi-092D1197 0.0.0.0 0.0.0.0 pending markuskey 2009-10-27T12:37:51.99Z eki-245816A6 eri-05D01618

lindenm@dyn-220:~/cloud$ euca-run-instances -k markuskey -t c1.medium --user-data "<< FOO >" --kernel eki-245816A6 --ramdisk eri-05D01618 emi-092D1197
RESERVATION r-28AE050D admin admin-default
INSTANCE i-4EFD095C emi-092D1197 0.0.0.0 0.0.0.0 pending markuskey 2009-10-27T12:44:14.422Z eki-245816A6 eri-05D01618

lindenm@dyn-220:~/cloud$ euca-run-instances -k markuskey -t c1.medium --user-data " << FOO >" --kernel eki-245816A6 --ramdisk eri-05D01618 emi-092D1197
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

I understand that euca-run-instances should work with any sort of user-data regardless of it's content. ec2-run-instances works flawlessly with eucalyptus in this case.

Note the used input strings:

"<< FOO >" (no space at the beginning, works)
" << FO >" (space, two character word, works)
" << FOO >" (space at the beginning, three character word, fails)
=================================

[Neil Soman (neilsoman)]

I am unable to reproduce this problem.

euca-run-instances -k mykey --kernel eki-0212156F --ramdisk eri-604B16B8 emi-97BE1397 -z mycluster2 --user-data " << FOO >" --addressing private
RESERVATION r-405607B1 admin admin-default
INSTANCE i-45F30812 emi-97BE1397 0.0.0.0 0.0.0.0 pending mykey 2009-10-27T17:06:34.508Z eki-0212156F eri-604B16B8

euca2ools revno 222, boto-1.8d.

[Markus Lindenberg (markusl)]

Hmm. I was using the latest karmic euca2ools package (bzr200) w/ just euca-run-instances "fixed". If anything else related to user data handling changed in between, this might be the cause. I'll be able to look into that tomorrow.

[Scott Moser (smoser)]

I'm also unable to reproduce Markus's problems with --user-data ' << FOO >' and a patched euca2ools.

I've built a ppa build of euca2ools with the patch at https://launchpad.net/~smoser/+archive/ppa .

I tested that the patch works against ec2 with user data of " << FOO >", and that it fails against eucalyptus as Markus said.

The issue is that boto (used by euca2ools) is encoding the supplied user-data in connection.py:run_instances with:
| base64.urlsafe_b64encode(user_data)

For many input strings it is true that:
  base64.b64decode(base64.urlsafe_b64encode(str)) == str

For '<< FOO >' it is not. It will throw an error.

My guess is that eucalyptus is taking the request from euca2ools and doing a b64decode rather than urlsafe_b64decode on it. This is throwing error and being returned to the user as a parse error.

Thus, I'm fairly sure that eucalyptus will have to be changed to expect user_data to be encoded via urlsafe_b64decode() rather than b64decode().

urlsafe_b64encode is explained at http://docs.python.org/library/base64.html as:
| Encode string s using a URL-safe alphabet, which substitutes - instead of +
| and _ instead of / in the standard Base64 alphabet. The result can still contain =.

I suspect that the eucalyptus

[Scott Moser (smoser)]

I just verified that ec2-api-tools work with such user data when pointed at a eucalyptus server:
$ ec2-run-instances -k mykey --user-data " << FOO >" emi-247011C0
RESERVATION r-45F60808 admin admin-default
INSTANCE i-523209EF emi-247011C0 0.0.0.0 0.0.0.0 pending mykey 0
                m1.small 2009-10-27T19:37:32+0000 sm-zone-1a eki-41E516CA eri-253A1644 monitoring-false
$ euca-run-instances -k mykey --user-data " << FOO >" emi-247011C0
Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
Failure: 403 Forbidden

$ dpkg-query --show euca2ools
euca2ools 1.0+bzr20091007-0ubuntu1.1~ppa

I also verified that inside the instance created by ec2-run-instances the meta-data server has the correct value for user-data.

[Scott Moser (smoser)]

I'm attaching a snippit from /var/log/eucalyptus/cloud-error.log, that occurs when you run the failing euca-run-instances command above.

[Neil Soman (neilsoman)]

Scott, sorry I don't understand what you are saying.

You said "I'm also unable to reproduce Markus's problems with --user-data ' << FOO >' and a patched euca2ools."

But you are still running into the "User authentication failed" issue?

I cannot reproduce this against the upsteam (current euca2ools or eucalyptus).

thanks
neil

[Scott Moser (smoser)]

On Tue, 27 Oct 2009, Neil Soman wrote:

> You said "I'm also unable to reproduce Markus's problems with --user-
> data ' << FOO >' and a patched euca2ools."

That should have said "*able* to reproduce" . I do see it.

To reproduce, you have to
a.) fix euca2ools so that it doesn't doubly encode user-data
    hand-patch the euca-run-instances, or use my ppa build

b.) run: ec2-run-instances -k mykey --user-data " << FOO >" ${EMI}

The result is:
| Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
| EC2ResponseError: 403 Forbidden
| Failure: 403 Forbidden

And the attached log message in cloud-error.log.

[Neil Soman (neilsoman)]

Yep, I confirm that it is a bug against eucalyptus (not euca2ools). I was using an older version of euca2ools that did double encoding of the user data.

thanks.

[Mathias Gug (mathiaz)]

To fix this issue an SRU is needed for karmic:

1. fix euca-run-instances to only encode user-data once to mimic what ec2-run-instances does
2. fix eucalyptus soap|rest interface to not fail when user-data is sent by euca-run-instances.

[Steve Langasek (vorlon)]

Documented at <https://wiki.ubuntu.com/KarmicKoala/ReleaseNotes#UEC%20user-data%20not%20usable%20by%20guest%20instances>:

When user data is passed to an instance started with {{{euca-run-instances}}} (using either the {{{-d}}}, {{{--user-data}}} option or the {{{-f}}}, {{{--user-data-file}}} option), the data returned at {{{http://169.254.169.254/latest/user-data}}} will be base64-encoded. {{{ec2-init}}} is unable make use of this user data because it must be decoded before use. A fix for this issue is expected to be provided in a post-release update immediately after the Ubuntu 9.10 release. (Bug:461156)

[Thierry Carrez (ttx)]

Test case A:
A1 euca-run-instances -k $MYKEY --user-data "FOO" $EMI -t c1.medium
A2 ssh -i $MYKEYPRIV ubuntu@$IP 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
Expected results: A2 should return "FOO"

Test case B:
B1 euca-run-instances -k $MYKEY --user-data " << FOO >" $EMI -t c1.medium
B2 ssh -i $MYKEYPRIV ubuntu@$IP 'wget -q http://169.254.169.254/latest/user-data -O -'; echo
Expected results: B2 should return " << FOO >"

With unpatched euca2ools and unpatched eucalyptus:
Test A fails with A2 returning "Rk9P" instead of "FOO"
Test B fails with B2 returning "IDw8IEZPTyA-" instead of " << FOO >"

With patched euca2ools (from smoser PPA) and unpatched eucalyptus:
Test A : PASS
Test B fails with B1 returning "403 Forbidden"

Note that python-boto does encode UserData using base64.b64encode (not urlsafe_b64encode as suggested in comment 13). The 403 Forbidden issue probably comes from URL decoding hitting special characters in HmacV2Handler.java, since they are no longer "protected" by double base64 encoding. Usage of canonicalString.replaceAll("\\+","%20") in there seems particularly relevant, since " << FOO >" translates to "IDw8IEZPTyA+"

[Scott Moser (smoser)]

> Note that python-boto does encode UserData using base64.b64encode
> (not urlsafe_b64encode as suggested in comment 13).

Thierry is correct. The commenter in comment 13 above must have been staring at a computer screen too long. boto uses base64.b64encode everywhere that i can see: http://code.google.com/p/boto/source/browse/trunk/boto/ec2/connection.py .

[Thierry Carrez (ttx)]

The "eucalyptus" part of this bug might well be bug 477776.

[Dustin Kirkland  (kirkland)]

Thierry/Scott-

Are there proposed fixes for this issue, at this point?

[chris grzegorczyk (chris-grze)]

@Thierry
- #21 the change of "+" is only for the purposes of computing the hmac, not processing the message contents.
- #23 the bug referenced is not related to this issue.

There look to be (at least) two issues here:
1) double base64'd data
2) authentication failure w/ certain user data strings

#1 was fixed in the euca2ools, afaict. #2 no longer manifests with any of the examples listed in this thread.

Is there still a pending issue here and, if so, is there a test case?

cheers.
chris

[chris grzegorczyk (chris-grze)]

------------------------------------------------------------
revno: 1079
committer: decker <decker@personal-army>
branch nick: 1.6
timestamp: Fri 2009-11-20 23:33:29 -0800
message:
  fixes authentication problem w/ userdata lp:#461156
------------------------------------------------------------

[Thierry Carrez (ttx)]

Fix committed to lp:~ttx/eucalyptus/karmic-sru2
Test with: eucalyptus 1.6~bzr931-0ubuntu7.4~ppa1 in https://launchpad.net/~ttx/+archive/ppa

With patched euca2ools (from smoser PPA) and patched eucalyptus (from ttx PPA):
Test A : PASS
Test B : PASS

[Thierry Carrez (ttx)]

Eucalyptus part of the fix now committed to lp:~ubuntu-core-dev/eucalyptus/ubuntu-karmic

[Launchpad Janitor (janitor)]

This bug was fixed in the package eucalyptus - 1.6.1~bzr1083-0ubuntu1

---------------
eucalyptus (1.6.1~bzr1083-0ubuntu1) lucid; urgency=low

  [ Dustin Kirkland ]
  * Merge upstream bzr revision 1082; the following bugs have been fixed
    upstream since the last merge:
    - LP: #378969 - private bug
    - LP: #404842 - init script fix
    - LP: #434283 - existing keys should be overwritten unconditionally
    - LP: #445990 - run instance will fail if no kernel or ramdisk specified
    - LP: #447457 - euca_conf --register-sc ... check the number of parameters
    - LP: #449874 - fix incorrect help text (--delete-nodes doesn't exist)
    - LP: #451795 - show registered images in elastic fox
    - LP: #454405 - return correct networkIndex values on describeInstances
    - LP: #456877 - init script fix
    - LP: #456878 - fix for libvirt xen driver
    - LP: #460085 - fix rampart memory leak
    - LP: #461156 - fix authentication problem w/ userdata
    - LP: #461394 - fix multiple concurrent snapshots on the same volume
    - LP: #461444 - fix memory leaks in NC getConsoleOutput and startup_thread
    - LP: #469984 - fix iptables rules issue
    - LP: #477776 - fix query string authentication
    - LP: #480783 - allow api connection over https
    - LP: #482249 - fix "Describe Regions"
    - LP: #484217 - create keypair should return an error if key exists
    - LP: #490623 - parse RFC 1123 formatted datetime
  * debian/control:
    - make all package lists one-per-line (makes changes henceforth more
      readable), sort lists
    - depend on rampart >= 1.3.0-0ubuntu6, which fixes some shared library
      installation issues
  * debian/patches/04-axis2c-1.6.0-rampart-1.3.0.patch: drop this patch,
    since Eucalyptus 1.6.1 natively supports axis2c 1.6.0 now
  * debian/eucalyptus-cloud.install,
    debian/eucalyptus-common.eucalyptus.upstart,
    debian/eucalyptus-java-common.install, debian/eucalyptus-sc.install,
    debian/eucalyptus-walrus.install: update static version number strings
    from "1.6-devel" to "1.6.1"; (we should really find a better way to do
    this)
  * debian/patches/03-DESTDIR.patch: ported forward for merge
 -- Dustin Kirkland <email address hidden> Tue, 01 Dec 2009 21:09:28 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package euca2ools - 1.0+bzr20091007-0ubuntu2

---------------
euca2ools (1.0+bzr20091007-0ubuntu2) lucid; urgency=low

  [ Scott Moser ]
  * Do not base64 encode user-data in euca-run-instances, as boto takes
    care of that (LP: #461156)
 -- Thierry Carrez <email address hidden> Wed, 02 Dec 2009 10:08:14 +0100

[Martin Pitt (pitti)]

Accepted eucalyptus into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Pitt (pitti)]

Accepted euca2ools into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Dustin Kirkland  (kirkland)]

Thierry-

Can you verify this package?

We need other fixes associated with this update here onsite at a partner.

[Thierry Carrez (ttx)]

Verification:
With karmic / karmic-updates : FAIL
With karmic-proposed (eucalyptus + euca2ools): PASS

[Launchpad Janitor (janitor)]

This bug was fixed in the package euca2ools - 1.0+bzr20091007-0ubuntu1.1

---------------
euca2ools (1.0+bzr20091007-0ubuntu1.1) karmic-proposed; urgency=low

  * Do not base64 encode user-data in euca-run-instances, as boto takes
    care of that (LP: #461156)
 -- Scott Moser <email address hidden> Mon, 26 Oct 2009 20:38:04 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package eucalyptus - 1.6~bzr931-0ubuntu7.4

---------------
eucalyptus (1.6~bzr931-0ubuntu7.4) karmic-proposed; urgency=low

  [ Thierry Carrez ]
  * cluster/handlers.c: Cherrypick upstream r946: initialize axis2c in the CC
    for each NC client connection, to avoid rampart memory leak (LP: #460085)
  * clc/modules/wsstack/src/main/java/com/eucalyptus/ws/handlers/HmacV2Handler.java:
    Cherrypick upstream r1079: Fix authentication issue when using a euca2ools
    that doesn't double base64encode userdata (LP: #461156)

  [ Dustin Kirkland ]
  * debian/eucalyptus-cc.upstart, debian/eucalyptus-common.eucalyptus.upstart:
    support CLEAN=1 on start/stop/restart of eucalyptus/euclayptus-cc; export
    the CLEAN env variable in eucalyptus.init, and handle it in both the
    pre-start and post-stop sections of eucalyptus-cc, (LP: #491254)
 -- Dustin Kirkland <email address hidden> Wed, 02 Dec 2009 17:58:18 -0600

[Scott Moser (smoser)]

Fairly sure this is fix-released in Eucalyptus, right?

[Dustin Kirkland  (kirkland)]

Should be, yes.