GZipped user-data causes 500 error to user requests

[Andy Botting]

I believe it's a valid use case for users to supply their user-data gzipped, especially in EC2 where user-data is limited to 16 KB.

In the case of users supplying gzipped user-data on an OpenStack instance, requests to http://169.254.169.254/latest/ (and others) will fail with a 500 error.

The ec2api metadata service then logs the following stacktrace:

ERROR ec2api.metadata [req-c6bfe852-4c50-4980-af97-75d18636b8ed - - - - -] Unexpected error.
ERROR ec2api.metadata Traceback (most recent call last):
ERROR ec2api.metadata File "/usr/lib/python2.7/dist-packages/ec2api/metadata/__init__.py", line 94, in __call__
ERROR ec2api.metadata resp = self._get_metadata(path_tokens, requester)
ERROR ec2api.metadata File "/usr/lib/python2.7/dist-packages/ec2api/metadata/__init__.py", line 273, in _get_metadata
ERROR ec2api.metadata requester['private_ip'])
ERROR ec2api.metadata File "/usr/lib/python2.7/dist-packages/ec2api/metadata/api.py", line 154, in get_metadata_item
ERROR ec2api.metadata os_instance_id, remote_ip)
ERROR ec2api.metadata File "/usr/lib/python2.7/dist-packages/ec2api/metadata/api.py", line 254, in _build_metadata
ERROR ec2api.metadata userdata = userdata.decode("utf-8")
ERROR ec2api.metadata File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
ERROR ec2api.metadata return codecs.utf_8_decode(input, errors, True)
ERROR ec2api.metadata UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte

[Andrey Pavlov (apavlov-e)]

@Andy,

1) does AWS instances allows gzipped user-data? I think - no. It defines user data as "you must provide base64-encoded text."

2) does nova itself allows to pass gzipped user-data? I also think - no.

3) does AWS protocol allows gzipped data?
4) Is it possible to pass gzipped data via CLI/UI?

ec2 must be compatible with AWS protocol but it allows some extensions due to OpenStack nature.
In this case I suggest that this feature can't be implemented.

[Andy Botting (andybotting)]

Hi Audrey.

Thanks for your comments.

I just had a quick Google, and came up with these couple of references from Amazon:

https://aws.amazon.com/blogs/developer/best-practices-for-local-file-parameters/ (See bottom EC2 User Data section)
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/amazon-linux-ami-basics.html (See Supported User-Data Formats, under the cloud-init heading)

They both suggest that cloud-init is the important factor here (not necessarily AWS) when dealing with GZip'd user-data, and cloud-init CAN handle it, if present.

My understanding is AWS/OpenStack Nova/whatever doesn't actually need to work with user-data itself, only store and serve it, so it's possible by not really dealing with it in any way.

So from the links above, I'd say:

1) does AWS instances allows gzipped user-data? Yes, cloud-init specifically deals with it (in Amazon Linux even).

2) does nova itself allows to pass gzipped user-data? Yes, I tested it by bypassing EC2API in our test cloud. Works.

3) does AWS protocol allows gzipped data? Yes, by actually allowing any data format, as long as the underlying processing handles it, and in the universal case of cloud-init, Yes.

4) Is it possible to pass gzipped data via CLI/UI? Yes.
I was testing with something like: 'nova boot --image xxx --flavor xxx --user-data user-data.yaml.gz testvm'
And my user-data.yaml before gzipping looked like:

#cloud-config
merge_type: 'list(append)+dict(recurse_array)+str()'
ssh_authorized_keys:
  - ssh-rsa AAAAB3NzaC1......

I hope I've supplied enough evidence here to demonstrate this bug should not be marked as 'Invalid'.

[Andrey Pavlov (apavlov-e)]

ok. I didn't understand issue correctly.

user-data is still base64 encoded in request but this encoded message contains gzipped data.

[Andy Botting (andybotting)]

So we're still hitting this issue.

The problem isn't so much that gzipped payloads aren't supported, but more that we're getting 500 errors returned to users when they attempt it.

It would be great to at least have this exception handled.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ec2-api 12.0.0.0rc1 release candidate.