EC2 credentials are cached on disk

[Scott Moser]

=== Begin SRU Template ===
[Impact]
On EC2, instance metadata can include credentials that remain valid for as
much as 6 hours. Reading these and allowing them to be pickled represents a
potential vulnerability if a snapshot of the disk is taken and shared as part
of an AMI.

The fix applied was simply to avoid reading the security credentials
in cloud-init.

[Test Case]
1. Launch an instance on Ec2.
2. Verify broken-ness by verifying 'security-credentials' exists in the
   pickled object in /var/lib/cloud/instance/obj.pkl
3. enable proposed, update, upgrade
4. clean instance
   rm -Rf /var/lib/cloud /var/log/cloud-init*
5. reboot
6. go back in and verify no 'security-credentials' are present.

[Regression Potential]
Low, but possible if someone was using the obj.pkl and expecting to
find credentials there. No one should be doing that.

Second possibility is if someone was using cloud-init's
get_instance_metadata and expected to have the security-credentials there.

=== End SRU Template ===
On EC2, instance metadata can include credentials that remain valid for as much
as 6 hours. Reading these and allowing them to be pickled represents a
potential vulnerability if a snapshot of the disk is taken and shared as part
of an AMI.

Note that:
a.) the simple fact of storing the credentials in a file that is readable only by root is not a serious problem as any attacker on the system has access to the network available data.
b.) General care needs to be taken for anyone "capturing" an ami and then making it public.

the suggested fix is to skip security-credentials when walking the meta-data tree.

[Scott Moser (smoser)]

Attaching the suggested fix per Andrew. I fixed flake8 complaints in the test, but that is all.

[Scott Moser (smoser)]

Andrew,
Please let me know if you want some path for this other than me making this a public bug and fixing.

[Andrew Jorgensen (ajorgens)]

Hi Scott,
Anthony Liguori will coordinate this with you.

[Scott Moser (smoser)]

Anthony, Andrew.
I'd fix this now, and we could start getting it back into 16.04 and the like, I'm waiting on you though.

[Andrew Jorgensen (ajorgens)]

I'm not sure why (or if?) folks on Amazon's side have dropped the ball here, but please go ahead and publish this fix at your convenience, if you haven't yet.

[Seth Arnold (seth-arnold)]

Has a CVE been assigned to this issue? Or does it fall into the category of "security hardening" and thus not qualify for a CVE?

Thanks

[Andrew Jorgensen (ajorgens)]

No CVE has been assigned, and in fact it seems only Amazon Linux was vulnerable, because Ubuntu and others were using an API version for instance metadata that did not include IAM instance credentials.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-19-ge987092-0ubuntu1

---------------
cloud-init (0.7.9-19-ge987092-0ubuntu1) zesty; urgency=medium

  * New upstream snapshot.
    - manual_cache_clean: When manually cleaning touch a file in instance dir.
    - Add tools/ds-identify to identify datasources available.
    - Fix small typo and change iso-filename for consistency [Robin Naundorf]
    - Fix eni rendering of multiple IPs per interface
      [Ryan Harper] (LP: #1657940)
    - tools/mock-meta: support python2 or python3 and ipv6 in both.
    - tests: remove executable bit on test_net, so it runs, and fix it.
    - tests: No longer monkey patch httpretty for python 3.4.2
    - Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
      [Lars Kellogg-Stedman] (LP: #1658174)
    - reset httppretty for each test [Lars Kellogg-Stedman] (LP: #1658200)
    - build: fix running Make on a branch with tags other than master
    - EC2: Do not cache security credentials on disk
      [Andrew Jorgensen] (LP: #1638312)
    - doc: Fix typos and clarify some aspects of the part-handler
      [Erik M. Bray]
    - doc: add some documentation on OpenStack datasource.
    - OpenStack: Use timeout and retries from config in get_data.
      [Lars Kellogg-Stedman] (LP: #1657130)
    - Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
    - Fix minor docs typo: perserve > preserve [Jeremy Bicha]
    - Use dnf instead of yum when available [Lars Kellogg-Stedman]
      (LP: #1647118)
    - validate-yaml: use python rather than explicitly python3
    - Get early logging logged, including failures of cmdline url.

 -- Scott Moser <email address hidden> Fri, 03 Feb 2017 21:54:39 -0500

[Jon Grimm (jgrimm)]

Are we able to turn this bug public?

[Chris Halse Rogers (raof)]

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-48-g1c795b9-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris Halse Rogers (raof)]

Hello Scott, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-48-g1c795b9-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

In order to see the security credentials, you must launch an instance with
iam role attached (--iam-instance-profile). Other wise they do not show up in the metadata service.

Ubuntu cloud-init did not actually show this bug because it read and stored
a version of the metadata service (2009-04-04) that does not have the iam/
credentials.

This can be verified simply by
$ sudo python3 -c 'from cloudinit.stages import _pkl_load as pl; import pprint; pprint.pprint(pl("/var/lib/cloud/instance/obj.pkl").metadata)'
{'ami-id': 'ami-78b7166e',
 'ami-launch-index': '0',
 'ami-manifest-path': '(unknown)',
 'block-device-mapping': {'ami': '/dev/sda1', 'root': '/dev/sda1'},
 'hostname': 'ip-10-0-0-160',
 'instance-action': 'none',
 'instance-id': 'i-00e90430c5dbe321c',
 'instance-type': 't2.micro',
 'local-hostname': 'ip-10-0-0-160',
 'local-ipv4': '10.0.0.160',
 'placement': {'availability-zone': 'us-east-1b'},
 'profile': 'default-hvm',
 'public-hostname': b'',
 'public-ipv4': '54.86.100.172',
 'public-keys': {'brickies': ['ssh-rsa '
                              'AAAAB3NzaC1yc2EAAAABIwAAAQEA3I7VUf2l5gSn5uavROsc5HRDpZdQueUq5ozemNSj8T7enqKHOEaFoU2VoPgGEWC9RyzSQVeyD6s7APMcE82EtmW4skVEgEGSbDc1pvxzxtchBj78hJP6Cf5TCMFSXw+Fz5rF1dR23QDbN1mkHs7adr8GW4kSWqU7Q7NDwfIrJJtO7Hi42GyXtvEONHbiRPOe8stqUly7MvUoN+5kfjBM8Qqpfl2+FNhTYWpMfYdPUnE7u536WqzFmsaqJctz3gBxH9Ex7dFtrxR4qiqEr9Qtlu3xGn7Bw07/+i1D+ey3ONkZLN+LQ714cgj8fRS4Hj29SCmXp5Kt5/82cD/VN3NtHw== '
                              'brickies']},
 'reservation-id': 'r-07f18d71dc1ebd0ab',
 'security-groups': 'wide-open'}

However, the get_instance_metadata() function would show the credentials.

$ python3 -c 'from cloudinit import ec2_utils; print(ec2_utils.get_instance_metadata("latest")["iam"])'
{'info': {'InstanceProfileArn': 'arn:aws:iam::950047163771:instance-profile/smtest-ec2-ro', 'InstanceProfileId': 'AIPAJ7VG3LHE4JOYDGNG6', 'Code': 'Success', 'LastUpdated': '2017-03-08T19:14:59Z'}, 'security-credentials': {'smtest-ec2-ro': {'Token': 'FQoDY...Bxgu=', 'AccessKeyId': 'XXXXXXXXXXXXXXXXXXXX', 'Expiration': '2017-03-09T01:43:31Z', 'Code': 'Success', 'SecretAccessKey': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'Type': 'AWS-HMAC', 'LastUpdated': '2017-03-08T19:14:40Z'}}}

[Scott Moser (smoser)]

# ami-78b7166e ubuntu/images-testing/hvm-ssd/ubuntu-yakkety-daily-amd64-server-20170307
$ ec2metadata --ami-id
ami-78b7166e

After upgrade..

$ dpkg-query --show cloud-init
cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.10.1

$ python3 -c 'from cloudinit import ec2_utils; print(ec2_utils.get_instance_metadata("latest")["iam"])'
{'info': {'InstanceProfileId': 'AIPAJ7VG3LHE4JOYDGNG6', 'InstanceProfileArn': 'arn:aws:iam::950047163771:instance-profile/smtest-ec2-ro', 'Code': 'Success', 'LastUpdated': '2017-03-08T19:33:11Z'}}

Notice that the 'security-credentials' dictionary is not present.

[Scott Moser (smoser)]

# ami-f4cc1de2 ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20170221
$ ec2metadata --ami-id
ami-f4cc1de2

$ dpkg-query --show cloud-init
cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.04.1

$ python3 -c 'from cloudinit import ec2_utils; print(ec2_utils.get_instance_metadata("latest")["iam"])'
{'info': {'Code': 'Success', 'InstanceProfileId': 'AIPAJ7VG3LHE4JOYDGNG6', 'InstanceProfileArn': 'arn:aws:iam::950047163771:instance-profile/smtest-ec2-ro', 'LastUpdated': '2017-03-08T19:15:50Z'}}

Notice that the 'security-credentials' entry is missing from the 'info' dict.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-48-g1c795b9-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-48-g1c795b9-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/rules: install Z99-cloudinit-warnings.sh to /etc/profile.d
  * debian/patches/ds-identify-behavior-xenial.patch: adjust default
    behavior of ds-identify for SRU (LP: #1669675, #1660385).
  * New upstream snapshot.
    - Support warning if the used datasource is not in ds-identify's list
      (LP: #1669675).
    - DatasourceEc2: add warning message when not on AWS. (LP: #1660385)
    - Z99-cloudinit-warnings: Add profile.d script for showing warnings on
    - Z99-cloud-locale-test.sh: convert tabs to spaces, remove unneccesary
      execute bit in permissions.
    - (RedHat) net: correct errors in cloudinit/net/sysconfig.py
      [Lars Kellogg-Stedman]
    - ec2_utils: fix MetadataLeafDecoder that returned bytes on empty
    - Fix eni rendering of multiple IPs per interface [Ryan Harper]
      (LP: #1657940)
    - Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
      [Lars Kellogg-Stedman]
    - EC2: Do not cache security credentials on disk [Andrew Jorgensen]
      (LP: #1638312)
    - OpenStack: Use timeout and retries from config in get_data.
      [Lars Kellogg-Stedman] (LP: #1657130)
    - Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
    - (RedHat) Use dnf instead of yum when available [Lars Kellogg-Stedman]
    - Get early logging logged, including failures of cmdline url.
    - test / doc / build environment changes
      - Remove style checking during build and add latest style checks to
        tox [Joshua Powers]
      - code-style: make master pass pycodestyle (2.3.1) cleanly, currently
        [Joshua Powers]
      - Fix small typo and change iso-filename for consistency
      - tools/mock-meta: support python2 or python3 and ipv6 in both.
      - tests: remove executable bit on test_net, so it runs, and fix it.
      - tests: No longer monkey patch httpretty for python 3.4.2
      - reset httppretty for each test [Lars Kellogg-Stedman]
      - build: fix running Make on a branch with tags other than master
      - doc: Fix typos and clarify some aspects of the part-handler
        [Erik M. Bray]
      - doc: add some documentation on OpenStack datasource.
      - Fix minor docs typo: perserve > preserve [Jeremy Bicha]
      - validate-yaml: use python rather than explicitly python3

 -- Scott Moser <email address hidden> Mon, 06 Mar 2017 16:34:10 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 0.7.9-48-g1c795b9-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-48-g1c795b9-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/rules: install Z99-cloudinit-warnings.sh to /etc/profile.d
  * debian/patches/ds-identify-behavior-yakkety.patch: adjust default
    behavior of ds-identify for SRU (LP: #1669675, #1660385).
  * New upstream snapshot.
    - Support warning if the used datasource is not in ds-identify's list
      (LP: #1669675).
    - DatasourceEc2: add warning message when not on AWS. (LP: #1660385)
    - Z99-cloudinit-warnings: Add profile.d script for showing warnings on
    - Z99-cloud-locale-test.sh: convert tabs to spaces, remove unneccesary
      execute bit in permissions.
    - (RedHat) net: correct errors in cloudinit/net/sysconfig.py
      [Lars Kellogg-Stedman]
    - ec2_utils: fix MetadataLeafDecoder that returned bytes on empty
    - Fix eni rendering of multiple IPs per interface [Ryan Harper]
      (LP: #1657940)
    - Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
      [Lars Kellogg-Stedman]
    - EC2: Do not cache security credentials on disk [Andrew Jorgensen]
      (LP: #1638312)
    - OpenStack: Use timeout and retries from config in get_data.
      [Lars Kellogg-Stedman] (LP: #1657130)
    - Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
    - (RedHat) Use dnf instead of yum when available [Lars Kellogg-Stedman]
    - Get early logging logged, including failures of cmdline url.
    - test / doc / build environment changes
      - Remove style checking during build and add latest style checks to
        tox [Joshua Powers]
      - code-style: make master pass pycodestyle (2.3.1) cleanly, currently
        [Joshua Powers]
      - Fix small typo and change iso-filename for consistency
      - tools/mock-meta: support python2 or python3 and ipv6 in both.
      - tests: remove executable bit on test_net, so it runs, and fix it.
      - tests: No longer monkey patch httpretty for python 3.4.2
      - reset httppretty for each test [Lars Kellogg-Stedman]
      - build: fix running Make on a branch with tags other than master
      - doc: Fix typos and clarify some aspects of the part-handler
        [Erik M. Bray]
      - doc: add some documentation on OpenStack datasource.
      - Fix minor docs typo: perserve > preserve [Jeremy Bicha]
      - validate-yaml: use python rather than explicitly python3

 -- Scott Moser <email address hidden> Mon, 06 Mar 2017 16:37:28 -0500

[Scott Moser (smoser)]

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2750