EC2 credentials are cached on disk

Bug #1638312 reported by Scott Moser on 2016-11-01
22
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Medium
Unassigned
cloud-init (Ubuntu)
Medium
Unassigned
Xenial
Medium
Unassigned
Yakkety
Medium
Unassigned

Bug Description

=== Begin SRU Template ===
[Impact]
On EC2, instance metadata can include credentials that remain valid for as
much as 6 hours. Reading these and allowing them to be pickled represents a
potential vulnerability if a snapshot of the disk is taken and shared as part
of an AMI.

The fix applied was simply to avoid reading the security credentials
in cloud-init.

[Test Case]
1. Launch an instance on Ec2.
2. Verify broken-ness by verifying 'security-credentials' exists in the
   pickled object in /var/lib/cloud/instance/obj.pkl
3. enable proposed, update, upgrade
4. clean instance
   rm -Rf /var/lib/cloud /var/log/cloud-init*
5. reboot
6. go back in and verify no 'security-credentials' are present.

[Regression Potential]
Low, but possible if someone was using the obj.pkl and expecting to
find credentials there. No one should be doing that.

Second possibility is if someone was using cloud-init's
get_instance_metadata and expected to have the security-credentials there.

=== End SRU Template ===
On EC2, instance metadata can include credentials that remain valid for as much
as 6 hours. Reading these and allowing them to be pickled represents a
potential vulnerability if a snapshot of the disk is taken and shared as part
of an AMI.

Note that:
a.) the simple fact of storing the credentials in a file that is readable only by root is not a serious problem as any attacker on the system has access to the network available data.
b.) General care needs to be taken for anyone "capturing" an ami and then making it public.

the suggested fix is to skip security-credentials when walking the meta-data tree.

Scott Moser (smoser) on 2016-11-01
information type: Private → Private Security
Changed in cloud-init:
status: New → Confirmed
importance: Undecided → Medium
Scott Moser (smoser) wrote :

Attaching the suggested fix per Andrew. I fixed flake8 complaints in the test, but that is all.

summary: - ec2 credentials cached on disk
+ EC2 credentials are cached on disk
Changed in cloud-init (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Scott Moser (smoser) wrote :

Andrew,
Please let me know if you want some path for this other than me making this a public bug and fixing.

Andrew Jorgensen (ajorgens) wrote :

Hi Scott,
Anthony Liguori will coordinate this with you.

Scott Moser (smoser) wrote :

Anthony, Andrew.
I'd fix this now, and we could start getting it back into 16.04 and the like, I'm waiting on you though.

Andrew Jorgensen (ajorgens) wrote :

I'm not sure why (or if?) folks on Amazon's side have dropped the ball here, but please go ahead and publish this fix at your convenience, if you haven't yet.

Seth Arnold (seth-arnold) wrote :

Has a CVE been assigned to this issue? Or does it fall into the category of "security hardening" and thus not qualify for a CVE?

Thanks

Andrew Jorgensen (ajorgens) wrote :

No CVE has been assigned, and in fact it seems only Amazon Linux was vulnerable, because Ubuntu and others were using an API version for instance metadata that did not include IAM instance credentials.

Scott Moser (smoser) on 2017-01-20
Changed in cloud-init:
status: Confirmed → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.9-19-ge987092-0ubuntu1

---------------
cloud-init (0.7.9-19-ge987092-0ubuntu1) zesty; urgency=medium

  * New upstream snapshot.
    - manual_cache_clean: When manually cleaning touch a file in instance dir.
    - Add tools/ds-identify to identify datasources available.
    - Fix small typo and change iso-filename for consistency [Robin Naundorf]
    - Fix eni rendering of multiple IPs per interface
      [Ryan Harper] (LP: #1657940)
    - tools/mock-meta: support python2 or python3 and ipv6 in both.
    - tests: remove executable bit on test_net, so it runs, and fix it.
    - tests: No longer monkey patch httpretty for python 3.4.2
    - Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
      [Lars Kellogg-Stedman] (LP: #1658174)
    - reset httppretty for each test [Lars Kellogg-Stedman] (LP: #1658200)
    - build: fix running Make on a branch with tags other than master
    - EC2: Do not cache security credentials on disk
      [Andrew Jorgensen] (LP: #1638312)
    - doc: Fix typos and clarify some aspects of the part-handler
      [Erik M. Bray]
    - doc: add some documentation on OpenStack datasource.
    - OpenStack: Use timeout and retries from config in get_data.
      [Lars Kellogg-Stedman] (LP: #1657130)
    - Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
    - Fix minor docs typo: perserve > preserve [Jeremy Bicha]
    - Use dnf instead of yum when available [Lars Kellogg-Stedman]
      (LP: #1647118)
    - validate-yaml: use python rather than explicitly python3
    - Get early logging logged, including failures of cmdline url.

 -- Scott Moser <email address hidden> Fri, 03 Feb 2017 21:54:39 -0500

Changed in cloud-init (Ubuntu):
status: Confirmed → Fix Released
Scott Moser (smoser) on 2017-03-03
Changed in cloud-init (Ubuntu Xenial):
status: New → Confirmed
Changed in cloud-init (Ubuntu Yakkety):
status: New → Confirmed
Changed in cloud-init (Ubuntu Xenial):
importance: Undecided → Medium
Changed in cloud-init (Ubuntu Yakkety):
importance: Undecided → Medium
Scott Moser (smoser) on 2017-03-03
description: updated
description: updated
Jon Grimm (jgrimm) wrote :

Are we able to turn this bug public?

Scott Moser (smoser) on 2017-03-04
information type: Private Security → Public

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-48-g1c795b9-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed
Chris Halse Rogers (raof) wrote :

Hello Scott, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-48-g1c795b9-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cloud-init (Ubuntu Yakkety):
status: Confirmed → Fix Committed
Scott Moser (smoser) wrote :

In order to see the security credentials, you must launch an instance with
iam role attached (--iam-instance-profile). Other wise they do not show up in the metadata service.

Ubuntu cloud-init did not actually show this bug because it read and stored
a version of the metadata service (2009-04-04) that does not have the iam/
credentials.

This can be verified simply by
$ sudo python3 -c 'from cloudinit.stages import _pkl_load as pl; import pprint; pprint.pprint(pl("/var/lib/cloud/instance/obj.pkl").metadata)'
{'ami-id': 'ami-78b7166e',
 'ami-launch-index': '0',
 'ami-manifest-path': '(unknown)',
 'block-device-mapping': {'ami': '/dev/sda1', 'root': '/dev/sda1'},
 'hostname': 'ip-10-0-0-160',
 'instance-action': 'none',
 'instance-id': 'i-00e90430c5dbe321c',
 'instance-type': 't2.micro',
 'local-hostname': 'ip-10-0-0-160',
 'local-ipv4': '10.0.0.160',
 'placement': {'availability-zone': 'us-east-1b'},
 'profile': 'default-hvm',
 'public-hostname': b'',
 'public-ipv4': '54.86.100.172',
 'public-keys': {'brickies': ['ssh-rsa '
                              'AAAAB3NzaC1yc2EAAAABIwAAAQEA3I7VUf2l5gSn5uavROsc5HRDpZdQueUq5ozemNSj8T7enqKHOEaFoU2VoPgGEWC9RyzSQVeyD6s7APMcE82EtmW4skVEgEGSbDc1pvxzxtchBj78hJP6Cf5TCMFSXw+Fz5rF1dR23QDbN1mkHs7adr8GW4kSWqU7Q7NDwfIrJJtO7Hi42GyXtvEONHbiRPOe8stqUly7MvUoN+5kfjBM8Qqpfl2+FNhTYWpMfYdPUnE7u536WqzFmsaqJctz3gBxH9Ex7dFtrxR4qiqEr9Qtlu3xGn7Bw07/+i1D+ey3ONkZLN+LQ714cgj8fRS4Hj29SCmXp5Kt5/82cD/VN3NtHw== '
                              'brickies']},
 'reservation-id': 'r-07f18d71dc1ebd0ab',
 'security-groups': 'wide-open'}

However, the get_instance_metadata() function would show the credentials.

$ python3 -c 'from cloudinit import ec2_utils; print(ec2_utils.get_instance_metadata("latest")["iam"])'
{'info': {'InstanceProfileArn': 'arn:aws:iam::950047163771:instance-profile/smtest-ec2-ro', 'InstanceProfileId': 'AIPAJ7VG3LHE4JOYDGNG6', 'Code': 'Success', 'LastUpdated': '2017-03-08T19:14:59Z'}, 'security-credentials': {'smtest-ec2-ro': {'Token': 'FQoDY...Bxgu=', 'AccessKeyId': 'XXXXXXXXXXXXXXXXXXXX', 'Expiration': '2017-03-09T01:43:31Z', 'Code': 'Success', 'SecretAccessKey': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'Type': 'AWS-HMAC', 'LastUpdated': '2017-03-08T19:14:40Z'}}}

Scott Moser (smoser) wrote :

# ami-78b7166e ubuntu/images-testing/hvm-ssd/ubuntu-yakkety-daily-amd64-server-20170307
$ ec2metadata --ami-id
ami-78b7166e

After upgrade..

$ dpkg-query --show cloud-init
cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.10.1

$ python3 -c 'from cloudinit import ec2_utils; print(ec2_utils.get_instance_metadata("latest")["iam"])'
{'info': {'InstanceProfileId': 'AIPAJ7VG3LHE4JOYDGNG6', 'InstanceProfileArn': 'arn:aws:iam::950047163771:instance-profile/smtest-ec2-ro', 'Code': 'Success', 'LastUpdated': '2017-03-08T19:33:11Z'}}

Notice that the 'security-credentials' dictionary is not present.

Scott Moser (smoser) wrote :

# ami-f4cc1de2 ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20170221
$ ec2metadata --ami-id
ami-f4cc1de2

$ dpkg-query --show cloud-init
cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.04.1

$ python3 -c 'from cloudinit import ec2_utils; print(ec2_utils.get_instance_metadata("latest")["iam"])'
{'info': {'Code': 'Success', 'InstanceProfileId': 'AIPAJ7VG3LHE4JOYDGNG6', 'InstanceProfileArn': 'arn:aws:iam::950047163771:instance-profile/smtest-ec2-ro', 'LastUpdated': '2017-03-08T19:15:50Z'}}

Notice that the 'security-credentials' entry is missing from the 'info' dict.

tags: added: verification-done-xenial verification-done-yakkety
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.9-48-g1c795b9-0ubuntu1~16.04.1

---------------
cloud-init (0.7.9-48-g1c795b9-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * debian/rules: install Z99-cloudinit-warnings.sh to /etc/profile.d
  * debian/patches/ds-identify-behavior-xenial.patch: adjust default
    behavior of ds-identify for SRU (LP: #1669675, #1660385).
  * New upstream snapshot.
    - Support warning if the used datasource is not in ds-identify's list
      (LP: #1669675).
    - DatasourceEc2: add warning message when not on AWS. (LP: #1660385)
    - Z99-cloudinit-warnings: Add profile.d script for showing warnings on
    - Z99-cloud-locale-test.sh: convert tabs to spaces, remove unneccesary
      execute bit in permissions.
    - (RedHat) net: correct errors in cloudinit/net/sysconfig.py
      [Lars Kellogg-Stedman]
    - ec2_utils: fix MetadataLeafDecoder that returned bytes on empty
    - Fix eni rendering of multiple IPs per interface [Ryan Harper]
      (LP: #1657940)
    - Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
      [Lars Kellogg-Stedman]
    - EC2: Do not cache security credentials on disk [Andrew Jorgensen]
      (LP: #1638312)
    - OpenStack: Use timeout and retries from config in get_data.
      [Lars Kellogg-Stedman] (LP: #1657130)
    - Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
    - (RedHat) Use dnf instead of yum when available [Lars Kellogg-Stedman]
    - Get early logging logged, including failures of cmdline url.
    - test / doc / build environment changes
      - Remove style checking during build and add latest style checks to
        tox [Joshua Powers]
      - code-style: make master pass pycodestyle (2.3.1) cleanly, currently
        [Joshua Powers]
      - Fix small typo and change iso-filename for consistency
      - tools/mock-meta: support python2 or python3 and ipv6 in both.
      - tests: remove executable bit on test_net, so it runs, and fix it.
      - tests: No longer monkey patch httpretty for python 3.4.2
      - reset httppretty for each test [Lars Kellogg-Stedman]
      - build: fix running Make on a branch with tags other than master
      - doc: Fix typos and clarify some aspects of the part-handler
        [Erik M. Bray]
      - doc: add some documentation on OpenStack datasource.
      - Fix minor docs typo: perserve > preserve [Jeremy Bicha]
      - validate-yaml: use python rather than explicitly python3

 -- Scott Moser <email address hidden> Mon, 06 Mar 2017 16:34:10 -0500

Changed in cloud-init (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cloud-init - 0.7.9-48-g1c795b9-0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-48-g1c795b9-0ubuntu1~16.10.1) yakkety; urgency=medium

  * debian/rules: install Z99-cloudinit-warnings.sh to /etc/profile.d
  * debian/patches/ds-identify-behavior-yakkety.patch: adjust default
    behavior of ds-identify for SRU (LP: #1669675, #1660385).
  * New upstream snapshot.
    - Support warning if the used datasource is not in ds-identify's list
      (LP: #1669675).
    - DatasourceEc2: add warning message when not on AWS. (LP: #1660385)
    - Z99-cloudinit-warnings: Add profile.d script for showing warnings on
    - Z99-cloud-locale-test.sh: convert tabs to spaces, remove unneccesary
      execute bit in permissions.
    - (RedHat) net: correct errors in cloudinit/net/sysconfig.py
      [Lars Kellogg-Stedman]
    - ec2_utils: fix MetadataLeafDecoder that returned bytes on empty
    - Fix eni rendering of multiple IPs per interface [Ryan Harper]
      (LP: #1657940)
    - Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
      [Lars Kellogg-Stedman]
    - EC2: Do not cache security credentials on disk [Andrew Jorgensen]
      (LP: #1638312)
    - OpenStack: Use timeout and retries from config in get_data.
      [Lars Kellogg-Stedman] (LP: #1657130)
    - Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
    - (RedHat) Use dnf instead of yum when available [Lars Kellogg-Stedman]
    - Get early logging logged, including failures of cmdline url.
    - test / doc / build environment changes
      - Remove style checking during build and add latest style checks to
        tox [Joshua Powers]
      - code-style: make master pass pycodestyle (2.3.1) cleanly, currently
        [Joshua Powers]
      - Fix small typo and change iso-filename for consistency
      - tools/mock-meta: support python2 or python3 and ipv6 in both.
      - tests: remove executable bit on test_net, so it runs, and fix it.
      - tests: No longer monkey patch httpretty for python 3.4.2
      - reset httppretty for each test [Lars Kellogg-Stedman]
      - build: fix running Make on a branch with tags other than master
      - doc: Fix typos and clarify some aspects of the part-handler
        [Erik M. Bray]
      - doc: add some documentation on OpenStack datasource.
      - Fix minor docs typo: perserve > preserve [Jeremy Bicha]
      - validate-yaml: use python rather than explicitly python3

 -- Scott Moser <email address hidden> Mon, 06 Mar 2017 16:37:28 -0500

Changed in cloud-init (Ubuntu Yakkety):
status: Fix Committed → Fix Released

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers