Fetching metadata via LB may result with wrong instance data

[Kobi Samoray]

While querying metadata from an instance via a loadbalancer, metadata service relies on X-Metadata-Provider to identify the correct instance by querying Neutron for subnets which are attached to the loadbalancer.
Then the subnet result is used to identify the instance by querying for ports which are attached to the subnets above.
Yet, when the first query result is empty due to deletion, bug or any other reason within the Neutron side, this may cause a security vulnerability, as Neutron will retrieve ports of _any_ instance which has the same IP address as the instance which is queried.
That could compromise key pairs and other sensitive data.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

I'm curious, how would an attacker go about obtaining an instance with an IP address which had no record in the metadata server? You mention deletion specifically as a way to trigger this, but what does the exploit scenario look like in detail (steps to reproduce)?

I guess this is specifically an issue with RFC 1918 private and/or IPv6 linklocal type addresses, which could exist in on multiple networks in different tenants? Less so with global addressing where the environment will attempt to prevent more than one instance/interface having the same address.

[Kobi Samoray (ksamoray)]

Nova API's metadata service can identify the requesting instance with several possible identifiers:
- OpenStack/Nova instance id, coded into X-Instance-ID header within the metadata HTTP request.
* This is what vanilla OpenStack deployments use *
This bug is irrelevant in this case.

- Use the combination of X-Metadata-Provider and X-Forwarded-For to query Neutron, and conclude the instance id. This is where the bug above is relevant.
A metadata provider is a router is set by an L3 object (e.g router) which is connected to the network, where the requesting instance is connected.
Within a cloud which has overlapping subnets, the combination of a router, which cannot be connected to two subnets that overlap (indeed mostly private IPs and link locals), and the instance IP - which is stored in X-Forwarded-For header, should be unique and identify the requesting instance.
However, if Neutron, due to a bug, responds with an empty list of subnets, Nova metadata service doesn't stop but still queries for ports of instances with the requesting instance's IP - of _any_network_.
So if the requesting instance of tenant A has IP X, it might receive the metadata of another instance, of tenant B, which also uses the same IP.
In the case above, tenant A could acquire SSH keys of tenant B - this is not the classic attacker case but more of a leak of confidential information from one tenant to the other.

[Jeremy Stanley (fungi)]

Under what circumstances would SSH keys be stored in instance metadata? I'm aware of reasons for people to place PUBLIC exponents corresponding to private SSH keys there, but not the actual private SSH keys. Keep in mind OpenStack already cautions in https://wiki.openstack.org/wiki/OSSN/OSSN-0074 that "Nova metadata service should not be used for sensitive information," so while unauthorized access to another instance's metadata is definitely worth fixing it's probably not as high of a risk as it sounds.

Between the above and the lack of clear exploit scenario for this bug (an attacker would need to be able to trigger another unspecified Neutron bug to leverage this), I think we can dispense with the overhead of an embargo and continue with broader discussion in public. If there is no objection to that plan, I'll switch the bug to public as a security hardening opportunity late this week.

[Kobi Samoray (ksamoray)]

In that case let's treat this as a regular bug.

[Jeremy Stanley (fungi)]

Reviewing our taxonomy, this probably fits closest as a class C1 report due to depending on the existence of other exploitable bugs to leverage. https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[Gage Hugo (gagehugo)]

Agreed on C1 for this.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679247
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=31773715687326c92d4ad46ebb32b14645bbc614
Submitter: Zuul
Branch: master

commit 31773715687326c92d4ad46ebb32b14645bbc614
Author: Kobi Samoray <email address hidden>
Date: Tue Aug 27 13:49:05 2019 +0300

    Avoid fetching metadata when no subnets found

    Metadata service uses the provider id to identify the requesting
    instance.
    However, when provider query doesn't find any networks, the request
    should fail.
    The same goes to the case where multiple ports are found.

    Closes-Bug: #1841933
    Change-Id: I8ce3703ca86a3a0769edd42a790d82796d1071d7