[OSSA 2015-021] secgroup rules doesn't work for instance immediately (CVE-2015-7713)

[suntao]

I have an OpenStack kilo setup on RHEL7.1 with a controller and a compute node (network-compute + network-network)，the config is following:

# /etc/nova.nova.conf on contrller node
[DEFAULT]
network_api_class = nova.network.api.API
security_group_api = nova

# /etc/nova/nova.conf on compute node
[DEFAULT]
network_api_class = nova.network.api.API
security_group_api = nova
firewall_driver = nova.virt.libvirt.firewall.IptablesFirewallDriver
network_manager = nova.network.manager.FlatDHCPManager
network_size = 254
allow_same_net_traffic = False
multi_host = True
send_arp_for_ha = True
share_dhcp_address = True
force_dhcp_release = True
flat_network_bridge = br100
flat_interface = eth0
public_interface = eth0

steps for test 1:
1) create and start VM instance-1 with secgroup default;
2) VM instance-1 ping br100: OK;
3) br100 ping VM instance-1: operation not permitted (because of no secgroup-rules for ICMP)
4) nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
5) br100 ping VM instance-1: i got the same wrong message, not expected.

steps for test 2:
1) nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0;
2) create and start VM instance-2 with secgroup default;
3) br100 ping instance-2: OK

It seems that command "nova secgroup-add-rule ..." doesn't work immediately for the existed or running VM instances?

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

So does new nova net security-group-rules are supposed to be applied on existing instance ?

[Jeremy Stanley (fungi)]

I'm switching this bug to public security since it's been independently reported in public security bug 1492961 today.

[Sreekumar S (sreesiv)]

>> So does new nova net security-group-rules are supposed to be applied on existing instance ?

I believe it should, especially important in cases where new rules are added to make the group more restrictive. The admin would think that his rule is applied and safe, whereas actually he needs to manually re-associate that group with 'one of the instances' for it to be applied across all VMs. More details can be seen in the public security bug 1492961 raised be me.

I think it has to do something with the sec group rules not being refreshed in the code path... "refresh_security_group_rules" func not being called I suppose!

I am new to OpenStack and would like to work on investigating and fixing this issue. I've assigned this to myself. Request reviewers to confirm and discuss the vulnerabilities and possible solutions.

[Tristan Cacqueray (tristan-cacqueray)]

Can someone familiar with nova-net security group please confirm the issue here ?

[Sreekumar S (sreesiv)]

I've found the root cause of this issue.

The call chain goes like this...

In "./nova/compute/api.py"
add_rules() AND remove_rules() calls
 trigger_rules_refresh() which in turn calls refresh_instance_security_rules() [RPC]

This ends up in ./nova/compute/manager.py
[RPC] _ComputeV4Proxy::refresh_instance_security_rules() which calls ComputeManager::refresh_instance_security_rules(). But the manager's function decorated with '@object_compat' calls decorated_function()
  which calls _load_instance(kwargs['instance']). This fails since there is no 'instance' key there and once again calls _load_instance from the except catch...
        except KeyError:
            args = (_load_instance(args[0]),) + args[1:]

From there it calls objects.Instance._from_db_object() with argument 'expected_attrs=metas' which is initialized to
metas = ['metadata', 'system_metadata']

The db instance dict doesn't have the keys in 'metas' because in trigger_rules_refresh() the sec groups are got from db by joining on the instances column, but it doesn't join on the metadata/system_metadata fields.

This again causes 'KeyError' because when db instance dict is converted to the Instance object, it expects fields that aren't in the dict. So the manager's function do not call refresh_instance_security_rules() on the LibvirtDriver and thereby IptablesFirewallDriver.

This same issue is mentioned in bug 1484738, although the end problems they cause differ. I've verified the fix proposed by Matt Riedemann and it resolves the issue.

More details: https://bugs.launchpad.net/nova/+bug/1484738/comments/11

Why this is not repro'ed in Liberty and why can't those changes be backported: https://bugs.launchpad.net/nova/+bug/1484738/comments/13

More details in commit msg: https://bugs.launchpad.net/nova/+bug/1484738/comments/17

So once the fix for bug 1484738 is merged to stable/kilo, it will resolve this issue as well.

[Matt Riedemann (mriedem)]

Per comment 5, this sounds like a duplicate of bug 1484738. Before comment 5 I was going to ask if there were any errors in the logs, but comment 5 seems to confirm that. So I wouldn't consider this a security issue per se, it's just a code bug that was preventing things from working properly and is now fixed in master (liberty) and backport to stable/kilo and stable/juno. As of this morning it's merged in stable/juno and the fix in stable/kilo is going through the gate.

[Jeremy Stanley (fungi)]

It sounds like the extent of the risk here is that a user may extend a security group with a change to block some malicious traffic or close off a vulnerable service, then assume they're protected from it without double-checking.

[Matt Riedemann (mriedem)]

Looking at the 2 security group APIs in nova, in the case of nova-network when you add a new secgroup rule it gets into the db but doesn't get down to the virt driver which actually does the iptables stuff (in the case of libvirt) if you hit the KeyError when it calls refresh_instance_security_rules. That failure doesn't get back to the user because the nova-network secgroup API does a cast to refresh the rules. So, yeah, the operation from the CLI would not return an error, but the rules aren't applied for the guest if you hit the KeyError, so you might have a false sense of security.

[Matt Riedemann (mriedem)]

I wouldn't consider Neutron affected by this since it doesn't do the cast to refresh_instance_security_rules.

[Sreekumar S (sreesiv)]

I am new to OpenStack and Python in general, so didn't understand the 'cast' part.

But as I mentioned in the bug I raised (bug 1492961, which was marked duplicate to this one) I've verified the same with neutron network and is working fine there. So it's nova-network only and is resolved with mriedem's changes to bug 1484738.

[Tristan Cacqueray (tristan-cacqueray)]

Since it's quite a security defect, and already fixed in supported stable release, I suggest we issue an OSSA with this impact description:

Title: Nova network security group changes are not applied to running instances
Products: Nova
Affects: versions through 2014.2.3, and 2015.1 versions through 2015.1.1

Description:
Security group changes silently fails to be applied to already running instances, potentially resulting in instances not being protected by security group. All Nova network setups are affected.

Now about attributions, It seems like this was independently reported by suntao, Sreekumar S. Are those your name and is there affiliations to mention along or are you independent contributors ?

[Sreekumar S (sreesiv)]

Yes, 'Sreekumar S' is my name, and I am an independent contributor.

[Matt Riedemann (mriedem)]

Tristan, this was fixed under bug 1484738 but the OSSA is on this bug - how do we resolve that? Should the OSSA move to 1484738?

[Tristan Cacqueray (tristan-cacqueray)]

Matt, that is fine to me, this bug describe the security impact of bug 1484738, and I propose to use it for the OSSA tasks. The advisory can reference the former bug 1484738.

@suntao, can you confirm your attribution is correct:

Title: Nova network security group changes are not applied to running instances
Reporter: Sreekumar S and Suntao
Products: Nova
Affects: versions through 2014.2.3, and 2015.1 versions through 2015.1.1

Description:
Sreekumar S and Suntao independently reported a vulnerability in Nova network. Security group changes silently fails to be applied to already running instances, potentially resulting in instances not being protected by security group. All Nova network setups are affected.

[Tristan Cacqueray (tristan-cacqueray)]

Oups, forgot to update the affect line:

Affects: <=2014.2.3, >=2015.1.0, <=2015.1.1

[Grant Murphy (gmurphy)]

+1 with affects line update

[Jeremy Stanley (fungi)]

Also a slight phrasing improvement to suggest over the second sentence of Tristan's impact description from comment #14:

"Security group changes silently fail to be applied to already running instances, potentially resulting in instances not being protected by the security group."

Otherwise looks good to me.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks, CVE requested with:

Title: Nova network security group changes are not applied to running instances
Reporter: Sreekumar S and Suntao
Products: Nova
Affects: <=2014.2.3, >=2015.1.0, <=2015.1.1

Description:
Sreekumar S and Suntao independently reported a vulnerability in Nova network. Security group changes silently fail to be applied to already running instances, potentially resulting in instances not being protected by the security group. All Nova network setups are affected.