Iptables can allow open access to instances in certain situations

[Sam Morrison]

Have been debugging some instance hacks on our cloud and we have tracked it down to nova setting some iptables to open access to all instances.

Environment:
We are using neutron and have reproduced this on Newton and Ocata, it looks like Mitaka is not affected

nova.conf options:
use_neutron=True
linuxnet_interface_driver = nova.network.linux_net.NeutronLinuxBridgeInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver

neutron is using ML2/linuxbridge

Steps to reproduce:
# Clear out all iptables rules
iptables -F; iptables -X; iptables -t nat -F; iptables -t nat -X; iptables -t mangle -F; iptables -t mangle -X; iptables -t raw -F; iptables -t raw -X
# restart neutron linuxbridge
systemctl restart neutron-linuxbridge-agent
# Wait a couple of seconds for rules to be added
sleep 5
# Restart nova-compute
systemctl restart nova-compute

If you look in iptables you will see in the FORWARD chain:

-A FORWARD -j nova-filter-top
-A FORWARD -j nova-compute-FORWARD
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-linuxbri-FORWARD

The nova-compute-FORWARD chain will have
-A nova-compute-FORWARD -i brq34bf77e6-d8 -j ACCEPT
-A nova-compute-FORWARD -o brq34bf77e6-d8 -j ACCEPT
-A nova-compute-FORWARD -o brq34bf77e6-d8 -j DROP

It seems that once neutron agent does something on this host, eg. a new instance gets booted/deleted etc. then the rules change to:

-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-linuxbri-FORWARD
-A FORWARD -j nova-filter-top
-A FORWARD -j nova-compute-FORWARD

So we don't see this much, just on quiet compute nodes it would seem.

I feel like nova shouldn't be doing any iptables stuff so I'm scared I've got some config wrong but I haven't found anything so far and it looks the same as our Mitaka hosts which don't see this issue.

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Sam Morrison (sorrison)]

Just wondering if I can get some feedback on this, have tried in IRC but no luck (plus I'm in Australian TZ so that is usual :-) )

[Jeremy Stanley (fungi)]

I've reached out privately in IRC to the three Nova core security reviewers asking them to take a look. (The silence is unfortunately unlikely to be a TZ issue as two of the three are also on your continent and the third is in an EMEA TZ.)

[Tony Breeds (o-tony)]

I think I understand your comments, and the open access id due to the re-ordering of the 4 rules in the FORWARD chain that is to say the nova-compute-FORWARD still contains the correct rule but we're never getting to it. Did I understand correctly?

If not can we get the rest of the FORWARD chains added to this bug?

[Sam Morrison (sorrison)]

I've been doing some digging and have found that this issue is due to the addition of the privsep vif_plug_linux_bridge module.

In mitaka there are no nova security group chains or rules at all. In newton there are and this module is the one creating them. Because we are using neutron there shouldn't be any nova secgroup chains or rules right?

[Sam Morrison (sorrison)]

So after some more digging it looks as if the privsep vif_plug_linux_bridge module was new in Newton and this is the reason we're now getting nova-* iptables rules.

There were no nova-* iptables rules in Mitaka.

I think this is due to the vif_plug_linux_bridge not taking into account the DEFAULT/firewall_driver config option which for us is set to noop because we use neutron.

[John Garbutt (johngarbutt)]

Sorry, was out at hospital yesterday, just seen the note from fungi now.

Sam, good digging.

VIF.plug() is called on nova-compute startup, so that is what triggers something to happen on nova-compute start. I think dansmith mentioned that was needed, so he might be good to rope in here (probably should add him to nova-core-sec).

Just double checking the neutron driver here (totally need to bring a networking and privsep expert into the discussion), is it linuxbridge and not ovs and not hybrid?

Seems we always modify iptables in some way:
https://github.com/openstack/nova/blob/c65f1bdc5e1eaf9c48781d925bc7421e03cdb43e/nova/network/linux_net.py#L1795

Spent about 30 mins staring out this going nowhere, time to rope some others into the discussion. I propose dansmith?

[Tony Breeds (o-tony)]

I've added Dan as I wasn't able to reproduce it or locate the source of the strange rules.

I don't think this is a prvsep issues per se It's just that os-vif uses privsep as opppsed to nova's rootwrap. os-vif shoudl be adding rules to the nova-compute-FORWARD chain which seems right to me

#confused

[Sam Morrison (sorrison)]

On our mitaka compute nodes we don’t have any nova-* chains or rules in iptables at all.

We have the firewall_driver set to noop in nova as neutron linux bridge does all the iptables.

[Sam Morrison (sorrison)]

Yeah not privsep but the addition on os-vif which came in with newton as wasn’t in mitaka

[John Garbutt (johngarbutt)]

In case this helps...

There are some neutron drivers that need a firewall driver other than noop. In addition it does look like the noop driver still adds some IP tables rules (which surprised me).

I think dan mentioned something about after a reboot need to re-write the rules on nova-compute start, in case we had a host reboot and the rules need adding. It seems like here we do a re-write but there is no trigger for the neutron agent to do its half of the rules, and that leaves a dangerous config? Not sure.

But this is getting deeper into the networking stack than my knowledge currently goes, so I don't feel I know enough to find the resolution / verify its a problem.

[Sam Morrison (sorrison)]

Just wondering what to do about this, it seems noone is 100% clear what is supposed to be happening. Should we be opening this bug up so more eyes can see it?

[Jeremy Stanley (fungi)]

As John suggested having Dan take a look and he's subscribed, I've reached out to him through other channels to see if he has time to weigh in.

That said, given this is turning out hard to reproduce and may be unlikely for an attacker to figure out how to cause directly, it seems like it's probably safe enough to switch to our public workflow. I'm open to that if others are.

[Dan Smith (danms)]

I am still looking through this, but just FYI, I think the mere import of linux_net.py will cause the nova-* chains to be _created_, and nova/virt/libvirt/vif.py will do that, which is imported always by the libvirt driver. When linux_net.py is imported, we create iptables_manager = IptablesManager(), which creates those chains in its __init__(). I haven't surveyed the mitaka/newton boundary, but I'm not sure why those wouldn't have been created in mitaka as well, unless newton code used to blow away all existing chains and them stopped doing that in newton.

Still looking...

[Dan Smith (danms)]

Sam, can you tell me what your config says (if anything) for these options?

 forward_bridge_interface
 iptables_drop_action

The NeutronLinuxBridgeInterfaceDriver.plug() will definitely insert things into the nova-* rules, but it's not clear to me that we actually hit any of that code from libvirt anymore. The interesting thing is that the os-vif path actually seems to do the things that the NLBI driver was doing, which does include adding the "gateway rules" to the nova-compute-FORWARD chain. these appear to be intended to be the final say on packets to/from the instance that aren't otherwise filtered, so that packets can flow over a default gateway interface. If forward_bridge_interface is default ([all]) then it will generate rules like you show, which is "-o $myinterface -j ACCEPT" and "-i $myinterface -j ACCEPT".

Note that none of this really has anything to do with the firewall driver setting, so I don't think it's being ignored somewhere it shouldn't.

All of this code tries to be semi-defensive against a system with pre-existing firewall rules (which is somewhat of a fool's errand). So I wonder if this is just a case of nova (os-vif in this case) injecting its rules ahead of the neutron rules (which it knows nothing about) and thus inserting the "gateway rules" ahead of neutron's firewall rules? That would explain why restart ordering affects when this bug appears.

Does that make any sense?

Also, I think we probably want to rope Jay Pipes and Sean Mooney into this, as they are the most knowledgeable about os-vif AFAIK.

[Jeremy Stanley (fungi)]

Thanks, Dan! I've subscribed Jay and Sean now as well, at your recommendation.

[Sam Morrison (sorrison)]

When I was debugging this the code in nova.network.linux_net wasn't being executed it was in vif_plug_linux_bridg.linux_net

[Jay Pipes (jaypipes)]

https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/iptables.py#L281-L326

Not surprising that the IPTablesManager.__init__() code is os-vif is setting these rules up, since this code was copied from the nova/network/linux_net.py module pretty much verbatim. The only real difference between the two is that the os-vif code doesn't look at global oslo.cfg CONF options in the __init__ method (instead, it accepts kwarg configuration parameters)

OK, so what's the solution here? Should we just delete the setup of the old nova-specific chains in the IPTablesManager.__init__ method?

[Dan Smith (danms)]

Sam, yep, I know that. The code is nearly identical and has the same config options. Can you get me those from your config (if set) so I can make sure it's behaving the way I expect?

Jay, no that won't solve it on its own because other code in linux_net will try to add rules to those chains. If we don't need those "gateway rules" on neutron linux bridge systems, then we could try to go about removing all of the stuff that ends up getting them added. However, I'd not want to do that without someone confirming that it would still work in all cases.

[Sam Morrison (sorrison)]

These are the only network related options I can see in our nova.conf

use_neutron=True
linuxnet_interface_driver = nova.network.linux_net.NeutronLinuxBridgeInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver

Let me know if you need more info, cheers. I can't figure out why we have none of these rules in mitaka as code looks pretty much the same except it doesn't use the vif_plug_linux_bridge module.

[Sam Morrison (sorrison)]

I can confirm that this issue has been fixed in Queens. When upgrading to queens I no longer get anymore nova-compute iptables rules so the behaviour is now the same as what it was in Mitaka.

Guess you can close this but in my mind this is a pretty severe security issue if you're using newton, ocata or pike.

[Sam Morrison (sorrison)]

I should also mention that we made no config changes between pike and queens to get this change of behaviour.

[sean mooney (sean-k-mooney)]

sorry i was not aware of this till now.
i was subscibed along time ago but must have missed it.
i will try to review this and the os-vif code to determin if there is something we can do for the older releases.

@sam im glad this appears to be fixed but without a deliberate effort to fix it it was likely not
intentional. as a result it would still be prudent to determin how this was fixed and ensure there is no regression in the future.

[sean mooney (sean-k-mooney)]

given the age of this bug and the fact that Sam reported this as fixed in Queens i am closing this as fix released.

we never found or recored how/why this was fixed but given that there has been no update on this in some time i think we can close this for now until there is evidence that it still exists on newer releaes.

[Jeremy Stanley (fungi)]

In an effort not to let bugs fester forever in private, any objections to switching this to public at least?

[Sam Morrison (sorrison)]

Fine with me

[Jeremy Stanley (fungi)]

Thanks Sam, I don't expect the Nova security reviewers to object as they're unable to reproduce this and consider the bug apparently fixed in the Queens release anyway. Setting the security advisory task to won't fix state, but can be revisited if any new information comes to light.