Security groups incorrectly applied on new additional interfaces

[Jan Collijs]

When launching an instance with one network interface and enabling 2 security groups everything is working as it supposed to be.

But when attaching additional network interfaces only the default security group is applied to those new interfaces. The additional security group isn't enabled at all on those extra interfaces.

We had to dig into the iptables chains to discover this behavior. Once adding the rule manually or adding them to the default security group everything is working fine.

[Manjeet Singh Bhatia (manjeet-s-bhatia)]

how I tried this.

- I created two networks with 2 subnets, one router and two instances one for each network
- I also created a sg and added icmp and tcp 22 rule to that
- for router i added two network interfaces and added gateway.
- two instances connected to network test1 and test2 and two rules applied to both.

- as i added icmp and tcp rule to new sg.

- I was able to ping both my instances which were associated floating ip.
- i was able to ssh to both my instances.

I- tried two sg on two instances on different networks attached to same router
which was working for me

does that cover your bug scenario or its something I am missing ?

[Jan Collijs (s-launchpat-f)]

What we do over here is as follows:

we use a vlan based network setup where we create on management network which is allocated a specific segmentation_id (actual vlan).

Spinning up instances directly with only this network attached and configured with 2 security groups default + web.

we also configured an extra network without a subnet allocating to a specific segmentation_id. Since those networks don't have a layer 3 configuration openstack refuses to attach this network at instance creating. So I attach them using the nova attach-interface command as soon as the instance is launched, which works fine networking wise.

The missing subnets are probably causing this behavior?

[yujie (16189455-d)]

Does the port attached to vm using nova has the tap/qbr/qvb interface in the Compute node it living? If it has, then what about the chain-rule in iptables in filter table? I know that the security-group takes place at iptables.

[Miguel Angel Ajo (mangelajo)]

Adding our colleague Shi Han Zhang

[shihanzhang (shihanzhang)]

ajo, thx for leting me know this bug, I will check it as soon as possible

[shihanzhang (shihanzhang)]

I can't reproduce this bug, my reproduce steps:
1. there are 3 security group default sg, sgA and sgB
2. create a VM with one network interface which belongs to sgA and sgB
3. attach a new network interface to this VM which belongs to default sg
but I check the iptables rules, it apply correctly,

@Jan Collijs , my reproduce steps are correctly? if not, please let me know how to reproduce, thank you very much!

[shihanzhang (shihanzhang)]

hi Jan Collijs, as your suggestion, I do as below:
1. create networkA with subnet 10.10.10.0/24, networkB without subnet
2. there are 2 security group default sg, and sgA which allows ssh access
3. create a VM with one network interface which in networkA and belongs default sg and sgA
4. use nova CLI "nova interface-attach --net-id" to add a network interface which in networkB and belongs default sg

I also check iptables rules are correct as below:

-A neutron-openvswi-INPUT -m physdev --physdev-in tapf5b77831-81 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-openvswi-of5b77831-8
-A neutron-openvswi-INPUT -m physdev --physdev-in tap2bd3ad0c-a7 --physdev-is-bridged -m comment --comment "Direct incoming traffic from VM to the security group chain." -j neutron-openvswi-o2bd3ad0c-a
-A neutron-openvswi-i2bd3ad0c-a -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
-A neutron-openvswi-i2bd3ad0c-a -m set --match-set NIPv4e22b4152-0e39-4c8c-b811- src -j RETURN
-A neutron-openvswi-i2bd3ad0c-a -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
-A neutron-openvswi-i2bd3ad0c-a -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-openvswi-sg-fallback
-A neutron-openvswi-if5b77831-8 -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
-A neutron-openvswi-if5b77831-8 -s 10.10.10.2/32 -p udp -m udp --sport 67 -m udp --dport 68 -j RETURN
-A neutron-openvswi-if5b77831-8 -m set --match-set NIPv4e22b4152-0e39-4c8c-b811- src -j RETURN
-A neutron-openvswi-if5b77831-8 -p tcp -m tcp --dport 22 -j RETURN
-A neutron-openvswi-if5b77831-8 -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
-A neutron-openvswi-if5b77831-8 -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-openvswi-sg-fallback
-A neutron-openvswi-o2bd3ad0c-a -p udp -m udp --sport 68 -m udp --dport 67 -m comment --comment "Allow DHCP client traffic." -j RETURN
-A neutron-openvswi-o2bd3ad0c-a -j neutron-openvswi-s2bd3ad0c-a
-A neutron-openvswi-o2bd3ad0c-a -p udp -m udp --sport 67 -m udp --dport 68 -m comment --comment "Prevent DHCP Spoofing by VM." -j DROP
-A neutron-openvswi-o2bd3ad0c-a -m state --state RELATED,ESTABLISHED -m comment --comment "Direct packets associated with a known session to the RETURN chain." -j RETURN
-A neutron-openvswi-o2bd3ad0c-a -j RETURN
-A neutron-openvswi-o2bd3ad0c-a -m state --state INVALID -m comment --comment "Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack." -j DROP
-A neutron-openvswi-o2bd3ad0c-a -m comment --comment "Send unmatched traffic to the fallback chain." -j neutron-openvswi-sg-fallback
-A neutron-openvswi-of5b77831-8 -p udp -m udp --sport 68 -m udp --dport 67 -m com...

[Jan Collijs (s-launchpat-f)]

So the 'feature' of attaching a network without a subnet through the nova cli with nova interface-attach is actually a bug in the first place?

Cause we are heavily relying on this one to have layer 2 connectivity without many dirty hacks..

The missing subnet would explain why the iptables rules are not applied to those extra added network interfaces off course.

[Sreenivas (sreenivas-pothukanoori)]

I have followed the below steps to reproduce the issue:

1. Created two security groups one is default security group with default rules and other one is mysecgrp with tcp & icmp rules. Below are the details:

root@ash2controller:/home/ashraf2# nova secgroup-list-rules mysecgrp
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+-----------+--------------+
| tcp | 22 | 22 | 0.0.0.0/0 | |
| icmp | -1 | -1 | 0.0.0.0/0 | |
+-------------+-----------+---------+-----------+--------------+

root@ash2controller:/home/ashraf2# nova secgroup-list-rules default
+-------------+-----------+---------+----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+----------+--------------+
| | | | | default |
| | | | | default |
+-------------+-----------+---------+----------+--------------+

2. Created a network with name mydemonet(172.52.16.0/24) and key-pair.

3. Booted VM(mydemovm) with mydemonet network and assigned default,mysecgrp security groups to it.

root@ash2controller:/home/ashraf2# nova boot --flavor m1.tiny --image cirros --nic net-id=78d6ad72-04af-426d-a7ed-bb312b5aa16f --security-group default,mysecgrp --key-name mykey mydemovm

4. Able to ping and do ssh to VM interface IP(172.52.16.9)

5. Now Created one more network i.e mydemonet2(172.52.18.0/24) and attached additional interface from this subnet to the mydemovm.

6. In VM details I am able to see both the security groups & interfaces.

root@ash2controller:/home/ashraf2# nova show mydemovm
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | ash2controller |
| OS-EXT-SRV-ATTR:hypervisor_hostname | ash2controller |
| OS-EXT-SRV-ATTR:instance_name | instance-00000042 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-03-31T08:08:04.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | ...

[Jan Collijs (s-launchpat-f)]

Yes that would be it, only we attached the second interface through the nova interface-attach command not through the GUI.

[Sreenivas (sreenivas-pothukanoori)]

Hi,

Please find below my observations while working on the bug. Ping to the additional interface is not working because of below reasons.

1. Security group not attached to additional interface.
By design(open stack it seems like additional interfaces will not inherit security groups applied on primary interface. Security group are specific to an interface. If additional interfaces have to be configured with security group there is a provision to do so using the below command.

neutron port-create --security-group default, mysecgrp mydemonet

Observations:
1. Security groups can be assigned to VM or to a port ( only at the time of creating port ).
2. Security groups configured at VM level are applied to all VNIC’s active at that instant.
3. Any new port can be configured with custom SG in addition to DSG only at the time of port creation.
4. VM level security groups not inherited by new port. This may be a design intent or constraints. This means that any new port created will only get assigned to DSG by default and not
           any other custom SG created and assigned to the VM prior to creating this new port.
           i.e. The new port need not inherit custom security group.

 While working on the bug we found below observation which is linked to the bug reported. For the additional interface to get IP, the interface should be configured at OS level.

2. Additional interface not coming up in instance.
This seems to be configuration issue at the Image level. The instance image should be configured for multiple interfaces. We think the initial Image originator tried was configured for a single NIC interface. Even if we add multiple interfaces from orchestration level(OpenStack) the VM Image is still configured for single NIC interface. Image is configured for multiple interfaces following below procedure.

In /etc/network/interfaces/
auto eth1
iface eth1 inet dhcp

reboot or restart network services.

Observations:
• seems to be VM image issue. The VM image should be configured to support multiple NIC interfaces.

Based on the above observations I think the issue reported may not be a bug. The behavior is as per open stack design intent.

Please let me know if we can go ahead and close this bug as the behavior observed is as per open stack design.

[Jan Collijs (s-launchpat-f)]

The image VM on our setup was configured to recognize hot plug network devices never had any issue on that part.

If the assignment of security-groups is intended not to assign the default security group to additional network interfaces which are attached after the boot process by openstack design I'm pretty amazed by it but fine and go ahead to close this ticket.

On our setup we completely disabled security groups since it's a cloud not exposed to the public without the need of this isolation and therefore we don't have any issues anymore with those additional attached interfaces and the iptables flows.

Thanks for your effort and clean explanation on this matter!

[zhaobo (zhaobo6)]

Hi, I tried this on the M release. I follow the steps above you mentioned and can not repo this bug.
The primary port's sg is separate with the new interface from another network(contain a subnet).

eg:
The primary port has its own sg, the new interface port has its own sg too.

And I check the iptables rules of 2 interfaces. All of them are the expected and following the behavior what they should be.

[Sreenivas (sreenivas-pothukanoori)]

Default security group and any additional security groups can be applied on additional interface at the time of port creation. However, when an additional interface is created and attached to the VM with out any security group, will always be assigned default security group only. The additional interface will not inherit any other security groups.

In this context, we will close this ticket as the functionality is in line with system design.

[Sreenivas (sreenivas-pothukanoori)]

I have changed this bug status to incomplete as am want to move this bug as invalid state based on comment#14.

Could you please check my observations mentioned in the comment#11, #14 and provide your confirmation to mark this bug as invalid.

[Jan Collijs (s-launchpat-f)]

As explained in #12 go right ahead.

[Armando Migliaccio (armando-migliaccio)]

correct

[Bjoern (bjoern-t)]

I do agree with the initial issue that this is bad design, especially if nova and also horizon are displaying security groups for the instances rather than neutron port.

Personally I would prefer that nova displays the security groups per port, per instance.