Security Group Rules not effective immediately

[Sreekumar S]

Exact version of Nova/OpenStack you are running:

devstack@dev0:~/devstack$ git log -1
commit 28247f3511b2eedfc740cf822f04dc8cb1ae7b08
Merge: 9d113a2 03a19bc
Author: Jenkins <email address hidden>
Date: Fri Jul 31 20:12:34 2015 +0000

    Merge "ceilometer: add tempest option to test events" into stable/kilo
devstack@dev0:~/devstack$ cd -
/opt/stack/nova
devstack@dev0:/opt/stack/nova$ git log -1
commit 90e1eacee8da05bed2b061b8df5fc4fbf5057bb2
Author: Alan Pevec <email address hidden>
Date: Tue Jul 28 21:25:51 2015 +0200

    Bump stable/kilo next version to 2015.1.2

    Change-Id: I180de9d4f1f4ae385682ce40a3a695eb3acc4c43
devstack@dev0:/opt/stack/nova$

Summary:
After editing sec group to add/remove rules, those changes are not reflected on the instances immediately. The changes take effect only after one of the instance's security group is removed and added back. Once this is done, all the instances where the particular security group is associated is updated with the new rules. Now the iptables routing rules get updated as well.

Reproduce steps (CLI):
-- ./stack.sh with nova-network configured.
-- Once the services are running, and horizon starts responding; Launch 2 instances from Horizon.
   These instances will have default sec group associated to them
-- Associate Floating IP to both of these instances...
devstack@dev0:~/devstack$ nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+
| 7cf44867-42f4-44de-aba5-019113d52612 | test0 | ACTIVE | - | Running | private=172.31.1.2, 10.0.2.249 |
| 0696c57e-7704-4d51-a3f2-627049946052 | test1 | ACTIVE | - | Running | private=172.31.1.3, 10.0.2.250 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+

-- From another node on the external network ping both the node on the FIPs. Ping will not go through...
-- Allow ICMP from all IPs...
devstack@dev0:~/devstack$ nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp | -1 | -1 | 0.0.0.0/0 | |
+-------------+-----------+---------+-----------+--------------+
devstack@dev0:~/devstack$ nova secgroup-list-rules 1
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp | -1 | -1 | 0.0.0.0/0 | |
+-------------+-----------+---------+-----------+--------------+

-- Ping again, still will not go through!!!

-- Remove the default group from one of the instances... Now the other node will start pinging, because now the new rule is applied to all instances.
nova remove-secgroup "0696c57e-7704-4d51-a3f2-627049946052" default

-- Add back the the default group to the instance, whose sec group was removed in the first place. Now that node starts pinging as well.
nova add-secgroup "0696c57e-7704-4d51-a3f2-627049946052" default

-- Same is the case with deletion of rules from sec group as well. Once you delete the ping allow rule, it will still allow pings until 'one of the instances is re-associated' with the sec group. This seems quite dangerous!!!

This bug is seen with both the CLI and Horizon Webapp with devstack running nova-network. I haven't checked the issue with nova-network on a multi node setup, although it should most probably be there as well.
    But I do have checked whether the same issue happens with neutron, in both devstack, and a 3 node (controller, network, compute1) setup as mentioned in the official installation guide for Kilo. In neutron the rules are immediately reflected.

Expected result: The new rules should be updated in iptables immediately after the sec group rules are modified and saved. All the instances associated to a sec group should immediately get affected by a rule change in that group. The admin/user need not have to manually detach and reattach the sec group to the instance.

Actual result: The modified rules don't immediately take effect.