euca-authorize messes up nova-compute-local chains on nodes with more than 1 running instance
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
High
|
Unassigned |
Bug Description
When modifying a security group with running instances, the nova-compute-local chain is incorrectly updated.
IPTables output before adding/revoking a rule:
Chain nova-compute-
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3002
0 0 nova-compute-
Chain nova-compute-
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3002
0 0 nova-compute-
Chain nova-compute-local (1 references)
pkts bytes target prot opt in out source destination
0 0 nova-compute-
0 0 nova-compute-
iptables output after revoking a rule (specifically tcp dpt 3002):
Chain nova-compute-
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
0 0 nova-compute-
Chain nova-compute-
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
0 0 nova-compute-
Chain nova-compute-local (1 references)
pkts bytes target prot opt in out source destination
0 0 nova-compute-
0 0 nova-compute-
Notice that the nova-compute-local chain has been updated to set destination to the first destination IP in the chain for all of them. I have tested that this happens with any number of instances > 1.
This incorrect state can be resolved by flushing the iptables rules and restarting compute, causing it to rebuild the chains:
Chain nova-compute-
pkts bytes target prot opt in out source destination
Chain nova-compute-
pkts bytes target prot opt in out source destination
Chain nova-compute-local (0 references)
pkts bytes target prot opt in out source destination
service nova-compute restart
nova-compute start/running, process 2797
Chain nova-compute-
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
0 0 nova-compute-
Chain nova-compute-
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT udp -- * * 10.16.1.1 0.0.0.0/0 udp spt:67 dpt:68
0 0 ACCEPT tcp -- * * 192.168.100.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 192.168.100.0/24 0.0.0.0/0
0 0 ACCEPT tcp -- * * 10.16.1.0/24 0.0.0.0/0 tcp dpt:22
0 0 ACCEPT icmp -- * * 10.16.1.0/24 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3000
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3001
0 0 nova-compute-
Chain nova-compute-local (1 references)
pkts bytes target prot opt in out source destination
0 0 nova-compute-
0 0 nova-compute-
This causes all other local chains to be moot and allows all traffic to every other instance. It only appears to effect instances in the same project (which makes sense).
Changed in nova: | |
importance: | Undecided → High |
status: | New → Confirmed |
tags: | added: security-group |
Changed in nova: | |
milestone: | none → diablo-4 |
Changed in nova: | |
milestone: | diablo-4 → 2011.3 |
status: | Fix Committed → Fix Released |
This is due to do_refresh_ security_ group sets network_info only on the first iteration of its loop and then recycles that for subsequent iterations.
network_info is a per-instance thing. It makes no sense to pass it to methods that act on more than one instance, so lets stop doing that. Ever again. :)