port security does not block router advertisements for instances

[George Shuklin]

Affected version: mitaka

Issue: If port security is enabled, IPv6 router advertisements may be send by any instance.

Network configuration: vlan, security groups disabled, port security enabled.

subnet:
{
  "description": "",
  "enable_dhcp": true,
  "network_id": "b71b7cc7-3534-481b-bb67-a473a8e083cc",
  "tenant_id": "4e632076f7004f908c8da67345a7592e",
  "created_at": "2017-04-21T12:39:13",
  "dns_nameservers": "",
  "updated_at": "2017-04-21T12:39:13",
  "ipv6_ra_mode": "",
  "allocation_pools": "{\"start\": \"2a00:xxxx:3:101::2\", \"end\": \"2a00:xxxx:3:101:ffff:ffff:ffff:ffff\"}",
  "gateway_ip": "2a00:xxxx:3:101::1",
  "ipv6_address_mode": "slaac",
  "ip_version": 6,
  "host_routes": "",
  "cidr": "2a00:xxxx:3:101::/64",
  "id": "789d4f41-7867-4b17-9f7b-220c1e689b0b",
  "subnetpool_id": "",
  "name": ""
}

When instance is configured by (malicious) user, it starts to send router advertisements (like it is a router) and those RAs may interrupt networking.

tcpdump from physical interface of compute node:
tcpdump -ni eth4 ip6
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth4, link-type EN10MB (Ethernet), capture size 262144 bytes
14:16:47.707480 IP6 fe80::52eb:1aff:fe77:de4f > ff02::1: ICMP6, router advertisement, length 64
14:16:48.709429 IP6 fe80::f816:3eff:fe69:6644 > ff02::1: ICMP6, router advertisement, length 56

first line is a valid router RA, second line (:6644) - by instance, which should be blocked by port security.

On a victim machine (same segment) routing table looks like this:

ip -6 route

default via fe80::52eb:1aff:fe77:de4f dev ens3 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium
default via fe80::f816:3eff:fe69:6644 dev ens3 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium

Last line - result of network hijacking from malicious instance, and shouldn't happen.

I'm not sure if this is a security issue or not.

[Brian Haley (brian-haley)]

Can you cut/paste the output of 'ip6tables-save' from the compute node and the router namespace on the network node? Could be there's a single rule missing, I just don't remember seeing a fix for something like this.

[Brian Haley (brian-haley)]

Actually, someone just found this change:

https://review.openstack.org/#/c/310648/

Do you have that applied?

But you posted above you don't have security groups enabled, which might be the issue.

[George Shuklin (george-shuklin)]

ip6tables-save
# Generated by ip6tables-save v1.4.21 on Wed Apr 26 14:31:17 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Wed Apr 26 14:31:17 2017
# Generated by ip6tables-save v1.4.21 on Wed Apr 26 14:31:17 2017
*mangle
:PREROUTING ACCEPT [8:536]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Wed Apr 26 14:31:17 2017
# Generated by ip6tables-save v1.4.21 on Wed Apr 26 14:31:17 2017
*raw
:PREROUTING ACCEPT [8:536]
:OUTPUT ACCEPT [0:0]
COMMIT
# Completed on Wed Apr 26 14:31:17 2017
# Generated by ip6tables-save v1.4.21 on Wed Apr 26 14:31:17 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RESTRICTED - [0:0]
COMMIT
# Completed on Wed Apr 26 14:31:17 2017

I think port-security is done on OVS level. Here ovs-ofctl dump-flows:

 cookie=0xaf2c3557e0b521d9, duration=447480.581s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=10,icmp6,in_port=6,icmp_type=136 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447479.924s, table=0, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=10,icmp6,in_port=3,icmp_type=136 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447479.366s, table=0, n_packets=811, n_bytes=68442, idle_age=42805, hard_age=65534, priority=10,icmp6,in_port=1,icmp_type=136 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447478.765s, table=0, n_packets=640, n_bytes=54288, idle_age=12316, hard_age=65534, priority=10,icmp6,in_port=5,icmp_type=136 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447478.203s, table=0, n_packets=665, n_bytes=56334, idle_age=13381, hard_age=65534, priority=10,icmp6,in_port=4,icmp_type=136 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447477.497s, table=0, n_packets=782, n_bytes=66228, idle_age=13382, hard_age=65534, priority=10,icmp6,in_port=2,icmp_type=136 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447480.422s, table=0, n_packets=4017, n_bytes=168714, idle_age=110, hard_age=65534, priority=10,arp,in_port=6 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447479.788s, table=0, n_packets=373, n_bytes=15666, idle_age=133, hard_age=65534, priority=10,arp,in_port=3 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447479.207s, table=0, n_packets=487, n_bytes=20454, idle_age=70, hard_age=65534, priority=10,arp,in_port=1 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447478.641s, table=0, n_packets=519, n_bytes=21798, idle_age=985, hard_age=65534, priority=10,arp,in_port=5 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447478.046s, table=0, n_packets=464, n_bytes=19488, idle_age=250, hard_age=65534, priority=10,arp,in_port=4 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447477.327s, table=0, n_packets=509, n_bytes=21378, idle_age=266, hard_age=65534, priority=10,arp,in_port=2 actions=resubmit(,24)
 cookie=0xaf2c3557e0b521d9, duration=447485.884s, table=0, n_packets=1437117, n_bytes=96650207, idle_age=1, hard_age=65534, priority=2,in_port=7 actions=drop
 cookie=0xaf2c3557e0b521d9, duratio...

[Brian Haley (brian-haley)]

I don't believe OVS port security worked in Mitaka, I'll ask someone tomorrow.

But ip6tables shows no rules or chains since security groups have been disabled.

[George Shuklin (george-shuklin)]

Port security definitively works in Mitaka. It was a main reason for our upgrade, and I personally checked it - it worked in a lab and it has been working in production.

We do not use security groups, so all port security happens inside OVS (with ovs-ofctl). Both IPv4 and IPv6 port security are working, but IPv6 port security blocks only neighbor discovery, ignoring routing advertisement, which is a bug.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/462285

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/472509

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/462285
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ce0352aa7b1609078e8f109b5b4c368d9a1baa89
Submitter: Jenkins
Branch: master

commit ce0352aa7b1609078e8f109b5b4c368d9a1baa89
Author: Brian Haley <email address hidden>
Date: Wed May 3 16:34:12 2017 -0400

    Drop IPv6 Router Advertisements in OVS firewall

    Only neutron routers should be sending RAs, and with
    the iptables firewall these are dropped, but there
    was no corresponding rule for the OVS firewall.

    Change-Id: I045c652ad8cbecf5ed8e98934306476ed7170e90
    Partial-bug: #1685237

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/474738

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/474739

[Ihar Hrachyshka (ihar-hrachyshka)]

Sounds like an issue that can expose the network to inside attacker. Raising the priority to High to get backports till Newton.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/472509
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0cb9b5254f412e9ec0d3f2cb6fd07e038c3a5097
Submitter: Jenkins
Branch: master

commit 0cb9b5254f412e9ec0d3f2cb6fd07e038c3a5097
Author: Brian Haley <email address hidden>
Date: Thu Jun 8 23:35:17 2017 -0400

    Split allowed ICMPv6 types into two constants

    There was only a single list of allowed ICMPv6
    types, but the defaults allowed for ingress
    and egress are different when it comes to
    Router Advertisements and Router Solicitations.

    Change-Id: I737f07065cf2fb0b574a7f0f49e084488bf23ac0
    Closes-bug: #1685237

[George Shuklin (george-shuklin)]

And to Mitaka, if possible. We are still on Mitaka and that bug is somehow give me sense of insecurity.

[Brian Haley (brian-haley)]

George - upstream stable/mitaka branch is EOL, so you'll need to bug your distro provider about backporting this change. Should be straight-forward.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/474739
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a916fc5173569c5f7720d07fe3e1424bccbfb017
Submitter: Jenkins
Branch: stable/newton

commit a916fc5173569c5f7720d07fe3e1424bccbfb017
Author: Brian Haley <email address hidden>
Date: Wed May 3 16:34:12 2017 -0400

    Drop IPv6 Router Advertisements in OVS firewall

    Only neutron routers should be sending RAs, and with
    the iptables firewall these are dropped, but there
    was no corresponding rule for the OVS firewall.

    Change-Id: I045c652ad8cbecf5ed8e98934306476ed7170e90
    Partial-bug: #1685237
    (cherry picked from commit ce0352aa7b1609078e8f109b5b4c368d9a1baa89)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/474738
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4aded9f16d14558ba1786f515db99ebf443aa820
Submitter: Jenkins
Branch: stable/ocata

commit 4aded9f16d14558ba1786f515db99ebf443aa820
Author: Brian Haley <email address hidden>
Date: Wed May 3 16:34:12 2017 -0400

    Drop IPv6 Router Advertisements in OVS firewall

    Only neutron routers should be sending RAs, and with
    the iptables firewall these are dropped, but there
    was no corresponding rule for the OVS firewall.

    Change-Id: I045c652ad8cbecf5ed8e98934306476ed7170e90
    Partial-bug: #1685237
    (cherry picked from commit ce0352aa7b1609078e8f109b5b4c368d9a1baa89)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.0.0b3 development milestone.

[James Page (james-page)]

Marking Fix Released; this was included in the recent Pike release and has been backported to both newton and ocata.