neutron security group rules not applied to nova-lxd containers

[James Page]

I noted this when testing the changes for lxd:isolated in Ubuntu Xenial; neutron sets up iptables rules against tap devices (as used in the libvirt driver); however nova-lxd does not use tap devices - LXD plumbs the instance in to the neutron bridge using an veth pair.

I think the net result of this is that security rules are just not getting applied in LXD instances.

[James Page (james-page)]

Section of iptables rules:

-A neutron-openvswi-sg-chain -m physdev --physdev-out tap51745526-12 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-i51745526-1
-A neutron-openvswi-sg-chain -m physdev --physdev-in tap51745526-12 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-o51745526-1
-A neutron-openvswi-sg-chain -m physdev --physdev-out tapb2231d9d-70 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-ib2231d9d-7
-A neutron-openvswi-sg-chain -m physdev --physdev-in tapb2231d9d-70 --physdev-is-bridged -m comment --comment "Jump to the VM specific chain." -j neutron-openvswi-ob2231d9d-7
-A neutron-openvswi-sg-chain -j ACCEPT

[James Page (james-page)]

tcpdump capture from qbrXXXXXX:

root@juju-e397c6-ceph-upgrade-testing-10:~# tcpdump -i qbrb2231d9d-70
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on qbrb2231d9d-70, link-type EN10MB (Ethernet), capture size 262144 bytes
15:12:57.392931 IP james-page-bastion.openstacklocal > 192.168.21.3: ICMP echo request, id 16897, seq 35, length 64
15:12:57.392980 IP 192.168.21.3 > james-page-bastion.openstacklocal: ICMP echo reply, id 16897, seq 35, length 64

[James Page (james-page)]

interfaces on compute hosts:

9: qbrb2231d9d-70: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e6:76:41:b8:33:aa brd ff:ff:ff:ff:ff:ff
    inet6 fe80::58ec:7ff:fec8:beea/64 scope link
       valid_lft forever preferred_lft forever
10: qvob2231d9d-70@qvbb2231d9d-70: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 9e:e6:09:18:79:b8 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9ce6:9ff:fe18:79b8/64 scope link
       valid_lft forever preferred_lft forever
11: qvbb2231d9d-70@qvob2231d9d-70: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master qbrb2231d9d-70 state UP group default qlen 1000
    link/ether e6:76:41:b8:33:aa brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e476:41ff:feb8:33aa/64 scope link
       valid_lft forever preferred_lft forever
13: veth5RHGD8@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master qbrb2231d9d-70 state UP group default qlen 1000
    link/ether fe:49:43:8f:d7:16 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::fc49:43ff:fe8f:d716/64 scope link
       valid_lft forever preferred_lft forever
14: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1
    link/gre 0.0.0.0 brd 0.0.0.0
15: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
16: gre_sys@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65490 qdisc pfifo_fast master ovs-system state UNKNOWN group default qlen 1000
    link/ether 42:f1:80:1f:aa:7b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::40f1:80ff:fe1f:aa7b/64 scope link
       valid_lft forever preferred_lft forever
17: qbr51745526-12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 76:90:01:e0:ce:18 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::300a:61ff:fe76:81e5/64 scope link
       valid_lft forever preferred_lft forever
18: qvo51745526-12@qvb51745526-12: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master ovs-system state UP group default qlen 1000
    link/ether 22:3f:9d:50:e2:45 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::203f:9dff:fe50:e245/64 scope link
       valid_lft forever preferred_lft forever
19: qvb51745526-12@qvo51745526-12: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc noqueue master qbr51745526-12 state UP group default qlen 1000
    link/ether 76:90:01:e0:ce:18 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7490:1ff:fee0:ce18/64 scope link
       valid_lft forever preferred_lft forever
21: vethPXK7KW@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master qbr51745526-12 state UP group default qlen 1000
    link/ether fe:3e:13:c0:f3:43 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::fc3e:13ff:fec0:f343/64 scope link
       valid_lft forever preferred_lft forever

[James Page (james-page)]

This may all have worked at some point in the past, but looking at mitaka onwards, rules are always applied to the tap device, not on a bridge.

I think the way forward is to create the tap device, plumbed to the bridge and then pass that as a physical device to the container - however I'm still not quite sure how iptables will handle all of that.

[James Page (james-page)]

xenial-newton (14.0.0-0ubuntu1~cloud0):

name: instance-00000002
config:
  boot.autostart: "True"
  limits.cpu: "1"
  limits.memory: 256MB
  raw.lxc: |
    lxc.console.logfile=/var/log/lxd/instance-00000002/console.log
description: ""
devices:
  qbr2dc32c41-b1:
    host_name: tap2dc32c41-b1
    hwaddr: fa:16:3e:e5:47:2c
    nictype: bridged
    parent: qbr2dc32c41-b1
    type: nic
  root:
    path: /
    size: 5GB
    type: disk

which actually looks correctly IMHO - the device is plumbed via the tap driver.

[James Page (james-page)]

ocata has the same code paths, so should be OK so this looks isolated to xenial/mitaka, which uses veth pairs to plumb the instance into the neutron bridge infront of the openvswitch bridges.

[James Page (james-page)]

xenial-mitaka:

name: instance-00000001
config:
  boot.autostart: "True"
  limits.cpu: "1"
  limits.memory: 2048MB
  raw.lxc: |
    lxc.console.logfile=/var/log/lxd/instance-00000001/console.log
description: ""
devices:
  qbr8eea8b0f-24:
    hwaddr: fa:16:3e:81:4e:58
    nictype: bridged
    parent: qbr8eea8b0f-24
    type: nic
  root:
    path: /
    size: 20GB
    type: disk

The appropriate host_name for the device pair is not applied in this version.

[James Page (james-page)]

Propose change for mitaka:

  https://review.openstack.org/#/c/427111/

which is basically a backport of the 'host_name' usage from later versions.

I've tested with a hot-patch into a deployed xenial-mitaka cloud; devices where named correctly and security group rules where applied.

[James Page (james-page)]

xenial-mitaka with patch applied

$ sudo lxc profile show instance-00000001
name: instance-00000001
config:
  boot.autostart: "True"
  limits.cpu: "1"
  limits.memory: 2048MB
  raw.lxc: |
    lxc.console.logfile=/var/log/lxd/instance-00000001/console.log
description: ""
devices:
  qbr644c9eb4-ed:
    host_name: tap644c9eb4-ed
    hwaddr: fa:16:3e:f1:8e:e7
    nictype: bridged
    parent: qbr644c9eb4-ed
    type: nic
  root:
    path: /
    size: 20GB
    type: disk

[James Page (james-page)]

Changed landed into stable/mitaka; we'll need to get that cherry-picked, or a point release made to fix this up.

[Tyler Hicks (tyhicks)]

Thanks for the debdiff, James!

It looks good to me. I only added one line to the changelog mentioning that a CVE has not yet been assigned.

The build log comparison between the patched and unpatched nova-lxd xenial packages looks good. I've uploaded the package to the public security-proposed PPA:

 https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa

The binary packages are being published in the PPA as I type. Please do QA on the nova-lxd packages in the PPA, as they will be copied to xenial-security, and report the results in this bug.

Since the fix is public, I'm going to make this bug public and request a CVE on the oss-security list.

[Tyler Hicks (tyhicks)]

CVE request: http://openwall.com/lists/oss-security/2017/02/08/4

[Tyler Hicks (tyhicks)]

CVE-2017-5936 was assigned: http://openwall.com/lists/oss-security/2017/02/09/3

[James Page (james-page)]

I've tested the package in the security proposed PPA; it resolves the issue, host veth naming is aligned to neutron's expectation and security group rules are correctly applied.

Note that the code changes don't update the host veth name for existing instances; its possible todo this manually directly on compute hosts (using lxd profile edit <instancename>) but it is a disruptive operation as the instance will loose connectivity.

[James Page (james-page)]

Part of 13.1.1 - Marking Fix Released for nova-lxd

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova-lxd - 13.2.0-0ubuntu1.16.04.1

---------------
nova-lxd (13.2.0-0ubuntu1.16.04.1) xenial-security; urgency=medium

  * SECURITY UPDATE: ensure correct application of security group rules.
    - d/p/host-device-naming.patch: Cherry pick fix to ensure that the
      host part of the veth pair used to wire LXD containers into neutron
      has the correct naming, resolving issues with application of
      neutron security group rules in container deployments (LP: #1656847).
    - CVE not yet assigned

 -- James Page <email address hidden> Tue, 07 Feb 2017 17:06:46 +0100

[Tyler Hicks (tyhicks)]

James, I'm going to include a reference to this bug in the USN text with a mention that existing instances will still be affected and that they must be manually updated. Is it possible for you to leave a comment with some more information about how to fix existing interfaces?

[BlueT - Matthew Lien - 練喆明 (bluet)]

Thanks for the fix!
Agree with @tyhicks, it would be nice to have a HowTo for users to fix existing interfaces.