Exploitable services exposed on community test nodes

[Jeremy Stanley]

One of the donor service providers for the upstream OpenStack Infrastructure CI pool has notified us that their security team's periodic vulnerability scans have been identifying systems at random within our environment as running open memcached servers. Job correlation from these reports indicates each was running one of the following:

kolla-ansible-oraclelinux-binary
kolla-ansible-oraclelinux-source
kolla-ansible-oraclelinux-source-ceph

Please adjust the configuration of your job framework to prevent these services from being exposed to the Internet (through iptables ingress filters, service ACLs, configuring them to not listen on globally-routable interfaces, whatever works). Thanks!

[Jeremy Stanley (fungi)]

Any update on the state of this? It's really pretty urgent. An example of _why_ it's a problem: http://www.openwall.com/lists/oss-security/2018/03/02/1

[Jeffrey Zhang (jeffrey4l)]

we are trying to use iptables to prevent other IP to connect the memcached port.

I think in devstack, it has the same issue, how it avoid the issue?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/550325

[Jeremy Stanley (fungi)]

We preinstall restrictive iptables rulesets on our images when building them via http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/install.d/20-iptables and devstack configures keystone's memcached_servers setting to localhost:11211 so that it traverses the loopback interface rather than an externally-reachable address.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jeffrey Zhang ([e-mail]) on branch: master
Review: https://review.openstack.org/550325
Reason: check https://review.openstack.org/#/c/549715/1

[Paul Bourke (pauldbourke)]

Hi Jeremy, thanks for the link. I'm going to look at adding these rules to kolla tomorrow, will update here asap.

[Jeremy Stanley (fungi)]

It's worth noting that all the test nodes on which jobs run boot up with the rules I linked above already applied. If memcached or other services are being exposed on reachable interfaces of the node then it can only be because you're altering or tearing down the existing iptables ruleset.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/550821
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=404d4d0a50f292b1fd6e916cf80813b260621840
Submitter: Zuul
Branch: master

commit 404d4d0a50f292b1fd6e916cf80813b260621840
Author: Paul Bourke <email address hidden>
Date: Thu Mar 8 12:55:05 2018 +0000

    Use zuul firewall rules in gate

    Till now we've been flusing iptables in the gates to allow cross node
    communication in the multi node ceph jobs. This raised security
    concerns, in particular it exposed memcached to the external net.

    This patch uses the infra provided role 'multi-node-firewall' in order
    to correctly configure iptables. Thanks to Jeremy Stanley and Jeffrey
    for help with this.

    Closes-Bug: #1749326
    Change-Id: Iafaf1cf1d9b0227b0f869969d0bd52fbde3791a0

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.0.0.0b2 development milestone.