Security vulnerability: OpenStack APIs and Horizon Web UI are prone to DOS attacks

[Adam Heczko]

Affected versions: MOS 7.0

It was observed that OpenStack APIs and Horizon are prone to DOS attack. Flooding http endpoints with large amount of malicious requests could lead to services malfunction.

Proposed solution:
Apply haproxy http rate request limiting. Consult scale team to develop appropriate limit values for various APIs, e.g. Glance image operations probably needs much lower values of http requests than Horizon.
Example haproxy configuration: https://github.com/dschneller/haproxy-http-based-rate-limiting/blob/master/haproxy.cfg

Explanation of that configuration:
https://blog.codecentric.de/en/2014/12/haproxy-http-header-rate-limiting/

[Matthew Mosesohn (raytrac3r)]

@Georgy, can you please research and propose some haproxy limits for each OpenStack API? Doing limits on the haproxy load balancer is more effective than the DB-intensive limits in the APIs themselves.

[Mike Scherbakov (mihgen)]

To implement haproxy config changes, recommendation are required from OpenStack team on those.

[Boris Bobrov (bbobrov)]

I've marked 1555563 as a duplicate of this bugreport and set milestones from there.

[Dina Belova (dbelova)]

Sorry, accidentally removed Newton series. Fuel drivers, please move it back.

[Vitaly Sedelnik (vsedelnik)]

Confirmed for updates milestones, priority downgraded from Critical to Medium (see below for explanation)

This is not security vulnerability. Keystone endpoint is public in our default configuration, and as any service it has limited capacity and could be overloaded with number of requests that exceeds that capacity. If DoS attack is happening one could take appropriate measures - ban IPs the requests are coming from, filter specific types of packets, etc. There is nothing Keystone specific or OpenStack specific here.

Now this looks as feature request to put some limits on number of incoming requests. That should be discussed with product mgmt, scale and architecture groups. Then we could look into what could be done for older releases and existing deployments.

[Alexey Stupnikov (astupnikov)]

Set this bug's status to Won't Fix (wontfix-munotapplic) for 5.1 and 6.0 branch since we don't deliver Fuel fixes in maintenance updates for 5.1.1 and 6.0.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Ivan Berezovskiy (iberezovskiy)]

It's too late to fix medium bug in 9.0

[Adam Heczko (aheczko-mirantis)]

Proposed approach is described here:
https://javapipe.com/iptables-ddos-protection

[Rodion Tikunov (rtikunov)]

Closing this bug as Won't Fix for MOS 6.1 as we have finished active support for those releases and don't merge non-critical and non-security fixes there anymore.

[Denis Meltsaykin (dmeltsaykin)]

Closing as Won't Fix for 8.0 as this is a medium importance arguable feature-request.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/384994

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/384994
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0fb99f1ca7fe0925051060d552368bde31734c9e
Submitter: Jenkins
Branch: master

commit 0fb99f1ca7fe0925051060d552368bde31734c9e
Author: Oleksiy Molchanov <email address hidden>
Date: Mon Oct 24 16:23:12 2016 +0300

    Add few ddos protection rules to iptables

    Change-Id: I771cfdf7db9acd0c4de7fba8b775f66166c0b461
    Partial-Bug: 1509986

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/399497

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/399498

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/399498
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=607701ab9ad5595ac5aa7a631c37b91285555b01
Submitter: Jenkins
Branch: stable/newton

commit 607701ab9ad5595ac5aa7a631c37b91285555b01
Author: Oleksiy Molchanov <email address hidden>
Date: Mon Oct 24 16:23:12 2016 +0300

    Add few ddos protection rules to iptables

    Change-Id: I771cfdf7db9acd0c4de7fba8b775f66166c0b461
    Partial-Bug: 1509986
    (cherry picked from commit 0fb99f1ca7fe0925051060d552368bde31734c9e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/399497
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=3c3fa05c3d1725d00c62ab7ba3fcb97c67b4a0b7
Submitter: Jenkins
Branch: stable/mitaka

commit 3c3fa05c3d1725d00c62ab7ba3fcb97c67b4a0b7
Author: Oleksiy Molchanov <email address hidden>
Date: Mon Oct 24 16:23:12 2016 +0300

    Add few ddos protection rules to iptables

    Change-Id: I771cfdf7db9acd0c4de7fba8b775f66166c0b461
    Partial-Bug: 1509986
    (cherry picked from commit 0fb99f1ca7fe0925051060d552368bde31734c9e)

[Dmitry Pyzhov (dpyzhov)]

All patches are merged.

[Ilya Bumarskov (ibumarskov)]

Verified on snapshot-id #822
Firewall rules were apllied on all nodes (include nodes with compute and cinder roles) in cluster:

root@node-2:~# iptables --list | grep block
DROP all -- anywhere anywhere /* 010 block invalid packets */ ctstate INVALID
DROP tcp -- anywhere anywhere tcp flags:!FIN,SYN,RST,ACK/SYN /* 020 block not-syn new packets */ ctstate NEW
DROP tcp -- anywhere anywhere /* 030 block uncommon mss values */ ctstate NEW tcpmss match !536:65535
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE /* 040 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN/FIN,SYN /* 050 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:SYN,RST/SYN,RST /* 060 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN/FIN,SYN /* 070 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,RST/FIN,RST /* 080 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,ACK/FIN /* 090 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:ACK,URG/URG /* 100 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,ACK/FIN /* 110 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:PSH,ACK/PSH /* 120 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,PSH,ACK,URG /* 130 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/NONE /* 140 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,PSH,URG /* 150 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,PSH,URG /* 160 block packets with bogus tcp flags */
DROP tcp -- anywhere anywhere tcp flags:FIN,SYN,RST,PSH,ACK,URG/FIN,SYN,RST,ACK,URG /* 170 block packets with bogus tcp flags */