[system-tests] Sometimes firewall check returns false positive alerts

[Artem Panchenko]

Sometimes Firewall tests fails with "Unused port can be accessed" message:

http://jenkins-product.srt.mirantis.net:8080/job/6.0-icehouse.staging.centos.bvt_1/1/testReport/junit/(root)/deploy_neutron_gre/deploy_neutron_gre/

but after environment revert this can't be reproduced manually and iptables configuration looks ok. This probably can be caused by temporary iptables rules (I think OCF scripts or puppet can add some additional rules while configuring services) or glitch in tests. So it's necessary to increase verbosity of firewall check (e.g. add saving of iptables config) to have ability to investigate an issue after test failure.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/126870

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/126870
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=82e1c6e502f5094bf1ff91dc8f5dd4e7c5aab094
Submitter: Jenkins
Branch: master

commit 82e1c6e502f5094bf1ff91dc8f5dd4e7c5aab094
Author: Artem Panchenko <email address hidden>
Date: Wed Oct 8 13:59:22 2014 +0300

    Increase verbosity of Firewall check results

    Create dump of iptables rules before running check
    and increase verbosity of log message if test fails.
    Also do not remove temporary check file, just rename it.

    Change-Id: I55fa25e7ef0d2a42899b98600297741ad53d931c
    Closes-bug: #1378745

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/128290

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/128290
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=04904a82efa43572d9369bfc6c08d971be252555
Submitter: Jenkins
Branch: stable/5.1

commit 04904a82efa43572d9369bfc6c08d971be252555
Author: Artem Panchenko <email address hidden>
Date: Wed Oct 8 13:59:22 2014 +0300

    Increase verbosity of Firewall check results

    Create dump of iptables rules before running check
    and increase verbosity of log message if test fails.
    Also do not remove temporary check file, just rename it.

    Change-Id: I55fa25e7ef0d2a42899b98600297741ad53d931c
    Closes-bug: #1378745
    (cherry picked from commit 82e1c6e502f5094bf1ff91dc8f5dd4e7c5aab094)

[Artem Panchenko (apanchenko-8)]

The issue is still rarely reproduced on CI, for example:

https://fuel-jenkins.mirantis.com/job/5_1_fuellib_review_systest_centos/131/console

Additional investigation is needed to figure out the cause of that.

[Tatyanka (tatyana-leontovich)]

Mark as Won't fix for 5.1.1, acording to Medium status. Also this is test related issue, so it should not block nothing

[Tatyanka (tatyana-leontovich)]

======================================================================
ERROR: Deploy cluster in HA mode with Neutron VLAN and public network
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron/fuelweb_test/helpers/decorators.py", line 51, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron/fuelweb_test/tests/test_neutron.py", line 380, in deploy_neutron_vlan_ha_with_public_network
    self.fuel_web.security.verify_firewall(cluster_id)
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/5.1_fuelmain.system_test.ubuntu.ha_neutron/fuelweb_test/helpers/security.py", line 119, in verify_firewall
    tmp_file_path))
Exception: Firewall vulnerability detected. Unused port 792/tcp can be accessed on slave-02_controller (node-1) node. Check /var/tmp/iptables_check_file.old and /var/tmp/iptables_check_file.dump files on the node for details

[Tatyanka (tatyana-leontovich)]

Still sometimes failed, so move as low to 5.1.2

[Tatyanka (tatyana-leontovich)]

http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.ubuntu.ha_neutron/45/consoleFull

[Artem Panchenko (apanchenko-8)]

this issue wasn't reproduced for more than 3 month, moving it to invalid

[Maksym Strukov (unbelll)]

6.1 iso ver 300

======================================================================
ERROR: Deploy cluster in HA mode with Neutron VLAN and public network
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron/fuelweb_test/helpers/decorators.py", line 66, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron/fuelweb_test/tests/test_neutron.py", line 423, in deploy_neutron_vlan_ha_with_public_network
    self.fuel_web.security.verify_firewall(cluster_id)
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.ha_neutron/fuelweb_test/helpers/security.py", line 119, in verify_firewall
    tmp_file_path))
Exception: Firewall vulnerability detected. Unused port 4296/tcp can be accessed on slave-03_controller (node-2) node. Check /var/tmp/iptables_check_file.old and /var/tmp/iptables_check_file.dump files on the node for details
==========

/var/tmp/iptables_check_file.old and /var/tmp/iptables_check_file.dump is here http://paste.openstack.org/show/202102/

[Artem Panchenko (apanchenko-8)]

According to attached iptables rules dump, firewall on slave node was configured properly. I believe that such false positive alerts are caused by glitches in virtual environments (most probably hypervisor hardware was highly loaded), so as a workaround we can add @retry for firewall check in system tests.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/186408

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/186408
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=17f89f2f5b1e28297e0fce34ca37918b911e6b5d
Submitter: Jenkins
Branch: master

commit 17f89f2f5b1e28297e0fce34ca37918b911e6b5d
Author: Artem Panchenko <email address hidden>
Date: Thu May 28 17:26:27 2015 +0300

    Add retries for Firewall check

    Sometimes Firewall check fails due to virtual
    networks/machines glitches, but from the second
    try it passes. Add @retry decorator with default
    properties (try to run check 3 times and sleep
    for 30s between tries).

    Change-Id: I51a0532038231c85a6614146e82ad9097fcc5156
    Closes-bug: #1378745

[Artem Panchenko (apanchenko-8)]

This issue affects BVT for 5.0.3 milestone, see: bug #1530803

[Alexey Stupnikov (astupnikov)]

Set this bug's status to Won't fix (wontfix-munotapplic) for 5.0 branch since we don't provide MU for 5.0 branch