Iptables calls cause faults during deployment

Bug #1605540 reported by Denis Kostryukov
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Alex Schultz
7.0.x
Fix Released
High
Denis Meltsaykin
8.0.x
Fix Released
High
Anton Chevychalov
Mitaka
Fix Released
High
Alex Schultz

Bug Description

The iptables calls try to reach external and unreachable DNS servers. These calls cause errors in the deployment. It could be prevented if the calls would contain switches -n and -w.
Please add these switches (-n -w) to the iptables calls in the files located in fuel/library/files/fuel-ha-utils/ocf folder enlisted below:

nova-network: ocf_run iptables -t filter -F $filter_chain
nova-network: ocf_run iptables -t nat -F $nat_chain
nova-network: ocf_run iptables -t mangle -F $prefix-POSTROUTING
ns_haproxy: iptables -n -t nat -L | grep -q masquerade-for-haproxy-namespace
ns_haproxy: ocf_run iptables -n -t nat -A POSTROUTING -s "${OCF_RESKEY_namespace_ip}" -j MASQUERADE -m comment --comment "masquerade-for-haproxy-namespace"
ns_vrouter: iptables -n -t nat -L | grep -q masquerade-for-vrouter-namespace
ns_vrouter: ocf_run iptables -n -t nat -A POSTROUTING -s "${OCF_RESKEY_namespace_ip}" -j MASQUERADE -m comment --comment "masquerade-for-vrouter-namespace"
rabbitmq: until $(iptables -nvL | grep -q 'temporary RMQ block') || [ $tries -eq 0 ]; do
rabbitmq: iptables -I INPUT -p tcp -m tcp --dport ${OCF_RESKEY_node_port} -m state --state NEW,RELATED,ESTABLISHED \
rabbitmq: for i in $(iptables -nvL --line-numbers | awk '/temporary RMQ block/ {print $1}'); do
rabbitmq: iptables -D INPUT -p tcp -m tcp --dport ${OCF_RESKEY_node_port} -m state --state NEW,RELATED,ESTABLISHED \

It can cause errors during deployment a new environment and on an already installed one.

How to reproduce:

Make the DNS service unreachable and start to stop/start CRM resources.

The description of the case enlists the files and the calls where the changes should be done. To find them:

fuel/library/files/fuel-ha-utils/ocf$ grep -r -i "iptables " *

MOS7.0 drop_w23.

VERSION:

  feature_groups:

    - mirantis

  production: "docker"

  release: "7.0"

  openstack_version: "2015.1.0-7.0"

  api: "1.0"

  build_number: "852"

  build_id: "852"

  nailgun_sha: "864ff12f07568f528cde18a798006703a6b1f47c"

  python-fuelclient_sha: "80a97c3946460d434066192eec7cf8bf98453247"

  fuel-agent_sha: "d02d30e61fdcbc20176e56aa3eae3ec74e94804b"

  fuel-nailgun-agent_sha: "06305379a1cc2e7db992dfc3db6566766bc8fc31"

  astute_sha: "bc04aee030d019c77485ef4326b1b2d63229fab1"

  fuel-library_sha: "c29ab66b03223eb2b9e84bf6dd9feef8042f69eb"

  fuel-ostf_sha: "fce90d4add96ca9b196bb30b18d4cc8b5319b686"

  fuelmain_sha: "f45ec7e3571999d22396272a14281f9264636be5

Please provide a backport for MOS 7.0

Changed in mos:
importance: Undecided → High
tags: added: sla1
tags: added: area-library
Revision history for this message
Alex Schultz (alex-schultz) wrote :
no longer affects: mos/10.0.x
no longer affects: mos
no longer affects: mos/7.0.x
no longer affects: mos/8.0.x
no longer affects: mos/9.x
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/349923

Dmitry Pyzhov (dpyzhov)
tags: added: 9.1-proposed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/348074
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=28e3108dcf778f76899dc6b7f3a031ed8d6c6393
Submitter: Jenkins
Branch: master

commit 28e3108dcf778f76899dc6b7f3a031ed8d6c6393
Author: Alex Schultz <email address hidden>
Date: Wed Jul 27 16:41:19 2016 -0600

    Add -n to iptables calls

    This change updates the calls that we use in the ocf scripts to
    determine if the rules are present to include the -n flag to prevent
    unnecessary dns lookups which can lead to deployment failures if dns is
    unavailable.

    Change-Id: I17d04fbad6def1217429fc3c92bed997fd510eb8
    Closes-Bug: #1605540

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/7.0)

Reviewed: https://review.openstack.org/349923
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=40088d3ddfd1e1bcd9dd1f9be5ff14855c298e66
Submitter: Jenkins
Branch: stable/7.0

commit 40088d3ddfd1e1bcd9dd1f9be5ff14855c298e66
Author: Alex Schultz <email address hidden>
Date: Wed Jul 27 16:41:19 2016 -0600

    Add -n to iptables calls

    This change updates the calls that we use in the ocf scripts to
    determine if the rules are present to include the -n flag to prevent
    unnecessary dns lookups which can lead to deployment failures if dns is
    unavailable.

    (cherry picked from 28e3108dcf778f76899dc6b7f3a031ed8d6c6393)

    Change-Id: I17d04fbad6def1217429fc3c92bed997fd510eb8
    Closes-Bug: #1605540

tags: added: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/355518

Revision history for this message
Alex Schultz (alex-schultz) wrote :

It should be noted that we consume the upstream rabbitmq in master, so the changes for 7,8,9 need to include the rabbitmq fixes as well.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/355542

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/355551

Revision history for this message
Anton Chevychalov (achevychalov) wrote :

> It should be noted that we consume the upstream rabbitmq in master, so the changes for 7,8,9 need to include the rabbitmq fixes as well.

Alex, as far as I know possibility of rabbitmq backporting is under investigation.

Currently I see no version related changes in your master branch commits so I intend to backport them.

Revision history for this message
Gergo Kekesi (kekesigergo) wrote :

Hi Gents,

1.
Will you backport the -w option to MOS7.0 track as well?
Maybe I missed but I cannot see.

2.
You haven't touched the nova-network script. Is it because it's coming from upstream?

BR.
Gergo

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/355953

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/355965

no longer affects: fuel/newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/7.0)

Reviewed: https://review.openstack.org/355965
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=8fffe24cb00047cec9bbb355134f378be8d11010
Submitter: Jenkins
Branch: stable/7.0

commit 8fffe24cb00047cec9bbb355134f378be8d11010
Author: Alex Schultz <email address hidden>
Date: Mon Aug 15 09:48:57 2016 -0600

    Add --wait to iptables calls

    This change updates the calls that we us in the ocf scripts to determine
    if the rules are present to include the -w flag to prevent the scripts
    from failing if another iptables call is currently running. It has been
    reported that this can occur when the ocf scripts are running in
    parallel to the puppet deployment (firewall task)

    Change-Id: Ia603f5643720a5fa5407de36ca75830a7c3f57fa
    Closes-Bug: #1605540

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/355542
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=10dd6a082e03c0289b3571d25219b4e075753f15
Submitter: Jenkins
Branch: master

commit 10dd6a082e03c0289b3571d25219b4e075753f15
Author: Alex Schultz <email address hidden>
Date: Mon Aug 15 09:48:57 2016 -0600

    Add --wait to iptables calls

    This change updates the calls that we us in the ocf scripts to determine
    if the rules are present to include the -w flag to prevent the scripts
    from failing if another iptables call is currently running. It has been
    reported that this can occur when the ocf scripts are running in
    parallel to the puppet deployment (firewall task)

    Change-Id: Ia603f5643720a5fa5407de36ca75830a7c3f57fa
    Closes-Bug: #1605540

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/348073
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=c62122351969308ffa446f63dc9efff7eae79cef
Submitter: Jenkins
Branch: stable/mitaka

commit c62122351969308ffa446f63dc9efff7eae79cef
Author: Alex Schultz <email address hidden>
Date: Wed Jul 27 16:41:19 2016 -0600

    Add -n to iptables calls

    This change updates the calls that we use in the ocf scripts to
    determine if the rules are present to include the -n flag to prevent
    unnecessary dns lookups which can lead to deployment failures if dns is
    unavailable.

    Change-Id: I17d04fbad6def1217429fc3c92bed997fd510eb8
    Closes-Bug: #1605540

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/355551
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a0973ab77e67a3828e078fad36a0e9190de9bd85
Submitter: Jenkins
Branch: stable/mitaka

commit a0973ab77e67a3828e078fad36a0e9190de9bd85
Author: Alex Schultz <email address hidden>
Date: Mon Aug 15 09:48:57 2016 -0600

    Add --wait to iptables calls

    This change updates the calls that we us in the ocf scripts to determine
    if the rules are present to include the -w flag to prevent the scripts
    from failing if another iptables call is currently running. It has been
    reported that this can occur when the ocf scripts are running in
    parallel to the puppet deployment (firewall task)

    Change-Id: Ia603f5643720a5fa5407de36ca75830a7c3f57fa
    Closes-Bug: #1605540

Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

The first fix https://review.openstack.org/#/c/349923/ is verified on MOS 7.0 + MU5 updates.

Steps to verify:
1. Make the DNS service unreachable. For it on all controllers perform:
'iptables -I INPUT -p udp --sport 53 -s 10.109.12.1,10.109.10.2,10.109.10.1 -j DROP',
where 10.109.12.1,10.109.10.2,10.109.10.1 are taken from /etc/resolv.conf.
2. crm resource stop clone_p_haproxy
3. crm resource start clone_p_haproxy
4. watch --interval 1 'ps aux|grep iptables'
5. In the other console of the same controller run 'iptables -t nat -L'.

Before fix:
4. There is the process 'iptables -t nat -L' in the output of "watch --interval 1 'ps aux|grep iptables'".
5. The following message is observed: "Another app is currently holding the xtables lock. Perhaps you want to use the -w option?"

After fix:
4. There is no process 'iptables -t nat -L' in the output of "watch --interval 1 'ps aux|grep iptables'".
5. 'iptables -t nat -L' is finished successfully.

Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

As for the second fix https://review.openstack.org/#/c/355965/ for 7.0:
This fix of customer-found bug that can be hardly tested on our environments. So it was agreed with dev team that fix shouldn't break legacy. Results of swarm tests with that fix - 95%, no bugs connected with this patch was found. For details see https://mirantis.testrail.com/index.php?/plans/view/18869

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/8.0)

Reviewed: https://review.openstack.org/355518
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=02d7b33ab8a7a31dd3ac7995d23a21f6f7ccf9fd
Submitter: Jenkins
Branch: stable/8.0

commit 02d7b33ab8a7a31dd3ac7995d23a21f6f7ccf9fd
Author: Alex Schultz <email address hidden>
Date: Wed Jul 27 16:41:19 2016 -0600

    Add -n to iptables calls

    This change updates the calls that we use in the ocf scripts to
    determine if the rules are present to include the -n flag to prevent
    unnecessary dns lookups which can lead to deployment failures if dns is
    unavailable.

    Change-Id: I17d04fbad6def1217429fc3c92bed997fd510eb8
    Closes-Bug: #1605540
    (cherry picked from 28e3108dcf778f76899dc6b7f3a031ed8d6c6393)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/355953
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d900c3e3efbc0fd9e6ef2f7b62da32975aed96ed
Submitter: Jenkins
Branch: stable/8.0

commit d900c3e3efbc0fd9e6ef2f7b62da32975aed96ed
Author: Alex Schultz <email address hidden>
Date: Mon Aug 15 09:48:57 2016 -0600

    Add --wait to iptables calls

    This change updates the calls that we us in the ocf scripts to determine
    if the rules are present to include the -w flag to prevent the scripts
    from failing if another iptables call is currently running. It has been
    reported that this can occur when the ocf scripts are running in
    parallel to the puppet deployment (firewall task)

    Change-Id: Ia603f5643720a5fa5407de36ca75830a7c3f57fa
    Closes-Bug: #1605540

tags: added: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on 9.1 snapshot #207.

tags: removed: on-verification
Revision history for this message
Dmitry (dtsapikov) wrote :

Verified on 8.0+mu3

tags: added: on-verification
tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0rc1

This issue was fixed in the openstack/fuel-library 10.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 10.0.0

This issue was fixed in the openstack/fuel-library 10.0.0 release.

Revision history for this message
Sergey Novikov (snovikov) wrote :

Verified on MOS 10.0 (RC #2)

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.