tripleo

newer versions of docker switch FORWARD chain to DROP by default

Bug #1750194 reported by Alex Schultz
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Alex Schultz
tripleo queens-rc1

Bug Description

see https://github.com/moby/moby/pull/28257

Newer versions of docker cause the FORWARD chain default action to be switched to DROP on the undercloud when it it is installed. This causes issues with deployments because traffic is no longer passing as expected.

Alex Schultz (alex-schultz)
Changed in tripleo:
assignee: nobody → Alex Schultz (alex-schultz)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/545707

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (master)

Fix proposed to branch: master
Review: https://review.openstack.org/545710

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/545707
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=21101149f269f1374a5a16d13c7f26228d4a7ec1
Submitter: Zuul
Branch: master

commit 21101149f269f1374a5a16d13c7f26228d4a7ec1
Author: Alex Schultz <email address hidden>
Date: Sun Feb 18 11:01:04 2018 -0700

    Add firewall chain support

    Add ability to manage firewall chains with the firewallchain resource.

    Change-Id: Ib75f97748540b9162d76c9c189d3ca7e082b3784
    Related-Bug: #1750194

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/545999

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (master)

Reviewed: https://review.openstack.org/545710
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=da31c6e5baaf4abf01a6c2232a0dbac2513f0e06
Submitter: Zuul
Branch: master

commit da31c6e5baaf4abf01a6c2232a0dbac2513f0e06
Author: Alex Schultz <email address hidden>
Date: Sun Feb 18 11:07:22 2018 -0700

    Ensure FORWARD is ACCEPT by default

    Newer versions of docker have switched to updating the FORWARD chain to
    be DROP by default. This causes issues with the deployment. Update the
    FORWARD chains to be ACCEPT by default.

    Depends-On: Ib75f97748540b9162d76c9c189d3ca7e082b3784
    Closes-Bug: #1750194

    Change-Id: I93be7138e6a61cf3aadf19f53097d67469befc17

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/545999
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a1ec856e61532daa49f38683857918fd2cc561aa
Submitter: Zuul
Branch: master

commit a1ec856e61532daa49f38683857918fd2cc561aa
Author: Alex Schultz <email address hidden>
Date: Mon Feb 19 15:10:01 2018 -0700

    Add firewall chain configuration

    Adds the ability to specify firewall chains via heat templates.
    Additionally newer versions of docker have switched to updating
    the FORWARD chain to DROP by default. Neutron needs this to be
    ACCEPT by default. This change adds the ability to specify
    firewall chains via templates.

    Depends-On: Ib75f97748540b9162d76c9c189d3ca7e082b3784
    Change-Id: I15ec9216013a1b0b935dcd1f5bc8281348777189
    Related-Bug: #1750194

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/546236

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/546237

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/546238

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to instack-undercloud (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/547221

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/547224

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (master)

Fix proposed to branch: master
Review: https://review.openstack.org/547281

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on instack-undercloud (master)

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/547224
Reason: https://review.openstack.org/547281

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/547221
Reason: https://review.openstack.org/547281

Bogdan Dobrelya (bogdando)
Changed in tripleo:
status: Fix Released → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/548398

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/548401

Alex Schultz (alex-schultz)
Changed in tripleo:
importance: High → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to instack-undercloud (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/548416

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (master)

Reviewed: https://review.openstack.org/547281
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=bfb758b5e792c83e5cde9847bcad424fcfaf071d
Submitter: Zuul
Branch: master

commit bfb758b5e792c83e5cde9847bcad424fcfaf071d
Author: Alex Schultz <email address hidden>
Date: Thu Feb 22 23:01:49 2018 -0700

    Fix bootstrap NAT

    Docker will switch the FORWARD filter to DROP if it sets the ip_forward
    to 1. Previously we were doing this in a post configuration element
    rather than in the puppet run itself. This change moves the ip_forward=1
    to puppet so it runs prior to docker being installed. Additionally we
    are ensuring that the full set of network rules are being added to the
    FORWARD filter because previously we were only setting half of them.
    This would allow us to actually not have to use ACCEPT as the default
    for the FORWARD filter but this would require additional testing.

    Previously we had tried switching the default policy back to ACCEPT,
    however given that docker is not configuring the iptables rule until
    it's installed and started, the puppet rules do not actually apply on
    the installation of the undercloud. The puppet management of the
    defaults for the FORWARD chain only gets updated on a subsequent run of
    the installer which will not work.

    Change-Id: Ieae6a74f7269bd64606fd80a2a08b2058c24d2c5
    Closes-Bug: #1750194
    Closes-Bug: #1750874

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/548616

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (stable/pike)

Reviewed: https://review.openstack.org/548616
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=50217d7a93dce7fdc17c0dfbb04260f86fd3ac7d
Submitter: Zuul
Branch: stable/pike

commit 50217d7a93dce7fdc17c0dfbb04260f86fd3ac7d
Author: Alex Schultz <email address hidden>
Date: Thu Feb 22 23:01:49 2018 -0700

    Fix bootstrap NAT

    Docker will switch the FORWARD filter to DROP if it sets the ip_forward
    to 1. Previously we were doing this in a post configuration element
    rather than in the puppet run itself. This change moves the ip_forward=1
    to puppet so it runs prior to docker being installed. Additionally we
    are ensuring that the full set of network rules are being added to the
    FORWARD filter because previously we were only setting half of them.
    This would allow us to actually not have to use ACCEPT as the default
    for the FORWARD filter but this would require additional testing.

    Conflicts:
     instack_undercloud/tests/test_undercloud.py
     instack_undercloud/undercloud.py

    Change-Id: Ieae6a74f7269bd64606fd80a2a08b2058c24d2c5
    Closes-Bug: #1750194
    Closes-Bug: #1750874
    (cherry picked from commit bfb758b5e792c83e5cde9847bcad424fcfaf071d)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on instack-undercloud (stable/pike)

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/546237

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on puppet-tripleo (stable/pike)

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/546236

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-heat-templates (stable/pike)

Change abandoned by Alex Schultz (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/546238

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/instack-undercloud 8.3.0

This issue was fixed in the openstack/instack-undercloud 8.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/549854

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on puppet-tripleo (master)

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/548398
Reason: this won't work due to stages, we need to be targeted

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on instack-undercloud (master)

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/548416
Reason: this doesn't work

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-tripleo (master)

Reviewed: https://review.openstack.org/549854
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=20cdbd3c6b18bb83e801d4cd7b1d23e38168e055
Submitter: Zuul
Branch: master

commit 20cdbd3c6b18bb83e801d4cd7b1d23e38168e055
Author: Alex Schultz <email address hidden>
Date: Mon Mar 5 12:23:05 2018 -0700

    Ensure ip_forward set before Docker

    Docker will attempt to configure iptables rules if it finds that
    ip_forward is not enabled prior to the docker daemon getting started.
    We should ensure that this gets configured prior to Docker being
    configured if we are configuring it with puppet.

    Change-Id: I6ea6fb8ed300d284c961e7474ff84d104f326255
    Needed-By: I557e4a41c4e5be3a2f50e5d5ddc86e17c1eb44e1
    Related-Bug: #1750194

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/550289

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/550290

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/instack-undercloud 7.4.10

This issue was fixed in the openstack/instack-undercloud 7.4.10 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-tripleo (stable/pike)

Reviewed: https://review.openstack.org/550290
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=7a1b8c7ab47ccece6a14d1d3c3fb414a18c83fde
Submitter: Zuul
Branch: stable/pike

commit 7a1b8c7ab47ccece6a14d1d3c3fb414a18c83fde
Author: Alex Schultz <email address hidden>
Date: Mon Mar 5 12:23:05 2018 -0700

    Ensure ip_forward set before Docker

    Docker will attempt to configure iptables rules if it finds that
    ip_forward is not enabled prior to the docker daemon getting started.
    We should ensure that this gets configured prior to Docker being
    configured if we are configuring it with puppet.

    Change-Id: I6ea6fb8ed300d284c961e7474ff84d104f326255
    Needed-By: I557e4a41c4e5be3a2f50e5d5ddc86e17c1eb44e1
    Related-Bug: #1750194
    (cherry picked from commit 20cdbd3c6b18bb83e801d4cd7b1d23e38168e055)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to puppet-tripleo (stable/queens)

Reviewed: https://review.openstack.org/550289
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=2083903250d4e5159831b7b4e47acd9116b5e1f0
Submitter: Zuul
Branch: stable/queens

commit 2083903250d4e5159831b7b4e47acd9116b5e1f0
Author: Alex Schultz <email address hidden>
Date: Mon Mar 5 12:23:05 2018 -0700

    Ensure ip_forward set before Docker

    Docker will attempt to configure iptables rules if it finds that
    ip_forward is not enabled prior to the docker daemon getting started.
    We should ensure that this gets configured prior to Docker being
    configured if we are configuring it with puppet.

    Change-Id: I6ea6fb8ed300d284c961e7474ff84d104f326255
    Needed-By: I557e4a41c4e5be3a2f50e5d5ddc86e17c1eb44e1
    Related-Bug: #1750194
    (cherry picked from commit 20cdbd3c6b18bb83e801d4cd7b1d23e38168e055)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/548401
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=75ee85b1e45b09ac3093d3ace1112d5c3be18074
Submitter: Zuul
Branch: master

commit 75ee85b1e45b09ac3093d3ace1112d5c3be18074
Author: Alex Schultz <email address hidden>
Date: Tue Feb 27 12:52:55 2018 -0700

    Add KernelIpForward configuration

    Expose the configuration of net.ipv4.ip_forward via the kernel service.

    Depends-On: I6ea6fb8ed300d284c961e7474ff84d104f326255
    Change-Id: I557e4a41c4e5be3a2f50e5d5ddc86e17c1eb44e1
    Related-Bug: #1750194

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/550567

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/550567
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=25eba575b5cc95991d890739cb551bc91f1316cb
Submitter: Zuul
Branch: stable/queens

commit 25eba575b5cc95991d890739cb551bc91f1316cb
Author: Alex Schultz <email address hidden>
Date: Tue Feb 27 12:52:55 2018 -0700

    Add KernelIpForward configuration

    Expose the configuration of net.ipv4.ip_forward via the kernel service.

    Depends-On: I6ea6fb8ed300d284c961e7474ff84d104f326255
    Change-Id: I557e4a41c4e5be3a2f50e5d5ddc86e17c1eb44e1
    Related-Bug: #1750194
    (cherry picked from commit 75ee85b1e45b09ac3093d3ace1112d5c3be18074)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/551335

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/551340

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (stable/newton)

Reviewed: https://review.openstack.org/551340
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=663dad2a37fed795e78c6911e0338c03977c66e6
Submitter: Zuul
Branch: stable/newton

commit 663dad2a37fed795e78c6911e0338c03977c66e6
Author: Alex Schultz <email address hidden>
Date: Thu Feb 22 23:01:49 2018 -0700

    Fix bootstrap NAT

    Docker will switch the FORWARD filter to DROP if it sets the ip_forward
    to 1. Previously we were doing this in a post configuration element
    rather than in the puppet run itself. This change moves the ip_forward=1
    to puppet so it runs prior to docker being installed. Additionally we
    are ensuring that the full set of network rules are being added to the
    FORWARD filter because previously we were only setting half of them.
    This would allow us to actually not have to use ACCEPT as the default
    for the FORWARD filter but this would require additional testing.

    Conflicts:
     elements/puppet-stack-config/puppet-stack-config.yaml.template
     elements/undercloud-install/os-refresh-config/post-configure.d/98-undercloud-setup

    Change-Id: Ieae6a74f7269bd64606fd80a2a08b2058c24d2c5
    Closes-Bug: #1750194
    Closes-Bug: #1750874
    (cherry picked from commit bfb758b5e792c83e5cde9847bcad424fcfaf071d)
    (cherry picked from commit 50217d7a93dce7fdc17c0dfbb04260f86fd3ac7d)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (stable/ocata)

Reviewed: https://review.openstack.org/551335
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=256fecbf508f0753175835e9c685e3e49399b88d
Submitter: Zuul
Branch: stable/ocata

commit 256fecbf508f0753175835e9c685e3e49399b88d
Author: Alex Schultz <email address hidden>
Date: Thu Feb 22 23:01:49 2018 -0700

    Fix bootstrap NAT

    Docker will switch the FORWARD filter to DROP if it sets the ip_forward
    to 1. Previously we were doing this in a post configuration element
    rather than in the puppet run itself. This change moves the ip_forward=1
    to puppet so it runs prior to docker being installed. Additionally we
    are ensuring that the full set of network rules are being added to the
    FORWARD filter because previously we were only setting half of them.
    This would allow us to actually not have to use ACCEPT as the default
    for the FORWARD filter but this would require additional testing.

    Conflicts:
     elements/puppet-stack-config/puppet-stack-config.yaml.template
     elements/undercloud-install/os-refresh-config/post-configure.d/98-undercloud-setup

    Change-Id: Ieae6a74f7269bd64606fd80a2a08b2058c24d2c5
    Closes-Bug: #1750194
    Closes-Bug: #1750874
    (cherry picked from commit bfb758b5e792c83e5cde9847bcad424fcfaf071d)
    (cherry picked from commit 50217d7a93dce7fdc17c0dfbb04260f86fd3ac7d)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/instack-undercloud 6.1.6

This issue was fixed in the openstack/instack-undercloud 6.1.6 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/instack-undercloud 5.3.8

This issue was fixed in the openstack/instack-undercloud 5.3.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.