CI: new centos7 image contains a regression that prevents multinode networking setup to work

[Emilien Macchi]

Affecting at least master and Pike (maybe more) but the undercloud fails to deploy because multinode setup hasn't finished correctly, the subnode-2 can't be pinged:

http://logs.openstack.org/75/508975/13/check/legacy-tripleo-ci-centos-7-nonha-multinode-oooq/9045427/job-output.txt.gz#_2017-10-20_18_29_35_247147

I did some quick research and I think this is in the new centos7 image, where we have updates on some core packages:
https://www.diffchecker.com/huccl3ol

See iptables, it was upgraded and could be the reason.
Release note of latest tag:
https://git.centos.org/blob/rpms!iptables.git/4d0bb22ecdc68576e7a4a7afd966207c50b664d2/SPECS!iptables.spec#L278

It's maybe not iptables, but this bug serves as placeholder.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/513848

[David Moreau Simard (dmsimard)]

Direct link to the spec diff between the two versions: https://git.centos.org/blobdiff/rpms!iptables.git/4d0bb22ecdc68576e7a4a7afd966207c50b664d2/SPECS!iptables.spec

[David Moreau Simard (dmsimard)]

It looks like there is a known issue in the last release of iptables: https://bugzilla.redhat.com/show_bug.cgi?id=1504647

[David Moreau Simard (dmsimard)]

Also: https://bugzilla.redhat.com/show_bug.cgi?id=1486803

[David Moreau Simard (dmsimard)]

Also note that a restart of iptables clears the rules, they are not properly persisted.

[David Moreau Simard (dmsimard)]

Persist iptables rules: https://review.openstack.org/#/c/513943/

[wes hayutin (weshayutin)]

Weird, things are working again w/o either:
https://review.openstack.org/513848 or
https://review.openstack.org/#/c/513943/

Don't see any discussion on #tripleo or #openstack-infra
The passing jobs are still using the new iptables
iptables-services.x86_64 1.4.21-18.2.el7_4 @updates

odd, anyone have any hints?

[wes hayutin (weshayutin)]

http://logs.openstack.org/57/509157/15/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq/06074ac/logs/undercloud/var/log/extra/yum-list-installed.txt.gz

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/513848

[Alex Schultz (alex-schultz)]

I think https://review.openstack.org/#/c/513943/ resolved this issue. We are not currently hitting this so closing it.