centos-8 ironic inspector deploy fails due to iptables --flush

[sean mooney]

https://opendev.org/openstack/kolla-ansible/commit/86e83faeb1fd088d44c5108a5ec835eba6316b2d

intoduced Flush and delete ironic-inspector iptables chain to support upgrades from using iptable to using dnsmask for dhcp filtering.

executing iptables --flush "ironic-inspector" on centos-8 results in iptables: Operation not supported.

removing the workaround allows deployment to complete.

this shoudl either be made configurable or remove on centos 8 or removed in general.

i encoutered this suing master kolla ansible with the train-centos8 source contaienr form docker hub.

i belive this is related to the fact that iptables on centos-8 is a different implemantion bast on nf filters intead of legacy iptables.

on ubuntu 18.04 there is a iptables-legacy pacakge avaiable but centos 8 only support the nft implementation. so the existing implementation is not compatiable with centos-8

[sean mooney (sean-k-mooney)]

just incase its relevent i was using the centos-8 stream release on the host

(kolla-venv) [sean@workstation kolla-work-dir]$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

(kolla-venv) [sean@workstation kolla-work-dir]$ cat /etc/redhat-release
CentOS Linux release 8.1.1911 (Core)
(kolla-venv) [sean@workstation kolla-work-dir]$ uname -a
Linux workstation 5.6.5-1.el8.elrepo.x86_64 #1 SMP Thu Apr 16 14:12:46 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

[Mark Goddard (mgoddard)]

Thanks for raising Sean, looks like we need to install iptables to the host.

[sean mooney (sean-k-mooney)]

iptables is installed on the host but centos uses a different version based on nf tables
which replaces the legacy implementation

if i just remove
https://github.com/openstack/kolla-ansible/blob/7a34f82a608d488dc10f46be9ff146ab80d08360/ansible/roles/ironic/tasks/deploy.yml#L27-L42

then the deploy compelte correctly.

executing iptables --flush "ironic-inspector" fails on the host possibly because the chain does not existit i can check that as it might be a delta in the nft based implematnion vs the legacy one.

i think we shoudl either remove that code or we shoudl add a check and only run it if the code exists. if we do that then you could also use the ansible module as apparently the reason you chose the commanline was the fact it ignored non existing chains.

[Mark Goddard (mgoddard)]

The code exists for users switching from iptables to dnsmasq as the PXE filter. It's not just a one time upgrade transition - it's configurable. So I think it needs to stay.

From the manpage:

-F, --flush [chain]
Flush the selected chain (all the chains in the table if none is given). This is equivalent to deleting all the rules one by one.

So we could fall back to deleting rules one by one. Possibly all at once if we can do some wildcard match:

-D, --delete chain rule-specification
-D, --delete chain rulenum
Delete one or more rules from the selected chain. There are two versions of this command: the rule can be specified as a number in the chain (starting at 1 for the first rule) or a rule to match.

[sean mooney (sean-k-mooney)]

that will not fix it.

the issue is the chain does not exist because this was a new install

sean@workstation ~]$ sudo iptables -N sean-test
[sean@workstation ~]$ sudo iptables --flush sean-test
[sean@workstation ~]$ sudo iptables -X sean-test
[sean@workstation ~]$ sudo iptables --flush sean-test
iptables: Operation not supported.

so the code need to be updated to check if the chain exsits and only delete it if it does

in my case the ironic-inspector chain neve exsited and in the nft implementation its is an error to delete a chain that does not exist.

[Radosław Piliszek (yoctozepto)]

Hmm, that makes sense. Wonder why CI is not picky about it hmm. There is a lot different from CentOS 8 ISO installation and our CI image, e.g. we get a collection of system-wide-pip-installed packages. This seems another differences, hence we are not testing any real deployment but very custom environment most likely.

[Mark Goddard (mgoddard)]

Testing on a CentOS 8.1 cloud image, this works fine:

$ sudo iptables -N test
$ sudo iptables --flush test
$ sudo iptables -X test
$ sudo iptables --flush test
iptables: No chain/target/match by that name.

I have the iptables package installed, but no nftables.

[sean mooney (sean-k-mooney)]

that is likely a delta between the centos 8 stream image and centos 8
the centos 8 stream release should track rhel 8 much more closely.
rhel8 only suport the ntabels verion of iptables which treats it as an error.
the centos 8 stream release also has the same behavior as rhel 8