Docker CE can cause ironic provisioning to fail
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
kolla-ansible |
Fix Released
|
High
|
Mark Goddard | ||
Rocky |
New
|
Medium
|
Mark Goddard | ||
Stein |
Fix Released
|
High
|
Mark Goddard | ||
Train |
Fix Released
|
High
|
Mark Goddard |
Bug Description
Recently, Kolla Ansible switched to installing Docker CE by default in the 'kolla-ansible bootstrap-servers' command. This moves us forward from the old 1.12 legacy package we were using previously. These changes will be included in the Stein release.
When using Ironic with Docker CE and the Ironic Inspector iptables PXE filter, provisioning of bare metal nodes fails. This has so far only been seen in Kayobe CI, and not yet reproduced on real hardware.
The cause of the issue appears to be that Docker sets the default policy of the FORWARD chain to DROP since version 13.1 [1]. In the nova role we set the following sysctl, meaning that bridged L2 traffic is processed by iptables:
net.bridge.
When ironic inspector is used with the iptables PXE filter (ironic_
For some reason, the combination of the default DROP policy on the FORWARD chain and the inspector iptables PXE filter causes DHCP packets to get dropped before they get to the neutron OVS bridges. I suspect this depends on the network topology, and might not affect systems with a separate inspection/
A few things I've tried that avoid this issue:
* Use a different PXE filter, e.g. 'dnsmasq'. This is actually recommended over iptables.
* Set the net.bridge.
Some things that might work that I haven't tried:
* Set the default iptables policy on the FORWARD chain to ACCEPT. Docker will probably try to revert this change if it is restarted
* Set the iptables Docker config option to false to prevent it from configuring iptables. This is fine for Kolla Ansible, but if it is used for any other containers that don't use host networking, then it will cause problems.
I think the sensible thing to do here is to change the default value of ironic_
[1] https:/
Note: support for ironic_ inspector_ pxe_filter was only added in the Stein release. We only use Docker CE in Rocky by default for Ubuntu, but it is possible to change docker_ legacy_ packages to false on other distros in Rocky to use Docker CE.