[OVN] Add support for Baremetal provisioning with ML2/OVN with IPv4

[Lucas Alvares Gomes]

The ML2/OVN driver does not yet officially support provisioning baremetal nodes. This bug is a tracker for that work.

I believe the work can be broken up in two parts:

Part 1: Add support for provisioning baremetal nodes with ML2/OVN + Neutron DHCP agent. Since the OVN built-in DHCP server does not yet fully support chainloading from PXE to iPXE the user will need to use the Neutron DHCP agent to handle the DHCP part but everything else, including metadata, should be done by OVN.

Part 2: This will have OVN built-in DHCP server handling the chainloading from PXE to iPXE. Neutron DHCP agent no longer will be needed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/840287

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/840316

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840287
Committed: https://opendev.org/openstack/neutron/commit/243c209eb2cf75570e6c850b7d5becb468e8d1ab
Submitter: "Zuul (22348)"
Branch: master

commit 243c209eb2cf75570e6c850b7d5becb468e8d1ab
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri Apr 30 09:42:17 2021 +0100

    [OVN] Add baremetal support with Neutron DHCP agent

    This patch now creates OVN "external" ports for Neutron ports with
    VNIC_BAREMETAL. This ports will be scheduled on the OpenStack Controller
    nodes (or OVN Gateway nodes) and are responsible for replying to the
    ARP requests coming from the baremetal nodes.

    This patch also disables OVN's built-in DHCP server for VNIC_BAREMETAL
    ports. This is because OVN DHCP server does not yet fully support
    chainloading from PXE to iPXE, this feature is work-in-progress right
    now. A following patch to this one will be sent in the future adding
    support for OVN's built-in DHCP server being used with baremetal nodes.

    This patch implements the "Part 1" from bug #1971431.

    Partial-bug: #1971431
    Change-Id: I6b234fbe1b7c54b41a1b8b430fdf0ac76993af96
    Signed-off-by: Lucas Alvares Gomes <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/840886

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/840887

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/840888

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/840889

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/840890

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840887
Committed: https://opendev.org/openstack/neutron/commit/bf65c0c33a05f5d44000c25772296828cd8dc4bc
Submitter: "Zuul (22348)"
Branch: stable/xena

commit bf65c0c33a05f5d44000c25772296828cd8dc4bc
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri Apr 30 09:42:17 2021 +0100

    [OVN] Add baremetal support with Neutron DHCP agent

    This patch now creates OVN "external" ports for Neutron ports with
    VNIC_BAREMETAL. This ports will be scheduled on the OpenStack Controller
    nodes (or OVN Gateway nodes) and are responsible for replying to the
    ARP requests coming from the baremetal nodes.

    This patch also disables OVN's built-in DHCP server for VNIC_BAREMETAL
    ports. This is because OVN DHCP server does not yet fully support
    chainloading from PXE to iPXE, this feature is work-in-progress right
    now. A following patch to this one will be sent in the future adding
    support for OVN's built-in DHCP server being used with baremetal nodes.

    This patch implements the "Part 1" from bug #1971431.

    Conflicts:
      neutron/common/ovn/constants.py
      neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py

    Partial-bug: #1971431
    Change-Id: I6b234fbe1b7c54b41a1b8b430fdf0ac76993af96
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 243c209eb2cf75570e6c850b7d5becb468e8d1ab)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840888
Committed: https://opendev.org/openstack/neutron/commit/2e1ba361577e59ca6f8a8227e75c3b2195ce4f93
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 2e1ba361577e59ca6f8a8227e75c3b2195ce4f93
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri Apr 30 09:42:17 2021 +0100

    [OVN] Add baremetal support with Neutron DHCP agent

    This patch now creates OVN "external" ports for Neutron ports with
    VNIC_BAREMETAL. This ports will be scheduled on the OpenStack Controller
    nodes (or OVN Gateway nodes) and are responsible for replying to the
    ARP requests coming from the baremetal nodes.

    This patch also disables OVN's built-in DHCP server for VNIC_BAREMETAL
    ports. This is because OVN DHCP server does not yet fully support
    chainloading from PXE to iPXE, this feature is work-in-progress right
    now. A following patch to this one will be sent in the future adding
    support for OVN's built-in DHCP server being used with baremetal nodes.

    This patch implements the "Part 1" from bug #1971431.

    Conflicts:
      neutron/common/ovn/constants.py
      neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py

    Partial-bug: #1971431
    Change-Id: I6b234fbe1b7c54b41a1b8b430fdf0ac76993af96
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 243c209eb2cf75570e6c850b7d5becb468e8d1ab)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840890
Committed: https://opendev.org/openstack/neutron/commit/d9fae7b3325ea21e815dea60700d20837593371e
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit d9fae7b3325ea21e815dea60700d20837593371e
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri Apr 30 09:42:17 2021 +0100

    [OVN] Add baremetal support with Neutron DHCP agent

    This patch now creates OVN "external" ports for Neutron ports with
    VNIC_BAREMETAL. This ports will be scheduled on the OpenStack Controller
    nodes (or OVN Gateway nodes) and are responsible for replying to the
    ARP requests coming from the baremetal nodes.

    This patch also disables OVN's built-in DHCP server for VNIC_BAREMETAL
    ports. This is because OVN DHCP server does not yet fully support
    chainloading from PXE to iPXE, this feature is work-in-progress right
    now. A following patch to this one will be sent in the future adding
    support for OVN's built-in DHCP server being used with baremetal nodes.

    This patch implements the "Part 1" from bug #1971431.

    Conflicts:
      neutron/common/ovn/constants.py
      neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py

    Partial-bug: #1971431
    Change-Id: I6b234fbe1b7c54b41a1b8b430fdf0ac76993af96
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 243c209eb2cf75570e6c850b7d5becb468e8d1ab)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840889
Committed: https://opendev.org/openstack/neutron/commit/bca80bfbfb63c805ff2d3e2ac5e2bfdaf27ce23d
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit bca80bfbfb63c805ff2d3e2ac5e2bfdaf27ce23d
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri Apr 30 09:42:17 2021 +0100

    [OVN] Add baremetal support with Neutron DHCP agent

    This patch now creates OVN "external" ports for Neutron ports with
    VNIC_BAREMETAL. This ports will be scheduled on the OpenStack Controller
    nodes (or OVN Gateway nodes) and are responsible for replying to the
    ARP requests coming from the baremetal nodes.

    This patch also disables OVN's built-in DHCP server for VNIC_BAREMETAL
    ports. This is because OVN DHCP server does not yet fully support
    chainloading from PXE to iPXE, this feature is work-in-progress right
    now. A following patch to this one will be sent in the future adding
    support for OVN's built-in DHCP server being used with baremetal nodes.

    This patch implements the "Part 1" from bug #1971431.

    Conflicts:
      neutron/common/ovn/constants.py
      neutron/tests/unit/plugins/ml2/drivers/ovn/mech_driver/test_mech_driver.py

    Partial-bug: #1971431
    Change-Id: I6b234fbe1b7c54b41a1b8b430fdf0ac76993af96
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 243c209eb2cf75570e6c850b7d5becb468e8d1ab)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840886
Committed: https://opendev.org/openstack/neutron/commit/f181a8f3b74376a497b38170e56dfc177b6cf6bc
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit f181a8f3b74376a497b38170e56dfc177b6cf6bc
Author: Lucas Alvares Gomes <email address hidden>
Date: Fri Apr 30 09:42:17 2021 +0100

    [OVN] Add baremetal support with Neutron DHCP agent

    This patch now creates OVN "external" ports for Neutron ports with
    VNIC_BAREMETAL. This ports will be scheduled on the OpenStack Controller
    nodes (or OVN Gateway nodes) and are responsible for replying to the
    ARP requests coming from the baremetal nodes.

    This patch also disables OVN's built-in DHCP server for VNIC_BAREMETAL
    ports. This is because OVN DHCP server does not yet fully support
    chainloading from PXE to iPXE, this feature is work-in-progress right
    now. A following patch to this one will be sent in the future adding
    support for OVN's built-in DHCP server being used with baremetal nodes.

    This patch implements the "Part 1" from bug #1971431.

    Partial-bug: #1971431
    Change-Id: I6b234fbe1b7c54b41a1b8b430fdf0ac76993af96
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 243c209eb2cf75570e6c850b7d5becb468e8d1ab)

[Liu Xie (liushy)]

Hi Lucas, very nice to see your work!
I have one dout about traffic between baremetal and vms based on one vlan network:
TOR would braodcasting the traffic because arp of baremetal nics are already ageing on TOR and ovn-controller would be arp-proxy for baremetal nics. So how could we avoid the braodcasting events occur?
Maybe it is a matter for ovn.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/840316
Committed: https://opendev.org/openstack/neutron/commit/e73a85f3dd15aea2564a34f36261cd4c03128450
Submitter: "Zuul (22348)"
Branch: master

commit e73a85f3dd15aea2564a34f36261cd4c03128450
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue May 3 14:37:46 2022 +0100

    [OVN] Add baremetal support without Neutron DHCP agent for IPv4

    This patch adds support for deploying baremetal nodes with OVN's
    built-in DHCP server for IPv4.

    Since Neutron API's for setting DHCP options is mostly a pass-thru,
    Ironic uses a dnsmasq syntax for setting the baremetal options [0].
    Since this syntax is unlikely to change and it's only a tiny subset of
    what dnsmasq can offer this patch does translate that syntax used by
    Ironic and convert it to OVN's equivalent options. In this way we do not
    need to re-design Neutron's DHCP options API nor change Ironic to use it
    with ML2/OVN.

    This option also adds a new configuration option called
    "disable_ovn_dhcp_for_baremetal_ports". PXE booting nodes can be very
    sensitive and operators may prefer to use a fully-fledged DHCP server to
    do it (even Ironic makes DHCP pluggable). So if operators wish to
    disable OVN's built-in DHCP server for baremetal provisioning they can
    do so by setting this new option to True. It defaults to False.

    This change has been tested with real hardware and it does work. That
    said, we found a problem in core OVN itself [1] while testing it that
    can affect PXE from reaching the TFTP server, we already communicated
    this with the core OVN folks and we hope it can be fixed soon. The
    change in core OVN should not affect the Neutron change tho.

    Not that the "server-ip-address" DHCP Option now points to the
    "next_server" option in OVN instead of the "tftp_server_address". The
    previous behavior was wrong, the "server-ip-address" should set the
    "siaddr" in the DHCP header and this has been introduced in OVN [2] as
    an option called "next_server".

    [0]
    https://github.com/openstack/ironic/blob/49113385e89c52b56152418d3a0c8c69ddaf8b6e/ironic/common/pxe_utils.py#L523-L538
    [1]
    https://mail.openvswitch.org/pipermail/ovs-discuss/2022-May/051821.html
    [2]
    https://patchwork<email address hidden>/

    Partial-Bug: #1971431
    Change-Id: Ia041f640293ba26abf9f70af915817e9861e8ffc
    Signed-off-by: Lucas Alvares Gomes <email address hidden>

[Przemyslaw Hausman (phausman)]

Lukas, thanks for working on this. I'm trying to test the patches and everything's working fine until I run `openstack baremetal node provide <node>`. Baremetal node won't PXE boot and in the neutron-server.log I can see the following errors:

```
Attempting to bind port bb174dfc-5747-4868-b8ff-e339b6e54944 on host ac19c223-481c-4171-ba4d-470dca67b1db for vnic_type baremetal with profile {"local_link_information": [{}]} bind_port /usr/lib/python3/dist-packages/neutron/plugins/ml2/managers.py:810

Attempting to bind port bb174dfc-5747-4868-b8ff-e339b6e54944 by drivers ovn on host ac19c223-481c-4171-ba4d-470dca67b1db at level 0 using segments [{'id': 'ed2201f9-ebde-4754-8bba-ab4d16e82740', 'network_type': 'flat', 'physical_network': 'physnet2', 'segmentation_id': None, 'network_id': '5d8a2ae4-ab47-4aa6-99f9-d58c1a2d87d4'}] _bind_port_level /usr/lib/python3/dist-packages/neutron/plugins/ml2/managers.py:835

Refusing to bind port due to unsupported vnic_type: baremetal with no switchdev capability bind_port /usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/mech_driver.py:955

Failed to bind port bb174dfc-5747-4868-b8ff-e339b6e54944 on host ac19c223-481c-4171-ba4d-470dca67b1db for vnic_type baremetal using segments [{'id': 'ed2201f9-ebde-4754-8bba-ab4d16e82740', 'network_type': 'flat', 'physical_network': 'physnet2', 'segmentation_id': None, 'network_id': '5d8a2ae4-ab47-4aa6-99f9-d58c1a2d87d4'}]
```

Am I still missing some patches?

[Lucas Alvares Gomes (lucasagomes)]

Hi phausman,

Hmmm... So that error per-se doesn't seem to be a problem because Baremetal ports are created as external ports in OVN [0], so OVN itself should bind it to a node that contains the "ovn-cms-options=enable-chassis-as-gw" set in it.

Using [0] can you verify if it's the case ?

Another thing may be missing is the OVN code itself, you have to make sure that the version of OVN being used also includes this patch here [1], without it the iPXE chainloading won't work with the built-in OVN DHCP server.

[0] https://docs.openstack.org/neutron/latest/admin/ovn/external_ports.html#ovn-external-ports
[1] https://github.com/ovn-org/ovn/commit/0057cde2a64749bd2dbbaff525f7a1edccbd9c8a

[Przemyslaw Hausman (phausman)]

Lukas, thanks for the feedback.

I have three nodes and all of them are configured with `ovn-cms-options=enable-chassis-as-gw` and `ovn-bridge-mappings="physnet1:br-data,physnet2:br-ironic"`.

```
root@juju-bd5e43-4-lxd-5:~# ovn-sbctl list chassis
_uuid : 4445c088-d34e-4460-83c9-f3bc7fa8df2c
encaps : [26f71ac6-c433-431d-918d-343d0e9229de]
external_ids : {}
hostname : node07.maas
name : node07.maas
nb_cfg : 0
other_config : {ct-no-masked-label="true", datapath-type=system, iface-types="bareudp,erspan,geneve,gre,gtpu,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", mac-binding-timestamp="true", ovn-bridge-mappings="physnet1:br-data,physnet2:br-ironic", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw, ovn-enable-lflow-cache="true", ovn-limit-lflow-cache="", ovn-memlimit-lflow-cache-kb="", ovn-monitor-all="false", ovn-trim-limit-lflow-cache="", ovn-trim-timeout-ms="", ovn-trim-wmark-perc-lflow-cache="", port-up-notif="true"}
transport_zones : []
vtep_logical_switches: []

[...]
```

Somehow they are not added to the ha_chassis_group though. `ovn-nbctl list ha_chassis_group` reports empty `ha_chassis` list. That's one problem.

```
root@juju-bd5e43-4-lxd-5:~# ovn-nbctl list ha_chassis_group
_uuid : 2c29aee3-4bac-4c5f-ab04-ac34638bc8e1
external_ids : {"neutron:availability_zone_hints"=""}
ha_chassis : []
name : neutron-eb006711-9904-40b9-ab7c-82279e4c49b4
```

However, I can manually add the chassis to ha_chassis_group with `ovn-nbctl ha-chassis-group-add-chassis [...]`. And as soon as I do, the OVN-provided DHCP server starts responding to DHCP requests.

But then baremetal node fails to download ipxe.efi. It seems that the TFTP requests are being send towards DHCP server instead of ironic-conductor's TFTP server. So I checked the dhcp_options and realized that `next_server` option was missing. That's the second problem.

As soon as I manually added `next_server` with `ovn-nbctl dhcp-options-set-options <dhcp-option> [...] next_server="10.0.100.9"`, the baremetal node successfully downloaded ipxe.efi and booted just fine.

I'm running OVN 22.09 so I think it should have the patch you mentioned already applied.

```
# ovn-nbctl --version
ovn-nbctl 22.09.0
Open vSwitch Library 3.0.1
DB Schema 6.3.0
```

I'll do some more digging but if you have any other ideas, please share. Thanks!

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hi Lucas:

We have [1] and [2] merged. Do we need any other patch? If not, please move the status to "Fix Released".

Thanks!

[1]https://review.opendev.org/c/openstack/neutron/+/840287
[2]https://review.opendev.org/c/openstack/neutron/+/840316