For grins, I decided to change the gateway IP for the provisioning subnet to 192.168.208.99. Now, when a machine is provisioned, we see the following in the flow table: root@lab-infra02:/home/jdenton# ovs-ofctl dump-flows br-int | grep 192.168.208 cookie=0xf36accc9, duration=40.502s, table=26, n_packets=0, n_bytes=0, idle_age=40, priority=100,udp,reg14=0x5,m etadata=0x1,dl_src=14:02:ec:32:3e:0c,nw_src=192.168.208.251,nw_dst=255.255.255.255,tp_src=68,tp_dst=67 actions=co ntroller(userdata=00.00.00.02.00.00.00.00.00.01.de.10.00.00.00.63.c0.a8.d0.fb.06.04.08.08.04.04.0f.14.63.6c.6f.75 .64.2e.61.72.63.61.6e.65.62.79.74.65.2e.63.6f.6d.33.04.00.00.a8.c0.1a.02.05.dc.01.04.ff.ff.fc.00.03.04.c0.a8.d0.6 3.36.04.c0.a8.d0.63,pause),resubmit(,27) cookie=0x88f3182e, duration=40.502s, table=27, n_packets=0, n_bytes=0, idle_age=40, priority=100,udp,reg0=0x8/0x 8,reg14=0x5,metadata=0x1,dl_src=14:02:ec:32:3e:0c,tp_src=68,tp_dst=67 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_D ST[],mod_dl_src:fa:16:3e:1f:ab:d3,mod_nw_src:192.168.208.99,mod_tp_src:67,mod_tp_dst:68,move:NXM_NX_REG14[]->NXM_ NX_REG15[],load:0x1->NXM_NX_REG10[0],resubmit(,37) Something to note is that while DHCP replies will be *sourced* as 192.168.208.99, the PXE client cannot actually communicate with that address, as shown here in these repeated ARP requests: 11:03:52.766526 14:02:ec:32:3e:0c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 405: vlan 6, p 0, ethertype IPv4 (0x0800), (tos 0x0, ttl 64, id 25461, offset 0, flags [none], proto UDP (17), length 387) 0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 14:02:ec:32:3e:0c, length 359, xid 0xd2d114eb, Flags [Broadcast] (0x8000) 11:03:52.767872 fa:16:3e:1f:ab:d3 > 14:02:ec:32:3e:0c, ethertype 802.1Q (0x8100), length 386: vlan 6, p 0, ethertype IPv4 (0x0800), (tos 0x0, ttl 64, id 25461, offset 0, flags [none], proto UDP (17), length 368) 192.168.208.99.67 > 255.255.255.255.68: [no cksum] BOOTP/DHCP, Reply, length 340, xid 0xd2d114eb, Flags [Broadcast] (0x8000) 11:03:52.775178 14:02:ec:32:3e:0c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 6, p 0, ethertype ARP (0x0806), Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.208.99 tell 192.168.208.251, length 46 11:03:53.263325 14:02:ec:32:3e:0c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 6, p 0, ethertype ARP (0x0806), Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.208.99 tell 192.168.208.251, length 46 11:03:53.763352 14:02:ec:32:3e:0c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 6, p 0, ethertype ARP (0x0806), Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.208.99 tell 192.168.208.251, length 46 It's not until I attach the provisioning subnet to a Neutron router do I start to see ARP replies: 11:04:04.766589 14:02:ec:32:3e:0c > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 64: vlan 6, p 0, ethertype ARP (0x0806), Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.208.99 tell 192.168.208.251, length 46 11:04:04.767108 fa:16:3e:e2:90:84 > 14:02:ec:32:3e:0c, ethertype 802.1Q (0x8100), length 64: vlan 6, p 0, ethertype ARP (0x0806), Ethernet (len 6), IPv4 (len 4), Reply 192.168.208.99 is-at fa:16:3e:e2:90:84, length 46 Why is it ARP'ing for that address, anyway? Well, once there is an ARP reply, the PXE client attempts to download via TFTP: 11:05:04.774315 14:02:ec:32:3e:0c > fa:16:3e:e2:90:84, ethertype 802.1Q (0x8100), length 84: vlan 6, p 0, ethertype IPv4 (0x0800), (tos 0x0, ttl 64, id 25510, offset 0, flags [none], proto UDP (17), length 66) 192.168.208.251.1978 > 192.168.208.99.69: [udp sum ok] TFTP, length 38, RRQ "ipxe.efi" octet tsize 0 blksize 1468 0x0000: 4500 0042 63a6 0000 4011 f454 c0a8 d0fb E..Bc...@..T.... 0x0010: c0a8 d063 07ba 0045 002e df6e 0001 6970 ...c...E...n..ip 0x0020: 7865 2e65 6669 006f 6374 6574 0074 7369 xe.efi.octet.tsi 0x0030: 7a65 0030 0062 6c6b 7369 7a65 0031 3436 ze.0.blksize.146 0x0040: 3800 Notice the destination is 192.168.208.99:69, the gateway IP (also server-identifier). I suspect the PXE client is relying on this address rather than option 66 due to the null terminated string issue mentioned earlier. Of course, there is no TFTP server at this address and the cycle loops forever. I saw this same behavior prior to changing the subnet gateway. The ARP request was answered by the physical firewall, since it owned the IP legitimately, and a pcap on the firewall showed attempts to hit 192.168.208.1:69. All of this is to say that while I understand the baremetal port will be scheduled to an OVN chassis/gateway node, I am not sure the expectation is that the provisioning network needs to be attached to a Neutron router that will ultimately need to provide connectivity to the Ironic API endpoint(s) for IPA to do its thing. Meaning, I don't think the server-id should be the gateway IP for the subnet, but should instead be a reserved DHCP port IP from the allocation pool[1]. Second, it seems there may be some compatibility issues with certain PXE implementations that will require the use of a null terminated string for option 66 and 67, and maybe it should be configurable. Or, perhaps PXE clients are expected to know how to handle the null terminated string (but not require it) while those that require it don't know how to handle its absence and it should be there 100% of the time[2]. FWIW, attempting to add the null character via the set command results in the following: ovn-nbctl set DHCP_Options e5132f1c-3314-4f3d-b6a1-137e5141190d options:tftp_server="192.168.208.22^@" 2023-02-16T11:58:57.945Z|00047|lflow|WARN|error parsing actions "reg0[3] = put_dhcp_opts(offerip = 192.168.208.220, bootfile_name = "http://192.168.208.22:8051/boot.ipxe", bootfile_name_alt = "ipxe.efi", dns_server = {8.8.4.4}, domain_name = "cloud.arcanebyte.com", lease_time = 43200, mtu = 1500, netmask = 255.255.252.0, router = 192.168.208.99, server_id = 192.168.208.99, tftp_server = 192.168.208.22^@, tftp_server_address = 192.168.208.22); next;": Invalid character `^' in input. -- [1] https://www.rfc-editor.org/rfc/rfc2131#section-2 - "A DHCP server always returns its own address in the 'server identifier' option." [2] https://www.rfc-editor.org/rfc/rfc2132#section-2 - "Options containing NVT ASCII data SHOULD NOT include a trailing NULL; however, the receiver of such options MUST be prepared to delete trailing nulls if they exist."