PXE doesn't working for HP DL380 with broadcom cards

[Roman Sokolkov]

Description:
HP DL380 unable to boot by PXE. DHCP are fine.
But node uses internal docker IP as TFTP server (i.e. 172.17.0.2) which is of course unreachable.

Environment:
- MOS 5.1.1
- HP proliant DL380p Gen8 (System ROM P70 02/10/2014)
- NICs: Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe (Broadcom UNDI PXE-2.1 v16.4.1)

Steps to reproduce:
- Enable PXE boot on Broadcom card
- Try to boot by PXE

Expected result:
- Node will boot into bootstrap image

Actual result:
- PXE boot fails on TFTP. Please see screenshot.

Workaround:
As i understand these broadcom cards operate as GPXE.
Remove "pxe-service" option with gpxe from /etc/cobbler/dnsmasq.template
and restart dnsmasq "cobbler sync"

Details:
In Fuel dnsmasq configured with special options in case of NO GPXE.
dhcp-match=gpxe,175
pxe-service=net:#gpxe,x86PC,"Install",pxelinux,10.30.0.2
But in case of GPXE some default behaviour is used.
For some reason PXE ROM gets Server-ID (Option 54): 172.17.0.2 as TFTP server that is internal docker container address.

DHCP offer from dnsmasq (tcpdump): http://paste.openstack.org/show/157037/
Screenshot of PXE booting: http://s009.radikal.ru/i309/1501/5e/3901d8f80c2e.png

[Bogdan Dobrelya (bogdando)]

Looks like as a major issue with docker configuration, raising to high

[Bogdan Dobrelya (bogdando)]

@Matthew, could you please take a look into this issue? There is a w/a provided by Roman

[Matthew Mosesohn (raytrac3r)]

Roman, do you know of any internal hardware available to test a fix?

[Matthew Mosesohn (raytrac3r)]

The most obvious fix seems to be to replace pxe-service with the following two lines:
pxe-service=net:!gpxe,x86PC,"Install",pxelinux,10.20.0.3 <- replace # to !
pxe-service=net:gpxe,x86PC,"Install",gpxelinux.0,10.20.0.3 <- add gpxelinux.0 image

And it would be necessary to add gpxelinux.0 to tftproot.

But we could short circuit this possibly on the dhcrelay side by rewriting option 54 (Server IP field).

I should point out that in Fuel 6.1 we will move to host networking and remove all the Docker NAT issues and this issue should correct itself. For 6.0.x and 5.1.x we will probably need a solution like above.

[Roman Sokolkov (rsokolkov)]

Matt, i've never met such internal hardware, ask IT.

[Matthew Mosesohn (raytrac3r)]

Roman, I've got some hosts for testing from Fuel Devops team. It is Dell hardware, but has the same Broadcom hardware. I've added a rule to match gPXE and non-gPXE, but the card doesn't actually present gPXE capability. I'm using the following dnsmasq template in /etc/cobbler/dnsmasq.template: http://paste.openstack.org/

Can you possibly reproduce this again with that config file? Change 10.20.0.2 to 10.30.0.2 or whatever is the IP of your Fuel Master. The Server-ID (opt 54) cannot be rewritten or else dnsmasq is unable to hear the DHCP ACKs. Also, if you enable "log-dhcp" in dnsmasq config, you can see a little more detail.

[Roman Sokolkov (rsokolkov)]

Matt, please resend correct link with dnsmasq.template and i will check.

[Matthew Mosesohn (raytrac3r)]

We'll sync up on April 9 to attempt fixing now. I spoke with Gleb Galkin on this.

[Matthew Mosesohn (raytrac3r)]

The lab where we can test a fix is not available until maybe April 29

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/178221

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/178221
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=407802d2f75adda351b2e828875984e71d5ae2f0
Submitter: Jenkins
Branch: master

commit 407802d2f75adda351b2e828875984e71d5ae2f0
Author: Matthew Mosesohn <email address hidden>
Date: Tue Apr 28 17:45:53 2015 +0300

    Remove pxe-service flag in Cobbler and GPXE exclusion

    These options block Broadcom BCM5720 cards from PXE
    boot and are not necessary for provisioning.

    Change-Id: I4c6520bca8549fe96264f25827bc1e534669b860
    Closes-Bug: #1410280

[Vitaly Sedelnik (vsedelnik)]

New hardware support - Won't Fix for already released versions.