[SRU] iPXE doesn't handle NAK requests when multiple DHCP server's offer
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MAAS |
Invalid
|
Critical
|
Blake Rouse | ||
2.2 |
Invalid
|
Critical
|
Unassigned | ||
ipxe (Ubuntu) |
Fix Released
|
Critical
|
Christian Ehrhardt | ||
Xenial |
Fix Released
|
Critical
|
Unassigned | ||
Zesty |
Invalid
|
Critical
|
Unassigned | ||
Artful |
Fix Released
|
Undecided
|
Unassigned | ||
Bionic |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[Impact]
When there are multiple DHCP servers on the network, iPXE doesn't handle NAK's for the DHCP servers. This causes iPXE to get blocked without attempting to re-discover, hence, never obtaining an IP address.
For example, in a MAAS HA environment with a DHCP master/slave configuration, the machine fails to PXE boot because at a certain point, the DHCP server is not fully in sync, which causes iPXE to get a NAK request. This prevents the machine from PXE booting.
[Test case]
The easiest way:
1. Install MAAS with two rack controllers
2. Configure HA
3. PXE boot KVM's.
[Regression Potential]
Minimal. This only ensures that iPXE attempts to re-discover the network when it receives a NACK.
[Original bug report]
A VM failed to PXE boot after receiving multiple DHCP offers.
You can see this here on a log from the secondary controller:
http://
The node is offered both 10.245.208.201 and 10.245.208.120, tries to get 10.245.208.120, and is refused.
One strange thing is that it seems like the DHCP server on both the primary controller and the secondary controller are responding. The primary controller's log doesn't have the offer for 10.245.208.120 - only the offer for 10.245.208.201:
http://
This is in an HA setup: region API's are at 10.245.208.30, 10.245.208.31 and 10.245.208.32. We're using hacluster to load balance, and a VIP in front at 10.245.208.33. There are rack controllers on 10.245.208.30 and 10.245.208.31. For the untagged vlan this VM is trying to boot from, 10.245.208.30 is set as the primary controller, and 10.245.208.31 is set as the secondary.
Primary postgres is on 10.245.208.30, it's being replicated to backup postgres on 10.245.208.31. It has a VIP at 10.245.208.34.
We don't hit this everytime - on this deployment only one machine out of about 30 hit this.
We've also seen this on single node MAAS setups - non HA. So, it's not an HA specific issue.
I've attached logs from the maas servers.
tags: |
added: foundations-engine removed: foundation-engine |
Changed in maas: | |
milestone: | none → 2.3.0 |
Changed in maas: | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in maas: | |
milestone: | 2.3.0 → 2.3.0beta2 |
Changed in maas: | |
importance: | High → Critical |
tags: | added: cdo-release-blocker |
summary: |
- pod VM fails to PXE boot after receiving multiple DHCP offers from both - primary and secondary rack controllers, for different IPs + pod VM fails to PXE boot after receiving multiple DHCP offers, for + different IPs, from the dhcp server |
description: | updated |
tags: | added: internal |
Changed in maas: | |
milestone: | 2.3.0beta2 → 2.3.0beta3 |
Changed in ipxe (Ubuntu): | |
importance: | Undecided → Critical |
Changed in maas: | |
status: | Triaged → Incomplete |
Changed in maas: | |
assignee: | nobody → Blake Rouse (blake-rouse) |
Changed in maas: | |
status: | Incomplete → Invalid |
Changed in ipxe (Ubuntu): | |
assignee: | nobody → Andres Rodriguez (andreserl) |
tags: | added: patch |
Changed in ipxe (Ubuntu): | |
assignee: | Andres Rodriguez (andreserl) → ChristianEhrhardt (paelzer) |
Changed in ipxe (Ubuntu Bionic): | |
status: | New → Fix Released |
summary: |
- pod VM fails to PXE boot after receiving multiple DHCP offers, for - different IPs, from the dhcp server + [SRU] iPXE doesn't handle NAK requests when multiple DHCP server's offer |
description: | updated |
Changed in ipxe (Ubuntu Zesty): | |
importance: | Undecided → Critical |
Changed in ipxe (Ubuntu Xenial): | |
importance: | Undecided → Critical |
We need to investigate network setting (MTU, Switch config)