[UI] Rescue Mode IP Addresses incorrect

[Jeff Lane ]

Trying to figure out why a newly deployed node didn't complete deployment (it failed the reboot). SO I thought I'd try rescue mode to examine the newly installed NVMe device.

I marked the node as Broken, then chose the Rescue action and waited for the node to reboot into the rescue mode.

Once MAAS told me that the node was in Rescue Mode (confirmed that it was booted into an ephemeral via attached monitor), I attempted to log in.

MAAS tells me through the Node Interfaces tab that it has three addresses ranging from 10.0.1.0 to 10.0.1.2.

HOWEVER, none of these work.

Digging through the maas.log, I can see that it actually gave out completely different IP addresses.

Rescue Mode does not tell me how to connect to the node once it's actually in rescue mode.

See the attached screen shot to see exactly what the MAAS UI shows for the IP addresses (Note, this shows all three interfaces configured for DHCP, for deployment I had ONLY configured enp2s0 and enp2s1 as "Auto Assign" not "DHCP" and likewise I had left eno1 UNCONFIGURED, not configured at all.

[Jeff Lane  (bladernr)]

The attached screen shot shows the MAAS UI reporting three completely different IP addresses that don't work, while the MAAS log instead shows two OTHER IP Addresses that DO work. Basically, the IP addresses below in the maas.log snippet are the IP addresses MAAS assigned when I set those two interfaces to "Auto Assign". When I entered Rescue Mode, MAAS UI tells me completely differetn IP addresses that do not work, while the 2.99 address DOES work.

2018-10-18T15:49:58.420531-04:00 critical-maas maas.node: [info] shilin: Status transition from READY to ALLOCATED
2018-10-18T15:49:58.435239-04:00 critical-maas maas.node: [info] shilin: allocated to user bladernr
2018-10-18T15:49:58.891890-04:00 critical-maas maas.interface: [info] Allocated automatic IP address 10.0.2.99 for enp2s0 (physical) on shilin.
2018-10-18T15:49:59.020529-04:00 critical-maas maas.interface: [info] Allocated automatic IP address 10.0.2.100 for enp3s0 (physical) on shilin.
2018-10-18T15:49:59.052759-04:00 critical-maas maas.node: [info] shilin: Status transition from ALLOCATED to DEPLOYING
2018-10-18T15:49:59.138464-04:00 critical-maas maas.power: [info] Changing power state (on) of node: shilin (aday6p)
2018-10-18T15:50:19.250680-04:00 critical-maas maas.power: [info] Changed power state (on) of node: shilin (aday6p)
2018-10-18T16:02:52.834245-04:00 critical-maas maas.node: [info] shilin: Releasing node
2018-10-18T16:02:52.837102-04:00 critical-maas maas.node: [info] shilin: Status transition from DEPLOYING to RELEASING
2018-10-18T16:02:52.865662-04:00 critical-maas maas.node: [info] shilin: Status transition from RELEASING to BROKEN
2018-10-18T16:02:52.971381-04:00 critical-maas maas.power: [info] Changing power state (off) of node: shilin (aday6p)
2018-10-18T16:03:01.703957-04:00 critical-maas maas.power: [info] Changed power state (off) of node: shilin (aday6p)
2018-10-18T16:03:06.517515-04:00 critical-maas maas.node: [info] shilin: Status transition from BROKEN to ENTERING_RESCUE_MODE
2018-10-18T16:03:06.593978-04:00 critical-maas maas.power: [info] Changing power state (cycle) of node: shilin (aday6p)
2018-10-18T16:03:06.595416-04:00 critical-maas maas.node: [info] shilin: Rescue mode starting
2018-10-18T16:03:27.178453-04:00 critical-maas maas.power: [info] Changed power state (on) of node: shilin (aday6p)
2018-10-18T16:04:48.985965-04:00 critical-maas maas.node: [info] shilin: Status transition from ENTERING_RESCUE_MODE to RESCUE_MODE

bladernr@critical-maas:~$ for x in 10.0.1.0 10.0.1.1 10.0.1.2 10.0.2.99 10.0.2.100; do
> ping -c 1 $x
> done
PING 10.0.1.0 (10.0.1.0) 56(84) bytes of data.
From 10.0.0.1 icmp_seq=1 Destination Host Unreachable

--- 10.0.1.0 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data.
From 10.0.0.1 icmp_seq=1 Destination Host Unreachable

--- 10.0.1.1 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PING 10.0.1.2 (10.0.1.2) 56(84) bytes of data.
From 10.0.0.1 icmp_seq=1 Destination Host Unreachable

--- 10.0.1.2 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

PIN...

[Andres Rodriguez (andreserl)]

A machine that failed to deploy still holds the IP addresses that were assigned to the machine. However, rescue mode and other ephemeral environments get network from the /dynamic/ range and not from the static network configuration because it could be the very same reason why it fails to deploy/configure/etc.

That said, I feel like this bug has already been previously filed. Either way, I have a couple quetsions:

1. As soon as the machine PXE booted of MAAS (before it even got to the OS), did the UI show the IP address the machine got at all ? If not, can you check?

2. The reason why I ask above is that MAAS can only know what IP address it has if, and only if, the DHCP notifies MAAS of the IP the machine has received. There's a bug in cloud-init-ramfs or related tools where they copy the config the initrams into the ephemeral environment and the machine won't renew the IP address, hence it gets automatically removed. So in your case, the UI may have been updated correctly, but you didn't see it because the lease expired and wasn't renewed, but the machine already owned that IP.

[Adam Collard (adam-collard)]

This bug has not seen any activity in the last 6 months, so it is being automatically closed.

If you are still experiencing this issue, please feel free to re-open.

MAAS Team