Node has to be powered off before putting in maintenace when clean fails

[Sai Sindhur Malleni]

When using ironic automated cleaning, if the node clean fails it is put into clean failed state and in maintenance. Ironic hence loses track of the power state, and reports power off even if the node is actually turned on. This causes very difficult to debug problems from an os-collect-config/os-net-config perspective, because the node that failed cleaning might still have an IP that the undercloud will use to provision new machines. So the "clean failed" will keep trying to get metadata from the undercloud and the new node that gets the same IP address doesn't get metadata/ gets it intermittently. This leads to failed deployments when machines are recycled from one workload to another.

[Dmitry Tantsur (divius)]

Hi! This bug cannot be fixed for two reasons:

1. If we power off on failed cleaning, we're losing the ability to debug it AND we can potentially brick the node, if it was doing something dangerous.

2. Not syncing power state during maintenance is one of the main features of maintenance.

Now, the whole situation may improve with the specific faults work https://review.openstack.org/#/c/334113/, but as it is we cannot really fix it.

[Jim Rollenhagen (jim-rollenhagen)]

What if we checked power state immediately when cleaning fails, so the database is up to date?

Or is this bug about getting out of sync if someone turns on the node while it's in maintenance?

[Sai Sindhur Malleni (smalleni)]

It would be helpful to alteast have the clast known power state show up before putting it in maintenance, instead of showing power off.

[Dmitry Tantsur (divius)]

I'd rather set power state to None on entering maintenance, and keep it this way, until the node gets out of it. Then we won't lie to users if any power actions are done to the node.

[Sai Sindhur Malleni (smalleni)]

dtantsur++ Can we triage this and set priority at your convenience please.

[Sam Betts (sambetts)]

This bug seems less about the power state, and more about the node holding onto an IP address in the flat network that it shouldn't have because the neutron port get's deleted so from neutron's perspective the IP is "free". Perhaps the solution should be that in the case of cleaning failed, ironic doesn't delete the neutron port for that node, keeping the IP assignment and preventing it from being assigned to a different server and then conflicting.