[all] too many NODE_FAILURES on periodic CI jobs

[Bhagyashri Shewale]

https://review.rdoproject.org/zuul/builds?result=NODE_FAILURE

We are getting below issue and failed to build node:

1. Detailed node error: Build of instance <instance-id> aborted: Failed to allocate the network(s), not rescheduling / "Failed to allocate the network(s), not rescheduling"
2. nodepool.exceptions.LaunchNetworkException: Unable to find public IP of server
3. No valid host found
4. Quota exceeded for ram

First one got resolve.

[Javier Peña (jpena-c)]

Looking at the nodepool logs, it all points to some error in the cloud provider side. I have opened a ticket on Vexxhost to track this.

[Javier Peña (jpena-c)]

The ticket was fixed by Vexxhost, mentioning issues in their routing infrastructure.

In addition to that, we had an additional issue with OVB jobs, where stacks were not cleaned up at the job end. This was eating up all resources in the tenant, causing more errors. https://review.rdoproject.org/r/32147 should have fixed that.

[Rabi Mishra (rabi)]

Looks like we're still seeing those. I just noticed them in https://review.opendev.org/c/openstack/tripleo-heat-templates/+/777294.

Not sure if it's the same set of issues or something new.