[R 2.20] Some VMs stuck in error state when nova-conductor connection to RabbitMQ gets torn down intermittently

[amit surana]

3-5 out of 32 VMs get stuck in error state when launched simultaneously. Seems like nova-compute (on one of those compute nodes) to conductor communication is broken. Ranjeeth has looked at the setup.

Version 2.20 build 34.

Logs are here:

10.84.5.100:/cs-shared/bugs/1463070/

[Ranjeet R (rranjeet-n)]

I see there are RabbitMQ reconnects and because of which some of the VM spawns fail at different places. One way to solve this to retry VM spawns when it fails which would allow the SNAT testing to go through.

[Ranjeet R (rranjeet-n)]

After updating the system with the OSLO messaging patches from - https://github.com/openstack/oslo.messaging/commit/858353394dd15101ebcb1c4b39e672bc4aa2bd86 - I was able to launch 100 VMs without any issues. There were patching issues in the cluster which was fixed. Given that, I recommend to remove the blocker status in this bug.

[Ranjeet R (rranjeet-n)]

In scale, we are intermittent issues where nova-conductor connection to rabbitmq gets torn down and then reconnects back. We will keep this bug open to track the nova-conductor reconnect issue.

[Sunil Bakhru (sbakhru)]

Once Oslo patch is in, the main issue seems to be resolved. The issue of intermittent nova-conductor connection reconnect at scaled config is something we will monitor but doesn't gate the release.

[OpenContrail Admin (ci-admin-f)]

bug update...

[Ranjeet R (rranjeet-n)]

We have moved to a new model of Rabbitmq connectivity where it is not going through HA Proxy. This is not reproducible anymore.