This was intermittent, and we don't see it anymore.
The root cause was that instances took a long time to power off, nova does a graceful shutdown and after a timeout it will force the instance to stop. An issue in the booted image is likely the reason the graceful shutdown took too long/failed. We don't know exactly what change, but a good guess is that some change in upstream CentOS improved the graceful shutdown behavior in out images.
Closing this issue.
This was intermittent, and we don't see it anymore.
The root cause was that instances took a long time to power off, nova does a graceful shutdown and after a timeout it will force the instance to stop. An issue in the booted image is likely the reason the graceful shutdown took too long/failed. We don't know exactly what change, but a good guess is that some change in upstream CentOS improved the graceful shutdown behavior in out images.