Cluster deployment failed with the error: Failed to execute ironic_post_swift_key/2

[Sofiia Andriichenko]

log swift-object-server on contlroller
http://paste.openstack.org/show/494225/
from log "failed to connect to 192.168.1.4" , but 192.168.1.4 doesn't exist, 192.168.1.3 and 192.168.1.5 exists

Steps:
1. Deploy cluster with the following configuration:

1 Controller
1 Compute
1 Ironic.

Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation, DVR enabled.
Storage Backends - Cinder LVM over iSCSI for volumes.

2. Add 1 Ironic and 1 Controller+Ironic node.
3. Deploy changes.
4. Run OSTF. Run happy-path to boot Ironic VM instance.
5. Delete 1 Ironic and 1 Controller+Ironic node.
6. Deploy changes.

Actuall:
Changes deployment failed in 99% of deployment on controller
ERROR. Deployment has failed. All nodes are finished. Failed tasks: Task[ironic_post_swift_key/2] Stopping the deployment process!

Expected:
Deployment pass

[Sofiia Andriichenko (sandriichenko)]

https://drive.google.com/a/mirantis.com/file/d/0BxPLDs6wcpbDZDk0aHIxSUtBNjQ/view

[Sofiia Andriichenko (sandriichenko)]

 https://drive.google.com/a/mirantis.com/file/d/0BxPLDs6wcpbDZDk0aHIxSUtBNjQ/view?usp=sharing
Diagnostic Snapshot

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Andrey Shestakov (ashestakov)]

Looks like swift scale down is not works properly.
After controller deletion swift ring should rebuild.

[Alyona Kiseleva (akiselyova)]

According to swift logs, swift wasn't configured properly in redeployment process.
Nodes should be removed from swift-ring-builder, rings should be rebalanced and object.ring.gz should be sent to nodes after it. I see error with it in swift log, so, something from this process was missed. Something like

swift-ring-builder /etc/swift/account.builder remove <ip_address>/<device_name>
swift-ring-builder /etc/swift/container.builder remove <ip_address>/<device_name>
swift-ring-builder /etc/swift/object.builder remove <ip_address>/<device_name>

swift-ring-builder /etc/swift/account.builder rebalance
swift-ring-builder /etc/swift/container.builder rebalance
swift-ring-builder /etc/swift/object.builder rebalance

swift-ring-builder /etc/swift/account.builder
swift-ring-builder /etc/swift/container.builder
swift-ring-builder /etc/swift/object.builder

It's a deployment problem.

[Nikita Karpin (mkarpin)]

the workaround is https://bugs.launchpad.net/mos/+bug/1549293/comments/7