upload_cirros fails on multirack because image doesn't leave queued state because swift is unavailable

[Mikhail Samoylov]

Steps to reproduce:
1. Revert snapshot with 5 slaves
2. Create cluster (HA) with Neutron VLAN/VXLAN/GRE
3. Add 3 controller nodes
4. Add 2 compute + cinder nodes
5. Upload 'two_nodegroups' network template
6. Deploy cluster
7. Run health checks (OSTF)
8. Check L3 network configuration on slaves
9. Check that services are listening on their networks only

Expected result:
Test passed

Actual result:
Test failed on task "Task[upload_cirros/1]"
2016-04-07 22:52:55 ERR (/Stage[main]/Main/Exec[upload_cirros_shell]/returns) change from notrun to 0 failed: /bin/bash "/etc/puppet/shell_manifests/upload_cirros_command.sh" returned 1 instead of one of [0]

Snapshot:
https://drive.google.com/open?id=0B2SenDuhfXPlODNWbkJscHo3cU0

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.multiracks_2/1/console

Fuel version:
fuel-9.0-172-2016-04-07%5F14-00-00.iso

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

version

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Maksim Malchuk (mmalchuk)]

cirros upload failed due the swift problem again, the root cause:

2016-04-07 22:52:55.133 17299 ERROR swiftclient Traceback (most recent call last):
2016-04-07 22:52:55.133 17299 ERROR swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
2016-04-07 22:52:55.133 17299 ERROR swiftclient service_token=self.service_token, **kwargs)
2016-04-07 22:52:55.133 17299 ERROR swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 929, in head_container
2016-04-07 22:52:55.133 17299 ERROR swiftclient http_response_content=body)
2016-04-07 22:52:55.133 17299 ERROR swiftclient ClientException: Container HEAD failed: http://10.109.11.3:8080/v1/AUTH_96bf033387a34855b749ffcdbc017b23/glance 503 Service Unavailable

[Maksim Malchuk (mmalchuk)]

swift deployment failed due:

<42>Apr 7 22:23:48 node-1 swift-object-server: UNCAUGHT EXCEPTION#012Traceback (most recent call last):#012 File "/usr/bin/swift-object-replicator", line 34, in <module>#012 run_daemon(ObjectReplicator, conf_file, **options)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 110, in
 run_daemon#012 klass(conf).run(once=once, **kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/common/daemon.py", line 57, in run#012 self.run_forever(**kwargs)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 802, in run_forever#012 self.replicate()#012 File "/usr/l
ib/python2.7/dist-packages/swift/obj/replicator.py", line 691, in replicate#012 self.my_replication_ips = self._get_my_replication_ips()#012 File "/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 138, in _get_my_replication_ips#012 self.load_object_ring(policy)#012 File "/usr/lib/python2
.7/dist-packages/swift/obj/replicator.py", line 166, in load_object_ring#012 policy.load_ring(self.swift_dir)#012 File "/usr/lib/python2.7/dist-packages/swift/common/storage_policy.py", line 371, in load_ring#012 self.object_ring = Ring(swift_dir, ring_name=self.ring_name)#012 File "/usr/lib/python2.7/
dist-packages/swift/common/ring/ring.py", line 170, in __init__#012 self._reload(force=True)#012 File "/usr/lib/python2.7/dist-packages/swift/common/ring/ring.py", line 175, in _reload#012 ring_data = RingData.load(self.serialized_path)#012 File "/usr/lib/python2.7/dist-packages/swift/common/ring/ring.
py", line 87, in load#012 gz_file = GzipFile(filename, 'rb')#012 File "/usr/lib/python2.7/gzip.py", line 94, in __init__#012 fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')#012IOError: [Errno 2] No such file or directory: '/etc/swift/object.ring.gz'

[Maksim Malchuk (mmalchuk)]

file /etc/swift/object.ring.gz exists on node-1 it is looks like the swift deploy problem

[Alex Schultz (alex-schultz)]

Not a duplicate of #1566434, as that bug is an enhancement and may not fix this.

[Maksim Malchuk (mmalchuk)]

https://bugs.launchpad.net/fuel/+bug/1563981
should partially fix this bug

[Alex Schultz (alex-schultz)]

The swift health check is wrong for this environment. I believe this is being addressed as part of Bug 1548275 so this is a duplicate of that issue.