[swarm] deployment failure: change from down to up failed: Timeout waiting for backend: 'object-storage' status to become: 'up' after 600 seconds

[Andrey Lavrentyev]

Detailed bug description:
Failure during cluster deployment for network template.
(/Stage[main]/Osnailyfacter::Ceph::Enable_rados/Osnailyfacter::Wait_for_backend[object-storage]/Haproxy_backend_status[object-storage]/ensure) change from down to up failed: Timeout waiting for backend: 'object-storage' status to become: 'up' after 600 seconds!

Swarm failure: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.network_templates/38/testReport/%28root%29/deploy_ceph_net_tmpl/

Steps to reproduce:
Execute 'deploy_ceph_net_tmpl' test with the following steps:
1. Revert snapshot with 5 slaves
2. Create cluster (HA) with Neutron VLAN/VXLAN/GRE
3. Add 3 controller + ceph nodes
4. Add 2 compute + ceph nodes
5. Upload 'ceph' network template
6. Create custom network groups basing on template endpoints assignments
7. Run network verification
8. Deploy cluster
9. Run network verification
10. Run health checks (OSTF)
11. Check L3 network configuration on slaves
12. Check that services are listening on their networks only

Expected results:
Deployment is a success

Actual result:
Deployment failure

Description of the environment:

9.1 snapshot #177

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuelmenu-9.0.0-1.mos275.noarch
 fuel-library9.0-9.0.0-1.mos8533.noarch
 fuel-nailgun-9.0.0-1.mos8768.noarch
 nailgun-mcagents-9.0.0-1.mos761.noarch
 fuel-notify-9.0.0-1.mos8533.noarch
 fuel-ui-9.0.0-1.mos2771.noarch
 shotgun-9.0.0-1.mos90.noarch
 python-fuelclient-9.0.0-1.mos337.noarch
 fuel-agent-9.0.0-1.mos288.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8768.noarch
 fuel-bootstrap-cli-9.0.0-1.mos288.noarch
 fuel-setup-9.0.0-1.mos6352.noarch
 network-checker-9.0.0-1.mos74.x86_64
 fuel-release-9.0.0-1.mos6352.noarch
 python-packetary-9.0.0-1.mos144.noarch
 fuel-misc-9.0.0-1.mos8533.noarch
 fuel-migrate-9.0.0-1.mos8533.noarch
 rubygem-astute-9.0.0-1.mos761.noarch
 fuel-mirror-9.0.0-1.mos144.noarch
 fuel-ostf-9.0.0-1.mos940.noarch
 fuel-utils-9.0.0-1.mos8533.noarch
 fuel-openstack-metadata-9.0.0-1.mos8768.noarch
 fuel-9.0.0-1.mos6352.noarch

UBUNTU_MIRROR_ID=ubuntu-2016-08-03-174238
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-08-24-172320
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-08-24-172320
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-07-18-162958
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

Logs: https://drive.google.com/open?id=0B5HPBFb7K7gXMi1mc2Zia1JERGc

[Kyrylo Galanov (kgalanov)]

There was radosGW failure with little logs: http://paste.openstack.org/show/565620/
Could you please provide en environment with broken service?

[Stanislaw Bogatkin (sbogatkin)]

I have manually checked all these failed tests for last several weeks and there is no such problem anymore. Closed as invalid.