ironic_deploy_ceph test failed after deployment on getting ironic-deploy-linux image

[Andrey Sledzinskiy]

iso - 8.0-566

Steps:
1. Create cluster
2. Add 1 node with controller+ceph-osd role
3. Add 2 nodes with controller+ironic+ceph-osd role
4. Add 1 node with compute role
5. Add 1 nodes with ironic role
6. Deploy the cluster
7. Upload image to glance - http://paste.openstack.org/show/486956/
8. Get 'ironic-deploy-linux' image from glance

Actual result - image is not present that caused test failure
http://paste.openstack.org/show/486957/

[Andrey Shestakov (ashestakov)]

To successful images upload rados should be available.
But as i see in logs, rados was not ready when task upload_images started.

puppet:
2016-02-15 06:08:00 +0000 /Stage[main]/Main/Service[radosgw]/ensure (notice): ensure changed 'stopped' to 'running'
glance:
<147>Feb 15 06:08:40 node-4 glance-api: 2016-02-15 06:08:40.911 180754 ERROR swiftclient [-] Container HEAD failed: http://10.109.54.3:8080/swift/v1/glance 503 Service Unavailable
last error in apache:
[Mon Feb 15 06:08:43.702711 2016] [proxy:error] [pid 64189:tid 139853966554880] AH00940: FCGI: disabled connection for (127.0.0.1)

So rados got available in 3 seconds later then upload task started.

Some validation should be added to task enable_rados.

[Timur Nurlygayanov (tnurlygayanov)]

Does the issue reproduced in 100 % of cases or it is a question of timeouts? If it is possible random fails we need to move it to MOS 8.0 MU and MOS 9.0, if it is reproducing in 100% of cases we need to fix it as soon as possible.

[Andrey Shestakov (ashestakov)]

We never got this situation. looks like very low chance

[Serge Kovaleff (serge-kovaleff)]

Should we close it as Cannot reproduce?

[Andrey Shestakov (ashestakov)]

Addressed by https://review.openstack.org/#/c/297633/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/303301
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=2ab7c95205af43aea1ee6a02eb51cc98d9c02953
Submitter: Jenkins
Branch: stable/mitaka

commit 2ab7c95205af43aea1ee6a02eb51cc98d9c02953
Author: Andrey Shestakov <email address hidden>
Date: Fri Mar 25 13:28:29 2016 +0200

    Add RadosGW readiness check

    After radosgw service start it requires some time to get operational state
    This change introduces check script which waits until radosgw start to serve
    http requests.

    Change-Id: Iaeb3f60c0da114de1fea3f0ec3aa69acd05b037a
    Closes-bug: #1545651
    (cherry picked from commit 3d159974341211ffa90045e9f0cc3b819f212f3e)

[Sofiia Andriichenko (sandriichenko)]

Verified

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 193
cat /etc/fuel_build_number:
 193
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6332.noarch
 fuel-misc-9.0.0-1.mos8288.noarch
 python-packetary-9.0.0-1.mos131.noarch
 fuel-openstack-metadata-9.0.0-1.mos8648.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8648.noarch
 python-fuelclient-9.0.0-1.mos306.noarch
 fuel-9.0.0-1.mos6332.noarch
 fuel-nailgun-9.0.0-1.mos8648.noarch
 rubygem-astute-9.0.0-1.mos738.noarch
 fuel-library9.0-9.0.0-1.mos8288.noarch
 fuel-agent-9.0.0-1.mos272.noarch
 fuel-ui-9.0.0-1.mos2658.noarch
 fuel-setup-9.0.0-1.mos6332.noarch
 nailgun-mcagents-9.0.0-1.mos738.noarch
 shotgun-9.0.0-1.mos87.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-bootstrap-cli-9.0.0-1.mos272.noarch
 fuel-migrate-9.0.0-1.mos8288.noarch
 fuelmenu-9.0.0-1.mos267.noarch
 fuel-notify-9.0.0-1.mos8288.noarch
 fuel-ostf-9.0.0-1.mos924.noarch
 fuel-mirror-9.0.0-1.mos131.noarch
 fuel-utils-9.0.0-1.mos8288.noarch

[Serge Kovaleff (serge-kovaleff)]

@Vitaly, do we need the bug fix in MOS 8.updates?