ironic_deploy_ceph test failed after deployment on getting ironic-deploy-linux image

Bug #1545651 reported by Andrey Sledzinskiy on 2016-02-15
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Andrey Shestakov
Mitaka
High
Andrey Shestakov
Newton
High
Andrey Shestakov
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
High
Andrey Shestakov
8.0.x
High
Vitaly Sedelnik
9.x
High
Andrey Shestakov

Bug Description

iso - 8.0-566

Steps:
1. Create cluster
2. Add 1 node with controller+ceph-osd role
3. Add 2 nodes with controller+ironic+ceph-osd role
4. Add 1 node with compute role
5. Add 1 nodes with ironic role
6. Deploy the cluster
7. Upload image to glance - http://paste.openstack.org/show/486956/
8. Get 'ironic-deploy-linux' image from glance

Actual result - image is not present that caused test failure
http://paste.openstack.org/show/486957/

tags: added: area-qa
Changed in fuel:
status: New → Confirmed
Changed in fuel:
assignee: Kyrylo Romanenko (kromanenko) → MOS Ironic (mos-ironic)
Changed in fuel:
assignee: MOS Ironic (mos-ironic) → Andrey Shestakov (ashestakov)
Andrey Shestakov (ashestakov) wrote :

To successful images upload rados should be available.
But as i see in logs, rados was not ready when task upload_images started.

puppet:
2016-02-15 06:08:00 +0000 /Stage[main]/Main/Service[radosgw]/ensure (notice): ensure changed 'stopped' to 'running'
glance:
<147>Feb 15 06:08:40 node-4 glance-api: 2016-02-15 06:08:40.911 180754 ERROR swiftclient [-] Container HEAD failed: http://10.109.54.3:8080/swift/v1/glance 503 Service Unavailable
last error in apache:
[Mon Feb 15 06:08:43.702711 2016] [proxy:error] [pid 64189:tid 139853966554880] AH00940: FCGI: disabled connection for (127.0.0.1)

So rados got available in 3 seconds later then upload task started.

Some validation should be added to task enable_rados.

tags: removed: area-qa
affects: fuel → mos
Changed in mos:
milestone: 9.0 → none
no longer affects: fuel/8.0.x
tags: added: ironic

Does the issue reproduced in 100 % of cases or it is a question of timeouts? If it is possible random fails we need to move it to MOS 8.0 MU and MOS 9.0, if it is reproducing in 100% of cases we need to fix it as soon as possible.

Andrey Shestakov (ashestakov) wrote :

We never got this situation. looks like very low chance

Serge Kovaleff (serge-kovaleff) wrote :

Should we close it as Cannot reproduce?

tags: added: area-ironic
removed: ironic
Changed in fuel:
assignee: nobody → Andrey Shestakov (ashestakov)
importance: Undecided → High
milestone: none → 10.0
status: New → Fix Committed

Reviewed: https://review.openstack.org/303301
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=2ab7c95205af43aea1ee6a02eb51cc98d9c02953
Submitter: Jenkins
Branch: stable/mitaka

commit 2ab7c95205af43aea1ee6a02eb51cc98d9c02953
Author: Andrey Shestakov <email address hidden>
Date: Fri Mar 25 13:28:29 2016 +0200

    Add RadosGW readiness check

    After radosgw service start it requires some time to get operational state
    This change introduces check script which waits until radosgw start to serve
    http requests.

    Change-Id: Iaeb3f60c0da114de1fea3f0ec3aa69acd05b037a
    Closes-bug: #1545651
    (cherry picked from commit 3d159974341211ffa90045e9f0cc3b819f212f3e)

Verified

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 193
cat /etc/fuel_build_number:
 193
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6332.noarch
 fuel-misc-9.0.0-1.mos8288.noarch
 python-packetary-9.0.0-1.mos131.noarch
 fuel-openstack-metadata-9.0.0-1.mos8648.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8648.noarch
 python-fuelclient-9.0.0-1.mos306.noarch
 fuel-9.0.0-1.mos6332.noarch
 fuel-nailgun-9.0.0-1.mos8648.noarch
 rubygem-astute-9.0.0-1.mos738.noarch
 fuel-library9.0-9.0.0-1.mos8288.noarch
 fuel-agent-9.0.0-1.mos272.noarch
 fuel-ui-9.0.0-1.mos2658.noarch
 fuel-setup-9.0.0-1.mos6332.noarch
 nailgun-mcagents-9.0.0-1.mos738.noarch
 shotgun-9.0.0-1.mos87.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-bootstrap-cli-9.0.0-1.mos272.noarch
 fuel-migrate-9.0.0-1.mos8288.noarch
 fuelmenu-9.0.0-1.mos267.noarch
 fuel-notify-9.0.0-1.mos8288.noarch
 fuel-ostf-9.0.0-1.mos924.noarch
 fuel-mirror-9.0.0-1.mos131.noarch
 fuel-utils-9.0.0-1.mos8288.noarch

Serge Kovaleff (serge-kovaleff) wrote :

@Vitaly, do we need the bug fix in MOS 8.updates?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers