[system-test] In ceph_ha_restart test we wait all Nova services but 1 compute was destroyed

Bug #1461570 reported by Dmitry Tyzhnenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Dmitry Tyzhnenko

Bug Description

Failed on CI - http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.ubuntu.thread_3/146/consoleFull

Fuel ISO 6.1-497
{
    u'build_id': u'2015-06-02_16-28-25',
    u'build_number': u'497',
    u'auth_required': True,
    u'fuel-ostf_sha': u'f899e16c4ce9a60f94e7128ecde1324ea41d09d4',
    u'fuel-library_sha': u'd757cd41e4f8273d36ef85b8207e554e5422c5b0',
    u'nailgun_sha': u'3830bdcb28ec050eed399fe782cc3dd5fbf31bde',
    u'openstack_version': u'2014.2.2-6.1',
    u'production': u'docker',
    u'api': u'1.0',
    u'python-fuelclient_sha': u'4fc55db0265bbf39c369df398b9dc7d6469ba13b',
    u'astute_sha': u'cbae24e9904be2ff8d1d49c0c48d1bdc33574228',
    u'fuelmain_sha': u'bcc909ffc5dd5156ba54cae348b6a07c1b607b24',
    u'feature_groups': [u'mirantis'],
    u'release': u'6.1',
    u'release_versions': {u'2014.2.2-6.1': {u'VERSION': {
        u'build_id': u'2015-06-02_16-28-25',
        u'build_number': u'497',
        u'fuel-library_sha': u'd757cd41e4f8273d36ef85b8207e554e5422c5b0',
        u'nailgun_sha': u'3830bdcb28ec050eed399fe782cc3dd5fbf31bde',
        u'fuel-ostf_sha': u'f899e16c4ce9a60f94e7128ecde1324ea41d09d4',
        u'production': u'docker',
        u'api': u'1.0',
        u'python-fuelclient_sha': u'4fc55db0265bbf39c369df398b9dc7d6469ba13b',
        u'astute_sha': u'cbae24e9904be2ff8d1d49c0c48d1bdc33574228',
        u'fuelmain_sha': u'bcc909ffc5dd5156ba54cae348b6a07c1b607b24',
        u'feature_groups': [u'mirantis'],
        u'release': u'6.1',
        u'openstack_version': u'2014.2.2-6.1',
        }}},
}

        Scenario:
            1. Create cluster
            2. Add 3 nodes with controller and ceph OSD roles
            3. Add 1 node with ceph OSD roles
            4. Add 2 nodes with compute and ceph OSD roles
            5. Deploy the cluster
            6. Waiting up galera and cinder
            7. Check ceph status
            8. Run OSTF
            9. Destroy osd-node
            10. Check ceph status
            11. Run OSTF
            12. Destroy one compute node
            13. Check ceph status
            14. Run OSTF
            15. Cold restart
            16. Waiting up galera and cinder
            17. Run single OSTF - Create volume and attach it to instance
            18. Run OSTF

nova service list from OSTF logs - http://paste.openstack.org/show/259507/

Expected results:
    All OSTF tests after all tries should pass

Actual result:
   after step 15 OSTF failed

Tags: system-tests
Revision history for this message
Dmitry Tyzhnenko (dtyzhnenko) wrote :
description: updated
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Oleksiy Molchanov (omolchanov)
status: New → Confirmed
Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

This is just a compute node reporting as down, not controller. We aren't managing uptime of nova-compute on controllers. Marking as medium and 7.0. A user would need to restart nova services on the controller to restore this node's functionality.

Revision history for this message
Matthew Mosesohn (raytrac3r) wrote :

Sorry, nova services on compute node #6

Changed in fuel:
importance: High → Medium
milestone: 6.1 → 7.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/188071

Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Dmitry Tyzhnenko (dtyzhnenko)
status: Confirmed → In Progress
Changed in fuel:
milestone: 7.0 → 6.1
summary: - Nova services didn't start after cold restart nodes in environment
+ [system-test] In ceph_ha_restart test we wait all Nova services but 1
+ compute was destroyed
tags: added: non-release system-tests
tags: removed: non-release
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/188071
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=521931f98723ad07141cd4a53def695db1cba922
Submitter: Jenkins
Branch: master

commit 521931f98723ad07141cd4a53def695db1cba922
Author: Dmitry Tyzhnenko <email address hidden>
Date: Wed Jun 3 19:32:53 2015 +0300

    Add should_fail 1 in assert_os_services_ready

    Because we destroy a compute node we should ignore
        nova services test

    Change-Id: If8219ecfafb15dde9f107e4da3054621c5ca54c0
    Closes-bug: #1461570

Changed in fuel:
status: In Progress → Fix Committed
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.