[10.0][swarm] TimeoutError: Node failed to become offline

[Dmitry Belyaninov]

There are some failed swarm test cases:

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.numa_cpu_pinning/147/testReport/(root)/reboot_cpu_pinning_compute/reboot_cpu_pinning_compute/

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.public_api/16/testReport/(root)/deploy_env_with_public_api/deploy_env_with_public_api/

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.ceph_ha_one_controller/159/testReport/(root)/check_ceph_partitions_after_reboot/check_ceph_partitions_after_reboot/

TimeoutError: Node u'slave-**' failed to become offline

[Dmitry Kaigarodеsev (dkaiharodsev)]

please describe how can ci-team help to solve this bug?

[Oleksiy Molchanov (omolchanov)]

@Dmitry, my bad, I thought the problem is related to getting node online.

@QA team, I have tried to diagnose a root cause using https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.numa_cpu_pinning/147/console test case.

04:00:29 - Node was rebooted using 'shutdown +1'
04:04:30 - Test failed with message 'Node failed to become offline'

Node had 3 minutes to go offline, but somehow it didn't manage. I think that it was related to high load, but not sure, because there are no logs related to this node in snapshot.

You can try to extend time we are waiting for node to become offline, also please don't use shutdown +1.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/428642

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/428642
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=18e4d59570c9334cbf32ccb6412102ef884871b0
Submitter: Jenkins
Branch: stable/newton

commit 18e4d59570c9334cbf32ccb6412102ef884871b0
Author: Dmitry Belyaninov <email address hidden>
Date: Fri Feb 3 08:45:12 2017 +0000

    Timeout changing for restart procedure

    We have few tests failed on call "warm_shutdown_nodes"
    with 4x60 timeout. But there are few rassed tests with
    10x60 timeout. So longer timeout should be used for
    all tests.

    Change-Id: I0c7e63b4e9156f35d6679a9dcdfb7b9249c6796a
    Partial-Bug: 1652000

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/430625

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/430625
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=185b824873d4fbc603c7a36a70b80e662340bf20
Submitter: Jenkins
Branch: master

commit 185b824873d4fbc603c7a36a70b80e662340bf20
Author: Dmitry Belyaninov <email address hidden>
Date: Fri Feb 3 08:45:12 2017 +0000

    Timeout changing for restart procedure

    We have few tests failed on call "warm_shutdown_nodes"
    with 4x60 timeout. But there are few rassed tests with
    10x60 timeout. So longer timeout should be used for
    all tests.

    Change-Id: I0c7e63b4e9156f35d6679a9dcdfb7b9249c6796a
    Partial-Bug: 1652000
    (cherry picked from commit 18e4d59570c9334cbf32ccb6412102ef884871b0)