Nodes failed to be deleted with non informal message like {"1"=>#<Astute::Node uid="1", error="Node not answered by RPC.">}

[Tatyanka]

https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.command_line/23/testReport/

4 tests where deletion of nodes take place failed with the same non descriptive message

http://paste.openstack.org/show/476839/

Step to reproduce:
1. Create cluster - deploy it try to delete cluster over cli

2. Create any cluster - deploy it -> try to delete some node using cli

Result:
node deletion is failed

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "163"
  build_id: "163"
  fuel-nailgun_sha: "1e9f5dfdf811795637d726c3d7d959e63a25bca1"
  python-fuelclient_sha: "b619bc86bba311b7bced66783833fa0454d99e9b"
  fuel-agent_sha: "49e2e8a08a3cf39d1b38ee155015332aa84e47fd"
  fuel-nailgun-agent_sha: "08e0a11cf1f29b705e4b910d9b9db5e9b708b6e3"
  astute_sha: "a090546d43c770ac27ca81c6f8c78ff0ba4a93e0"
  fuel-library_sha: "a0044771e18b404de1346e2a995fd215dddb5806"
  fuel-ostf_sha: "17a093c05b972c2dd47f2c57775086d96caa5dfb"
  fuel-createmirror_sha: "fe81731b76071dc0a4ffdf039080732243f5a871"
  fuelmain_sha: "a2d78bdf7b37d8f2658c9f0f1f43708bdf85f8cd"

[Maciej Kwiek (maciej-iai)]

Could not reproduce, we need to check whether swarm test confirm it's existence again.

[Tatyanka (tatyana-leontovich)]

looks like we have reproduce of problem on the latest swarm run, so update report with snapshot

[Tatyanka (tatyana-leontovich)]

Look at the env - the issue is on Qa side, scenario is next - we revert prepared environment, and get only master node active, slaves failed to be started, so that marked as ofline, we do not check this in tests and try to delete env, as result env failed to be deletedfor 6 minutes(we try several times and wait rpc answer) and tests marked as fail. If we revert failed snapshot - waiting for additional several minutes - env successfully deleted

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/240555

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/240555
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=4509b95152f85f919e345ef7a088f6763a4f9b50
Submitter: Jenkins
Branch: master

commit 4509b95152f85f919e345ef7a088f6763a4f9b50
Author: Tatyana Leontovich <email address hidden>
Date: Fri Oct 30 16:41:13 2015 +0200

    Increase timeout for env deletion over cli

    In test delete cluster over cli we use
    revert mechanism, but do not check if all the nodes
    become online after revert. This lead to the situation
    when cluster deletion takes a little bit more time.
    So In this patch delete timeout was increased.
    Also add check for node statuses after revert
    and log with error level if some of them is offline.
    We don't interrupt tests here according to cluster should be deleted,even all the nodes are offline

    Change-Id: I769e5a17875595b632d111d6b5f1a782989e3a37
    Closes-Bug: #1508007

[Tatyanka (tatyana-leontovich)]

verified https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.command_line/34/