Astute can refuse connection in any actions - Connection refused - connect(2)

[Ilya Bumarskov]

Fuel 10 build #1501

Scenario:
            1. Create cluster in Ha mode with 1 controller
            2. Add 1 node with controller role
            3. Add 1 node with compute role
            4. Verify network
            5. Deploy cluster
            6. Reset settings <- This step is failed
            7. Update net
            8. Re-deploy cluster
            9. Verify network
            10. Run OSTF

Actual result:
Task 'reset_environment' has incorrect status. error != ready, 'Method reset_environment. Connection refused - connect(2). Inspect Astute logs for the details'.

Astute logs:
2017-03-21 23:15:58 ERROR [17143] Error running RPC method reset_environment: Connection refused - connect(2), trace:
["/usr/share/ruby/net/http.rb:878:in `initialize'",
 "/usr/share/ruby/net/http.rb:878:in `open'",
 "/usr/share/ruby/net/http.rb:878:in `block in connect'",
 "/usr/share/ruby/timeout.rb:66:in `timeout'",
 "/usr/share/ruby/net/http.rb:877:in `connect'",
 "/usr/share/ruby/net/http.rb:862:in `do_start'",
 "/usr/share/ruby/net/http.rb:857:in `start'",
 "/usr/share/ruby/xmlrpc/client.rb:472:in `do_rpc'",
 "/usr/share/ruby/xmlrpc/client.rb:279:in `call2'",
 "/usr/share/ruby/xmlrpc/client.rb:260:in `call'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/cobbler.rb:47:in `token'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/cobbler.rb:116:in `sync'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/cobbler_manager.rb:249:in `sync'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/cobbler_manager.rb:124:in `ensure in edit_nodes'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/cobbler_manager.rb:124:in `edit_nodes'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/provision.rb:185:in `remove_nodes'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/orchestrator.rb:134:in `remove_nodes'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:181:in `remove_nodes'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/dispatcher.rb:198:in `reset_environment'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:172:in `dispatch_message'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:131:in `block in dispatch'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:in `each_with_index'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:128:in `dispatch'",
 "/usr/share/gems/gems/astute-10.0.0/lib/astute/server/server.rb:106:in `block in perform_main_job'"]

[Oleksiy Molchanov (omolchanov)]

Please attach diagnostic snapshot.

[Vladimir Khlyunev (vkhlyunev)]

Looks really similar - https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.thread_3/219/testReport/(root)/ceph_ha_restart/ceph_ha_restart/

[Nastya Urlapova (aurlapova)]

Since that issue hasn't workaround priority was increased to Critical.

[Dmitry Pyzhov (dpyzhov)]

We reverted this feature. So the bug should disappear.

[Vladimir Kuklin (vkuklin)]

The issue still remains in the test - we need to wait for nodes to become online after env destroy. They still exist as online in the database and mcollective may still timeout.

[Ilya Bumarskov (ibumarskov)]

Bug wasn't reproduced latest 9 runs.