Error occured: connection closed by remote host after stopping deployment at the end of provisioning

[Anastasia Palkina]

"build_id": "2014-05-15_01-10-31",
"mirantis": "yes",
"build_number": "205",
"ostf_sha": "69bdb9729cbf16d2c8ba927e946d64a8898909af",
"nailgun_sha": "0c88805f28d658be82b305eaeafc5466b563569d",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "fc249cd1fe62d5303f531cf00d9e31ee2ab642d2",
"astute_sha": "2b19f9cda281a4e53fb4a2e241d882e29ab9724b",
"release": "5.0",
"fuellib_sha": "1c7d598a65309f8bda1391360f0b94bfb5015180"

1. Create new environment (CentOS, simple mode)
2. Choose VLAN segmentation
3. Choose Ceph for images
4. Add controller, compute, cinder and 2 ceph
5. Start deployment
6. Stop deployment at the end of provisioning of one node. Other node should be in status 'CentOS is installed' (see screen)
7. Deployment has failed
8. There are error in orchestrator logs:

[416] Error running RPC method stop_deploy_task: connection closed by remote host, trace: ["/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/transport/packet_stream.rb:87:in `next_packet'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:178:in `block in poll_message'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:173:in `loop'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:173:in `poll_message'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:460:in `dispatch_incoming_packets'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:221:in `preprocess'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/server.rb:204:in `preprocess'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/session.rb:449:in `block in preprocess'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/server_list.rb:47:in `block in each'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/server_list.rb:45:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/server_list.rb:45:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/session.rb:449:in `preprocess'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/session.rb:430:in `process'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/session.rb:417:in `block in loop'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/session.rb:417:in `loop'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi/session.rb:417:in `loop'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/ssh.rb:99:in `block (2 levels) in run_remote_command'", "/usr/lib64/ruby/2.1.0/timeout.rb:91:in `block in timeout'", "/usr/lib64/ruby/2.1.0/timeout.rb:35:in `block in catch'", "/usr/lib64/ruby/2.1.0/timeout.rb:35:in `catch'", "/usr/lib64/ruby/2.1.0/timeout.rb:35:in `catch'", "/usr/lib64/ruby/2.1.0/timeout.rb:106:in `timeout'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/ssh.rb:99:in `block in run_remote_command'", "/usr/lib64/ruby/gems/2.1.0/gems/net-ssh-multi-1.2.0/lib/net/ssh/multi.rb:62:in `start'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/ssh.rb:82:in `run_remote_command'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/ssh.rb:32:in `block in execute'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/ssh.rb:28:in `times'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/ssh.rb:28:in `execute'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:161:in `stop_provision'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:192:in `stop_current_task'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:158:in `stop_deploy_task'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:128:in `dispatch_message'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:89:in `block in dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:87:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:87:in `each_with_index'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:87:in `dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:82:in `block in perform_service_job'"]

I reproduced this bug twice.

[Vladimir Kuklin (vkuklin)]

This is related to the fact that we really can have problems with provisioning stop. Should be actually addressed by Ironic as provisioning engine along with IPMI drivers.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/96736

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96736
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=1fc4cf583700244adf3604bfb84decf7cbc5f487
Submitter: Jenkins
Branch: master

commit 1fc4cf583700244adf3604bfb84decf7cbc5f487
Author: Vladimir Sharshov <email address hidden>
Date: Fri May 30 15:54:40 2014 +0400

    Prevent fall with exception if remote node close connection

    Change-Id: I50a43c1b81d150ced64435f04173d22365d61192
    Closes-Bug: #1319883

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/113185

[Andrey Sledzinskiy (asledzinskiy)]

Bug is reproduced on {

    "build_id": "2014-08-08_00-31-17",
    "mirantis": "yes",
    "build_number": "168",
    "ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
    "nailgun_sha": "04ada3cd7ef14f6741a05fd5d6690260f9198095",
    "production": "docker",
    "api": "1.0",
    "fuelmain_sha": "2e8814b3e1823d02ebd2456d4653d60e829cc97a",
    "astute_sha": "6db5f5031b74e67b92fcac1f7998eaa296d68025",
    "release": "5.0.1",
    "fuellib_sha": "dec8421a18e3ccfa35978b36b5de3292b53e6b69"

}

[Vladimir Sharshov (vsharshov)]

We should merge https://review.openstack.org/96736 after 5.0.1 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladimir Sharshov (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/113185
Reason: For 5.0.2 we only provide changes for packages and manifests