Env reseting hangs

[Sergey Galkin]

After couple resets next reset hangs without any messages.
Picture from attached screenshot not changing about 20 hours

Strage from logs
astute/astute.log

2016-02-19 15:16:30 DEBUG [1089] Cobbler syncing
2016-02-19 15:16:49 WARNING [1074] Cobbler problem. Try to repeat: 1 attempt
2016-02-19 15:17:00 WARNING [1089] Cobbler problem. Try to repeat: 1 attempt
2016-02-19 15:17:28 ERROR [1074] Error running RPC method reset_environment: end of file reached, trace:
["/usr/share/ruby/net/protocol.rb:153:in `read_nonblock'",
 "/usr/share/ruby/net/protocol.rb:153:in `rbuf_fill'",
 "/usr/share/ruby/net/protocol.rb:134:in `readuntil'",
 "/usr/share/ruby/net/protocol.rb:144:in `readline'",
 "/usr/share/ruby/net/http/response.rb:39:in `read_status_line'",
 "/usr/share/ruby/net/http/response.rb:28:in `read_new'",
 "/usr/share/ruby/net/http.rb:1406:in `block in transport_request'",
 "/usr/share/ruby/net/http.rb:1403:in `catch'",
 "/usr/share/ruby/net/http.rb:1403:in `transport_request'",
 "/usr/share/ruby/net/http.rb:1376:in `request'",
 "/usr/share/ruby/net/http.rb:1322:in `request_post'",
 "/usr/share/ruby/xmlrpc/client.rb:475:in `do_rpc'",
 "/usr/share/ruby/xmlrpc/client.rb:279:in `call2'",
 "/usr/share/ruby/xmlrpc/client.rb:260:in `call'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler.rb:89:in `item_exists'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler.rb:101:in `system_exists?'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler.rb:149:in `netboot'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler_manager.rb:131:in `block in netboot_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler_manager.rb:127:in `each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler_manager.rb:127:in `netboot_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/provision.rb:180:in `remove_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:149:in `remove_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:200:in `remove_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:217:in `reset_environment'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:189:in `dispatch_message'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `block in dispatch'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `each_with_index'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `dispatch'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:123:in `block in perform_main_job'"]

2016-02-19 15:17:28 INFO [1074] Casting message to Nailgun:
{"method"=>"reset_environment_resp",
 "args"=>
  {"task_uuid"=>"7e9be354-a184-49d1-9b1f-8acd60329dd2",
   "status"=>"error",
   "error"=>
    "Method reset_environment. end of file reached.\nInspect Astute logs for the details"}}

2016-02-19 15:17:38 INFO [1089] Changing node netboot state node-181
2016-02-19 15:17:38 DEBUG [1089] Cobbler syncing
2016-02-19 15:17:55 ERROR [1089] Error running RPC method reset_environment: end of file reached, trace:
["/usr/share/ruby/net/protocol.rb:153:in `read_nonblock'",
 "/usr/share/ruby/net/protocol.rb:153:in `rbuf_fill'",
 "/usr/share/ruby/net/protocol.rb:134:in `readuntil'",
 "/usr/share/ruby/net/protocol.rb:144:in `readline'",
 "/usr/share/ruby/net/http/response.rb:39:in `read_status_line'",
 "/usr/share/ruby/net/http/response.rb:28:in `read_new'",
 "/usr/share/ruby/net/http.rb:1406:in `block in transport_request'",
 "/usr/share/ruby/net/http.rb:1403:in `catch'",
 "/usr/share/ruby/net/http.rb:1403:in `transport_request'",
 "/usr/share/ruby/net/http.rb:1376:in `request'",
 "/usr/share/ruby/net/http.rb:1322:in `request_post'",
 "/usr/share/ruby/xmlrpc/client.rb:475:in `do_rpc'",
 "/usr/share/ruby/xmlrpc/client.rb:279:in `call2'",
 "/usr/share/ruby/xmlrpc/client.rb:260:in `call'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler.rb:89:in `item_exists'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler.rb:101:in `system_exists?'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler.rb:149:in `netboot'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler_manager.rb:131:in `block in netboot_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler_manager.rb:127:in `each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/cobbler_manager.rb:127:in `netboot_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/provision.rb:180:in `remove_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:149:in `remove_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:200:in `remove_nodes'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:217:in `reset_environment'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:189:in `dispatch_message'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `block in dispatch'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `each_with_index'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `dispatch'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:123:in `block in perform_main_job'"]

2016-02-19 15:17:55 INFO [1089] Casting message to Nailgun:
{"method"=>"reset_environment_resp",
 "args"=>
  {"task_uuid"=>"356b01bb-ce6a-4090-b729-d9bf2e0044ba",
   "status"=>"error",
   "error"=>
    "Method reset_environment. end of file reached.\nInspect Astute logs for the details"}}

nailgun/app.log
Fri Feb 19 16:39:29 2016 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /api/notifications?_=1455894755057 (ip 192.168.238.1) !!!
Fri Feb 19 16:39:29 2016 - uwsgi_response_write_body_do(): Broken pipe [core/writer.c line 331] during GET /api/notifications?_=1455894755057 (192.168.238.1)
IOError: write error

FUEL VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "573"
  build_id: "573"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "643a1ef27c7dccc1c2a2ad26b85c09226b35a67d"

[Sergey Galkin (sgalkin)]

Snaposhot
http://mos-scale-share.mirantis.com/fuel-snapshot-2016-02-20_09-34-28.tar.xz

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Dina Belova (dbelova)]

Need to be reproduced one again. Assigning to mos-scale

[Leontii Istomin (listomin)]

1. deploy 9.0-mos-401
2. curl -s 'https://review.openstack.org/gitweb?p=openstack/fuel-web.git;a=patch;h=e22577fded3a09c97983a5232326e09b715bd03d' | patch -b -d /usr/lib/python2.7/site-packages/ -p2
yum -y install python-uwsgidecorator
service receiverd restart && service nailgun restart
due https://bugs.launchpad.net/fuel/+bug/1569859
3. fuel-agent has been updated: http://paste.openstack.org/show/505660/ due https://bugs.launchpad.net/fuel/+bug/1543233
4. patch has been applied to keep rotated logs http://paste.openstack.org/show/495857/
5. curl -s 'https://review.openstack.org/gitweb?p=openstack/fuel-library.git;a=patch;h=36d3cfdfc05ce970d2101b62fcb49702e1bb1d5b' | patch -b -d /etc/puppet/modules -p3 due https://bugs.launchpad.net/mos/+bug/1566802
6.LMA, ElasticSearch, Grafana plugins have been installed:
yum -y install createrepo rpm rpm-build dpkg-devel git
easy_install pip
pip install fuel-plugin-builder
git clone https://github.com/openstack/fuel-plugin-lma-collector.git
fpb --check ./fuel-plugin-lma-collector
fpb --build ./fuel-plugin-lma-collector
fuel plugins --install ./fuel-plugin-lma-collector/*.noarch.rpm
git clone https://github.com/openstack/fuel-plugin-elasticsearch-kibana.git
fpb --check ./fuel-plugin-elasticsearch-kibana
fpb --build ./fuel-plugin-elasticsearch-kibana
fuel plugins --install ./fuel-plugin-elasticsearch-kibana/*.noarch.rpm
git clone https://github.com/openstack/fuel-plugin-influxdb-grafana.git
fpb --check ./fuel-plugin-influxdb-grafana
fpb --build ./fuel-plugin-influxdb-grafana
fuel plugins --install ./fuel-plugin-influxdb-grafana/*.noarch.rpm
curl -s 'http://paste.openstack.org/raw/505662/' | patch -b -d / -p1
fuel plugins --sync
7. try to deploy an env
deploy was failed
8. reset the env - SUCCESS