MCollective call failed in agent 'puppetsync'

[Serg Lystopad]

{"build_id": "2014-07-31_23-42-53", "mirantis": "yes", "build_number": "542", "ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f", "nailgun_sha": "9112643217a8e591da78e554c3d499b2835943a3", "production": "docker", "api": "1.0", "fuelmain_sha": "335cfd6401d17530ccd34d0e4a0c5161e821be11", "astute_sha": "6db5f5031b74e67b92fcac1f7998eaa296d68025", "release": "5.0.1", "fuellib_sha": "6e6bf6cba3dd8a105f7e7f52f8de68c5b4bc32fe"}

MOS: 5.0.1
stack: Ubuntu 12.04
mode: HA (3 controllers, 3 ceph, 3 mongo, 2 computes), ceilometer is turned on via UI

Deployment fails with:
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:02 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 41 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:00:02 warning: [575] Rsync problem. Try to repeat: 1 attempt
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:22 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 42 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:00:22 warning: [575] Rsync problem. Try to repeat: 2 attempt
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:00:43 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 42 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:00:43 warning: [575] Rsync problem. Try to repeat: 3 attempt
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:03 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 42 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:01:03 warning: [575] Rsync problem. Try to repeat: 4 attempt
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:24 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 42 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:01:24 warning: [575] Rsync problem. Try to repeat: 5 attempt
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:01:44 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 42 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:01:44 warning: [575] Rsync problem. Try to repeat: 6 attempt
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:05 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 38 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:02:05 warning: [575] Rsync problem. Try to repeat: 7 attempt
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:26 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 42 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:02:26 warning: [575] Rsync problem. Try to repeat: 8 attempt
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:02:46 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 31 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:02:46 warning: [575] Rsync problem. Try to repeat: 9 attempt
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"42", :statuscode=>1, :statusmsg=>"Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.\n Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)\nrsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]\n", :data=>{:msg=>nil}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"36", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"34", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"39", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"32", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"31", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"38", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"41", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"40", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"35", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 debug: [575] 0a93f725-19a5-45cd-8724-123b1ab61b15: MC agent 'puppetsync', method 'rsync', results: {:sender=>"33", :statuscode=>0, :statusmsg=>"OK", :data=>{:msg=>"Puppet modules and manifests was sync!"}}
2014-08-11T16:03:07 err: [575] MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 42 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

2014-08-11T16:03:07 err: [575] Unexpected error 0a93f725-19a5-45cd-8724-123b1ab61b15: MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 42 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

 traceback /usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/mclient.rb:116:in `check_results_with_retries'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/mclient.rb:62:in `method_missing'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deployment_engine.rb:138:in `sync_puppet_manifests'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deployment_engine.rb:52:in `block in deploy'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deployment_engine.rb:43:in `each_slice'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deployment_engine.rb:43:in `deploy'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:42:in `deploy'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:105:in `deploy'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:128:in `dispatch_message'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:91:in `block in dispatch'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `call'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `block in each'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:89:in `each_with_index'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:89:in `dispatch'
/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:77:in `block in perform_main_job'
2014-08-11T16:03:07 err: [575] Error running RPC method deploy: 0a93f725-19a5-45cd-8724-123b1ab61b15: MCollective call failed in agent 'puppetsync', method 'rsync', failed nodes:
ID: 42 - Reason: Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

, trace: ["/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/mclient.rb:116:in `check_results_with_retries'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/mclient.rb:62:in `method_missing'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deployment_engine.rb:138:in `sync_puppet_manifests'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deployment_engine.rb:52:in `block in deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deployment_engine.rb:43:in `each_slice'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/deployment_engine.rb:43:in `deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/orchestrator.rb:42:in `deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/dispatcher.rb:105:in `deploy'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:128:in `dispatch_message'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:91:in `block in dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `call'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:64:in `block in each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/task_queue.rb:56:in `each'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:89:in `each_with_index'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:89:in `dispatch'", "/usr/lib64/ruby/gems/2.1.0/gems/astute-0.0.2/lib/astute/server/server.rb:77:in `block in perform_main_job'"]
2014-08-11T16:03:07 info: [575] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"0a93f725-19a5-45cd-8724-123b1ab61b15", "status"=>"error", "error"=>"Error occurred while running method 'deploy'. Inspect Orchestrator logs for the details."}}

As I can see failed node change from repeat to repeat

[Serg Lystopad (slystopad)]

All nodes reply to ping from fuel master

# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online
---|--------|------------------|---------|------------|-------------------|------------|---------------|-------
39 | error | Untitled (c3:e0) | 7 | 10.20.0.20 | 38:ea:a7:35:c3:e0 | mongo | | True
33 | error | Untitled (c2:d4) | 7 | 10.20.0.12 | 38:ea:a7:35:c2:d4 | ceph-osd | | True
31 | error | Untitled (c3:78) | 7 | 10.20.0.10 | 38:ea:a7:35:c3:78 | ceph-osd | | True
34 | error | Untitled (c0:80) | 7 | 10.20.0.13 | 38:ea:a7:35:c0:80 | controller | | True
40 | error | Untitled (c2:4c) | 7 | 10.20.0.15 | 38:ea:a7:35:c2:4c | compute | | True
32 | error | Untitled (51:60) | 7 | 10.20.0.11 | 38:ea:a7:36:51:60 | ceph-osd | | True
38 | error | Untitled (a0:84) | 7 | 10.20.0.19 | 38:ea:a7:90:a0:84 | mongo | | True
42 | error | Untitled (bf:d0) | 7 | 10.20.0.17 | 38:ea:a7:35:bf:d0 | controller | | True
41 | error | Untitled (e9:c0) | 7 | 10.20.0.16 | 38:ea:a7:35:e9:c0 | compute | | True
36 | error | Untitled (c0:a8) | 7 | 10.20.0.14 | 38:ea:a7:35:c0:a8 | controller | | True
35 | error | Untitled (c2:fc) | 7 | 10.20.0.18 | 38:ea:a7:35:c2:fc | mongo | | True

[Serg Lystopad (slystopad)]

[root@fuel w]# mco rpc --agent puppetsync --action rsync -I /^[0-9]/
Discovering hosts using the mc method for 2 second(s) .... 11

 * [ ============================================================> ] 11 / 11

42 Request Aborted
   Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

Finished processing 11 / 11 hosts in 20528.52 ms
[root@fuel w]# ssh node-42
Warning: Permanently added 'node-42' (RSA) to the list of known hosts.
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.11.0-18-generic x86_64)

 * Documentation: https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@node-42:~# rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/
skipping non-regular file "firewall/spec/fixtures/modules/firewall/lib"
skipping non-regular file "firewall/spec/fixtures/modules/firewall/manifests"
skipping non-regular file "l23network/spec/fixtures/modules/l23network/lib"
skipping non-regular file "l23network/spec/fixtures/modules/l23network/manifests"
skipping non-regular file "l23network/spec/fixtures/modules/l23network/templates"
skipping non-regular file "nova/Gemfile"

[Serg Lystopad (slystopad)]

Below are 5 consecutive runs of puppetsync on all nodes:

[root@fuel w]# for i in $(seq 1 5); do mco rpc --agent puppetsync --action rsync -I /^[0-9]/; done
Discovering hosts using the mc method for 2 second(s) .... 11

 * [ ============================================================> ] 11 / 11

38 Request Aborted
   Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

Finished processing 11 / 11 hosts in 20567.41 ms
Discovering hosts using the mc method for 2 second(s) .... 11

 * [ ============================================================> ] 11 / 11

35 Request Aborted
   Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

Finished processing 11 / 11 hosts in 20523.61 ms
Discovering hosts using the mc method for 2 second(s) .... 11

 * [ ============================================================> ] 11 / 11

35 Request Aborted
   Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

Finished processing 11 / 11 hosts in 20525.56 ms
Discovering hosts using the mc method for 2 second(s) .... 11

 * [ ============================================================> ] 11 / 11

39 Request Aborted
   Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

Finished processing 11 / 11 hosts in 20550.84 ms
Discovering hosts using the mc method for 2 second(s) .... 11

 * [ ============================================================> ] 11 / 11

40 Request Aborted
   Fail to upload folder using command rsync -c -r --delete rsync://10.20.0.2:/puppet/modules/ /etc/puppet/modules/.
                       Exit code: 12, stderr: rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(764) [Receiver=3.0.9]

Finished processing 11 / 11 hosts in 20524.98 ms

[Vladimir Sharshov (vsharshov)]

Hi, Sergey!

It is look like reproducing https://bugs.launchpad.net/fuel/+bug/1322577

In fresh 5.0.1 this fix must be present: https://github.com/stackforge/fuel-library/commit/b27bd563eddc456b5e15f800c562f4a664e3fe92

Please provide a versions of main components: http://10.20.0.2:8000/api/version

[Vladimir Sharshov (vsharshov)]

Update: https://github.com/stackforge/fuel-library/commit/c0c0c135ef85bcf2c05f1b79dee65e61660e1f37#diff-d41d8cd98f00b204e9800998ecf8427e (correct link for patchset in 5.0)

[Serg Lystopad (slystopad)]

{u'build_id': u'2014-07-31_23-42-53', u'ostf_sha': u'09b6bccf7d476771ac859bb3c76c9ebec9da9e1f', u'build_number': u'542', u'api': u'1.0', u'nailgun_sha': u'9112643217a8e591da78e554c3d499b2835943a3', u'production': u'docker', u'mirantis': u'yes', u'fuelmain_sha': u'335cfd6401d17530ccd34d0e4a0c5161e821be11', u'astute_sha': u'6db5f5031b74e67b92fcac1f7998eaa296d68025', u'release': u'5.0.1', u'fuellib_sha': u'6e6bf6cba3dd8a105f7e7f52f8de68c5b4bc32fe'}

[Vladimir Sharshov (vsharshov)]

Let's try to decrease number of nodes which Astute try to manipulate in same time.

Steps:

- go to master node
- `dockerctl shell astute`
- `yum install -y vi`
- `vi /etc/astute/astuted.conf`
- add "max_nodes_per_call: 7"
- kill astute master to apply settings `ps -ef | grep '[a]stute master' | grep -v grep | awk '{print $2}' | xargs kill`
- `exit`

[Serg Lystopad (slystopad)]

Inside rsync docker container
I added to /etc/xinetd.d/rsync :
        cps = 512 10
        flags = IPv4
        per_source = UNLIMITED

service xinetd restart

Deployment is in progress . 4 nodes are already ready.
[root@fuel w]# fuel node | grep ready
39 | ready | Untitled (c3:e0) | 7 | 10.20.0.20 | 9a:e1:d2:9f:89:45 | mongo | | True
34 | ready | Untitled (c0:80) | 7 | 10.20.0.13 | 82:df:40:9c:3f:40 | controller | | True
38 | ready | Untitled (a0:84) | 7 | 10.20.0.19 | 2a:00:25:f5:45:44 | mongo | | True
35 | ready | Untitled (c2:fc) | 7 | 10.20.0.18 | ce:b1:c4:dd:9f:4e | mongo | | True

[Vladimir Sharshov (vsharshov)]

Sergiy, very interesting, thank you. It will be cool if you try to repeat problem and after it reproduced deployment with possible fix.

[Serg Lystopad (slystopad)]

All nodes were deployed successfully. Seems xinetd's reconfigurations helped.

Should I reconfigure astute as proposed in https://bugs.launchpad.net/fuel/+bug/1355281/comments/7 ?

[Vladimir Sharshov (vsharshov)]

I think no, because https://bugs.launchpad.net/fuel/+bug/1355281/comments/7 is a workaround and as i can see, you found solution.

[Vladimir Sharshov (vsharshov)]

sergiy (slystopad), thank you. We commit your solution for 5.0.1 and 5.1 version. You can look details here: https://bugs.launchpad.net/fuel/+bug/1322577