master log not synchronized to remote log

[Rico Lin]

master log not synchronized to remote log
When error occurred in puppet which led to deploy failed.

1. astute log in master node will shows that:

2014-12-18 10:15:40 WARNING
[423] This nodes: [{"uid"=>"5", "role"=>"compute"}, {"uid"=>"6", "role"=>"ceph-osd"}, {"uid"=>"7", "role"=>"ceph-osd"}] will not deploy because at least one critical node deployment fail
2014-12-18 10:15:40 WARNING
[423] This nodes: [{"uid"=>"4", "role"=>"controller"}] will not deploy because at least one critical node deployment fail
2014-12-18 10:15:40 WARNING
[423] 91e1ee1a-b243-4e62-be50-84bcd8599915: Critical nodes with uids: 3 fail to deploy. Stop deployment
2014-12-18 10:15:40 INFO
[423] 91e1ee1a-b243-4e62-be50-84bcd8599915: Finished deployment of nodes => roles: [{"3"=>"primary-controller"}]
2014-12-18 10:15:40 INFO
[423] 91e1ee1a-b243-4e62-be50-84bcd8599915: Spent 63.44353457 seconds on puppet run for following nodes(uids): 3
2014-12-18 10:15:40 INFO
[423] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"91e1ee1a-b243-4e62-be50-84bcd8599915", "nodes"=>[{"uid"=>"3", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller"}]}}
2014-12-18 10:15:40 DEBUG
[423] Data send by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"3", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller"}]}
2014-12-18 10:15:40 DEBUG
[423] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"3", "status"=>"error", "error_type"=>"deploy", "role"=>"primary-controller"}]}
2014-12-18 10:15:40 DEBUG
[423] Node "3" has failed to deploy. There is no more retries for puppet run.
2014-12-18 10:15:40 DEBUG
[423] Nodes statuses: {"succeed"=>[], "error"=>["3"], "running"=>[]}
2014-12-18 10:15:40 DEBUG
[423] 91e1ee1a-b243-4e62-be50-84bcd8599915: MC agent 'puppetd', method 'last_run_summary', results: {:sender=>"3", :statuscode=>0, :statusmsg=>"OK", :data=>{:status=>"stopped", :resources=>{"out_of_sync"=>0, "total"=>0, "changed"=>0, "restarted"=>0, "failed"=>1}, :runtime=>1418897740, :lastrun=>0, :version=>nil, :err_msg=>"Process not running but not empty lockfile is present. Trying to remove lockfile...ok.", :enabled=>1, :output=>"Currently stopped; last completed run 1418897740 seconds ago", :time=>{"last_run"=>0}, :idling=>0, :running=>0, :changes=>nil, :stopped=>1, :events=>nil}}
2014-12-18 10:15:38 DEBUG
[423] Data received by DeploymentProxyReporter to report it up: {"nodes"=>[{"uid"=>"3", "progress"=>0, "status"=>"deploying", "role"=>"primary-controller"}]}
2014-12-18 10:15:38 DEBUG
[423] Got progress for nodes: [{"uid"=>"3", "progress"=>0}]
2014-12-18 10:15:38 DEBUG
[423] Nodes statuses: {"succeed"=>[], "error"=>[], "running"=>["3"]}

2. Cached puppet log for node 3 in master node did not shows any error message

3. When dive into node 3. Error message( puppet (err) ) is there at the button of puppet.log

Fuel need more synchronized log message
Make sure users really can find out what happened by monitoring logs in master node and report an more precision diagnostic snapshot to others.

[Mike Scherbakov (mihgen)]

Hi Rico,
thanks for bug report. Can you please attach your logs, or even diagnostic snapshot (better option). Thanks!

[Oleksiy Molchanov (omolchanov)]

Hi Rico,

Please attach diagnostic snapshot for further investigation.

Thanks in advance.

[Stanislaw Bogatkin (sbogatkin)]

This bug was incomplete for more than 4 weeks. We cannot investigate it further so we are setting the status to Invalid. If you think it is not correct, please feel free to provide requested information and reopen the bug, and we will look into it further.