Process not running but not empty lockfile is present

[Aleksey Kasatkin]

Problems with deployment observed in logs provided by Joseph Mori in https://bugs.launchpad.net/fuel/+bug/1304469 is caused by the following reason. There is the same error everywhere in that logs that lead to fail of controller deployment:

2014-05-09T18:57:20 debug: [8859] 402da9e3-6a73-4014-a3d7-80ce8bab9c08: MC agent 'puppetd', method 'last_run_summary', results: {:sender=>"1", :statuscode=>0, :statusmsg=>"OK", :data=>{:status=>"stopped", :runtime=>8, :changes=>{"total"=>818}, :err_msg=>"Process not running but not empty lockfile is present. Trying to remove lockfile...ok.", :enabled=>1, :events=>{"failure"=>8, "total"=>826, "success"=>818}, :version=>{"config"=>1399660419, "puppet"=>"2.7.23"}, :idling=>0, :output=>"Currently stopped; last completed run 8 seconds ago", :stopped=>1, :time=>{"nova_paste_api_ini"=>0.060911, "cinder_config"=>0.063597, "glance_registry_config"=>0.04679, "sysctl"=>0.007922, "ring_container_device"=>5.166411, "service"=>119.635331, "cs_property"=>19.247938, "anchor"=>0.008439, "rabbitmq_user"=>2.576989, "file"=>1.283383, "file_line"=>0.001503, "rabbitmq_user_permissions"=>0.377321, "l3_if_downup"=>0.001907, "config_retrieval"=>37.280536, "package"=>633.97545, "l2_ovs_bridge"=>4.464839, "nova_config"=>0.147581, "cron"=>0.100862, "cs_shadow"=>10.925087, "l2_ovs_patch"=>1.433786, "last_run"=>1399661832, "ceilometer_config"=>0.30722, "glance_api_config"=>0.055452, "ring_object_device"=>3.242823, "firewall"=>16.83787, "neutron_dhcp_agent_config"=>0.054695, "rabbitmq_vhost"=>0.17639, "ring_devices"=>0.000317, "database"=>1.691244, "install_ssh_keys"=>0.011396, "cinder_api_paste_ini"=>0.027768, "l2_ovs_port"=>1.324343, "total"=>1008.071975, "cs_commit"=>8.578894, "database_grant"=>23.171662, "database_user"=>7.358681, "ring_account_device"=>3.249833, "cs_resource"=>18.846675, "heat_config"=>0.744471, "glance_cache_config"=>0.02279, "group"=>0.325504, "user"=>1.610422, "host"=>0.0134, "filebucket"=>0.000166, "exec"=>83.583636, "a2mod"=>0.000576, "notify"=>0.011234, "ceph_conf"=>0.01793}, :running=>0, :resources=>{"changed"=>795, "total"=>1465, "skipped"=>549, "scheduled"=>0, "failed_to_restart"=>1, "restarted"=>61, "failed"=>8, "out_of_sync"=>803}, :lastrun=>1399661832}}

err_msg=>"Process not running but not empty lockfile is present. Trying to remove lockfile...ok."

After 3rd try deployment is stopped.
This situation is repeated for about 10 times in that logs.

[Joseph Mori (jmori)]

So what is the fix for this as I cannot deploy any nodes.

Running Fuel 4.1

Thanks

[Aleksandr Didenko (adidenko)]

This message about lockfile is not a problem. Here is the real problem that breaks deployments (from puppet-apply.log):

2014-05-13T14:49:21.573368+00:00 err: (/Stage[main]/Nova::Utilities/Package[guestmount]/ensure) change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install guestmount' returned 100: Reading package lists...
2014-05-13T14:49:35.381004+00:00 err: (/Stage[main]/Cinder::Api/Package[cinder-api]/ensure) change from purged to latest failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install cinder-api' returned 100: Reading package lists...
2014-05-13T14:49:38.531630+00:00 err: (/Stage[main]/Nova::Scheduler/Nova::Generic_service[scheduler]/Package[nova-scheduler]/ensure) change from purged to latest failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install nova-scheduler' returned 100: Reading package lists...
2014-05-13T14:50:31.982913+00:00 err: (/Stage[main]/Keystone/Package[keystone]/ensure) change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install keystone' returned 100: Reading package lists...

And there are a lot of such errors for all nodes. Those errors mean you have some intermittent issues with Fuel admin network (the one where eth0 interfaces of your servers are connected to) - please check your switch and cables. You can also check logs on the Fuel node for any errors like "eth0: link is not ready".